TCP Releases Over 4,000 New EEBO-TCP Texts

What the Public Release of ECCO-TCP Texts Means for You, Now and in the Future

Unfortunately, they didn’t make the texts easily accessible. To obtain them one had to apply by email to be subscribed to a Dropbox folder. There is a database and search interface, but it requires registration, and is unclear as to who qualifies for an account. I think that the database holds the marked up, XML texts, which have not (yet) been publicly released.

So I have created a package via the Open Knowledge Foundation’s Data Hub. You can download the zip package and an index in csv format from ckan. Note the zip bundle is around 142 mb. Don’t try this on dial-up. Check the index first. When I have time, I’ll work on a web interface that allows easy searching and sorting of it. I hope also that these texts will be made available individually, but given the number of them that’s not a trivial task.

What to do with these texts will be discussed tomorrow, Saturday 13th August, at Textcamp (London); the twitter hashtag is #tcamp11.

Update: xml (.tei) and epub versions are available from tei-Oxford.

One part of answering these questions is simply looking at what others are doing. So I’ve begun to compile a list of Digital Humanities projects where GIS (Geographical Information Systems) has a leading part. Aside from my own bookmarks, I’ve drawn on two similar lists: that at Historical GIS Research Network and the AAG Historical GIS Clearing House. It is a list of academic projects: although there are many excellent extra-mural mapping projects I specifically wanted to see how the digital and the humanities are combining in the university. It is also heavily weighted towards history and literary studies, as those are what I am involved in and know about. Please tell me of any other projects through the comments.

I’ve used GIS in a rather loose way, taking in what has been termed ‘neogeography’ and ‘webmapping.’ A couple of the projects I’ve listed don’t even aim to produce maps, but gazetteers of old place names, and utilize text processing technologies rather than anything that could be considered GIS. Part of this exercise is to see how space and place are being analysed, and what technologies are being used to do so; GIS seemed a useful catch-all term. I hope the purists will forgive me.

This list takes a snapshot of the state of the ‘spatial turn’ in (some of) the (digital) humanities up to early 2011. The technologies used fall into four types: flash animations, Google Maps, server-side delivery and old-style downloadable shapefiles. The focus is frequently based on geographical units – cities, regions, countries, continents – and less often on particular subjects. Suprisingly, there’s only one project on the Holocaust and that barely begun; I heard of two other projects, but both seem to be defunct. Further analysis will follow as time allows.

Thanks to all who responded to my query on the Humanist list; the relevant postings can be found in the March 2011 archives.

Cohen’s specific historical object is the Victorian ‘frame of mind.’ How did they think, how did they see the world, and how did they believe? His method is to use Google’s vast digitization program to read the Victorians, or at least those who were published, en masse, rather than rely on a canon of notable authors. The move from the anecdotal and elite selection of Houghton’s The Victorian Frame of Mind, 1830-1870, to a truly comprehensive survey of all Victorian authors, will hopefully give a broader, more accurate and more subtle view of Victorian modes of thought, and perhaps a more open one that allows for discordance and diversity.

This isn’t a simple matter of chucking a load of material into a database, pushing a button and then having the computer throw out unambiguous facts and truths. Cohen and Gibbs have posted some caveats: the data isn’t perfect, meaning of words change over time, as yet only the titles of books are being mined, no collocation or context is given. It also requires some careful methodology, and weighing for all sorts of extraneous factors: William Briggs has done some very interesting analysis bringing in population statistics. But with freely available data, anyone with a spreadsheet program can try out ideas and run checks, allowing for the collaborative development of analytical techniques.

Percentage of British books with 'revolution' in the title, 1789-1914

Of the words Cohen and Gibbs have chosen, one stands out as being more temporal than the others: revolution. None of the other terms is so event-related, or has a specific chronological location. Many are abstract, like ‘God’ or ‘honour’; some are names (‘Aristotle, ‘Jesus’, ‘Plato’ and ‘Socrates’); and there’s one place, Rome. That does not mean that there is no relation between these words and contemporary events – Rome has a startling peak in 1851, possibly related to the French occupation in the aftermath of 1848. Nor does revolution refer only to moments of uprising; it can equally mean the movement of the planets and the development of industry (Google’s ngram machine has the latter taking off in the 1880s). But it is the only chosen term that has a specific chronological collorary. Although the project is oriented around more long-term and subtle concerns, the changes in Victorian mentalities, I began to wonder how much the data reflected more immediate responses to human affairs.

Unsurprisingly, in the case of revolution, we have a mass of titles registering in the 1790s, and a very sharp peak in 1848. There are two other clear spikes in 1817 and 1830/1. A little bit of scrutiny, and you’ll see that 1871, the year of the Paris Commune, shows a marked increase. From prior knowledge of revolutions and threats of them, we can validate the data as reflecting events. As yet the statistics are not telling us anything new. There are some differences if one visualizes the data as the number of publications rather than percentages. 1830-1 and 1848 still stand out, 1817 and the Paris Commune less so. There also seems to be a different distribution: the last few decades have far more occurrences more evenly distributed than the first half of the century.

Graph of no. of English books published 1789 - 1914 with 'Revolution' in the title

Although it is important to check the data against what is already known, one must guard against presumptions of correlation. Can we be sure we know what revolution is being reflected? 1848 saw revolutions throughout Europe, but were the titles referring to all of them, a subset, or even just the domestic radicalism of the Chartists? Similarly, Cohen considers the 1830 spike to point to “the successful 1830 revolution in France”; but given the figures for 1831, it could be a result of the turmoil preceding the reform act of 1832. Merthyr Tydfil saw perhaps the first industrial working class uprising in Britain; Bristol and Nottingham saw state institutions go up in flames; there were incidents across the country, from Exeter to Huddersfield. The British publishing trade may have taken more note of this than three glorious days in Paris: the small rise around 1871 may also indicate that British publishing would register domestic concerns far more dramatically than events abroad. Against this, the jump in 1857 is probably due to the Indian Mutiny. In turn, the 1831 figures could indicate that the situation in Britain was far more volatile than todays historians have judged it.

So although there is evidence of a causal relationship between events and book titles, it is not transparent. It is further clouded by changes in the meaning of the word. The sustained increase over the last 25 years suggests a change in the conception of revolution from taking to the streets to building working class organizations, from riot and insurgency to factory strikes and the new unionism, from an immediate event to a longer term social struggle.  This indicates a fundamental change in class structure – the growth of an industrial proletariat – and consistent class antagonism. But note that events still affect the numbers: the increase from 1904 to 1905 is probably due to the first Russian revolution.

The greater concern with domestic events and the change in meaning of the word ‘revolution’ are working hypotheses. Hopefully, the full corpus from which the numbers are drawn will be opened up, allowing these to be checked. I’d also like to investigate the 1853 spike, after the defeat of the Chartists and with no foreign correlate that I can think of.

Finally, a curious absence, and a warning against presuming an easy reflection of reality in words. The following graph is of the occurrence of the word ‘money’ in book titles, expressed as a percentage.

Percentage of British books with 'money' in the title, 1789-1914

See that dip for 1825? Yet there was a banking crisis that year!

* Insert standard complaint about proprietary file formats here. However, it’s a simple spreadsheet, and neither Open Office nor Libre Office had any difficulties opening it.

It’s easy to play. Just go to http://ngrams.googlelabs.com/, put in some words or phrases, separating multiples with commas, adjust the settings as you wish, and press the button. Up comes a graph showing distribution across your chosen time period. Thrill to the peaks! Gasp at the troughs! Wonder at the abundances! Curse the absences!

Like all good games, there’s much wisdom to be gained from playing. For one thing, it tests the sources, both the original works and their translation into digital format. (Some of Google’s metadata is bizarrely inaccurate.) It makes one think about possible reasons: whatever the results of a query are, they never explain why. It questions the technologies of language, whether printed or digital, including orthography, typeface and grammar. (Note that it only covers  written language, not that spoken or sung, with rhythm, accents and inflections.) In all, it indicates the ambiguity and instability of language, as against any needs or claims of clarity and transparency of meaning.

Below are four jests: testing a grandiose claim, a revealing anachronism, typographical obscenity and the deleterious effects of popular culture on the Queen’s English.

Benjamin was right!

By searching on four major Western cities, we can see that Walter Benjamin was right to consider Paris ‘the capital of the nineteenth century‘ [pdf].

Google Ngram for Paris, London, Berlin, New York, 1800-1900

Link

And never mind population, trade, dominions and suchlike. But note that had I included ‘Rome’, Benjamin would have been refuted by the number of works on ancient history.

Surrealism: A Victorian Creation?

Although the word ‘surrealism’ was originally coined by Apollinaire in 1917, and given substance in the 1920s by Andre Breton, Google finds it in mid-Victorian English:

Google Ngram for 'surrealism.'

Link

There’s an interesting error behind this. In parsing The London Review, volume 15, 1860-1, the last hypenated word of one page, and the first of the next, the chapter heading rather than the continuation of the text proper, have been run together by Google’s OCR software. Such bugs, the ‘revealing errors‘  of logic, can be considered a manifestation of the surrealist spirit.

FFS!

Early modern printing is notorious for using ‘f’ in place of ‘s.’ Oh the comedic potential, as is born out by this graph:

Google Ngram for 'fuck.'

Link

The alternative explanation is, of course, that we were a foulmouthed bunch until the Victorians, and only since the 1950s have we begun to throw off those moral shackles.

Star Trek and the split infinitive

That most controversial of grammatical issues had a historical turn in the 1980s, when “to boldly go” overtook “to go boldly.” A way of interrogating the corpus for the frequency of split infinitives isn’t obvious, but the results would be very interesting.

Google Ngram: 'to go boldly' and 'to boldly go.'

Link. Wikipedia on Split Infinitives

Google have provided some basic, but literate, documentation. And the datasets are freely available under a creative commons license. A Guardian article serves as a decent introduction, although it exaggerates the originality of the techniques. See also the New York Times. And don’t ever, ever, use the pseudo-word ‘culturomics.’

As part of this article – which may still come – I was going to raise the question of how the internet could revitalize critical history. History Workshop was as important for its social role as for publishing papers and theses. Whether it be the news, the less-formal articles on historical passions or the correspondence in the journal, or the meetings and attendant socializing, it brought people together. What chance that today critical historians can use modern technologies to once again find each other?

The good news is that History Workshop is now establishing a proper online presence, with a website (powered, thanks to WordPress, by free software) due to launch fully in January 2011. A Call For Papers has been issued, stating:

The History Workshop Journal editorial collective is launching History Workshop Online, a website devoted to the practice of politically-engaged history. Affiliated to the journal but entirely separate in its content, the site will serve as a forum, laboratory, and virtual coffeehouse for anyone interested in connecting historical exploration with the politics of the present, whether through engagement with public history, social history, the history of sexuality, or intimate histories of everyday life. In the spirit of the original history workshop movement, we’re keen to explore the diverse (and now multi-media) ways in which progressive history is being “done”, in and out of universities and the museum and heritage sector.

For the launch of the website in January 2011, we welcome all pertinent contributions: reports on public history initiatives; multimedia essays and articles; flagged events for our noticeboard; fulminations, rants, and raves.
For further information please contact the site’s editor, Marybeth Hamilton, at marybeth@historyworkshop.org.uk

The crucial phrase: “in the spirit of the original history workshop movement”, as befits this web that we weave.

The highlight of the Professional Reflection strand was Claire Ross‘ Pointless Babble or Enabled Backchannel, a witty and zippy analysis of twitter usage during three Digital Humanities conferences in 2009. Far more than 140 characters, without any excess and plenty of time for questions.

The History strand saw two very good presentations. And one that had me gawping in disbelief. Roorda’s Letters, Ideas and Information Technology, on visualizing seventeenth century correspondence, and Sainte’s Reading Darwin Between The Lines, analysing Darwin’s rare use of the term ‘evolution’, were very fine. But Blaney’s Developing a Collaborative Online Environment for History – The Experience of British History Online was a trip into the digital netherworld.

What British History Online wanted to do was crowdsource the Calendars of State Papers, those abstracts of government paperwork compiled in Victorian Times and now showing their age. So what do they do? Raise obstacles to participation. First, the CSP are behind a paywall, and as far as I can tell, there are no institutional subscriptions available. So the academics they hoped would annotate the documents had to pay for the honour. Then, to minimise contributions either malicious or erroneous, they deliberately put in obstacles and constraints to make annotation difficult. *rollseyes* Do they have any idea what crowdsourcing is?

Contributions were, unsurprisingly, sparse.

One of the audience asked about re-use. We were informed that the XML was locked up, the documents copyrighted (even though much of the material on BHO has long since passed into the public domain), but generously, we can print off as many copies as we wish. This was the only time I heard such sentiments expressed at DH2010; everyone else understood the importance of openness, of re-use, of contributing corrections and improvements, of sharing. It’s called community. And if you look at the graphic below, you’ll see it’s one of the prominent words (used 25 times) in the closing address from Melissa Terras, Present, Not Voting.

(Click to view full size)

‘Transcribe’ and ‘Bentham’ also feature as this is a crowdsourcing project Terras is involved in. As she says:

one of the things we want to do with Transcribe Bentham is to provide access to the resulting XML files so that others can reuse the information (via web-services, etc). The hosting and transcription environment we are developing will be open source, so that others can use it. And this sea change, from working in small groups, to really reaching out to users is something we have to embrace, and learn to work with.

The prospect of easily setting up such collaborations is mouthwatering. Access, re-use, reaching out, yes yes yes. Sharing is fundamental to what we do, and we are stronger when we share. And right now the Digital Humanities community – like everyone else – faces terrible pressure, from government and university management, and needs to get stuck in:

We need people who are not just prepared to whine but prepared to roll up their sleeves and do things to improve our associations, our community, and our presence in academia.

Her whole speech was barnstorming, critical but not despondent, electrifying the audience, and the highlight of a conference that, for all the heat and rushing around and getting up way too early, truly inspired me.

For me, the important presentation in the Teaching/Managing strand was Nowviskie and Porter’s “The Graceful Degradation Survey: Managing Digital Humanities Projects Through Times of Transition and Decline.” The afterlife of digital projects – and websites in general – is not only very important, but quite neglected, seemingly being done on an ad-hoc, voluntary basis. It was more to do with project management, organization and funding; I had hoped to hear something about technical solutions. It did suggest that there is a move to creating smaller, more preservable packets of information: a granular approach insuring against complete meltdown.

Another suggestion was that Digihum projects are increasingly being operated outside the academy. There’s a subterranean current here at DH2010 of extra-academic projects, ‘fragile vessels’ (as mentioned yesterday), small unfunded projects. One of those – a graduate project now continuing independently   – is contextus, which featured in the Scanning Between the Lines: The Search for the Semantic Story panel in the afternoon. Aside from being a very clear and useful introduction to RDFa (foaf etc), and being sprinkled with Doctor Who references, the speakers showed the great potential of the ‘semantic web’, about which I’d previously been a bit doubtful.

Many of the posters displayed, as on day two, were also for small, semi-independent or semi-official projects, using whatever tools are available free (in the financial sense). Somehow, this aspect of the Digital Humanities isn’t getting the full recognition it deserves. The lack of money shouldn’t mean abandoning a good or interesting idea, nor should it be considered a denial of permission to do what we want to do. It’s an obstacle, yes, but not insurmountable. Ways of operating on a shoestring need to be shared. And there is the advantage that without funds, one isn’t beholden to funders.

The first day of the conference proper (there’s been many associated events in the last few days) was mainly dealing with organization, with only a few events. I missed the second day of THATCamp London, twitter proving more frustrating than informative as it just made me want to be there more than ever, but managed to catch Dan Cohen afterwards for my first interview.

The only event I attended, was the launch of the CHARM (Centre for the History and Analysis of Recorded Music) sound files. These are digitisations of out-of-copyright, lesser known, 20s and 30s 78 rpm records, and are freely downloadable. Hallelujah for free, because there’s some gems to be discovered. Check out Mischa Spoliansky’s excellent, jaunty version of Gershwin’s Rhapsody in Blue (seemingly no static URLs, but the search interface is easy to use). And thank you to CHARM for not locking the music up: both the speakers spoke with an enthusiasm they wanted to share. Got interviews with them too.

Duties meant I missed the opening ceremony – which also featured CHARM – but had a snigger at the tweets about paleography provoked by the words of Kings’ lamentable principal.

Serious seminars start tomorrow. Perhaps serious blog posts too.

Notwithstanding his protestations to be speaking “as dispassionately and factually as I can”, it was a partisan and aggressive statement for the so-called “creative industries.” The usual suspects were lined up: the BBC, file sharers, the public sector, search engines, digital utopians et al. The notorious Tera report was cited, apocalyptic visions of redundancies painted (how ironic coming from Wapping), government enforcement of “basic property rights” demanded, Sky iPhone apps and Fox films promoted.

(It was also semi-literate: “almost exactly”; “the era of Pope, and Johnson, and writers after them.” As for the trite Tolkein quotation, was he trying to show he was down with the geeks?)

But it was the attack on the British Library’s newspaper digitisation program that garnered most of the headlines; see, for example, The Indy and The Guardian. The library’s project aims to turn some 40 million pages of their newspaper holdings into searchable, preservable, accessible, distributable text; a great resource for historians. The majority of this material is clearly out of copyright and in the public domain. Where it is not, there will be agreement with, and remuneration for, the copyright holders. The press release states:

…. the partnership will also seek to digitise a range of in-copyright material, with the agreement of the relevant rightsholders. This copyright material will, with the express permission of the publishers, be made available via the online resource – providing fuller coverage for users and a much-needed revenue stream for the rightsholders.

So what’s Murdoch getting so angry about? Immediately, competition with archive.timesonline.co.uk. It’s curious he didn’t take the opportunity to plug that product along with all the others. But there’s something else: free content.

Just yesterday, the Library announced the digitisation of their newspaper archive – originally given to them by publishers as a matter of legal obligation. This is not simply being done for posterity, nor to make free access for library users easier, but also for commercial gain via a paid‐for website. The move is strongly opposed by major publishers. If it goes ahead, free content would not only be a justification for more funding, but actually become a source of funds for a public body.

As the old saw goes, there’s free as in freedom, and free as in beer. One means the freedom to use a resource in any fashion, the other simply not to pay for something. The British Library project is not free in either sense, as I’ve previously shown. Likewise the Times archive.

Clearly, the “free content” referred to is material on which copyright has expired and which is now in the public domain. It can be used in any way anyone desires. The problem is obtaining it. That means only certain institutions, the British Library and News International alike, can take advantage of this common wealth, and enclose it with an array of technological (DRM), financial (paywalls) or contractual (the terms of use and copyright claims over remastering into digital formats) fences.

For Murdoch, this isn’t enough. He believes the Times archive is his inviolable property. Although in practice only the British Library can compete with him, the root problem is that this material is in the public domain. Consequently, not only does he berate public sector competition for “profiting from work they do not create”, but demands that only the creative industries should be allowed to “develop and protect the value of what they create”,  even where copyright has lapsed. This privatisation envisages the digital humanities as an adjunct to commercial exploitation, in line with current ideologies of the “impact agenda” and business-driven education; and it dramatically diminishes our historical commons.

Standard caveats: I am not a lawyer. Nor do I play one on TV. Nor am I a mind-reader with privileged access to the inner workings of the mindset of the wealthy.

See also  Richard Lewis and The Guardian.

You can read the transcript here and here. Curiously, neither page has an explicit copyright statement, so presumably both sites are claiming the rights to the lecture!

This is a phenomenal amount of data: over 3 million names, 1,300 to 1,600 separate manuscripts, around a quarter of a million pages, some 40 million words in all. Which means it needs a decent interface and facilities, or else one will just be swamped by raw, muddy data. Judging by the demonstration given by Sharon Howard, it looks like it has been designed with researchers in mind, with the ability to cross reference records and link entries together in sets.

Two examples of using the site were given. Tim Hitchcock deftly restored the reputation of the much-maligned nurse Hannah Poole, falsely accused of being uncaring and negligent; Bob Shoemaker showed a perhaps surprising distance between the criminal world and the poor, members of one rarely showing up in the records of the other.

Digital projects have to have a research question, and the one behind London Lives is: how did the poor and the plebian relate to the welfare authorities in early modern London? There were a number of questions raised as to which term to use – the project was originally entitled ‘Plebian Lives and the Making of Modern London’, which was dropped because of the difficulties of defining class in this era (not to mention the presence in the records of numerous higher class individuals, such as administrators). Hence the new title ‘London Lives’, both more specific in location and more general in scope.

There is one aspect to this question that concerns me: it frames the poor in terms of their institutional existence. Of course, there are few records of the poor that come ‘from below’, from their own mouths, especially when illiteracy is the norm. And of course, these institutional sources offer much rich material for understanding the lives of the lower classes, not necessarily as passive as one might presume. But there was much more to their lives than interactions with the authorities, and quite possibly a considerable degree of social autonomy as well.

Uncovering this independent existence is central to my Alsatia project, and devilishly difficult to do.

The London Lives Project isn’t live yet; it should be opening this May, but in the meantime there’s a blog for news and information about the forthcoming unconference in July.