pubmed.ncbi.nlm.nih.gov

Quantitative analysis of culture using millions of digitized books - PubMed

  • ️Sat Jan 01 2011

. 2011 Jan 14;331(6014):176-82.

doi: 10.1126/science.1199644. Epub 2010 Dec 16.

Affiliations

Quantitative analysis of culture using millions of digitized books

Jean-Baptiste Michel et al. Science. 2011.

Abstract

We constructed a corpus of digitized texts containing about 4% of all books ever printed. Analysis of this corpus enables us to investigate cultural trends quantitatively. We survey the vast terrain of 'culturomics,' focusing on linguistic and cultural phenomena that were reflected in the English language between 1800 and 2000. We show how this approach can provide insights about fields as diverse as lexicography, the evolution of grammar, collective memory, the adoption of technology, the pursuit of fame, censorship, and historical epidemiology. Culturomics extends the boundaries of rigorous quantitative inquiry to a wide array of new phenomena spanning the social sciences and the humanities.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1

‘Culturomic’ analyses study millions of books at once. (A) Top row: authors have been writing for millennia; ~129 million book editions have been published since the advent of the printing press (upper left). Second row: Libraries and publishing houses provide books to Google for scanning (middle left). Over 15 million books have been digitized. Third row: each book is associated with metadata. Five million books are chosen for computational analysis (bottom left). Bottom row: a culturomic ‘timeline’ shows the frequency of ‘apple’ in English books over time (1800-2000). (B) Usage frequency of ‘slavery’. The Civil War (1861-1865) and the civil rights movement (1955-1968) are highlighted in red. The number in the upper left (1e-4) is the unit of frequency. (C) Usage frequency over time for ‘the Great War’ (blue), ‘World War I’ (green), and ‘World War II’ (red).

Fig. 2
Fig. 2

Culturomics has profound consequences for the study of language, lexicography, and grammar. (A) The size of the English lexicon over time. Tick marks show the number of single words in three dictionaries (see text). (B) Fraction of words in the lexicon that appear in two different dictionaries as a function of usage frequency. (C) Five words added by the AHD in its 2000 update. Inset: Median frequency of new words added to AHD4 in 2000. The frequency of half of these words exceeded 10-9 as far back as 1890 (white dot). (D) Obsolete words added to AHD4 in 2000. Inset: Mean frequency of the 2220 AHD headwords whose current usage frequency is less than 10-9. (E) Usage frequency of irregular verbs (red) and their regular counterparts (blue). Some verbs (chide/chided) have regularized during the last two centuries. The trajectories for ‘speeded’ and ‘speed up’ (green) are similar, reflecting the role of semantic factors in this instance of regularization. The verb ‘burn’ first regularized in the US (US flag) and later in the UK (UK flag). The irregular ‘snuck’ is rapidly gaining on ‘sneaked’. (F) Scatter plot of the irregular verbs; each verb's position depends on its regularity (see text) in the early 19th century (x-coordinate) and in the late 20th century (y-coordinate). For 16% of the verbs, the change in regularity was greater than 10% (large font). Dashed lines separate irregular verbs (regularity<50%) from regular verbs (regularity>50%). Six verbs became regular (upper left quadrant, blue), while two became irregular (lower right quadrant, red). Inset: the regularity of ‘chide’ over time. (G) Median regularity of verbs whose past tense is often signified with a –t suffix instead of –ed (burn, smell, spell, spill, dwell, learn, and spoil) in US (black) and UK (grey) books.

Fig. 3
Fig. 3

Cultural turnover is accelerating. (A) We forget: frequency of 1883 (blue), 1910 (green) and 1950 (red). Inset: We forget faster. The half-life of the curves (grey dots) is getting shorter (grey line: moving average). (B) Cultural adoption occurs faster. Median trajectory for three cohorts of inventions from three different time periods (1800-1840: blue, 1840-1880: green, 1880-1920: red). Inset: The telephone (green, date of invention: green arrow) and radio (blue, date of invention: blue arrow). (C) Fame of various personalities born between 1920 and 1930. (D) Frequency of the 50 most famous people born in 1871 (grey lines; median: dark gray). Five examples are highlighted. (E) The median trajectory of the 1865 cohort is characterized by four parameters: (i) initial ‘age of celebrity’ (34 years old, tick mark); (ii) doubling time of the subsequent rise to fame (4 years, blue line); (iii) ‘age of peak celebrity’ (70 years after birth, tick mark), and (iv) half-life of the post-peak ‘forgetting’ phase (73 years, red line). Inset: The doubling time and half-life over time. (F) The median trajectory of the 25 most famous personalities born between 1800 and 1920 in various careers.

Fig. 4
Fig. 4

Culturomics can be used to detect censorship. (A) Usage frequency of ‘Marc Chagall’ in German (red) as compared to English (blue). (B) Suppression of Leon Trotsky (blue), Grigory Zinoviev (green), and Lev Kamenev (red) in Russian texts, with noteworthy events indicated: Trotsky's assassination (blue arrow), Zinoviev and Kamenev executed (red arrow), the ‘Great Purge’ (red highlight), perestroika (grey arrow). (C) The 1976 and 1989 Tiananmen Square incidents both lead to elevated discussion in English texts. Response to the 1989 incident is largely absent in Chinese texts (blue), suggesting government censorship. (D) After the ‘Hollywood Ten’ were blacklisted (red highlight) from American movie studios, their fame declined (median: wide grey). None of them were credited in a film until 1960's (aptly named) ‘Exodus’. (E) Writers in various disciplines were suppressed by the Nazi regime (red highlight). In contrast, the Nazis themselves (thick red) exhibited a strong fame peak during the war years. (F) Distribution of suppression indices for both English (blue) and German (red) for the period from 1933-1945. Three victims of Nazi suppression are highlighted at left (red arrows). Inset: Calculation of the suppression index for ‘Henri Matisse’.

Fig. 5
Fig. 5

Culturomics provides quantitative evidence for scholars in many fields. (A) Historical Epidemiology: ‘influenza’ is shown in blue; the Russian, Spanish, and Asian flu epidemics are highlighted. (B) History of the Civil War. (C) Comparative History. (D) Gender studies. (E,F) History of Science. (G) Historical Gastronomy. (H) History of Religion: ‘God’.

Comment in

Similar articles

Cited by

References

    1. Wilson Edward O. Consilience. Knopf; New York: 1998.
    1. Sperber Dan. Anthropology and psychology: Towards an epidemiology of representations. Man. 1985;20:73–89.
    1. Lieberson Stanley, Horwich Joel. Implication analysis: a pragmatic proposal for linking theory and data in the social sciences. Sociological Methodology. 2008 December;38:1–50.
    1. Cavalli-Sforza LL, Feldman Marcus W. Cultural Transmission and Evolution. Princeton UP; Princeton, NJ: 1981. - PubMed
    1. Niyogi Partha. The Computational Nature of Language Learning and Evolution. MIT; Cambridge, MA: 2006.

Publication types

MeSH terms