Shannon's 1951 letter statistics hold on old and new English; his third-order value reads high

In 1951 Claude Shannon published a table of how many bits each letter of printed English carries when the reader knows 0, 1 or 2 letters before it. We recomputed that table on three pooled corpora: books from 1788 to 1859, fiction from 1892 to 1925, and present-day Wikipedia featured articles, each reduced to 26 letters and a space. The single-letter and letter-pair rows match Shannon's within 0.12 bits on all three corpora.c1 His third-order value of 3.1 bits sits 0.39 bits above ours on the old books and 0.23 above ours on Wikipedia.c2 We also trained a letter-count reader on each corpus and scored it on held-out text. With the corrected split, the reader needs 1.63 to 2.41 bits per letter on its own kind of text and never reaches Shannon's 1.3.c3 Within fiction, a change of era costs the reader 0.10 and 0.25 bits per letter, and a change from fiction to The Federalist in the same era costs 0.44, so the preregistered claim that era costs more than genre failed.c4 A planted Markov text with known entropy was recovered within 0.004 bits.

Shannon [1] described two ways to measure how predictable English is. The first counts letters. Let the text be reduced to 26 letters and a space, so 27 symbols. The N-gram entropyDefinition: N-gram entropy information theory, after Shannon 1951 The uncertainty of the next letter when the N minus 1 letters before it are known, averaged over the text. Shannon wrote it F with a subscript N, so F1 uses single-letter counts, F2 one letter of context and F3 two. Excludes the entropy of the language as a whole, which is the limit as N grows. full definition is the uncertainty of the next letter when the N minus 1 letters before it are known; F1 uses single-letter frequencies, F2 knows one letter back and F3 two. All are in bits per letterDefinition: bits per letter information theory The average number of binary digits needed to write down each letter of a text under a given model of the text. One bit halves the number of equally likely choices, so 4.75 bits per letter is the cost of picking blindly among 27 symbols. Excludes a count of bytes in a file, and any statement about meaning. full definition, the average number of yes-or-no answers needed per letter. Blind choice among 27 symbols costs 4.75 bits. His table for 27 symbols reads 4.76, 4.03, 3.32 and 3.1 for F0 to F3, and 2.14 bits per letter from word statistics. The second way asks a person to predict the next letter of a passage. From that experiment he put an upper bound near 1.3 bits per letter on the entropy of English when 100 letters of context are known.

Later work kept both routes. Cover and King [2] replaced guessing with betting, so a subject's stake on each letter gives a probability, and reported estimates near 1.3 bits per character. Brown and colleagues [3] built a word trigram model, measured its cross-entropy on a balanced sample of English and gave an upper bound of 1.75 bits per character. Takahira, Tanaka-Ishii and Debowski [4] compressed large corpora in six languages with prediction by partial matching [5] and extrapolated the code length to infinitely long text to estimate the entropy rate itself.

Those studies push the estimate of the entropy rate downward. Our question is narrower. Do the low rows of Shannon's table, which are easy to recompute, still hold on text of other eras and kinds? And when a reader built from letter counts is trained on one kind of text and reads another, which costs more, a change of era or a change of genre?

A preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition is a dated plan fixed and hash-locked before any result is seen. This study had two. The first was locked on 1 November 2026 at 08:13 UTC. It named the corpora, the statistics, the reader, the planted text, the control, nine numeric forecasts and the kill conditionDefinition: kill condition research method (our usage, after Popper's falsification) A rule written in the preregistration that, when met, decides against the claim with no room for judgement after the run. Excludes reasons found after the run for setting a result aside. full definition, a rule that decides against the run with no judgement after it. The decision it informed was which claims in material we had saved still hold when the text or the reader changes, and so which ones we can reuse.

The first run had a fault of ours. The code promised in a comment to shuffle the books before splitting training text from test text and never did. The old corpus's test text was therefore all The Federalist, and the mid corpus's all Fitzgerald, so era was read through one book. We wrote a second preregistration to fix the split and locked it at 08:20 UTC the same day. It was not blind: it was written after the first run had been read. The first run stands as recorded, and both are reported here.

Claims and kill condition, as locked:

The books are public-domain texts from Project Gutenberg [6]: The Federalist (1788), Austen (1813), Bronte (1847), Melville (1851) and Dickens (1859) make the old corpus; Doyle (1892), Joyce (1914 and 1916), Wodehouse (1920) and Fitzgerald (1925) make the mid corpus. The new corpus is a sample of Wikipedia featured articles [7], about 0.77 MB, fetched through the Wikipedia API on 1 November 2026. Wikipedia text is licensed CC BY-SA; we use it for statistics only and release no text from it. The source record names each file and its size; the text files themselves were not hashed at download, so a reproduction may draw slightly different editions.

Each text was cut to the Project Gutenberg start and end markers, lowercased, and every run of characters outside a to z was replaced by a single space. In the first run each corpus takes an equal slice from each of its books, 750,000 characters in all, and the first 600,000 train the reader. In the second run every book gives its first 80 percent to training and its last 20 percent to test, and the sets are kept apart by genre: old fiction (four novels), mid fiction (five works), old nonfiction (The Federalist) and new nonfiction (Wikipedia). Each set has 600,000 training characters, except Wikipedia with 588,712, and 100,000 test characters.

Shannon's own test passages are not openly available, so we compare against his printed table. The table values were recorded in the first preregistration from a scan of the 1951 paper.

For a block $b_i$ of N minus 1 letters and a next letter $j$, the N-gram entropy is

with probabilities taken as relative frequencies in the pooled corpus. This is a plug-in estimateDefinition: plug-in estimate statistics An entropy computed by putting the observed relative frequencies of a sample straight into the entropy formula, with no correction for the size of the sample. Excludes corrected estimators that add a term for unseen or rare events. full definition: observed frequencies go straight into the formula with no correction for sample size. F0 is ${\log }_{}27$, 4.755 bits.

The word-based figure divides the entropy of the word frequency distribution, $H_w$, by the mean number of characters per word including the following space, $L$:

Shannon reached his 2.14 by extending a fitted Zipf's lawDefinition: Zipf's law quantitative linguistics The observation that the frequency of a word falls roughly in inverse proportion to its rank in the frequency list. Excludes an exact law that holds for every word or every text. full definition tail, the rule that a word's frequency falls in inverse proportion to its rank. We counted words directly.

The reader is a context modelDefinition: context model data compression and statistical language modelling A model that predicts the next symbol from counts of what followed the same few preceding symbols in training text, blending long and short contexts. Prediction by partial matching (PPM) is the classic form. Excludes models that use word meaning, grammar or knowledge outside the letter counts. full definition: it predicts the next letter from counts of what followed the same preceding letters in training text, in the manner of prediction by partial matching. It blends contexts of length 0 to 6. Starting from a uniform distribution over 27 symbols, each context length $k$ with $n_k$ counts over $d_k$ distinct next letters updates the estimate as

where $c_k\left(x\right)$ is the count of letter $x$ after that context. The reader is scored by its log lossDefinition: log loss information theory and machine learning; also called cross-entropy The average of minus the base-2 logarithm of the probability a model gave to the letter that came next, in bits per letter. It is the length of the code an ideal coder driven by the model would write. Excludes the share of letters guessed right, and the entropy of the text itself, which log loss can only bound from above. full definition, the average of minus ${\log }_{}$ of the probability it gave to the letter that came next:

over $n$ letters of held-out textDefinition: held-out text statistics and machine learning Text kept apart from the text a model was trained on and used only to score it. Excludes any text whose counts entered the model. full definition, text the reader was not trained on. The first run scores 40,000 letters of each test text and the second 30,000.

The reader also plays Shannon's guessing gameDefinition: guessing game information theory, after Shannon 1951 A reader guesses the next letter in order of preference until right. The share of letters found at the first, second and later guesses gives an upper and a lower bound on the entropy. Excludes a direct measure of the probabilities a reader holds. full definition: it ranks the 27 symbols and we record the rank $r$ of the true letter. With $q_r$ the share of letters found at rank $r$, Shannon's upper and lower bounds are

A transfer penaltyDefinition: transfer penalty our usage, from machine learning practice on domain shift The extra log loss a reader pays on another kind of text compared with a reader trained on that kind of text, both scored on the same held-out passage. Excludes the difference in difficulty between two texts read by their own readers. full definition is the extra log loss a reader pays on another kind of text, against a reader trained on that kind, on the same held-out passage. With $R(a,b)$ the log loss of the reader trained on set $a$ reading the test text of set $b$,

The era penalty in fiction uses $a$ and $b$ from old and mid fiction. The genre penalty uses old fiction and The Federalist, which share an era.

A planted checkDefinition: planted check research method; called a positive control in experimental science Data built so that the right answer is known, run through the same analysis before the real data, to show that the analysis can see the effect when it is there. Excludes any check made on the real data, which cannot show what the analysis misses. full definition is data built so the right answer is known. Ours is a random order-2 Markov chainDefinition: order-2 Markov chain probability A random text in which each symbol depends only on the two symbols before it, through a fixed table of probabilities, so its entropy per symbol can be computed exactly. Excludes real text, whose dependence reaches further back. full definition over 27 symbols, 2,000,000 symbols long, with transition probabilities drawn from a Dirichlet distribution. Its entropy per symbol is 2.673 bits by eq. 1 with N of 3. A reader of context length 2 was trained on the first half of the chain and scored on 20,000 later symbols. It missed the true entropy by 0.004 bits, against a kill threshold of 0.1.c5 The void condition did not fire.

A controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition is a version of the data with the structure under test removed and scored the same way. Here the held-out test text has its characters shuffled 20 times, 4,000 letters each, and the trained reader reads each shuffle. The real text should cost far less than any shuffle. In the first run the shuffles cost a mean of 8.51 bits per letter against 2.73 for the real Wikipedia text; in the second, 9.26 against 1.93 for old fiction. The shuffles cost more than 4.75 bits because the reader is confident and wrong on scrambled text. The real text beat all 20 shuffles in both runs. The p value from drawsDefinition: p value (from draws) statistics The share of control draws scoring at or above the real score, with one added to the count and to the number of draws, so it is never zero. Excludes a p value from a textbook formula that assumes a distribution. full definition, $p=(k+1)/(n+1)$ with $k$ shuffles at or below the real loss out of $n$, is 0.048, the smallest value 20 draws allow.

Table 1 and Figure 1 set Shannon's table against ours. The first run computed the letter statistics on each pooled corpus.

The single-letter and letter-pair rows match Shannon's within 0.12 bits on all three corpora.c1 The largest of those gaps is F2 on Wikipedia, 0.120. His third-order value of 3.1 bits sits 0.39 bits above ours on the old books and 0.23 above ours on Wikipedia.c2 The first-run verdict on the old table failed on that row alone, and the verdict on the new table passed, since 0.228 is inside the 0.3 bound. The word-based figure is 0.29 to 0.39 bits below his. Words in the Wikipedia sample are longer, 5.92 characters with the space against 5.41 in the old books, and their frequency distribution carries 10.34 bits per word against 9.74.

In the first run the reader trained on the old corpus needed 2.13 bits per letter on the old held-out text, inside the forecast band of 1.85 to 2.35. Because of the faulty split, that held-out text was all The Federalist. The guessing game gave bounds of 1.34 and 2.35 by eqs. 6 and 5, and the reader's first guess was right on 58 percent of letters. The claim that this reader reaches 1.3 bits failed, as forecast.

The second run, with the split fixed, gives the matrix in Figure 2. With the corrected split, the reader needs 1.63 to 2.41 bits per letter on its own kind of text and never reaches Shannon's 1.3.c3 The Federalist is the easiest text for its own reader, 1.63, and Wikipedia the hardest, 2.41.

Table 2 gives the transfer penalties of the second run against their forecasts.

Within fiction, a change of era costs the reader 0.10 and 0.25 bits per letter, and a change from fiction to The Federalist in the same era costs 0.44, so the preregistered claim that era costs more than genre failed.c4 We forecast that failure. A mid-fiction reader on Wikipedia pays 0.92, where era, genre and topic all change at once.

Of the numeric forecasts, 4 of 9 held in the first run and 4 of 5 held after the fix. Appendix A lists each one.

The low rows of Shannon's table describe English letters across two centuries of text. Single-letter frequencies and letter-pair statistics differ by 0.03 and 0.09 bits between the oldest books and Wikipedia. That is the part of the table a reader can quote.

F3 is different. Our value is lower than his on every corpus, by more than our forecast allowed on the old books. We have not found the cause. Two are open, and neither was tested. A plug-in estimate falls below the true value as the number of contexts grows against the sample, and 27 squared, 729, two-letter contexts is many for 750,000 characters if rare pairs matter. Shannon's F3 also rests on letter counts from other text than ours, and a difference in the source counts would move it. Either way the third row should be quoted with its source and method.

The word-based gap fits a difference of method. Counting words directly misses the long tail of rare words that a fitted Zipf curve adds, and a missing tail lowers the entropy. We did not refit his tail to confirm it.

The reader result agrees with the line of work after Shannon [2] [3]: letter counts alone do not reach a human reader's figure. Our context model sits at 1.63 to 2.41 bits per letter on held-out text. Shannon's 1.3 is a bound on a reader who knows words, grammar and the world.

The transfer result was the condition we most wanted to vary. A reader from one era of fiction reads another era at a cost of 0.10 to 0.25 bits, and the same reader reads The Federalist at 0.44. What a text is about and how it is argued changes letter statistics more than when it was written.