A small language model squeezes the UDHR to 0.27 bits a character, and fresh text to 1.59

A language model that predicts the next word well is also a compressor, and its compression rate is often quoted as a measure of how well it models text. We used two small open models, SmolLM2-360M and Qwen3-0.6B, as arithmetic coders and compared them with xz and with a PPM context model. SmolLM2 encodes the English Universal Declaration of Human Rights at 0.27 bits per character, and Qwen3-0.6B at 0.57.c1 The best classical coder, PPM trained on 5 million characters of Project Gutenberg, needs 1.99. We then added a control we had not planned: 30,000 characters of our own technical writing from October 2026, written after both models were released. On text written after their training, the two models need 1.59 and 1.66 bits per character.c2 Cold PPM, which has seen nothing, finds the unseen text only slightly harder than the declaration: 2.70 against 2.57 bits per character.c3 The models' advantage on the declaration therefore points at their having seen it; one fresh text cannot size that share. An arithmetic encoder and decoder round-tripped a 120-token passage exactly with both models.c5

A model that assigns a probability to each next symbol can drive an arithmetic coderDefinition: arithmetic coding data compression A method that turns a model's probability for each next symbol into a single binary number, so a symbol given probability p costs close to minus log2 p bits, and a decoder with the same model recovers the text exactly. Excludes codes that give each symbol a fixed whole number of bits, such as Huffman codes. full definition, which spends close to minus ${\log }_{}p$ bits on a symbol given probability $p$ and lets a decoder with the same model recover the text exactly. The better the predictions, the shorter the code. Delétang and colleagues [1] used this equivalence to show that large language models are strong general-purpose compressors, and pointed out that counting the model's own size changes the picture. Huang and colleagues [2] found that compression on external text corpora tracks benchmark scores almost linearly across about thirty public models, and proposed compression as an evaluation measure.

Compression on public text has a known weakness. If the text was in the training data, a model can predict it from memorisationDefinition: memorisation machine learning; in evaluation also called training-data contamination A model's ability to predict a specific text because that text, or close copies of it, was in its training data, as distinct from predicting text of the same kind it has not seen. Excludes any claim about how the model stores the text internally. full definition, recall of that text, and not from a general model of language. Li and colleagues [3] built an evaluation on this point: they measured compression on Wikipedia, news, code and arXiv text dated before and after each model's training cutoff and reported that the rate worsens after the cutoff for many models.

Our question is smaller and concrete. How many bits per characterDefinition: bits per character information theory and data compression The length of a compressed text in bits divided by the number of characters in the original. Lower is better, and 8 is the cost of plain one-byte text. Excludes bits per token or per byte, which divide by a different count. full definition do two small open models need on a famous public document, against classical compressors, and how much of the result survives on text written after they were trained?

This test is one part of a larger study of language as a code. Its preregistrationDefinition: preregistration research method (metascience) A dated written plan for a study, fixed and hash-locked before any result is seen: the question, the data, the statistic, the controls, the forecasts and the kill conditions. Excludes a plan revised after looking at the outcome, and a protocol written only for the record after the run. full definition, a dated plan fixed and hash-locked before any result is seen, is a forecast file written and hashed on 1 November 2026 at 03:14 UTC, before any measurement. A first work session wrote it and stopped with no measurement saved. A second work session wrote a file of scoring rules 13 minutes later, before computing anything, and ran every measurement after that. A point forecast holds if the measured value is within 25 percent of it, and an inequality forecast holds when the inequality is true.

For this part the forecast file fixed:

The unseen-text control was not in the forecast file. It was added after the declaration came in at 0.27 bits per character under SmolLM2, to test whether that number was memorisation. It is a post hoc controlDefinition: post hoc control research method A control added after the first results were seen. It can test an explanation but carries less weight than a control fixed in advance, because the choice of control could follow the result. Excludes controls named in the preregistration. full definition: it can test an explanation, and it carries less weight than a control fixed in advance.

SmolLM2's technical report appeared in February 2025 and Qwen3's in May 2025 [4] [5]. That the unseen text was written after both models were trained rests on those dates and on the log's own dates.

Each text is split into tokensDefinition: token natural language processing The unit a language model reads and predicts, a whole word, part of a word or a single character taken from a fixed vocabulary of tens of thousands of pieces. Excludes a character or a byte. full definition, the word pieces a model reads, with the model's own tokenizer. Every token $t_i$ is priced at minus ${\log }_{}$ of the probability the model gave it after the tokens before it. The ideal code lengthDefinition: ideal code length information theory The sum over a text of minus log2 of the probability the model gave to each symbol that came next. A real arithmetic coder comes within a few bits of it. Excludes a measured file size, and any cost of storing the model. full definition of a text of $C$ characters and $T$ tokens is

in bits per character, with the sum over tokens 2 to $T$. The first token is priced at ${\log }_{}V$ bits for a vocabulary of $V$ tokens: 49,152 for SmolLM2 and 151,936 for Qwen3. Long texts are read in windows of 512 tokens that advance by 256, so every token after the first window is predicted from at least 256 tokens of context. The models ran in half precision on one GPU. Decoding the tokens gave back the exact original text for the declaration and Alice under both tokenizers.

An ideal length is only a claim until a coder achieves it. For each model, an arithmetic encoder turned the first 120 tokens of the declaration into a bit string, using a 64-token context, single precision weights on the CPU and probabilities rounded to integer frequencies out of $2^{24}$. The decoder, given the bit string, the first token and the same model, rebuilt the 120 tokens. This is the kill condition in the design.

xzDefinition: xz data compression software A general-purpose file compressor using the LZMA method, which replaces repeated strings with references to earlier copies. Here at its strongest preset. Excludes any model of language beyond repeated byte strings. full definition is a general-purpose compressor that replaces repeated strings with references to earlier copies; it ran at its strongest preset, and its size in bits is divided by the number of characters. PPMDefinition: PPM data compression Prediction by partial matching, a compressor that predicts each character from counts of what followed the same preceding characters, falling back to shorter contexts when a longer one has not been seen. Here method C with exclusion and contexts of up to 5 characters. Excludes models with learned weights, such as neural language models. full definition, prediction by partial matching [6], predicts each character from counts of what followed the same preceding characters, here up to 5 characters back, with method C escapes and exclusion. A character never seen in any context costs ${\log }_{}$ of the number of Unicode code points. PPM cold starts empty and learns as it codes. PPM trained starts from counts over the 5,000,000 training characters and keeps learning as it codes; the counts are restored after each text.

A compressor that must be sent along with the message pays for its own size. The payback lengthDefinition: payback length our usage, after the two-part code of minimum description length The number of characters a model must compress before the bits it saves over another compressor add up to the size of the model itself, when the model has to be sent along with the message. Excludes a compression ratio on one text, which ignores the cost of the model. full definition is the number of characters after which the bits saved over xz equal the model's size in bits:

where $S$ is the model's size in bytes and $b_{\mathrm{xz}}$ and $b_m$ are bits per character under xz and under the model. For trained PPM, $S$ is the training text compressed with xz, 1,458,524 bytes.

For a prompt such as "The capital of Norway is" or "5 + 6 =", the cost of an answer is the code length of the answer's tokens given the prompt. Each prompt has a right answer and a wrong one. There are 20 capital-city prompts and 30 each of one-digit and three-digit sums.

A controlDefinition: control statistics and experimental design A version of the data with the structure under test removed (for example shuffled or pooled), scored the same way as the real data, so the real score has something to beat. Excludes a comparison with a different dataset or with a published number. full definition is a version of the data with the structure under test removed and scored the same way. Three serve here.

The round trip of the methods section is a check that the model rows are real codes. An arithmetic encoder and decoder round-tripped a 120-token passage exactly with both models.c5 The kill condition did not fire. The long-document figures remain ideal code lengths.

PPM was retrained on the same 5,000,000 characters with every character shuffled, and again with every word shuffled. On the declaration, character-shuffled training gave 3.59 bits per character, worse than cold PPM at 2.57, so the trained model's gain over cold is gone and more. Word-shuffled training gave 2.10 against 1.99 for normal training, a loss of a fifth of the gain. The context model gains from letter order inside words much more than from word order, as it should.

The unseen text is the control for prior exposure. A coder that has seen nothing, cold PPM, gives a measure of how hard each text is in itself.

Table 1 gives every coder on every text, and Figure 1 draws the same numbers.

SmolLM2 encodes the English Universal Declaration of Human Rights at 0.27 bits per character, and Qwen3-0.6B at 0.57.c1 Against xz on the same text that is a tenfold and a fivefold lead. Our forecasts were 1.2 and 1.0, and both failed by far more than the 25 percent allowed. The Gutenberg book shows the same pattern more weakly: 0.89 under SmolLM2, against a forecast of 2.0.

On text written after their training, the two models need 1.59 and 1.66 bits per character.c2 SmolLM2 needs about six times the bits it needs on the declaration. Its lead over xz falls from tenfold to under twofold.

Cold PPM, which has seen nothing, finds the unseen text only slightly harder than the declaration: 2.70 against 2.57 bits per character.c3 xz finds it easier, 2.74 against 2.83. A sixfold jump for a language model on a text that classical coders find about as hard as the declaration points at prior exposure to the declaration.

Trained PPM does worse than cold PPM on the unseen text, 2.89 against 2.70 bits per character.c4 Its training on literary English helps on the declaration and on Alice and hurts on our technical writing.

Model size changes the ranking for short messages. By eq. 2, SmolLM2 pays for its 723,674,912 bytes against xz after 2.6 billion characters of Alice-like text, which held the preregistered bound of at least one billion. On the unseen text, where its lead over xz is 1.15 bits per character, the same arithmetic gives about 5 billion characters. Trained PPM pays back after 11.9 million characters of Alice-like text.

Figure 2 shows the cost of sending a known answer. A right capital city costs about 1 bit under either model, and a wrong one 18 to 20. A one-digit sum costs Qwen3 1.29 bits and SmolLM2 4.01, close to its 4.63 for a wrong sum. A three-digit sum costs 8.5 to 9.1 bits even when right, and a wrong one costs only about 5 bits more. The answer costs fit recall of capital cities and show no sign of reliable three-digit addition.

Appendix A sets every preregistered forecast for this part against its outcome.

The headline number, 0.27 bits per character, is true of the declaration and says little about language. The declaration is copied widely on the public web, so we infer that it sits in the training data of both models, probably many times; we did not check their training sets. A model that has seen it often can code it from recall. The unseen text removes that route, and the lead over classical coders drops to what a model of language buys.

That residual lead is still real. On text neither model could have seen, both need about 1.6 bits per character against 2.70 to 2.89 for the classical coders. This agrees with the main claim of Delétang and colleagues [1] that language models are strong compressors, and with their point that the model's size belongs in the count: on fresh text, SmolLM2 needs billions of characters before it pays for itself against xz.

Our result fits the evaluation design of Li and colleagues [3], who score compression on text from after each model's training cutoff. It also bears on compression used as a measure of capability [2]. A corpus that overlaps the training data rewards recall, and two models can rank differently on recall than on language. Here the smaller model, SmolLM2, leads by a factor of two on the declaration and by 4 percent on the unseen text.

The trained PPM result is a smaller lesson of the same kind. Counts from Gutenberg books help on literary text and hurt on a technical log. A compressor's training text is part of its result, for a context model as much as for a neural one.