The film
About 4:48, hosted by Pylon. It asks you one question and waits for your answer before it shows you. The narrated cut runs 3:51, voiced by Pylon in a synthesized voice made from the author's own voice; the author approved these cuts for publishing. Captions are on by default. The live version under it draws the same film in your browser, waits for your answer at each question, and runs a little longer; the full words are also below.
Nothing loads until you press play.
The lab: the customs shed
Type any text and cut it with merges learned on the guides' sentences, on Moby-Dick's opening, or on your own text. Slide the number of merges and switch the base to bytes. On the guides' 47,002 words, every word is one piece after 136 merges; Moby-Dick's opening takes 878.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
Tokens
Tokens. Cutting text into crates.
Everything that comes through my customs shed goes into a standard crate. Text too, before any language model can touch it.
How big a crate
A language model doesn't read letters. It reads a list of pieces, each one a number from its vocabulary. The question is how big to cut the pieces.
Cut by letter, with one extra mark for the end of each word, and the guides' sentences take about 4.39 pieces per word, counting each full stop and question mark as a word: tiny crates, a great many trips.
Cut by whole word, and every word is one piece. But a word the model has never seen has no crate at all.
Merge the common pairs
Byte-pair encoding meets in the middle. Start from single letters. Count every pair of neighbouring pieces, inside words. The most common pair gets a crate of its own. Then count again.
On the guides' sentences, the first merge joins an e to the end-of-word mark, so an e that ends a word gets one crate. By the fifth merge, the whole word 'the' has its own crate.
After fifty merges, how many pieces does a word take? (the film pauses here)
After 50 merges, a word takes 1.80 pieces on average. After 136, every one of the guides' words is a single piece.
A word nobody shipped before
Now a word that never came through the shed: lamplit. There's no crate for it.
So it's cut into pieces that do have crates: l, am, p, and lit at the end of a word. Nothing is thrown away. Rico Sennrich and his colleagues used this method, in a 2016 paper, to translate rare words.
Real English
The guides only know 44 words and 2 marks. So here's real English: the opening of Moby-Dick, about 599 words of it, to learn the merges from.
Then a page it never saw, the opening of Pride and Prejudice. With 400 merges, the new page takes 2.14 pieces per word. The word lighthouse, never seen, comes out as li, gh, thou and se.
Why the common pairs
Does it matter which pairs we merge? Merge random pairs instead, the same number of times, and over ten tries the guides' words still take 1.32 to 1.67 pieces each, instead of one.
The smallest crates
There's one more choice: what the very first pieces are. Characters run out fast. The page from Pride and Prejudice has curly quotation marks, and a capital D and E, that Moby-Dick never used, so with characters as the base, they have no crate.
GPT-2 starts from bytes instead. There are only 256 of them, and any text at all is some list of bytes. An a is one byte, an e with its accent built in is two, a curly apostrophe three, and a seal emoji four.
Start from bytes, and that new page has 0 pieces without a crate, instead of 16.
The lighthouse model
The little language model from the attention lesson skips all of this. It has one crate per word, 46 of them counting the full stop and question mark, because its whole world is those 46.
Byte-pair encoding needs 136 merges, and a vocabulary of 160, to cut the same sentences into whole words, because every piece it made on the way stays in the vocabulary.
Goodnight
Common pairs get their own crates, rare words come apart into pieces, and bytes mean nothing is ever left on the dock.
Next: giving every crate a place on the chart. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's tokenizer runs, computed for this film. media/school/tokens/facts.json, written by tools/school/fill_tokens.mjs from system/school/tokens.mjs: BPE learned on the guides' corpus (3,000 generated episodes, 47,002 words) and on the first five paragraphs of Moby-Dick (716 words), measured on held-out text. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/tokens
Used for: Guides' corpus: 4.39 tokens per word at 0 merges (characters plus one end-of-word symbol per word; '.' and '?' count as words), 1.80 at 50 merges, 1.00 at 136 merges (vocabulary 160). The first merge is e with the end-of-word mark; the fifth makes 'the'. Unseen 'lamplit' is cut into l, am, p, lit (end of word). Ten runs of 136 random merges leave 1.32 to 1.67 tokens per word. English: with 400 merges learned on Moby-Dick (716 pre-tokens, 599 words split on spaces), the opening of Pride and Prejudice takes 2.14 tokens per word; 'lighthouse' is cut into li, gh, thou, se, the same at 400 and at 878 merges. The attention model is word-level with 46 entries: 44 words and two marks ('.' and '?').
Does not prove: Anything about large tokenizers; small texts chosen to show the mechanism.
Sennrich, Haddow and Birch 2016, Neural Machine Translation of Rare Words with Subword Units. Sennrich, R., Haddow, B., Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arXiv:1508.07909. https://arxiv.org/abs/1508.07909
Used for: Byte-pair encoding for word segmentation: start from characters with an end-of-word symbol, repeatedly merge the most frequent adjacent pair of symbols within words, and add the merged symbol to the vocabulary (section 3.2, Algorithm 1). Rare and unseen words are represented as sequences of subword units, which lets a translation model handle them.
Does not prove: That BPE is the best segmentation for every language or task.
Gage 1994, A New Algorithm for Data Compression. Gage, P. (1994). A New Algorithm for Data Compression. The C Users Journal 12(2), 23-38. https://www.derczynski.com/papers/archive/BPE_Gage.pdf
Used for: Byte pair encoding began as compression: replace the most frequent adjacent pair of bytes with a byte that does not occur in the data, and repeat.
Does not prove: Anything about language models.
Melville 1851, Moby-Dick. Melville, H. (1851). Moby-Dick; or, The Whale. Chapter 1, paragraphs 1 to 5. Project Gutenberg eBook #2701. https://www.gutenberg.org/ebooks/2701
Used for: Public-domain English text used as training text for the English BPE.
Does not prove: Nothing; it is sample text.
Austen 1813, Pride and Prejudice. Austen, J. (1813). Pride and Prejudice. Chapter 1, paragraphs 1 to 10. Project Gutenberg eBook #1342. https://www.gutenberg.org/ebooks/1342
Used for: Public-domain English text used as held-out text.
Does not prove: Nothing; it is sample text.
Radford and others 2019, Language Models are Unsupervised Multitask Learners (GPT-2). Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Used for: GPT-2 applies BPE to bytes rather than Unicode code points: a base vocabulary of 256 instead of over 130,000, so any string can be encoded (section 2.2). Its vocabulary has 50,257 entries (section 2.3).
Does not prove: That byte-level BPE is better on every measure; ours compresses slightly worse than character-level BPE on the held-out page.
UTF-8 byte counts and byte-level BPE, computed for this film. media/school/tokens/facts.json (bytes section): TextEncoder byte counts; character-level and byte-level BPE with 878 merges on Moby-Dick, measured on Pride and Prejudice. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/tokens
Used for: UTF-8: a is 1 byte, a precomposed accented e (U+00E9) 2 (an e with a combining accent is 3), a curly apostrophe 3, the seal emoji 4. On the held-out page, character-level BPE leaves 16 unknown tokens (7 open and 7 close curly quotes, one D and one E never seen in training); byte-level BPE leaves 0.
Does not prove: That bytes are the right base for every language model.
What this lesson leaves out
The merges here are learned on small texts, so the pieces are short and odd; production tokenizers learn tens of thousands of merges on far more text.
Real tokenizers add rules this one leaves out (how spaces attach, special tokens, normalisation), so their pieces for the same text will differ.
Before this: The held-out set (F6). Next: Embeddings (L8).