The film
About 3:38, hosted by Tallow. It asks you one question and waits for your answer before it shows you. The narrated cut runs 2:53, voiced by Tallow in a synthesized voice made from the author's own voice; the author approved these cuts for publishing. Captions are on by default. The live version under it draws the same film in your browser, waits for your answer at each question, and runs a little longer; the full words are also below.
Nothing loads until you press play.
The lab: the lamp room
Train your own skip-gram on the guides' sentences: choose the seed, the numbers per word, the window and the noise words, then step through its snapshots and pick a word to see its neighbours. The film's run ends with 65% of words beside a word of the same kind, against about 23% shuffled.
Everything is computed on your computer, in this page. Nothing is sent anywhere.
Everything the film says
Embeddings
Embeddings. Every word gets coordinates.
Pylon showed you that a word in a model is a list of numbers, a place. Tonight, where those places come from.
Before anything is learned
At the start, every word gets a list of 32 small random numbers. The places mean nothing yet.
Check the neighbours: only 26 percent of words sit next to a word of the same kind, about what a shuffle gives.
You are the company you keep
The trick is a guessing game. Take a word in a sentence, and the words around it.
Each word gets two lists: one for when it's the word in the middle, one for when it's a neighbour. Nudge the lists so that the middle word's list and each real neighbour's neighbour-list get a bigger dot product, the measure from the vectors lesson.
And for every real neighbour, pick 5 words at random, common words more often than rare ones, and push those the other way. Tomas Mikolov and his colleagues called it negative sampling, in 2013.
The cards drift
Run the game over the guides' sentences, 5 times through, and watch the places move.
Will the neighbours keep getting tidier all the way through training? (the film pauses here)
Very early, by 5 percent of the way, the neighbours look their tidiest of any checkpoint: 72 percent of words sit next to a word of the same kind. By the end, in this run, it has slipped back to 65, three words fewer out of 46, while the guessing game keeps getting better on sentences it never trained on.
Better at the game is not always tidier on the chart. The game is the goal; the neat neighbours are a side effect.
Another roll of the dice
Start from different random numbers, four more times, and you get different coordinates, but much the same neighbours: against the first run, 75 to 80 percent of each word's three nearest neighbours agree.
Two ways to the same chart
The language model from the attention lesson learned its word lists a different way: as a side effect of guessing the next word, through two layers of attention.
Its lists put 80 percent of words next to a word of the same kind. And between the two tables, about 33 percent of each word's three nearest neighbours agree. By chance it would be about 7 percent.
So the two methods agree on far more than chance, and far less than two runs of the same method agree with each other. Neither one is the true map. Each is a map of what its own game needed.
Goodnight
Nobody draws the chart. A guessing game moves the cards, and words used alike end up close.
Next in the lighthouse: guessing the next word, and keeping score. Goodnight.
Sources
Every claim in the film, with where it comes from and what that source does not prove. The numbers about the model are computed from its weights, which you can load and check: facts.json.
Night School's skip-gram, trained for this film. media/school/embeddings/facts.json, written by tools/school/fill_embeddings.mjs from system/school/embeddings.mjs: skip-gram with negative sampling (32 numbers per word, window 1 to 2, 5 noise words, 5 epochs, seed 1) on the guides' corpus; a finite-difference check of the hand-written gradient (worst 5.3e-7). https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/embeddings
Used for: Same-kind nearest neighbour share: 26% at the start (shuffle control mean 22%, 95% interval 9% to 39%); highest of the 12 snapshots at 5% of training (72%), 65% at the end, from one seed; held-out loss falls at every snapshot (2.14 at 5%, 1.92 at the end). Seed 1 against each of four other seeds: 75% to 80% of each word's three nearest neighbours agree (pairs among seeds 2 to 5 not measured).
Does not prove: Anything about large embedding tables.
Mikolov and others 2013, Distributed Representations of Words and Phrases and their Compositionality. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS 26. arXiv:1310.4546. https://arxiv.org/abs/1310.4546
Used for: The skip-gram model learns an input and an output vector for every word, trained so a word's vector predicts the words around it (section 2). Negative sampling replaces the full softmax: each real (word, context) pair is pushed together and k words drawn from a noise distribution (the unigram distribution to the 3/4 power) are pushed apart (section 2.2, equation 4).
Does not prove: That the learned neighbours reflect meaning beyond how words are used in the training text.
Night School's tiny transformer, its word table. media/school/attention/model.json (wte), compared in media/school/embeddings/facts.json. https://github.com/HarperZ9/HarperZ9.github.io/tree/main/media/school/embeddings
Used for: The attention model's word table puts 80% of words next to a same-kind word; its top-3 neighbour lists overlap the skip-gram's by 33% (chance about 7%; shuffled control 2% to 12%).
Does not prove: Which table is better; they were trained for different games.
What this lesson leaves out
Forty-six words from a made-up corpus are enough to see the effect and far too few to show what large tables learn; the categories used to score neighbours are ours.
The chart shows two directions out of many, so two cards can look close and be far apart; neighbours are measured in all the dimensions.
Before this: Tokens (L7). Next: Next-word prediction (L9).