{"about":"Lesson L11, the transformer block. Computed by tools/school/fill_block.mjs from system/school/block.mjs and the attention lab's trained model. Do not edit by hand. Losses are in nats (natural log).","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","model":{"seed":7,"dataSeed":11,"steps":2000,"heldOutSeed":1592593325,"heldOutRows":256,"positions":6144,"nCopy":506,"nLocal":5638,"evaluateLoss":0.687407},"baselines":{"unigramLoss":3.272,"copyChance":0.1008},"params":{"about":"Parameter counts by part, read off the weight tensors. embeddings: word table (46 x 32) plus position table (24 x 32). attention: per layer the query, key and value matrix (32 x 96) with its bias and the output projection (32 x 32) with its bias. mlp: per layer 32 to 64 and back, with biases, GELU in between. norms: gain and bias of two layer norms per layer plus the final one. output: the readout matrix (32 x 46), not tied to the word table. A count of parameters says nothing about which ones matter; the knockouts measure that.","counts":{"embeddings":2240,"attention":8448,"mlp":8384,"norms":320,"output":1472,"total":20864},"shares":{"embeddings":0.1074,"attention":0.4049,"mlp":0.4018,"norms":0.0153,"output":0.0706},"config":{"dModel":32,"nHeads":2,"headDim":16,"nLayers":2,"dFF":64,"ctx":24,"vocab":46}},"stream":{"about":"The residual stream is a 32-number vector per position that every sub-layer reads (through its own layer norm) and adds to. steps: mean over the 6144 held-out positions of the stream's L2 norm after the embedding and after each sub-layer's add. adds: for each sub-layer, the mean L2 norm of what it added, the mean per-position ratio of that norm to the norm of the stream it was added to, and the mean cosine between the two (0: a new direction; 1: the same direction, only longer). untrained is the same network at its starting weights (seed 7), the control. One trained model and one held-out set: norms show where the stream grows in this network, not a law of transformers.","trained":{"steps":[{"at":"embed","norm":3.2179},{"at":"l0.attn","norm":4.4892},{"at":"l0.mlp","norm":44.2519},{"at":"l1.attn","norm":44.9752},{"at":"l1.mlp","norm":52.3363}],"adds":[{"at":"l0.attn","addNorm":3.0591,"ratioToStream":0.972,"cosineWithStream":-0.0068},{"at":"l0.mlp","addNorm":42.0254,"ratioToStream":10.0522,"cosineWithStream":0.4259},{"at":"l1.attn","addNorm":4.6957,"ratioToStream":0.1825,"cosineWithStream":0.013},{"at":"l1.mlp","addNorm":13.2366,"ratioToStream":0.4054,"cosineWithStream":0.2651}]},"untrained":{"steps":[{"at":"embed","norm":0.1623},{"at":"l0.attn","norm":0.1629},{"at":"l0.mlp","norm":0.1661},{"at":"l1.attn","norm":0.1667},{"at":"l1.mlp","norm":0.1694}],"adds":[{"at":"l0.attn","addNorm":0.0127,"ratioToStream":0.0799,"cosineWithStream":0.0021},{"at":"l0.mlp","addNorm":0.0278,"ratioToStream":0.1728,"cosineWithStream":0.0224},{"at":"l1.attn","addNorm":0.0138,"ratioToStream":0.0845,"cosineWithStream":0.0061},{"at":"l1.mlp","addNorm":0.0273,"ratioToStream":0.1664,"cosineWithStream":0.0195}]}},"lens":{"about":"Logit lens (nostalgebraist 2020, a blog post, not a peer-reviewed paper): apply the model's final layer norm and output matrix to the stream after each sub-step, as if the network ended there, and score those predictions on the 6144 held-out targets. loss in nats; top1 is the share of targets that are the first choice; copy and local split the targets as the corpus labels them. The last row is the model's real output (equal to its evaluated loss, 0.687407). normOnly skips the final norm's learned gain and bias. untrained is the control. The readout was trained only on the last stream, so an early row measures what that readout can decode from an earlier stream; a poor score there does not prove the information is absent.","trained":[{"at":"embed","loss":2.3304,"top1":0.3963,"copyAccuracy":0.1245,"copyLoss":3.2372,"localAccuracy":0.4207,"localLoss":2.2491},{"at":"l0.attn","loss":1.7256,"top1":0.5094,"copyAccuracy":0.2095,"copyLoss":3.8022,"localAccuracy":0.5364,"localLoss":1.5392},{"at":"l0.mlp","loss":0.8631,"top1":0.6792,"copyAccuracy":0.4289,"copyLoss":1.9902,"localAccuracy":0.7017,"localLoss":0.7619},{"at":"l1.attn","loss":0.7496,"top1":0.7077,"copyAccuracy":0.8162,"copyLoss":0.6379,"localAccuracy":0.6979,"localLoss":0.7596},{"at":"l1.mlp","loss":0.6874,"top1":0.7243,"copyAccuracy":0.9921,"copyLoss":0.056,"localAccuracy":0.7002,"localLoss":0.7441}],"normOnly":[{"at":"embed","loss":2.2796,"top1":0.3192,"copyAccuracy":0.085,"copyLoss":3.1434,"localAccuracy":0.3402,"localLoss":2.2021},{"at":"l0.attn","loss":2.0382,"top1":0.4048,"copyAccuracy":0.0613,"copyLoss":3.8954,"localAccuracy":0.4356,"localLoss":1.8716},{"at":"l0.mlp","loss":0.9585,"top1":0.6548,"copyAccuracy":0.1877,"copyLoss":2.418,"localAccuracy":0.6967,"localLoss":0.8275},{"at":"l1.attn","loss":0.8651,"top1":0.6813,"copyAccuracy":0.5158,"copyLoss":1.3506,"localAccuracy":0.6962,"localLoss":0.8215},{"at":"l1.mlp","loss":0.7367,"top1":0.7209,"copyAccuracy":0.9921,"copyLoss":0.1363,"localAccuracy":0.6965,"localLoss":0.7906}],"untrained":[{"at":"embed","loss":3.8418,"top1":0.0202,"copyAccuracy":0.0711,"copyLoss":3.8317,"localAccuracy":0.0156,"localLoss":3.8427},{"at":"l0.attn","loss":3.8427,"top1":0.021,"copyAccuracy":0.0791,"copyLoss":3.8335,"localAccuracy":0.0158,"localLoss":3.8436},{"at":"l0.mlp","loss":3.8439,"top1":0.0243,"copyAccuracy":0.0771,"copyLoss":3.8321,"localAccuracy":0.0195,"localLoss":3.845},{"at":"l1.attn","loss":3.8417,"top1":0.0247,"copyAccuracy":0.0771,"copyLoss":3.8308,"localAccuracy":0.02,"localLoss":3.8427},{"at":"l1.mlp","loss":3.8414,"top1":0.0244,"copyAccuracy":0.0692,"copyLoss":3.8297,"localAccuracy":0.0204,"localLoss":3.8425}]},"ablations":{"about":"Knockouts on 256 held-out rows of 24 positions (6144 next-word targets: 506 look-back 'copy' targets, 5638 'local' targets), drawn from the corpus generator with the held-out seed, a separate stream from training. Each row removes one part and scores the whole network: l<L>.attn is layer L's attention sub-layer output, l<L>.h<H> is one head's 16-number slice of the attention output before the output projection, l<L>.mlp is layer L's MLP output, all.attn and all.mlp remove that sub-layer in every layer. Modes: zero replaces it with zeros; mean with its average over all held-out positions; shuffle with the value the same part produced at the same position of the next held-out row (real activations, wrong context). dLoss is the change in held-out loss. Zero knockouts are checked in the tests against zeroing the matching weights. The untrained network is the control: its knockouts should move nothing. Single knockouts on one model: a part whose removal costs little may still matter when another part is also gone, and these numbers do not show how a part does its job.","trained":{"base":{"loss":0.6874,"top1":0.7243,"copyAccuracy":0.9921,"copyLoss":0.056,"localAccuracy":0.7002,"localLoss":0.7441},"rows":[{"name":"l0.attn","mode":"zero","loss":2.1443,"top1":0.5039,"copyAccuracy":0.2925,"copyLoss":2.9298,"localAccuracy":0.5229,"localLoss":2.0738,"dLoss":1.4569},{"name":"l0.h0","mode":"zero","loss":0.9781,"top1":0.6742,"copyAccuracy":0.9407,"copyLoss":0.3596,"localAccuracy":0.6502,"localLoss":1.0337,"dLoss":0.2907},{"name":"l0.h1","mode":"zero","loss":1.7457,"top1":0.5775,"copyAccuracy":0.3755,"copyLoss":3.02,"localAccuracy":0.5956,"localLoss":1.6313,"dLoss":1.0583},{"name":"l0.mlp","mode":"zero","loss":5.9605,"top1":0.1211,"copyAccuracy":0.4783,"copyLoss":2.4248,"localAccuracy":0.089,"localLoss":6.2778,"dLoss":5.2731},{"name":"l0.attn","mode":"mean","loss":2.1843,"top1":0.4976,"copyAccuracy":0.2925,"copyLoss":3.3171,"localAccuracy":0.516,"localLoss":2.0826,"dLoss":1.4969},{"name":"l0.h0","mode":"mean","loss":1.0245,"top1":0.666,"copyAccuracy":0.9387,"copyLoss":0.3604,"localAccuracy":0.6415,"localLoss":1.0841,"dLoss":0.3371},{"name":"l0.h1","mode":"mean","loss":1.7378,"top1":0.5799,"copyAccuracy":0.3538,"copyLoss":3.2664,"localAccuracy":0.6002,"localLoss":1.6006,"dLoss":1.0504},{"name":"l0.mlp","mode":"mean","loss":3.5623,"top1":0.1631,"copyAccuracy":0.0435,"copyLoss":3.5564,"localAccuracy":0.1738,"localLoss":3.5629,"dLoss":2.8749},{"name":"l0.attn","mode":"shuffle","loss":2.8893,"top1":0.4704,"copyAccuracy":0.3281,"copyLoss":3.8418,"localAccuracy":0.4832,"localLoss":2.8038,"dLoss":2.2019},{"name":"l0.h0","mode":"shuffle","loss":1.1975,"top1":0.6484,"copyAccuracy":0.7905,"copyLoss":0.8591,"localAccuracy":0.6357,"localLoss":1.2278,"dLoss":0.5101},{"name":"l0.h1","mode":"shuffle","loss":2.2389,"top1":0.5456,"copyAccuracy":0.4269,"copyLoss":3.4662,"localAccuracy":0.5562,"localLoss":2.1287,"dLoss":1.5514},{"name":"l0.mlp","mode":"shuffle","loss":9.1978,"top1":0.153,"copyAccuracy":0.0395,"copyLoss":11.194,"localAccuracy":0.1632,"localLoss":9.0186,"dLoss":8.5103},{"name":"l1.attn","mode":"zero","loss":0.8486,"top1":0.6794,"copyAccuracy":0.4071,"copyLoss":2.0122,"localAccuracy":0.7038,"localLoss":0.7442,"dLoss":0.1612},{"name":"l1.h0","mode":"zero","loss":0.6872,"top1":0.7262,"copyAccuracy":0.9921,"copyLoss":0.0569,"localAccuracy":0.7024,"localLoss":0.7438,"dLoss":-0.0002},{"name":"l1.h1","mode":"zero","loss":0.8491,"top1":0.6771,"copyAccuracy":0.4032,"copyLoss":2.0165,"localAccuracy":0.7017,"localLoss":0.7443,"dLoss":0.1617},{"name":"l1.mlp","mode":"zero","loss":0.7496,"top1":0.7077,"copyAccuracy":0.8162,"copyLoss":0.6379,"localAccuracy":0.6979,"localLoss":0.7596,"dLoss":0.0622},{"name":"l1.attn","mode":"mean","loss":0.9144,"top1":0.6779,"copyAccuracy":0.3953,"copyLoss":2.8083,"localAccuracy":0.7033,"localLoss":0.7445,"dLoss":0.227},{"name":"l1.h0","mode":"mean","loss":0.6872,"top1":0.7271,"copyAccuracy":0.9921,"copyLoss":0.0569,"localAccuracy":0.7033,"localLoss":0.7438,"dLoss":-0.0002},{"name":"l1.h1","mode":"mean","loss":0.9235,"top1":0.6764,"copyAccuracy":0.4012,"copyLoss":2.9181,"localAccuracy":0.7011,"localLoss":0.7445,"dLoss":0.2361},{"name":"l1.mlp","mode":"mean","loss":0.7522,"top1":0.7095,"copyAccuracy":0.7648,"copyLoss":0.708,"localAccuracy":0.7045,"localLoss":0.7562,"dLoss":0.0648},{"name":"l1.attn","mode":"shuffle","loss":0.9785,"top1":0.6733,"copyAccuracy":0.3933,"copyLoss":3.4996,"localAccuracy":0.6985,"localLoss":0.7523,"dLoss":0.2911},{"name":"l1.h0","mode":"shuffle","loss":0.6886,"top1":0.7253,"copyAccuracy":0.9921,"copyLoss":0.0643,"localAccuracy":0.7013,"localLoss":0.7446,"dLoss":0.0011},{"name":"l1.h1","mode":"shuffle","loss":0.9858,"top1":0.6733,"copyAccuracy":0.3972,"copyLoss":3.5976,"localAccuracy":0.6981,"localLoss":0.7514,"dLoss":0.2984},{"name":"l1.mlp","mode":"shuffle","loss":0.9934,"top1":0.6611,"copyAccuracy":0.5375,"copyLoss":2.0419,"localAccuracy":0.6722,"localLoss":0.8993,"dLoss":0.306},{"name":"all.attn","mode":"zero","loss":2.205,"top1":0.4932,"copyAccuracy":0.1719,"copyLoss":3.3834,"localAccuracy":0.522,"localLoss":2.0993,"dLoss":1.5176},{"name":"all.mlp","mode":"zero","loss":3.653,"top1":0.2332,"copyAccuracy":0.334,"copyLoss":2.508,"localAccuracy":0.2242,"localLoss":3.7558,"dLoss":2.9656},{"name":"all.attn","mode":"mean","loss":2.2231,"top1":0.4938,"copyAccuracy":0.1719,"copyLoss":3.7919,"localAccuracy":0.5227,"localLoss":2.0823,"dLoss":1.5357},{"name":"all.mlp","mode":"mean","loss":3.6973,"top1":0.151,"copyAccuracy":0,"copyLoss":3.6303,"localAccuracy":0.1646,"localLoss":3.7033,"dLoss":3.0099},{"name":"all.attn","mode":"shuffle","loss":2.9558,"top1":0.4463,"copyAccuracy":0.1502,"copyLoss":4.8454,"localAccuracy":0.4729,"localLoss":2.7862,"dLoss":2.2684},{"name":"all.mlp","mode":"shuffle","loss":9.4553,"top1":0.1509,"copyAccuracy":0.0395,"copyLoss":11.561,"localAccuracy":0.1609,"localLoss":9.2663,"dLoss":8.7678}]},"untrained":{"base":{"loss":3.8414,"top1":0.0244,"copyAccuracy":0.0692,"copyLoss":3.8297,"localAccuracy":0.0204,"localLoss":3.8425},"rows":[{"name":"l0.attn","mode":"zero","loss":3.8403,"top1":0.0265,"copyAccuracy":0.0692,"copyLoss":3.8282,"localAccuracy":0.0227,"localLoss":3.8414,"dLoss":-0.0011},{"name":"l0.h0","mode":"zero","loss":3.8402,"top1":0.0234,"copyAccuracy":0.0672,"copyLoss":3.8291,"localAccuracy":0.0195,"localLoss":3.8412,"dLoss":-0.0012},{"name":"l0.h1","mode":"zero","loss":3.8415,"top1":0.0262,"copyAccuracy":0.0692,"copyLoss":3.8288,"localAccuracy":0.0223,"localLoss":3.8426,"dLoss":0.0001},{"name":"l0.mlp","mode":"zero","loss":3.8394,"top1":0.0299,"copyAccuracy":0.0711,"copyLoss":3.8317,"localAccuracy":0.0263,"localLoss":3.8401,"dLoss":-0.002},{"name":"l0.attn","mode":"mean","loss":3.8411,"top1":0.0241,"copyAccuracy":0.0652,"copyLoss":3.8295,"localAccuracy":0.0204,"localLoss":3.8422,"dLoss":-0.0003},{"name":"l0.h0","mode":"mean","loss":3.8412,"top1":0.0234,"copyAccuracy":0.0652,"copyLoss":3.8297,"localAccuracy":0.0197,"localLoss":3.8423,"dLoss":-0.0002},{"name":"l0.h1","mode":"mean","loss":3.8413,"top1":0.0246,"copyAccuracy":0.0692,"copyLoss":3.8296,"localAccuracy":0.0206,"localLoss":3.8423,"dLoss":-0.0001},{"name":"l0.mlp","mode":"mean","loss":3.8402,"top1":0.0273,"copyAccuracy":0.0751,"copyLoss":3.832,"localAccuracy":0.0231,"localLoss":3.8409,"dLoss":-0.0012},{"name":"l0.attn","mode":"shuffle","loss":3.8412,"top1":0.028,"copyAccuracy":0.0672,"copyLoss":3.8293,"localAccuracy":0.0245,"localLoss":3.8423,"dLoss":-0.0002},{"name":"l0.h0","mode":"shuffle","loss":3.8415,"top1":0.0244,"copyAccuracy":0.0692,"copyLoss":3.8298,"localAccuracy":0.0204,"localLoss":3.8426,"dLoss":0.0001},{"name":"l0.h1","mode":"shuffle","loss":3.8411,"top1":0.0264,"copyAccuracy":0.0692,"copyLoss":3.8292,"localAccuracy":0.0225,"localLoss":3.8422,"dLoss":-0.0003},{"name":"l0.mlp","mode":"shuffle","loss":3.8403,"top1":0.0262,"copyAccuracy":0.0632,"copyLoss":3.8322,"localAccuracy":0.0229,"localLoss":3.8411,"dLoss":-0.0011},{"name":"l1.attn","mode":"zero","loss":3.8435,"top1":0.026,"copyAccuracy":0.0672,"copyLoss":3.8309,"localAccuracy":0.0223,"localLoss":3.8446,"dLoss":0.0021},{"name":"l1.h0","mode":"zero","loss":3.8419,"top1":0.0241,"copyAccuracy":0.0672,"copyLoss":3.8301,"localAccuracy":0.0202,"localLoss":3.843,"dLoss":0.0005},{"name":"l1.h1","mode":"zero","loss":3.8429,"top1":0.0236,"copyAccuracy":0.0652,"copyLoss":3.8305,"localAccuracy":0.0199,"localLoss":3.844,"dLoss":0.0015},{"name":"l1.mlp","mode":"zero","loss":3.8417,"top1":0.0247,"copyAccuracy":0.0771,"copyLoss":3.8308,"localAccuracy":0.02,"localLoss":3.8427,"dLoss":0.0003},{"name":"l1.attn","mode":"mean","loss":3.8432,"top1":0.0238,"copyAccuracy":0.0652,"copyLoss":3.8309,"localAccuracy":0.02,"localLoss":3.8443,"dLoss":0.0018},{"name":"l1.h0","mode":"mean","loss":3.8419,"top1":0.0308,"copyAccuracy":0.0672,"copyLoss":3.83,"localAccuracy":0.0275,"localLoss":3.8429,"dLoss":0.0005},{"name":"l1.h1","mode":"mean","loss":3.8427,"top1":0.0233,"copyAccuracy":0.0652,"copyLoss":3.8307,"localAccuracy":0.0195,"localLoss":3.8438,"dLoss":0.0013},{"name":"l1.mlp","mode":"mean","loss":3.8434,"top1":0.0246,"copyAccuracy":0.0771,"copyLoss":3.831,"localAccuracy":0.0199,"localLoss":3.8445,"dLoss":0.002},{"name":"l1.attn","mode":"shuffle","loss":3.842,"top1":0.0262,"copyAccuracy":0.0652,"copyLoss":3.8305,"localAccuracy":0.0227,"localLoss":3.8431,"dLoss":0.0006},{"name":"l1.h0","mode":"shuffle","loss":3.8413,"top1":0.0259,"copyAccuracy":0.0652,"copyLoss":3.8299,"localAccuracy":0.0223,"localLoss":3.8423,"dLoss":-0.0001},{"name":"l1.h1","mode":"shuffle","loss":3.8422,"top1":0.0249,"copyAccuracy":0.0652,"copyLoss":3.8304,"localAccuracy":0.0213,"localLoss":3.8432,"dLoss":0.0008},{"name":"l1.mlp","mode":"shuffle","loss":3.8416,"top1":0.0283,"copyAccuracy":0.0751,"copyLoss":3.8293,"localAccuracy":0.0241,"localLoss":3.8427,"dLoss":0.0002},{"name":"all.attn","mode":"zero","loss":3.8422,"top1":0.0236,"copyAccuracy":0.0711,"copyLoss":3.8291,"localAccuracy":0.0193,"localLoss":3.8433,"dLoss":0.0008},{"name":"all.mlp","mode":"zero","loss":3.8401,"top1":0.0231,"copyAccuracy":0.0791,"copyLoss":3.8323,"localAccuracy":0.0181,"localLoss":3.8408,"dLoss":-0.0013},{"name":"all.attn","mode":"mean","loss":3.8428,"top1":0.0223,"copyAccuracy":0.0652,"copyLoss":3.8304,"localAccuracy":0.0184,"localLoss":3.844,"dLoss":0.0014},{"name":"all.mlp","mode":"mean","loss":3.8423,"top1":0.0226,"copyAccuracy":0.081,"copyLoss":3.8331,"localAccuracy":0.0174,"localLoss":3.8431,"dLoss":0.0009},{"name":"all.attn","mode":"shuffle","loss":3.8419,"top1":0.0257,"copyAccuracy":0.0672,"copyLoss":3.8298,"localAccuracy":0.022,"localLoss":3.8429,"dLoss":0.0004},{"name":"all.mlp","mode":"shuffle","loss":3.8404,"top1":0.0313,"copyAccuracy":0.0751,"copyLoss":3.8314,"localAccuracy":0.0273,"localLoss":3.8412,"dLoss":-0.001}]}},"residual":{"about":"What the residual add does, measured: each row runs the network with one sub-layer's output replacing the stream instead of being added to it (no residual connection around that one sub-layer, everything else intact), on 256 held-out rows of 24 positions (6144 next-word targets: 506 look-back 'copy' targets, 5638 'local' targets), drawn from the corpus generator with the held-out seed, a separate stream from training. Compare with the zero knockout of the same sub-layer (no output, residual kept). Where a sub-layer's output is much larger than the stream it joins (see the stream section), replacing and adding give nearly the same stream, so dropping that one residual connection costs little. Trained once with the residual connections in place: this does not show how a network trained without them would do.","trained":{"base":{"loss":0.6874,"top1":0.7243,"copyAccuracy":0.9921,"copyLoss":0.056,"localAccuracy":0.7002,"localLoss":0.7441},"rows":[{"name":"l0.attn","mode":"replace","loss":4.9254,"top1":0.2612,"copyAccuracy":0.1818,"copyLoss":10.991,"localAccuracy":0.2684,"localLoss":4.381,"dLoss":4.2379},{"name":"l0.mlp","mode":"replace","loss":0.6905,"top1":0.7264,"copyAccuracy":0.9881,"copyLoss":0.075,"localAccuracy":0.7029,"localLoss":0.7458,"dLoss":0.0031},{"name":"l1.attn","mode":"replace","loss":6.6192,"top1":0.0884,"copyAccuracy":0.5731,"copyLoss":2.6513,"localAccuracy":0.0449,"localLoss":6.9753,"dLoss":5.9318},{"name":"l1.mlp","mode":"replace","loss":2.8514,"top1":0.4046,"copyAccuracy":0.6877,"copyLoss":1.3424,"localAccuracy":0.3792,"localLoss":2.9869,"dLoss":2.164}]},"untrained":{"base":{"loss":3.8414,"top1":0.0244,"copyAccuracy":0.0692,"copyLoss":3.8297,"localAccuracy":0.0204,"localLoss":3.8425},"rows":[{"name":"l0.attn","mode":"replace","loss":3.8556,"top1":0.0176,"copyAccuracy":0.0336,"copyLoss":3.8357,"localAccuracy":0.0161,"localLoss":3.8573,"dLoss":0.0142},{"name":"l0.mlp","mode":"replace","loss":3.858,"top1":0.0164,"copyAccuracy":0.0316,"copyLoss":3.8147,"localAccuracy":0.0151,"localLoss":3.8619,"dLoss":0.0166},{"name":"l1.attn","mode":"replace","loss":3.8343,"top1":0.0208,"copyAccuracy":0.0237,"copyLoss":3.8357,"localAccuracy":0.0206,"localLoss":3.8342,"dLoss":-0.0071},{"name":"l1.mlp","mode":"replace","loss":3.8319,"top1":0.027,"copyAccuracy":0.0119,"copyLoss":3.8268,"localAccuracy":0.0284,"localLoss":3.8323,"dLoss":-0.0095}]},"zeroForComparison":[{"name":"l0.attn","mode":"zero","loss":2.1443,"top1":0.5039,"copyAccuracy":0.2925,"copyLoss":2.9298,"localAccuracy":0.5229,"localLoss":2.0738,"dLoss":1.4569},{"name":"l0.mlp","mode":"zero","loss":5.9605,"top1":0.1211,"copyAccuracy":0.4783,"copyLoss":2.4248,"localAccuracy":0.089,"localLoss":6.2778,"dLoss":5.2731},{"name":"l1.attn","mode":"zero","loss":0.8486,"top1":0.6794,"copyAccuracy":0.4071,"copyLoss":2.0122,"localAccuracy":0.7038,"localLoss":0.7442,"dLoss":0.1612},{"name":"l1.mlp","mode":"zero","loss":0.7496,"top1":0.7077,"copyAccuracy":0.8162,"copyLoss":0.6379,"localAccuracy":0.6979,"localLoss":0.7596,"dLoss":0.0622}]},"cites":{"about":"What the code does, checked against the papers. The block follows the GPT-2 arrangement, not the original: Vaswani et al. 2017 (arXiv 1706.03762) put the layer norm after the residual add, LayerNorm(x + Sublayer(x)); Radford et al. 2019 (GPT-2 report) moved layer norm to the input of each sub-block and added one after the final block, which is what this model does (h + Sublayer(LN(h)), then a final LN). Other differences from Vaswani et al.: GELU in place of ReLU, an MLP width of 2 x dModel (theirs is 4 x, 2048 for 512), learned position embeddings (they used sines and cosines, and report learned ones gave nearly identical results), and an output matrix not shared with the word table (they share it). Residual output projections start smaller by 1/sqrt(2 x layers), the GPT-2 report's 1/sqrt(N) with N residual layers.","vaswani2017":"arXiv 1706.03762","radford2019":"Language Models are Unsupervised Multitask Learners (OpenAI report)","nostalgebraist2020":"interpreting GPT: the logit lens (LessWrong blog post, 31 August 2020)"},"seconds":7.97}