{"about":"Lesson L14, scaling. Computed by tools/school/fill_scaling.mjs from system/school/scaling.mjs and the attention lab's own model and trainer. Do not edit by hand. Losses are in nats (natural log).","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","floor":0.6585,"grid":{"about":"Widths 8, 16, 32 and 64 with feed-forward width twice the model width and 2 heads, depths 1, 2 and 3: 12 models, each trained 4000 steps of 16 rows of 24 words (fresh rows every step, data seed 11, model seed 7, learning rate 0.01 with 30 warmup steps then constant, the lesson model's recipe unchanged), scored on the same 256 held-out rows at steps 25, 50, 100, 200, 300, 500, 750, 1000, 1500, 2000, 3000, 4000. params.nonEmbedding leaves out the token table, the position table and the untied readout (Kaplan et al. count N this way); params.total counts everything. flops: training compute so far from the model's own FLOPS count (3 x forward). sixND: Kaplan's 6 x N x D estimate, for comparison. Width 8 has 4 numbers per head. The learning rate and initial scale were tuned for width 32 and not re-tuned per size, which can flatter the middle of the grid. One seed per point except the noise runs.","runs":[{"name":"d8L1","config":{"dModel":8,"nLayers":1,"nHeads":2,"dFF":16,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":1544,"nonEmbedding":616,"embedding":928,"perLayer":600},"flopsPerStep":2488320,"final":{"step":4000,"heldOut":0.81276,"excess":0.15426,"copyAccuracy":0.35968,"flops":9953280000,"tokens":1536000},"best":{"step":4000,"heldOut":0.81276,"excess":0.15426,"copyAccuracy":0.35968},"start":3.8329,"checkpoints":[{"step":25,"flops":62208000,"sixND":35481600,"tokens":9600,"heldOut":3.081,"excess":2.4225,"copyAccuracy":0.049407,"localAccuracy":0.33434,"train":3.6075},{"step":50,"flops":124416000,"sixND":70963200,"tokens":19200,"heldOut":1.8358,"excess":1.1773,"copyAccuracy":0.14427,"localAccuracy":0.50479,"train":2.3642},{"step":100,"flops":248832000,"sixND":141926400,"tokens":38400,"heldOut":1.2444,"excess":0.58587,"copyAccuracy":0.1996,"localAccuracy":0.59968,"train":1.43},{"step":200,"flops":497664000,"sixND":283852800,"tokens":76800,"heldOut":1.054,"excess":0.39548,"copyAccuracy":0.17589,"localAccuracy":0.64562,"train":1.114},{"step":300,"flops":746496000,"sixND":425779200,"tokens":115200,"heldOut":0.98686,"excess":0.32836,"copyAccuracy":0.21542,"localAccuracy":0.66744,"train":0.98781},{"step":500,"flops":1244160000,"sixND":709632000,"tokens":192000,"heldOut":0.91508,"excess":0.25658,"copyAccuracy":0.2253,"localAccuracy":0.68429,"train":0.91175},{"step":750,"flops":1866240000,"sixND":1064448000,"tokens":288000,"heldOut":0.88213,"excess":0.22363,"copyAccuracy":0.16601,"localAccuracy":0.69315,"train":0.86815},{"step":1000,"flops":2488320000,"sixND":1419264000,"tokens":384000,"heldOut":0.87263,"excess":0.21413,"copyAccuracy":0.2332,"localAccuracy":0.69475,"train":0.85467},{"step":1500,"flops":3732480000,"sixND":2128896000,"tokens":576000,"heldOut":0.85577,"excess":0.19727,"copyAccuracy":0.3083,"localAccuracy":0.69706,"train":0.83614},{"step":2000,"flops":4976640000,"sixND":2838528000,"tokens":768000,"heldOut":0.84017,"excess":0.18167,"copyAccuracy":0.33202,"localAccuracy":0.69901,"train":0.82424},{"step":3000,"flops":7464960000,"sixND":4257792000,"tokens":1152000,"heldOut":0.82503,"excess":0.16653,"copyAccuracy":0.32411,"localAccuracy":0.70273,"train":0.81585},{"step":4000,"flops":9953280000,"sixND":5677056000,"tokens":1536000,"heldOut":0.81276,"excess":0.15426,"copyAccuracy":0.35968,"localAccuracy":0.70167,"train":0.80783}],"seconds":8.424},{"name":"d8L2","config":{"dModel":8,"nLayers":2,"nHeads":2,"dFF":16,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":2144,"nonEmbedding":1216,"embedding":928,"perLayer":600},"flopsPerStep":4128768,"final":{"step":4000,"heldOut":0.73284,"excess":0.074342,"copyAccuracy":0.74308,"flops":16515072000,"tokens":1536000},"best":{"step":4000,"heldOut":0.73284,"excess":0.074342,"copyAccuracy":0.74308},"start":3.8273,"checkpoints":[{"step":25,"flops":103219200,"sixND":70041600,"tokens":9600,"heldOut":3.0715,"excess":2.413,"copyAccuracy":0.13043,"localAccuracy":0.42072,"train":3.5906},{"step":50,"flops":206438400,"sixND":140083200,"tokens":19200,"heldOut":1.711,"excess":1.0525,"copyAccuracy":0.15613,"localAccuracy":0.55179,"train":2.3082},{"step":100,"flops":412876800,"sixND":280166400,"tokens":38400,"heldOut":1.2142,"excess":0.55566,"copyAccuracy":0.20751,"localAccuracy":0.60536,"train":1.3644},{"step":200,"flops":825753600,"sixND":560332800,"tokens":76800,"heldOut":1.0266,"excess":0.36811,"copyAccuracy":0.16798,"localAccuracy":0.65945,"train":1.0559},{"step":300,"flops":1238630400,"sixND":840499200,"tokens":115200,"heldOut":0.93588,"excess":0.27738,"copyAccuracy":0.16601,"localAccuracy":0.68677,"train":0.93},{"step":500,"flops":2064384000,"sixND":1400832000,"tokens":192000,"heldOut":0.89052,"excess":0.23202,"copyAccuracy":0.20751,"localAccuracy":0.69244,"train":0.88181},{"step":750,"flops":3096576000,"sixND":2101248000,"tokens":288000,"heldOut":0.86566,"excess":0.20716,"copyAccuracy":0.27075,"localAccuracy":0.69794,"train":0.85564},{"step":1000,"flops":4128768000,"sixND":2801664000,"tokens":384000,"heldOut":0.85336,"excess":0.19486,"copyAccuracy":0.32016,"localAccuracy":0.7022,"train":0.83911},{"step":1500,"flops":6193152000,"sixND":4202496000,"tokens":576000,"heldOut":0.84869,"excess":0.19019,"copyAccuracy":0.29051,"localAccuracy":0.69794,"train":0.82345},{"step":2000,"flops":8257536000,"sixND":5603328000,"tokens":768000,"heldOut":0.82949,"excess":0.17099,"copyAccuracy":0.35968,"localAccuracy":0.70167,"train":0.81421},{"step":3000,"flops":12386304000,"sixND":8404992000,"tokens":1152000,"heldOut":0.76068,"excess":0.10218,"copyAccuracy":0.73123,"localAccuracy":0.70752,"train":0.78749},{"step":4000,"flops":16515072000,"sixND":11206656000,"tokens":1536000,"heldOut":0.73284,"excess":0.074342,"copyAccuracy":0.74308,"localAccuracy":0.70362,"train":0.72169}],"seconds":14.439},{"name":"d8L3","config":{"dModel":8,"nLayers":3,"nHeads":2,"dFF":16,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":2744,"nonEmbedding":1816,"embedding":928,"perLayer":600},"flopsPerStep":5769216,"final":{"step":4000,"heldOut":0.75729,"excess":0.098791,"copyAccuracy":0.67589,"flops":23076864000,"tokens":1536000},"best":{"step":4000,"heldOut":0.75729,"excess":0.098791,"copyAccuracy":0.67589},"start":3.8291,"checkpoints":[{"step":25,"flops":144230400,"sixND":104601600,"tokens":9600,"heldOut":3.0656,"excess":2.4071,"copyAccuracy":0.043478,"localAccuracy":0.44094,"train":3.5911},{"step":50,"flops":288460800,"sixND":209203200,"tokens":19200,"heldOut":1.7379,"excess":1.0794,"copyAccuracy":0.12648,"localAccuracy":0.55126,"train":2.3079},{"step":100,"flops":576921600,"sixND":418406400,"tokens":38400,"heldOut":1.2178,"excess":0.55934,"copyAccuracy":0.1996,"localAccuracy":0.6089,"train":1.3925},{"step":200,"flops":1153843200,"sixND":836812800,"tokens":76800,"heldOut":1.016,"excess":0.35745,"copyAccuracy":0.17391,"localAccuracy":0.65945,"train":1.0652},{"step":300,"flops":1730764800,"sixND":1255219200,"tokens":115200,"heldOut":0.93412,"excess":0.27562,"copyAccuracy":0.19368,"localAccuracy":0.6834,"train":0.95366},{"step":500,"flops":2884608000,"sixND":2092032000,"tokens":192000,"heldOut":0.90648,"excess":0.24798,"copyAccuracy":0.23715,"localAccuracy":0.69564,"train":0.90044},{"step":750,"flops":4326912000,"sixND":3138048000,"tokens":288000,"heldOut":0.89934,"excess":0.24084,"copyAccuracy":0.24506,"localAccuracy":0.69156,"train":0.87018},{"step":1000,"flops":5769216000,"sixND":4184064000,"tokens":384000,"heldOut":0.86993,"excess":0.21143,"copyAccuracy":0.30435,"localAccuracy":0.69777,"train":0.84662},{"step":1500,"flops":8653824000,"sixND":6276096000,"tokens":576000,"heldOut":0.82798,"excess":0.16948,"copyAccuracy":0.40316,"localAccuracy":0.69741,"train":0.83162},{"step":2000,"flops":11538432000,"sixND":8368128000,"tokens":768000,"heldOut":0.79415,"excess":0.13565,"copyAccuracy":0.61462,"localAccuracy":0.6983,"train":0.80067},{"step":3000,"flops":17307648000,"sixND":12552192000,"tokens":1152000,"heldOut":0.76148,"excess":0.10298,"copyAccuracy":0.67589,"localAccuracy":0.70344,"train":0.7653},{"step":4000,"flops":23076864000,"sixND":16736256000,"tokens":1536000,"heldOut":0.75729,"excess":0.098791,"copyAccuracy":0.67589,"localAccuracy":0.70078,"train":0.74862}],"seconds":19.618},{"name":"d16L1","config":{"dModel":16,"nLayers":1,"nHeads":2,"dFF":32,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":4112,"nonEmbedding":2256,"embedding":1856,"perLayer":2224},"flopsPerStep":7335936,"final":{"step":4000,"heldOut":0.68473,"excess":0.026235,"copyAccuracy":0.96443,"flops":29343744000,"tokens":1536000},"best":{"step":4000,"heldOut":0.68473,"excess":0.026235,"copyAccuracy":0.96443},"start":3.8259,"checkpoints":[{"step":25,"flops":183398400,"sixND":129945600,"tokens":9600,"heldOut":2.4489,"excess":1.7904,"copyAccuracy":0.16403,"localAccuracy":0.51117,"train":3.3553},{"step":50,"flops":366796800,"sixND":259891200,"tokens":19200,"heldOut":1.2831,"excess":0.62455,"copyAccuracy":0.19565,"localAccuracy":0.59276,"train":1.6813},{"step":100,"flops":733593600,"sixND":519782400,"tokens":38400,"heldOut":1.0617,"excess":0.40317,"copyAccuracy":0.17984,"localAccuracy":0.64633,"train":1.1305},{"step":200,"flops":1467187200,"sixND":1039564800,"tokens":76800,"heldOut":0.90533,"excess":0.24683,"copyAccuracy":0.19368,"localAccuracy":0.68819,"train":0.93215},{"step":300,"flops":2200780800,"sixND":1559347200,"tokens":115200,"heldOut":0.87618,"excess":0.21768,"copyAccuracy":0.2332,"localAccuracy":0.69706,"train":0.86502},{"step":500,"flops":3667968000,"sixND":2598912000,"tokens":192000,"heldOut":0.85159,"excess":0.19309,"copyAccuracy":0.27668,"localAccuracy":0.70451,"train":0.84127},{"step":750,"flops":5501952000,"sixND":3898368000,"tokens":288000,"heldOut":0.78486,"excess":0.12636,"copyAccuracy":0.65217,"localAccuracy":0.69847,"train":0.79375},{"step":1000,"flops":7335936000,"sixND":5197824000,"tokens":384000,"heldOut":0.77012,"excess":0.11162,"copyAccuracy":0.70158,"localAccuracy":0.70326,"train":0.73502},{"step":1500,"flops":11003904000,"sixND":7796736000,"tokens":576000,"heldOut":0.75021,"excess":0.091707,"copyAccuracy":0.71542,"localAccuracy":0.70255,"train":0.7235},{"step":2000,"flops":14671872000,"sixND":10395648000,"tokens":768000,"heldOut":0.74077,"excess":0.082269,"copyAccuracy":0.78656,"localAccuracy":0.70149,"train":0.71617},{"step":3000,"flops":22007808000,"sixND":15593472000,"tokens":1152000,"heldOut":0.69943,"excess":0.040933,"copyAccuracy":0.96245,"localAccuracy":0.70114,"train":0.67741},{"step":4000,"flops":29343744000,"sixND":20791296000,"tokens":1536000,"heldOut":0.68473,"excess":0.026235,"copyAccuracy":0.96443,"localAccuracy":0.70255,"train":0.6659}],"seconds":16.911},{"name":"d16L2","config":{"dModel":16,"nLayers":2,"nHeads":2,"dFF":32,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":6336,"nonEmbedding":4480,"embedding":1856,"perLayer":2224},"flopsPerStep":12976128,"final":{"step":4000,"heldOut":0.67458,"excess":0.016079,"copyAccuracy":0.99605,"flops":51904512000,"tokens":1536000},"best":{"step":4000,"heldOut":0.67458,"excess":0.016079,"copyAccuracy":0.99605},"start":3.8236,"checkpoints":[{"step":25,"flops":324403200,"sixND":258048000,"tokens":9600,"heldOut":2.4605,"excess":1.802,"copyAccuracy":0.16798,"localAccuracy":0.43118,"train":3.3514},{"step":50,"flops":648806400,"sixND":516096000,"tokens":19200,"heldOut":1.358,"excess":0.69947,"copyAccuracy":0.18577,"localAccuracy":0.59844,"train":1.7601},{"step":100,"flops":1297612800,"sixND":1032192000,"tokens":38400,"heldOut":1.0514,"excess":0.39294,"copyAccuracy":0.16008,"localAccuracy":0.64172,"train":1.1555},{"step":200,"flops":2595225600,"sixND":2064384000,"tokens":76800,"heldOut":0.90914,"excess":0.25064,"copyAccuracy":0.16403,"localAccuracy":0.68393,"train":0.93386},{"step":300,"flops":3892838400,"sixND":3096576000,"tokens":115200,"heldOut":0.88216,"excess":0.22366,"copyAccuracy":0.16996,"localAccuracy":0.69546,"train":0.87425},{"step":500,"flops":6488064000,"sixND":5160960000,"tokens":192000,"heldOut":0.8821,"excess":0.2236,"copyAccuracy":0.26087,"localAccuracy":0.69741,"train":0.85309},{"step":750,"flops":9732096000,"sixND":7741440000,"tokens":288000,"heldOut":0.79805,"excess":0.13955,"copyAccuracy":0.50593,"localAccuracy":0.69883,"train":0.81817},{"step":1000,"flops":12976128000,"sixND":10321920000,"tokens":384000,"heldOut":0.73014,"excess":0.071644,"copyAccuracy":0.8498,"localAccuracy":0.70788,"train":0.73783},{"step":1500,"flops":19464192000,"sixND":15482880000,"tokens":576000,"heldOut":0.71528,"excess":0.056779,"copyAccuracy":0.96245,"localAccuracy":0.70238,"train":0.69682},{"step":2000,"flops":25952256000,"sixND":20643840000,"tokens":768000,"heldOut":0.70122,"excess":0.042716,"copyAccuracy":0.9664,"localAccuracy":0.70184,"train":0.6749},{"step":3000,"flops":38928384000,"sixND":30965760000,"tokens":1152000,"heldOut":0.68262,"excess":0.02412,"copyAccuracy":0.98419,"localAccuracy":0.70788,"train":0.66751},{"step":4000,"flops":51904512000,"sixND":41287680000,"tokens":1536000,"heldOut":0.67458,"excess":0.016079,"copyAccuracy":0.99605,"localAccuracy":0.7022,"train":0.66125}],"seconds":28.81},{"name":"d16L3","config":{"dModel":16,"nLayers":3,"nHeads":2,"dFF":32,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":8560,"nonEmbedding":6704,"embedding":1856,"perLayer":2224},"flopsPerStep":18616320,"final":{"step":4000,"heldOut":0.68486,"excess":0.02636,"copyAccuracy":0.99012,"flops":74465280000,"tokens":1536000},"best":{"step":3000,"heldOut":0.68328,"excess":0.024776,"copyAccuracy":1},"start":3.8257,"checkpoints":[{"step":25,"flops":465408000,"sixND":386150400,"tokens":9600,"heldOut":2.4519,"excess":1.7934,"copyAccuracy":0.17194,"localAccuracy":0.52749,"train":3.3574},{"step":50,"flops":930816000,"sixND":772300800,"tokens":19200,"heldOut":1.3604,"excess":0.70191,"copyAccuracy":0.14625,"localAccuracy":0.57928,"train":1.7236},{"step":100,"flops":1861632000,"sixND":1544601600,"tokens":38400,"heldOut":1.0187,"excess":0.36017,"copyAccuracy":0.18972,"localAccuracy":0.66797,"train":1.1477},{"step":200,"flops":3723264000,"sixND":3089203200,"tokens":76800,"heldOut":0.92078,"excess":0.26228,"copyAccuracy":0.16601,"localAccuracy":0.68801,"train":0.92934},{"step":300,"flops":5584896000,"sixND":4633804800,"tokens":115200,"heldOut":0.89103,"excess":0.23253,"copyAccuracy":0.2668,"localAccuracy":0.69173,"train":0.873},{"step":500,"flops":9308160000,"sixND":7723008000,"tokens":192000,"heldOut":0.86768,"excess":0.20918,"copyAccuracy":0.34783,"localAccuracy":0.69457,"train":0.8424},{"step":750,"flops":13962240000,"sixND":11584512000,"tokens":288000,"heldOut":0.77512,"excess":0.11662,"copyAccuracy":0.59684,"localAccuracy":0.69918,"train":0.80364},{"step":1000,"flops":18616320000,"sixND":15446016000,"tokens":384000,"heldOut":0.75069,"excess":0.092186,"copyAccuracy":0.71739,"localAccuracy":0.71213,"train":0.73726},{"step":1500,"flops":27924480000,"sixND":23169024000,"tokens":576000,"heldOut":0.71147,"excess":0.052969,"copyAccuracy":0.87747,"localAccuracy":0.70238,"train":0.71209},{"step":2000,"flops":37232640000,"sixND":30892032000,"tokens":768000,"heldOut":0.69281,"excess":0.034314,"copyAccuracy":0.98419,"localAccuracy":0.69865,"train":0.6722},{"step":3000,"flops":55848960000,"sixND":46338048000,"tokens":1152000,"heldOut":0.68328,"excess":0.024776,"copyAccuracy":1,"localAccuracy":0.70096,"train":0.66111},{"step":4000,"flops":74465280000,"sixND":61784064000,"tokens":1536000,"heldOut":0.68486,"excess":0.02636,"copyAccuracy":0.99012,"localAccuracy":0.69847,"train":0.65688}],"seconds":39.806},{"name":"d32L1","config":{"dModel":32,"nLayers":1,"nHeads":2,"dFF":64,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":12320,"nonEmbedding":8608,"embedding":3712,"perLayer":8544},"flopsPerStep":24109056,"final":{"step":4000,"heldOut":0.76645,"excess":0.10795,"copyAccuracy":0.57115,"flops":96436224000,"tokens":1536000},"best":{"step":3000,"heldOut":0.76475,"excess":0.10625,"copyAccuracy":0.56917},"start":3.8436,"checkpoints":[{"step":25,"flops":602726400,"sixND":495820800,"tokens":9600,"heldOut":1.7068,"excess":1.0483,"copyAccuracy":0.17589,"localAccuracy":0.56101,"train":3.0216},{"step":50,"flops":1205452800,"sixND":991641600,"tokens":19200,"heldOut":1.1592,"excess":0.50073,"copyAccuracy":0.21739,"localAccuracy":0.61582,"train":1.3024},{"step":100,"flops":2410905600,"sixND":1983283200,"tokens":38400,"heldOut":0.97431,"excess":0.31581,"copyAccuracy":0.18182,"localAccuracy":0.67134,"train":1.0356},{"step":200,"flops":4821811200,"sixND":3966566400,"tokens":76800,"heldOut":0.88086,"excess":0.22236,"copyAccuracy":0.17984,"localAccuracy":0.70096,"train":0.88696},{"step":300,"flops":7232716800,"sixND":5949849600,"tokens":115200,"heldOut":0.86417,"excess":0.20567,"copyAccuracy":0.19565,"localAccuracy":0.69635,"train":0.85292},{"step":500,"flops":12054528000,"sixND":9916416000,"tokens":192000,"heldOut":0.81728,"excess":0.15878,"copyAccuracy":0.43874,"localAccuracy":0.70184,"train":0.81341},{"step":750,"flops":18081792000,"sixND":14874624000,"tokens":288000,"heldOut":0.80938,"excess":0.15088,"copyAccuracy":0.41304,"localAccuracy":0.70043,"train":0.79014},{"step":1000,"flops":24109056000,"sixND":19832832000,"tokens":384000,"heldOut":0.78617,"excess":0.12767,"copyAccuracy":0.57312,"localAccuracy":0.70043,"train":0.77259},{"step":1500,"flops":36163584000,"sixND":29749248000,"tokens":576000,"heldOut":0.77798,"excess":0.11948,"copyAccuracy":0.57905,"localAccuracy":0.69777,"train":0.7622},{"step":2000,"flops":48218112000,"sixND":39665664000,"tokens":768000,"heldOut":0.78024,"excess":0.12174,"copyAccuracy":0.58103,"localAccuracy":0.69812,"train":0.75668},{"step":3000,"flops":72327168000,"sixND":59498496000,"tokens":1152000,"heldOut":0.76475,"excess":0.10625,"copyAccuracy":0.56917,"localAccuracy":0.70699,"train":0.75177},{"step":4000,"flops":96436224000,"sixND":79331328000,"tokens":1536000,"heldOut":0.76645,"excess":0.10795,"copyAccuracy":0.57115,"localAccuracy":0.70043,"train":0.74938}],"seconds":43.864},{"name":"d32L2","config":{"dModel":32,"nLayers":2,"nHeads":2,"dFF":64,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":20864,"nonEmbedding":17152,"embedding":3712,"perLayer":8544},"flopsPerStep":44826624,"final":{"step":4000,"heldOut":0.67348,"excess":0.014979,"copyAccuracy":0.99209,"flops":179306496000,"tokens":1536000},"best":{"step":4000,"heldOut":0.67348,"excess":0.014979,"copyAccuracy":0.99209},"start":3.8414,"checkpoints":[{"step":25,"flops":1120665600,"sixND":987955200,"tokens":9600,"heldOut":1.7252,"excess":1.0667,"copyAccuracy":0.15217,"localAccuracy":0.54878,"train":2.9816},{"step":50,"flops":2241331200,"sixND":1975910400,"tokens":19200,"heldOut":1.1656,"excess":0.50708,"copyAccuracy":0.21344,"localAccuracy":0.60784,"train":1.3223},{"step":100,"flops":4482662400,"sixND":3951820800,"tokens":38400,"heldOut":0.96542,"excess":0.30692,"copyAccuracy":0.15613,"localAccuracy":0.67435,"train":1.0402},{"step":200,"flops":8965324800,"sixND":7903641600,"tokens":76800,"heldOut":0.88124,"excess":0.22274,"copyAccuracy":0.21344,"localAccuracy":0.69759,"train":0.90257},{"step":300,"flops":13447987200,"sixND":11855462400,"tokens":115200,"heldOut":0.84356,"excess":0.18506,"copyAccuracy":0.33992,"localAccuracy":0.70255,"train":0.84187},{"step":500,"flops":22413312000,"sixND":19759104000,"tokens":192000,"heldOut":0.81818,"excess":0.15968,"copyAccuracy":0.42885,"localAccuracy":0.70575,"train":0.8205},{"step":750,"flops":33619968000,"sixND":29638656000,"tokens":288000,"heldOut":0.71501,"excess":0.056506,"copyAccuracy":0.96838,"localAccuracy":0.69918,"train":0.7604},{"step":1000,"flops":44826624000,"sixND":39518208000,"tokens":384000,"heldOut":0.69809,"excess":0.039594,"copyAccuracy":0.99605,"localAccuracy":0.70131,"train":0.68092},{"step":1500,"flops":67239936000,"sixND":59277312000,"tokens":576000,"heldOut":0.69442,"excess":0.035919,"copyAccuracy":0.98814,"localAccuracy":0.70043,"train":0.66928},{"step":2000,"flops":89653248000,"sixND":79036416000,"tokens":768000,"heldOut":0.68741,"excess":0.028907,"copyAccuracy":0.99209,"localAccuracy":0.70025,"train":0.66438},{"step":3000,"flops":134479872000,"sixND":118554624000,"tokens":1152000,"heldOut":0.6749,"excess":0.016396,"copyAccuracy":0.99802,"localAccuracy":0.70557,"train":0.65853},{"step":4000,"flops":179306496000,"sixND":158072832000,"tokens":1536000,"heldOut":0.67348,"excess":0.014979,"copyAccuracy":0.99209,"localAccuracy":0.70362,"train":0.65505}],"seconds":82.525},{"name":"d32L3","config":{"dModel":32,"nLayers":3,"nHeads":2,"dFF":64,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":29408,"nonEmbedding":25696,"embedding":3712,"perLayer":8544},"flopsPerStep":65544192,"final":{"step":4000,"heldOut":0.67044,"excess":0.011939,"copyAccuracy":1,"flops":262176768000,"tokens":1536000},"best":{"step":4000,"heldOut":0.67044,"excess":0.011939,"copyAccuracy":1},"start":3.8278,"checkpoints":[{"step":25,"flops":1638604800,"sixND":1480089600,"tokens":9600,"heldOut":1.6713,"excess":1.0128,"copyAccuracy":0.15217,"localAccuracy":0.55552,"train":2.9519},{"step":50,"flops":3277209600,"sixND":2960179200,"tokens":19200,"heldOut":1.1481,"excess":0.48959,"copyAccuracy":0.20356,"localAccuracy":0.61635,"train":1.3055},{"step":100,"flops":6554419200,"sixND":5920358400,"tokens":38400,"heldOut":0.97354,"excess":0.31504,"copyAccuracy":0.16206,"localAccuracy":0.68056,"train":1.0442},{"step":200,"flops":13108838400,"sixND":11840716800,"tokens":76800,"heldOut":0.89285,"excess":0.23435,"copyAccuracy":0.19763,"localAccuracy":0.69564,"train":0.90507},{"step":300,"flops":19663257600,"sixND":17761075200,"tokens":115200,"heldOut":0.8576,"excess":0.1991,"copyAccuracy":0.33992,"localAccuracy":0.70114,"train":0.85568},{"step":500,"flops":32772096000,"sixND":29601792000,"tokens":192000,"heldOut":0.82724,"excess":0.16874,"copyAccuracy":0.43874,"localAccuracy":0.70025,"train":0.82389},{"step":750,"flops":49158144000,"sixND":44402688000,"tokens":288000,"heldOut":0.72974,"excess":0.071242,"copyAccuracy":0.86957,"localAccuracy":0.69936,"train":0.74661},{"step":1000,"flops":65544192000,"sixND":59203584000,"tokens":384000,"heldOut":0.70429,"excess":0.045786,"copyAccuracy":0.98221,"localAccuracy":0.70255,"train":0.68934},{"step":1500,"flops":98316288000,"sixND":88805376000,"tokens":576000,"heldOut":0.69093,"excess":0.032432,"copyAccuracy":0.99802,"localAccuracy":0.7006,"train":0.67757},{"step":2000,"flops":131088384000,"sixND":118407168000,"tokens":768000,"heldOut":0.69098,"excess":0.032479,"copyAccuracy":0.99605,"localAccuracy":0.70096,"train":0.66586},{"step":3000,"flops":196632576000,"sixND":177610752000,"tokens":1152000,"heldOut":0.69171,"excess":0.033205,"copyAccuracy":0.99209,"localAccuracy":0.69918,"train":0.6613},{"step":4000,"flops":262176768000,"sixND":236814336000,"tokens":1536000,"heldOut":0.67044,"excess":0.011939,"copyAccuracy":1,"localAccuracy":0.7006,"train":0.65617}],"seconds":111.78},{"name":"d64L1","config":{"dModel":64,"nLayers":1,"nHeads":2,"dFF":128,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":41024,"nonEmbedding":33600,"embedding":7424,"perLayer":33472},"flopsPerStep":85966848,"final":{"step":4000,"heldOut":0.67981,"excess":0.021314,"copyAccuracy":0.96443,"flops":343867392000,"tokens":1536000},"best":{"step":4000,"heldOut":0.67981,"excess":0.021314,"copyAccuracy":0.96443},"start":3.8309,"checkpoints":[{"step":25,"flops":2149171200,"sixND":1935360000,"tokens":9600,"heldOut":1.3033,"excess":0.64476,"copyAccuracy":0.18182,"localAccuracy":0.57095,"train":2.4746},{"step":50,"flops":4298342400,"sixND":3870720000,"tokens":19200,"heldOut":1.1087,"excess":0.45017,"copyAccuracy":0.1996,"localAccuracy":0.63267,"train":1.1848},{"step":100,"flops":8596684800,"sixND":7741440000,"tokens":38400,"heldOut":0.96765,"excess":0.30915,"copyAccuracy":0.17984,"localAccuracy":0.67648,"train":1.0029},{"step":200,"flops":17193369600,"sixND":15482880000,"tokens":76800,"heldOut":0.88901,"excess":0.23051,"copyAccuracy":0.21937,"localAccuracy":0.6944,"train":0.88274},{"step":300,"flops":25790054400,"sixND":23224320000,"tokens":115200,"heldOut":0.85131,"excess":0.19281,"copyAccuracy":0.39328,"localAccuracy":0.69351,"train":0.84654},{"step":500,"flops":42983424000,"sixND":38707200000,"tokens":192000,"heldOut":0.7829,"excess":0.1244,"copyAccuracy":0.59486,"localAccuracy":0.70167,"train":0.77781},{"step":750,"flops":64475136000,"sixND":58060800000,"tokens":288000,"heldOut":0.72101,"excess":0.062505,"copyAccuracy":0.93874,"localAccuracy":0.69954,"train":0.71875},{"step":1000,"flops":85966848000,"sixND":77414400000,"tokens":384000,"heldOut":0.7087,"excess":0.0502,"copyAccuracy":0.9585,"localAccuracy":0.70273,"train":0.68468},{"step":1500,"flops":128950272000,"sixND":116121600000,"tokens":576000,"heldOut":0.69883,"excess":0.040332,"copyAccuracy":0.96443,"localAccuracy":0.69369,"train":0.6774},{"step":2000,"flops":171933696000,"sixND":154828800000,"tokens":768000,"heldOut":0.70285,"excess":0.044353,"copyAccuracy":0.96047,"localAccuracy":0.69635,"train":0.66981},{"step":3000,"flops":257900544000,"sixND":232243200000,"tokens":1152000,"heldOut":0.68421,"excess":0.02571,"copyAccuracy":0.97036,"localAccuracy":0.7022,"train":0.66368},{"step":4000,"flops":343867392000,"sixND":309657600000,"tokens":1536000,"heldOut":0.67981,"excess":0.021314,"copyAccuracy":0.96443,"localAccuracy":0.70326,"train":0.65904}],"seconds":123.86},{"name":"d64L2","config":{"dModel":64,"nLayers":2,"nHeads":2,"dFF":128,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":74496,"nonEmbedding":67072,"embedding":7424,"perLayer":33472},"flopsPerStep":165150720,"final":{"step":4000,"heldOut":0.67496,"excess":0.016463,"copyAccuracy":0.98617,"flops":660602880000,"tokens":1536000},"best":{"step":4000,"heldOut":0.67496,"excess":0.016463,"copyAccuracy":0.98617},"start":3.8328,"checkpoints":[{"step":25,"flops":4128768000,"sixND":3863347200,"tokens":9600,"heldOut":1.2976,"excess":0.63915,"copyAccuracy":0.16403,"localAccuracy":0.58106,"train":2.4511},{"step":50,"flops":8257536000,"sixND":7726694400,"tokens":19200,"heldOut":1.1142,"excess":0.45571,"copyAccuracy":0.20356,"localAccuracy":0.61139,"train":1.1884},{"step":100,"flops":16515072000,"sixND":15453388800,"tokens":38400,"heldOut":0.96433,"excess":0.30583,"copyAccuracy":0.14822,"localAccuracy":0.68464,"train":1.0155},{"step":200,"flops":33030144000,"sixND":30906777600,"tokens":76800,"heldOut":0.8772,"excess":0.2187,"copyAccuracy":0.18182,"localAccuracy":0.69457,"train":0.88782},{"step":300,"flops":49545216000,"sixND":46360166400,"tokens":115200,"heldOut":0.85805,"excess":0.19955,"copyAccuracy":0.3083,"localAccuracy":0.6967,"train":0.85639},{"step":500,"flops":82575360000,"sixND":77266944000,"tokens":192000,"heldOut":0.79254,"excess":0.13404,"copyAccuracy":0.54348,"localAccuracy":0.69723,"train":0.81493},{"step":750,"flops":123863040000,"sixND":115900416000,"tokens":288000,"heldOut":0.79697,"excess":0.13847,"copyAccuracy":0.56522,"localAccuracy":0.69422,"train":0.77883},{"step":1000,"flops":165150720000,"sixND":154533888000,"tokens":384000,"heldOut":0.79053,"excess":0.13203,"copyAccuracy":0.56324,"localAccuracy":0.70149,"train":0.76864},{"step":1500,"flops":247726080000,"sixND":231800832000,"tokens":576000,"heldOut":0.78276,"excess":0.12426,"copyAccuracy":0.57905,"localAccuracy":0.69635,"train":0.76497},{"step":2000,"flops":330301440000,"sixND":309067776000,"tokens":768000,"heldOut":0.76236,"excess":0.10386,"copyAccuracy":0.6087,"localAccuracy":0.69741,"train":0.7518},{"step":3000,"flops":495452160000,"sixND":463601664000,"tokens":1152000,"heldOut":0.68521,"excess":0.026711,"copyAccuracy":0.97826,"localAccuracy":0.70291,"train":0.70786},{"step":4000,"flops":660602880000,"sixND":618135552000,"tokens":1536000,"heldOut":0.67496,"excess":0.016463,"copyAccuracy":0.98617,"localAccuracy":0.70131,"train":0.65824}],"seconds":247.97},{"name":"d64L3","config":{"dModel":64,"nLayers":3,"nHeads":2,"dFF":128,"vocab":46,"ctx":24},"modelSeed":7,"dataSeed":11,"params":{"total":107968,"nonEmbedding":100544,"embedding":7424,"perLayer":33472},"flopsPerStep":244334592,"final":{"step":4000,"heldOut":0.68103,"excess":0.022533,"copyAccuracy":0.99209,"flops":977338368000,"tokens":1536000},"best":{"step":3000,"heldOut":0.67778,"excess":0.019276,"copyAccuracy":0.99802},"start":3.8484,"checkpoints":[{"step":25,"flops":6108364800,"sixND":5791334400,"tokens":9600,"heldOut":1.3915,"excess":0.73301,"copyAccuracy":0.16008,"localAccuracy":0.55676,"train":2.4629},{"step":50,"flops":12216729600,"sixND":11582668800,"tokens":19200,"heldOut":1.1457,"excess":0.48717,"copyAccuracy":0.21146,"localAccuracy":0.6144,"train":1.2258},{"step":100,"flops":24433459200,"sixND":23165337600,"tokens":38400,"heldOut":0.99836,"excess":0.33986,"copyAccuracy":0.1502,"localAccuracy":0.6536,"train":1.0343},{"step":200,"flops":48866918400,"sixND":46330675200,"tokens":76800,"heldOut":0.881,"excess":0.2225,"copyAccuracy":0.19565,"localAccuracy":0.69847,"train":0.90411},{"step":300,"flops":73300377600,"sixND":69496012800,"tokens":115200,"heldOut":0.87123,"excess":0.21273,"copyAccuracy":0.21937,"localAccuracy":0.69564,"train":0.85848},{"step":500,"flops":122167296000,"sixND":115826688000,"tokens":192000,"heldOut":0.83281,"excess":0.17431,"copyAccuracy":0.41897,"localAccuracy":0.69688,"train":0.83304},{"step":750,"flops":183250944000,"sixND":173740032000,"tokens":288000,"heldOut":0.72169,"excess":0.063192,"copyAccuracy":0.85178,"localAccuracy":0.69883,"train":0.75472},{"step":1000,"flops":244334592000,"sixND":231653376000,"tokens":384000,"heldOut":0.70179,"excess":0.043286,"copyAccuracy":0.98419,"localAccuracy":0.70362,"train":0.70214},{"step":1500,"flops":366501888000,"sixND":347480064000,"tokens":576000,"heldOut":0.68502,"excess":0.026515,"copyAccuracy":0.99407,"localAccuracy":0.7038,"train":0.67367},{"step":2000,"flops":488669184000,"sixND":463306752000,"tokens":768000,"heldOut":0.68694,"excess":0.028437,"copyAccuracy":0.99407,"localAccuracy":0.70043,"train":0.67103},{"step":3000,"flops":733003776000,"sixND":694960128000,"tokens":1152000,"heldOut":0.67778,"excess":0.019276,"copyAccuracy":0.99802,"localAccuracy":0.70521,"train":0.66284},{"step":4000,"flops":977338368000,"sixND":926613504000,"tokens":1536000,"heldOut":0.68103,"excess":0.022533,"copyAccuracy":0.99209,"localAccuracy":0.70238,"train":0.65862}],"seconds":357.31}]},"fits":{"about":"Straight lines on log-log axes, by least squares on log10 values. excess = held-out loss minus the generator floor 0.6585 nats. byParams: final excess (step 4000) against non-embedding parameters, all 12 grid runs, and the depth-2 runs alone (width only). byParamsTotal: the same against all parameters. byTokens: for each run, excess against training tokens over its checkpoints from step 500 on (how fast longer training still pays). se: standard error of the slope from the residuals, which treats the points as independent; seeds were not varied across the grid, so read slopes against the noise section. xRange: the span the line was fitted on; nothing outside it is claimed. The runs do not reach convergence at the larger sizes, so byParams mixes the effect of size with the effect of a fixed step budget.","byParams":{"n":12,"slope":-0.38111,"se":0.12403,"r2":0.48564,"intercept":0.013128,"xRange":[600,100000]},"byParamsWidthOnly":{"n":4,"slope":-0.34062,"se":0.2064,"r2":0.57659,"intercept":-0.28741,"xRange":[1000,67000]},"byParamsTotal":{"n":12,"slope":-0.42963,"se":0.14728,"r2":0.45974,"intercept":0.26736,"xRange":[2000,110000]},"byTokens":[{"name":"d8L1","n":7,"slope":-0.23515,"se":0.0081627,"r2":0.99401,"intercept":0.64446,"xRange":[200000,1500000]},{"name":"d8L2","n":7,"slope":-0.50927,"se":0.1061,"r2":0.82167,"intercept":2.1211,"xRange":[200000,1500000]},{"name":"d8L3","n":7,"slope":-0.50473,"se":0.045289,"r2":0.9613,"intercept":2.1101,"xRange":[200000,1500000]},{"name":"d16L1","n":7,"slope":-0.881,"se":0.10689,"r2":0.93144,"intercept":3.9683,"xRange":[200000,1500000]},{"name":"d16L2","n":7,"slope":-1.2267,"se":0.067209,"r2":0.98521,"intercept":5.8087,"xRange":[200000,1500000]},{"name":"d16L3","n":7,"slope":-1.0652,"se":0.094011,"r2":0.96252,"intercept":4.8959,"xRange":[200000,1500000]},{"name":"d32L1","n":7,"slope":-0.19697,"se":0.027096,"r2":0.91356,"intercept":0.23284,"xRange":[200000,1500000]},{"name":"d32L2","n":7,"slope":-1.0243,"se":0.13977,"r2":0.91483,"intercept":4.4509,"xRange":[200000,1500000]},{"name":"d32L3","n":7,"slope":-1.0091,"se":0.18238,"r2":0.8596,"intercept":4.4211,"xRange":[200000,1500000]},{"name":"d64L1","n":7,"slope":-0.74606,"se":0.098234,"r2":0.92023,"intercept":2.9414,"xRange":[200000,1500000]},{"name":"d64L2","n":7,"slope":-1.0092,"se":0.28072,"r2":0.72105,"intercept":4.6763,"xRange":[200000,1500000]},{"name":"d64L3","n":7,"slope":-0.93044,"se":0.19071,"r2":0.8264,"intercept":3.9452,"xRange":[200000,1500000]}]},"frontier":{"about":"Every checkpoint of every grid run (seed-7 runs only) as a point: training compute so far (FLOPS of the model, forward and backward, attention included) and held-out loss there. points: the compute frontier, each checkpoint that beats every cheaper or equal-cost one. budgets: at four budgets per factor of ten, the best run and step that fits, with its non-embedding size and tokens per non-embedding parameter. fitExcess: frontier excess against compute. fitSizeAtBudget: the best model's non-embedding size against budget, the quantity Kaplan et al. and Hoffmann et al. disagree on; with 4 widths and 3 depths it can only take 12 values, so its slope is coarse. Checkpoints come from runs with a constant learning rate and no decay; Hoffmann et al. show that intermediate points of a run tuned for a longer horizon sit above what a run tuned for that length reaches, so the frontier here is an upper bound on what each budget can do.","points":[{"compute":62208000,"loss":3.081,"excess":2.4225,"run":"d8L1","step":25,"nonEmbedding":616,"total":1544,"tokens":9600},{"compute":103219200,"loss":3.0715,"excess":2.413,"run":"d8L2","step":25,"nonEmbedding":1216,"total":2144,"tokens":9600},{"compute":124416000,"loss":1.8358,"excess":1.1773,"run":"d8L1","step":50,"nonEmbedding":616,"total":1544,"tokens":19200},{"compute":206438400,"loss":1.711,"excess":1.0525,"run":"d8L2","step":50,"nonEmbedding":1216,"total":2144,"tokens":19200},{"compute":248832000,"loss":1.2444,"excess":0.58587,"run":"d8L1","step":100,"nonEmbedding":616,"total":1544,"tokens":38400},{"compute":412876800,"loss":1.2142,"excess":0.55566,"run":"d8L2","step":100,"nonEmbedding":1216,"total":2144,"tokens":38400},{"compute":497664000,"loss":1.054,"excess":0.39548,"run":"d8L1","step":200,"nonEmbedding":616,"total":1544,"tokens":76800},{"compute":746496000,"loss":0.98686,"excess":0.32836,"run":"d8L1","step":300,"nonEmbedding":616,"total":1544,"tokens":115200},{"compute":1238630400,"loss":0.93588,"excess":0.27738,"run":"d8L2","step":300,"nonEmbedding":1216,"total":2144,"tokens":115200},{"compute":1244160000,"loss":0.91508,"excess":0.25658,"run":"d8L1","step":500,"nonEmbedding":616,"total":1544,"tokens":192000},{"compute":1467187200,"loss":0.90533,"excess":0.24683,"run":"d16L1","step":200,"nonEmbedding":2256,"total":4112,"tokens":76800},{"compute":1866240000,"loss":0.88213,"excess":0.22363,"run":"d8L1","step":750,"nonEmbedding":616,"total":1544,"tokens":288000},{"compute":2200780800,"loss":0.87618,"excess":0.21768,"run":"d16L1","step":300,"nonEmbedding":2256,"total":4112,"tokens":115200},{"compute":2488320000,"loss":0.87263,"excess":0.21413,"run":"d8L1","step":1000,"nonEmbedding":616,"total":1544,"tokens":384000},{"compute":3096576000,"loss":0.86566,"excess":0.20716,"run":"d8L2","step":750,"nonEmbedding":1216,"total":2144,"tokens":288000},{"compute":3667968000,"loss":0.85159,"excess":0.19309,"run":"d16L1","step":500,"nonEmbedding":2256,"total":4112,"tokens":192000},{"compute":4976640000,"loss":0.84017,"excess":0.18167,"run":"d8L1","step":2000,"nonEmbedding":616,"total":1544,"tokens":768000},{"compute":5501952000,"loss":0.78486,"excess":0.12636,"run":"d16L1","step":750,"nonEmbedding":2256,"total":4112,"tokens":288000},{"compute":7335936000,"loss":0.77012,"excess":0.11162,"run":"d16L1","step":1000,"nonEmbedding":2256,"total":4112,"tokens":384000},{"compute":11003904000,"loss":0.75021,"excess":0.091707,"run":"d16L1","step":1500,"nonEmbedding":2256,"total":4112,"tokens":576000},{"compute":12976128000,"loss":0.73014,"excess":0.071644,"run":"d16L2","step":1000,"nonEmbedding":4480,"total":6336,"tokens":384000},{"compute":19464192000,"loss":0.71528,"excess":0.056779,"run":"d16L2","step":1500,"nonEmbedding":4480,"total":6336,"tokens":576000},{"compute":22007808000,"loss":0.69943,"excess":0.040933,"run":"d16L1","step":3000,"nonEmbedding":2256,"total":4112,"tokens":1152000},{"compute":29343744000,"loss":0.68473,"excess":0.026235,"run":"d16L1","step":4000,"nonEmbedding":2256,"total":4112,"tokens":1536000},{"compute":38928384000,"loss":0.68262,"excess":0.02412,"run":"d16L2","step":3000,"nonEmbedding":4480,"total":6336,"tokens":1152000},{"compute":51904512000,"loss":0.67458,"excess":0.016079,"run":"d16L2","step":4000,"nonEmbedding":4480,"total":6336,"tokens":1536000},{"compute":179306496000,"loss":0.67348,"excess":0.014979,"run":"d32L2","step":4000,"nonEmbedding":17152,"total":20864,"tokens":1536000},{"compute":262176768000,"loss":0.67044,"excess":0.011939,"run":"d32L3","step":4000,"nonEmbedding":25696,"total":29408,"tokens":1536000}],"fitExcess":{"n":28,"slope":-0.64767,"se":0.021917,"r2":0.97109,"intercept":5.3687,"xRange":[60000000,260000000000]},"fitSizeAtBudget":{"n":16,"slope":0.41372,"se":0.045225,"r2":0.85669,"intercept":-0.71857,"xRange":[100000000,560000000000]},"budgets":[{"budget":100000000,"run":"d8L1","step":25,"heldOut":3.081,"nonEmbedding":616,"tokensPerParam":15.584},{"budget":177830000,"run":"d8L1","step":50,"heldOut":1.8358,"nonEmbedding":616,"tokensPerParam":31.169},{"budget":316230000,"run":"d8L1","step":100,"heldOut":1.2444,"nonEmbedding":616,"tokensPerParam":62.338},{"budget":562340000,"run":"d8L1","step":200,"heldOut":1.054,"nonEmbedding":616,"tokensPerParam":124.68},{"budget":1000000000,"run":"d8L1","step":300,"heldOut":0.98686,"nonEmbedding":616,"tokensPerParam":187.01},{"budget":1778300000,"run":"d16L1","step":200,"heldOut":0.90533,"nonEmbedding":2256,"tokensPerParam":34.043},{"budget":3162300000,"run":"d8L2","step":750,"heldOut":0.86566,"nonEmbedding":1216,"tokensPerParam":236.84},{"budget":5623400000,"run":"d16L1","step":750,"heldOut":0.78486,"nonEmbedding":2256,"tokensPerParam":127.66},{"budget":10000000000,"run":"d16L1","step":1000,"heldOut":0.77012,"nonEmbedding":2256,"tokensPerParam":170.21},{"budget":17783000000,"run":"d16L2","step":1000,"heldOut":0.73014,"nonEmbedding":4480,"tokensPerParam":85.714},{"budget":31623000000,"run":"d16L1","step":4000,"heldOut":0.68473,"nonEmbedding":2256,"tokensPerParam":680.85},{"budget":56234000000,"run":"d16L2","step":4000,"heldOut":0.67458,"nonEmbedding":4480,"tokensPerParam":342.86},{"budget":100000000000,"run":"d16L2","step":4000,"heldOut":0.67458,"nonEmbedding":4480,"tokensPerParam":342.86},{"budget":177830000000,"run":"d16L2","step":4000,"heldOut":0.67458,"nonEmbedding":4480,"tokensPerParam":342.86},{"budget":316230000000,"run":"d32L3","step":4000,"heldOut":0.67044,"nonEmbedding":25696,"tokensPerParam":59.776},{"budget":562340000000,"run":"d32L3","step":4000,"heldOut":0.67044,"nonEmbedding":25696,"tokensPerParam":59.776}]},"noise":{"about":"Repeat runs at two grid points (width 16 and 32, depth 2) with model seeds 7, 8, 9 and data seeds 11, 12, 13. log10Excess spread: how far one size moves by luck alone on the log axis the fits use; a slope times the log10 step between two sizes smaller than this sd is not resolved. reproduce: the d32 L2 grid run is the lesson model's own recipe, so its checkpoints up to step 2000 must equal the stored training curve (4 decimals).","d16":{"runs":["d16L2","d16L2s8","d16L2s9"],"heldOut":{"n":3,"mean":0.68242,"sd":0.0067868,"min":0.67458,"max":0.68635},"log10Excess":{"n":3,"mean":-1.6348,"sd":0.13761,"min":-1.7937,"max":-1.5551},"copyAccuracy":{"n":3,"mean":0.99407,"sd":0.0052291,"min":0.98814,"max":0.99802}},"d32":{"runs":["d32L2","d32L2s8","d32L2s9"],"heldOut":{"n":3,"mean":0.6793,"sd":0.0052433,"min":0.67348,"max":0.68365},"log10Excess":{"n":3,"mean":-1.692,"sd":0.11772,"min":-1.8245,"max":-1.5995},"copyAccuracy":{"n":3,"mean":0.99341,"sd":0.0060392,"min":0.98814,"max":1}},"reproduce":{"points":11,"pointsEqual":11,"worstAbsDiff":0,"finalAt4000":{"step":4000,"heldOut":0.67348,"excess":0.014979,"copyAccuracy":0.99209,"flops":179306496000,"tokens":1536000}}},"papers":{"about":"What the two papers claim, read from the papers, beside what this toy grid can and cannot say. Kaplan et al. 2020 (arXiv 2001.08361): loss falls as a power law in non-embedding parameters N (embeddings excluded), dataset size D and compute C, with exponents near 0.076, 0.095 and 0.050 over models of about 768 to 1.5 billion non-embedding parameters on WebText2; L(N) is measured on models trained near convergence; for a compute budget they find the best N grows as C to the 0.73, so most extra compute should buy a larger model trained short of convergence. Hoffmann et al. 2022 (arXiv 2203.15556, Chinchilla): over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens, with the learning-rate schedule matched to each run's length, three methods give the best N growing as C to about 0.46 to 0.50 and tokens as C to about 0.50 to 0.54, so parameters and tokens should grow together (about 20 tokens per parameter at their scale); their parametric fit L = E + A / N^0.34 + B / D^0.28 has a constant E for the text's own entropy, the term this lesson knows exactly (the generator floor). They attribute Kaplan's larger model-size exponent mainly to a fixed schedule and small models. This grid has 12 models of 616 to 100544 non-embedding parameters on a 46-word synthetic grammar with a known floor, 4,000 steps at most, one learning rate, a constant schedule, and one seed at most points. Its slopes are descriptions of this toy over this range; they are not estimates of either paper's exponents, and agreement or disagreement with those numbers is not evidence about large language models.","kaplan":{"arxiv":"2001.08361","alphaN":0.076,"alphaD":0.095,"alphaCmin":0.05,"nOptExponent":0.73,"nRange":"about 768 to 1.5e9 non-embedding parameters"},"hoffmann":{"arxiv":"2203.15556","a":[0.5,0.49,0.46],"b":[0.5,0.51,0.54],"E":1.69,"alpha":0.34,"beta":0.28,"models":"over 400, 70e6 to over 16e9 parameters, 5e9 to 500e9 tokens","chinchilla":{"params":70000000000,"tokens":1400000000000},"gopher":{"params":280000000000,"tokens":300000000000}}},"runSeconds":{"d32L3":111.78,"d64L1":123.86,"d32L2":82.525,"d32L2s8":84.938,"d64L2":247.97,"d32L1":43.864,"d32L2s9":80.84,"d16L2":28.81,"d16L3":39.806,"d16L2s8":28.611,"d16L1":16.911,"d16L2s9":29.325,"d8L2":14.439,"d8L1":8.424,"d8L3":19.618,"d64L3":357.31},"seconds":357.43}