{"about":"Lesson L12, the training loop. Computed by tools/school/fill_trainloop.mjs from system/school/trainloop.mjs and the attention lab's own trainer. Do not edit by hand. Losses are in nats (natural log).","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","recipe":{"about":"What the lab's trainer does each step, read off the code. Batch: 16 rows of 24 words cut from a seeded stream of whole corpus episodes. Forward, then mean cross-entropy over all 384 next-word targets. Backward: a hand-written gradient for every parameter (checked against finite differences in the tinyformer tests). Clip: if the global L2 norm of the whole gradient exceeds 1.0, every gradient is scaled by 1.0 / norm. Step: Adam (Kingma and Ba 2015, arXiv 1412.6980, Algorithm 1: bias-corrected first and second moment estimates) with beta1 0.9, beta2 0.99 (the paper's suggested default is 0.999), epsilon 1e-8, and decoupled weight decay 0.01 as in AdamW (Loshchilov and Hutter 2019, arXiv 1711.05101), applied to weight matrices and embeddings only, never to gains or biases. One difference from the paper's Algorithm 2: there the decay term is scaled by the schedule multiplier only; here it is scaled by the full learning rate (the convention of common libraries), so the per-step shrink is lr x 0.01. Learning rate: linear warmup from lr/30 to lr over the first 30 steps, then constant; no decay schedule.","defaults":{"batch":16,"lr":0.01,"seed":1,"warmup":30,"weightDecay":0.01,"clip":1,"gradSign":1},"model":{"seed":7,"dataSeed":11,"steps":2000,"batch":16,"lr":0.01,"nParams":20864},"flops":{"forward":14942208,"step":44826624},"warmupLr":[{"step":1,"lr":0.000333},{"step":2,"lr":0.000667},{"step":10,"lr":0.003333},{"step":30,"lr":0.01},{"step":31,"lr":0.01}]},"reproduce":{"about":"Re-trained from the model's seeds (model seed 7, data seed 11) with the lab's own trainer for the full 2000 steps, scored every 25 steps on the same 256 held-out rows, and compared with the curve stored in model.json after rounding to 4 decimals. tensorsEqual: weight tensors whose float32 bytes match model.json exactly. gradNormsEqual: logged gradient norms (every 25 steps) equal after rounding. Bit-identical on this engine (V8 in Node); the code notes another engine may differ in the last bit of exp and tanh and drift from there, which this run does not test.","points":81,"pointsEqual":81,"worstAbsDiff":0,"tensorsEqual":29,"tensorsTotal":29,"gradNormsEqual":80,"gradNormsLogged":80,"summary":{"endStep":2000,"heldOut":0.6874,"copyAccuracy":0.9921,"localAccuracy":0.7002,"bestHeldOut":0.6775,"bestStep":1850,"firstCopyStep":700,"diverged":false},"curve":[{"step":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":25,"train":2.9816,"heldOut":1.7252,"copyAccuracy":0.1522,"localAccuracy":0.5488},{"step":50,"train":1.3223,"heldOut":1.1656,"copyAccuracy":0.2134,"localAccuracy":0.6078},{"step":75,"train":1.0861,"heldOut":1.0454,"copyAccuracy":0.1976,"localAccuracy":0.6483},{"step":100,"train":0.9942,"heldOut":0.9654,"copyAccuracy":0.1561,"localAccuracy":0.6744},{"step":125,"train":0.9327,"heldOut":0.9599,"copyAccuracy":0.1798,"localAccuracy":0.6823},{"step":150,"train":0.8998,"heldOut":0.9239,"copyAccuracy":0.2016,"localAccuracy":0.6907},{"step":175,"train":0.8934,"heldOut":0.9101,"copyAccuracy":0.166,"localAccuracy":0.693},{"step":200,"train":0.8844,"heldOut":0.8812,"copyAccuracy":0.2134,"localAccuracy":0.6976},{"step":225,"train":0.8422,"heldOut":0.8643,"copyAccuracy":0.2431,"localAccuracy":0.7001},{"step":250,"train":0.8468,"heldOut":0.878,"copyAccuracy":0.2826,"localAccuracy":0.6965},{"step":275,"train":0.8422,"heldOut":0.8454,"copyAccuracy":0.3024,"localAccuracy":0.7008},{"step":300,"train":0.8362,"heldOut":0.8436,"copyAccuracy":0.3399,"localAccuracy":0.7026},{"step":325,"train":0.8286,"heldOut":0.8536,"copyAccuracy":0.3281,"localAccuracy":0.694},{"step":350,"train":0.8238,"heldOut":0.8397,"copyAccuracy":0.3261,"localAccuracy":0.6969},{"step":375,"train":0.8355,"heldOut":0.8523,"copyAccuracy":0.3142,"localAccuracy":0.7017},{"step":400,"train":0.8032,"heldOut":0.8456,"copyAccuracy":0.3221,"localAccuracy":0.6971},{"step":425,"train":0.841,"heldOut":0.8448,"copyAccuracy":0.3162,"localAccuracy":0.6908},{"step":450,"train":0.8254,"heldOut":0.832,"copyAccuracy":0.3458,"localAccuracy":0.6997},{"step":475,"train":0.8116,"heldOut":0.824,"copyAccuracy":0.4229,"localAccuracy":0.7077},{"step":500,"train":0.7949,"heldOut":0.8182,"copyAccuracy":0.4289,"localAccuracy":0.7057},{"step":525,"train":0.7991,"heldOut":0.8344,"copyAccuracy":0.4723,"localAccuracy":0.6836},{"step":550,"train":0.7913,"heldOut":0.8073,"copyAccuracy":0.5119,"localAccuracy":0.6919},{"step":575,"train":0.7913,"heldOut":0.8039,"copyAccuracy":0.5277,"localAccuracy":0.696},{"step":600,"train":0.7993,"heldOut":0.7937,"copyAccuracy":0.5751,"localAccuracy":0.702},{"step":625,"train":0.7871,"heldOut":0.7775,"copyAccuracy":0.6028,"localAccuracy":0.7015},{"step":650,"train":0.7516,"heldOut":0.7728,"copyAccuracy":0.6601,"localAccuracy":0.6995},{"step":675,"train":0.7525,"heldOut":0.7455,"copyAccuracy":0.7569,"localAccuracy":0.7022},{"step":700,"train":0.7395,"heldOut":0.7289,"copyAccuracy":0.9348,"localAccuracy":0.6898},{"step":725,"train":0.6994,"heldOut":0.7114,"copyAccuracy":0.9763,"localAccuracy":0.6994},{"step":750,"train":0.6929,"heldOut":0.715,"copyAccuracy":0.9684,"localAccuracy":0.6992},{"step":775,"train":0.6882,"heldOut":0.7218,"copyAccuracy":0.9684,"localAccuracy":0.702},{"step":800,"train":0.6876,"heldOut":0.7033,"copyAccuracy":0.9763,"localAccuracy":0.7045},{"step":825,"train":0.6905,"heldOut":0.7058,"copyAccuracy":0.9822,"localAccuracy":0.7001},{"step":850,"train":0.6853,"heldOut":0.7002,"copyAccuracy":0.9941,"localAccuracy":0.7018},{"step":875,"train":0.6766,"heldOut":0.7192,"copyAccuracy":0.9605,"localAccuracy":0.7081},{"step":900,"train":0.6752,"heldOut":0.7048,"copyAccuracy":0.9921,"localAccuracy":0.6999},{"step":925,"train":0.6712,"heldOut":0.6969,"copyAccuracy":0.9901,"localAccuracy":0.6994},{"step":950,"train":0.6799,"heldOut":0.7053,"copyAccuracy":0.9941,"localAccuracy":0.7026},{"step":975,"train":0.6835,"heldOut":0.7108,"copyAccuracy":0.9901,"localAccuracy":0.6958},{"step":1000,"train":0.6711,"heldOut":0.6981,"copyAccuracy":0.996,"localAccuracy":0.7013},{"step":1025,"train":0.6718,"heldOut":0.6933,"copyAccuracy":0.9941,"localAccuracy":0.7059},{"step":1050,"train":0.6785,"heldOut":0.6871,"copyAccuracy":1,"localAccuracy":0.7034},{"step":1075,"train":0.6685,"heldOut":0.6913,"copyAccuracy":0.9921,"localAccuracy":0.7088},{"step":1100,"train":0.663,"heldOut":0.6942,"copyAccuracy":0.998,"localAccuracy":0.699},{"step":1125,"train":0.6573,"heldOut":0.6939,"copyAccuracy":0.996,"localAccuracy":0.6937},{"step":1150,"train":0.677,"heldOut":0.7227,"copyAccuracy":0.9466,"localAccuracy":0.6962},{"step":1175,"train":0.6877,"heldOut":0.7006,"copyAccuracy":0.9822,"localAccuracy":0.6967},{"step":1200,"train":0.6915,"heldOut":0.6998,"copyAccuracy":0.9585,"localAccuracy":0.6999},{"step":1225,"train":0.6709,"heldOut":0.6928,"copyAccuracy":0.9881,"localAccuracy":0.702},{"step":1250,"train":0.6798,"heldOut":0.6964,"copyAccuracy":0.9941,"localAccuracy":0.701},{"step":1275,"train":0.6682,"heldOut":0.6867,"copyAccuracy":0.9941,"localAccuracy":0.7045},{"step":1300,"train":0.6593,"heldOut":0.6898,"copyAccuracy":0.9921,"localAccuracy":0.6971},{"step":1325,"train":0.6622,"heldOut":0.6923,"copyAccuracy":0.9941,"localAccuracy":0.6967},{"step":1350,"train":0.6736,"heldOut":0.6964,"copyAccuracy":0.996,"localAccuracy":0.7026},{"step":1375,"train":0.6611,"heldOut":0.6937,"copyAccuracy":0.996,"localAccuracy":0.6962},{"step":1400,"train":0.6596,"heldOut":0.6821,"copyAccuracy":0.998,"localAccuracy":0.699},{"step":1425,"train":0.6634,"heldOut":0.6795,"copyAccuracy":0.996,"localAccuracy":0.7027},{"step":1450,"train":0.6668,"heldOut":0.6836,"copyAccuracy":0.998,"localAccuracy":0.6969},{"step":1475,"train":0.6606,"heldOut":0.6828,"copyAccuracy":0.998,"localAccuracy":0.7068},{"step":1500,"train":0.665,"heldOut":0.6944,"copyAccuracy":0.9881,"localAccuracy":0.7004},{"step":1525,"train":0.6643,"heldOut":0.6946,"copyAccuracy":0.9881,"localAccuracy":0.6988},{"step":1550,"train":0.6637,"heldOut":0.6917,"copyAccuracy":0.9941,"localAccuracy":0.7061},{"step":1575,"train":0.6583,"heldOut":0.6895,"copyAccuracy":1,"localAccuracy":0.6981},{"step":1600,"train":0.66,"heldOut":0.6814,"copyAccuracy":0.9921,"localAccuracy":0.7054},{"step":1625,"train":0.6716,"heldOut":0.6936,"copyAccuracy":0.9941,"localAccuracy":0.6985},{"step":1650,"train":0.6793,"heldOut":0.7088,"copyAccuracy":0.9881,"localAccuracy":0.6967},{"step":1675,"train":0.6832,"heldOut":0.6867,"copyAccuracy":0.9901,"localAccuracy":0.702},{"step":1700,"train":0.6602,"heldOut":0.6875,"copyAccuracy":0.9881,"localAccuracy":0.6942},{"step":1725,"train":0.6564,"heldOut":0.6858,"copyAccuracy":0.998,"localAccuracy":0.7065},{"step":1750,"train":0.659,"heldOut":0.6884,"copyAccuracy":0.998,"localAccuracy":0.6946},{"step":1775,"train":0.6686,"heldOut":0.6938,"copyAccuracy":0.9802,"localAccuracy":0.7015},{"step":1800,"train":0.6638,"heldOut":0.6838,"copyAccuracy":0.9901,"localAccuracy":0.704},{"step":1825,"train":0.6577,"heldOut":0.6893,"copyAccuracy":0.998,"localAccuracy":0.6997},{"step":1850,"train":0.6666,"heldOut":0.6775,"copyAccuracy":0.998,"localAccuracy":0.6978},{"step":1875,"train":0.6578,"heldOut":0.6832,"copyAccuracy":0.998,"localAccuracy":0.7024},{"step":1900,"train":0.6651,"heldOut":0.6863,"copyAccuracy":0.996,"localAccuracy":0.6983},{"step":1925,"train":0.6695,"heldOut":0.6868,"copyAccuracy":0.9842,"localAccuracy":0.7065},{"step":1950,"train":0.6679,"heldOut":0.6829,"copyAccuracy":0.996,"localAccuracy":0.7034},{"step":1975,"train":0.6624,"heldOut":0.6799,"copyAccuracy":1,"localAccuracy":0.7056},{"step":2000,"train":0.6524,"heldOut":0.6874,"copyAccuracy":0.9921,"localAccuracy":0.7002}],"milestones":[{"step":0,"heldOut":3.8414,"copyAccuracy":0.0692},{"step":100,"heldOut":0.9654,"copyAccuracy":0.1561},{"step":500,"heldOut":0.8182,"copyAccuracy":0.4289},{"step":750,"heldOut":0.715,"copyAccuracy":0.9684},{"step":1000,"heldOut":0.6981,"copyAccuracy":0.996},{"step":2000,"heldOut":0.6874,"copyAccuracy":0.9921}]},"gradients":{"about":"The global L2 norm of the gradient at every one of the 2000 steps of the reproduced run, measured before clipping. clipped: steps where the norm exceeded 1, so the clip scaled the step down. byWindow: the same per block of 250 steps. One run: how often the clip fires depends on the seed, the batch size and the learning rate.","stats":{"n":2000,"min":0.1359,"p10":0.2065,"median":0.3516,"p90":0.719,"max":2.4331,"clipped":76,"clipShare":0.038},"perStep":[2.0877,1.9371,1.8974,1.4748,1.1369,1.1334,1.2372,1.2382,1.3456,1.2513,1.6539,1.4796,1.4576,1.4003,1.2611,1.4539,1.2107,1.2453,1.2767,1.0803,1.3521,0.9782,1.1106,1.2044,0.8414,1.7558,0.9618,1.529,1.3009,0.8265,1.7241,1.5785,0.8964,1.7097,1.2449,0.7469,0.6572,0.9118,0.9285,0.6804,1.2161,0.8926,0.9913,0.804,1.0491,0.7501,1.2035,0.6477,0.9616,0.7496,0.8532,0.7743,0.7177,1.0602,0.9758,0.9097,0.8184,0.7941,0.7628,0.6854,0.9328,0.8528,0.7321,0.6797,1.0137,0.7678,0.9647,0.9993,0.6238,1.3809,0.7031,0.9908,1.455,1.0096,1.3882,1.3484,0.6907,0.9051,1.17,0.6256,0.8151,1.3574,1.0096,1.0095,1.3555,0.7836,0.7521,0.792,0.9747,1.0518,0.617,1.1339,1.0629,0.7866,0.9649,0.9035,1.1112,0.752,0.9241,1.013,0.6058,1.1446,0.787,0.7083,0.6693,0.7276,0.8149,0.5887,0.6618,0.7794,0.9365,1.6334,0.7491,1.341,0.8187,0.8749,0.9141,1.0381,0.8403,2.2569,0.7631,0.6865,0.9641,0.642,0.7745,0.6231,0.8602,0.779,0.6373,0.5459,0.7322,1.0263,0.6367,0.6746,0.5404,0.617,0.7216,0.591,0.7259,0.5725,0.7069,0.6164,1.2649,0.4438,1.6189,1.0688,0.7424,1.4705,0.9236,0.7669,0.9678,0.5526,0.8241,0.9165,0.7043,0.7511,1.1256,0.5205,0.7248,0.8707,0.6346,0.5336,0.6475,0.6063,0.7324,0.5168,0.8174,0.6983,0.6776,0.4942,0.5444,0.782,0.4964,0.7559,0.7054,0.743,0.7485,0.7821,0.6326,0.6374,0.7566,0.8193,0.5347,0.6279,0.6532,0.6632,0.7087,0.7434,0.5608,0.557,0.5998,0.8082,0.5751,0.455,0.6003,0.575,0.5555,0.5568,0.5649,0.5577,0.4142,0.386,0.4488,0.4313,0.4834,0.499,0.4009,0.468,0.4946,0.3984,0.5809,0.4392,0.494,0.5945,0.5209,0.5398,0.6379,0.5731,0.4117,0.5745,0.3941,0.444,0.5797,0.6727,0.5062,0.6978,0.6136,0.5143,0.5544,0.6045,1.0565,0.554,0.7736,0.7916,0.5946,0.7785,0.7087,0.7723,0.5676,0.8479,0.8706,0.5441,0.4723,0.5395,0.484,0.4721,0.5185,0.386,0.4458,0.5246,0.4103,0.5076,0.5344,0.4439,0.4847,0.558,0.4853,0.5002,0.414,0.4412,0.5214,0.4972,0.4372,0.4463,0.4633,0.526,0.4511,0.5227,0.4587,0.5019,0.3977,0.3721,0.4587,0.5613,0.5276,0.4581,0.5094,0.4636,0.5079,0.552,0.5346,0.5417,0.4528,0.4137,0.3744,0.5072,0.5796,0.602,0.4449,0.4416,0.5161,0.3601,0.4623,0.5545,0.4303,0.4249,0.4587,0.4585,0.5602,0.3817,0.3783,0.3232,0.4157,0.3876,0.4427,0.4475,0.3576,0.4635,0.4159,0.4549,0.4045,0.4249,0.5122,0.417,0.2782,0.405,0.47,0.3684,0.4205,0.4647,0.3924,0.4153,0.5141,0.3844,0.5041,0.4568,0.3729,0.4228,0.3962,0.7014,0.4483,0.4111,0.4603,0.4537,0.3972,0.4681,0.9168,0.4397,0.4249,0.4495,0.6293,0.4749,0.4343,0.4078,0.5805,0.5338,0.5914,0.4731,0.77,0.3277,0.3533,0.4163,0.6668,0.489,0.6774,0.487,0.719,0.3941,0.4819,0.4997,0.4681,0.4886,0.4702,0.4436,0.5667,0.6069,0.3772,0.4383,0.4654,0.5202,0.444,0.4925,0.5375,0.4399,0.5493,0.5103,0.3573,0.3628,0.5489,0.4179,0.3709,0.3808,0.5969,0.6911,0.4799,0.4066,0.7279,0.3337,0.4112,0.3963,0.5478,0.3211,0.48,0.3933,0.3468,0.3692,0.5346,0.3613,0.3681,0.393,0.5087,0.3864,0.3444,0.4092,0.3721,0.4178,0.4643,0.4535,0.5531,0.401,0.4479,0.3647,0.3832,0.3876,0.4277,0.3979,0.4112,0.4228,0.4152,0.4181,0.4586,0.3235,0.492,0.5043,0.7213,0.3638,0.4489,0.308,0.4276,0.4011,0.373,0.366,0.3993,0.8246,0.3822,0.4327,0.378,0.3597,0.3506,0.4207,0.4366,0.6103,0.4172,0.4957,0.4684,0.4861,0.4438,0.3881,0.4671,0.418,0.3455,0.4198,0.4684,0.3545,0.4227,0.2865,0.4024,0.3363,0.5381,0.3696,0.4806,0.3374,0.3613,0.5422,0.4898,0.3266,0.391,0.4615,0.3853,0.5733,0.3151,0.5176,0.3214,0.4428,0.3431,0.3895,0.3348,0.3316,0.3441,0.2729,0.3852,0.3457,0.3117,0.3156,0.2802,0.3848,0.3673,0.3316,0.3485,0.4612,0.5119,0.4615,0.3445,0.4422,0.408,0.2957,0.3411,0.3609,0.3792,0.4004,0.3873,0.4882,0.3907,0.37,0.4012,0.3454,0.4195,0.3397,0.3206,0.3314,0.4055,0.3131,0.3813,0.3115,0.3716,0.3989,0.3752,0.3379,0.4274,0.3422,0.6745,0.5547,0.4107,0.3618,0.4468,0.5975,0.4363,0.3852,0.3644,0.3173,0.4431,0.4877,0.5549,0.3664,0.3382,0.3044,0.4382,0.36,0.5252,0.383,0.3049,0.3165,0.2765,0.4115,0.4068,0.4628,0.3931,0.3579,0.667,0.3413,0.3621,0.3604,0.482,0.4971,0.53,0.4384,0.4429,0.4203,0.3265,0.3764,0.4638,0.4043,0.3522,0.4059,0.3891,0.4661,0.4666,0.3586,0.3605,0.4479,0.6453,0.4641,0.3699,0.6838,0.3961,0.6732,0.408,0.5455,0.4637,0.5566,0.6349,0.3575,0.4729,0.2986,0.5232,0.4843,0.4688,0.564,0.3397,0.5643,0.4383,0.5481,0.36,0.3391,0.5584,0.4298,0.5357,0.3941,0.4111,0.6042,0.3699,0.3726,0.3206,0.529,0.4341,0.4917,0.4861,0.4475,0.9529,0.465,0.5274,0.4242,0.481,0.4017,0.4196,0.3635,0.3796,0.3677,0.3019,0.4538,0.3742,0.3756,0.3222,0.3422,0.3584,0.3052,0.2891,0.3467,0.4141,0.3239,0.3167,0.3076,0.4399,0.5345,0.3782,0.3634,0.333,0.4979,0.4599,0.5094,0.458,0.348,0.3691,0.4321,0.3119,0.2889,0.4109,0.4483,0.3052,0.3606,0.2989,0.3132,0.421,0.4051,0.2947,0.4174,0.3751,0.4124,0.3898,0.3722,0.3622,0.4219,0.5417,0.3611,0.2787,0.3985,0.7211,0.5072,0.3175,0.4459,0.3401,0.5167,0.3133,0.4995,0.4644,0.3502,0.4463,0.414,0.5155,0.5832,0.4043,0.5518,0.3485,0.5676,0.4822,0.4235,0.5958,0.5379,0.3725,0.435,0.3388,0.346,0.3917,0.7256,0.4751,0.5438,0.4126,0.489,0.5596,0.3283,0.5024,0.4444,0.3828,0.3536,0.3513,0.4286,0.6148,0.4103,0.379,0.3504,0.5103,0.3268,0.3235,0.4816,0.3617,0.4021,0.3149,0.4076,0.5508,0.3293,0.5053,0.4018,0.398,0.3792,0.47,0.2724,0.6066,0.7868,0.6409,0.3003,0.6189,0.5518,0.3941,0.3448,0.317,0.2672,0.6983,0.7881,0.4367,0.3767,0.3554,0.3298,0.4095,0.5293,0.7292,0.4164,0.2672,0.486,0.6503,1.2494,0.6484,0.5709,0.4329,0.3201,0.4657,0.3818,0.3747,0.4112,0.3037,0.4203,0.8646,0.652,0.4167,0.3389,0.4404,0.6664,0.3627,0.3613,0.3535,0.3805,0.3937,0.6339,0.2828,0.3759,0.3026,0.4649,0.2959,0.8042,0.9112,0.3644,0.4782,0.3391,0.4952,0.5032,0.3953,0.4665,0.4778,0.5033,0.4829,0.3189,0.3463,0.5145,0.4959,0.4855,0.486,0.3509,0.2835,0.3212,0.2909,0.4925,0.3866,0.54,1.1132,0.2865,0.2765,0.5168,0.4298,0.3716,0.4552,0.3788,0.7758,0.3702,0.5896,0.4626,0.6896,0.3254,0.3,0.3719,0.6757,0.3907,0.4562,0.3706,0.4235,0.3008,0.4766,0.3418,0.3814,0.46,0.3784,1.0979,0.4945,0.3431,0.8618,0.4262,0.8011,0.5454,0.2902,0.3957,0.2957,0.3543,0.3868,0.2996,0.3424,0.3516,0.3612,0.6321,0.2768,0.3691,0.2711,0.3977,0.2917,0.2749,0.4959,0.2955,0.3279,0.444,0.4462,0.5841,1.8824,0.4136,0.6666,0.365,0.3489,0.2782,0.3812,0.5895,0.3651,0.2937,0.318,0.378,0.4244,1.1121,0.4378,0.3202,0.3292,0.5191,0.5742,1.1346,0.3234,0.3433,0.4528,0.3409,0.26,0.35,0.347,0.3765,0.3915,0.377,0.7122,0.329,0.3234,0.2868,0.3882,0.6179,0.2414,0.3428,0.384,0.3525,0.3212,0.3744,0.4438,0.3525,0.3535,0.4231,0.2839,0.2838,0.3389,0.3177,0.6275,0.4274,0.3481,0.5521,0.2844,0.4378,0.2797,0.47,0.2462,0.5349,0.4044,0.2294,0.3306,0.2818,0.2753,0.2559,0.284,0.391,1.3682,0.2867,0.2998,0.5151,0.2842,0.2754,0.4203,0.311,0.3203,0.5905,0.4756,0.274,0.3249,0.3138,0.3322,0.3668,0.3129,0.3479,0.2325,0.2516,0.3198,0.28,1.0852,0.3017,0.4482,0.3217,0.3292,0.5216,0.3475,1.1354,0.2748,0.3758,0.4982,0.5709,0.6505,0.5329,0.3754,0.2583,0.4903,0.3372,0.4893,0.4129,0.4135,0.4033,0.2657,0.2558,0.4935,0.2533,0.3318,0.5018,0.2588,0.3244,0.3428,0.4277,0.3116,0.2698,0.2529,0.2314,0.3368,0.3529,0.5029,0.2445,0.3285,0.2512,0.3088,0.3301,0.3264,0.2791,0.534,0.3272,0.5312,0.6064,0.2423,0.301,0.4037,0.3565,0.2429,0.4605,0.408,0.2271,0.3586,0.3508,0.3408,0.2945,0.2935,0.2779,0.3815,0.2296,0.3492,0.3543,0.4828,0.2912,0.3532,0.4235,0.4035,0.4762,0.3366,0.541,0.275,0.2888,0.2953,0.2064,0.3234,0.4177,0.3074,0.2829,0.2704,0.2278,0.2121,0.4314,0.307,0.253,0.2855,0.2547,0.2892,0.2869,0.3096,0.3068,0.2541,0.192,0.2145,0.3488,0.3154,0.601,0.2428,0.2985,0.247,0.204,0.2362,0.2566,0.2113,0.5081,0.2525,0.3045,0.2294,0.2132,0.2704,0.252,0.2874,0.2776,0.262,0.2694,0.551,0.2699,0.2307,0.3259,0.243,0.1985,0.3538,0.2862,0.1871,0.2702,0.2229,0.2952,0.2299,0.2872,0.2777,0.2253,0.2826,0.3982,0.3119,0.2038,0.2781,0.2669,0.2586,0.2169,0.2163,0.3378,0.2126,0.2041,0.2211,0.2484,0.239,0.26,0.2149,0.2661,0.2318,0.2117,0.2824,0.3115,0.2098,0.2209,0.2872,0.2749,0.7115,0.2879,0.2189,0.2751,0.2176,0.2598,0.2728,0.2727,0.3041,0.2916,0.278,0.2365,0.2925,0.2602,0.2289,0.4391,0.2488,0.2713,0.2844,0.1928,0.295,0.4128,0.2412,0.2233,0.286,0.5359,0.2561,0.3204,0.2366,0.2568,0.2608,0.3422,0.8833,0.2989,0.3482,0.4584,0.5639,0.3686,0.2607,0.4961,0.7016,0.4359,0.3554,0.315,0.3847,0.445,0.9317,0.514,0.3135,0.4908,0.456,0.2939,0.3175,0.2747,0.306,0.3273,0.3789,0.2844,0.3149,0.2533,0.3058,0.2469,0.256,0.8984,0.2585,0.8929,0.367,0.2951,0.4721,0.8542,0.3751,0.5558,0.4397,0.5195,0.3934,0.5783,0.4652,0.3449,0.3432,0.5595,0.2565,0.4719,0.5325,0.275,0.3664,0.2771,0.3817,0.445,0.2716,0.9692,0.4502,0.7304,0.2742,0.4492,0.4221,0.4582,0.5977,0.2714,0.6589,0.2447,0.4215,0.4029,0.6564,0.3204,0.3628,0.2658,0.4128,0.3465,0.3377,0.2275,0.4881,0.2395,0.3528,0.4041,0.2279,0.457,0.27,0.2603,0.2166,0.2215,0.236,0.3171,0.7732,0.3752,0.7045,0.3955,0.3518,0.4299,0.3402,0.7902,0.2562,1.0145,0.2555,0.5749,0.5623,0.4903,0.2048,0.4247,0.325,0.3422,0.5043,0.2537,0.2597,0.1894,0.1989,0.4979,0.2183,0.3771,0.2459,0.2368,0.2791,0.2199,0.3045,0.3029,0.3631,0.1976,0.3075,0.7494,0.3017,0.3138,0.3118,0.2278,0.2399,0.1991,0.2975,0.2935,0.4685,0.3491,0.3219,0.2838,0.2746,0.2255,0.2402,0.258,0.2995,0.2172,0.1986,0.2401,0.1799,0.2515,0.2793,0.2388,0.2802,0.2111,0.3142,0.2147,0.1962,0.287,0.2422,0.2131,0.2277,0.549,0.2563,0.2096,0.2325,0.2253,0.3786,0.8773,0.2076,0.3021,0.3617,0.2417,0.2895,0.2659,0.2762,0.2909,0.7363,0.2585,0.2797,0.2358,0.2817,0.7782,0.491,0.4899,0.2054,0.2501,0.6485,0.4038,0.2614,0.3426,0.5955,0.2358,0.8175,0.5051,0.3019,0.4183,0.3367,0.2837,0.2727,0.3977,0.2383,0.3218,0.3237,0.5853,0.4715,0.5068,0.2835,0.2271,0.2631,0.3449,0.2701,0.2904,0.3622,0.3665,0.345,0.3105,0.2182,0.2357,0.2761,0.2773,0.2316,0.2562,0.2016,0.3332,0.2581,0.2871,0.2179,0.2375,0.2208,0.1844,0.2864,0.252,0.2919,0.22,0.1855,0.1843,0.1946,0.2793,0.2709,0.2224,0.2666,0.2396,0.2236,0.2547,0.2147,0.2721,0.2945,0.219,0.2321,0.195,0.2182,0.1812,0.2167,0.2065,0.2128,0.2648,0.2558,0.2367,0.2822,0.2852,0.2662,0.2406,0.2043,0.288,0.3165,0.2049,0.1529,0.2555,0.2437,0.2847,0.2846,0.1987,0.2586,0.2036,0.2595,0.2315,0.3154,0.2146,0.2083,0.1678,0.2579,0.2568,0.2151,0.2068,0.1792,0.219,0.2065,0.2786,0.2069,0.2267,0.2721,0.2398,0.2656,0.1811,0.2186,0.2111,0.2214,0.1689,0.1886,0.2365,0.2454,0.1833,0.3336,0.1474,0.192,0.3126,0.3016,0.2274,0.2651,0.2097,0.2329,0.2361,0.2491,0.2483,0.239,0.2015,0.1933,0.1977,0.2488,0.2032,0.2448,0.2363,0.1737,0.3391,0.2506,0.2464,0.2448,0.2278,0.2037,0.2457,0.2221,0.2502,0.1767,0.2593,0.2918,0.2088,0.3639,0.4231,0.2897,0.2711,0.1786,0.3342,0.3649,0.2222,0.2805,0.2089,0.1814,0.2471,0.2695,0.1957,0.2165,0.2289,0.7893,0.3714,0.2462,0.2531,0.2748,0.5827,0.3832,0.2384,0.2508,0.267,0.2726,0.2188,0.3019,0.2261,0.3323,0.257,0.4199,0.2412,0.567,0.2075,0.2933,0.2624,0.2747,0.748,0.2229,0.2858,0.1914,0.2205,0.2883,0.26,0.2441,0.2916,0.2738,0.2784,0.2151,0.2306,0.3511,0.2401,0.2543,0.2613,0.3344,0.2504,0.2556,0.2035,0.2759,0.2772,0.3598,0.1948,0.2552,0.215,0.2603,0.2159,0.1798,0.2493,0.1857,0.177,0.2438,0.1793,0.1938,0.1815,0.3025,0.2025,0.1877,0.2324,0.2034,0.196,0.2679,0.2242,0.2633,0.1911,0.3089,0.2114,0.2071,0.1961,0.2588,0.2379,0.1812,0.1562,0.2959,0.2343,0.2299,0.1912,0.1895,0.17,0.225,0.2431,0.2851,0.1982,0.202,0.3024,0.2523,0.2273,0.22,0.1691,0.1865,0.2261,0.3058,0.2148,0.2025,0.2765,0.2304,0.2061,0.2853,0.2449,0.1835,0.414,0.2246,0.205,0.192,0.2142,0.4733,0.209,0.2103,0.1944,0.446,0.2011,0.2302,0.5047,0.4501,0.2074,0.1613,0.605,0.2872,0.3559,0.5783,0.5355,1.0907,0.5565,0.1832,0.2642,0.6121,0.3788,0.3576,0.5411,0.2388,0.2798,0.2828,0.2268,0.4127,0.2219,0.264,0.2181,0.3521,0.5029,0.4541,0.4326,0.2402,0.6015,0.212,0.2854,0.41,0.504,0.2788,0.3842,0.33,0.3348,0.232,0.3193,0.3857,0.2722,0.9694,0.2828,0.3686,0.4555,0.548,0.3723,0.9633,0.4838,0.2849,0.2448,0.2963,0.4691,0.232,0.4848,0.2024,0.2762,0.2298,0.3392,0.2662,0.22,0.1999,0.2435,0.2036,0.2827,0.2341,0.2379,0.2327,0.1677,0.2216,0.3103,0.2166,0.1692,0.2452,0.3122,0.2049,0.1802,0.1886,0.1952,0.1791,0.3275,0.3313,0.1955,0.1848,0.4283,0.2247,0.2377,0.2143,0.1958,0.2309,0.1999,0.2357,0.2793,0.2228,0.2287,0.1916,0.3553,0.1735,0.2547,0.219,0.2507,0.2051,0.2223,0.4428,0.2724,0.168,0.2025,0.1877,0.2434,0.3124,0.2476,0.2422,0.1494,0.1854,0.1713,0.3605,0.2953,0.2325,0.2267,0.166,0.2313,0.1711,0.2667,0.2017,0.1704,1.3463,0.2214,0.2115,0.5425,0.2204,0.3208,0.3097,0.2554,0.2368,0.3991,0.3565,0.2526,0.5283,0.2764,0.261,0.3722,0.2486,0.2645,0.219,0.2126,0.3521,0.8685,0.2902,0.4047,0.3284,0.2226,0.7831,0.235,0.263,0.3932,0.4728,0.2665,0.2154,0.1657,0.2497,0.3174,0.2988,0.3306,0.3767,2.1383,0.2723,0.2,0.1767,0.6787,0.2055,0.3399,0.1808,0.2555,0.1887,0.2703,0.2327,0.2369,0.2546,0.2375,0.1666,0.1774,0.3109,0.2966,0.2346,0.2712,0.1875,0.3028,0.2462,0.3083,0.2131,0.2216,0.1671,0.2705,0.3201,0.2163,0.2117,0.2003,0.1671,0.1729,0.2067,0.1882,0.1999,0.2205,0.2183,0.2252,0.1707,0.2267,0.1879,0.1576,0.1745,0.2622,0.2552,0.2632,0.2461,0.1993,0.2002,0.2634,0.2062,0.246,0.1938,0.1613,0.2634,0.1815,0.18,0.1885,0.2242,0.2586,0.2458,0.184,0.2112,0.1893,0.1975,0.1635,0.1824,0.2351,0.2392,0.1857,0.1674,0.1798,0.1978,0.2438,0.2466,0.1974,0.1962,0.1651,0.1901,0.218,0.1867,0.2109,0.1659,0.187,0.1523,0.2772,0.1905,0.1639,0.2173,0.2342,0.1795,0.1654,0.3409,0.1898,0.2714,0.1933,0.2148,0.2086,0.1854,0.2075,0.318,0.2631,0.2662,0.2043,0.198,0.2357,0.4606,0.1989,0.2125,0.8197,0.1772,0.1946,0.2049,0.2126,0.3405,0.2114,0.2241,0.2808,0.3114,0.2662,0.4583,0.7446,0.5242,0.7183,0.2633,0.1899,0.3545,0.5121,0.225,0.3802,0.2033,0.22,0.3835,0.2095,0.2205,0.4114,0.1947,0.2034,0.2096,0.196,0.1864,0.2982,0.4778,0.1718,0.1965,2.4331,0.6876,0.2781,0.2325,0.5453,0.4331,0.8445,0.2312,0.2311,0.2583,0.2272,0.4364,0.3035,0.2897,0.2309,0.2349,0.2477,0.1871,0.196,0.2455,0.2108,0.2026,0.1456,0.186,0.1855,0.2343,0.2122,0.2106,0.3363,0.192,0.3565,0.1948,0.1841,0.2529,0.2424,0.2357,0.1645,0.2532,0.211,0.1681,0.2618,0.2323,0.367,0.1893,0.1383,0.1932,0.1836,0.234,0.1703,0.2827,0.2022,0.1705,0.2091,0.2217,0.272,0.2694,0.1856,0.1844,0.1822,0.2477,0.2248,0.2184,0.2641,0.1849,0.2154,0.2941,0.1941,0.1875,0.212,0.2201,0.1726,0.2165,0.1359,0.1906,0.5658,0.2931,0.2374],"byWindow":[{"from":1,"to":250,"clipped":62,"median":0.7628},{"from":251,"to":500,"clipped":0,"median":0.4372},{"from":501,"to":750,"clipped":1,"median":0.4076},{"from":751,"to":1000,"clipped":8,"median":0.3719},{"from":1001,"to":1250,"clipped":1,"median":0.2953},{"from":1251,"to":1500,"clipped":0,"median":0.252},{"from":1501,"to":1750,"clipped":2,"median":0.2438},{"from":1751,"to":2000,"clipped":2,"median":0.2216}]},"variants":{"about":"One change each from the reference recipe (batch 16, lr 0.01 with 30 warmup steps, clip 1, weight decay 0.01, data seed 11, model seed 7), trained 1000 steps where not stated and scored every 50 steps on the same 256 held-out rows. reference is the reproduced run's first 1000 steps. Batch runs: batch4Steps keeps the step count (a quarter of the words); batch4Tokens and batch64Tokens keep the words seen (384000 target words) and so change the step count; the learning rate is not rescaled with batch size. noise: the reference recipe with data seeds 12 and 13, to show how far two runs that differ only in luck land apart; a variant inside that spread is not shown to differ. firstCopyStep: first scored step with copy accuracy at or above 0.9 (null: never in this run). One seed per variant: a single run each, so differences smaller than the noise runs' spread are not evidence.","reference":{"name":"reference","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.6981,"copyAccuracy":0.996,"localAccuracy":0.7013,"bestHeldOut":0.6981,"bestStep":1000,"firstCopyStep":700,"diverged":false},"curve":[{"step":0,"tokens":0,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"heldOut":1.1656,"copyAccuracy":0.2134,"localAccuracy":0.6078},{"step":100,"tokens":38400,"heldOut":0.9654,"copyAccuracy":0.1561,"localAccuracy":0.6744},{"step":150,"tokens":57600,"heldOut":0.9239,"copyAccuracy":0.2016,"localAccuracy":0.6907},{"step":200,"tokens":76800,"heldOut":0.8812,"copyAccuracy":0.2134,"localAccuracy":0.6976},{"step":250,"tokens":96000,"heldOut":0.878,"copyAccuracy":0.2826,"localAccuracy":0.6965},{"step":300,"tokens":115200,"heldOut":0.8436,"copyAccuracy":0.3399,"localAccuracy":0.7026},{"step":350,"tokens":134400,"heldOut":0.8397,"copyAccuracy":0.3261,"localAccuracy":0.6969},{"step":400,"tokens":153600,"heldOut":0.8456,"copyAccuracy":0.3221,"localAccuracy":0.6971},{"step":450,"tokens":172800,"heldOut":0.832,"copyAccuracy":0.3458,"localAccuracy":0.6997},{"step":500,"tokens":192000,"heldOut":0.8182,"copyAccuracy":0.4289,"localAccuracy":0.7057},{"step":550,"tokens":211200,"heldOut":0.8073,"copyAccuracy":0.5119,"localAccuracy":0.6919},{"step":600,"tokens":230400,"heldOut":0.7937,"copyAccuracy":0.5751,"localAccuracy":0.702},{"step":650,"tokens":249600,"heldOut":0.7728,"copyAccuracy":0.6601,"localAccuracy":0.6995},{"step":700,"tokens":268800,"heldOut":0.7289,"copyAccuracy":0.9348,"localAccuracy":0.6898},{"step":750,"tokens":288000,"heldOut":0.715,"copyAccuracy":0.9684,"localAccuracy":0.6992},{"step":800,"tokens":307200,"heldOut":0.7033,"copyAccuracy":0.9763,"localAccuracy":0.7045},{"step":850,"tokens":326400,"heldOut":0.7002,"copyAccuracy":0.9941,"localAccuracy":0.7018},{"step":900,"tokens":345600,"heldOut":0.7048,"copyAccuracy":0.9921,"localAccuracy":0.6999},{"step":950,"tokens":364800,"heldOut":0.7053,"copyAccuracy":0.9941,"localAccuracy":0.7026},{"step":1000,"tokens":384000,"heldOut":0.6981,"copyAccuracy":0.996,"localAccuracy":0.7013}]},"runs":[{"name":"lrLow","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.7835,"copyAccuracy":0.5395,"localAccuracy":0.7057,"bestHeldOut":0.7835,"bestStep":1000,"firstCopyStep":null,"diverged":false},"norms":{"n":1000,"min":0.8682,"p10":1.2007,"median":1.6268,"p90":2.5873,"max":6.3791,"clipped":989,"clipShare":0.989},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":3.3769,"heldOut":2.6992,"copyAccuracy":0.1818,"localAccuracy":0.5261},{"step":100,"tokens":38400,"train":2.1651,"heldOut":1.7542,"copyAccuracy":0.1798,"localAccuracy":0.5832},{"step":150,"tokens":57600,"train":1.5058,"heldOut":1.3694,"copyAccuracy":0.1719,"localAccuracy":0.6233},{"step":200,"tokens":76800,"train":1.2455,"heldOut":1.1961,"copyAccuracy":0.1818,"localAccuracy":0.6442},{"step":250,"tokens":96000,"train":1.1061,"heldOut":1.0837,"copyAccuracy":0.1779,"localAccuracy":0.6708},{"step":300,"tokens":115200,"train":1.0312,"heldOut":1.0118,"copyAccuracy":0.1739,"localAccuracy":0.6894},{"step":350,"tokens":134400,"train":0.9586,"heldOut":0.956,"copyAccuracy":0.2075,"localAccuracy":0.693},{"step":400,"tokens":153600,"train":0.9076,"heldOut":0.9197,"copyAccuracy":0.1739,"localAccuracy":0.6988},{"step":450,"tokens":172800,"train":0.8978,"heldOut":0.8933,"copyAccuracy":0.1818,"localAccuracy":0.6992},{"step":500,"tokens":192000,"train":0.8641,"heldOut":0.8779,"copyAccuracy":0.2806,"localAccuracy":0.702},{"step":550,"tokens":211200,"train":0.8521,"heldOut":0.8593,"copyAccuracy":0.2925,"localAccuracy":0.7045},{"step":600,"tokens":230400,"train":0.8487,"heldOut":0.861,"copyAccuracy":0.3043,"localAccuracy":0.6963},{"step":650,"tokens":249600,"train":0.8267,"heldOut":0.8441,"copyAccuracy":0.3518,"localAccuracy":0.7034},{"step":700,"tokens":268800,"train":0.8239,"heldOut":0.8269,"copyAccuracy":0.3893,"localAccuracy":0.7045},{"step":750,"tokens":288000,"train":0.803,"heldOut":0.8093,"copyAccuracy":0.4763,"localAccuracy":0.704},{"step":800,"tokens":307200,"train":0.7935,"heldOut":0.8066,"copyAccuracy":0.4466,"localAccuracy":0.6995},{"step":850,"tokens":326400,"train":0.7857,"heldOut":0.8027,"copyAccuracy":0.4842,"localAccuracy":0.6995},{"step":900,"tokens":345600,"train":0.7712,"heldOut":0.7978,"copyAccuracy":0.498,"localAccuracy":0.6976},{"step":950,"tokens":364800,"train":0.7711,"heldOut":0.7915,"copyAccuracy":0.5573,"localAccuracy":0.6994},{"step":1000,"tokens":384000,"train":0.7682,"heldOut":0.7835,"copyAccuracy":0.5395,"localAccuracy":0.7057}],"options":{"lr":0.001}},{"name":"lrMid","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.8353,"copyAccuracy":0.3439,"localAccuracy":0.7026,"bestHeldOut":0.8226,"bestStep":950,"firstCopyStep":null,"diverged":false},"norms":{"n":1000,"min":0.1227,"p10":0.1777,"median":0.2769,"p90":0.6382,"max":2.0877,"clipped":34,"clipShare":0.034},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":1.7981,"heldOut":1.1476,"copyAccuracy":0.2075,"localAccuracy":0.6164},{"step":100,"tokens":38400,"train":1.0726,"heldOut":1.0615,"copyAccuracy":0.1779,"localAccuracy":0.643},{"step":150,"tokens":57600,"train":0.9473,"heldOut":0.9926,"copyAccuracy":0.1976,"localAccuracy":0.6673},{"step":200,"tokens":76800,"train":0.9101,"heldOut":0.8906,"copyAccuracy":0.1996,"localAccuracy":0.6974},{"step":250,"tokens":96000,"train":0.8644,"heldOut":0.8881,"copyAccuracy":0.1838,"localAccuracy":0.6972},{"step":300,"tokens":115200,"train":0.8691,"heldOut":0.8737,"copyAccuracy":0.1601,"localAccuracy":0.6997},{"step":350,"tokens":134400,"train":0.8513,"heldOut":0.8686,"copyAccuracy":0.1957,"localAccuracy":0.7029},{"step":400,"tokens":153600,"train":0.8438,"heldOut":0.863,"copyAccuracy":0.2589,"localAccuracy":0.6974},{"step":450,"tokens":172800,"train":0.8559,"heldOut":0.8457,"copyAccuracy":0.2727,"localAccuracy":0.6971},{"step":500,"tokens":192000,"train":0.8391,"heldOut":0.8556,"copyAccuracy":0.3261,"localAccuracy":0.6967},{"step":550,"tokens":211200,"train":0.8257,"heldOut":0.8333,"copyAccuracy":0.3083,"localAccuracy":0.7043},{"step":600,"tokens":230400,"train":0.8263,"heldOut":0.8443,"copyAccuracy":0.3261,"localAccuracy":0.7004},{"step":650,"tokens":249600,"train":0.8221,"heldOut":0.8418,"copyAccuracy":0.3419,"localAccuracy":0.7002},{"step":700,"tokens":268800,"train":0.8385,"heldOut":0.867,"copyAccuracy":0.3458,"localAccuracy":0.6885},{"step":750,"tokens":288000,"train":0.8344,"heldOut":0.8485,"copyAccuracy":0.334,"localAccuracy":0.6935},{"step":800,"tokens":307200,"train":0.8203,"heldOut":0.8269,"copyAccuracy":0.3557,"localAccuracy":0.7068},{"step":850,"tokens":326400,"train":0.8156,"heldOut":0.8467,"copyAccuracy":0.3221,"localAccuracy":0.6924},{"step":900,"tokens":345600,"train":0.8209,"heldOut":0.8354,"copyAccuracy":0.3123,"localAccuracy":0.6997},{"step":950,"tokens":364800,"train":0.8103,"heldOut":0.8226,"copyAccuracy":0.3696,"localAccuracy":0.6972},{"step":1000,"tokens":384000,"train":0.8036,"heldOut":0.8353,"copyAccuracy":0.3439,"localAccuracy":0.7026}],"options":{"lr":0.03}},{"name":"lrHigh","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.9506,"copyAccuracy":0.2174,"localAccuracy":0.6903,"bestHeldOut":0.9258,"bestStep":500,"firstCopyStep":null,"diverged":false},"norms":{"n":1000,"min":0.1795,"p10":0.269,"median":0.4291,"p90":0.9117,"max":15.3551,"clipped":71,"clipShare":0.071},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":1.7417,"heldOut":1.262,"copyAccuracy":0.1957,"localAccuracy":0.5706},{"step":100,"tokens":38400,"train":1.1709,"heldOut":1.1629,"copyAccuracy":0.1779,"localAccuracy":0.613},{"step":150,"tokens":57600,"train":1.0986,"heldOut":1.037,"copyAccuracy":0.1917,"localAccuracy":0.6582},{"step":200,"tokens":76800,"train":1.0696,"heldOut":1.077,"copyAccuracy":0.1561,"localAccuracy":0.6522},{"step":250,"tokens":96000,"train":1.0065,"heldOut":1.0996,"copyAccuracy":0.1838,"localAccuracy":0.6449},{"step":300,"tokens":115200,"train":1.0199,"heldOut":0.9829,"copyAccuracy":0.1542,"localAccuracy":0.6708},{"step":350,"tokens":134400,"train":0.9233,"heldOut":0.9296,"copyAccuracy":0.1798,"localAccuracy":0.6896},{"step":400,"tokens":153600,"train":0.9548,"heldOut":0.9827,"copyAccuracy":0.1818,"localAccuracy":0.6735},{"step":450,"tokens":172800,"train":1.009,"heldOut":1.0099,"copyAccuracy":0.1739,"localAccuracy":0.6626},{"step":500,"tokens":192000,"train":0.919,"heldOut":0.9258,"copyAccuracy":0.1581,"localAccuracy":0.6877},{"step":550,"tokens":211200,"train":0.8894,"heldOut":0.9285,"copyAccuracy":0.2233,"localAccuracy":0.688},{"step":600,"tokens":230400,"train":0.9423,"heldOut":0.9283,"copyAccuracy":0.1877,"localAccuracy":0.6845},{"step":650,"tokens":249600,"train":0.9278,"heldOut":1.0389,"copyAccuracy":0.1542,"localAccuracy":0.6598},{"step":700,"tokens":268800,"train":1.004,"heldOut":0.9927,"copyAccuracy":0.2115,"localAccuracy":0.6752},{"step":750,"tokens":288000,"train":0.9721,"heldOut":0.9485,"copyAccuracy":0.2075,"localAccuracy":0.6761},{"step":800,"tokens":307200,"train":0.9484,"heldOut":0.9941,"copyAccuracy":0.1522,"localAccuracy":0.658},{"step":850,"tokens":326400,"train":0.9978,"heldOut":0.9822,"copyAccuracy":0.2134,"localAccuracy":0.6728},{"step":900,"tokens":345600,"train":0.9349,"heldOut":1.0085,"copyAccuracy":0.166,"localAccuracy":0.6715},{"step":950,"tokens":364800,"train":0.9964,"heldOut":1.0023,"copyAccuracy":0.1759,"localAccuracy":0.6676},{"step":1000,"tokens":384000,"train":0.938,"heldOut":0.9506,"copyAccuracy":0.2174,"localAccuracy":0.6903}],"options":{"lr":0.1}},{"name":"noClip","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.714,"copyAccuracy":0.9506,"localAccuracy":0.7006,"bestHeldOut":0.714,"bestStep":1000,"firstCopyStep":850,"diverged":false},"norms":{"n":1000,"min":0.2621,"p10":0.325,"median":0.4375,"p90":0.8084,"max":2.0877,"clipped":64,"clipShare":0.064},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":2.1586,"heldOut":1.1867,"copyAccuracy":0.2174,"localAccuracy":0.6165},{"step":100,"tokens":38400,"train":1.0487,"heldOut":1.034,"copyAccuracy":0.1601,"localAccuracy":0.6568},{"step":150,"tokens":57600,"train":0.9297,"heldOut":0.9253,"copyAccuracy":0.1858,"localAccuracy":0.691},{"step":200,"tokens":76800,"train":0.8729,"heldOut":0.8739,"copyAccuracy":0.2115,"localAccuracy":0.7036},{"step":250,"tokens":96000,"train":0.8532,"heldOut":0.8945,"copyAccuracy":0.1739,"localAccuracy":0.6896},{"step":300,"tokens":115200,"train":0.8583,"heldOut":0.8557,"copyAccuracy":0.1858,"localAccuracy":0.7018},{"step":350,"tokens":134400,"train":0.8419,"heldOut":0.8596,"copyAccuracy":0.1719,"localAccuracy":0.6992},{"step":400,"tokens":153600,"train":0.8356,"heldOut":0.8578,"copyAccuracy":0.2115,"localAccuracy":0.6997},{"step":450,"tokens":172800,"train":0.8459,"heldOut":0.8411,"copyAccuracy":0.2826,"localAccuracy":0.701},{"step":500,"tokens":192000,"train":0.8223,"heldOut":0.8444,"copyAccuracy":0.3419,"localAccuracy":0.6997},{"step":550,"tokens":211200,"train":0.8174,"heldOut":0.8411,"copyAccuracy":0.3063,"localAccuracy":0.6933},{"step":600,"tokens":230400,"train":0.8174,"heldOut":0.8342,"copyAccuracy":0.3538,"localAccuracy":0.7008},{"step":650,"tokens":249600,"train":0.8038,"heldOut":0.8243,"copyAccuracy":0.3557,"localAccuracy":0.6979},{"step":700,"tokens":268800,"train":0.8086,"heldOut":0.8144,"copyAccuracy":0.417,"localAccuracy":0.6935},{"step":750,"tokens":288000,"train":0.7887,"heldOut":0.7983,"copyAccuracy":0.5455,"localAccuracy":0.6979},{"step":800,"tokens":307200,"train":0.7626,"heldOut":0.7616,"copyAccuracy":0.7925,"localAccuracy":0.7008},{"step":850,"tokens":326400,"train":0.7299,"heldOut":0.7412,"copyAccuracy":0.9229,"localAccuracy":0.7004},{"step":900,"tokens":345600,"train":0.6931,"heldOut":0.7252,"copyAccuracy":0.9328,"localAccuracy":0.6947},{"step":950,"tokens":364800,"train":0.6851,"heldOut":0.7266,"copyAccuracy":0.9328,"localAccuracy":0.6976},{"step":1000,"tokens":384000,"train":0.6922,"heldOut":0.714,"copyAccuracy":0.9506,"localAccuracy":0.7006}],"options":{"clip":"none"}},{"name":"noDecay","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.6995,"copyAccuracy":0.9822,"localAccuracy":0.7038,"bestHeldOut":0.6928,"bestStep":950,"firstCopyStep":700,"diverged":false},"norms":{"n":1000,"min":0.1988,"p10":0.3009,"median":0.4576,"p90":0.876,"max":2.281,"clipped":67,"clipShare":0.067},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":2.1517,"heldOut":1.1653,"copyAccuracy":0.2134,"localAccuracy":0.6075},{"step":100,"tokens":38400,"train":1.0399,"heldOut":0.9636,"copyAccuracy":0.1601,"localAccuracy":0.6728},{"step":150,"tokens":57600,"train":0.9164,"heldOut":0.9741,"copyAccuracy":0.1858,"localAccuracy":0.6878},{"step":200,"tokens":76800,"train":0.8899,"heldOut":0.8787,"copyAccuracy":0.1877,"localAccuracy":0.7045},{"step":250,"tokens":96000,"train":0.8457,"heldOut":0.8817,"copyAccuracy":0.2628,"localAccuracy":0.696},{"step":300,"tokens":115200,"train":0.8436,"heldOut":0.8481,"copyAccuracy":0.2431,"localAccuracy":0.7052},{"step":350,"tokens":134400,"train":0.8306,"heldOut":0.8453,"copyAccuracy":0.3004,"localAccuracy":0.7011},{"step":400,"tokens":153600,"train":0.8168,"heldOut":0.822,"copyAccuracy":0.4229,"localAccuracy":0.6995},{"step":450,"tokens":172800,"train":0.8057,"heldOut":0.8034,"copyAccuracy":0.498,"localAccuracy":0.6981},{"step":500,"tokens":192000,"train":0.7666,"heldOut":0.7648,"copyAccuracy":0.7332,"localAccuracy":0.7018},{"step":550,"tokens":211200,"train":0.7327,"heldOut":0.7502,"copyAccuracy":0.8399,"localAccuracy":0.6935},{"step":600,"tokens":230400,"train":0.7272,"heldOut":0.7454,"copyAccuracy":0.8241,"localAccuracy":0.6956},{"step":650,"tokens":249600,"train":0.711,"heldOut":0.7316,"copyAccuracy":0.8676,"localAccuracy":0.6971},{"step":700,"tokens":268800,"train":0.7003,"heldOut":0.7075,"copyAccuracy":0.9526,"localAccuracy":0.696},{"step":750,"tokens":288000,"train":0.6862,"heldOut":0.7084,"copyAccuracy":0.9723,"localAccuracy":0.702},{"step":800,"tokens":307200,"train":0.6864,"heldOut":0.7013,"copyAccuracy":0.9802,"localAccuracy":0.7045},{"step":850,"tokens":326400,"train":0.6753,"heldOut":0.7045,"copyAccuracy":0.9941,"localAccuracy":0.6994},{"step":900,"tokens":345600,"train":0.6678,"heldOut":0.6982,"copyAccuracy":0.998,"localAccuracy":0.6974},{"step":950,"tokens":364800,"train":0.6648,"heldOut":0.6928,"copyAccuracy":0.9881,"localAccuracy":0.699},{"step":1000,"tokens":384000,"train":0.6756,"heldOut":0.6995,"copyAccuracy":0.9822,"localAccuracy":0.7038}],"options":{"weightDecay":0}},{"name":"batch4Steps","steps":1000,"tokens":96000,"summary":{"endStep":1000,"heldOut":0.8702,"copyAccuracy":0.2292,"localAccuracy":0.6981,"bestHeldOut":0.8702,"bestStep":1000,"firstCopyStep":null,"diverged":false},"norms":{"n":1000,"min":0.3473,"p10":0.5046,"median":0.7863,"p90":1.3101,"max":2.4823,"clipped":285,"clipShare":0.285},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":4800,"train":2.2997,"heldOut":1.3561,"copyAccuracy":0.1957,"localAccuracy":0.5573},{"step":100,"tokens":9600,"train":1.2229,"heldOut":1.1964,"copyAccuracy":0.1561,"localAccuracy":0.597},{"step":150,"tokens":14400,"train":1.1598,"heldOut":1.1151,"copyAccuracy":0.2095,"localAccuracy":0.6174},{"step":200,"tokens":19200,"train":1.072,"heldOut":1.0932,"copyAccuracy":0.1996,"localAccuracy":0.6385},{"step":250,"tokens":24000,"train":1.0646,"heldOut":1.0151,"copyAccuracy":0.1976,"localAccuracy":0.6611},{"step":300,"tokens":28800,"train":1.0076,"heldOut":0.9738,"copyAccuracy":0.1779,"localAccuracy":0.6697},{"step":350,"tokens":33600,"train":0.9738,"heldOut":0.9494,"copyAccuracy":0.1996,"localAccuracy":0.6809},{"step":400,"tokens":38400,"train":0.9574,"heldOut":0.9366,"copyAccuracy":0.168,"localAccuracy":0.6914},{"step":450,"tokens":43200,"train":0.9354,"heldOut":0.9515,"copyAccuracy":0.1779,"localAccuracy":0.6846},{"step":500,"tokens":48000,"train":0.913,"heldOut":0.9282,"copyAccuracy":0.1877,"localAccuracy":0.6878},{"step":550,"tokens":52800,"train":0.8818,"heldOut":0.9134,"copyAccuracy":0.1877,"localAccuracy":0.6942},{"step":600,"tokens":57600,"train":0.8834,"heldOut":0.9159,"copyAccuracy":0.1779,"localAccuracy":0.6862},{"step":650,"tokens":62400,"train":0.8706,"heldOut":0.8982,"copyAccuracy":0.1759,"localAccuracy":0.6924},{"step":700,"tokens":67200,"train":0.8688,"heldOut":0.8989,"copyAccuracy":0.1996,"localAccuracy":0.688},{"step":750,"tokens":72000,"train":0.8965,"heldOut":0.8996,"copyAccuracy":0.1779,"localAccuracy":0.6841},{"step":800,"tokens":76800,"train":0.8853,"heldOut":0.8961,"copyAccuracy":0.1917,"localAccuracy":0.6955},{"step":850,"tokens":81600,"train":0.8688,"heldOut":0.8974,"copyAccuracy":0.1522,"localAccuracy":0.696},{"step":900,"tokens":86400,"train":0.8585,"heldOut":0.8993,"copyAccuracy":0.166,"localAccuracy":0.6846},{"step":950,"tokens":91200,"train":0.8584,"heldOut":0.894,"copyAccuracy":0.2055,"localAccuracy":0.6919},{"step":1000,"tokens":96000,"train":0.8585,"heldOut":0.8702,"copyAccuracy":0.2292,"localAccuracy":0.6981}],"options":{"batch":4}},{"name":"batch4Tokens","steps":4000,"tokens":384000,"summary":{"endStep":4000,"heldOut":0.7636,"copyAccuracy":0.6581,"localAccuracy":0.7075,"bestHeldOut":0.757,"bestStep":3600,"firstCopyStep":null,"diverged":false},"norms":{"n":4000,"min":0.2263,"p10":0.3201,"median":0.4591,"p90":0.9809,"max":3.6731,"clipped":378,"clipShare":0.0945},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":200,"tokens":19200,"train":1.4386,"heldOut":1.0932,"copyAccuracy":0.1996,"localAccuracy":0.6385},{"step":400,"tokens":38400,"train":1.0009,"heldOut":0.9366,"copyAccuracy":0.168,"localAccuracy":0.6914},{"step":600,"tokens":57600,"train":0.9034,"heldOut":0.9159,"copyAccuracy":0.1779,"localAccuracy":0.6862},{"step":800,"tokens":76800,"train":0.8803,"heldOut":0.8961,"copyAccuracy":0.1917,"localAccuracy":0.6955},{"step":1000,"tokens":96000,"train":0.8611,"heldOut":0.8702,"copyAccuracy":0.2292,"localAccuracy":0.6981},{"step":1200,"tokens":115200,"train":0.8629,"heldOut":0.8526,"copyAccuracy":0.3261,"localAccuracy":0.6994},{"step":1400,"tokens":134400,"train":0.8324,"heldOut":0.8466,"copyAccuracy":0.3142,"localAccuracy":0.6921},{"step":1600,"tokens":153600,"train":0.8266,"heldOut":0.8517,"copyAccuracy":0.3024,"localAccuracy":0.6924},{"step":1800,"tokens":172800,"train":0.8399,"heldOut":0.8333,"copyAccuracy":0.3281,"localAccuracy":0.7029},{"step":2000,"tokens":192000,"train":0.8121,"heldOut":0.8166,"copyAccuracy":0.3992,"localAccuracy":0.6995},{"step":2200,"tokens":211200,"train":0.7973,"heldOut":0.8263,"copyAccuracy":0.4269,"localAccuracy":0.6983},{"step":2400,"tokens":230400,"train":0.7934,"heldOut":0.8098,"copyAccuracy":0.5178,"localAccuracy":0.6988},{"step":2600,"tokens":249600,"train":0.7834,"heldOut":0.7933,"copyAccuracy":0.5455,"localAccuracy":0.6994},{"step":2800,"tokens":268800,"train":0.7822,"heldOut":0.7885,"copyAccuracy":0.5553,"localAccuracy":0.6983},{"step":3000,"tokens":288000,"train":0.7725,"heldOut":0.7808,"copyAccuracy":0.5415,"localAccuracy":0.7001},{"step":3200,"tokens":307200,"train":0.7606,"heldOut":0.7735,"copyAccuracy":0.6344,"localAccuracy":0.702},{"step":3400,"tokens":326400,"train":0.7552,"heldOut":0.7849,"copyAccuracy":0.6245,"localAccuracy":0.7029},{"step":3600,"tokens":345600,"train":0.7459,"heldOut":0.757,"copyAccuracy":0.6561,"localAccuracy":0.6987},{"step":3800,"tokens":364800,"train":0.7472,"heldOut":0.7676,"copyAccuracy":0.6443,"localAccuracy":0.6976},{"step":4000,"tokens":384000,"train":0.7468,"heldOut":0.7636,"copyAccuracy":0.6581,"localAccuracy":0.7075}],"options":{"batch":4}},{"name":"batch64Tokens","steps":250,"tokens":384000,"summary":{"endStep":250,"heldOut":0.752,"copyAccuracy":0.7233,"localAccuracy":0.7049,"bestHeldOut":0.752,"bestStep":250,"firstCopyStep":null,"diverged":false},"norms":{"n":250,"min":0.2789,"p10":0.3613,"median":0.5528,"p90":1.1873,"max":2.5411,"clipped":36,"clipShare":0.144},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":25,"tokens":38400,"train":2.9359,"heldOut":1.6213,"copyAccuracy":0.1779,"localAccuracy":0.5738},{"step":50,"tokens":76800,"train":1.1977,"heldOut":1.0426,"copyAccuracy":0.1897,"localAccuracy":0.6508},{"step":75,"tokens":115200,"train":0.9393,"heldOut":0.9222,"copyAccuracy":0.2055,"localAccuracy":0.6887},{"step":100,"tokens":153600,"train":0.8741,"heldOut":0.8843,"copyAccuracy":0.1996,"localAccuracy":0.7015},{"step":125,"tokens":192000,"train":0.8543,"heldOut":0.8469,"copyAccuracy":0.3043,"localAccuracy":0.702},{"step":150,"tokens":230400,"train":0.826,"heldOut":0.826,"copyAccuracy":0.4209,"localAccuracy":0.7022},{"step":175,"tokens":268800,"train":0.8052,"heldOut":0.807,"copyAccuracy":0.5178,"localAccuracy":0.702},{"step":200,"tokens":307200,"train":0.7864,"heldOut":0.7901,"copyAccuracy":0.5356,"localAccuracy":0.7043},{"step":225,"tokens":345600,"train":0.7619,"heldOut":0.7685,"copyAccuracy":0.6502,"localAccuracy":0.705},{"step":250,"tokens":384000,"train":0.7376,"heldOut":0.752,"copyAccuracy":0.7233,"localAccuracy":0.7049}],"options":{"batch":64}}],"noise":[{"name":"dataSeed12","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.7,"copyAccuracy":0.9802,"localAccuracy":0.7031,"bestHeldOut":0.6911,"bestStep":900,"firstCopyStep":650,"diverged":false},"norms":{"n":1000,"min":0.2031,"p10":0.2975,"median":0.4591,"p90":0.8667,"max":2.0388,"clipped":65,"clipShare":0.065},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":2.1357,"heldOut":1.1468,"copyAccuracy":0.1858,"localAccuracy":0.6197},{"step":100,"tokens":38400,"train":1.0117,"heldOut":0.9501,"copyAccuracy":0.1779,"localAccuracy":0.6859},{"step":150,"tokens":57600,"train":0.9102,"heldOut":0.9199,"copyAccuracy":0.1818,"localAccuracy":0.6893},{"step":200,"tokens":76800,"train":0.8677,"heldOut":0.896,"copyAccuracy":0.1502,"localAccuracy":0.696},{"step":250,"tokens":96000,"train":0.8583,"heldOut":0.898,"copyAccuracy":0.2727,"localAccuracy":0.6949},{"step":300,"tokens":115200,"train":0.8475,"heldOut":0.8608,"copyAccuracy":0.334,"localAccuracy":0.6949},{"step":350,"tokens":134400,"train":0.8285,"heldOut":0.8538,"copyAccuracy":0.334,"localAccuracy":0.7002},{"step":400,"tokens":153600,"train":0.8219,"heldOut":0.8265,"copyAccuracy":0.3814,"localAccuracy":0.696},{"step":450,"tokens":172800,"train":0.8027,"heldOut":0.8252,"copyAccuracy":0.5553,"localAccuracy":0.7004},{"step":500,"tokens":192000,"train":0.7859,"heldOut":0.7666,"copyAccuracy":0.6759,"localAccuracy":0.6983},{"step":550,"tokens":211200,"train":0.7477,"heldOut":0.7759,"copyAccuracy":0.6779,"localAccuracy":0.7002},{"step":600,"tokens":230400,"train":0.7445,"heldOut":0.7404,"copyAccuracy":0.747,"localAccuracy":0.6979},{"step":650,"tokens":249600,"train":0.7158,"heldOut":0.7249,"copyAccuracy":0.9368,"localAccuracy":0.6988},{"step":700,"tokens":268800,"train":0.6881,"heldOut":0.7081,"copyAccuracy":0.9822,"localAccuracy":0.6987},{"step":750,"tokens":288000,"train":0.6836,"heldOut":0.7158,"copyAccuracy":0.9783,"localAccuracy":0.6971},{"step":800,"tokens":307200,"train":0.6766,"heldOut":0.6975,"copyAccuracy":0.9921,"localAccuracy":0.6873},{"step":850,"tokens":326400,"train":0.6688,"heldOut":0.6994,"copyAccuracy":0.998,"localAccuracy":0.7015},{"step":900,"tokens":345600,"train":0.6631,"heldOut":0.6911,"copyAccuracy":0.998,"localAccuracy":0.699},{"step":950,"tokens":364800,"train":0.674,"heldOut":0.6995,"copyAccuracy":0.9743,"localAccuracy":0.7011},{"step":1000,"tokens":384000,"train":0.6777,"heldOut":0.7,"copyAccuracy":0.9802,"localAccuracy":0.7031}],"options":{"seed":12}},{"name":"dataSeed13","steps":1000,"tokens":384000,"summary":{"endStep":1000,"heldOut":0.7019,"copyAccuracy":0.9921,"localAccuracy":0.7013,"bestHeldOut":0.6916,"bestStep":950,"firstCopyStep":750,"diverged":false},"norms":{"n":1000,"min":0.1935,"p10":0.2935,"median":0.4654,"p90":0.9125,"max":3.2457,"clipped":72,"clipShare":0.072},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":2.1393,"heldOut":1.1344,"copyAccuracy":0.1937,"localAccuracy":0.621},{"step":100,"tokens":38400,"train":1.0356,"heldOut":0.9956,"copyAccuracy":0.2134,"localAccuracy":0.668},{"step":150,"tokens":57600,"train":0.9263,"heldOut":0.9013,"copyAccuracy":0.168,"localAccuracy":0.6953},{"step":200,"tokens":76800,"train":0.8759,"heldOut":0.8831,"copyAccuracy":0.1937,"localAccuracy":0.6919},{"step":250,"tokens":96000,"train":0.8705,"heldOut":0.8671,"copyAccuracy":0.2332,"localAccuracy":0.6987},{"step":300,"tokens":115200,"train":0.8439,"heldOut":0.8515,"copyAccuracy":0.3419,"localAccuracy":0.6962},{"step":350,"tokens":134400,"train":0.823,"heldOut":0.822,"copyAccuracy":0.4901,"localAccuracy":0.6914},{"step":400,"tokens":153600,"train":0.8,"heldOut":0.8157,"copyAccuracy":0.4447,"localAccuracy":0.7026},{"step":450,"tokens":172800,"train":0.7643,"heldOut":0.7721,"copyAccuracy":0.6858,"localAccuracy":0.6955},{"step":500,"tokens":192000,"train":0.7481,"heldOut":0.7465,"copyAccuracy":0.747,"localAccuracy":0.704},{"step":550,"tokens":211200,"train":0.7375,"heldOut":0.7438,"copyAccuracy":0.7431,"localAccuracy":0.6926},{"step":600,"tokens":230400,"train":0.7282,"heldOut":0.7633,"copyAccuracy":0.6976,"localAccuracy":0.6992},{"step":650,"tokens":249600,"train":0.726,"heldOut":0.7599,"copyAccuracy":0.7668,"localAccuracy":0.6979},{"step":700,"tokens":268800,"train":0.7198,"heldOut":0.7526,"copyAccuracy":0.8162,"localAccuracy":0.6923},{"step":750,"tokens":288000,"train":0.6995,"heldOut":0.6983,"copyAccuracy":0.9881,"localAccuracy":0.6971},{"step":800,"tokens":307200,"train":0.674,"heldOut":0.6982,"copyAccuracy":0.9822,"localAccuracy":0.6978},{"step":850,"tokens":326400,"train":0.6819,"heldOut":0.7056,"copyAccuracy":0.9704,"localAccuracy":0.6972},{"step":900,"tokens":345600,"train":0.6851,"heldOut":0.7027,"copyAccuracy":0.9901,"localAccuracy":0.6978},{"step":950,"tokens":364800,"train":0.6718,"heldOut":0.6916,"copyAccuracy":0.9941,"localAccuracy":0.7001},{"step":1000,"tokens":384000,"train":0.6712,"heldOut":0.7019,"copyAccuracy":0.9921,"localAccuracy":0.7013}],"options":{"seed":13}}]},"controls":{"about":"Two broken loops that must fail to learn, from the same seeds, 200 steps each. lrZero: learning rate 0, so every update is zero and the held-out loss must stay at its starting value. uphill: the gradient's sign flipped before the optimiser, so each step climbs the loss. startLoss is the untrained network's held-out loss on the same rows. A control that fails shows the measurement can tell learning from not learning; it does not show the normal recipe is the best one.","startLoss":3.8414,"runs":[{"name":"lrZero","steps":200,"tokens":76800,"summary":{"endStep":200,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204,"bestHeldOut":3.8414,"bestStep":0,"firstCopyStep":null,"diverged":false},"norms":{"n":200,"min":1.7676,"p10":1.8427,"median":1.9322,"p90":2.0494,"max":2.2418,"clipped":200,"clipShare":1},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":3.843,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":100,"tokens":38400,"train":3.8435,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":150,"tokens":57600,"train":3.8417,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":200,"tokens":76800,"train":3.8428,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204}],"end":{"endStep":200,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204,"bestHeldOut":3.8414,"bestStep":0,"firstCopyStep":null,"diverged":false},"options":{"lr":0}},{"name":"uphill","steps":200,"tokens":76800,"summary":{"endStep":200,"heldOut":68.4964,"copyAccuracy":0,"localAccuracy":0.0076,"bestHeldOut":3.8414,"bestStep":0,"firstCopyStep":null,"diverged":false},"norms":{"n":200,"min":1.2448,"p10":2.5632,"median":17.836,"p90":30.0621,"max":626.938,"clipped":200,"clipShare":1},"curve":[{"step":0,"tokens":0,"train":null,"heldOut":3.8414,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":50,"tokens":19200,"train":12.0749,"heldOut":32.4275,"copyAccuracy":0,"localAccuracy":0},{"step":100,"tokens":38400,"train":52.848,"heldOut":63.4371,"copyAccuracy":0,"localAccuracy":0.0048},{"step":150,"tokens":57600,"train":64.8302,"heldOut":66.949,"copyAccuracy":0,"localAccuracy":0.0007},{"step":200,"tokens":76800,"train":67.9197,"heldOut":68.4964,"copyAccuracy":0,"localAccuracy":0.0076}],"end":{"endStep":200,"heldOut":68.4964,"copyAccuracy":0,"localAccuracy":0.0076,"bestHeldOut":3.8414,"bestStep":0,"firstCopyStep":null,"diverged":false},"options":{"gradSign":-1}}]},"speed":{"about":"Time per training step in Node (V8, one thread, float32 buffers), measured alone after the training runs finished: anatomy times 60 steps split into drawing the batch, a forward pass, forward plus backward, and the optimiser update. tokensPerSecond counts target words per second of a full step. reproduceMsPerStep is the wall time per step of the reproduced run, which shared the machine with up to 3 other runs. One machine and one engine: a browser tab or another CPU will differ.","anatomy":{"n":60,"batch":16,"ctx":24,"ms":{"batch":0.1204,"forward":6.3171,"forwardBackward":20.3267,"update":0.2266},"backwardOnly":14.0096},"msPerStep":20.674,"tokensPerSecond":18574,"reproduceMsPerStep":22.943,"reproduceSeconds":55.15,"flopsPerStep":44826624,"mflopsPerSecond":2168.3},"seconds":82.6}