{"about":"Lesson L16, evaluating a language model. Computed by tools/school/fill_evaluate.mjs from system/school/evaluate.mjs, evaluate-judge.mjs, evaluate-train.mjs, the attention lab's own model recipe and the corpus generator. Do not edit by hand. Losses are in nats per word. Intervals are 95 percent percentile bootstrap intervals resampling rows (errorbars.mjs), seeded, so every number is reproducible. A difference is resolved when its interval excludes zero, as lesson E23 says it.","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","recipe":{"genSeed":500,"pairSeed":77,"pairRows":4096,"maxNew":10,"promptWords":"3 to 10","shuffleSeed":4242,"sets":{"A":{"seed":90001,"rows":256},"B":{"seed":90002,"rows":512}},"seeds":[1,2,3],"marks":[50,200,400,600,800,1000,1500,2000],"leaks":[[2000,256,1],[2000,256,4],[2000,256,16],[2000,32,100],[2000,32,250],[600,256,8]],"steps":2000,"trainingSeeds":[{"model":7,"data":11},{"model":8,"data":12},{"model":9,"data":13}],"batch":16,"lr":0.01},"sets":{"about":"Two held-out sets from the corpus generator with seeds 90001 (A, the set leaked into contaminated runs) and 90002 (B, never trained on). oneAnswer counts look-back targets where the generator's exact odds give the target probability 1. generatorExact is the generator's own top word on those words: right every time, the pipeline check. overlap: rows of each set that occur word for word among a clean run's 32000 training rows (chance overlap, before any leak).","A":{"rows":256,"words":6144,"oneAnswer":519,"who":393,"copy":126,"generatorExact":{"right":519,"n":519}},"B":{"rows":512,"words":12288,"oneAnswer":1013,"who":760,"copy":253,"generatorExact":{"right":1013,"n":1013}},"overlap":{"1":{"A":{"rows":256,"seenInTraining":10,"trainingRows":32000},"B":{"rows":512,"seenInTraining":17,"trainingRows":32000}},"2":{"A":{"rows":256,"seenInTraining":9,"trainingRows":32000},"B":{"rows":512,"seenInTraining":19,"trainingRows":32000}},"3":{"A":{"rows":256,"seenInTraining":11,"trainingRows":32000},"B":{"rows":512,"seenInTraining":13,"trainingRows":32000}}},"reproducesModelJson":true},"checkpoints":{"about":"The attention lab's recipe retrained at seeds 1, 2, 3 (seed 1 is model.json's own), scored on fresh set B (512 rows, never trained on) at steps 50, 200, 400, 600, 800, 1000, 1500, 2000. loss: nats per word over every word. exact: share of one-answer words (generator probability 1, look-back targets) where the model's top word is right; who and copy split it by episode. perfect and fluent: share of rows whose free continuation each judge accepts. Each is mean [95 percent row bootstrap interval]. across: mean and sd over seeds. stopped: share of continuations that ended with \".\" within 10 words; len: mean continuation length in words.","perSeed":{"1":{"50":{"loss":{"mean":1.1421,"lo":1.1248,"hi":1.1585},"exact":{"mean":0.156,"lo":0.1357,"hi":0.1764},"who":{"mean":0.1816,"lo":0.157,"hi":0.2084},"copy":{"mean":0.0791,"lo":0.0474,"hi":0.1138},"perfect":{"mean":0.4375,"lo":0.3926,"hi":0.4805},"fluent":{"mean":0.7832,"lo":0.748,"hi":0.8164},"stopped":0.9668,"len":4.1523},"200":{"loss":{"mean":0.8687,"lo":0.8538,"hi":0.8837},"exact":{"mean":0.1925,"lo":0.1661,"hi":0.2151},"who":{"mean":0.2276,"lo":0.1968,"hi":0.257},"copy":{"mean":0.087,"lo":0.0539,"hi":0.1235},"perfect":{"mean":0.75,"lo":0.7109,"hi":0.7871},"fluent":{"mean":0.9844,"lo":0.9746,"hi":0.9941},"stopped":1,"len":3.4922},"400":{"loss":{"mean":0.8335,"lo":0.8208,"hi":0.846},"exact":{"mean":0.3021,"lo":0.2811,"hi":0.3232},"who":{"mean":0.3684,"lo":0.3455,"hi":0.3908},"copy":{"mean":0.1028,"lo":0.069,"hi":0.1439},"perfect":{"mean":0.8008,"lo":0.7637,"hi":0.8359},"fluent":{"mean":0.9902,"lo":0.9805,"hi":0.998},"stopped":1,"len":3.3965},"600":{"loss":{"mean":0.789,"lo":0.7777,"hi":0.7995},"exact":{"mean":0.5202,"lo":0.4925,"hi":0.5464},"who":{"mean":0.6618,"lo":0.6378,"hi":0.6852},"copy":{"mean":0.0949,"lo":0.0627,"hi":0.1336},"perfect":{"mean":0.7949,"lo":0.7578,"hi":0.8282},"fluent":{"mean":0.9883,"lo":0.9785,"hi":0.9961},"stopped":1,"len":3.3887},"800":{"loss":{"mean":0.6844,"lo":0.6735,"hi":0.6954},"exact":{"mean":0.9773,"lo":0.9675,"hi":0.986},"who":{"mean":0.9776,"lo":0.967,"hi":0.9875},"copy":{"mean":0.9763,"lo":0.9555,"hi":0.9922},"perfect":{"mean":0.9648,"lo":0.9473,"hi":0.9805},"fluent":{"mean":0.9902,"lo":0.9824,"hi":0.998},"stopped":1,"len":3.373},"1000":{"loss":{"mean":0.6807,"lo":0.6691,"hi":0.6925},"exact":{"mean":0.9951,"lo":0.9904,"hi":0.999},"who":{"mean":0.9934,"lo":0.9872,"hi":0.9987},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9785,"lo":0.9668,"hi":0.9902},"fluent":{"mean":0.9961,"lo":0.9902,"hi":1},"stopped":1,"len":3.4043},"1500":{"loss":{"mean":0.6787,"lo":0.6671,"hi":0.6895},"exact":{"mean":0.9891,"lo":0.9826,"hi":0.9948},"who":{"mean":0.9855,"lo":0.977,"hi":0.993},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9805,"lo":0.9668,"hi":0.9922},"fluent":{"mean":0.9941,"lo":0.9863,"hi":1},"stopped":1,"len":3.3848},"2000":{"loss":{"mean":0.6659,"lo":0.6548,"hi":0.6767},"exact":{"mean":0.997,"lo":0.9931,"hi":1},"who":{"mean":0.9961,"lo":0.9908,"hi":1},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9863,"lo":0.9766,"hi":0.9961},"fluent":{"mean":1,"lo":1,"hi":1},"stopped":1,"len":3.3965}},"2":{"50":{"loss":{"mean":1.1449,"lo":1.126,"hi":1.1658},"exact":{"mean":0.1629,"lo":0.1426,"hi":0.183},"who":{"mean":0.1921,"lo":0.1677,"hi":0.2169},"copy":{"mean":0.0751,"lo":0.0453,"hi":0.1102},"perfect":{"mean":0.4746,"lo":0.4335,"hi":0.5117},"fluent":{"mean":0.7949,"lo":0.7539,"hi":0.8301},"stopped":0.9863,"len":3.6973},"200":{"loss":{"mean":0.886,"lo":0.8705,"hi":0.9},"exact":{"mean":0.1856,"lo":0.1625,"hi":0.2085},"who":{"mean":0.2039,"lo":0.1773,"hi":0.2323},"copy":{"mean":0.1304,"lo":0.0909,"hi":0.171},"perfect":{"mean":0.7285,"lo":0.6914,"hi":0.7656},"fluent":{"mean":0.9609,"lo":0.9414,"hi":0.9766},"stopped":0.9941,"len":3.5195},"400":{"loss":{"mean":0.8052,"lo":0.794,"hi":0.816},"exact":{"mean":0.46,"lo":0.4345,"hi":0.4843},"who":{"mean":0.5816,"lo":0.556,"hi":0.6063},"copy":{"mean":0.0949,"lo":0.0615,"hi":0.129},"perfect":{"mean":0.8105,"lo":0.7754,"hi":0.8418},"fluent":{"mean":0.9922,"lo":0.9824,"hi":0.998},"stopped":1,"len":3.4609},"600":{"loss":{"mean":0.7385,"lo":0.7275,"hi":0.7497},"exact":{"mean":0.7433,"lo":0.7194,"hi":0.7641},"who":{"mean":0.775,"lo":0.7513,"hi":0.7974},"copy":{"mean":0.6482,"lo":0.5872,"hi":0.7037},"perfect":{"mean":0.8789,"lo":0.8496,"hi":0.9063},"fluent":{"mean":0.9707,"lo":0.9551,"hi":0.9844},"stopped":0.998,"len":3.3574},"800":{"loss":{"mean":0.6751,"lo":0.6646,"hi":0.6852},"exact":{"mean":0.9842,"lo":0.9765,"hi":0.9912},"who":{"mean":0.9803,"lo":0.9704,"hi":0.9893},"copy":{"mean":0.996,"lo":0.9877,"hi":1},"perfect":{"mean":0.9707,"lo":0.9551,"hi":0.9844},"fluent":{"mean":0.9922,"lo":0.9844,"hi":0.998},"stopped":1,"len":3.3574},"1000":{"loss":{"mean":0.6799,"lo":0.6702,"hi":0.6891},"exact":{"mean":0.9842,"lo":0.9748,"hi":0.9918},"who":{"mean":0.9803,"lo":0.9676,"hi":0.9904},"copy":{"mean":0.996,"lo":0.9877,"hi":1},"perfect":{"mean":0.9648,"lo":0.9492,"hi":0.9785},"fluent":{"mean":0.9902,"lo":0.9805,"hi":0.998},"stopped":1,"len":3.332},"1500":{"loss":{"mean":0.6743,"lo":0.6634,"hi":0.6852},"exact":{"mean":0.9783,"lo":0.9692,"hi":0.9865},"who":{"mean":0.9763,"lo":0.9652,"hi":0.9862},"copy":{"mean":0.9842,"lo":0.9677,"hi":0.9962},"perfect":{"mean":0.9805,"lo":0.9688,"hi":0.9902},"fluent":{"mean":0.998,"lo":0.9941,"hi":1},"stopped":1,"len":3.4141},"2000":{"loss":{"mean":0.6706,"lo":0.6608,"hi":0.6802},"exact":{"mean":0.9872,"lo":0.9787,"hi":0.9939},"who":{"mean":0.9829,"lo":0.9714,"hi":0.9916},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9727,"lo":0.959,"hi":0.9863},"fluent":{"mean":0.9961,"lo":0.9902,"hi":1},"stopped":1,"len":3.3184}},"3":{"50":{"loss":{"mean":1.1359,"lo":1.1201,"hi":1.1513},"exact":{"mean":0.1658,"lo":0.1421,"hi":0.1887},"who":{"mean":0.1934,"lo":0.1647,"hi":0.2225},"copy":{"mean":0.083,"lo":0.0514,"hi":0.1188},"perfect":{"mean":0.4219,"lo":0.377,"hi":0.4648},"fluent":{"mean":0.8125,"lo":0.7773,"hi":0.8438},"stopped":0.9844,"len":3.7598},"200":{"loss":{"mean":0.863,"lo":0.8485,"hi":0.8763},"exact":{"mean":0.1589,"lo":0.1384,"hi":0.1818},"who":{"mean":0.1724,"lo":0.1482,"hi":0.1994},"copy":{"mean":0.1186,"lo":0.0792,"hi":0.1585},"perfect":{"mean":0.75,"lo":0.709,"hi":0.7852},"fluent":{"mean":0.9746,"lo":0.9609,"hi":0.9863},"stopped":1,"len":3.5215},"400":{"loss":{"mean":0.7819,"lo":0.7695,"hi":0.7945},"exact":{"mean":0.5933,"lo":0.5608,"hi":0.6212},"who":{"mean":0.6197,"lo":0.5861,"hi":0.6542},"copy":{"mean":0.5138,"lo":0.4545,"hi":0.5744},"perfect":{"mean":0.834,"lo":0.8027,"hi":0.8652},"fluent":{"mean":0.9648,"lo":0.9492,"hi":0.9785},"stopped":0.998,"len":3.4473},"600":{"loss":{"mean":0.6866,"lo":0.6762,"hi":0.6965},"exact":{"mean":0.9793,"lo":0.9702,"hi":0.9873},"who":{"mean":0.9789,"lo":0.9687,"hi":0.9881},"copy":{"mean":0.9802,"lo":0.9605,"hi":0.9959},"perfect":{"mean":0.9629,"lo":0.9473,"hi":0.9785},"fluent":{"mean":0.9824,"lo":0.9707,"hi":0.9922},"stopped":1,"len":3.4023},"800":{"loss":{"mean":0.6732,"lo":0.6637,"hi":0.6826},"exact":{"mean":0.9872,"lo":0.9797,"hi":0.9932},"who":{"mean":0.9842,"lo":0.9747,"hi":0.9923},"copy":{"mean":0.996,"lo":0.9877,"hi":1},"perfect":{"mean":0.9766,"lo":0.9629,"hi":0.9883},"fluent":{"mean":0.9941,"lo":0.9863,"hi":1},"stopped":1,"len":3.3848},"1000":{"loss":{"mean":0.6762,"lo":0.6651,"hi":0.6865},"exact":{"mean":0.999,"lo":0.9969,"hi":1},"who":{"mean":0.9987,"lo":0.9958,"hi":1},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9766,"lo":0.9629,"hi":0.9883},"fluent":{"mean":0.9922,"lo":0.9824,"hi":0.998},"stopped":1,"len":3.4707},"1500":{"loss":{"mean":0.664,"lo":0.6538,"hi":0.6737},"exact":{"mean":0.998,"lo":0.9949,"hi":1},"who":{"mean":0.9974,"lo":0.9932,"hi":1},"copy":{"mean":1,"lo":1,"hi":1},"perfect":{"mean":0.9863,"lo":0.9765,"hi":0.9961},"fluent":{"mean":0.998,"lo":0.9941,"hi":1},"stopped":1,"len":3.3906},"2000":{"loss":{"mean":0.6816,"lo":0.67,"hi":0.6939},"exact":{"mean":0.9852,"lo":0.9775,"hi":0.9919},"who":{"mean":0.9816,"lo":0.9714,"hi":0.9902},"copy":{"mean":0.996,"lo":0.9874,"hi":1},"perfect":{"mean":0.9824,"lo":0.9707,"hi":0.9941},"fluent":{"mean":0.998,"lo":0.9922,"hi":1},"stopped":1,"len":3.4453}}},"across":{"50":{"loss":{"values":[1.1421,1.1449,1.1359],"mean":1.141,"sd":0.0046},"exact":{"values":[0.156,0.1629,0.1658],"mean":0.1616,"sd":0.0051},"who":{"values":[0.1816,0.1921,0.1934],"mean":0.189,"sd":0.0065},"copy":{"values":[0.0791,0.0751,0.083],"mean":0.0791,"sd":0.004},"perfect":{"values":[0.4375,0.4746,0.4219],"mean":0.4447,"sd":0.0271},"fluent":{"values":[0.7832,0.7949,0.8125],"mean":0.7969,"sd":0.0147},"stopped":{"values":[0.9668,0.9863,0.9844],"mean":0.9792,"sd":0.0108},"len":{"values":[4.1523,3.6973,3.7598],"mean":3.8698,"sd":0.2467}},"200":{"loss":{"values":[0.8687,0.886,0.863],"mean":0.8726,"sd":0.012},"exact":{"values":[0.1925,0.1856,0.1589],"mean":0.179,"sd":0.0177},"who":{"values":[0.2276,0.2039,0.1724],"mean":0.2013,"sd":0.0277},"copy":{"values":[0.087,0.1304,0.1186],"mean":0.112,"sd":0.0225},"perfect":{"values":[0.75,0.7285,0.75],"mean":0.7428,"sd":0.0124},"fluent":{"values":[0.9844,0.9609,0.9746],"mean":0.9733,"sd":0.0118},"stopped":{"values":[1,0.9941,1],"mean":0.998,"sd":0.0034},"len":{"values":[3.4922,3.5195,3.5215],"mean":3.5111,"sd":0.0164}},"400":{"loss":{"values":[0.8335,0.8052,0.7819],"mean":0.8069,"sd":0.0258},"exact":{"values":[0.3021,0.46,0.5933],"mean":0.4518,"sd":0.1458},"who":{"values":[0.3684,0.5816,0.6197],"mean":0.5232,"sd":0.1354},"copy":{"values":[0.1028,0.0949,0.5138],"mean":0.2372,"sd":0.2396},"perfect":{"values":[0.8008,0.8105,0.834],"mean":0.8151,"sd":0.0171},"fluent":{"values":[0.9902,0.9922,0.9648],"mean":0.9824,"sd":0.0153},"stopped":{"values":[1,1,0.998],"mean":0.9993,"sd":0.0011},"len":{"values":[3.3965,3.4609,3.4473],"mean":3.4349,"sd":0.034}},"600":{"loss":{"values":[0.789,0.7385,0.6866],"mean":0.7381,"sd":0.0512},"exact":{"values":[0.5202,0.7433,0.9793],"mean":0.7476,"sd":0.2295},"who":{"values":[0.6618,0.775,0.9789],"mean":0.8053,"sd":0.1607},"copy":{"values":[0.0949,0.6482,0.9802],"mean":0.5744,"sd":0.4473},"perfect":{"values":[0.7949,0.8789,0.9629],"mean":0.8789,"sd":0.084},"fluent":{"values":[0.9883,0.9707,0.9824],"mean":0.9805,"sd":0.009},"stopped":{"values":[1,0.998,1],"mean":0.9993,"sd":0.0011},"len":{"values":[3.3887,3.3574,3.4023],"mean":3.3828,"sd":0.023}},"800":{"loss":{"values":[0.6844,0.6751,0.6732],"mean":0.6776,"sd":0.006},"exact":{"values":[0.9773,0.9842,0.9872],"mean":0.9829,"sd":0.0051},"who":{"values":[0.9776,0.9803,0.9842],"mean":0.9807,"sd":0.0033},"copy":{"values":[0.9763,0.996,0.996],"mean":0.9895,"sd":0.0114},"perfect":{"values":[0.9648,0.9707,0.9766],"mean":0.9707,"sd":0.0059},"fluent":{"values":[0.9902,0.9922,0.9941],"mean":0.9922,"sd":0.002},"stopped":{"values":[1,1,1],"mean":1,"sd":0},"len":{"values":[3.373,3.3574,3.3848],"mean":3.3717,"sd":0.0137}},"1000":{"loss":{"values":[0.6807,0.6799,0.6762],"mean":0.6789,"sd":0.0024},"exact":{"values":[0.9951,0.9842,0.999],"mean":0.9928,"sd":0.0077},"who":{"values":[0.9934,0.9803,0.9987],"mean":0.9908,"sd":0.0095},"copy":{"values":[1,0.996,1],"mean":0.9987,"sd":0.0023},"perfect":{"values":[0.9785,0.9648,0.9766],"mean":0.9733,"sd":0.0074},"fluent":{"values":[0.9961,0.9902,0.9922],"mean":0.9928,"sd":0.003},"stopped":{"values":[1,1,1],"mean":1,"sd":0},"len":{"values":[3.4043,3.332,3.4707],"mean":3.4023,"sd":0.0694}},"1500":{"loss":{"values":[0.6787,0.6743,0.664],"mean":0.6723,"sd":0.0075},"exact":{"values":[0.9891,0.9783,0.998],"mean":0.9885,"sd":0.0099},"who":{"values":[0.9855,0.9763,0.9974],"mean":0.9864,"sd":0.0106},"copy":{"values":[1,0.9842,1],"mean":0.9947,"sd":0.0091},"perfect":{"values":[0.9805,0.9805,0.9863],"mean":0.9824,"sd":0.0034},"fluent":{"values":[0.9941,0.998,0.998],"mean":0.9967,"sd":0.0023},"stopped":{"values":[1,1,1],"mean":1,"sd":0},"len":{"values":[3.3848,3.4141,3.3906],"mean":3.3965,"sd":0.0155}},"2000":{"loss":{"values":[0.6659,0.6706,0.6816],"mean":0.6727,"sd":0.0081},"exact":{"values":[0.997,0.9872,0.9852],"mean":0.9898,"sd":0.0063},"who":{"values":[0.9961,0.9829,0.9816],"mean":0.9868,"sd":0.008},"copy":{"values":[1,1,0.996],"mean":0.9987,"sd":0.0023},"perfect":{"values":[0.9863,0.9727,0.9824],"mean":0.9805,"sd":0.007},"fluent":{"values":[1,0.9961,0.998],"mean":0.998,"sd":0.002},"stopped":{"values":[1,1,1],"mean":1,"sd":0},"len":{"values":[3.3965,3.3184,3.4453],"mean":3.3867,"sd":0.064}}}},"disagreements":{"about":"Paired change of each metric between two checkpoints of the same run, on the same rows of set B; positive means the later checkpoint is better (lower loss, higher share). Pairs: consecutive checkpoints, and every checkpoint against the last. A sign is +1 or -1 when the 95 percent paired row bootstrap interval excludes 0, else 0. opposite: pairs where one metric is resolved better and another resolved worse. lossOnly: loss resolved better while exact match is unresolved. judgeBlind: the perfect judge resolved better while the fluent judge is unresolved.","pairs":[{"seed":1,"from":50,"to":200,"loss":{"mean":0.2734,"lo":0.2617,"hi":0.2851},"exact":{"mean":0.0365,"lo":0.0109,"hi":0.0648},"perfect":{"mean":0.3125,"lo":0.2676,"hi":0.3555},"fluent":{"mean":0.2012,"lo":0.166,"hi":0.2363},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":200,"to":400,"loss":{"mean":0.0352,"lo":0.0271,"hi":0.0425},"exact":{"mean":0.1096,"lo":0.0785,"hi":0.1442},"perfect":{"mean":0.0508,"lo":0.0273,"hi":0.0742},"fluent":{"mean":0.0059,"lo":-0.0059,"hi":0.0176},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":0}},{"seed":1,"from":400,"to":600,"loss":{"mean":0.0444,"lo":0.0376,"hi":0.0513},"exact":{"mean":0.2182,"lo":0.1936,"hi":0.2418},"perfect":{"mean":-0.0059,"lo":-0.0273,"hi":0.0156},"fluent":{"mean":-0.002,"lo":-0.0117,"hi":0.0059},"signs":{"loss":1,"exact":1,"perfect":0,"fluent":0}},{"seed":1,"from":600,"to":800,"loss":{"mean":0.1046,"lo":0.0975,"hi":0.1124},"exact":{"mean":0.4571,"lo":0.429,"hi":0.4849},"perfect":{"mean":0.1699,"lo":0.1367,"hi":0.2051},"fluent":{"mean":0.002,"lo":-0.0078,"hi":0.0117},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":0}},{"seed":1,"from":800,"to":1000,"loss":{"mean":0.0037,"lo":-0.0029,"hi":0.0106},"exact":{"mean":0.0178,"lo":0.0097,"hi":0.027},"perfect":{"mean":0.0137,"lo":-0.0039,"hi":0.0313},"fluent":{"mean":0.0059,"lo":-0.002,"hi":0.0137},"signs":{"loss":0,"exact":1,"perfect":0,"fluent":0}},{"seed":1,"from":1000,"to":1500,"loss":{"mean":0.002,"lo":-0.0054,"hi":0.0094},"exact":{"mean":-0.0059,"lo":-0.0138,"hi":0.002},"perfect":{"mean":0.002,"lo":-0.0137,"hi":0.0195},"fluent":{"mean":-0.002,"lo":-0.0098,"hi":0.0039},"signs":{"loss":0,"exact":0,"perfect":0,"fluent":0}},{"seed":1,"from":1500,"to":2000,"loss":{"mean":0.0128,"lo":0.007,"hi":0.0186},"exact":{"mean":0.0079,"lo":0.002,"hi":0.0147},"perfect":{"mean":0.0059,"lo":-0.0039,"hi":0.0176},"fluent":{"mean":0.0059,"lo":0,"hi":0.0137},"signs":{"loss":1,"exact":1,"perfect":0,"fluent":0}},{"seed":1,"from":50,"to":2000,"loss":{"mean":0.4762,"lo":0.4628,"hi":0.4896},"exact":{"mean":0.8411,"lo":0.8211,"hi":0.8596},"perfect":{"mean":0.5488,"lo":0.5078,"hi":0.5899},"fluent":{"mean":0.2168,"lo":0.1797,"hi":0.252},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":200,"to":2000,"loss":{"mean":0.2028,"lo":0.1924,"hi":0.2128},"exact":{"mean":0.8045,"lo":0.7794,"hi":0.8293},"perfect":{"mean":0.2363,"lo":0.1992,"hi":0.2773},"fluent":{"mean":0.0156,"lo":0.0059,"hi":0.0273},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":400,"to":2000,"loss":{"mean":0.1676,"lo":0.1598,"hi":0.1758},"exact":{"mean":0.695,"lo":0.6743,"hi":0.7161},"perfect":{"mean":0.1855,"lo":0.1504,"hi":0.2227},"fluent":{"mean":0.0098,"lo":0.002,"hi":0.0195},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":600,"to":2000,"loss":{"mean":0.1231,"lo":0.1162,"hi":0.1305},"exact":{"mean":0.4768,"lo":0.4498,"hi":0.5041},"perfect":{"mean":0.1914,"lo":0.1563,"hi":0.2266},"fluent":{"mean":0.0117,"lo":0.0039,"hi":0.0215},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":800,"to":2000,"loss":{"mean":0.0185,"lo":0.0125,"hi":0.0246},"exact":{"mean":0.0197,"lo":0.0109,"hi":0.0295},"perfect":{"mean":0.0215,"lo":0.0059,"hi":0.0391},"fluent":{"mean":0.0098,"lo":0.002,"hi":0.0195},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":1,"from":1000,"to":2000,"loss":{"mean":0.0148,"lo":0.0085,"hi":0.0217},"exact":{"mean":0.002,"lo":-0.0039,"hi":0.0078},"perfect":{"mean":0.0078,"lo":-0.0078,"hi":0.0234},"fluent":{"mean":0.0039,"lo":0,"hi":0.0098},"signs":{"loss":1,"exact":0,"perfect":0,"fluent":0}},{"seed":2,"from":50,"to":200,"loss":{"mean":0.2589,"lo":0.246,"hi":0.2722},"exact":{"mean":0.0227,"lo":-0.0098,"hi":0.0524},"perfect":{"mean":0.2539,"lo":0.209,"hi":0.3008},"fluent":{"mean":0.166,"lo":0.1289,"hi":0.2012},"signs":{"loss":1,"exact":0,"perfect":1,"fluent":1}},{"seed":2,"from":200,"to":400,"loss":{"mean":0.0808,"lo":0.0722,"hi":0.0899},"exact":{"mean":0.2744,"lo":0.2418,"hi":0.3071},"perfect":{"mean":0.082,"lo":0.0508,"hi":0.1114},"fluent":{"mean":0.0313,"lo":0.0156,"hi":0.0469},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":2,"from":400,"to":600,"loss":{"mean":0.0667,"lo":0.0597,"hi":0.0739},"exact":{"mean":0.2833,"lo":0.2552,"hi":0.3099},"perfect":{"mean":0.0684,"lo":0.0391,"hi":0.1035},"fluent":{"mean":-0.0215,"lo":-0.0372,"hi":-0.0059},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":-1}},{"seed":2,"from":600,"to":800,"loss":{"mean":0.0634,"lo":0.0571,"hi":0.07},"exact":{"mean":0.2409,"lo":0.2187,"hi":0.2652},"perfect":{"mean":0.0918,"lo":0.0625,"hi":0.1211},"fluent":{"mean":0.0215,"lo":0.0059,"hi":0.0391},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":2,"from":800,"to":1000,"loss":{"mean":-0.0048,"lo":-0.0096,"hi":0.0000445},"exact":{"mean":0,"lo":-0.0087,"hi":0.0081},"perfect":{"mean":-0.0059,"lo":-0.0215,"hi":0.0098},"fluent":{"mean":-0.002,"lo":-0.0098,"hi":0.0059},"signs":{"loss":0,"exact":0,"perfect":0,"fluent":0}},{"seed":2,"from":1000,"to":1500,"loss":{"mean":0.0056,"lo":-0.0002,"hi":0.011},"exact":{"mean":-0.0059,"lo":-0.0151,"hi":0.0031},"perfect":{"mean":0.0156,"lo":0.0039,"hi":0.0293},"fluent":{"mean":0.0078,"lo":0.002,"hi":0.0156},"signs":{"loss":0,"exact":0,"perfect":1,"fluent":1}},{"seed":2,"from":1500,"to":2000,"loss":{"mean":0.0037,"lo":-0.0017,"hi":0.009},"exact":{"mean":0.0089,"lo":0,"hi":0.0178},"perfect":{"mean":-0.0078,"lo":-0.0254,"hi":0.0078},"fluent":{"mean":-0.002,"lo":-0.0098,"hi":0.0039},"signs":{"loss":0,"exact":0,"perfect":0,"fluent":0}},{"seed":2,"from":50,"to":2000,"loss":{"mean":0.4743,"lo":0.4584,"hi":0.4895},"exact":{"mean":0.8243,"lo":0.801,"hi":0.8458},"perfect":{"mean":0.498,"lo":0.4551,"hi":0.5411},"fluent":{"mean":0.2012,"lo":0.1641,"hi":0.2383},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":2,"from":200,"to":2000,"loss":{"mean":0.2154,"lo":0.2065,"hi":0.2245},"exact":{"mean":0.8016,"lo":0.778,"hi":0.8245},"perfect":{"mean":0.2441,"lo":0.207,"hi":0.2832},"fluent":{"mean":0.0352,"lo":0.0176,"hi":0.0528},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":2,"from":400,"to":2000,"loss":{"mean":0.1345,"lo":0.1274,"hi":0.1419},"exact":{"mean":0.5271,"lo":0.5015,"hi":0.5532},"perfect":{"mean":0.1621,"lo":0.1289,"hi":0.1973},"fluent":{"mean":0.0039,"lo":-0.0039,"hi":0.0137},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":0}},{"seed":2,"from":600,"to":2000,"loss":{"mean":0.0678,"lo":0.0615,"hi":0.0741},"exact":{"mean":0.2438,"lo":0.221,"hi":0.2669},"perfect":{"mean":0.0938,"lo":0.0625,"hi":0.1211},"fluent":{"mean":0.0254,"lo":0.0098,"hi":0.043},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":2,"from":800,"to":2000,"loss":{"mean":0.0044,"lo":-0.0005,"hi":0.0093},"exact":{"mean":0.003,"lo":-0.006,"hi":0.0119},"perfect":{"mean":0.002,"lo":-0.0176,"hi":0.0215},"fluent":{"mean":0.0039,"lo":-0.0059,"hi":0.0137},"signs":{"loss":0,"exact":0,"perfect":0,"fluent":0}},{"seed":2,"from":1000,"to":2000,"loss":{"mean":0.0092,"lo":0.0047,"hi":0.0138},"exact":{"mean":0.003,"lo":-0.005,"hi":0.0109},"perfect":{"mean":0.0078,"lo":-0.0117,"hi":0.0273},"fluent":{"mean":0.0059,"lo":-0.0039,"hi":0.0176},"signs":{"loss":1,"exact":0,"perfect":0,"fluent":0}},{"seed":3,"from":50,"to":200,"loss":{"mean":0.2729,"lo":0.2619,"hi":0.2836},"exact":{"mean":-0.0069,"lo":-0.0416,"hi":0.0275},"perfect":{"mean":0.3281,"lo":0.2832,"hi":0.3691},"fluent":{"mean":0.1621,"lo":0.1289,"hi":0.1954},"signs":{"loss":1,"exact":0,"perfect":1,"fluent":1}},{"seed":3,"from":200,"to":400,"loss":{"mean":0.081,"lo":0.0726,"hi":0.089},"exact":{"mean":0.4344,"lo":0.3903,"hi":0.478},"perfect":{"mean":0.084,"lo":0.0508,"hi":0.1191},"fluent":{"mean":-0.0098,"lo":-0.0293,"hi":0.0078},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":0}},{"seed":3,"from":400,"to":600,"loss":{"mean":0.0953,"lo":0.087,"hi":0.104},"exact":{"mean":0.386,"lo":0.355,"hi":0.416},"perfect":{"mean":0.1289,"lo":0.0996,"hi":0.1602},"fluent":{"mean":0.0176,"lo":0.0039,"hi":0.0332},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":3,"from":600,"to":800,"loss":{"mean":0.0134,"lo":0.0082,"hi":0.0182},"exact":{"mean":0.0079,"lo":-0.002,"hi":0.0174},"perfect":{"mean":0.0137,"lo":-0.0039,"hi":0.0313},"fluent":{"mean":0.0117,"lo":0.002,"hi":0.0234},"signs":{"loss":1,"exact":0,"perfect":0,"fluent":1}},{"seed":3,"from":800,"to":1000,"loss":{"mean":-0.003,"lo":-0.008,"hi":0.0023},"exact":{"mean":0.0118,"lo":0.0051,"hi":0.0196},"perfect":{"mean":0,"lo":-0.0176,"hi":0.0195},"fluent":{"mean":-0.002,"lo":-0.0098,"hi":0.0059},"signs":{"loss":0,"exact":1,"perfect":0,"fluent":0}},{"seed":3,"from":1000,"to":1500,"loss":{"mean":0.0122,"lo":0.0076,"hi":0.0168},"exact":{"mean":-0.001,"lo":-0.0049,"hi":0.002},"perfect":{"mean":0.0098,"lo":-0.0039,"hi":0.0254},"fluent":{"mean":0.0059,"lo":-0.002,"hi":0.0137},"signs":{"loss":1,"exact":0,"perfect":0,"fluent":0}},{"seed":3,"from":1500,"to":2000,"loss":{"mean":-0.0176,"lo":-0.0243,"hi":-0.0106},"exact":{"mean":-0.0128,"lo":-0.0211,"hi":-0.0059},"perfect":{"mean":-0.0039,"lo":-0.0176,"hi":0.0078},"fluent":{"mean":0,"lo":-0.0059,"hi":0.0059},"signs":{"loss":-1,"exact":-1,"perfect":0,"fluent":0}},{"seed":3,"from":50,"to":2000,"loss":{"mean":0.4543,"lo":0.4409,"hi":0.468},"exact":{"mean":0.8193,"lo":0.7941,"hi":0.8439},"perfect":{"mean":0.5605,"lo":0.5176,"hi":0.5996},"fluent":{"mean":0.1855,"lo":0.1523,"hi":0.2188},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":3,"from":200,"to":2000,"loss":{"mean":0.1814,"lo":0.1713,"hi":0.1921},"exact":{"mean":0.8263,"lo":0.8032,"hi":0.8494},"perfect":{"mean":0.2324,"lo":0.1953,"hi":0.2695},"fluent":{"mean":0.0234,"lo":0.0117,"hi":0.0371},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":3,"from":400,"to":2000,"loss":{"mean":0.1003,"lo":0.091,"hi":0.1096},"exact":{"mean":0.3919,"lo":0.3627,"hi":0.4238},"perfect":{"mean":0.1484,"lo":0.1172,"hi":0.1797},"fluent":{"mean":0.0332,"lo":0.0195,"hi":0.0488},"signs":{"loss":1,"exact":1,"perfect":1,"fluent":1}},{"seed":3,"from":600,"to":2000,"loss":{"mean":0.005,"lo":-0.0027,"hi":0.0121},"exact":{"mean":0.0059,"lo":-0.005,"hi":0.0171},"perfect":{"mean":0.0195,"lo":0.002,"hi":0.0371},"fluent":{"mean":0.0156,"lo":0.0059,"hi":0.0273},"signs":{"loss":0,"exact":0,"perfect":1,"fluent":1}},{"seed":3,"from":800,"to":2000,"loss":{"mean":-0.0084,"lo":-0.0156,"hi":-0.0015},"exact":{"mean":-0.002,"lo":-0.0095,"hi":0.005},"perfect":{"mean":0.0059,"lo":-0.0078,"hi":0.0195},"fluent":{"mean":0.0039,"lo":-0.0039,"hi":0.0117},"signs":{"loss":-1,"exact":0,"perfect":0,"fluent":0}},{"seed":3,"from":1000,"to":2000,"loss":{"mean":-0.0054,"lo":-0.0129,"hi":0.0015},"exact":{"mean":-0.0138,"lo":-0.0229,"hi":-0.0066},"perfect":{"mean":0.0059,"lo":-0.0098,"hi":0.0215},"fluent":{"mean":0.0059,"lo":0,"hi":0.0137},"signs":{"loss":0,"exact":-1,"perfect":0,"fluent":0}}],"counts":{"pairs":39,"opposite":1,"lossOnly":6,"judgeBlind":4},"opposite":["seed 2, step 400 to 600: loss resolved better (0.067 [0.06, 0.074]); exact resolved better (0.283 [0.255, 0.31]); perfect resolved better (0.068 [0.039, 0.104]); fluent resolved worse (-0.021 [-0.037, -0.006])."],"lossOnly":["seed 1, step 1000 to 2000: loss resolved better (0.015 [0.008, 0.022]); exact unresolved (0.002 [-0.004, 0.008]); perfect unresolved (0.008 [-0.008, 0.023]); fluent unresolved (0.004 [0, 0.01]).","seed 2, step 50 to 200: loss resolved better (0.259 [0.246, 0.272]); exact unresolved (0.023 [-0.01, 0.052]); perfect resolved better (0.254 [0.209, 0.301]); fluent resolved better (0.166 [0.129, 0.201]).","seed 2, step 1000 to 2000: loss resolved better (0.009 [0.005, 0.014]); exact unresolved (0.003 [-0.005, 0.011]); perfect unresolved (0.008 [-0.012, 0.027]); fluent unresolved (0.006 [-0.004, 0.018]).","seed 3, step 50 to 200: loss resolved better (0.273 [0.262, 0.284]); exact unresolved (-0.007 [-0.042, 0.027]); perfect resolved better (0.328 [0.283, 0.369]); fluent resolved better (0.162 [0.129, 0.195]).","seed 3, step 600 to 800: loss resolved better (0.013 [0.008, 0.018]); exact unresolved (0.008 [-0.002, 0.017]); perfect unresolved (0.014 [-0.004, 0.031]); fluent resolved better (0.012 [0.002, 0.023]).","seed 3, step 1000 to 1500: loss resolved better (0.012 [0.008, 0.017]); exact unresolved (-0.001 [-0.005, 0.002]); perfect unresolved (0.01 [-0.004, 0.025]); fluent unresolved (0.006 [-0.002, 0.014])."],"judgeBlind":["seed 1, step 200 to 400: loss resolved better (0.035 [0.027, 0.042]); exact resolved better (0.11 [0.078, 0.144]); perfect resolved better (0.051 [0.027, 0.074]); fluent unresolved (0.006 [-0.006, 0.018]).","seed 1, step 600 to 800: loss resolved better (0.105 [0.097, 0.112]); exact resolved better (0.457 [0.429, 0.485]); perfect resolved better (0.17 [0.137, 0.205]); fluent unresolved (0.002 [-0.008, 0.012]).","seed 2, step 400 to 2000: loss resolved better (0.135 [0.127, 0.142]); exact resolved better (0.527 [0.502, 0.553]); perfect resolved better (0.162 [0.129, 0.197]); fluent unresolved (0.004 [-0.004, 0.014]).","seed 3, step 200 to 400: loss resolved better (0.081 [0.073, 0.089]); exact resolved better (0.434 [0.39, 0.478]); perfect resolved better (0.084 [0.051, 0.119]); fluent unresolved (-0.01 [-0.029, 0.008])."]},"misses":{"about":"What each metric misses, read off the blocks above (all set B unless named).","loss":"Loss counts every word, so it keeps moving where exact match has stopped: 6 checkpoint pairs have loss resolved better and exact match unresolved. Loss is also an average over every word: from step 50 to step 200 it falls from 1.141 to 0.873 nats (final 0.673) while exact match on the one-answer words moves only from 0.162 to 0.179 (final 0.99); a falling loss does not say which words got better.","exact":"Exact match sees only the one-answer words (1013 of 12288 words of B) and saturates: at step 2000 it is 0.99, so it cannot show a leak; with 32 rows leaked 250 times each the loss gap on them is 0.305 nats (seeds 0.35, 0.29, 0.276) and the exact-match gap 0.01.","fluent":"The fluent judge reads word pairs, so it passes text whose long-range facts are wrong: at step 200 it accepts 0.973 of continuations against the grammar's 0.743, and of the continuations the grammar rejects it accepts 0.938, 0.856, 0.898 (seeds 1, 2, 3).","perfect":"The perfect judge is perfect only for this toy language: it says whether the generator could have written the text, not whether the text is useful, and it scores one sampled continuation per row, so its number moves with the sampling seed as well as the model. Outside a toy no such parser exists, which is why real evaluations reach for an LLM judge and then have to measure that judge."},"judges":{"about":"The fluent judge (every adjacent word pair seen in 4,096 rows of generator text) against the perfect judge (the generator's grammar) on the same continuations of set B. agree: share of rows where they give the same verdict. kappa: Cohen's kappa, agreement beyond what their two acceptance rates give by chance (1 is perfect, 0 is chance). falseAccept: of the continuations the grammar rejects, the share the fluent judge accepts. falseReject: of those the grammar accepts, the share the fluent judge rejects. Intervals resample rows (1,000 resamples).","perSeed":{"1":{"50":{"n":512,"agree":0.6543,"kappa":0.3543,"falseAccept":0.6146,"falseReject":0,"counts":{"both":224,"onlyPerfect":0,"onlyFluent":177,"neither":111},"kappaIv":{"mean":0.3543,"lo":0.2974,"hi":0.4181},"falseAcceptIv":{"mean":0.6146,"lo":0.5563,"hi":0.6691}},"200":{"n":512,"agree":0.7656,"kappa":0.0909,"falseAccept":0.9375,"falseReject":0,"counts":{"both":384,"onlyPerfect":0,"onlyFluent":120,"neither":8},"kappaIv":{"mean":0.0909,"lo":0.0355,"hi":0.1493},"falseAcceptIv":{"mean":0.9375,"lo":0.896,"hi":0.9762}},"400":{"n":512,"agree":0.8105,"kappa":0.0763,"falseAccept":0.951,"falseReject":0,"counts":{"both":410,"onlyPerfect":0,"onlyFluent":97,"neither":5},"kappaIv":{"mean":0.0763,"lo":0.0172,"hi":0.1386},"falseAcceptIv":{"mean":0.951,"lo":0.9091,"hi":0.9894}},"600":{"n":512,"agree":0.8066,"kappa":0.0879,"falseAccept":0.9429,"falseReject":0,"counts":{"both":407,"onlyPerfect":0,"onlyFluent":99,"neither":6},"kappaIv":{"mean":0.0879,"lo":0.0296,"hi":0.1563},"falseAcceptIv":{"mean":0.9429,"lo":0.8962,"hi":0.9811}},"800":{"n":512,"agree":0.9746,"kappa":0.426,"falseAccept":0.7222,"falseReject":0,"counts":{"both":494,"onlyPerfect":0,"onlyFluent":13,"neither":5},"kappaIv":{"mean":0.426,"lo":0.1508,"hi":0.6605},"falseAcceptIv":{"mean":0.7222,"lo":0.5,"hi":0.9167}},"1000":{"n":512,"agree":0.9824,"kappa":0.3031,"falseAccept":0.8182,"falseReject":0,"counts":{"both":501,"onlyPerfect":0,"onlyFluent":9,"neither":2},"kappaIv":{"mean":0.3031,"lo":0,"hi":0.6199},"falseAcceptIv":{"mean":0.8182,"lo":0.5455,"hi":1}},"1500":{"n":512,"agree":0.9863,"kappa":0.4566,"falseAccept":0.7,"falseReject":0,"counts":{"both":502,"onlyPerfect":0,"onlyFluent":7,"neither":3},"kappaIv":{"mean":0.4566,"lo":0,"hi":0.7664},"falseAcceptIv":{"mean":0.7,"lo":0.375,"hi":1}},"2000":{"n":512,"agree":0.9863,"kappa":0,"falseAccept":1,"falseReject":0,"counts":{"both":505,"onlyPerfect":0,"onlyFluent":7,"neither":0},"kappaIv":{"mean":0,"lo":0,"hi":0},"falseAcceptIv":{"mean":1,"lo":1,"hi":1}}},"2":{"50":{"n":512,"agree":0.6797,"kappa":0.378,"falseAccept":0.6097,"falseReject":0,"counts":{"both":243,"onlyPerfect":0,"onlyFluent":164,"neither":105},"kappaIv":{"mean":0.378,"lo":0.3204,"hi":0.4387},"falseAcceptIv":{"mean":0.6097,"lo":0.5523,"hi":0.6643}},"200":{"n":512,"agree":0.7676,"kappa":0.1967,"falseAccept":0.8561,"falseReject":0,"counts":{"both":373,"onlyPerfect":0,"onlyFluent":119,"neither":20},"kappaIv":{"mean":0.1967,"lo":0.1244,"hi":0.2693},"falseAcceptIv":{"mean":0.8561,"lo":0.7986,"hi":0.9111}},"400":{"n":512,"agree":0.8184,"kappa":0.0652,"falseAccept":0.9588,"falseReject":0,"counts":{"both":415,"onlyPerfect":0,"onlyFluent":93,"neither":4},"kappaIv":{"mean":0.0652,"lo":0.0152,"hi":0.1236},"falseAcceptIv":{"mean":0.9588,"lo":0.9208,"hi":0.9904}},"600":{"n":512,"agree":0.9082,"kappa":0.3594,"falseAccept":0.7581,"falseReject":0,"counts":{"both":450,"onlyPerfect":0,"onlyFluent":47,"neither":15},"kappaIv":{"mean":0.3594,"lo":0.2289,"hi":0.4874},"falseAcceptIv":{"mean":0.7581,"lo":0.6471,"hi":0.8548}},"800":{"n":512,"agree":0.9785,"kappa":0.4138,"falseAccept":0.7333,"falseReject":0,"counts":{"both":497,"onlyPerfect":0,"onlyFluent":11,"neither":4},"kappaIv":{"mean":0.4138,"lo":0.1142,"hi":0.6614},"falseAcceptIv":{"mean":0.7333,"lo":0.5,"hi":0.9376}},"1000":{"n":512,"agree":0.9746,"kappa":0.426,"falseAccept":0.7222,"falseReject":0,"counts":{"both":494,"onlyPerfect":0,"onlyFluent":13,"neither":5},"kappaIv":{"mean":0.426,"lo":0.1302,"hi":0.6632},"falseAcceptIv":{"mean":0.7222,"lo":0.5,"hi":0.9286}},"1500":{"n":512,"agree":0.9824,"kappa":0.1789,"falseAccept":0.9,"falseReject":0,"counts":{"both":502,"onlyPerfect":0,"onlyFluent":9,"neither":1},"kappaIv":{"mean":0.1789,"lo":0,"hi":0.4956},"falseAcceptIv":{"mean":0.9,"lo":0.6667,"hi":1}},"2000":{"n":512,"agree":0.9766,"kappa":0.2448,"falseAccept":0.8571,"falseReject":0,"counts":{"both":498,"onlyPerfect":0,"onlyFluent":12,"neither":2},"kappaIv":{"mean":0.2448,"lo":0,"hi":0.5196},"falseAcceptIv":{"mean":0.8571,"lo":0.6427,"hi":1}}},"3":{"50":{"n":512,"agree":0.6094,"kappa":0.2883,"falseAccept":0.6757,"falseReject":0,"counts":{"both":216,"onlyPerfect":0,"onlyFluent":200,"neither":96},"kappaIv":{"mean":0.2883,"lo":0.2353,"hi":0.3477},"falseAcceptIv":{"mean":0.6757,"lo":0.625,"hi":0.7291}},"200":{"n":512,"agree":0.7754,"kappa":0.145,"falseAccept":0.8984,"falseReject":0,"counts":{"both":384,"onlyPerfect":0,"onlyFluent":115,"neither":13},"kappaIv":{"mean":0.145,"lo":0.0778,"hi":0.2134},"falseAcceptIv":{"mean":0.8984,"lo":0.8451,"hi":0.9474}},"400":{"n":512,"agree":0.8691,"kappa":0.3094,"falseAccept":0.7882,"falseReject":0,"counts":{"both":427,"onlyPerfect":0,"onlyFluent":67,"neither":18},"kappaIv":{"mean":0.3094,"lo":0.1969,"hi":0.4199},"falseAcceptIv":{"mean":0.7882,"lo":0.6966,"hi":0.8721}},"600":{"n":512,"agree":0.9805,"kappa":0.6341,"falseAccept":0.5263,"falseReject":0,"counts":{"both":493,"onlyPerfect":0,"onlyFluent":10,"neither":9},"kappaIv":{"mean":0.6341,"lo":0.3901,"hi":0.8296},"falseAcceptIv":{"mean":0.5263,"lo":0.2855,"hi":0.7502}},"800":{"n":512,"agree":0.9824,"kappa":0.3943,"falseAccept":0.75,"falseReject":0,"counts":{"both":500,"onlyPerfect":0,"onlyFluent":9,"neither":3},"kappaIv":{"mean":0.3943,"lo":0,"hi":0.6655},"falseAcceptIv":{"mean":0.75,"lo":0.4992,"hi":1}},"1000":{"n":512,"agree":0.9844,"kappa":0.4941,"falseAccept":0.6667,"falseReject":0,"counts":{"both":500,"onlyPerfect":0,"onlyFluent":8,"neither":4},"kappaIv":{"mean":0.4941,"lo":0.1216,"hi":0.774},"falseAcceptIv":{"mean":0.6667,"lo":0.3636,"hi":0.9334}},"1500":{"n":512,"agree":0.9883,"kappa":0.2474,"falseAccept":0.8571,"falseReject":0,"counts":{"both":505,"onlyPerfect":0,"onlyFluent":6,"neither":1},"kappaIv":{"mean":0.2474,"lo":0,"hi":0.6649},"falseAcceptIv":{"mean":0.8571,"lo":0.5,"hi":1}},"2000":{"n":512,"agree":0.9844,"kappa":0.1972,"falseAccept":0.8889,"falseReject":0,"counts":{"both":503,"onlyPerfect":0,"onlyFluent":8,"neither":1},"kappaIv":{"mean":0.1972,"lo":0,"hi":0.5666},"falseAcceptIv":{"mean":0.8889,"lo":0.6,"hi":1}}}}},"pairwise":{"about":"Pairwise judging of an earlier checkpoint (x) against the last (y) on the same prompts of set B, each pair judged twice, with x shown first (XY) and with y shown first (YX). The fluent pairwise judge must name a winner: the answer passing the fluent check wins, else the longer, else the one shown first (position and length rules written in on purpose, as analogues of the biases Zheng and others 2023 measured in LLM judges). The perfect pairwise judge prefers the grammatical answer and ties otherwise. same: share of pairs whose two texts are identical. consistent: same winner in both orders; firstBoth: the first-shown answer won in both orders; the Differ fields count only pairs with different texts. decidedBy: which rule decided the XY verdict. agreeOnDecided: on pairs the perfect judge decides, the share where the fluent judge names the same winner in both orders. longerPicked: of pairs where the perfect judge prefers the shorter answer, the share where the fluent judge picks the longer one (XY order). perfectLaterWinsDecided: on pairs the perfect judge decides, the share won by the later checkpoint.","pairs":{"1.200":{"seed":1,"earlier":200,"later":2000,"n":512,"perfectX":0.0078,"perfectY":0.2441,"perfectTie":0.748,"fluentXwinsXY":0.9492,"fluentXwinsYX":0.0664,"consistent":0.1172,"firstBoth":0.8828,"decidedBy":{"position":0.8828,"length":0.1016,"fluency":0.0156},"agreeOnDecided":0.0698,"decided":129,"same":0.5723,"differ":219,"consistentDiffer":0.274,"firstBothDiffer":0.726,"positionDecidedDiffer":0.726,"shorterRight":20,"longerPicked":0.8,"perfectLaterWinsDecided":{"mean":0.969,"lo":0.9333,"hi":0.9928},"consistentDifferIv":{"mean":0.274,"lo":0.2192,"hi":0.3361}},"1.600":{"seed":1,"earlier":600,"later":2000,"n":512,"perfectX":0.0039,"perfectY":0.1953,"perfectTie":0.8008,"fluentXwinsXY":0.9609,"fluentXwinsYX":0.0313,"consistent":0.0703,"firstBoth":0.9297,"decidedBy":{"position":0.9297,"length":0.0586,"fluency":0.0117},"agreeOnDecided":0.0686,"decided":102,"same":0.6348,"differ":187,"consistentDiffer":0.1925,"firstBothDiffer":0.8075,"positionDecidedDiffer":0.8075,"shorterRight":6,"longerPicked":0.5,"perfectLaterWinsDecided":{"mean":0.9804,"lo":0.9478,"hi":1},"consistentDifferIv":{"mean":0.1925,"lo":0.1392,"hi":0.246}},"1.1000":{"seed":1,"earlier":1000,"later":2000,"n":512,"perfectX":0.0117,"perfectY":0.0195,"perfectTie":0.9688,"fluentXwinsXY":0.9434,"fluentXwinsYX":0.043,"consistent":0.0996,"firstBoth":0.9004,"decidedBy":{"position":0.9004,"length":0.0957,"fluency":0.0039},"agreeOnDecided":0.1875,"decided":16,"same":0.7656,"differ":120,"consistentDiffer":0.425,"firstBothDiffer":0.575,"positionDecidedDiffer":0.575,"shorterRight":6,"longerPicked":0.8333,"perfectLaterWinsDecided":{"mean":0.625,"lo":0.3845,"hi":0.8667},"consistentDifferIv":{"mean":0.425,"lo":0.3495,"hi":0.5093}},"2.200":{"seed":2,"earlier":200,"later":2000,"n":512,"perfectX":0.0098,"perfectY":0.2539,"perfectTie":0.7363,"fluentXwinsXY":0.9297,"fluentXwinsYX":0.0918,"consistent":0.1621,"firstBoth":0.8379,"decidedBy":{"position":0.8379,"length":0.1191,"fluency":0.043},"agreeOnDecided":0.163,"decided":135,"same":0.5508,"differ":230,"consistentDiffer":0.3609,"firstBothDiffer":0.6391,"positionDecidedDiffer":0.6391,"shorterRight":46,"longerPicked":0.7391,"perfectLaterWinsDecided":{"mean":0.963,"lo":0.929,"hi":0.9927},"consistentDifferIv":{"mean":0.3609,"lo":0.3021,"hi":0.4258}},"2.600":{"seed":2,"earlier":600,"later":2000,"n":512,"perfectX":0.0137,"perfectY":0.1074,"perfectTie":0.8789,"fluentXwinsXY":0.9492,"fluentXwinsYX":0.0371,"consistent":0.0879,"firstBoth":0.9121,"decidedBy":{"position":0.9121,"length":0.0547,"fluency":0.0332},"agreeOnDecided":0.2419,"decided":62,"same":0.666,"differ":171,"consistentDiffer":0.2632,"firstBothDiffer":0.7368,"positionDecidedDiffer":0.7368,"shorterRight":11,"longerPicked":0.5455,"perfectLaterWinsDecided":{"mean":0.8871,"lo":0.8,"hi":0.9623},"consistentDifferIv":{"mean":0.2632,"lo":0.1965,"hi":0.3294}},"2.1000":{"seed":2,"earlier":1000,"later":2000,"n":512,"perfectX":0.0215,"perfectY":0.0293,"perfectTie":0.9492,"fluentXwinsXY":0.9434,"fluentXwinsYX":0.043,"consistent":0.0996,"firstBoth":0.9004,"decidedBy":{"position":0.9004,"length":0.0859,"fluency":0.0137},"agreeOnDecided":0.2692,"decided":26,"same":0.7402,"differ":133,"consistentDiffer":0.3835,"firstBothDiffer":0.6165,"positionDecidedDiffer":0.6165,"shorterRight":12,"longerPicked":0.75,"perfectLaterWinsDecided":{"mean":0.5769,"lo":0.3702,"hi":0.7895},"consistentDifferIv":{"mean":0.3835,"lo":0.3051,"hi":0.4656}},"3.200":{"seed":3,"earlier":200,"later":2000,"n":512,"perfectX":0.0059,"perfectY":0.2383,"perfectTie":0.7559,"fluentXwinsXY":0.9395,"fluentXwinsYX":0.0625,"consistent":0.123,"firstBoth":0.877,"decidedBy":{"position":0.877,"fluency":0.0234,"length":0.0996},"agreeOnDecided":0.112,"decided":125,"same":0.5156,"differ":248,"consistentDiffer":0.254,"firstBothDiffer":0.746,"positionDecidedDiffer":0.746,"shorterRight":15,"longerPicked":0.6,"perfectLaterWinsDecided":{"mean":0.976,"lo":0.9469,"hi":1},"consistentDifferIv":{"mean":0.254,"lo":0.203,"hi":0.308}},"3.600":{"seed":3,"earlier":600,"later":2000,"n":512,"perfectX":0.0117,"perfectY":0.0313,"perfectTie":0.957,"fluentXwinsXY":0.9473,"fluentXwinsYX":0.0215,"consistent":0.0742,"firstBoth":0.9258,"decidedBy":{"position":0.9258,"length":0.0586,"fluency":0.0156},"agreeOnDecided":0.3636,"decided":22,"same":0.7988,"differ":103,"consistentDiffer":0.3689,"firstBothDiffer":0.6311,"positionDecidedDiffer":0.6311,"shorterRight":5,"longerPicked":0.6,"perfectLaterWinsDecided":{"mean":0.7273,"lo":0.5172,"hi":0.9091},"consistentDifferIv":{"mean":0.3689,"lo":0.2743,"hi":0.458}},"3.1000":{"seed":3,"earlier":1000,"later":2000,"n":512,"perfectX":0.0117,"perfectY":0.0176,"perfectTie":0.9707,"fluentXwinsXY":0.9531,"fluentXwinsYX":0.043,"consistent":0.0898,"firstBoth":0.9102,"decidedBy":{"position":0.9102,"length":0.084,"fluency":0.0059},"agreeOnDecided":0.2,"decided":15,"same":0.7715,"differ":117,"consistentDiffer":0.3932,"firstBothDiffer":0.6068,"positionDecidedDiffer":0.6068,"shorterRight":6,"longerPicked":0.8333,"perfectLaterWinsDecided":{"mean":0.6,"lo":0.3333,"hi":0.8571},"consistentDifferIv":{"mean":0.3932,"lo":0.3027,"hi":0.4822}}}},"contamination":{"about":"Contamination (Sainz and others 2023): the clean run and a leaked twin share the model seed and the data stream; the leaked twin also has the first `rows` rows of set A written into its batches (`repeats` shuffled passes, spread evenly over the run, replacing the last rows of a batch). Both are scored on those rows of A and on fresh B. leakedShare is the share of all training rows that were leaked rows. onA and onB: the leaked model's improvement over its clean twin (clean minus leaked for loss, leaked minus clean for shares), paired over rows. gap = onA - onB, with a 95 percent interval resampling A's rows and B's rows separately (2,000 resamples). A positive gap means the score on the leaked rows is inflated beyond any change on fresh rows.","arms":{"2000.256x1":{"steps":2000,"rows":256,"repeats":1,"leakedRows":256,"leakedShare":0.008,"perSeed":{"1":{"loss":{"cleanA":0.6495,"leakedA":0.654,"cleanB":0.6659,"leakedB":0.6679,"onA":-0.0045,"onB":-0.0019,"gap":-0.0026,"lo":-0.0078,"hi":0.0025},"exact":{"cleanA":0.9981,"leakedA":0.9904,"cleanB":0.997,"leakedB":0.9911,"onA":-0.0077,"onB":-0.0059,"gap":-0.0018,"lo":-0.0128,"hi":0.0089},"perfect":{"cleanA":0.9805,"leakedA":0.9805,"cleanB":0.9863,"leakedB":0.9785,"onA":0,"onB":-0.0078,"gap":0.0078,"lo":-0.0098,"hi":0.0254}},"2":{"loss":{"cleanA":0.6605,"leakedA":0.6598,"cleanB":0.6706,"leakedB":0.6755,"onA":0.0006,"onB":-0.0048,"gap":0.0055,"lo":-0.0016,"hi":0.0137},"exact":{"cleanA":0.9865,"leakedA":0.9769,"cleanB":0.9872,"leakedB":0.9793,"onA":-0.0096,"onB":-0.0079,"gap":-0.0017,"lo":-0.0193,"hi":0.0161},"perfect":{"cleanA":0.9805,"leakedA":0.9727,"cleanB":0.9727,"leakedB":0.9844,"onA":-0.0078,"onB":0.0117,"gap":-0.0195,"lo":-0.043,"hi":0.0039}},"3":{"loss":{"cleanA":0.6678,"leakedA":0.6653,"cleanB":0.6816,"leakedB":0.6763,"onA":0.0025,"onB":0.0053,"gap":-0.0028,"lo":-0.0149,"hi":0.0087},"exact":{"cleanA":0.9942,"leakedA":0.9807,"cleanB":0.9852,"leakedB":0.9882,"onA":-0.0135,"onB":0.003,"gap":-0.0164,"lo":-0.0332,"hi":-0.0008},"perfect":{"cleanA":0.9648,"leakedA":0.9648,"cleanB":0.9824,"leakedB":0.9785,"onA":0,"onB":-0.0039,"gap":0.0039,"lo":-0.0254,"hi":0.0313}}},"gap":{"loss":{"values":[-0.0026,0.0055,-0.0028],"mean":0.0000258,"sd":0.0047,"resolvedSeeds":0},"exact":{"values":[-0.0018,-0.0017,-0.0164],"mean":-0.0067,"sd":0.0085,"resolvedSeeds":1},"perfect":{"values":[0.0078,-0.0195,0.0039],"mean":-0.0026,"sd":0.0148,"resolvedSeeds":0}}},"2000.256x4":{"steps":2000,"rows":256,"repeats":4,"leakedRows":1024,"leakedShare":0.032,"perSeed":{"1":{"loss":{"cleanA":0.6495,"leakedA":0.652,"cleanB":0.6659,"leakedB":0.6715,"onA":-0.0026,"onB":-0.0056,"gap":0.0031,"lo":-0.0035,"hi":0.0093},"exact":{"cleanA":0.9981,"leakedA":0.9827,"cleanB":0.997,"leakedB":0.9812,"onA":-0.0154,"onB":-0.0158,"gap":0.0004,"lo":-0.0154,"hi":0.0156},"perfect":{"cleanA":0.9805,"leakedA":0.9805,"cleanB":0.9863,"leakedB":0.9805,"onA":0,"onB":-0.0059,"gap":0.0059,"lo":-0.0215,"hi":0.0313}},"2":{"loss":{"cleanA":0.6605,"leakedA":0.6684,"cleanB":0.6706,"leakedB":0.6914,"onA":-0.0079,"onB":-0.0207,"gap":0.0128,"lo":0.0024,"hi":0.024},"exact":{"cleanA":0.9865,"leakedA":0.9711,"cleanB":0.9872,"leakedB":0.9585,"onA":-0.0154,"onB":-0.0286,"gap":0.0132,"lo":-0.0064,"hi":0.0326},"perfect":{"cleanA":0.9805,"leakedA":0.9492,"cleanB":0.9727,"leakedB":0.9355,"onA":-0.0312,"onB":-0.0371,"gap":0.0059,"lo":-0.0293,"hi":0.0391}},"3":{"loss":{"cleanA":0.6678,"leakedA":0.6563,"cleanB":0.6816,"leakedB":0.6708,"onA":0.0115,"onB":0.0108,"gap":0.0008,"lo":-0.0086,"hi":0.01},"exact":{"cleanA":0.9942,"leakedA":0.9981,"cleanB":0.9852,"leakedB":0.9951,"onA":0.0039,"onB":0.0099,"gap":-0.006,"lo":-0.0173,"hi":0.0052},"perfect":{"cleanA":0.9648,"leakedA":0.9844,"cleanB":0.9824,"leakedB":0.9863,"onA":0.0195,"onB":0.0039,"gap":0.0156,"lo":-0.0059,"hi":0.0391}}},"gap":{"loss":{"values":[0.0031,0.0128,0.0008],"mean":0.0056,"sd":0.0064,"resolvedSeeds":1},"exact":{"values":[0.0004,0.0132,-0.006],"mean":0.0025,"sd":0.0098,"resolvedSeeds":0},"perfect":{"values":[0.0059,0.0059,0.0156],"mean":0.0091,"sd":0.0056,"resolvedSeeds":0}}},"2000.256x16":{"steps":2000,"rows":256,"repeats":16,"leakedRows":4096,"leakedShare":0.128,"perSeed":{"1":{"loss":{"cleanA":0.6495,"leakedA":0.6507,"cleanB":0.6659,"leakedB":0.6715,"onA":-0.0013,"onB":-0.0056,"gap":0.0044,"lo":-0.0015,"hi":0.0102},"exact":{"cleanA":0.9981,"leakedA":1,"cleanB":0.997,"leakedB":0.997,"onA":0.0019,"onB":0,"gap":0.0019,"lo":-0.003,"hi":0.0079},"perfect":{"cleanA":0.9805,"leakedA":0.9766,"cleanB":0.9863,"leakedB":0.9883,"onA":-0.0039,"onB":0.002,"gap":-0.0059,"lo":-0.0312,"hi":0.0195}},"2":{"loss":{"cleanA":0.6605,"leakedA":0.6538,"cleanB":0.6706,"leakedB":0.671,"onA":0.0067,"onB":-0.0004,"gap":0.007,"lo":0.0002,"hi":0.0156},"exact":{"cleanA":0.9865,"leakedA":0.9865,"cleanB":0.9872,"leakedB":0.9862,"onA":0,"onB":-0.001,"gap":0.001,"lo":-0.0134,"hi":0.016},"perfect":{"cleanA":0.9805,"leakedA":0.9727,"cleanB":0.9727,"leakedB":0.9766,"onA":-0.0078,"onB":0.0039,"gap":-0.0117,"lo":-0.041,"hi":0.0176}},"3":{"loss":{"cleanA":0.6678,"leakedA":0.65,"cleanB":0.6816,"leakedB":0.6718,"onA":0.0178,"onB":0.0098,"gap":0.008,"lo":-0.0018,"hi":0.0182},"exact":{"cleanA":0.9942,"leakedA":0.9904,"cleanB":0.9852,"leakedB":0.9812,"onA":-0.0039,"onB":-0.0039,"gap":0.0000951,"lo":-0.0171,"hi":0.016},"perfect":{"cleanA":0.9648,"leakedA":0.9766,"cleanB":0.9824,"leakedB":0.9805,"onA":0.0117,"onB":-0.002,"gap":0.0137,"lo":-0.0098,"hi":0.0391}}},"gap":{"loss":{"values":[0.0044,0.007,0.008],"mean":0.0065,"sd":0.0019,"resolvedSeeds":1},"exact":{"values":[0.0019,0.001,0.0000951],"mean":0.001,"sd":0.0009,"resolvedSeeds":0},"perfect":{"values":[-0.0059,-0.0117,0.0137],"mean":-0.0013,"sd":0.0133,"resolvedSeeds":0}}},"2000.32x100":{"steps":2000,"rows":32,"repeats":100,"leakedRows":3200,"leakedShare":0.1,"perSeed":{"1":{"loss":{"cleanA":0.6345,"leakedA":0.5997,"cleanB":0.6659,"leakedB":0.6715,"onA":0.0348,"onB":-0.0056,"gap":0.0404,"lo":0.0298,"hi":0.0512},"exact":{"cleanA":1,"leakedA":1,"cleanB":0.997,"leakedB":0.9911,"onA":0,"onB":-0.0059,"gap":0.0059,"lo":-0.0000234,"hi":0.0127},"perfect":{"cleanA":0.9688,"leakedA":0.9688,"cleanB":0.9863,"leakedB":0.9805,"onA":0,"onB":-0.0059,"gap":0.0059,"lo":-0.0059,"hi":0.0195}},"2":{"loss":{"cleanA":0.647,"leakedA":0.6142,"cleanB":0.6706,"leakedB":0.6791,"onA":0.0328,"onB":-0.0085,"gap":0.0413,"lo":0.0255,"hi":0.0581},"exact":{"cleanA":0.9848,"leakedA":1,"cleanB":0.9872,"leakedB":0.9803,"onA":0.0152,"onB":-0.0069,"gap":0.0221,"lo":-0.0019,"hi":0.0598},"perfect":{"cleanA":0.9688,"leakedA":1,"cleanB":0.9727,"leakedB":0.9727,"onA":0.0313,"onB":0,"gap":0.0313,"lo":-0.0137,"hi":0.1016}},"3":{"loss":{"cleanA":0.6536,"leakedA":0.596,"cleanB":0.6816,"leakedB":0.6866,"onA":0.0576,"onB":-0.005,"gap":0.0626,"lo":0.0466,"hi":0.0784},"exact":{"cleanA":1,"leakedA":1,"cleanB":0.9852,"leakedB":0.9704,"onA":0,"onB":-0.0148,"gap":0.0148,"lo":0.0028,"hi":0.0273},"perfect":{"cleanA":0.9688,"leakedA":0.9375,"cleanB":0.9824,"leakedB":0.9863,"onA":-0.0312,"onB":0.0039,"gap":-0.0352,"lo":-0.1016,"hi":0.0059}}},"gap":{"loss":{"values":[0.0404,0.0413,0.0626],"mean":0.0481,"sd":0.0126,"resolvedSeeds":3},"exact":{"values":[0.0059,0.0221,0.0148],"mean":0.0143,"sd":0.0081,"resolvedSeeds":1},"perfect":{"values":[0.0059,0.0313,-0.0352],"mean":0.0007,"sd":0.0335,"resolvedSeeds":0}}},"2000.32x250":{"steps":2000,"rows":32,"repeats":250,"leakedRows":8000,"leakedShare":0.25,"perSeed":{"1":{"loss":{"cleanA":0.6345,"leakedA":0.3345,"cleanB":0.6659,"leakedB":0.7156,"onA":0.3,"onB":-0.0497,"gap":0.3498,"lo":0.3217,"hi":0.3807},"exact":{"cleanA":1,"leakedA":1,"cleanB":0.997,"leakedB":0.9763,"onA":0,"onB":-0.0207,"gap":0.0207,"lo":0.0116,"hi":0.0312},"perfect":{"cleanA":0.9688,"leakedA":1,"cleanB":0.9863,"leakedB":0.9863,"onA":0.0313,"onB":0,"gap":0.0313,"lo":-0.0098,"hi":0.0996}},"2":{"loss":{"cleanA":0.647,"leakedA":0.378,"cleanB":0.6706,"leakedB":0.6914,"onA":0.269,"onB":-0.0207,"gap":0.2897,"lo":0.262,"hi":0.3178},"exact":{"cleanA":0.9848,"leakedA":1,"cleanB":0.9872,"leakedB":0.9862,"onA":0.0152,"onB":-0.001,"gap":0.0161,"lo":-0.0071,"hi":0.0543},"perfect":{"cleanA":0.9688,"leakedA":0.9688,"cleanB":0.9727,"leakedB":0.9824,"onA":0,"onB":0.0098,"gap":-0.0098,"lo":-0.1074,"hi":0.0781}},"3":{"loss":{"cleanA":0.6536,"leakedA":0.3921,"cleanB":0.6816,"leakedB":0.6962,"onA":0.2615,"onB":-0.0146,"gap":0.2761,"lo":0.2486,"hi":0.303},"exact":{"cleanA":1,"leakedA":1,"cleanB":0.9852,"leakedB":0.9921,"onA":0,"onB":0.0069,"gap":-0.0069,"lo":-0.0157,"hi":0.001},"perfect":{"cleanA":0.9688,"leakedA":1,"cleanB":0.9824,"leakedB":0.9883,"onA":0.0313,"onB":0.0059,"gap":0.0254,"lo":-0.0156,"hi":0.0938}}},"gap":{"loss":{"values":[0.3498,0.2897,0.2761],"mean":0.3052,"sd":0.0392,"resolvedSeeds":3},"exact":{"values":[0.0207,0.0161,-0.0069],"mean":0.01,"sd":0.0148,"resolvedSeeds":1},"perfect":{"values":[0.0313,-0.0098,0.0254],"mean":0.0156,"sd":0.0222,"resolvedSeeds":0}}},"600.256x8":{"steps":600,"rows":256,"repeats":8,"leakedRows":2048,"leakedShare":0.2133,"perSeed":{"1":{"loss":{"cleanA":0.7684,"leakedA":0.6803,"cleanB":0.789,"leakedB":0.7068,"onA":0.088,"onB":0.0822,"gap":0.0058,"lo":-0.0051,"hi":0.0163},"exact":{"cleanA":0.5356,"leakedA":0.9114,"cleanB":0.5202,"leakedB":0.8786,"onA":0.3757,"onB":0.3583,"gap":0.0174,"lo":-0.0368,"hi":0.0737},"perfect":{"cleanA":0.7969,"leakedA":0.8945,"cleanB":0.7949,"leakedB":0.9277,"onA":0.0977,"onB":0.1328,"gap":-0.0352,"lo":-0.0918,"hi":0.0254}},"2":{"loss":{"cleanA":0.7312,"leakedA":0.7376,"cleanB":0.7385,"leakedB":0.7602,"onA":-0.0065,"onB":-0.0217,"gap":0.0152,"lo":0.005,"hi":0.0257},"exact":{"cleanA":0.7322,"leakedA":0.6744,"cleanB":0.7433,"leakedB":0.6782,"onA":-0.0578,"onB":-0.0652,"gap":0.0073,"lo":-0.0337,"hi":0.0505},"perfect":{"cleanA":0.8438,"leakedA":0.8203,"cleanB":0.8789,"leakedB":0.8594,"onA":-0.0234,"onB":-0.0195,"gap":-0.0039,"lo":-0.0527,"hi":0.043}},"3":{"loss":{"cleanA":0.6766,"leakedA":0.6892,"cleanB":0.6866,"leakedB":0.7169,"onA":-0.0126,"onB":-0.0302,"gap":0.0176,"lo":0.0074,"hi":0.0279},"exact":{"cleanA":0.9807,"leakedA":0.9229,"cleanB":0.9793,"leakedB":0.9457,"onA":-0.0578,"onB":-0.0336,"gap":-0.0242,"lo":-0.0518,"hi":0.0036},"perfect":{"cleanA":0.9492,"leakedA":0.918,"cleanB":0.9629,"leakedB":0.9297,"onA":-0.0312,"onB":-0.0332,"gap":0.002,"lo":-0.0469,"hi":0.0469}}},"gap":{"loss":{"values":[0.0058,0.0152,0.0176],"mean":0.0129,"sd":0.0062,"resolvedSeeds":2},"exact":{"values":[0.0174,0.0073,-0.0242],"mean":0.0002,"sd":0.0217,"resolvedSeeds":0},"perfect":{"values":[-0.0352,-0.0039,0.002],"mean":-0.0124,"sd":0.0199,"resolvedSeeds":0}}}},"verdict":["2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), loss: on A the leaked model moved -0.005, 0.001, 0.002 against its clean twin, on fresh B -0.002, -0.005, 0.005; gap -0.003 [-0.008, 0.003], 0.005 [-0.002, 0.014], -0.003 [-0.015, 0.009] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), exact: on A the leaked model moved -0.008, -0.01, -0.013 against its clean twin, on fresh B -0.006, -0.008, 0.003; gap -0.002 [-0.013, 0.009], -0.002 [-0.019, 0.016], -0.016 [-0.033, -0.001] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), perfect: on A the leaked model moved 0, -0.008, 0 against its clean twin, on fresh B -0.008, 0.012, -0.004; gap 0.008 [-0.01, 0.025], -0.02 [-0.043, 0.004], 0.004 [-0.025, 0.031] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), loss: on A the leaked model moved -0.003, -0.008, 0.012 against its clean twin, on fresh B -0.006, -0.021, 0.011; gap 0.003 [-0.004, 0.009], 0.013 [0.002, 0.024], 0.001 [-0.009, 0.01] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), exact: on A the leaked model moved -0.015, -0.015, 0.004 against its clean twin, on fresh B -0.016, -0.029, 0.01; gap 0 [-0.015, 0.016], 0.013 [-0.006, 0.033], -0.006 [-0.017, 0.005] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), perfect: on A the leaked model moved 0, -0.031, 0.02 against its clean twin, on fresh B -0.006, -0.037, 0.004; gap 0.006 [-0.021, 0.031], 0.006 [-0.029, 0.039], 0.016 [-0.006, 0.039] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), loss: on A the leaked model moved -0.001, 0.007, 0.018 against its clean twin, on fresh B -0.006, 0, 0.01; gap 0.004 [-0.002, 0.01], 0.007 [0, 0.016], 0.008 [-0.002, 0.018] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), exact: on A the leaked model moved 0.002, 0, -0.004 against its clean twin, on fresh B 0, -0.001, -0.004; gap 0.002 [-0.003, 0.008], 0.001 [-0.013, 0.016], 0.0000951 [-0.017, 0.016] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), perfect: on A the leaked model moved -0.004, -0.008, 0.012 against its clean twin, on fresh B 0.002, 0.004, -0.002; gap -0.006 [-0.031, 0.02], -0.012 [-0.041, 0.018], 0.014 [-0.01, 0.039] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), loss: on A the leaked model moved 0.035, 0.033, 0.058 against its clean twin, on fresh B -0.006, -0.009, -0.005; gap 0.04 [0.03, 0.051], 0.041 [0.026, 0.058], 0.063 [0.047, 0.078] (seeds 1, 2, 3; 3 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), exact: on A the leaked model moved 0, 0.015, 0 against its clean twin, on fresh B -0.006, -0.007, -0.015; gap 0.006 [-0.0000234, 0.013], 0.022 [-0.002, 0.06], 0.015 [0.003, 0.027] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), perfect: on A the leaked model moved 0, 0.031, -0.031 against its clean twin, on fresh B -0.006, 0, 0.004; gap 0.006 [-0.006, 0.02], 0.031 [-0.014, 0.102], -0.035 [-0.102, 0.006] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), loss: on A the leaked model moved 0.3, 0.269, 0.261 against its clean twin, on fresh B -0.05, -0.021, -0.015; gap 0.35 [0.322, 0.381], 0.29 [0.262, 0.318], 0.276 [0.249, 0.303] (seeds 1, 2, 3; 3 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), exact: on A the leaked model moved 0, 0.015, 0 against its clean twin, on fresh B -0.021, -0.001, 0.007; gap 0.021 [0.012, 0.031], 0.016 [-0.007, 0.054], -0.007 [-0.016, 0.001] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), perfect: on A the leaked model moved 0.031, 0, 0.031 against its clean twin, on fresh B 0, 0.01, 0.006; gap 0.031 [-0.01, 0.1], -0.01 [-0.107, 0.078], 0.025 [-0.016, 0.094] (seeds 1, 2, 3; 0 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), loss: on A the leaked model moved 0.088, -0.006, -0.013 against its clean twin, on fresh B 0.082, -0.022, -0.03; gap 0.006 [-0.005, 0.016], 0.015 [0.005, 0.026], 0.018 [0.007, 0.028] (seeds 1, 2, 3; 2 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), exact: on A the leaked model moved 0.376, -0.058, -0.058 against its clean twin, on fresh B 0.358, -0.065, -0.034; gap 0.017 [-0.037, 0.074], 0.007 [-0.034, 0.05], -0.024 [-0.052, 0.004] (seeds 1, 2, 3; 0 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), perfect: on A the leaked model moved 0.098, -0.023, -0.031 against its clean twin, on fresh B 0.133, -0.02, -0.033; gap -0.035 [-0.092, 0.025], -0.004 [-0.053, 0.043], 0.002 [-0.047, 0.047] (seeds 1, 2, 3; 0 of 3 resolved)."]},"verdicts":["At step 2000 (seeds 1, 2, 3): loss 0.673 (sd 0.008), exact match 0.99 (sd 0.006), perfect judge 0.98, fluent judge 0.998. At step 50: loss 1.141, exact 0.162, perfect 0.445, fluent 0.797.","Disagreements over 39 checkpoint pairs: 1 with one metric resolved better and another resolved worse; 6 with loss resolved better and exact match unresolved; 4 with the perfect judge resolved better and the fluent judge unresolved.","Seed 1: fluent against perfect judge, kappa 0.091 [0.035, 0.149] at step 200 and 0 [0, 0] at step 2000; false accepts 0.938 and 1.","Seed 2: fluent against perfect judge, kappa 0.197 [0.124, 0.269] at step 200 and 0.245 [0, 0.52] at step 2000; false accepts 0.856 and 0.857.","Seed 3: fluent against perfect judge, kappa 0.145 [0.078, 0.213] at step 200 and 0.197 [0, 0.567] at step 2000; false accepts 0.898 and 0.889.","Seed 1, step 200 against 2000: identical texts 0.572; on the 219 different pairs the fluent judge kept its verdict after a swap in 0.274 and the first-shown answer won both orders in 0.726; on pairs the perfect judge decides, the later model wins 0.969 [0.933, 0.993], and the fluent judge agrees in both orders on 0.07.","Seed 1, step 600 against 2000: identical texts 0.635; on the 187 different pairs the fluent judge kept its verdict after a swap in 0.193 and the first-shown answer won both orders in 0.807; on pairs the perfect judge decides, the later model wins 0.98 [0.948, 1], and the fluent judge agrees in both orders on 0.069.","Seed 1, step 1000 against 2000: identical texts 0.766; on the 120 different pairs the fluent judge kept its verdict after a swap in 0.425 and the first-shown answer won both orders in 0.575; on pairs the perfect judge decides, the later model wins 0.625 [0.385, 0.867], and the fluent judge agrees in both orders on 0.188.","Seed 2, step 200 against 2000: identical texts 0.551; on the 230 different pairs the fluent judge kept its verdict after a swap in 0.361 and the first-shown answer won both orders in 0.639; on pairs the perfect judge decides, the later model wins 0.963 [0.929, 0.993], and the fluent judge agrees in both orders on 0.163.","Seed 2, step 600 against 2000: identical texts 0.666; on the 171 different pairs the fluent judge kept its verdict after a swap in 0.263 and the first-shown answer won both orders in 0.737; on pairs the perfect judge decides, the later model wins 0.887 [0.8, 0.962], and the fluent judge agrees in both orders on 0.242.","Seed 2, step 1000 against 2000: identical texts 0.74; on the 133 different pairs the fluent judge kept its verdict after a swap in 0.383 and the first-shown answer won both orders in 0.617; on pairs the perfect judge decides, the later model wins 0.577 [0.37, 0.789], and the fluent judge agrees in both orders on 0.269.","Seed 3, step 200 against 2000: identical texts 0.516; on the 248 different pairs the fluent judge kept its verdict after a swap in 0.254 and the first-shown answer won both orders in 0.746; on pairs the perfect judge decides, the later model wins 0.976 [0.947, 1], and the fluent judge agrees in both orders on 0.112.","Seed 3, step 600 against 2000: identical texts 0.799; on the 103 different pairs the fluent judge kept its verdict after a swap in 0.369 and the first-shown answer won both orders in 0.631; on pairs the perfect judge decides, the later model wins 0.727 [0.517, 0.909], and the fluent judge agrees in both orders on 0.364.","Seed 3, step 1000 against 2000: identical texts 0.771; on the 117 different pairs the fluent judge kept its verdict after a swap in 0.393 and the first-shown answer won both orders in 0.607; on pairs the perfect judge decides, the later model wins 0.6 [0.333, 0.857], and the fluent judge agrees in both orders on 0.2.","2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), loss: on A the leaked model moved -0.005, 0.001, 0.002 against its clean twin, on fresh B -0.002, -0.005, 0.005; gap -0.003 [-0.008, 0.003], 0.005 [-0.002, 0.014], -0.003 [-0.015, 0.009] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), exact: on A the leaked model moved -0.008, -0.01, -0.013 against its clean twin, on fresh B -0.006, -0.008, 0.003; gap -0.002 [-0.013, 0.009], -0.002 [-0.019, 0.016], -0.016 [-0.033, -0.001] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 1 times each (0.8 percent of training rows), perfect: on A the leaked model moved 0, -0.008, 0 against its clean twin, on fresh B -0.008, 0.012, -0.004; gap 0.008 [-0.01, 0.025], -0.02 [-0.043, 0.004], 0.004 [-0.025, 0.031] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), loss: on A the leaked model moved -0.003, -0.008, 0.012 against its clean twin, on fresh B -0.006, -0.021, 0.011; gap 0.003 [-0.004, 0.009], 0.013 [0.002, 0.024], 0.001 [-0.009, 0.01] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), exact: on A the leaked model moved -0.015, -0.015, 0.004 against its clean twin, on fresh B -0.016, -0.029, 0.01; gap 0 [-0.015, 0.016], 0.013 [-0.006, 0.033], -0.006 [-0.017, 0.005] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 4 times each (3.2 percent of training rows), perfect: on A the leaked model moved 0, -0.031, 0.02 against its clean twin, on fresh B -0.006, -0.037, 0.004; gap 0.006 [-0.021, 0.031], 0.006 [-0.029, 0.039], 0.016 [-0.006, 0.039] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), loss: on A the leaked model moved -0.001, 0.007, 0.018 against its clean twin, on fresh B -0.006, 0, 0.01; gap 0.004 [-0.002, 0.01], 0.007 [0, 0.016], 0.008 [-0.002, 0.018] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), exact: on A the leaked model moved 0.002, 0, -0.004 against its clean twin, on fresh B 0, -0.001, -0.004; gap 0.002 [-0.003, 0.008], 0.001 [-0.013, 0.016], 0.0000951 [-0.017, 0.016] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 256 rows of set A leaked 16 times each (12.8 percent of training rows), perfect: on A the leaked model moved -0.004, -0.008, 0.012 against its clean twin, on fresh B 0.002, 0.004, -0.002; gap -0.006 [-0.031, 0.02], -0.012 [-0.041, 0.018], 0.014 [-0.01, 0.039] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), loss: on A the leaked model moved 0.035, 0.033, 0.058 against its clean twin, on fresh B -0.006, -0.009, -0.005; gap 0.04 [0.03, 0.051], 0.041 [0.026, 0.058], 0.063 [0.047, 0.078] (seeds 1, 2, 3; 3 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), exact: on A the leaked model moved 0, 0.015, 0 against its clean twin, on fresh B -0.006, -0.007, -0.015; gap 0.006 [-0.0000234, 0.013], 0.022 [-0.002, 0.06], 0.015 [0.003, 0.027] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 32 rows of set A leaked 100 times each (10 percent of training rows), perfect: on A the leaked model moved 0, 0.031, -0.031 against its clean twin, on fresh B -0.006, 0, 0.004; gap 0.006 [-0.006, 0.02], 0.031 [-0.014, 0.102], -0.035 [-0.102, 0.006] (seeds 1, 2, 3; 0 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), loss: on A the leaked model moved 0.3, 0.269, 0.261 against its clean twin, on fresh B -0.05, -0.021, -0.015; gap 0.35 [0.322, 0.381], 0.29 [0.262, 0.318], 0.276 [0.249, 0.303] (seeds 1, 2, 3; 3 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), exact: on A the leaked model moved 0, 0.015, 0 against its clean twin, on fresh B -0.021, -0.001, 0.007; gap 0.021 [0.012, 0.031], 0.016 [-0.007, 0.054], -0.007 [-0.016, 0.001] (seeds 1, 2, 3; 1 of 3 resolved).","2000 steps, 32 rows of set A leaked 250 times each (25 percent of training rows), perfect: on A the leaked model moved 0.031, 0, 0.031 against its clean twin, on fresh B 0, 0.01, 0.006; gap 0.031 [-0.01, 0.1], -0.01 [-0.107, 0.078], 0.025 [-0.016, 0.094] (seeds 1, 2, 3; 0 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), loss: on A the leaked model moved 0.088, -0.006, -0.013 against its clean twin, on fresh B 0.082, -0.022, -0.03; gap 0.006 [-0.005, 0.016], 0.015 [0.005, 0.026], 0.018 [0.007, 0.028] (seeds 1, 2, 3; 2 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), exact: on A the leaked model moved 0.376, -0.058, -0.058 against its clean twin, on fresh B 0.358, -0.065, -0.034; gap 0.017 [-0.037, 0.074], 0.007 [-0.034, 0.05], -0.024 [-0.052, 0.004] (seeds 1, 2, 3; 0 of 3 resolved).","600 steps, 256 rows of set A leaked 8 times each (21.3 percent of training rows), perfect: on A the leaked model moved 0.098, -0.023, -0.031 against its clean twin, on fresh B 0.133, -0.02, -0.033; gap -0.035 [-0.092, 0.025], -0.004 [-0.053, 0.043], 0.002 [-0.047, 0.047] (seeds 1, 2, 3; 0 of 3 resolved)."],"speed":{"about":"Milliseconds in Node (V8), one process, nothing else running. msPer16Rows: one scorer step of the lab (16 rows of set B: loss and exact match by one forward pass, then a free continuation and both judges per row); a browser lab running 40 ms slices takes one step per slice. msPerTrainStep: one training step at batch 16. inPool: mean training step inside the fill's runs, 4 processes at once.","msBuildSets":104.696,"msPairTable":11.4404,"msPer16Rows":19.4435,"msPer16RowsMax":27.8069,"msPerTrainStep":22.5182,"inPool":30.6476,"rowsPer40ms":32},"film":{"about":"What the films draw. curve: seed 1's four metrics on set B at every checkpoint with 95 percent intervals. examples: the first 6 rows of B, the prompt and the continuation written at step 200 and at step 2000, with both judges' verdicts. fooled: rows where the step-200 continuation passes the fluent judge and fails the grammar. leak: per seed, loss on A and B for each clean and leaked pair.","curve":[{"step":50,"loss":{"mean":1.1421,"lo":1.1248,"hi":1.1585},"exact":{"mean":0.156,"lo":0.1357,"hi":0.1764},"perfect":{"mean":0.4375,"lo":0.3926,"hi":0.4805},"fluent":{"mean":0.7832,"lo":0.748,"hi":0.8164}},{"step":200,"loss":{"mean":0.8687,"lo":0.8538,"hi":0.8837},"exact":{"mean":0.1925,"lo":0.1661,"hi":0.2151},"perfect":{"mean":0.75,"lo":0.7109,"hi":0.7871},"fluent":{"mean":0.9844,"lo":0.9746,"hi":0.9941}},{"step":400,"loss":{"mean":0.8335,"lo":0.8208,"hi":0.846},"exact":{"mean":0.3021,"lo":0.2811,"hi":0.3232},"perfect":{"mean":0.8008,"lo":0.7637,"hi":0.8359},"fluent":{"mean":0.9902,"lo":0.9805,"hi":0.998}},{"step":600,"loss":{"mean":0.789,"lo":0.7777,"hi":0.7995},"exact":{"mean":0.5202,"lo":0.4925,"hi":0.5464},"perfect":{"mean":0.7949,"lo":0.7578,"hi":0.8282},"fluent":{"mean":0.9883,"lo":0.9785,"hi":0.9961}},{"step":800,"loss":{"mean":0.6844,"lo":0.6735,"hi":0.6954},"exact":{"mean":0.9773,"lo":0.9675,"hi":0.986},"perfect":{"mean":0.9648,"lo":0.9473,"hi":0.9805},"fluent":{"mean":0.9902,"lo":0.9824,"hi":0.998}},{"step":1000,"loss":{"mean":0.6807,"lo":0.6691,"hi":0.6925},"exact":{"mean":0.9951,"lo":0.9904,"hi":0.999},"perfect":{"mean":0.9785,"lo":0.9668,"hi":0.9902},"fluent":{"mean":0.9961,"lo":0.9902,"hi":1}},{"step":1500,"loss":{"mean":0.6787,"lo":0.6671,"hi":0.6895},"exact":{"mean":0.9891,"lo":0.9826,"hi":0.9948},"perfect":{"mean":0.9805,"lo":0.9668,"hi":0.9922},"fluent":{"mean":0.9941,"lo":0.9863,"hi":1}},{"step":2000,"loss":{"mean":0.6659,"lo":0.6548,"hi":0.6767},"exact":{"mean":0.997,"lo":0.9931,"hi":1},"perfect":{"mean":0.9863,"lo":0.9766,"hi":0.9961},"fluent":{"mean":1,"lo":1,"hi":1}}],"examples":[{"prompt":["tallow","found","a","rope",".","rerun","knocked","on","a","shell"],"early":{"words":["."],"perfect":1,"fluent":1},"late":{"words":["."],"perfect":1,"fluent":1}},{"prompt":["tallow","kept","the","lamp","bright",".","rerun","knocked","on","a"],"early":{"words":["shell","."],"perfect":1,"fluent":1},"late":{"words":["shell","."],"perfect":1,"fluent":1}},{"prompt":["rerun","mended","the","boat",".","the","bell","rang",".","tallow"],"early":{"words":["lit","the","lamp","."],"perfect":1,"fluent":1},"late":{"words":["lit","the","lamp","."],"perfect":1,"fluent":1}},{"prompt":["pylon","painted","the","rope",".","who","painted","the"],"early":{"words":["lamp","?","pylon","."],"perfect":0,"fluent":1},"late":{"words":["rope","?","pylon","."],"perfect":1,"fluent":1}},{"prompt":["tallow","found","a","card",".","the","lamp","was","bright","."],"early":{"words":["rerun","knocked","twice","."],"perfect":1,"fluent":1},"late":{"words":["rerun","knocked","twice","."],"perfect":1,"fluent":1}},{"prompt":["tallow","kept","the","lamp","bright","."],"early":{"words":["the","tide","came","in","."],"perfect":1,"fluent":1},"late":{"words":["the","tide","came","in","."],"perfect":1,"fluent":1}}],"fooled":[{"prompt":["pylon","painted","the","rope",".","who","painted","the"],"early":{"words":["lamp","?","pylon","."],"perfect":0,"fluent":1},"late":{"words":["rope","?","pylon","."],"perfect":1,"fluent":1}},{"prompt":["rerun","dropped","the","net",".","who","dropped"],"early":{"words":["the","lamp","?","tallow","."],"perfect":0,"fluent":1},"late":{"words":["the","net","?","rerun","."],"perfect":1,"fluent":1}},{"prompt":["pylon","found","a","kettle","."],"early":{"words":["pylon","put","a","cone","on","the","crate","."],"perfect":0,"fluent":1},"late":{"words":["pylon","put","a","cone","on","the","kettle","."],"perfect":1,"fluent":1}},{"prompt":["tallow","found","a","card",".","the","bell","was","cold","."],"early":{"words":["pylon","put","a","cone","on","the","rope","."],"perfect":0,"fluent":1},"late":{"words":["pylon","put","a","cone","on","the","card","."],"perfect":1,"fluent":1}}],"leak":[{"steps":2000,"rows":256,"repeats":1,"loss":[{"seed":1,"cleanA":0.6495,"leakedA":0.654,"cleanB":0.6659,"leakedB":0.6679,"gap":-0.0026,"lo":-0.0078,"hi":0.0025},{"seed":2,"cleanA":0.6605,"leakedA":0.6598,"cleanB":0.6706,"leakedB":0.6755,"gap":0.0055,"lo":-0.0016,"hi":0.0137},{"seed":3,"cleanA":0.6678,"leakedA":0.6653,"cleanB":0.6816,"leakedB":0.6763,"gap":-0.0028,"lo":-0.0149,"hi":0.0087}]},{"steps":2000,"rows":256,"repeats":4,"loss":[{"seed":1,"cleanA":0.6495,"leakedA":0.652,"cleanB":0.6659,"leakedB":0.6715,"gap":0.0031,"lo":-0.0035,"hi":0.0093},{"seed":2,"cleanA":0.6605,"leakedA":0.6684,"cleanB":0.6706,"leakedB":0.6914,"gap":0.0128,"lo":0.0024,"hi":0.024},{"seed":3,"cleanA":0.6678,"leakedA":0.6563,"cleanB":0.6816,"leakedB":0.6708,"gap":0.0008,"lo":-0.0086,"hi":0.01}]},{"steps":2000,"rows":256,"repeats":16,"loss":[{"seed":1,"cleanA":0.6495,"leakedA":0.6507,"cleanB":0.6659,"leakedB":0.6715,"gap":0.0044,"lo":-0.0015,"hi":0.0102},{"seed":2,"cleanA":0.6605,"leakedA":0.6538,"cleanB":0.6706,"leakedB":0.671,"gap":0.007,"lo":0.0002,"hi":0.0156},{"seed":3,"cleanA":0.6678,"leakedA":0.65,"cleanB":0.6816,"leakedB":0.6718,"gap":0.008,"lo":-0.0018,"hi":0.0182}]},{"steps":2000,"rows":32,"repeats":100,"loss":[{"seed":1,"cleanA":0.6345,"leakedA":0.5997,"cleanB":0.6659,"leakedB":0.6715,"gap":0.0404,"lo":0.0298,"hi":0.0512},{"seed":2,"cleanA":0.647,"leakedA":0.6142,"cleanB":0.6706,"leakedB":0.6791,"gap":0.0413,"lo":0.0255,"hi":0.0581},{"seed":3,"cleanA":0.6536,"leakedA":0.596,"cleanB":0.6816,"leakedB":0.6866,"gap":0.0626,"lo":0.0466,"hi":0.0784}]},{"steps":2000,"rows":32,"repeats":250,"loss":[{"seed":1,"cleanA":0.6345,"leakedA":0.3345,"cleanB":0.6659,"leakedB":0.7156,"gap":0.3498,"lo":0.3217,"hi":0.3807},{"seed":2,"cleanA":0.647,"leakedA":0.378,"cleanB":0.6706,"leakedB":0.6914,"gap":0.2897,"lo":0.262,"hi":0.3178},{"seed":3,"cleanA":0.6536,"leakedA":0.3921,"cleanB":0.6816,"leakedB":0.6962,"gap":0.2761,"lo":0.2486,"hi":0.303}]},{"steps":600,"rows":256,"repeats":8,"loss":[{"seed":1,"cleanA":0.7684,"leakedA":0.6803,"cleanB":0.789,"leakedB":0.7068,"gap":0.0058,"lo":-0.0051,"hi":0.0163},{"seed":2,"cleanA":0.7312,"leakedA":0.7376,"cleanB":0.7385,"leakedB":0.7602,"gap":0.0152,"lo":0.005,"hi":0.0257},{"seed":3,"cleanA":0.6766,"leakedA":0.6892,"cleanB":0.6866,"leakedB":0.7169,"gap":0.0176,"lo":0.0074,"hi":0.0279}]}]},"runSeconds":{"leak.2000.256.1.1":65.2,"ckpt.1":72.1,"ckpt.2":72.2,"ckpt.3":72.8,"leak.2000.256.1.2":62.4,"leak.2000.256.4.1":59.6,"leak.2000.256.4.2":59.7,"leak.2000.256.1.3":60.9,"leak.2000.256.4.3":60.5,"leak.2000.256.16.3":59.6,"leak.2000.256.16.1":60.6,"leak.2000.256.16.2":60.4,"leak.2000.32.100.1":62,"leak.2000.32.100.3":60,"leak.2000.32.100.2":61.7,"leak.2000.32.250.1":62.4,"leak.600.256.8.2":20.7,"leak.600.256.8.1":23.9,"leak.600.256.8.3":17.9,"leak.2000.32.250.2":62.9,"leak.2000.32.250.3":61.6},"seconds":330}