{"about":"Lesson S17, distillation. Computed by tools/school/fill_distill.mjs from system/school/distill.mjs and the attention lab's own model, trainer and corpus generator. Do not edit by hand. Losses are in nats (natural log). Every student arm uses the same rows for the same steps; only the training target differs.","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","recipe":{"steps":2000,"batch":16,"lr":0.01,"seeds":[1,2,3],"heldOutRows":256,"checkpoints":[0,100,250,500,1000,1500,2000],"loss":"alpha * T^2 * KL(softmax(teacher/T) || softmax(student/T)) + (1 - alpha) * CE(label, softmax(student))"},"teacher":{"about":"The teacher is the attention lab's trained model (seed 7, 2000 steps), scored on the standard 256 held-out rows (6144 next-word targets, seed 1592593325). floor is the corpus generator's own loss on those rows (no model can beat it on average); excess is loss minus floor. unigramLoss is a word-count model fitted on 1,024 training rows: a model that ignores context.","nParams":20864,"config":{"vocab":46,"ctx":24,"dModel":32,"nHeads":2,"nLayers":2,"dFF":64,"attnMode":"normal"},"heldOut":0.6874,"floor":0.6585,"excess":0.0289,"copyAccuracy":0.9921,"accuracy":0.7243,"unigramLoss":3.2725},"students":{"about":"Students are the same architecture, smaller (feed-forward twice the width, two heads). The main student is d16x1: the smallest tried, about a fifth of the teacher, and still able to learn the look-back skill, which leaves the most room between student and teacher. d16x2 (two layers) was run for hard labels and T 2 only, full data and the 32-row pool, three seeds, as a check that the picture holds at another size. Same 2000 steps, batch 16, learning rate 0.01 with the lab's warmup, for every arm.","main":"d16x1","sizes":{"d16x1":{"config":{"dModel":16,"nLayers":1,"nHeads":2,"dFF":32},"nParams":4112,"nonEmbedding":2256,"shareOfTeacher":0.1971},"d16x2":{"config":{"dModel":16,"nLayers":2,"nHeads":2,"dFF":32},"nParams":6336,"nonEmbedding":4480,"shareOfTeacher":0.3037}},"d16x2":{"full":{"hard":{"arm":"hard","kind":"hard","seeds":[1,2,3],"heldOut":{"values":[0.6864,0.6859,0.6879],"mean":0.6868,"sd":0.001,"min":0.6859,"max":0.6879},"excess":{"values":[0.0279,0.0274,0.0294],"mean":0.0282,"sd":0.001,"min":0.0274,"max":0.0294},"copyAccuracy":{"values":[0.9941,0.9881,0.9802],"mean":0.9875,"sd":0.0069,"min":0.9802,"max":0.9941},"agreement":{"values":[0.8441,0.8271,0.8511],"mean":0.8408,"sd":0.0123,"min":0.8271,"max":0.8511},"bestHeldOut":{"values":[0.6864,0.6859,0.6879],"mean":0.6868,"sd":0.001,"min":0.6859,"max":0.6879},"bestStep":[2000,2000,2000],"curveSeed1":[{"step":0,"heldOut":3.8315,"copyAccuracy":0,"agreement":0.0591},{"step":100,"heldOut":1.055,"copyAccuracy":0.1897,"agreement":0.7074},{"step":250,"heldOut":0.8956,"copyAccuracy":0.1957,"agreement":0.7228},{"step":500,"heldOut":0.8401,"copyAccuracy":0.3399,"agreement":0.741},{"step":1000,"heldOut":0.6998,"copyAccuracy":0.9684,"agreement":0.8267},{"step":1500,"heldOut":0.7119,"copyAccuracy":0.9862,"agreement":0.8322},{"step":2000,"heldOut":0.6864,"copyAccuracy":0.9941,"agreement":0.8441}]},"soft2":{"arm":"soft2","kind":"soft","T":2,"seeds":[1,2,3],"heldOut":{"values":[0.6972,0.6925,0.689],"mean":0.6929,"sd":0.0041,"min":0.689,"max":0.6972},"excess":{"values":[0.0387,0.034,0.0304],"mean":0.0344,"sd":0.0041,"min":0.0304,"max":0.0387},"copyAccuracy":{"values":[0.996,0.9921,0.9941],"mean":0.9941,"sd":0.002,"min":0.9921,"max":0.996},"agreement":{"values":[0.939,0.8939,0.9565],"mean":0.9298,"sd":0.0323,"min":0.8939,"max":0.9565},"bestHeldOut":{"values":[0.6896,0.6925,0.6884],"mean":0.6902,"sd":0.0021,"min":0.6884,"max":0.6925},"bestStep":[1500,2000,1500],"vsHard":{"heldOut":{"diffs":[0.0108,0.0066,0.001],"mean":0.0061,"sd":0.0049,"resolved":true},"copyAccuracy":{"diffs":[0.002,0.004,0.0138],"mean":0.0066,"sd":0.0064,"resolved":false},"agreement":{"diffs":[0.0949,0.0667,0.1055],"mean":0.089,"sd":0.02,"resolved":true},"bestHeldOut":{"diffs":[0.0032,0.0066,0.0005],"mean":0.0034,"sd":0.0031,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.8315,"copyAccuracy":0,"agreement":0.0591},{"step":100,"heldOut":1.121,"copyAccuracy":0.1937,"agreement":0.6812},{"step":250,"heldOut":0.9302,"copyAccuracy":0.1818,"agreement":0.7518},{"step":500,"heldOut":0.8397,"copyAccuracy":0.3715,"agreement":0.8317},{"step":1000,"heldOut":0.7011,"copyAccuracy":0.9921,"agreement":0.9212},{"step":1500,"heldOut":0.6896,"copyAccuracy":0.9921,"agreement":0.9631},{"step":2000,"heldOut":0.6972,"copyAccuracy":0.996,"agreement":0.939}]}},"pool32":{"hard":{"arm":"hard","kind":"hard","seeds":[1,2,3],"heldOut":{"values":[4.4602,4.436,4.3393],"mean":4.4119,"sd":0.064,"min":4.3393,"max":4.4602},"excess":{"values":[3.8017,3.7775,3.6808],"mean":3.7533,"sd":0.064,"min":3.6808,"max":3.8017},"copyAccuracy":{"values":[0.3221,0.2668,0.2352],"mean":0.2747,"sd":0.044,"min":0.2352,"max":0.3221},"agreement":{"values":[0.5884,0.5959,0.5812],"mean":0.5885,"sd":0.0073,"min":0.5812,"max":0.5959},"bestHeldOut":{"values":[1.5928,1.6468,1.6355],"mean":1.625,"sd":0.0284,"min":1.5928,"max":1.6468},"bestStep":[100,100,100],"curveSeed1":[{"step":0,"heldOut":3.8315,"copyAccuracy":0,"agreement":0.0591},{"step":100,"heldOut":1.5928,"copyAccuracy":0.2253,"agreement":0.6325},{"step":250,"heldOut":3.0566,"copyAccuracy":0.3004,"agreement":0.5838},{"step":500,"heldOut":3.6642,"copyAccuracy":0.2984,"agreement":0.5924},{"step":1000,"heldOut":4.1171,"copyAccuracy":0.3142,"agreement":0.5843},{"step":1500,"heldOut":4.3194,"copyAccuracy":0.2964,"agreement":0.5854},{"step":2000,"heldOut":4.4602,"copyAccuracy":0.3221,"agreement":0.5884}]},"soft2":{"arm":"soft2","kind":"soft","T":2,"seeds":[1,2,3],"heldOut":{"values":[1.7398,1.6346,1.7311],"mean":1.7018,"sd":0.0584,"min":1.6346,"max":1.7398},"excess":{"values":[1.0813,0.976,1.0726],"mean":1.0433,"sd":0.0584,"min":0.976,"max":1.0813},"copyAccuracy":{"values":[0.3478,0.334,0.3024],"mean":0.3281,"sd":0.0233,"min":0.3024,"max":0.3478},"agreement":{"values":[0.6823,0.772,0.7767],"mean":0.7437,"sd":0.0532,"min":0.6823,"max":0.7767},"bestHeldOut":{"values":[1.3765,1.4693,1.4285],"mean":1.4248,"sd":0.0465,"min":1.3765,"max":1.4693},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-2.7204,-2.8014,-2.6082],"mean":-2.71,"sd":0.097,"resolved":true},"copyAccuracy":{"diffs":[0.0257,0.0672,0.0672],"mean":0.0534,"sd":0.024,"resolved":true},"agreement":{"diffs":[0.0939,0.1761,0.1955],"mean":0.1552,"sd":0.0539,"resolved":true},"bestHeldOut":{"diffs":[-0.2163,-0.1774,-0.207],"mean":-0.2002,"sd":0.0203,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.8315,"copyAccuracy":0,"agreement":0.0591},{"step":100,"heldOut":1.3765,"copyAccuracy":0.1818,"agreement":0.6553},{"step":250,"heldOut":1.6113,"copyAccuracy":0.3202,"agreement":0.7225},{"step":500,"heldOut":1.7692,"copyAccuracy":0.3202,"agreement":0.7292},{"step":1000,"heldOut":1.7951,"copyAccuracy":0.3498,"agreement":0.7529},{"step":1500,"heldOut":1.7686,"copyAccuracy":0.3221,"agreement":0.7246},{"step":2000,"heldOut":1.7398,"copyAccuracy":0.3478,"agreement":0.6823}]}}}},"darkKnowledge":{"about":"What a soft target carries that a hard label drops. The generator often leaves the next word open: on the held-out rows, torn is the share of positions where its most likely word has at most 0.75 of the probability, entropy its mean next-word entropy in nats. A hard label names one word; the soft target keeps the split. Probes: the generator's exact odds after a few contexts and the teacher's shares at T 1 (words above 0.02 in either).","positions":6144,"torn":0.3989,"entropy":0.6529,"probes":[{"words":["rerun","knocked"],"odds":[{"word":"twice","generator":0.5,"teacher":0.5431},{"word":"on","generator":0.5,"teacher":0.4547}]},{"words":["tallow","found","a"],"odds":[{"word":"kettle","generator":0.1,"teacher":0.1428},{"word":"map","generator":0.1,"teacher":0.126},{"word":"crate","generator":0.1,"teacher":0.1189},{"word":"card","generator":0.1,"teacher":0.0985},{"word":"lamp","generator":0.1,"teacher":0.0931},{"word":"net","generator":0.1,"teacher":0.093},{"word":"bell","generator":0.1,"teacher":0.086},{"word":"shell","generator":0.1,"teacher":0.0847},{"word":"boat","generator":0.1,"teacher":0.0837},{"word":"rope","generator":0.1,"teacher":0.0689}]},{"words":["tallow"],"odds":[{"word":"found","generator":0.4211,"teacher":0.5613},{"word":"lit","generator":0.0789,"teacher":0.0504},{"word":"kept","generator":0.0789,"teacher":0.045},{"word":"dropped","generator":0.0702,"teacher":0.0829},{"word":"painted","generator":0.0702,"teacher":0.0648},{"word":"carried","generator":0.0702,"teacher":0.0536},{"word":"checked","generator":0.0702,"teacher":0.0536},{"word":"mended","generator":0.0702,"teacher":0.0465},{"word":"counted","generator":0.0702,"teacher":0.0341}]},{"words":["the"],"odds":[{"word":"tide","generator":0.5,"teacher":0.548},{"word":"bell","generator":0.5,"teacher":0.4297}]},{"words":["rerun","found","a","kettle","."],"odds":[{"word":"pylon","generator":0.4417,"teacher":0.5092},{"word":"the","generator":0.3417,"teacher":0.2716},{"word":"tallow","generator":0.1083,"teacher":0.1139},{"word":"rerun","generator":0.1083,"teacher":0.1036}]}]},"fullData":{"about":"Fresh rows every step (data seed 100 + student seed), 2000 steps, student seeds 1, 2, 3, scored at the end on the 256 held-out rows. heldOut and excess in nats; copyAccuracy is top-1 accuracy on look-back targets; agreement is the share of all held-out positions where the student's top word is the teacher's top word (on a 50/50 split the teacher's pick is arbitrary, so agreement there measures imitation, not skill). vsHard is the paired difference (arm minus hard labels, same seed and rows). mix2 is half soft (T 2) and half hard. oracle uses the generator's exact odds as the soft target. argmax trains on the teacher's single top word as if it were certain: on the many positions where the next word is open (darkKnowledge.torn) the student becomes sure of one word and pays a large loss whenever the text picks another, so its loss is high while its look-back accuracy and agreement with the teacher are high. This shows the effect for this teacher, this student and this data; it does not show what happens with a larger gap between teacher and student.","arms":{"hard":{"arm":"hard","kind":"hard","seeds":[1,2,3],"heldOut":{"values":[0.6949,0.774,0.6949],"mean":0.7213,"sd":0.0457,"min":0.6949,"max":0.774},"excess":{"values":[0.0364,0.1155,0.0363],"mean":0.0627,"sd":0.0457,"min":0.0363,"max":0.1155},"copyAccuracy":{"values":[0.9605,0.5672,0.9783],"mean":0.8353,"sd":0.2324,"min":0.5672,"max":0.9783},"agreement":{"values":[0.8381,0.786,0.8706],"mean":0.8315,"sd":0.0427,"min":0.786,"max":0.8706},"bestHeldOut":{"values":[0.6949,0.774,0.6949],"mean":0.7213,"sd":0.0457,"min":0.6949,"max":0.774},"bestStep":[2000,2000,2000],"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0561,"copyAccuracy":0.1561,"agreement":0.6815},{"step":250,"heldOut":0.8908,"copyAccuracy":0.1937,"agreement":0.7163},{"step":500,"heldOut":0.8627,"copyAccuracy":0.2095,"agreement":0.7153},{"step":1000,"heldOut":0.7288,"copyAccuracy":0.9308,"agreement":0.8211},{"step":1500,"heldOut":0.7077,"copyAccuracy":0.9545,"agreement":0.8416},{"step":2000,"heldOut":0.6949,"copyAccuracy":0.9605,"agreement":0.8381}]},"argmax":{"arm":"argmax","kind":"argmax","seeds":[1,2,3],"heldOut":{"values":[3.9528,3.8409,3.735],"mean":3.8429,"sd":0.1089,"min":3.735,"max":3.9528},"excess":{"values":[3.2942,3.1823,3.0765],"mean":3.1843,"sd":0.1089,"min":3.0765,"max":3.2942},"copyAccuracy":{"values":[0.9387,0.9368,0.9664],"mean":0.9473,"sd":0.0166,"min":0.9368,"max":0.9664},"agreement":{"values":[0.9808,0.9821,0.9854],"mean":0.9827,"sd":0.0023,"min":0.9808,"max":0.9854},"bestHeldOut":{"values":[1.9051,2.0062,1.9834],"mean":1.9649,"sd":0.0531,"min":1.9051,"max":2.0062},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[3.2579,3.0669,3.0401],"mean":3.1216,"sd":0.1187,"resolved":true},"copyAccuracy":{"diffs":[-0.0217,0.3696,-0.0119],"mean":0.112,"sd":0.2231,"resolved":false},"agreement":{"diffs":[0.1427,0.1961,0.1147],"mean":0.1512,"sd":0.0413,"resolved":true},"bestHeldOut":{"diffs":[1.2101,1.2322,1.2885],"mean":1.2436,"sd":0.0404,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.9051,"copyAccuracy":0.1937,"agreement":0.8429},{"step":250,"heldOut":2.4869,"copyAccuracy":0.3202,"agreement":0.9124},{"step":500,"heldOut":3.0649,"copyAccuracy":0.832,"agreement":0.9629},{"step":1000,"heldOut":3.5258,"copyAccuracy":0.9229,"agreement":0.9725},{"step":1500,"heldOut":3.8731,"copyAccuracy":0.9289,"agreement":0.9772},{"step":2000,"heldOut":3.9528,"copyAccuracy":0.9387,"agreement":0.9808}]},"soft1":{"arm":"soft1","kind":"soft","T":1,"seeds":[1,2,3],"heldOut":{"values":[0.7046,0.6999,0.7014],"mean":0.7019,"sd":0.0024,"min":0.6999,"max":0.7046},"excess":{"values":[0.046,0.0414,0.0428],"mean":0.0434,"sd":0.0024,"min":0.0414,"max":0.046},"copyAccuracy":{"values":[0.9565,0.9664,0.9565],"mean":0.9598,"sd":0.0057,"min":0.9565,"max":0.9664},"agreement":{"values":[0.918,0.96,0.9023],"mean":0.9268,"sd":0.0298,"min":0.9023,"max":0.96},"bestHeldOut":{"values":[0.7046,0.6999,0.7014],"mean":0.7019,"sd":0.0024,"min":0.6999,"max":0.7046},"bestStep":[2000,2000,2000],"vsHard":{"heldOut":{"diffs":[0.0096,-0.0741,0.0065],"mean":-0.0193,"sd":0.0475,"resolved":false},"copyAccuracy":{"diffs":[-0.004,0.3992,-0.0217],"mean":0.1245,"sd":0.2381,"resolved":false},"agreement":{"diffs":[0.0799,0.174,0.0317],"mean":0.0952,"sd":0.0723,"resolved":true},"bestHeldOut":{"diffs":[0.0096,-0.0741,0.0065],"mean":-0.0193,"sd":0.0475,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0393,"copyAccuracy":0.1621,"agreement":0.7738},{"step":250,"heldOut":0.877,"copyAccuracy":0.2213,"agreement":0.8193},{"step":500,"heldOut":0.8189,"copyAccuracy":0.5751,"agreement":0.8674},{"step":1000,"heldOut":0.717,"copyAccuracy":0.9387,"agreement":0.9313},{"step":1500,"heldOut":0.7101,"copyAccuracy":0.9625,"agreement":0.9329},{"step":2000,"heldOut":0.7046,"copyAccuracy":0.9565,"agreement":0.918}]},"soft2":{"arm":"soft2","kind":"soft","T":2,"seeds":[1,2,3],"heldOut":{"values":[0.7026,0.7217,0.7824],"mean":0.7355,"sd":0.0417,"min":0.7026,"max":0.7824},"excess":{"values":[0.044,0.0631,0.1239],"mean":0.077,"sd":0.0417,"min":0.044,"max":0.1239},"copyAccuracy":{"values":[0.9664,0.9644,0.5632],"mean":0.8314,"sd":0.2322,"min":0.5632,"max":0.9664},"agreement":{"values":[0.9119,0.936,0.9155],"mean":0.9212,"sd":0.013,"min":0.9119,"max":0.936},"bestHeldOut":{"values":[0.7026,0.719,0.7824],"mean":0.7347,"sd":0.0422,"min":0.7026,"max":0.7824},"bestStep":[2000,1500,2000],"vsHard":{"heldOut":{"diffs":[0.0076,-0.0523,0.0875],"mean":0.0143,"sd":0.0702,"resolved":false},"copyAccuracy":{"diffs":[0.0059,0.3972,-0.415],"mean":-0.004,"sd":0.4062,"resolved":false},"agreement":{"diffs":[0.0739,0.1501,0.0449],"mean":0.0896,"sd":0.0543,"resolved":true},"bestHeldOut":{"diffs":[0.0076,-0.055,0.0875],"mean":0.0134,"sd":0.0715,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.1548,"copyAccuracy":0.1917,"agreement":0.6642},{"step":250,"heldOut":0.9426,"copyAccuracy":0.1423,"agreement":0.7666},{"step":500,"heldOut":0.8617,"copyAccuracy":0.2352,"agreement":0.7627},{"step":1000,"heldOut":0.744,"copyAccuracy":0.8024,"agreement":0.8584},{"step":1500,"heldOut":0.7077,"copyAccuracy":0.9526,"agreement":0.9251},{"step":2000,"heldOut":0.7026,"copyAccuracy":0.9664,"agreement":0.9119}]},"soft4":{"arm":"soft4","kind":"soft","T":4,"seeds":[1,2,3],"heldOut":{"values":[0.7145,0.7266,0.7276],"mean":0.7229,"sd":0.0073,"min":0.7145,"max":0.7276},"excess":{"values":[0.056,0.0681,0.0691],"mean":0.0644,"sd":0.0073,"min":0.056,"max":0.0691},"copyAccuracy":{"values":[0.9605,0.9427,0.8735],"mean":0.9256,"sd":0.0459,"min":0.8735,"max":0.9605},"agreement":{"values":[0.8826,0.9188,0.8708],"mean":0.8907,"sd":0.025,"min":0.8708,"max":0.9188},"bestHeldOut":{"values":[0.7145,0.7258,0.7276],"mean":0.7226,"sd":0.0071,"min":0.7145,"max":0.7276},"bestStep":[2000,1500,2000],"vsHard":{"heldOut":{"diffs":[0.0196,-0.0474,0.0327],"mean":0.0016,"sd":0.043,"resolved":false},"copyAccuracy":{"diffs":[0,0.3755,-0.1047],"mean":0.0903,"sd":0.2525,"resolved":false},"agreement":{"diffs":[0.0446,0.1328,0.0002],"mean":0.0592,"sd":0.0675,"resolved":false},"bestHeldOut":{"diffs":[0.0196,-0.0483,0.0327],"mean":0.0014,"sd":0.0435,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.2276,"copyAccuracy":0.1383,"agreement":0.6377},{"step":250,"heldOut":0.9447,"copyAccuracy":0.1779,"agreement":0.7762},{"step":500,"heldOut":0.8402,"copyAccuracy":0.3814,"agreement":0.8687},{"step":1000,"heldOut":0.7283,"copyAccuracy":0.9368,"agreement":0.9022},{"step":1500,"heldOut":0.7188,"copyAccuracy":0.9585,"agreement":0.9215},{"step":2000,"heldOut":0.7145,"copyAccuracy":0.9605,"agreement":0.8826}]},"soft8":{"arm":"soft8","kind":"soft","T":8,"seeds":[1,2,3],"heldOut":{"values":[0.7451,0.8025,0.736],"mean":0.7612,"sd":0.036,"min":0.736,"max":0.8025},"excess":{"values":[0.0866,0.1439,0.0774],"mean":0.1026,"sd":0.036,"min":0.0774,"max":0.1439},"copyAccuracy":{"values":[0.9486,0.5632,0.9625],"mean":0.8248,"sd":0.2266,"min":0.5632,"max":0.9625},"agreement":{"values":[0.8914,0.8244,0.8875],"mean":0.8678,"sd":0.0376,"min":0.8244,"max":0.8914},"bestHeldOut":{"values":[0.7451,0.8025,0.736],"mean":0.7612,"sd":0.036,"min":0.736,"max":0.8025},"bestStep":[2000,2000,2000],"vsHard":{"heldOut":{"diffs":[0.0502,0.0284,0.0411],"mean":0.0399,"sd":0.0109,"resolved":true},"copyAccuracy":{"diffs":[-0.0119,-0.004,-0.0158],"mean":-0.0105,"sd":0.006,"resolved":true},"agreement":{"diffs":[0.0534,0.0384,0.0169],"mean":0.0362,"sd":0.0183,"resolved":true},"bestHeldOut":{"diffs":[0.0502,0.0284,0.0411],"mean":0.0399,"sd":0.0109,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.3972,"copyAccuracy":0.0929,"agreement":0.5373},{"step":250,"heldOut":0.9592,"copyAccuracy":0.1502,"agreement":0.7576},{"step":500,"heldOut":0.819,"copyAccuracy":0.6996,"agreement":0.8675},{"step":1000,"heldOut":0.7576,"copyAccuracy":0.9368,"agreement":0.903},{"step":1500,"heldOut":0.7484,"copyAccuracy":0.9348,"agreement":0.8901},{"step":2000,"heldOut":0.7451,"copyAccuracy":0.9486,"agreement":0.8914}]},"mix2":{"arm":"mix2","kind":"soft","T":2,"alpha":0.5,"seeds":[1,2,3],"heldOut":{"values":[0.6963,0.7004,0.6923],"mean":0.6963,"sd":0.0041,"min":0.6923,"max":0.7004},"excess":{"values":[0.0377,0.0419,0.0337],"mean":0.0378,"sd":0.0041,"min":0.0337,"max":0.0419},"copyAccuracy":{"values":[0.9723,0.9565,0.9723],"mean":0.9671,"sd":0.0091,"min":0.9565,"max":0.9723},"agreement":{"values":[0.9185,0.8667,0.8774],"mean":0.8875,"sd":0.0273,"min":0.8667,"max":0.9185},"bestHeldOut":{"values":[0.6963,0.7004,0.6923],"mean":0.6963,"sd":0.0041,"min":0.6923,"max":0.7004},"bestStep":[2000,2000,2000],"vsHard":{"heldOut":{"diffs":[0.0014,-0.0736,-0.0026],"mean":-0.025,"sd":0.0422,"resolved":false},"copyAccuracy":{"diffs":[0.0119,0.3893,-0.0059],"mean":0.1318,"sd":0.2232,"resolved":false},"agreement":{"diffs":[0.0804,0.0807,0.0068],"mean":0.056,"sd":0.0426,"resolved":true},"bestHeldOut":{"diffs":[0.0014,-0.0736,-0.0026],"mean":-0.025,"sd":0.0422,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0863,"copyAccuracy":0.1838,"agreement":0.6626},{"step":250,"heldOut":0.8875,"copyAccuracy":0.1522,"agreement":0.7658},{"step":500,"heldOut":0.8479,"copyAccuracy":0.3063,"agreement":0.7441},{"step":1000,"heldOut":0.7277,"copyAccuracy":0.7925,"agreement":0.8293},{"step":1500,"heldOut":0.7094,"copyAccuracy":0.9447,"agreement":0.8451},{"step":2000,"heldOut":0.6963,"copyAccuracy":0.9723,"agreement":0.9185}]},"oracle1":{"arm":"oracle1","kind":"oracle","T":1,"seeds":[1,2,3],"heldOut":{"values":[0.6805,0.6956,0.6895],"mean":0.6885,"sd":0.0076,"min":0.6805,"max":0.6956},"excess":{"values":[0.0219,0.0371,0.0309],"mean":0.03,"sd":0.0076,"min":0.0219,"max":0.0371},"copyAccuracy":{"values":[0.9625,0.9664,0.9723],"mean":0.9671,"sd":0.005,"min":0.9625,"max":0.9723},"agreement":{"values":[0.8148,0.8371,0.8143],"mean":0.822,"sd":0.013,"min":0.8143,"max":0.8371},"bestHeldOut":{"values":[0.6805,0.6847,0.6846],"mean":0.6833,"sd":0.0024,"min":0.6805,"max":0.6847},"bestStep":[2000,1500,1500],"vsHard":{"heldOut":{"diffs":[-0.0145,-0.0784,-0.0054],"mean":-0.0328,"sd":0.0398,"resolved":false},"copyAccuracy":{"diffs":[0.002,0.3992,-0.0059],"mean":0.1318,"sd":0.2317,"resolved":false},"agreement":{"diffs":[-0.0233,0.0511,-0.0563],"mean":-0.0095,"sd":0.055,"resolved":false},"bestHeldOut":{"diffs":[-0.0145,-0.0893,-0.0102],"mean":-0.038,"sd":0.0445,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0181,"copyAccuracy":0.1858,"agreement":0.715},{"step":250,"heldOut":0.8554,"copyAccuracy":0.2391,"agreement":0.7759},{"step":500,"heldOut":0.764,"copyAccuracy":0.6067,"agreement":0.8255},{"step":1000,"heldOut":0.7111,"copyAccuracy":0.9466,"agreement":0.7694},{"step":1500,"heldOut":0.6838,"copyAccuracy":0.9625,"agreement":0.8245},{"step":2000,"heldOut":0.6805,"copyAccuracy":0.9625,"agreement":0.8148}]},"oracle2":{"arm":"oracle2","kind":"oracle","T":2,"seeds":[1,2,3],"heldOut":{"values":[0.6971,0.7073,0.6891],"mean":0.6978,"sd":0.0091,"min":0.6891,"max":0.7073},"excess":{"values":[0.0385,0.0488,0.0306],"mean":0.0393,"sd":0.0091,"min":0.0306,"max":0.0488},"copyAccuracy":{"values":[0.9704,0.9644,0.9743],"mean":0.9697,"sd":0.005,"min":0.9644,"max":0.9743},"agreement":{"values":[0.7783,0.839,0.786],"mean":0.8011,"sd":0.0331,"min":0.7783,"max":0.839},"bestHeldOut":{"values":[0.6971,0.7073,0.6891],"mean":0.6978,"sd":0.0091,"min":0.6891,"max":0.7073},"bestStep":[2000,2000,2000],"vsHard":{"heldOut":{"diffs":[0.0021,-0.0667,-0.0058],"mean":-0.0234,"sd":0.0377,"resolved":false},"copyAccuracy":{"diffs":[0.0099,0.3972,-0.004],"mean":0.1344,"sd":0.2277,"resolved":false},"agreement":{"diffs":[-0.0597,0.0531,-0.0846],"mean":-0.0304,"sd":0.0734,"resolved":false},"bestHeldOut":{"diffs":[0.0021,-0.0667,-0.0058],"mean":-0.0234,"sd":0.0377,"resolved":false}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0978,"copyAccuracy":0.1779,"agreement":0.6426},{"step":250,"heldOut":0.9156,"copyAccuracy":0.1976,"agreement":0.7314},{"step":500,"heldOut":0.8454,"copyAccuracy":0.2569,"agreement":0.7983},{"step":1000,"heldOut":0.7231,"copyAccuracy":0.9407,"agreement":0.825},{"step":1500,"heldOut":0.7107,"copyAccuracy":0.9526,"agreement":0.8327},{"step":2000,"heldOut":0.6971,"copyAccuracy":0.9704,"agreement":0.7783}]},"untrained2":{"arm":"untrained2","kind":"soft","T":2,"untrained":true,"seeds":[1,2,3],"heldOut":{"values":[3.8394,3.8389,3.8391],"mean":3.8391,"sd":0.0002,"min":3.8389,"max":3.8394},"excess":{"values":[3.1808,3.1804,3.1806],"mean":3.1806,"sd":0.0002,"min":3.1804,"max":3.1808},"copyAccuracy":{"values":[0.0751,0.0751,0.0731],"mean":0.0744,"sd":0.0011,"min":0.0731,"max":0.0751},"agreement":{"values":[0.0213,0.0234,0.0212],"mean":0.022,"sd":0.0013,"min":0.0212,"max":0.0234},"bestHeldOut":{"values":[3.834,3.8294,3.8384],"mean":3.8339,"sd":0.0045,"min":3.8294,"max":3.8384},"bestStep":[0,0,250],"vsHard":{"heldOut":{"diffs":[3.1444,3.0649,3.1442],"mean":3.1179,"sd":0.0459,"resolved":true},"copyAccuracy":{"diffs":[-0.8854,-0.4921,-0.9051],"mean":-0.7609,"sd":0.233,"resolved":true},"agreement":{"diffs":[-0.8167,-0.7625,-0.8494],"mean":-0.8096,"sd":0.0439,"resolved":true},"bestHeldOut":{"diffs":[3.1391,3.0553,3.1436],"mean":3.1127,"sd":0.0497,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":3.8385,"copyAccuracy":0.0889,"agreement":0.0254},{"step":250,"heldOut":3.839,"copyAccuracy":0.0632,"agreement":0.0225},{"step":500,"heldOut":3.839,"copyAccuracy":0.0692,"agreement":0.0225},{"step":1000,"heldOut":3.8389,"copyAccuracy":0.0731,"agreement":0.0213},{"step":1500,"heldOut":3.8392,"copyAccuracy":0.0692,"agreement":0.0207},{"step":2000,"heldOut":3.8394,"copyAccuracy":0.0751,"agreement":0.0213}]}},"verdicts":["full data, soft1: held-out loss 0.7019 (sd 0.0024) against hard labels' 0.7213; lower by 0.0193 nats on average over 3 paired seeds, not resolved (the seeds disagree in sign or the mean is within two standard errors).","full data, soft2: held-out loss 0.7355 (sd 0.0417) against hard labels' 0.7213; higher by 0.0143 nats on average over 3 paired seeds, not resolved (the seeds disagree in sign or the mean is within two standard errors).","full data, soft4: held-out loss 0.7229 (sd 0.0073) against hard labels' 0.7213; higher by 0.0016 nats on average over 3 paired seeds, not resolved (the seeds disagree in sign or the mean is within two standard errors).","full data, soft8: held-out loss 0.7612 (sd 0.036) against hard labels' 0.7213; higher by 0.0399 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","full data, mix2: held-out loss 0.6963 (sd 0.0041) against hard labels' 0.7213; lower by 0.025 nats on average over 3 paired seeds, not resolved (the seeds disagree in sign or the mean is within two standard errors).","full data, argmax: held-out loss 3.8429 (sd 0.1089) against hard labels' 0.7213; higher by 3.1216 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","full data, oracle1: held-out loss 0.6885 (sd 0.0076) against hard labels' 0.7213; lower by 0.0328 nats on average over 3 paired seeds, not resolved (the seeds disagree in sign or the mean is within two standard errors)."]},"temperature":{"about":"The temperature sweep (pure soft targets, alpha 1) side by side for full data and the 32-row pool, three seeds each; vsHard is the paired difference in held-out loss. vsT2 is each temperature's paired difference from T 2 (same seeds and rows), with the same resolved rule. The film's main temperature is 2, the middle of the sweep: a first full run of this fill showed no temperature that stands clear of the others.","mainT":2,"full":[{"T":1,"heldOut":{"values":[0.7046,0.6999,0.7014],"mean":0.7019,"sd":0.0024,"min":0.6999,"max":0.7046},"copyAccuracy":{"values":[0.9565,0.9664,0.9565],"mean":0.9598,"sd":0.0057,"min":0.9565,"max":0.9664},"agreement":{"values":[0.918,0.96,0.9023],"mean":0.9268,"sd":0.0298,"min":0.9023,"max":0.96},"vsHard":{"diffs":[0.0096,-0.0741,0.0065],"mean":-0.0193,"sd":0.0475,"resolved":false},"vsT2":{"diffs":[0.002,-0.0218,-0.081],"mean":-0.0336,"sd":0.0428,"resolved":false}},{"T":2,"heldOut":{"values":[0.7026,0.7217,0.7824],"mean":0.7355,"sd":0.0417,"min":0.7026,"max":0.7824},"copyAccuracy":{"values":[0.9664,0.9644,0.5632],"mean":0.8314,"sd":0.2322,"min":0.5632,"max":0.9664},"agreement":{"values":[0.9119,0.936,0.9155],"mean":0.9212,"sd":0.013,"min":0.9119,"max":0.936},"vsHard":{"diffs":[0.0076,-0.0523,0.0875],"mean":0.0143,"sd":0.0702,"resolved":false},"vsT2":null},{"T":4,"heldOut":{"values":[0.7145,0.7266,0.7276],"mean":0.7229,"sd":0.0073,"min":0.7145,"max":0.7276},"copyAccuracy":{"values":[0.9605,0.9427,0.8735],"mean":0.9256,"sd":0.0459,"min":0.8735,"max":0.9605},"agreement":{"values":[0.8826,0.9188,0.8708],"mean":0.8907,"sd":0.025,"min":0.8708,"max":0.9188},"vsHard":{"diffs":[0.0196,-0.0474,0.0327],"mean":0.0016,"sd":0.043,"resolved":false},"vsT2":{"diffs":[0.0119,0.0049,-0.0548],"mean":-0.0126,"sd":0.0367,"resolved":false}},{"T":8,"heldOut":{"values":[0.7451,0.8025,0.736],"mean":0.7612,"sd":0.036,"min":0.736,"max":0.8025},"copyAccuracy":{"values":[0.9486,0.5632,0.9625],"mean":0.8248,"sd":0.2266,"min":0.5632,"max":0.9625},"agreement":{"values":[0.8914,0.8244,0.8875],"mean":0.8678,"sd":0.0376,"min":0.8244,"max":0.8914},"vsHard":{"diffs":[0.0502,0.0284,0.0411],"mean":0.0399,"sd":0.0109,"resolved":true},"vsT2":{"diffs":[0.0426,0.0808,-0.0464],"mean":0.0256,"sd":0.0653,"resolved":false}}],"pool32":[{"T":1,"heldOut":{"values":[1.5017,1.476,1.801],"mean":1.5929,"sd":0.1807,"min":1.476,"max":1.801},"copyAccuracy":{"values":[0.3834,0.2569,0.2549],"mean":0.2984,"sd":0.0736,"min":0.2549,"max":0.3834},"agreement":{"values":[0.7393,0.7804,0.7498],"mean":0.7565,"sd":0.0214,"min":0.7393,"max":0.7804},"vsHard":{"diffs":[-3.1209,-3.3563,-3.1471],"mean":-3.2081,"sd":0.129,"resolved":true},"vsT2":{"diffs":[-0.0892,-0.3144,0.271],"mean":-0.0442,"sd":0.2953,"resolved":false}},{"T":2,"heldOut":{"values":[1.5908,1.7904,1.5299],"mean":1.6371,"sd":0.1362,"min":1.5299,"max":1.7904},"copyAccuracy":{"values":[0.3794,0.336,0.3874],"mean":0.3676,"sd":0.0277,"min":0.336,"max":0.3874},"agreement":{"values":[0.7638,0.7397,0.7956],"mean":0.7664,"sd":0.028,"min":0.7397,"max":0.7956},"vsHard":{"diffs":[-3.0318,-3.0419,-3.4182],"mean":-3.1639,"sd":0.2202,"resolved":true},"vsT2":null},{"T":4,"heldOut":{"values":[1.4524,1.7835,1.4891],"mean":1.575,"sd":0.1815,"min":1.4524,"max":1.7835},"copyAccuracy":{"values":[0.4743,0.2708,0.3696],"mean":0.3715,"sd":0.1018,"min":0.2708,"max":0.4743},"agreement":{"values":[0.7969,0.745,0.7537],"mean":0.7652,"sd":0.0278,"min":0.745,"max":0.7969},"vsHard":{"diffs":[-3.1702,-3.0488,-3.459],"mean":-3.226,"sd":0.2108,"resolved":true},"vsT2":{"diffs":[-0.1384,-0.0069,-0.0409],"mean":-0.0621,"sd":0.0683,"resolved":false}},{"T":8,"heldOut":{"values":[1.5752,1.659,1.4061],"mean":1.5468,"sd":0.1288,"min":1.4061,"max":1.659},"copyAccuracy":{"values":[0.3419,0.2372,0.3123],"mean":0.2971,"sd":0.054,"min":0.2372,"max":0.3419},"agreement":{"values":[0.7367,0.7303,0.7606],"mean":0.7425,"sd":0.016,"min":0.7303,"max":0.7606},"vsHard":{"diffs":[-3.0474,-3.1733,-3.5421],"mean":-3.2542,"sd":0.2571,"resolved":true},"vsT2":{"diffs":[-0.0156,-0.1314,-0.1239],"mean":-0.0903,"sd":0.0648,"resolved":true}}]},"lowData":{"about":"Hinton and colleagues' strongest claim: soft targets carry generalisation from little data. Every arm trains on the same small fixed pool (32 or 256 rows of 24 words, pool seed 200 + student seed, served in shuffled passes) for 2000 steps; the teacher's soft targets are computed on those same pool rows. bestHeldOut is the lowest held-out loss at the checkpoints (0, 100, 250, 500, 1000, 1500, 2000): what stopping early at the best checkpoint would give, read off the held-out rows themselves, so it flatters every arm equally. The teacher itself was trained on fresh rows, so its soft targets carry what it learned from far more data than the pool: this measures transfer through soft targets, not learning from 32 rows alone.","pool":{"small":32,"mid":256,"words":24,"passesSmall":1000,"passesMid":125},"pool32":{"hard":{"arm":"hard","kind":"hard","seeds":[1,2,3],"heldOut":{"values":[4.6226,4.8323,4.9481],"mean":4.801,"sd":0.165,"min":4.6226,"max":4.9481},"excess":{"values":[3.964,4.1737,4.2896],"mean":4.1425,"sd":0.165,"min":3.964,"max":4.2896},"copyAccuracy":{"values":[0.2688,0.2925,0.3043],"mean":0.2885,"sd":0.0181,"min":0.2688,"max":0.3043},"agreement":{"values":[0.5675,0.5654,0.589],"mean":0.574,"sd":0.0131,"min":0.5654,"max":0.589},"bestHeldOut":{"values":[1.7304,1.6576,1.7731],"mean":1.7204,"sd":0.0584,"min":1.6576,"max":1.7731},"bestStep":[100,100,100],"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.7304,"copyAccuracy":0.17,"agreement":0.5941},{"step":250,"heldOut":3.2119,"copyAccuracy":0.2372,"agreement":0.5833},{"step":500,"heldOut":3.9424,"copyAccuracy":0.253,"agreement":0.5566},{"step":1000,"heldOut":4.5369,"copyAccuracy":0.2767,"agreement":0.5763},{"step":1500,"heldOut":4.7962,"copyAccuracy":0.2628,"agreement":0.5669},{"step":2000,"heldOut":4.6226,"copyAccuracy":0.2688,"agreement":0.5675}]},"argmax":{"arm":"argmax","kind":"argmax","seeds":[1,2,3],"heldOut":{"values":[6.4833,7.6891,7.9361],"mean":7.3695,"sd":0.7773,"min":6.4833,"max":7.9361},"excess":{"values":[5.8248,7.0306,7.2776],"mean":6.711,"sd":0.7773,"min":5.8248,"max":7.2776},"copyAccuracy":{"values":[0.4862,0.3063,0.2984],"mean":0.3636,"sd":0.1062,"min":0.2984,"max":0.4862},"agreement":{"values":[0.7972,0.792,0.7948],"mean":0.7947,"sd":0.0026,"min":0.792,"max":0.7972},"bestHeldOut":{"values":[2.7484,2.4991,2.7475],"mean":2.665,"sd":0.1437,"min":2.4991,"max":2.7484},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[1.8608,2.8568,2.988],"mean":2.5685,"sd":0.6164,"resolved":true},"copyAccuracy":{"diffs":[0.2174,0.0138,-0.0059],"mean":0.0751,"sd":0.1236,"resolved":false},"agreement":{"diffs":[0.2297,0.2266,0.2057],"mean":0.2206,"sd":0.013,"resolved":true},"bestHeldOut":{"diffs":[1.018,0.8415,0.9744],"mean":0.9446,"sd":0.0919,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":2.7484,"copyAccuracy":0.2253,"agreement":0.7803},{"step":250,"heldOut":4.0967,"copyAccuracy":0.3874,"agreement":0.7878},{"step":500,"heldOut":4.7432,"copyAccuracy":0.417,"agreement":0.798},{"step":1000,"heldOut":5.7296,"copyAccuracy":0.4328,"agreement":0.8016},{"step":1500,"heldOut":6.761,"copyAccuracy":0.4308,"agreement":0.8008},{"step":2000,"heldOut":6.4833,"copyAccuracy":0.4862,"agreement":0.7972}]},"soft1":{"arm":"soft1","kind":"soft","T":1,"seeds":[1,2,3],"heldOut":{"values":[1.5017,1.476,1.801],"mean":1.5929,"sd":0.1807,"min":1.476,"max":1.801},"excess":{"values":[0.8431,0.8175,1.1424],"mean":0.9343,"sd":0.1807,"min":0.8175,"max":1.1424},"copyAccuracy":{"values":[0.3834,0.2569,0.2549],"mean":0.2984,"sd":0.0736,"min":0.2549,"max":0.3834},"agreement":{"values":[0.7393,0.7804,0.7498],"mean":0.7565,"sd":0.0214,"min":0.7393,"max":0.7804},"bestHeldOut":{"values":[1.2493,1.2444,1.277],"mean":1.2569,"sd":0.0176,"min":1.2444,"max":1.277},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-3.1209,-3.3563,-3.1471],"mean":-3.2081,"sd":0.129,"resolved":true},"copyAccuracy":{"diffs":[0.1146,-0.0356,-0.0494],"mean":0.0099,"sd":0.091,"resolved":false},"agreement":{"diffs":[0.1717,0.215,0.1608],"mean":0.1825,"sd":0.0287,"resolved":true},"bestHeldOut":{"diffs":[-0.4811,-0.4133,-0.4961],"mean":-0.4635,"sd":0.0442,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.2493,"copyAccuracy":0.2292,"agreement":0.7256},{"step":250,"heldOut":1.4782,"copyAccuracy":0.2984,"agreement":0.7476},{"step":500,"heldOut":1.5697,"copyAccuracy":0.2964,"agreement":0.7264},{"step":1000,"heldOut":1.4706,"copyAccuracy":0.3735,"agreement":0.7606},{"step":1500,"heldOut":1.535,"copyAccuracy":0.3498,"agreement":0.7736},{"step":2000,"heldOut":1.5017,"copyAccuracy":0.3834,"agreement":0.7393}]},"soft2":{"arm":"soft2","kind":"soft","T":2,"seeds":[1,2,3],"heldOut":{"values":[1.5908,1.7904,1.5299],"mean":1.6371,"sd":0.1362,"min":1.5299,"max":1.7904},"excess":{"values":[0.9323,1.1318,0.8714],"mean":0.9785,"sd":0.1362,"min":0.8714,"max":1.1318},"copyAccuracy":{"values":[0.3794,0.336,0.3874],"mean":0.3676,"sd":0.0277,"min":0.336,"max":0.3874},"agreement":{"values":[0.7638,0.7397,0.7956],"mean":0.7664,"sd":0.028,"min":0.7397,"max":0.7956},"bestHeldOut":{"values":[1.3205,1.4453,1.3906],"mean":1.3854,"sd":0.0626,"min":1.3205,"max":1.4453},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-3.0318,-3.0419,-3.4182],"mean":-3.1639,"sd":0.2202,"resolved":true},"copyAccuracy":{"diffs":[0.1107,0.0435,0.083],"mean":0.0791,"sd":0.0338,"resolved":true},"agreement":{"diffs":[0.1963,0.1743,0.2065],"mean":0.1924,"sd":0.0165,"resolved":true},"bestHeldOut":{"diffs":[-0.41,-0.2124,-0.3826],"mean":-0.335,"sd":0.1071,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.3205,"copyAccuracy":0.1719,"agreement":0.6761},{"step":250,"heldOut":1.5331,"copyAccuracy":0.2628,"agreement":0.7376},{"step":500,"heldOut":1.5481,"copyAccuracy":0.336,"agreement":0.7399},{"step":1000,"heldOut":1.5689,"copyAccuracy":0.3577,"agreement":0.75},{"step":1500,"heldOut":1.6051,"copyAccuracy":0.3676,"agreement":0.7554},{"step":2000,"heldOut":1.5908,"copyAccuracy":0.3794,"agreement":0.7638}]},"soft4":{"arm":"soft4","kind":"soft","T":4,"seeds":[1,2,3],"heldOut":{"values":[1.4524,1.7835,1.4891],"mean":1.575,"sd":0.1815,"min":1.4524,"max":1.7835},"excess":{"values":[0.7939,1.125,0.8305],"mean":0.9165,"sd":0.1815,"min":0.7939,"max":1.125},"copyAccuracy":{"values":[0.4743,0.2708,0.3696],"mean":0.3715,"sd":0.1018,"min":0.2708,"max":0.4743},"agreement":{"values":[0.7969,0.745,0.7537],"mean":0.7652,"sd":0.0278,"min":0.745,"max":0.7969},"bestHeldOut":{"values":[1.4038,1.5302,1.4239],"mean":1.4526,"sd":0.0679,"min":1.4038,"max":1.5302},"bestStep":[100,100,250],"vsHard":{"heldOut":{"diffs":[-3.1702,-3.0488,-3.459],"mean":-3.226,"sd":0.2108,"resolved":true},"copyAccuracy":{"diffs":[0.2055,-0.0217,0.0652],"mean":0.083,"sd":0.1147,"resolved":false},"agreement":{"diffs":[0.2293,0.1795,0.1647],"mean":0.1912,"sd":0.0339,"resolved":true},"bestHeldOut":{"diffs":[-0.3266,-0.1274,-0.3492],"mean":-0.2677,"sd":0.1221,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.4038,"copyAccuracy":0.1719,"agreement":0.6444},{"step":250,"heldOut":1.4763,"copyAccuracy":0.247,"agreement":0.7262},{"step":500,"heldOut":1.552,"copyAccuracy":0.3241,"agreement":0.7746},{"step":1000,"heldOut":1.485,"copyAccuracy":0.4348,"agreement":0.7677},{"step":1500,"heldOut":1.4785,"copyAccuracy":0.4644,"agreement":0.779},{"step":2000,"heldOut":1.4524,"copyAccuracy":0.4743,"agreement":0.7969}]},"soft8":{"arm":"soft8","kind":"soft","T":8,"seeds":[1,2,3],"heldOut":{"values":[1.5752,1.659,1.4061],"mean":1.5468,"sd":0.1288,"min":1.4061,"max":1.659},"excess":{"values":[0.9167,1.0004,0.7475],"mean":0.8882,"sd":0.1288,"min":0.7475,"max":1.0004},"copyAccuracy":{"values":[0.3419,0.2372,0.3123],"mean":0.2971,"sd":0.054,"min":0.2372,"max":0.3419},"agreement":{"values":[0.7367,0.7303,0.7606],"mean":0.7425,"sd":0.016,"min":0.7303,"max":0.7606},"bestHeldOut":{"values":[1.5662,1.4812,1.3931],"mean":1.4802,"sd":0.0866,"min":1.3931,"max":1.5662},"bestStep":[100,250,100],"vsHard":{"heldOut":{"diffs":[-3.0474,-3.1733,-3.5421],"mean":-3.2542,"sd":0.2571,"resolved":true},"copyAccuracy":{"diffs":[0.0731,-0.0553,0.0079],"mean":0.0086,"sd":0.0642,"resolved":false},"agreement":{"diffs":[0.1691,0.1649,0.1715],"mean":0.1685,"sd":0.0034,"resolved":true},"bestHeldOut":{"diffs":[-0.1642,-0.1764,-0.38],"mean":-0.2402,"sd":0.1212,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.5662,"copyAccuracy":0.1047,"agreement":0.5293},{"step":250,"heldOut":1.5923,"copyAccuracy":0.2312,"agreement":0.7017},{"step":500,"heldOut":1.6083,"copyAccuracy":0.2727,"agreement":0.7171},{"step":1000,"heldOut":1.6101,"copyAccuracy":0.3557,"agreement":0.7406},{"step":1500,"heldOut":1.6211,"copyAccuracy":0.336,"agreement":0.7397},{"step":2000,"heldOut":1.5752,"copyAccuracy":0.3419,"agreement":0.7367}]},"mix2":{"arm":"mix2","kind":"soft","T":2,"alpha":0.5,"seeds":[1,2,3],"heldOut":{"values":[2.1039,2.0301,2.0724],"mean":2.0688,"sd":0.037,"min":2.0301,"max":2.1039},"excess":{"values":[1.4454,1.3716,1.4138],"mean":1.4102,"sd":0.037,"min":1.3716,"max":1.4454},"copyAccuracy":{"values":[0.2787,0.2312,0.2036],"mean":0.2378,"sd":0.038,"min":0.2036,"max":0.2787},"agreement":{"values":[0.6169,0.6326,0.5993],"mean":0.6163,"sd":0.0167,"min":0.5993,"max":0.6326},"bestHeldOut":{"values":[1.2995,1.4653,1.3991],"mean":1.388,"sd":0.0835,"min":1.2995,"max":1.4653},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-2.5187,-2.8022,-2.8758],"mean":-2.7322,"sd":0.1885,"resolved":true},"copyAccuracy":{"diffs":[0.0099,-0.0613,-0.1008],"mean":-0.0507,"sd":0.0561,"resolved":false},"agreement":{"diffs":[0.0493,0.0672,0.0103],"mean":0.0423,"sd":0.0291,"resolved":true},"bestHeldOut":{"diffs":[-0.4309,-0.1923,-0.374],"mean":-0.3324,"sd":0.1246,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.2995,"copyAccuracy":0.1739,"agreement":0.6335},{"step":250,"heldOut":1.7936,"copyAccuracy":0.2391,"agreement":0.6009},{"step":500,"heldOut":1.9624,"copyAccuracy":0.2549,"agreement":0.5949},{"step":1000,"heldOut":2.1253,"copyAccuracy":0.2727,"agreement":0.6024},{"step":1500,"heldOut":2.1073,"copyAccuracy":0.2905,"agreement":0.6105},{"step":2000,"heldOut":2.1039,"copyAccuracy":0.2787,"agreement":0.6169}]},"oracle1":{"arm":"oracle1","kind":"oracle","T":1,"seeds":[1,2,3],"heldOut":{"values":[1.8024,1.6665,1.9335],"mean":1.8008,"sd":0.1335,"min":1.6665,"max":1.9335},"excess":{"values":[1.1438,1.0079,1.275],"mean":1.1422,"sd":0.1335,"min":1.0079,"max":1.275},"copyAccuracy":{"values":[0.3518,0.3004,0.3083],"mean":0.3202,"sd":0.0277,"min":0.3004,"max":0.3518},"agreement":{"values":[0.6914,0.6683,0.6603],"mean":0.6733,"sd":0.0161,"min":0.6603,"max":0.6914},"bestHeldOut":{"values":[1.2221,1.2311,1.252],"mean":1.2351,"sd":0.0153,"min":1.2221,"max":1.252},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-2.8202,-3.1658,-3.0146],"mean":-3.0002,"sd":0.1733,"resolved":true},"copyAccuracy":{"diffs":[0.083,0.0079,0.004],"mean":0.0316,"sd":0.0445,"resolved":false},"agreement":{"diffs":[0.1239,0.1029,0.0713],"mean":0.0993,"sd":0.0265,"resolved":true},"bestHeldOut":{"diffs":[-0.5083,-0.4265,-0.5212],"mean":-0.4853,"sd":0.0513,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.2221,"copyAccuracy":0.1858,"agreement":0.658},{"step":250,"heldOut":1.4829,"copyAccuracy":0.2668,"agreement":0.6722},{"step":500,"heldOut":1.5471,"copyAccuracy":0.2984,"agreement":0.667},{"step":1000,"heldOut":1.8498,"copyAccuracy":0.2747,"agreement":0.6784},{"step":1500,"heldOut":1.8239,"copyAccuracy":0.3221,"agreement":0.6823},{"step":2000,"heldOut":1.8024,"copyAccuracy":0.3518,"agreement":0.6914}]},"oracle2":{"arm":"oracle2","kind":"oracle","T":2,"seeds":[1,2,3],"heldOut":{"values":[3.4793,2.942,3.1932],"mean":3.2048,"sd":0.2688,"min":2.942,"max":3.4793},"excess":{"values":[2.8207,2.2834,2.5346],"mean":2.5463,"sd":0.2688,"min":2.2834,"max":2.8207},"copyAccuracy":{"values":[0.3162,0.251,0.2352],"mean":0.2675,"sd":0.043,"min":0.2352,"max":0.3162},"agreement":{"values":[0.6418,0.6405,0.6549],"mean":0.6457,"sd":0.008,"min":0.6405,"max":0.6549},"bestHeldOut":{"values":[1.3197,1.3842,1.4571],"mean":1.387,"sd":0.0688,"min":1.3197,"max":1.4571},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[-1.1433,-1.8903,-1.755],"mean":-1.5962,"sd":0.398,"resolved":true},"copyAccuracy":{"diffs":[0.0474,-0.0415,-0.0692],"mean":-0.0211,"sd":0.0609,"resolved":false},"agreement":{"diffs":[0.0742,0.075,0.0659],"mean":0.0717,"sd":0.005,"resolved":true},"bestHeldOut":{"diffs":[-0.4108,-0.2735,-0.316],"mean":-0.3334,"sd":0.0703,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.3197,"copyAccuracy":0.1779,"agreement":0.6577},{"step":250,"heldOut":2.2337,"copyAccuracy":0.2826,"agreement":0.6401},{"step":500,"heldOut":2.7328,"copyAccuracy":0.3043,"agreement":0.6411},{"step":1000,"heldOut":3.0231,"copyAccuracy":0.3419,"agreement":0.6494},{"step":1500,"heldOut":2.7882,"copyAccuracy":0.3458,"agreement":0.6532},{"step":2000,"heldOut":3.4793,"copyAccuracy":0.3162,"agreement":0.6418}]}},"pool256":{"hard":{"arm":"hard","kind":"hard","seeds":[1,2,3],"heldOut":{"values":[1.2595,1.3918,1.1751],"mean":1.2755,"sd":0.1092,"min":1.1751,"max":1.3918},"excess":{"values":[0.601,0.7333,0.5166],"mean":0.617,"sd":0.1092,"min":0.5166,"max":0.7333},"copyAccuracy":{"values":[0.8221,0.8182,0.8261],"mean":0.8221,"sd":0.004,"min":0.8182,"max":0.8261},"agreement":{"values":[0.7352,0.717,0.7599],"mean":0.7374,"sd":0.0216,"min":0.717,"max":0.7599},"bestHeldOut":{"values":[0.9079,0.9255,0.8802],"mean":0.9045,"sd":0.0228,"min":0.8802,"max":0.9255},"bestStep":[250,500,500],"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.0401,"copyAccuracy":0.166,"agreement":0.6898},{"step":250,"heldOut":0.9079,"copyAccuracy":0.1798,"agreement":0.7673},{"step":500,"heldOut":0.9235,"copyAccuracy":0.4466,"agreement":0.7321},{"step":1000,"heldOut":0.9629,"copyAccuracy":0.7747,"agreement":0.7583},{"step":1500,"heldOut":1.1402,"copyAccuracy":0.8083,"agreement":0.7489},{"step":2000,"heldOut":1.2595,"copyAccuracy":0.8221,"agreement":0.7352}]},"argmax":{"arm":"argmax","kind":"argmax","seeds":[1,2,3],"heldOut":{"values":[5.2635,5.661,5.6844],"mean":5.5363,"sd":0.2365,"min":5.2635,"max":5.6844},"excess":{"values":[4.605,5.0024,5.0259],"mean":4.8778,"sd":0.2365,"min":4.605,"max":5.0259},"copyAccuracy":{"values":[0.9071,0.8933,0.9407],"mean":0.9137,"sd":0.0244,"min":0.8933,"max":0.9407},"agreement":{"values":[0.9647,0.9639,0.9764],"mean":0.9683,"sd":0.007,"min":0.9639,"max":0.9764},"bestHeldOut":{"values":[2.0654,2.111,1.9849],"mean":2.0538,"sd":0.0639,"min":1.9849,"max":2.111},"bestStep":[100,100,100],"vsHard":{"heldOut":{"diffs":[4.004,4.2692,4.5093],"mean":4.2608,"sd":0.2528,"resolved":true},"copyAccuracy":{"diffs":[0.085,0.0751,0.1146],"mean":0.0916,"sd":0.0206,"resolved":true},"agreement":{"diffs":[0.2295,0.2469,0.2165],"mean":0.231,"sd":0.0153,"resolved":true},"bestHeldOut":{"diffs":[1.1575,1.1855,1.1047],"mean":1.1492,"sd":0.041,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":2.0654,"copyAccuracy":0.2036,"agreement":0.8374},{"step":250,"heldOut":2.6914,"copyAccuracy":0.4051,"agreement":0.911},{"step":500,"heldOut":3.6301,"copyAccuracy":0.8024,"agreement":0.946},{"step":1000,"heldOut":4.3042,"copyAccuracy":0.8854,"agreement":0.9613},{"step":1500,"heldOut":4.9291,"copyAccuracy":0.8577,"agreement":0.9549},{"step":2000,"heldOut":5.2635,"copyAccuracy":0.9071,"agreement":0.9647}]},"soft2":{"arm":"soft2","kind":"soft","T":2,"seeds":[1,2,3],"heldOut":{"values":[0.7309,0.7177,0.7336],"mean":0.7274,"sd":0.0085,"min":0.7177,"max":0.7336},"excess":{"values":[0.0723,0.0592,0.0751],"mean":0.0689,"sd":0.0085,"min":0.0592,"max":0.0751},"copyAccuracy":{"values":[0.9368,0.9407,0.9407],"mean":0.9394,"sd":0.0023,"min":0.9368,"max":0.9407},"agreement":{"values":[0.9403,0.932,0.9315],"mean":0.9346,"sd":0.0049,"min":0.9315,"max":0.9403},"bestHeldOut":{"values":[0.7253,0.7177,0.7223],"mean":0.7218,"sd":0.0038,"min":0.7177,"max":0.7253},"bestStep":[1500,2000,1500],"vsHard":{"heldOut":{"diffs":[-0.5287,-0.6741,-0.4415],"mean":-0.5481,"sd":0.1175,"resolved":true},"copyAccuracy":{"diffs":[0.1146,0.1225,0.1146],"mean":0.1173,"sd":0.0046,"resolved":true},"agreement":{"diffs":[0.2051,0.215,0.1715],"mean":0.1972,"sd":0.0228,"resolved":true},"bestHeldOut":{"diffs":[-0.1826,-0.2077,-0.1579],"mean":-0.1828,"sd":0.0249,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.1919,"copyAccuracy":0.1779,"agreement":0.6553},{"step":250,"heldOut":0.9069,"copyAccuracy":0.1897,"agreement":0.7782},{"step":500,"heldOut":0.8027,"copyAccuracy":0.6383,"agreement":0.8462},{"step":1000,"heldOut":0.7274,"copyAccuracy":0.9348,"agreement":0.916},{"step":1500,"heldOut":0.7253,"copyAccuracy":0.9368,"agreement":0.9344},{"step":2000,"heldOut":0.7309,"copyAccuracy":0.9368,"agreement":0.9403}]},"oracle1":{"arm":"oracle1","kind":"oracle","T":1,"seeds":[1,2,3],"heldOut":{"values":[0.7125,0.7401,0.7161],"mean":0.7229,"sd":0.015,"min":0.7125,"max":0.7401},"excess":{"values":[0.054,0.0815,0.0576],"mean":0.0643,"sd":0.015,"min":0.054,"max":0.0815},"copyAccuracy":{"values":[0.9289,0.919,0.9289],"mean":0.9256,"sd":0.0057,"min":0.919,"max":0.9289},"agreement":{"values":[0.8286,0.8631,0.8327],"mean":0.8415,"sd":0.0189,"min":0.8286,"max":0.8631},"bestHeldOut":{"values":[0.7125,0.7244,0.7069],"mean":0.7146,"sd":0.009,"min":0.7069,"max":0.7244},"bestStep":[2000,1500,1000],"vsHard":{"heldOut":{"diffs":[-0.547,-0.6518,-0.459],"mean":-0.5526,"sd":0.0965,"resolved":true},"copyAccuracy":{"diffs":[0.1067,0.1008,0.1028],"mean":0.1034,"sd":0.003,"resolved":true},"agreement":{"diffs":[0.0934,0.1462,0.0728],"mean":0.1041,"sd":0.0379,"resolved":true},"bestHeldOut":{"diffs":[-0.1954,-0.2011,-0.1733],"mean":-0.1899,"sd":0.0147,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":1.002,"copyAccuracy":0.1601,"agreement":0.7344},{"step":250,"heldOut":0.8746,"copyAccuracy":0.2569,"agreement":0.7607},{"step":500,"heldOut":0.7677,"copyAccuracy":0.834,"agreement":0.8302},{"step":1000,"heldOut":0.7453,"copyAccuracy":0.8775,"agreement":0.8166},{"step":1500,"heldOut":0.7189,"copyAccuracy":0.9249,"agreement":0.8174},{"step":2000,"heldOut":0.7125,"copyAccuracy":0.9289,"agreement":0.8286}]}},"verdicts":["32 rows, soft2: held-out loss 1.6371 (sd 0.1362) against hard labels' 4.801; lower by 3.1639 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","32 rows, soft4: held-out loss 1.575 (sd 0.1815) against hard labels' 4.801; lower by 3.226 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","32 rows, mix2: held-out loss 2.0688 (sd 0.037) against hard labels' 4.801; lower by 2.7322 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","32 rows, argmax: held-out loss 7.3695 (sd 0.7773) against hard labels' 4.801; higher by 2.5685 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","32 rows, oracle1: held-out loss 1.8008 (sd 0.1335) against hard labels' 4.801; lower by 3.0002 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","256 rows, soft2: held-out loss 0.7274 (sd 0.0085) against hard labels' 1.2755; lower by 0.5481 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","256 rows, argmax: held-out loss 5.5363 (sd 0.2365) against hard labels' 1.2755; higher by 4.2608 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors).","256 rows, oracle1: held-out loss 0.7229 (sd 0.015) against hard labels' 1.2755; lower by 0.5526 nats on average over 3 paired seeds, resolved (every seed agrees in sign and the mean is over two standard errors)."]},"controls":{"about":"Controls that must fail. untrainedTeacher: soft targets at T 2 from a teacher with the same config and seed but no training (its shares are near uniform), full data, three seeds; it must not teach past word counts (unigramLoss). gradCheck: central finite differences of the soft-target loss (T 2, alpha 0.7) for every entry of all 29 tensors of a small float64 model, worst tensor-level relative error; broken is the same check on a backward pass that drops the 1/T of the softened softmax, which the check must catch.","unigramLoss":3.2725,"untrainedTeacher":{"arm":"untrained2","kind":"soft","T":2,"untrained":true,"seeds":[1,2,3],"heldOut":{"values":[3.8394,3.8389,3.8391],"mean":3.8391,"sd":0.0002,"min":3.8389,"max":3.8394},"excess":{"values":[3.1808,3.1804,3.1806],"mean":3.1806,"sd":0.0002,"min":3.1804,"max":3.1808},"copyAccuracy":{"values":[0.0751,0.0751,0.0731],"mean":0.0744,"sd":0.0011,"min":0.0731,"max":0.0751},"agreement":{"values":[0.0213,0.0234,0.0212],"mean":0.022,"sd":0.0013,"min":0.0212,"max":0.0234},"bestHeldOut":{"values":[3.834,3.8294,3.8384],"mean":3.8339,"sd":0.0045,"min":3.8294,"max":3.8384},"bestStep":[0,0,250],"vsHard":{"heldOut":{"diffs":[3.1444,3.0649,3.1442],"mean":3.1179,"sd":0.0459,"resolved":true},"copyAccuracy":{"diffs":[-0.8854,-0.4921,-0.9051],"mean":-0.7609,"sd":0.233,"resolved":true},"agreement":{"diffs":[-0.8167,-0.7625,-0.8494],"mean":-0.8096,"sd":0.0439,"resolved":true},"bestHeldOut":{"diffs":[3.1391,3.0553,3.1436],"mean":3.1127,"sd":0.0497,"resolved":true}},"curveSeed1":[{"step":0,"heldOut":3.834,"copyAccuracy":0.0316,"agreement":0.0094},{"step":100,"heldOut":3.8385,"copyAccuracy":0.0889,"agreement":0.0254},{"step":250,"heldOut":3.839,"copyAccuracy":0.0632,"agreement":0.0225},{"step":500,"heldOut":3.839,"copyAccuracy":0.0692,"agreement":0.0225},{"step":1000,"heldOut":3.8389,"copyAccuracy":0.0731,"agreement":0.0213},{"step":1500,"heldOut":3.8392,"copyAccuracy":0.0692,"agreement":0.0207},{"step":2000,"heldOut":3.8394,"copyAccuracy":0.0751,"agreement":0.0213}]},"gradCheck":{"ok":5.42e-9,"broken":0.4714,"tensors":29}},"speed":{"about":"Milliseconds per training step of the main student in Node (V8), one process, nothing else running, 40 steps after 5 warm-up steps: hard labels, soft targets (includes the teacher's forward pass) and oracle targets (includes the generator's exact odds for the batch). inPool is the mean per-step time inside the fill's runs, with 4 processes at once. A browser lab running 40 ms slices fits about 40 / ms steps per slice.","msPerStep":{"hard":2.8602,"soft":9.5847,"oracle":3.9743},"inPool":{"hard":3.527,"soft2":10.7177,"oracle1":4.6812},"stepsPer40ms":{"hard":13,"soft":4,"oracle":10}},"film":{"about":"Share arrays for the film's wheels: for each context, 46 shares in vocabulary order (vocab). teacher.T1, T2 and T4 are the teacher's shares softened at that temperature; generator is the corpus generator's exact odds; studentHard and studentSoft are the main student (d16x1, seed 1, full data, 2000 steps) after training on hard labels and on the teacher's soft targets at T 2 (the main temperature, the middle of the sweep; the sweep does not single out one temperature, see temperature.vsT2).","vocab":[".","?","the","a","who","found","put","cone","on","was","tallow","pylon","rerun","crate","shell","lamp","rope","boat","map","card","kettle","net","bell","checked","carried","mended","painted","counted","dropped","wet","old","cold","bright","empty","red","lit","kept","knocked","twice","watched","stood","dock","rang","tide","came","in"],"mainT":2,"contexts":[{"words":["tallow","found","a"],"teacher":{"T1":[0.0002,0.00000783,0.0002,0.0000138,0.0000885,0.0000397,0.000027,0.0000261,0.0001,0.0000453,0.0001,0.0001,0.0001,0.1189,0.0847,0.0931,0.0689,0.0837,0.126,0.0985,0.1428,0.093,0.086,0.0000903,0.0000464,0.0000396,0.0000284,0.0000364,0.0000881,0.0000317,0.0000303,0.0000414,0.0000347,0.0000434,0.0000458,0.0000561,0.0000611,0.0001,0.0000796,0.0000702,0.0000796,0.0000244,0.0000654,0.0024,1.62e-7,0.00000634],"T2":[0.0036,0.0008,0.0046,0.0011,0.0027,0.0018,0.0015,0.0015,0.0031,0.002,0.003,0.0031,0.003,0.1002,0.0845,0.0886,0.0762,0.084,0.1031,0.0911,0.1097,0.0885,0.0852,0.0028,0.002,0.0018,0.0015,0.0018,0.0027,0.0016,0.0016,0.0019,0.0017,0.0019,0.002,0.0022,0.0023,0.003,0.0026,0.0024,0.0026,0.0014,0.0023,0.0141,0.0001,0.0007],"T4":[0.0128,0.0061,0.0144,0.007,0.0111,0.0091,0.0082,0.0082,0.0117,0.0094,0.0116,0.0118,0.0116,0.0672,0.0617,0.0632,0.0586,0.0615,0.0682,0.0641,0.0703,0.0632,0.062,0.0112,0.0094,0.0091,0.0084,0.0089,0.0111,0.0086,0.0085,0.0092,0.0088,0.0093,0.0094,0.0099,0.0101,0.0117,0.0108,0.0105,0.0108,0.008,0.0103,0.0252,0.0023,0.0057]},"generator":[0,0,0,0,0,0,0,0,0,0,0,0,0,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"studentHard":[0.00000431,0.0000232,0.0003,0.0000241,0.0000118,0.0001,0.0004,0.0000197,0.0000495,0.00000266,0.0002,0.0000392,0.0001,0.1032,0.1212,0.1395,0.0683,0.0976,0.0879,0.1092,0.1095,0.0816,0.077,0.0000925,0.000013,0.0001,0.0000344,0.0001,0.0000369,0.0000175,0.0000163,0.0000224,0.00000528,0.0000174,0.0000174,0.0000664,0.000084,0.0001,0.0000609,0.0002,0.0002,0.0000847,0.00000384,0.0024,2.88e-8,0.0000401],"studentSoft":[0.0002,0.00000782,0.0003,0.0000079,0.0002,0.0000416,0.000021,0.0000367,0.0001,0.000051,0.0001,0.0001,0.0000966,0.113,0.0762,0.0834,0.0628,0.0679,0.1816,0.0913,0.1341,0.0918,0.0914,0.0000594,0.0000379,0.0000568,0.0000348,0.0000306,0.0000816,0.0000365,0.0000282,0.0000365,0.0000329,0.0000389,0.0000397,0.000063,0.0000829,0.0000651,0.000057,0.0001,0.0001,0.0000407,0.0000721,0.0041,3.28e-7,0.0000148]},{"words":["rerun","knocked"],"teacher":{"T1":[0.00000501,0.00000514,0.0000202,0.0002,0.0000643,0.0000347,0.00000252,0.0000345,0.4547,0.0000915,0.0000272,0.0000737,0.000055,0.0000543,0.0000186,0.0000447,0.000013,0.0000578,0.0000396,0.0000425,0.00000756,0.0001,0.0000101,0.0000566,0.0000172,0.00000999,0.0000149,0.0000251,0.0000136,0.0001,0.0000611,0.0000956,0.00000753,0.0001,0.000078,0.0000894,0.0000471,0.0000603,0.5431,1.73e-7,2.38e-7,0.0003,0.0000344,0.0000099,0.0000161,0.00000253],"T2":[0.0013,0.0013,0.0027,0.0092,0.0048,0.0035,0.0009,0.0035,0.401,0.0057,0.0031,0.0051,0.0044,0.0044,0.0026,0.004,0.0021,0.0045,0.0037,0.0039,0.0016,0.006,0.0019,0.0045,0.0025,0.0019,0.0023,0.003,0.0022,0.0063,0.0046,0.0058,0.0016,0.0063,0.0053,0.0056,0.0041,0.0046,0.4382,0.0002,0.0003,0.0106,0.0035,0.0019,0.0024,0.0009],"T4":[0.0095,0.0096,0.0135,0.0251,0.018,0.0154,0.008,0.0154,0.1653,0.0197,0.0145,0.0187,0.0173,0.0173,0.0132,0.0165,0.0121,0.0176,0.016,0.0163,0.0106,0.0202,0.0113,0.0175,0.013,0.0113,0.0125,0.0142,0.0122,0.0208,0.0178,0.0199,0.0105,0.0207,0.0189,0.0196,0.0167,0.0177,0.1728,0.0041,0.0044,0.0269,0.0154,0.0113,0.0128,0.008]},"generator":[0,0,0,0,0,0,0,0,0.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.5,0,0,0,0,0,0,0],"studentHard":[0.00000321,0.0000194,1.55e-7,0.0000575,2.02e-7,0.0000121,0.0000314,0.000012,0.4122,0.0000101,1.66e-7,3.93e-8,0.0000968,0.0000176,0.0000597,0.0000419,0.0000214,0.0004,0.00000519,0.0000135,0.00000859,0.0005,0.00000858,7.79e-7,1.12e-7,0.0008,0.0003,2.56e-7,0.0012,9.98e-7,5.92e-7,8.41e-7,3.38e-7,6.28e-7,6.13e-7,2.75e-7,2.19e-7,0.0000171,0.5836,0.000052,0.0000251,0.00000866,0.0000451,0.0004,0.000048,9.03e-9],"studentSoft":[0.00000318,7.26e-7,0.00000372,0.0001,0.0000365,0.00002,0.0000021,0.0000211,0.3974,0.0000886,0.0000228,0.0001,0.0000854,0.0001,0.0000175,0.0001,0.00000649,0.0001,0.0000372,0.0001,0.0000104,0.0003,0.0000144,0.0000631,0.0000148,0.00000245,0.00000718,0.0000265,0.0000213,0.0002,0.0001,0.0002,0.0000184,0.0002,0.0001,0.0000818,0.0000298,0.0000211,0.5989,7.16e-9,7.48e-9,0.0013,0.0000133,0.00000828,0.00000654,0.00000359]}]},"runSeconds":{"full.d16x1.hard.2":7.5,"full.d16x1.hard.1":7.6,"full.d16x1.hard.3":7.6,"full.d16x1.argmax.1":18.3,"full.d16x1.argmax.2":18.5,"full.d16x1.argmax.3":18.5,"full.d16x1.soft1.1":20.2,"full.d16x1.soft1.2":20.5,"full.d16x1.soft2.1":21.4,"full.d16x1.soft1.3":21.5,"full.d16x1.soft2.2":21.8,"full.d16x1.soft2.3":22.8,"full.d16x1.soft4.1":21.7,"full.d16x1.soft4.2":21.9,"full.d16x1.soft4.3":21.5,"full.d16x1.soft8.1":20.1,"full.d16x1.soft8.2":19.9,"full.d16x1.soft8.3":20,"full.d16x1.mix2.1":19.9,"full.d16x1.oracle1.1":9.9,"full.d16x1.oracle1.2":9.9,"full.d16x1.mix2.2":20.2,"full.d16x1.mix2.3":20.1,"full.d16x1.oracle1.3":10,"full.d16x1.oracle2.1":10.1,"full.d16x1.oracle2.2":10,"full.d16x1.oracle2.3":10.1,"pool32.d16x1.hard.1":8.1,"full.d16x1.untrained2.1":19.2,"full.d16x1.untrained2.2":19.2,"full.d16x1.untrained2.3":19.3,"pool32.d16x1.hard.2":8.5,"pool32.d16x1.hard.3":8.5,"pool32.d16x1.argmax.1":20.4,"pool32.d16x1.argmax.2":20.1,"pool32.d16x1.argmax.3":20.4,"pool32.d16x1.soft1.1":21.9,"pool32.d16x1.soft1.2":22,"pool32.d16x1.soft1.3":21.9,"pool32.d16x1.soft2.1":21.7,"pool32.d16x1.soft2.2":21.4,"pool32.d16x1.soft2.3":20.2,"pool32.d16x1.soft4.1":20.2,"pool32.d16x1.soft4.2":20.2,"pool32.d16x1.soft4.3":20.2,"pool32.d16x1.soft8.1":19.9,"pool32.d16x1.soft8.2":20.1,"pool32.d16x1.soft8.3":20,"pool32.d16x1.mix2.1":20,"pool32.d16x1.oracle1.1":9.9,"pool32.d16x1.oracle1.2":9.8,"pool32.d16x1.mix2.2":19.9,"pool32.d16x1.mix2.3":20.1,"pool32.d16x1.oracle1.3":9.6,"pool32.d16x1.oracle2.1":10,"pool32.d16x1.oracle2.2":9.9,"pool32.d16x1.oracle2.3":10,"pool256.d16x1.hard.1":7.6,"pool256.d16x1.hard.2":7.7,"pool256.d16x1.hard.3":7.8,"pool256.d16x1.argmax.1":18.2,"pool256.d16x1.argmax.2":18.4,"pool256.d16x1.argmax.3":18.5,"pool256.d16x1.soft2.1":20,"pool256.d16x1.oracle1.1":10.4,"pool256.d16x1.oracle1.2":10.4,"pool256.d16x1.soft2.2":20.6,"pool256.d16x1.soft2.3":20.9,"pool256.d16x1.oracle1.3":10.5,"full.d16x2.hard.1":14.4,"full.d16x2.hard.2":14,"full.d16x2.hard.3":13.9,"pool32.d16x2.hard.1":14.3,"full.d16x2.soft2.1":27,"full.d16x2.soft2.2":27,"full.d16x2.soft2.3":26.8,"pool32.d16x2.hard.2":14,"pool32.d16x2.hard.3":14.1,"pool32.d16x2.soft2.1":26.6,"pool32.d16x2.soft2.2":26.5,"pool32.d16x2.soft2.3":26.1},"seconds":356.5}