{"about":"Lesson L9, next-word prediction and the loss. Computed by tools/school/fill_nextword.mjs from system/school/nextword.mjs and the attention lab's trained model. Do not edit by hand. Losses are in nats (natural log).","words":[".","?","the","a","who","found","put","cone","on","was","tallow","pylon","rerun","crate","shell","lamp","rope","boat","map","card","kettle","net","bell","checked","carried","mended","painted","counted","dropped","wet","old","cold","bright","empty","red","lit","kept","knocked","twice","watched","stood","dock","rang","tide","came","in"],"modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","model":{"seed":7,"dataSeed":11,"steps":2000,"batch":16,"ctx":24,"vocab":46,"nParams":20864},"worked":{"about":"Four words with scores 2, 1, 0 and -1; the word that came is the first. Softmax: exp of each score over the sum of the four. Loss: minus the natural log of the right word's probability. Gradient check: p minus one-hot against central finite differences (step 1e-5); the control drops the minus one and must fail. Error is max |a - f| / max(|a| + |f|, 1e-4). shiftInvariant: adding 100 to every score leaves the probabilities unchanged (max relative difference). A four-word example shows the arithmetic; it says nothing about the trained model.","words":["lamp","map","net","bell"],"logits":[2,1,0,-1],"answer":"lamp","exp":[7.38906,2.71828,1,0.36788],"probs":[0.64391,0.23688,0.08714,0.03206],"loss":0.44019,"perplexity":1.553,"grad":[-0.35609,0.23688,0.08714,0.03206],"finiteDiff":[-0.35609,0.23688,0.08714,0.03206],"gradError":1.0947055634296893e-10,"brokenGrad":[0.64391,0.23688,0.08714,0.03206],"brokenError":1,"shiftInvariant":0,"passes":true,"controlFails":true},"torn":{"about":"Three contexts where the next word is genuinely open. model: the trained model's eight most likely next words after the last word. truth: the corpus generator's exact odds there (parsed with its own rules). entropy in nats; 'between' is exp(entropy), the number of equally likely words that would be as uncertain. Three contexts show the idea; held-out scores measure it. modelDist and truthDist: all 46 shares in vocabulary order.","rows":[{"words":["pylon"],"model":[{"word":"found","p":0.498},{"word":"painted","p":0.0868},{"word":"stood","p":0.0855},{"word":"watched","p":0.0851},{"word":"mended","p":0.0733},{"word":"dropped","p":0.0557},{"word":"checked","p":0.0378},{"word":"carried","p":0.0366}],"truth":[{"word":"found","p":0.4211},{"word":"watched","p":0.0789},{"word":"stood","p":0.0789},{"word":"checked","p":0.0702},{"word":"carried","p":0.0702},{"word":"mended","p":0.0702},{"word":"painted","p":0.0702},{"word":"counted","p":0.0702}],"modelDist":[0.0001,0,0,0,0,0.498,0.0035,0,0,0,0.0001,0,0.0002,0,0,0,0,0,0,0,0,0,0.0002,0.0378,0.0366,0.0733,0.0868,0.0323,0.0557,0,0,0,0,0,0,0.001,0.0011,0.0022,0,0.0851,0.0855,0,0.0001,0,0,0],"truthDist":[0,0,0,0,0,0.4211,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.0702,0.0702,0.0702,0.0702,0.0702,0.0702,0,0,0,0,0,0,0,0,0,0,0.0789,0.0789,0,0,0,0,0],"modelEntropy":1.7451,"truthEntropy":1.8837,"modelBetween":5.73,"truthBetween":6.58},{"words":["tallow","found","a"],"model":[{"word":"kettle","p":0.1428},{"word":"map","p":0.126},{"word":"crate","p":0.1189},{"word":"card","p":0.0985},{"word":"lamp","p":0.0931},{"word":"net","p":0.093},{"word":"bell","p":0.086},{"word":"shell","p":0.0847}],"truth":[{"word":"crate","p":0.1},{"word":"shell","p":0.1},{"word":"lamp","p":0.1},{"word":"rope","p":0.1},{"word":"boat","p":0.1},{"word":"map","p":0.1},{"word":"card","p":0.1},{"word":"kettle","p":0.1}],"modelDist":[0.0002,0,0.0002,0,0.0001,0,0,0,0.0001,0,0.0001,0.0001,0.0001,0.1189,0.0847,0.0931,0.0689,0.0837,0.126,0.0985,0.1428,0.093,0.086,0.0001,0,0,0,0,0.0001,0,0,0,0,0,0,0.0001,0.0001,0.0001,0.0001,0.0001,0.0001,0,0.0001,0.0024,0,0],"truthDist":[0,0,0,0,0,0,0,0,0,0,0,0,0,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"modelEntropy":2.3091,"truthEntropy":2.3026,"modelBetween":10.07,"truthBetween":10},{"words":["rerun","knocked"],"model":[{"word":"twice","p":0.5431},{"word":"on","p":0.4547}],"truth":[{"word":"on","p":0.5},{"word":"twice","p":0.5}],"modelDist":[0,0,0,0.0002,0.0001,0,0,0,0.4547,0.0001,0,0.0001,0.0001,0.0001,0,0,0,0.0001,0,0,0,0.0001,0,0.0001,0,0,0,0,0,0.0001,0.0001,0.0001,0,0.0001,0.0001,0.0001,0,0.0001,0.5431,0,0,0.0003,0,0,0,0],"truthDist":[0,0,0,0,0,0,0,0,0.5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.5,0,0,0,0,0,0,0],"modelEntropy":0.7108,"truthEntropy":0.6931,"modelBetween":2.04,"truthBetween":2}]},"probes":{"about":"The attention lab's six probe sentences. top5: the model's five most likely next words after the last word, with probabilities. loss: minus the log probability of the expected answer there. meanLossInside: the mean loss over every earlier position of the probe whose next word the probe contains. One sentence each: a probe shows a behaviour, it does not measure a rate.","rows":[{"words":["rerun","found","a","kettle",".","pylon","put","a","cone","on","the"],"query":10,"answer":"kettle","kind":"copy","top5":[{"word":"kettle","p":0.996},{"word":"map","p":0.0013},{"word":"shell","p":0.0011},{"word":"crate","p":0.0006},{"word":"bell","p":0.0003}],"pAnswer":0.996,"loss":0.004,"rank":0,"meanLossInside":0.3579,"positionsInside":10,"note":"Copy: the cone goes on the thing Rerun found seven words back."},{"words":["tallow","found","a","map",".","the","bell","rang",".","pylon","put","a","cone","on","the"],"query":14,"answer":"map","kind":"copy","top5":[{"word":"map","p":0.9981},{"word":"tallow","p":0.0004},{"word":"net","p":0.0004},{"word":"kettle","p":0.0004},{"word":"on","p":0.0003}],"pAnswer":0.9981,"loss":0.0019,"rank":0,"meanLossInside":0.4745,"positionsInside":14,"note":"Copy past a distractor: bell is the nearer thing, but map was found."},{"words":["pylon","checked","the","crate",".","the","bell","rang",".","who","checked","the","crate","?"],"query":13,"answer":"pylon","kind":"copy","top5":[{"word":"pylon","p":0.9944},{"word":"rerun","p":0.0033},{"word":"tallow","p":0.0016},{"word":"the","p":0.0006},{"word":"bright","p":0}],"pAnswer":0.9944,"loss":0.0056,"rank":0,"meanLossInside":0.7475,"positionsInside":13,"note":"Who did it: the answer is the name at the start of the matching sentence."},{"words":["tallow","mended","the","net",".","rerun","knocked","twice",".","who","mended","the"],"query":11,"answer":"net","kind":"copy","top5":[{"word":"net","p":0.9965},{"word":"crate","p":0.0014},{"word":"map","p":0.0006},{"word":"rerun","p":0.0005},{"word":"card","p":0.0004}],"pAnswer":0.9965,"loss":0.0035,"rank":0,"meanLossInside":0.8186,"positionsInside":11,"note":"Which thing: the question repeats the verb, so the object must be looked up."},{"words":["rerun","painted","the","boat",".","pylon","watched","the","dock",".","the","rope","was","wet",".","who","painted","the","boat","?"],"query":19,"answer":"rerun","kind":"copy","top5":[{"word":"rerun","p":0.9978},{"word":"tallow","p":0.0005},{"word":"the","p":0.0002},{"word":"pylon","p":0.0002},{"word":"card","p":0.0002}],"pAnswer":0.9978,"loss":0.0022,"rank":0,"meanLossInside":0.7033,"positionsInside":19,"note":"Long distance: another guide is named in between, and the answer is 19 words back."},{"words":["pylon","watched","the","dock",".","tallow","lit","the"],"query":7,"answer":"lamp","kind":"local","top5":[{"word":"lamp","p":0.9998},{"word":"rope","p":0.0001},{"word":"card","p":0},{"word":"bright","p":0},{"word":"put","p":0}],"pAnswer":0.9998,"loss":0.0002,"rank":0,"meanLossInside":0.7733,"positionsInside":7,"note":"Local: after 'lit the' the next word is lamp; the previous word is enough."}],"untrained":[{"answer":"kettle","pAnswer":0.0209,"loss":3.8682,"top1":"checked"},{"answer":"map","pAnswer":0.0197,"loss":3.9258,"top1":"tallow"},{"answer":"pylon","pAnswer":0.0191,"loss":3.9578,"top1":"in"},{"answer":"net","pAnswer":0.0221,"loss":3.813,"top1":"the"},{"answer":"rerun","pAnswer":0.017,"loss":4.074,"top1":"."},{"answer":"lamp","pAnswer":0.0242,"loss":3.7227,"top1":"checked"}]},"held":{"about":"256 held-out rows of 24 next-word predictions each (6144 positions, generator seed 1592593325, a separate stream from training; the grammar is small, so every held-out sentence also occurs somewhere in training, and 6 of the 256 rows equal a training row word for word: the rows are new, the sentences are not). loss: mean cross-entropy in nats; perplexity: exp(loss); top1 / top5: share of positions where the word that came is the model's first choice / among its five most likely. 'copy' positions need a word from earlier in the row, 'local' positions do not (the corpus labels them). Controls: 'untrained' is the same network at its starting weights (seed 7), 'uniform' gives every word 1/46. The n-gram rows count the exact 768000 training targets the model saw (2000 batches of 16 rows, data seed 11): unigram counts words, bigram counts word after previous word, trigram counts word after the previous two (bigram at a row's first position). Add-one smoothing unless the key says alpha 0.01. 'generator' is the floor: the corpus generator's own next-word probabilities, computed exactly by parsing each row with the generator's rules and odds (corpus-grammar.mjs); no model can beat its loss on average, because the text is random where the generator rolls dice. Top-1 and top-5 count tied words fractionally (random tie order), so the uniform guess scores 1/46 and 5/46. One held-out set: these are point estimates; with 6144 positions a share near 0.7 has a binomial standard error near 0.006. A lower loss here does not show the model would do well on text from any other source.","positions":6144,"nCopy":506,"nLocal":5638,"trainingTokens":768000,"scores":{"trained":{"all":{"n":6144,"loss":0.6874,"perplexity":1.9886,"top1":0.7243,"top5":0.9284},"copy":{"n":506,"loss":0.056,"perplexity":1.0576,"top1":0.9921,"top5":1},"local":{"n":5638,"loss":0.7441,"perplexity":2.1045,"top1":0.7002,"top5":0.922}},"untrained":{"all":{"n":6144,"loss":3.8414,"perplexity":46.5909,"top1":0.0244,"top5":0.1237},"copy":{"n":506,"loss":3.8297,"perplexity":46.0509,"top1":0.0692,"top5":0.1403},"local":{"n":5638,"loss":3.8425,"perplexity":46.6396,"top1":0.0204,"top5":0.1222}},"uniform":{"all":{"n":6144,"loss":3.8286,"perplexity":46,"top1":0.0217,"top5":0.1087},"copy":{"n":506,"loss":3.8286,"perplexity":46,"top1":0.0217,"top5":0.1087},"local":{"n":5638,"loss":3.8286,"perplexity":46,"top1":0.0217,"top5":0.1087}},"unigram":{"all":{"n":6144,"loss":3.2721,"perplexity":26.3654,"top1":0.1686,"top5":0.4583},"copy":{"n":506,"loss":4.0973,"perplexity":60.1766,"top1":0,"top5":0.0731},"local":{"n":5638,"loss":3.198,"perplexity":24.4833,"top1":0.1838,"top5":0.4929}},"bigram":{"all":{"n":6144,"loss":1.2806,"perplexity":3.5989,"top1":0.5065,"top5":0.8854},"copy":{"n":506,"loss":1.9615,"perplexity":7.1101,"top1":0.1858,"top5":0.6719},"local":{"n":5638,"loss":1.2195,"perplexity":3.3856,"top1":0.5353,"top5":0.9046}},"trigram":{"all":{"n":6144,"loss":1.0693,"perplexity":2.9133,"top1":0.5632,"top5":0.8988},"copy":{"n":506,"loss":1.9265,"perplexity":6.8654,"top1":0.1907,"top5":0.6779},"local":{"n":5638,"loss":0.9924,"perplexity":2.6976,"top1":0.5967,"top5":0.9187}},"generator":{"all":{"n":6144,"loss":0.6585,"perplexity":1.932,"top1":0.732,"top5":0.9313},"copy":{"n":506,"loss":0,"perplexity":1,"top1":1,"top5":1},"local":{"n":5638,"loss":0.7176,"perplexity":2.0496,"top1":0.708,"top5":0.9251}},"bigramAlpha001":{"all":{"n":6144,"loss":1.278,"perplexity":3.5895,"top1":0.5065,"top5":0.8854},"copy":{"n":506,"loss":1.9601,"perplexity":7.0998,"top1":0.1858,"top5":0.6719},"local":{"n":5638,"loss":1.2168,"perplexity":3.3764,"top1":0.5353,"top5":0.9046}},"trigramAlpha001":{"all":{"n":6144,"loss":1.0618,"perplexity":2.8915,"top1":0.5632,"top5":0.8988},"copy":{"n":506,"loss":1.9175,"perplexity":6.804,"top1":0.1907,"top5":0.6779},"local":{"n":5638,"loss":0.985,"perplexity":2.6778,"top1":0.5967,"top5":0.9187}}},"check":{"about":"Consistency checks: the lab's own evaluate() loss on the same rows, the unigram loss from corpus.mjs's unigramCrossEntropy on the same tokens (both add-one), and log 46 for the uniform guess.","evaluateLoss":0.687407,"thisLoss":0.6874,"unigramCorpusMjs":3.272053,"uniformLog46":3.828641},"lossByPosition":{"about":"Mean loss at each position of a row (0 = first word of the row, the least context).","trained":[1.9433,0.042,1.7508,0.0751,1.2314,1.2335,0.3317,0.5118,0.3292,0.6838,0.6676,0.4978,0.5591,0.4142,0.3579,0.4756,0.4211,0.8132,0.5865,0.5505,0.4792,0.7404,0.9382,0.8638],"bigram":[2.2532,0.0598,2.3088,0.6287,1.4386,1.8002,0.9107,1.2722,0.8751,1.1485,1.7129,0.927,1.3025,1.179,1.0842,1.4436,1.2023,1.405,1.1712,1.3178,1.2064,1.3224,1.4632,1.3019],"trigram":[2.2532,0.0688,1.7517,0.491,1.3885,1.6178,0.7623,0.7981,0.6722,1.0266,1.3842,0.8018,1.0542,0.9313,0.9037,1.034,0.9988,1.2759,1.0198,0.9836,0.894,1.1392,1.2698,1.1426],"generator":[1.8767,0.0433,1.7449,0.0715,1.1914,1.1589,0.3169,0.491,0.3158,0.6572,0.648,0.462,0.5453,0.4067,0.3477,0.4674,0.412,0.7863,0.5219,0.5332,0.4691,0.7101,0.8469,0.7808]},"excessOverGenerator":{"all":0.0289,"copy":0.056,"local":0.0265}},"curve":{"about":"Lifted from model.json, not recomputed. 'train' is the mean loss of the previous 25 training batches, each measured before its update; 'heldOut' is the loss on the 256 held-out rows after the update; perplexity is exp(heldOut). Step 0 is the untrained network. One training run (model seed 7, data seed 11): the curve's shape for other seeds is not shown here.","steps":2000,"every":25,"batch":16,"lr":0.01,"points":[{"step":0,"train":null,"heldOut":3.8414,"perplexity":46.591,"copyAccuracy":0.0692,"localAccuracy":0.0204},{"step":25,"train":2.9816,"heldOut":1.7252,"perplexity":5.614,"copyAccuracy":0.1522,"localAccuracy":0.5488},{"step":50,"train":1.3223,"heldOut":1.1656,"perplexity":3.208,"copyAccuracy":0.2134,"localAccuracy":0.6078},{"step":75,"train":1.0861,"heldOut":1.0454,"perplexity":2.845,"copyAccuracy":0.1976,"localAccuracy":0.6483},{"step":100,"train":0.9942,"heldOut":0.9654,"perplexity":2.626,"copyAccuracy":0.1561,"localAccuracy":0.6744},{"step":125,"train":0.9327,"heldOut":0.9599,"perplexity":2.611,"copyAccuracy":0.1798,"localAccuracy":0.6823},{"step":150,"train":0.8998,"heldOut":0.9239,"perplexity":2.519,"copyAccuracy":0.2016,"localAccuracy":0.6907},{"step":175,"train":0.8934,"heldOut":0.9101,"perplexity":2.485,"copyAccuracy":0.166,"localAccuracy":0.693},{"step":200,"train":0.8844,"heldOut":0.8812,"perplexity":2.414,"copyAccuracy":0.2134,"localAccuracy":0.6976},{"step":225,"train":0.8422,"heldOut":0.8643,"perplexity":2.373,"copyAccuracy":0.2431,"localAccuracy":0.7001},{"step":250,"train":0.8468,"heldOut":0.878,"perplexity":2.406,"copyAccuracy":0.2826,"localAccuracy":0.6965},{"step":275,"train":0.8422,"heldOut":0.8454,"perplexity":2.329,"copyAccuracy":0.3024,"localAccuracy":0.7008},{"step":300,"train":0.8362,"heldOut":0.8436,"perplexity":2.325,"copyAccuracy":0.3399,"localAccuracy":0.7026},{"step":325,"train":0.8286,"heldOut":0.8536,"perplexity":2.348,"copyAccuracy":0.3281,"localAccuracy":0.694},{"step":350,"train":0.8238,"heldOut":0.8397,"perplexity":2.316,"copyAccuracy":0.3261,"localAccuracy":0.6969},{"step":375,"train":0.8355,"heldOut":0.8523,"perplexity":2.345,"copyAccuracy":0.3142,"localAccuracy":0.7017},{"step":400,"train":0.8032,"heldOut":0.8456,"perplexity":2.329,"copyAccuracy":0.3221,"localAccuracy":0.6971},{"step":425,"train":0.841,"heldOut":0.8448,"perplexity":2.328,"copyAccuracy":0.3162,"localAccuracy":0.6908},{"step":450,"train":0.8254,"heldOut":0.832,"perplexity":2.298,"copyAccuracy":0.3458,"localAccuracy":0.6997},{"step":475,"train":0.8116,"heldOut":0.824,"perplexity":2.28,"copyAccuracy":0.4229,"localAccuracy":0.7077},{"step":500,"train":0.7949,"heldOut":0.8182,"perplexity":2.266,"copyAccuracy":0.4289,"localAccuracy":0.7057},{"step":525,"train":0.7991,"heldOut":0.8344,"perplexity":2.303,"copyAccuracy":0.4723,"localAccuracy":0.6836},{"step":550,"train":0.7913,"heldOut":0.8073,"perplexity":2.242,"copyAccuracy":0.5119,"localAccuracy":0.6919},{"step":575,"train":0.7913,"heldOut":0.8039,"perplexity":2.234,"copyAccuracy":0.5277,"localAccuracy":0.696},{"step":600,"train":0.7993,"heldOut":0.7937,"perplexity":2.212,"copyAccuracy":0.5751,"localAccuracy":0.702},{"step":625,"train":0.7871,"heldOut":0.7775,"perplexity":2.176,"copyAccuracy":0.6028,"localAccuracy":0.7015},{"step":650,"train":0.7516,"heldOut":0.7728,"perplexity":2.166,"copyAccuracy":0.6601,"localAccuracy":0.6995},{"step":675,"train":0.7525,"heldOut":0.7455,"perplexity":2.107,"copyAccuracy":0.7569,"localAccuracy":0.7022},{"step":700,"train":0.7395,"heldOut":0.7289,"perplexity":2.073,"copyAccuracy":0.9348,"localAccuracy":0.6898},{"step":725,"train":0.6994,"heldOut":0.7114,"perplexity":2.037,"copyAccuracy":0.9763,"localAccuracy":0.6994},{"step":750,"train":0.6929,"heldOut":0.715,"perplexity":2.044,"copyAccuracy":0.9684,"localAccuracy":0.6992},{"step":775,"train":0.6882,"heldOut":0.7218,"perplexity":2.058,"copyAccuracy":0.9684,"localAccuracy":0.702},{"step":800,"train":0.6876,"heldOut":0.7033,"perplexity":2.02,"copyAccuracy":0.9763,"localAccuracy":0.7045},{"step":825,"train":0.6905,"heldOut":0.7058,"perplexity":2.025,"copyAccuracy":0.9822,"localAccuracy":0.7001},{"step":850,"train":0.6853,"heldOut":0.7002,"perplexity":2.014,"copyAccuracy":0.9941,"localAccuracy":0.7018},{"step":875,"train":0.6766,"heldOut":0.7192,"perplexity":2.053,"copyAccuracy":0.9605,"localAccuracy":0.7081},{"step":900,"train":0.6752,"heldOut":0.7048,"perplexity":2.023,"copyAccuracy":0.9921,"localAccuracy":0.6999},{"step":925,"train":0.6712,"heldOut":0.6969,"perplexity":2.008,"copyAccuracy":0.9901,"localAccuracy":0.6994},{"step":950,"train":0.6799,"heldOut":0.7053,"perplexity":2.024,"copyAccuracy":0.9941,"localAccuracy":0.7026},{"step":975,"train":0.6835,"heldOut":0.7108,"perplexity":2.036,"copyAccuracy":0.9901,"localAccuracy":0.6958},{"step":1000,"train":0.6711,"heldOut":0.6981,"perplexity":2.01,"copyAccuracy":0.996,"localAccuracy":0.7013},{"step":1025,"train":0.6718,"heldOut":0.6933,"perplexity":2,"copyAccuracy":0.9941,"localAccuracy":0.7059},{"step":1050,"train":0.6785,"heldOut":0.6871,"perplexity":1.988,"copyAccuracy":1,"localAccuracy":0.7034},{"step":1075,"train":0.6685,"heldOut":0.6913,"perplexity":1.996,"copyAccuracy":0.9921,"localAccuracy":0.7088},{"step":1100,"train":0.663,"heldOut":0.6942,"perplexity":2.002,"copyAccuracy":0.998,"localAccuracy":0.699},{"step":1125,"train":0.6573,"heldOut":0.6939,"perplexity":2.002,"copyAccuracy":0.996,"localAccuracy":0.6937},{"step":1150,"train":0.677,"heldOut":0.7227,"perplexity":2.06,"copyAccuracy":0.9466,"localAccuracy":0.6962},{"step":1175,"train":0.6877,"heldOut":0.7006,"perplexity":2.015,"copyAccuracy":0.9822,"localAccuracy":0.6967},{"step":1200,"train":0.6915,"heldOut":0.6998,"perplexity":2.013,"copyAccuracy":0.9585,"localAccuracy":0.6999},{"step":1225,"train":0.6709,"heldOut":0.6928,"perplexity":1.999,"copyAccuracy":0.9881,"localAccuracy":0.702},{"step":1250,"train":0.6798,"heldOut":0.6964,"perplexity":2.007,"copyAccuracy":0.9941,"localAccuracy":0.701},{"step":1275,"train":0.6682,"heldOut":0.6867,"perplexity":1.987,"copyAccuracy":0.9941,"localAccuracy":0.7045},{"step":1300,"train":0.6593,"heldOut":0.6898,"perplexity":1.993,"copyAccuracy":0.9921,"localAccuracy":0.6971},{"step":1325,"train":0.6622,"heldOut":0.6923,"perplexity":1.998,"copyAccuracy":0.9941,"localAccuracy":0.6967},{"step":1350,"train":0.6736,"heldOut":0.6964,"perplexity":2.007,"copyAccuracy":0.996,"localAccuracy":0.7026},{"step":1375,"train":0.6611,"heldOut":0.6937,"perplexity":2.001,"copyAccuracy":0.996,"localAccuracy":0.6962},{"step":1400,"train":0.6596,"heldOut":0.6821,"perplexity":1.978,"copyAccuracy":0.998,"localAccuracy":0.699},{"step":1425,"train":0.6634,"heldOut":0.6795,"perplexity":1.973,"copyAccuracy":0.996,"localAccuracy":0.7027},{"step":1450,"train":0.6668,"heldOut":0.6836,"perplexity":1.981,"copyAccuracy":0.998,"localAccuracy":0.6969},{"step":1475,"train":0.6606,"heldOut":0.6828,"perplexity":1.979,"copyAccuracy":0.998,"localAccuracy":0.7068},{"step":1500,"train":0.665,"heldOut":0.6944,"perplexity":2.003,"copyAccuracy":0.9881,"localAccuracy":0.7004},{"step":1525,"train":0.6643,"heldOut":0.6946,"perplexity":2.003,"copyAccuracy":0.9881,"localAccuracy":0.6988},{"step":1550,"train":0.6637,"heldOut":0.6917,"perplexity":1.997,"copyAccuracy":0.9941,"localAccuracy":0.7061},{"step":1575,"train":0.6583,"heldOut":0.6895,"perplexity":1.993,"copyAccuracy":1,"localAccuracy":0.6981},{"step":1600,"train":0.66,"heldOut":0.6814,"perplexity":1.977,"copyAccuracy":0.9921,"localAccuracy":0.7054},{"step":1625,"train":0.6716,"heldOut":0.6936,"perplexity":2.001,"copyAccuracy":0.9941,"localAccuracy":0.6985},{"step":1650,"train":0.6793,"heldOut":0.7088,"perplexity":2.032,"copyAccuracy":0.9881,"localAccuracy":0.6967},{"step":1675,"train":0.6832,"heldOut":0.6867,"perplexity":1.987,"copyAccuracy":0.9901,"localAccuracy":0.702},{"step":1700,"train":0.6602,"heldOut":0.6875,"perplexity":1.989,"copyAccuracy":0.9881,"localAccuracy":0.6942},{"step":1725,"train":0.6564,"heldOut":0.6858,"perplexity":1.985,"copyAccuracy":0.998,"localAccuracy":0.7065},{"step":1750,"train":0.659,"heldOut":0.6884,"perplexity":1.991,"copyAccuracy":0.998,"localAccuracy":0.6946},{"step":1775,"train":0.6686,"heldOut":0.6938,"perplexity":2.001,"copyAccuracy":0.9802,"localAccuracy":0.7015},{"step":1800,"train":0.6638,"heldOut":0.6838,"perplexity":1.981,"copyAccuracy":0.9901,"localAccuracy":0.704},{"step":1825,"train":0.6577,"heldOut":0.6893,"perplexity":1.992,"copyAccuracy":0.998,"localAccuracy":0.6997},{"step":1850,"train":0.6666,"heldOut":0.6775,"perplexity":1.969,"copyAccuracy":0.998,"localAccuracy":0.6978},{"step":1875,"train":0.6578,"heldOut":0.6832,"perplexity":1.98,"copyAccuracy":0.998,"localAccuracy":0.7024},{"step":1900,"train":0.6651,"heldOut":0.6863,"perplexity":1.986,"copyAccuracy":0.996,"localAccuracy":0.6983},{"step":1925,"train":0.6695,"heldOut":0.6868,"perplexity":1.987,"copyAccuracy":0.9842,"localAccuracy":0.7065},{"step":1950,"train":0.6679,"heldOut":0.6829,"perplexity":1.98,"copyAccuracy":0.996,"localAccuracy":0.7034},{"step":1975,"train":0.6624,"heldOut":0.6799,"perplexity":1.974,"copyAccuracy":1,"localAccuracy":0.7056},{"step":2000,"train":0.6524,"heldOut":0.6874,"perplexity":1.989,"copyAccuracy":0.9921,"localAccuracy":0.7002}],"modelJsonBaselines":{"unigramLoss":3.272,"copyChance":0.1008}},"seconds":{"counts":0.06,"total":0.51}}