{"about":"Computed by tools/school/fill_tokens.mjs. Do not edit by hand. Byte-pair encoding as in Sennrich, Haddow and Birch (2016), section 3.2: characters plus an end-of-word symbol, merge the most frequent adjacent pair within words, repeat.","method":{"citation":"Sennrich, Haddow, Birch (2016), Neural Machine Translation of Rare Words with Subword Units, ACL, section 3.2 and Algorithm 1.","endOfWord":"</w>","pretokenizer":"a run of letters, a run of digits, or one other non-space character","ties":"equal counts: the pair whose left symbol sorts first (JavaScript string order), then the right symbol","vocabSize":"base symbols plus one per merge, as the paper states; symbolsInUse counts symbols still present in the training words","tokensPerWord":"tokens divided by pre-tokenized words (punctuation marks count as words, as in the attention model)","charsPerToken":"letters, digits and punctuation (no whitespace) divided by tokens; the end-of-word symbol is part of a token, not a character"},"corpus":{"about":"The guides' corpus as text: 3000 episodes from corpus.mjs (seed 7), \".\" and \"?\" attached to the word before; held out: 500 episodes from the held-out seed. BPE learned until every word is one symbol.","sample":["tallow found a bell. rerun knocked on a shell. tallow kept the lamp bright. pylon put a cone on the bell.","rerun carried the shell. tallow kept the lamp bright. rerun knocked twice. who carried the shell? rerun.","tallow dropped the lamp. the shell was wet. the map was empty. who dropped the lamp? tallow.","rerun found a map. the tide came in. pylon put a cone on the map.","rerun counted the crate. who counted the crate? rerun.","pylon found a crate. pylon put a cone on the crate."],"chars":197193,"words":47002,"wordTypes":46,"heldWords":7875,"letterWordTypes":44,"markTypes":2,"base":[".","</w>","?","a","b","c","d","e","f","g","h","i","k","l","m","n","o","p","r","s","t","u","w","y"],"baseSize":24,"totalMerges":136,"firstMerges":[{"a":"e","b":"</w>","made":"e</w>","count":10141},{"a":".","b":"</w>","made":".</w>","count":8115},{"a":"n","b":"</w>","made":"n</w>","count":7033},{"a":"h","b":"e</w>","made":"he</w>","count":6275},{"a":"t","b":"he</w>","made":"the</w>","count":6275},{"a":"d","b":"</w>","made":"d</w>","count":5825},{"a":"o","b":"n</w>","made":"on</w>","count":4903},{"a":"l","b":"l","made":"ll","count":3564},{"a":"t","b":"</w>","made":"t</w>","count":3560},{"a":"e","b":"d</w>","made":"ed</w>","count":3451},{"a":"l","b":"on</w>","made":"lon</w>","count":3060},{"a":"p","b":"y","made":"py","count":3060},{"a":"py","b":"lon</w>","made":"pylon</w>","count":3060},{"a":"a","b":"</w>","made":"a</w>","count":2758},{"a":"c","b":"o","made":"co","count":1838},{"a":"e","b":"r","made":"er","count":1833},{"a":"er","b":"u","made":"eru","count":1833},{"a":"eru","b":"n</w>","made":"erun</w>","count":1833},{"a":"r","b":"erun</w>","made":"rerun</w>","count":1833},{"a":"a","b":"ll","made":"all","count":1787}],"allMerges":[["e","</w>",10141],[".","</w>",8115],["n","</w>",7033],["h","e</w>",6275],["t","he</w>",6275],["d","</w>",5825],["o","n</w>",4903],["l","l",3564],["t","</w>",3560],["e","d</w>",3451],["l","on</w>",3060],["p","y",3060],["py","lon</w>",3060],["a","</w>",2758],["c","o",1838],["e","r",1833],["er","u",1833],["eru","n</w>",1833],["r","erun</w>",1833],["a","ll",1787],["all","o",1787],["allo","w",1787],["allow","</w>",1787],["t","allow</w>",1787],["e","ll",1777],["ell","</w>",1777],["p","</w>",1701],["u","n",1690],["c","k",1616],["a","m",1403],["co","n",1242],["con","e</w>",1242],["f","o",1242],["fo","un",1242],["foun","d</w>",1242],["p","u",1242],["pu","t</w>",1242],["?","</w>",1219],["h","o",1219],["ho","</w>",1219],["w","ho</w>",1219],["o","ck",1198],["w","a",1116],["am","p</w>",1106],["l","amp</w>",1106],["a","r",961],["c","ar",961],["b","ell</w>",904],["o","p",902],["r","op",902],["t","ed</w>",892],["h","ell</w>",873],["s","hell</w>",873],["r","a",868],["r","i",868],["k","e",852],["s","</w>",843],["wa","s</w>",843],["e","t</w>",743],["c","h",691],["n","et</w>",602],["d","ock",600],["dock","</w>",600],["k","n",598],["kn","ock",598],["knock","ed</w>",598],["a","p</w>",595],["m","ap</w>",595],["c","ra",573],["cra","t",573],["crat","e</w>",573],["ke","t",571],["ket","t",571],["kett","l",571],["kettl","e</w>",571],["a","t</w>",569],["b","o",569],["bo","at</w>",569],["rop","e</w>",562],["car","d</w>",539],["co","un",448],["coun","ted</w>",448],["b","ri",446],["bri","g",446],["brig","h",446],["brigh","t</w>",446],["a","i",444],["ai","n",444],["ain","ted</w>",444],["p","ainted</w>",444],["car","ri",422],["carri","ed</w>",422],["ch","e",418],["che","ck",418],["check","ed</w>",418],["d","ed</w>",366],["e","n",366],["en","ded</w>",366],["m","ended</w>",366],["d","rop",340],["drop","p",340],["dropp","ed</w>",340],["o","d</w>",327],["o","od</w>",327],["s","t",327],["st","ood</w>",327],["c","e</w>",324],["i","ce</w>",324],["t","w",324],["tw","ice</w>",324],["am","e</w>",297],["c","ame</w>",297],["d","e</w>",297],["i","de</w>",297],["i","n</w>",297],["t","ide</w>",297],["g","</w>",295],["n","g</w>",295],["ra","ng</w>",295],["ke","p",281],["kep","t</w>",281],["i","t</w>",279],["l","it</w>",279],["ch","ed</w>",273],["t","ched</w>",273],["wa","tched</w>",273],["l","d</w>",266],["co","ld</w>",148],["r","ed</w>",142],["w","et</w>",141],["e","m",129],["em","p",129],["emp","t",129],["empt","y",129],["empty","</w>",129],["o","ld</w>",118]],"curveTrain":[{"merges":0,"vocab":24,"tokens":206528,"tokensPerWord":4.394,"charsPerToken":0.7724},{"merges":5,"vocab":29,"tokens":168689,"tokensPerWord":3.589,"charsPerToken":0.9457},{"merges":10,"vocab":34,"tokens":147386,"tokensPerWord":3.1357,"charsPerToken":1.0824},{"merges":15,"vocab":39,"tokens":133610,"tokensPerWord":2.8426,"charsPerToken":1.194},{"merges":20,"vocab":44,"tokens":124491,"tokensPerWord":2.6486,"charsPerToken":1.2814},{"merges":30,"vocab":54,"tokens":107379,"tokensPerWord":2.2846,"charsPerToken":1.4856},{"merges":40,"vocab":64,"tokens":95028,"tokensPerWord":2.0218,"charsPerToken":1.6787},{"merges":50,"vocab":74,"tokens":84653,"tokensPerWord":1.8011,"charsPerToken":1.8845},{"merges":60,"vocab":84,"tokens":76307,"tokensPerWord":1.6235,"charsPerToken":2.0906},{"merges":80,"vocab":104,"tokens":64710,"tokensPerWord":1.3767,"charsPerToken":2.4652},{"merges":100,"vocab":124,"tokens":56352,"tokensPerWord":1.1989,"charsPerToken":2.8309},{"merges":120,"vocab":144,"tokens":50120,"tokensPerWord":1.0663,"charsPerToken":3.1829},{"merges":136,"vocab":160,"tokens":47002,"tokensPerWord":1,"charsPerToken":3.394}],"curveHeld":[{"merges":0,"vocab":24,"tokens":34866,"tokensPerWord":4.4274,"charsPerToken":0.7741},{"merges":5,"vocab":29,"tokens":28481,"tokensPerWord":3.6166,"charsPerToken":0.9477},{"merges":10,"vocab":34,"tokens":24896,"tokensPerWord":3.1614,"charsPerToken":1.0842},{"merges":15,"vocab":39,"tokens":22696,"tokensPerWord":2.882,"charsPerToken":1.1892},{"merges":20,"vocab":44,"tokens":21226,"tokensPerWord":2.6954,"charsPerToken":1.2716},{"merges":30,"vocab":54,"tokens":18205,"tokensPerWord":2.3117,"charsPerToken":1.4826},{"merges":40,"vocab":64,"tokens":16332,"tokensPerWord":2.0739,"charsPerToken":1.6526},{"merges":50,"vocab":74,"tokens":14452,"tokensPerWord":1.8352,"charsPerToken":1.8676},{"merges":60,"vocab":84,"tokens":13077,"tokensPerWord":1.6606,"charsPerToken":2.064},{"merges":80,"vocab":104,"tokens":11155,"tokensPerWord":1.4165,"charsPerToken":2.4196},{"merges":100,"vocab":124,"tokens":9753,"tokensPerWord":1.2385,"charsPerToken":2.7675},{"merges":120,"vocab":144,"tokens":8525,"tokensPerWord":1.0825,"charsPerToken":3.1661},{"merges":136,"vocab":160,"tokens":7875,"tokensPerWord":1,"charsPerToken":3.4274}],"randomControl":{"seeds":10,"train":[{"merges":0,"mean":4.394,"min":4.394,"max":4.394},{"merges":5,"mean":4.2466,"min":4.1104,"max":4.3175},{"merges":10,"mean":4.1278,"min":3.8848,"max":4.2586},{"merges":15,"mean":3.9609,"min":3.7283,"max":4.1536},{"merges":20,"mean":3.8182,"min":3.5538,"max":4.0308},{"merges":30,"mean":3.6113,"min":3.3941,"max":3.8123},{"merges":40,"mean":3.3514,"min":3.0966,"max":3.577},{"merges":50,"mean":3.1539,"min":2.8123,"max":3.345},{"merges":60,"mean":2.9511,"min":2.6327,"max":3.1304},{"merges":80,"mean":2.4846,"min":2.2036,"max":2.8066},{"merges":100,"mean":2.1051,"min":1.8988,"max":2.5005},{"merges":120,"mean":1.7437,"min":1.5103,"max":2.0306},{"merges":136,"mean":1.4823,"min":1.319,"max":1.6703}],"held":[{"merges":0,"mean":4.4274,"min":4.4274,"max":4.4274},{"merges":5,"mean":4.2791,"min":4.1458,"max":4.3533},{"merges":10,"mean":4.1583,"min":3.9203,"max":4.2848},{"merges":15,"mean":3.992,"min":3.7669,"max":4.1703},{"merges":20,"mean":3.8446,"min":3.5844,"max":4.0458},{"merges":30,"mean":3.6317,"min":3.4104,"max":3.8338},{"merges":40,"mean":3.369,"min":3.1098,"max":3.5931},{"merges":50,"mean":3.1673,"min":2.8361,"max":3.358},{"merges":60,"mean":2.9606,"min":2.6448,"max":3.1299},{"merges":80,"mean":2.4916,"min":2.2173,"max":2.7952},{"merges":100,"mean":2.1101,"min":1.8926,"max":2.5007},{"merges":120,"mean":1.7474,"min":1.5027,"max":2.0264},{"merges":136,"mean":1.4831,"min":1.3206,"max":1.6612}]},"unseen":[{"word":"lamplit","seenInTraining":false,"pieces":["l","am","p","lit</w>"],"unknownPieces":[]},{"word":"shellfish","seenInTraining":false,"pieces":["s","h","ell","f","i","s","h","</w>"],"unknownPieces":[]},{"word":"redcoat","seenInTraining":false,"pieces":["r","e","d","co","at</w>"],"unknownPieces":[]},{"word":"kettlebell","seenInTraining":false,"pieces":["kettl","e","bell</w>"],"unknownPieces":[]},{"word":"lighthouse","seenInTraining":false,"pieces":["l","i","g","h","t","ho","u","s","e</w>"],"unknownPieces":[]},{"word":"harbour","seenInTraining":false,"pieces":["h","ar","bo","u","r","</w>"],"unknownPieces":[]}]},"english":{"about":"BPE learned on the first five paragraphs of Moby-Dick chapter 1 until every word is one symbol; held out: the first ten paragraphs of Pride and Prejudice chapter 1. Characters per token can start below 1 because the end-of-word symbol is its own token until a merge absorbs it.","sources":{"MOBY_DICK":{"author":"Herman Melville","title":"Moby-Dick; or, The Whale","year":1851,"part":"chapter 1, Loomings, paragraphs 1 to 5","source":"Project Gutenberg eBook #2701","url":"https://www.gutenberg.org/files/2701/2701-0.txt"},"PRIDE_AND_PREJUDICE":{"author":"Jane Austen","title":"Pride and Prejudice","year":1813,"part":"chapter 1, paragraphs 1 to 10 (italic underscores removed, illustration caption omitted)","source":"Project Gutenberg eBook #1342","url":"https://www.gutenberg.org/files/1342/1342-0.txt"}},"trainText":"Call me Ishmael. Some years ago\u2014never mind how long precisely\u2014having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodically knocking people\u2019s hats off\u2014then, I account it high time to get to sea as soon as I can. This is my substitute for pistol and ball. With a philosophical flourish Cato throws himself upon his sword; I quietly take to the ship. There is nothing surprising in this. If they but knew it, almost all men in their degree, some time or other, cherish very nearly the same feelings towards the ocean with me.\n\nThere now is your insular city of the Manhattoes, belted round by wharves as Indian isles by coral reefs\u2014commerce surrounds it with her surf. Right and left, the streets take you waterward. Its extreme downtown is the battery, where that noble mole is washed by waves, and cooled by breezes, which a few hours previous were out of sight of land. Look at the crowds of water-gazers there.\n\nCircumambulate the city of a dreamy Sabbath afternoon. Go from Corlears Hook to Coenties Slip, and from thence, by Whitehall, northward. What do you see?\u2014Posted like silent sentinels all around the town, stand thousands upon thousands of mortal men fixed in ocean reveries. Some leaning against the spiles; some seated upon the pier-heads; some looking over the bulwarks of ships from China; some high aloft in the rigging, as if striving to get a still better seaward peep. But these are all landsmen; of week days pent up in lath and plaster\u2014tied to counters, nailed to benches, clinched to desks. How then is this? Are the green fields gone? What do they here?\n\nBut look! here come more crowds, pacing straight for the water, and seemingly bound for a dive. Strange! Nothing will content them but the extremest limit of the land; loitering under the shady lee of yonder warehouses will not suffice. No. They must get just as nigh the water as they possibly can without falling in. And there they stand\u2014miles of them\u2014leagues. Inlanders all, they come from lanes and alleys, streets and avenues\u2014north, east, south, and west. Yet here they all unite. Tell me, does the magnetic virtue of the needles of the compasses of all those ships attract them thither?\n\nOnce more. Say you are in the country; in some high land of lakes. Take almost any path you please, and ten to one it carries you down in a dale, and leaves you there by a pool in the stream. There is magic in it. Let the most absent-minded of men be plunged in his deepest reveries\u2014stand that man on his legs, set his feet a-going, and he will infallibly lead you to water, if water there be in all that region. Should you ever be athirst in the great American desert, try this experiment, if your caravan happen to be supplied with a metaphysical professor. Yes, as every one knows, meditation and water are wedded for ever.","heldText":"It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.\n\nHowever little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered as the rightful property of some one or other of their daughters.\n\n\u201cMy dear Mr. Bennet,\u201d said his lady to him one day, \u201chave you heard that Netherfield Park is let at last?\u201d\n\nMr. Bennet replied that he had not.\n\n\u201cBut it is,\u201d returned she; \u201cfor Mrs. Long has just been here, and she told me all about it.\u201d\n\nMr. Bennet made no answer.\n\n\u201cDo not you want to know who has taken it?\u201d cried his wife, impatiently.\n\n\u201cYou want to tell me, and I have no objection to hearing it.\u201d\n\nThis was invitation enough.\n\n\u201cWhy, my dear, you must know, Mrs. Long says that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and some of his servants are to be in the house by the end of next week.\u201d","chars":3383,"words":716,"wordTypes":362,"spaceWords":599,"heldSpaceWords":233,"heldWords":281,"heldWordsUnseen":117,"base":["!",",","-",".",";","</w>","?","A","B","C","G","H","I","L","M","N","O","P","R","S","T","W","Y","a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","\u2014","\u2019"],"baseSize":51,"totalMerges":878,"firstMerges":[{"a":"e","b":"</w>","made":"e</w>","count":110},{"a":"t","b":"h","made":"th","count":83},{"a":"s","b":"</w>","made":"s</w>","count":81},{"a":"t","b":"</w>","made":"t</w>","count":64},{"a":"e","b":"r","made":"er","count":57},{"a":"d","b":"</w>","made":"d</w>","count":54},{"a":"i","b":"n","made":"in","count":54},{"a":"a","b":"n","made":"an","count":44},{"a":"y","b":"</w>","made":"y</w>","count":43},{"a":",","b":"</w>","made":",</w>","count":39},{"a":"o","b":"u","made":"ou","count":34},{"a":"th","b":"e</w>","made":"the</w>","count":33},{"a":".","b":"</w>","made":".</w>","count":31},{"a":"f","b":"</w>","made":"f</w>","count":30},{"a":"an","b":"d</w>","made":"and</w>","count":26},{"a":"l","b":"</w>","made":"l</w>","count":26},{"a":"e","b":"n","made":"en","count":25},{"a":"e","b":"s</w>","made":"es</w>","count":24},{"a":"er","b":"</w>","made":"er</w>","count":24},{"a":"a","b":"r","made":"ar","count":23}],"vocabCurve":[{"merges":0,"vocab":51,"symbolsInUse":51,"trainTokens":3497},{"merges":25,"vocab":76,"symbolsInUse":74,"trainTokens":2487},{"merges":50,"vocab":101,"symbolsInUse":98,"trainTokens":2150},{"merges":75,"vocab":126,"symbolsInUse":122,"trainTokens":1951},{"merges":100,"vocab":151,"symbolsInUse":145,"trainTokens":1812},{"merges":125,"vocab":176,"symbolsInUse":167,"trainTokens":1711},{"merges":150,"vocab":201,"symbolsInUse":192,"trainTokens":1624},{"merges":175,"vocab":226,"symbolsInUse":216,"trainTokens":1549},{"merges":200,"vocab":251,"symbolsInUse":237,"trainTokens":1494},{"merges":225,"vocab":276,"symbolsInUse":256,"trainTokens":1444},{"merges":250,"vocab":301,"symbolsInUse":277,"trainTokens":1394},{"merges":275,"vocab":326,"symbolsInUse":295,"trainTokens":1344},{"merges":300,"vocab":351,"symbolsInUse":314,"trainTokens":1294},{"merges":325,"vocab":376,"symbolsInUse":321,"trainTokens":1269},{"merges":350,"vocab":401,"symbolsInUse":327,"trainTokens":1244},{"merges":375,"vocab":426,"symbolsInUse":332,"trainTokens":1219},{"merges":400,"vocab":451,"symbolsInUse":346,"trainTokens":1194},{"merges":425,"vocab":476,"symbolsInUse":352,"trainTokens":1169},{"merges":450,"vocab":501,"symbolsInUse":358,"trainTokens":1144},{"merges":475,"vocab":526,"symbolsInUse":368,"trainTokens":1119},{"merges":500,"vocab":551,"symbolsInUse":373,"trainTokens":1094},{"merges":525,"vocab":576,"symbolsInUse":371,"trainTokens":1069},{"merges":550,"vocab":601,"symbolsInUse":372,"trainTokens":1044},{"merges":575,"vocab":626,"symbolsInUse":375,"trainTokens":1019},{"merges":600,"vocab":651,"symbolsInUse":379,"trainTokens":994},{"merges":625,"vocab":676,"symbolsInUse":385,"trainTokens":969},{"merges":650,"vocab":701,"symbolsInUse":388,"trainTokens":944},{"merges":675,"vocab":726,"symbolsInUse":391,"trainTokens":919},{"merges":700,"vocab":751,"symbolsInUse":393,"trainTokens":894},{"merges":725,"vocab":776,"symbolsInUse":396,"trainTokens":869},{"merges":750,"vocab":801,"symbolsInUse":393,"trainTokens":844},{"merges":775,"vocab":826,"symbolsInUse":397,"trainTokens":819},{"merges":800,"vocab":851,"symbolsInUse":386,"trainTokens":794},{"merges":825,"vocab":876,"symbolsInUse":382,"trainTokens":769},{"merges":850,"vocab":901,"symbolsInUse":371,"trainTokens":744},{"merges":875,"vocab":926,"symbolsInUse":364,"trainTokens":719},{"merges":878,"vocab":929,"symbolsInUse":362,"trainTokens":716}],"curveTrain":[{"merges":0,"vocab":51,"tokens":3497,"tokensPerWord":4.8841,"charsPerToken":0.7953},{"merges":10,"vocab":61,"tokens":2868,"tokensPerWord":4.0056,"charsPerToken":0.9697},{"merges":25,"vocab":76,"tokens":2487,"tokensPerWord":3.4735,"charsPerToken":1.1182},{"merges":50,"vocab":101,"tokens":2150,"tokensPerWord":3.0028,"charsPerToken":1.2935},{"merges":75,"vocab":126,"tokens":1951,"tokensPerWord":2.7249,"charsPerToken":1.4254},{"merges":100,"vocab":151,"tokens":1812,"tokensPerWord":2.5307,"charsPerToken":1.5348},{"merges":150,"vocab":201,"tokens":1624,"tokensPerWord":2.2682,"charsPerToken":1.7124},{"merges":200,"vocab":251,"tokens":1494,"tokensPerWord":2.0866,"charsPerToken":1.8614},{"merges":300,"vocab":351,"tokens":1294,"tokensPerWord":1.8073,"charsPerToken":2.1491},{"merges":400,"vocab":451,"tokens":1194,"tokensPerWord":1.6676,"charsPerToken":2.3291},{"merges":500,"vocab":551,"tokens":1094,"tokensPerWord":1.5279,"charsPerToken":2.542},{"merges":600,"vocab":651,"tokens":994,"tokensPerWord":1.3883,"charsPerToken":2.7978},{"merges":700,"vocab":751,"tokens":894,"tokensPerWord":1.2486,"charsPerToken":3.1107},{"merges":800,"vocab":851,"tokens":794,"tokensPerWord":1.1089,"charsPerToken":3.5025},{"merges":878,"vocab":929,"tokens":716,"tokensPerWord":1,"charsPerToken":3.8841}],"curveHeld":[{"merges":0,"vocab":51,"tokens":1250,"tokensPerWord":4.4484,"charsPerToken":0.7752},{"merges":10,"vocab":61,"tokens":1032,"tokensPerWord":3.6726,"charsPerToken":0.939},{"merges":25,"vocab":76,"tokens":923,"tokensPerWord":3.2847,"charsPerToken":1.0498},{"merges":50,"vocab":101,"tokens":804,"tokensPerWord":2.8612,"charsPerToken":1.2052},{"merges":75,"vocab":126,"tokens":746,"tokensPerWord":2.6548,"charsPerToken":1.2989},{"merges":100,"vocab":151,"tokens":712,"tokensPerWord":2.5338,"charsPerToken":1.361},{"merges":150,"vocab":201,"tokens":667,"tokensPerWord":2.3737,"charsPerToken":1.4528},{"merges":200,"vocab":251,"tokens":642,"tokensPerWord":2.2847,"charsPerToken":1.5093},{"merges":300,"vocab":351,"tokens":606,"tokensPerWord":2.1566,"charsPerToken":1.599},{"merges":400,"vocab":451,"tokens":602,"tokensPerWord":2.1423,"charsPerToken":1.6096},{"merges":500,"vocab":551,"tokens":595,"tokensPerWord":2.1174,"charsPerToken":1.6286},{"merges":600,"vocab":651,"tokens":587,"tokensPerWord":2.089,"charsPerToken":1.6508},{"merges":700,"vocab":751,"tokens":568,"tokensPerWord":2.0214,"charsPerToken":1.706},{"merges":800,"vocab":851,"tokens":554,"tokensPerWord":1.9715,"charsPerToken":1.7491},{"merges":878,"vocab":929,"tokens":551,"tokensPerWord":1.9609,"charsPerToken":1.7586}],"randomControl":{"seeds":10,"train":[{"merges":0,"mean":4.8841,"min":4.8841,"max":4.8841},{"merges":10,"mean":4.7737,"min":4.648,"max":4.831},{"merges":25,"mean":4.6409,"min":4.3827,"max":4.7193},{"merges":50,"mean":4.4701,"min":4.2751,"max":4.5908},{"merges":75,"mean":4.308,"min":4.0908,"max":4.4581},{"merges":100,"mean":4.1556,"min":3.9749,"max":4.3031},{"merges":150,"mean":3.8961,"min":3.648,"max":4.0349},{"merges":200,"mean":3.638,"min":3.4274,"max":3.7947},{"merges":300,"mean":3.2552,"min":3.0908,"max":3.4483},{"merges":400,"mean":2.9479,"min":2.7696,"max":3.1704},{"merges":500,"mean":2.6274,"min":2.4623,"max":2.8268},{"merges":600,"mean":2.3582,"min":2.2207,"max":2.5922},{"merges":700,"mean":2.0908,"min":1.9134,"max":2.2542},{"merges":800,"mean":1.8637,"min":1.7179,"max":1.9846},{"merges":878,"mean":1.677,"min":1.5419,"max":1.7877}],"held":[{"merges":0,"mean":4.4484,"min":4.4484,"max":4.4484},{"merges":10,"mean":4.3605,"min":4.274,"max":4.3986},{"merges":25,"mean":4.2488,"min":4.0605,"max":4.3203},{"merges":50,"mean":4.0936,"min":3.9573,"max":4.1957},{"merges":75,"mean":3.9441,"min":3.8078,"max":4.0996},{"merges":100,"mean":3.8359,"min":3.6512,"max":3.9751},{"merges":150,"mean":3.6477,"min":3.4128,"max":3.8221},{"merges":200,"mean":3.4715,"min":3.1993,"max":3.605},{"merges":300,"mean":3.1893,"min":2.9964,"max":3.3523},{"merges":400,"mean":3.006,"min":2.8185,"max":3.1886},{"merges":500,"mean":2.81,"min":2.6584,"max":2.9609},{"merges":600,"mean":2.6263,"min":2.4662,"max":2.8292},{"merges":700,"mean":2.469,"min":2.3665,"max":2.5801},{"merges":800,"mean":2.3552,"min":2.242,"max":2.4698},{"merges":878,"mean":2.253,"min":2.0854,"max":2.3452}]},"unseen":[{"word":"lighthouse","seenInTraining":false,"pieces":["li","gh","thou","se</w>"],"unknownPieces":[]},{"word":"seafaring","seenInTraining":false,"pieces":["se","af","ar","ing</w>"],"unknownPieces":[]},{"word":"whaleboats","seenInTraining":false,"pieces":["w","h","al","e","b","o","ats</w>"],"unknownPieces":[]},{"word":"Pylon","seenInTraining":false,"pieces":["P","y","l","on</w>"],"unknownPieces":[]},{"word":"neighbourhood","seenInTraining":false,"pieces":["ne","i","gh","b","ou","r","h","oo","d</w>"],"unknownPieces":[]},{"word":"Netherfield","seenInTraining":false,"pieces":["N","e","th","er","fi","el","d</w>"],"unknownPieces":[]}],"heldUnknownChars":["\u201c","\u201d","D","E"],"heldUnknownTokens":16,"crossDomain":{"about":"Pride and Prejudice cut with the 136 merges learned on the guides' corpus, against the first 136 merges learned on Moby-Dick.","corpusMerges":3.2456,"englishMerges":2.4057}},"bytes":{"about":"UTF-8 bytes per character from TextEncoder, and a byte-level BPE learned on the same Moby-Dick text from a base of all 256 bytes (Radford et al. 2019, section 2.2). Control: one byte per character by keeping the low 8 bits of the code point, which cannot be decoded back.","table":[{"char":"a","codePoint":"U+0061","bytes":[97],"hex":"61","nBytes":1},{"char":"\u00e9","codePoint":"U+00E9","bytes":[195,169],"hex":"c3 a9","nBytes":2},{"char":"\u2019","codePoint":"U+2019","bytes":[226,128,153],"hex":"e2 80 99","nBytes":3},{"char":"\u20ac","codePoint":"U+20AC","bytes":[226,130,172],"hex":"e2 82 ac","nBytes":3},{"char":"\ud83e\uddad","codePoint":"U+1F9AD","bytes":[240,159,166,173],"hex":"f0 9f a6 ad","nBytes":4}],"passages":[{"name":"MOBY_DICK","chars":3383,"bytes":3405,"nonAscii":["\u2014","\u2019"]},{"name":"PRIDE_AND_PREJUDICE","chars":1210,"bytes":1238,"nonAscii":["\u201c","\u201d"]}],"naiveControl":[{"char":"a","utf8RoundTrip":true,"naiveRoundTrip":true,"naiveByte":97},{"char":"\u00e9","utf8RoundTrip":true,"naiveRoundTrip":true,"naiveByte":233},{"char":"\u2019","utf8RoundTrip":true,"naiveRoundTrip":false,"naiveByte":25},{"char":"\u20ac","utf8RoundTrip":true,"naiveRoundTrip":false,"naiveByte":172},{"char":"\ud83e\uddad","utf8RoundTrip":true,"naiveRoundTrip":false,"naiveByte":173}],"byteBPE":{"baseSize":257,"merges":878,"firstMerges":[{"a":"e","b":"</w>","made":"e</w>","count":110},{"a":"t","b":"h","made":"th","count":83},{"a":"s","b":"</w>","made":"s</w>","count":81},{"a":"t","b":"</w>","made":"t</w>","count":64},{"a":"e","b":"r","made":"er","count":57},{"a":"d","b":"</w>","made":"d</w>","count":54},{"a":"i","b":"n","made":"in","count":54},{"a":"a","b":"n","made":"an","count":44},{"a":"y","b":"</w>","made":"y</w>","count":43},{"a":",","b":"</w>","made":",</w>","count":39},{"a":"o","b":"u","made":"ou","count":34},{"a":"th","b":"e</w>","made":"the</w>","count":33},{"a":".","b":"</w>","made":".</w>","count":31},{"a":"f","b":"</w>","made":"f</w>","count":30},{"a":"an","b":"d</w>","made":"and</w>","count":26},{"a":"l","b":"</w>","made":"l</w>","count":26},{"a":"e","b":"n","made":"en","count":25},{"a":"e","b":"s</w>","made":"es</w>","count":24},{"a":"er","b":"</w>","made":"er</w>","count":24},{"a":"a","b":"r","made":"ar","count":23}],"heldTokensPerWord":2.0107,"heldUnknownTokens":0},"charBPE":{"baseSize":51,"merges":878,"heldTokensPerWord":1.9609,"heldUnknownTokens":16},"probe":{"text":"Pylon\u2019s caf\u00e9 \u201cbell\u201d \ud83e\uddad","charLevel":{"pieces":[["P","y","l","on</w>"],["\u2019</w>"],["s</w>"],["c","af","\u00e9","</w>"],["\u201c","</w>"],["bel","l</w>"],["\u201d","</w>"],["\ud83e\uddad","</w>"]],"unknown":4},"byteLevel":{"pieces":[["P","y","l","on</w>"],["<e2><80>","<99>","</w>"],["s</w>"],["c","af","<c3>","<a9>","</w>"],["<e2><80>","<9c>","</w>"],["bel","l</w>"],["<e2><80>","<9d>","</w>"],["<f0>","<9f>","<a6>","<ad>","</w>"]],"unknown":0}},"gpt2":{"citation":"Radford, Wu, Child, Luan, Amodei, Sutskever (2019), Language Models are Unsupervised Multitask Learners, OpenAI, section 2.2 Input Representation and section 2.3 Model.","says":["Reference BPE implementations work on Unicode code points; covering all of Unicode that way needs a base vocabulary of over 130,000 symbols before any merge (section 2.2).","A byte-level BPE needs a base vocabulary of only 256 (section 2.2).","Plain BPE on bytes wasted slots on variants such as dog. dog! dog?, so GPT-2 blocks merges across character categories, with an exception for spaces (section 2.2).","Because any Unicode string is a byte string, the model can assign a probability to any string (section 2.2).","The GPT-2 vocabulary has 50,257 tokens (section 2.3)."]}},"tieIn":{"about":"The attention model is word-level with 46 words. The same training corpus cut by BPE at a few vocabulary sizes (base characters plus merges). Every corpus word is a single token after 136 merges (vocabulary 160).","attentionVocab":46,"rows":[{"vocab":24,"merges":0,"tokens":206528,"tokensPerWord":4.394,"wordLevelTokens":47002},{"vocab":46,"merges":22,"tokens":120917,"tokensPerWord":2.5726,"wordLevelTokens":47002},{"vocab":64,"merges":40,"tokens":95028,"tokensPerWord":2.0218,"wordLevelTokens":47002},{"vocab":100,"merges":76,"tokens":66949,"tokensPerWord":1.4244,"wordLevelTokens":47002},{"vocab":160,"merges":136,"tokens":47002,"tokensPerWord":1,"wordLevelTokens":47002}],"everyWordWholeAtMerges":136},"heldOutSeed":1592593325,"seconds":17.24}