{"about":"Lesson K23, fused attention. Computed by tools/school/fill_fused.mjs from system/school/fused.mjs, fused-emu.mjs, fused-wgsl.mjs and the attention lab's trained model. Do not edit by hand. No GPU was used: every count is from the emulator, every error from a float32 emulation against float64. The WGSL kernels were validated by naga 30.0.1 on the CPU (no GPU) when this lesson was written; that is a compile check, not a run. Timings on a real GPU happen only in the lab, on the reader's device, labelled with their adapter.","modelSha256":"67600098679443bf87c78f0f3ef1a1399ae51642b3e9608517105d7831219d5e","modelFile":"media/school/attention/model.json","sourceSha256":{"fused.mjs":"c1a55c8d89694d754bc5eae8eb09453cc41277cd20ad92a3124016034a11dccc","fused-emu.mjs":"c7e2a58cdc5a41778085769352995d33702538ae8895387663a1d7b57716103c","fused-wgsl.mjs":"67faeef3112e921a41b113c6139133b606c81935faa90579593c000af3370cdf"},"sources":{"about":"What the sources claim, read for this lesson (arXiv PDFs read in full; the WGSL and WebGPU specifications are W3C Candidate Recommendation Drafts of September 2026). Every count in the other blocks is this lesson's own emulator on its own kernels; none of it is a measurement of the papers' GPUs. Differences: the kernels here run one head, without masking (the JavaScript tiled attention also runs causal), keep the query loop outside and the key loop inside (the order of FlashAttention-2, not Algorithm 1's outer loop over K and V), and write only O, not the statistics m and l that Algorithm 1 keeps in HBM for the backward pass.","flashAttention":{"cite":"Dao, Fu, Ermon, Rudra and Re, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022, arXiv:2205.14135","claims":["Algorithm 0 (section 2.2): load Q, K by blocks, compute S = Q K^T, write S to HBM; read S, compute P = softmax(S), write P to HBM; load P and V by blocks, compute O = P V, write O","Algorithm 1 (section 3.1): block sizes Bc = ceil(M / 4d), Br = min(ceil(M / 4d), d) for on-chip SRAM of size M; O, l and m live in HBM and are written back after each block","section 3.1: softmax is computed one block at a time by keeping the statistics m(x) and l(x) and rescaling by e^(m(x_i) - m(x))","Theorem 1: exact attention with O(N^2 d) FLOPs and O(N) additional memory","Theorem 2: with d <= M <= N d, standard attention needs Theta(N d + N^2) HBM accesses and FlashAttention Theta(N^2 d^2 / M)","Proposition 3: no exact attention algorithm needs o(N^2 d^2 / M) HBM accesses for all M in [d, N d]","section 2.1: an A100 has 40 to 80 GB of HBM at 1.5 to 2.0 TB/s and 192 KB of SRAM per each of 108 streaming multiprocessors, SRAM bandwidth estimated around 19 TB/s","Figure 2 (A100, GPT-2 medium, N 1024, d 64, 16 heads, batch 64): HBM reads and writes 40.3 GB standard against 4.4 GB FlashAttention; runtime 41.7 ms against 7.3 ms; 66.6 against 75.2 GFLOPs","abstract and section 4: 15 percent end-to-end speedup on BERT-large (N 512) against the MLPerf 1.1 record; 3x on GPT-2 (N 1K) against HuggingFace, 1.8x against Megatron; memory linear in N"]},"onlineSoftmax":{"cite":"Milakov and Gimelshein, Online normalizer calculation for softmax, arXiv:1805.02867 (v2, 2018, a preprint)","claims":["Algorithm 3: m_j = max(m_(j-1), x_j); d_j = d_(j-1) e^(m_(j-1) - m_j) + e^(x_j - m_j), with m_0 = -infinity and d_0 = 0; Theorem 1 proves d_V = sum_j e^(x_j - m_V)","section 2: the safe softmax (max, then sum, then divide) makes 4 memory accesses per element; the naive one makes 3 but is unsafe; section 3: the online normaliser brings the safe one to 3","section 3.1: the (m, d) pairs combine with an associative, commutative operator, so the reduction can run in parallel","section 5 (Tesla V100 PCIe 16 GB, fp32): softmax up to 1.3x faster at large vectors (about V 4000, batch 4000); softmax and top-k fused up to 5x"]},"higham":{"cite":"Higham, Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM 2002, equation 3.5","use":"gamma_n = n u / (1 - n u) bounds a float inner product of length n; it is used for the score and sum terms of the bound in fused.mjs. The bound as a whole is derived for this lesson, first order in u, and is not a theorem from the book."},"wgsl":{"cite":"WebGPU Shading Language, W3C Candidate Recommendation Draft 21 September 2026, section 15.7.4.1 Accuracy of Concrete Floating Point Expressions","exp":"exp(x) in f32: 3 + 2 |x| ULP (in f16: 1 + 2 |x| ULP)","use":"the bound allows this exp error, so a GPU's exp is held to the same check as the emulator's"},"webgpu":{"cite":"WebGPU, W3C Candidate Recommendation Draft 15 September 2026, section 3.6.2 Limits","maxComputeWorkgroupStorageSize":16384}},"traffic":{"about":"Global memory traffic of one attention head, N query and key rows of width d, float32, counted by walking every kernel's loops (compute: false). Tiles: B = 128 at d 16, 32 at d 64 rows, the most that fit a tile of K and one of V in 16,384 bytes of workgroup memory (capped at 128). globalReads and globalWrites are 4-byte element requests; globalBytes is both together. Closed forms with G = ceil(N / B): unfused reads N d + 2 G N d + 2 N^2 and writes 2 N^2 + N d; fused reads N d + 2 G N d and writes N d. So the unfused kernels move exactly 4 N^2 more floats (S written, read, P written, read); scoreShare is that over all unfused traffic. extraMemoryBytes is the N x N score buffer the unfused path must allocate. overTheta divides the traffic by FlashAttention's Theorem 2 orders (standard N d + N^2; fused N^2 d^2 / M with M = 2 B d, the workgroup memory in floats); it settles to a constant as N grows, which is what Theta claims. Requests, not DRAM transactions: caches may serve some.","rows":[{"N":64,"d":16,"B":128,"kind":"unfused","globalReads":11264,"globalWrites":9216,"fma":131072,"exps":4096,"barriers":1220,"sharedReads":229120,"sharedWrites":67456,"globalBytes":81920,"extraMemoryBytes":16384,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4,"scoreShare":0.8,"overFused":5},{"N":64,"d":16,"B":128,"kind":"fused","globalReads":3072,"globalWrites":1024,"fma":131072,"exps":4160,"barriers":2,"sharedReads":131072,"sharedWrites":2048,"globalBytes":16384,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":16,"scoreShare":0,"overFused":1},{"N":128,"d":16,"B":128,"kind":"unfused","globalReads":38912,"globalWrites":34816,"fma":524288,"exps":16384,"barriers":2436,"sharedReads":720384,"sharedWrites":134912,"globalBytes":294912,"extraMemoryBytes":65536,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4,"scoreShare":0.8889,"overFused":9},{"N":128,"d":16,"B":128,"kind":"fused","globalReads":6144,"globalWrites":2048,"fma":524288,"exps":16512,"barriers":2,"sharedReads":524288,"sharedWrites":4096,"globalBytes":32768,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":8,"scoreShare":0,"overFused":1},{"N":256,"d":16,"B":128,"kind":"unfused","globalReads":151552,"globalWrites":135168,"fma":2097152,"exps":65536,"barriers":4880,"sharedReads":2489344,"sharedWrites":278016,"globalBytes":1146880,"extraMemoryBytes":262144,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4.1176,"scoreShare":0.9143,"overFused":11.6667},{"N":256,"d":16,"B":128,"kind":"fused","globalReads":20480,"globalWrites":4096,"fma":2097152,"exps":66048,"barriers":8,"sharedReads":2097152,"sharedWrites":16384,"globalBytes":98304,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":6,"scoreShare":0,"overFused":1},{"N":512,"d":16,"B":128,"kind":"unfused","globalReads":598016,"globalWrites":532480,"fma":8388608,"exps":262144,"barriers":9792,"sharedReads":9172992,"sharedWrites":588800,"globalBytes":4521984,"extraMemoryBytes":1048576,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4.1818,"scoreShare":0.9275,"overFused":13.8},{"N":512,"d":16,"B":128,"kind":"fused","globalReads":73728,"globalWrites":8192,"fma":8388608,"exps":264192,"barriers":32,"sharedReads":8388608,"sharedWrites":65536,"globalBytes":327680,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":5,"scoreShare":0,"overFused":1},{"N":1024,"d":16,"B":128,"kind":"unfused","globalReads":2375680,"globalWrites":2113536,"fma":33554432,"exps":1048576,"barriers":19712,"sharedReads":35123200,"sharedWrites":1308672,"globalBytes":17956864,"extraMemoryBytes":4194304,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4.2154,"scoreShare":0.9343,"overFused":15.2222},{"N":1024,"d":16,"B":128,"kind":"fused","globalReads":278528,"globalWrites":16384,"fma":33554432,"exps":1056768,"barriers":128,"sharedReads":33554432,"sharedWrites":262144,"globalBytes":1179648,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.5,"scoreShare":0,"overFused":1},{"N":2048,"d":16,"B":128,"kind":"unfused","globalReads":9469952,"globalWrites":8421376,"fma":134217728,"exps":4194304,"barriers":39936,"sharedReads":137355264,"sharedWrites":3141632,"globalBytes":71565312,"extraMemoryBytes":16777216,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4.2326,"scoreShare":0.9377,"overFused":16.0588},{"N":2048,"d":16,"B":128,"kind":"fused","globalReads":1081344,"globalWrites":32768,"fma":134217728,"exps":4227072,"barriers":512,"sharedReads":134217728,"sharedWrites":1048576,"globalBytes":4456448,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.25,"scoreShare":0,"overFused":1},{"N":4096,"d":16,"B":128,"kind":"unfused","globalReads":37814272,"globalWrites":33619968,"fma":536870912,"exps":16777216,"barriers":81920,"sharedReads":543145984,"sharedWrites":8380416,"globalBytes":285736960,"extraMemoryBytes":67108864,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":4.2412,"scoreShare":0.9394,"overFused":16.5152},{"N":4096,"d":16,"B":128,"kind":"fused","globalReads":4259840,"globalWrites":65536,"fma":536870912,"exps":16908288,"barriers":2048,"sharedReads":536870912,"sharedWrites":4194304,"globalBytes":17301504,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.125,"scoreShare":0,"overFused":1},{"N":64,"d":64,"B":32,"kind":"unfused","globalReads":28672,"globalWrites":12288,"fma":524288,"exps":4096,"barriers":1232,"sharedReads":622336,"sharedWrites":81792,"globalBytes":163840,"extraMemoryBytes":16384,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":5,"scoreShare":0.4,"overFused":1.6667},{"N":64,"d":64,"B":32,"kind":"fused","globalReads":20480,"globalWrites":4096,"fma":524288,"exps":4224,"barriers":8,"sharedReads":524288,"sharedWrites":16384,"globalBytes":98304,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":6,"scoreShare":0,"overFused":1},{"N":128,"d":64,"B":32,"kind":"unfused","globalReads":106496,"globalWrites":40960,"fma":2097152,"exps":16384,"barriers":2496,"sharedReads":2293248,"sharedWrites":196352,"globalBytes":589824,"extraMemoryBytes":65536,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":6,"scoreShare":0.4444,"overFused":1.8},{"N":128,"d":64,"B":32,"kind":"fused","globalReads":73728,"globalWrites":8192,"fma":2097152,"exps":16896,"barriers":32,"sharedReads":2097152,"sharedWrites":65536,"globalBytes":327680,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":5,"scoreShare":0,"overFused":1},{"N":256,"d":64,"B":32,"kind":"unfused","globalReads":409600,"globalWrites":147456,"fma":8388608,"exps":65536,"barriers":5120,"sharedReads":8780800,"sharedWrites":523776,"globalBytes":2228224,"extraMemoryBytes":262144,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":6.8,"scoreShare":0.4706,"overFused":1.8889},{"N":256,"d":64,"B":32,"kind":"fused","globalReads":278528,"globalWrites":16384,"fma":8388608,"exps":67584,"barriers":128,"sharedReads":8388608,"sharedWrites":262144,"globalBytes":1179648,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.5,"scoreShare":0,"overFused":1},{"N":512,"d":64,"B":32,"kind":"unfused","globalReads":1605632,"globalWrites":557056,"fma":33554432,"exps":262144,"barriers":10752,"sharedReads":34338816,"sharedWrites":1571840,"globalBytes":8650752,"extraMemoryBytes":1048576,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":7.3333,"scoreShare":0.4848,"overFused":1.9412},{"N":512,"d":64,"B":32,"kind":"fused","globalReads":1081344,"globalWrites":32768,"fma":33554432,"exps":270336,"barriers":512,"sharedReads":33554432,"sharedWrites":1048576,"globalBytes":4456448,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.25,"scoreShare":0,"overFused":1},{"N":1024,"d":64,"B":32,"kind":"unfused","globalReads":6356992,"globalWrites":2162688,"fma":134217728,"exps":1048576,"barriers":23552,"sharedReads":135786496,"sharedWrites":5240832,"globalBytes":34078720,"extraMemoryBytes":4194304,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":7.6471,"scoreShare":0.4923,"overFused":1.9697},{"N":1024,"d":64,"B":32,"kind":"fused","globalReads":4259840,"globalWrites":65536,"fma":134217728,"exps":1081344,"barriers":2048,"sharedReads":134217728,"sharedWrites":4194304,"globalBytes":17301504,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.125,"scoreShare":0,"overFused":1},{"N":2048,"d":64,"B":32,"kind":"unfused","globalReads":25296896,"globalWrites":8519680,"fma":536870912,"exps":4194304,"barriers":55296,"sharedReads":540008448,"sharedWrites":18870272,"globalBytes":135266304,"extraMemoryBytes":16777216,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":7.8182,"scoreShare":0.4961,"overFused":1.9846},{"N":2048,"d":64,"B":32,"kind":"fused","globalReads":16908288,"globalWrites":131072,"fma":536870912,"exps":4325376,"barriers":8192,"sharedReads":536870912,"sharedWrites":16777216,"globalBytes":68157440,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.0625,"scoreShare":0,"overFused":1},{"N":4096,"d":64,"B":32,"kind":"unfused","globalReads":100925440,"globalWrites":33816576,"fma":2147483648,"exps":16777216,"barriers":143360,"sharedReads":2153758720,"sharedWrites":71294976,"globalBytes":538968064,"extraMemoryBytes":67108864,"sharedPerWorkgroupBytes":8192,"matchesClosedForm":true,"overTheta":7.9077,"scoreShare":0.4981,"overFused":1.9922},{"N":4096,"d":64,"B":32,"kind":"fused","globalReads":67371008,"globalWrites":262144,"fma":2147483648,"exps":17301504,"barriers":32768,"sharedReads":2147483648,"sharedWrites":67108864,"globalBytes":270532608,"extraMemoryBytes":0,"sharedPerWorkgroupBytes":16384,"matchesClosedForm":true,"overTheta":4.0313,"scoreShare":0,"overFused":1}],"verdict":["At N 4096: d 64 unfused 514 MiB against fused 258 MiB (1.99x); d 16 unfused 272.5 MiB against fused 16.5 MiB (16.52x). The score table is 49.8 percent of the unfused traffic at d 64 and 93.9 percent at d 16.","Unfused over fused traffic by N at d 64: 1.67, 1.8, 1.89, 1.94, 1.97, 1.98, 1.99; at d 16: 5, 9, 11.67, 13.8, 15.22, 16.06, 16.52 (N = 64, 128, 256, 512, 1024, 2048, 4096). The unfused path also allocates 64 MiB for S at N 4096; the fused one allocates nothing extra.","All 28 rows match the closed forms: true. Traffic over the Theorem 2 order at N = 64, 128, 256, 512, 1024, 2048, 4096: fused d 64 6, 5, 4.5, 4.25, 4.13, 4.06, 4.03; unfused d 64 5, 6, 6.8, 7.33, 7.65, 7.82, 7.91. The fused constant tends to 4 (2 G N d = 4 N^2 d^2 / M); the unfused one tends to 4 + 2 d / B (4.25 at d 16, 8 at d 64), because these unfused kernels re-read K and V once per query block (2 G N d) where Algorithm 0's analysis counts them once."]},"correctness":{"about":"Full float32 emulations against a float64 reference (standard softmax attention with the max subtracted) on the same inputs (uniform in [-2, 2), seed 1, scale 1 / sqrt(d)). The tolerance per output is fused.mjs's bound: 2 (exp(2 delta + 2 e) - 1 + gamma_K) sum_j p_ij |v_j| plus an underflow floor, with delta the score error (gamma_(d+1) scale sum |q k|, Higham's equation 3.5) and e WGSL's f32 exp accuracy (3 + 2 |x| ULP); it is derived for this lesson, first order in u. maxRatio is the largest error over its bound. countsAgree: the full emulation and the count-only walk give the same counts. sameBitsAsTiled: the fused kernel's float32 output equals fused.mjs's tiledAttention bit for bit (the same operations in the same order).","rows":[{"N":64,"d":16,"B":128,"kind":"unfused","ok":true,"mismatches":0,"outputs":1024,"maxErr":5.33e-7,"maxRatio":0.0105,"countsAgree":true,"sameBitsAsTiled":null},{"N":64,"d":16,"B":128,"kind":"fused","ok":true,"mismatches":0,"outputs":1024,"maxErr":4.14e-7,"maxRatio":0.0086,"countsAgree":true,"sameBitsAsTiled":true},{"N":64,"d":16,"B":128,"kind":"broken","ok":true,"mismatches":0,"outputs":1024,"maxErr":4.14e-7,"maxRatio":0.0086,"countsAgree":true,"sameBitsAsTiled":null},{"N":256,"d":16,"B":128,"kind":"unfused","ok":true,"mismatches":0,"outputs":4096,"maxErr":5.67e-7,"maxRatio":0.0067,"countsAgree":true,"sameBitsAsTiled":null},{"N":256,"d":16,"B":128,"kind":"fused","ok":true,"mismatches":0,"outputs":4096,"maxErr":3.89e-7,"maxRatio":0.005,"countsAgree":true,"sameBitsAsTiled":true},{"N":256,"d":16,"B":128,"kind":"broken","ok":false,"mismatches":1754,"outputs":4096,"maxErr":0.2722,"maxRatio":4150.9478,"countsAgree":true,"sameBitsAsTiled":null},{"N":1024,"d":16,"B":128,"kind":"unfused","ok":true,"mismatches":0,"outputs":16384,"maxErr":3.35e-7,"maxRatio":0.0021,"countsAgree":true,"sameBitsAsTiled":null},{"N":1024,"d":16,"B":128,"kind":"fused","ok":true,"mismatches":0,"outputs":16384,"maxErr":3.43e-7,"maxRatio":0.0021,"countsAgree":true,"sameBitsAsTiled":true},{"N":1024,"d":16,"B":128,"kind":"broken","ok":false,"mismatches":14086,"outputs":16384,"maxErr":0.2526,"maxRatio":1383.7279,"countsAgree":true,"sameBitsAsTiled":null},{"N":64,"d":64,"B":32,"kind":"unfused","ok":true,"mismatches":0,"outputs":4096,"maxErr":4.69e-7,"maxRatio":0.0026,"countsAgree":true,"sameBitsAsTiled":null},{"N":64,"d":64,"B":32,"kind":"fused","ok":true,"mismatches":0,"outputs":4096,"maxErr":4.38e-7,"maxRatio":0.0024,"countsAgree":true,"sameBitsAsTiled":true},{"N":64,"d":64,"B":32,"kind":"broken","ok":false,"mismatches":2114,"outputs":4096,"maxErr":0.5678,"maxRatio":3004.5855,"countsAgree":true,"sameBitsAsTiled":null},{"N":256,"d":64,"B":32,"kind":"unfused","ok":true,"mismatches":0,"outputs":16384,"maxErr":6.59e-7,"maxRatio":0.0031,"countsAgree":true,"sameBitsAsTiled":null},{"N":256,"d":64,"B":32,"kind":"fused","ok":true,"mismatches":0,"outputs":16384,"maxErr":6.83e-7,"maxRatio":0.0033,"countsAgree":true,"sameBitsAsTiled":true},{"N":256,"d":64,"B":32,"kind":"broken","ok":false,"mismatches":14303,"outputs":16384,"maxErr":0.4559,"maxRatio":2203.4958,"countsAgree":true,"sameBitsAsTiled":null},{"N":1024,"d":64,"B":32,"kind":"unfused","ok":true,"mismatches":0,"outputs":65536,"maxErr":7.79e-7,"maxRatio":0.0024,"countsAgree":true,"sameBitsAsTiled":null},{"N":1024,"d":64,"B":32,"kind":"fused","ok":true,"mismatches":0,"outputs":65536,"maxErr":8.26e-7,"maxRatio":0.0027,"countsAgree":true,"sameBitsAsTiled":true},{"N":1024,"d":64,"B":32,"kind":"broken","ok":false,"mismatches":62352,"outputs":65536,"maxErr":0.31,"maxRatio":919.2886,"countsAgree":true,"sameBitsAsTiled":null}],"verdict":"Unfused and fused pass at every size: true, the largest error at most 0.0105 of the bound (the bound is loose on purpose: it allows a GPU's exp error of 3 + 2 |x| ULP, where the emulator's exp is V8's, rounded once to float32). Fused equals the JavaScript tiled attention bit for bit: true. The broken kernel fails wherever a row spans more than one tile: d 16 N 64 (B 128): passes, one tile; d 16 N 256 (B 128): 1754 of 4096 wrong; d 16 N 1024 (B 128): 14086 of 16384 wrong; d 64 N 64 (B 32): 2114 of 4096 wrong; d 64 N 256 (B 32): 14303 of 16384 wrong; d 64 N 1024 (B 32): 62352 of 65536 wrong."},"probe":{"about":"The attention lab's trained model on the probe sentence \"rerun found a kettle . pylon put a cone on the\" (11 words, causal, head width 16): its own Q, K and V at each layer and head, tiled attention with tiles of 4 keys against the float64 reference within the bound, the model's own attention output (computed by tinyformer-attn.mjs, the standard way) against the same reference, and the broken rescale. row: the scaled scores of query position 10 (the word \"the\", where the answer \"kettle\" is predicted) over keys 0 to 10. trace: that row's running max and running sum after each tile of 4 keys. rowsWithMaxRise: query rows whose max rose after their first tile, the rows where the broken rescale does harm.","heads":[{"layer":0,"head":0,"ok":true,"maxRatio":0.0111,"modelOk":true,"modelMaxRatio":0.0074,"brokenOk":false,"brokenMismatches":64,"outputs":176,"rowsWithMaxRise":4,"row":[-3.1784,0.9175,-3.1788,-2.9439,-3.6449,-4.5474,-1.7949,-2.6287,2.1808,-1.2282,-2.878],"rowMax":2.1808,"argmax":8,"trace":[{"tile":0,"keys":[0,3],"m":0.9175,"l":1.0543},{"tile":1,"keys":[4,7],"m":0.9175,"l":1.1642},{"tile":2,"keys":[8,10],"m":2.1808,"l":1.3686}]},{"layer":0,"head":1,"ok":true,"maxRatio":0.0094,"modelOk":true,"modelMaxRatio":0.0072,"brokenOk":false,"brokenMismatches":96,"outputs":176,"rowsWithMaxRise":6,"row":[-5.9714,-5.5046,-4.6662,-6.5762,-2.9902,-4.0567,-2.604,-3.0409,0.9549,-0.1413,-3.8159],"rowMax":0.9549,"argmax":8,"trace":[{"tile":0,"keys":[0,3],"m":-4.6662,"l":1.8516},{"tile":1,"keys":[4,7],"m":-2.604,"l":2.7951},{"tile":2,"keys":[8,10],"m":0.9549,"l":1.4222}]},{"layer":1,"head":0,"ok":true,"maxRatio":0.0102,"modelOk":true,"modelMaxRatio":0.0039,"brokenOk":false,"brokenMismatches":64,"outputs":176,"rowsWithMaxRise":4,"row":[0.7124,-3.218,0.241,-1.7562,-0.497,-1.5879,-4.321,0.9106,-0.8725,0.8463,1.3328],"rowMax":1.3328,"argmax":10,"trace":[{"tile":0,"keys":[0,3],"m":0.7124,"l":1.7285},{"tile":1,"keys":[4,7],"m":0.9106,"l":2.75},{"tile":2,"keys":[8,10],"m":1.3328,"l":3.5278}]},{"layer":1,"head":1,"ok":true,"maxRatio":0.0073,"modelOk":true,"modelMaxRatio":0.0047,"brokenOk":false,"brokenMismatches":80,"outputs":176,"rowsWithMaxRise":5,"row":[-2.757,-3.3306,-0.5553,10.3194,-1.067,-3.4398,-2.755,-5.3251,1.0504,-2.3313,-3.7547],"rowMax":10.3194,"argmax":3,"trace":[{"tile":0,"keys":[0,3],"m":10.3194,"l":1},{"tile":1,"keys":[4,7],"m":10.3194,"l":1},{"tile":2,"keys":[8,10],"m":10.3194,"l":1.0001}]}],"verdict":"Tiled passes on all four heads: true (worst 0.0111 of the bound); the model's own attention output passes the same check: true. The broken rescale fails on every head: L0 H0 64 of 176, L0 H1 96 of 176, L1 H0 64 of 176, L1 H1 80 of 176. Largest score in row 10: 10.319 at layer 1 head 1, key 3 (\"kettle\")."},"overflow":{"about":"Why the running max matters, with the model's numbers. In fp16 (largest finite 65504) e^s rounds to infinity once s >= ln 65520 = 11.0901. probeRow: the probe's row 10 at the head with its largest score, softmaxed in fp16 without the max subtracted (naive: exp, sum and divide each rounded to fp16) and with it (safe); overflowFactor is the smallest factor by which that row's scores could be scaled before the naive fp16 sum overflows. heldOut: every causal score of 256 held-out rows of 24 words (the float32 forward pass), per layer and head: the largest and smallest score and how many reach the overflow threshold. The model computes in float32 and subtracts the max, so none of this happens in the lab's model; it is what fp16 would do.","threshold":11.0901,"probeRow":{"layer":1,"head":1,"max":10.3194,"naiveSum":30320,"naiveFinite":true,"expOfMax":30320,"safeSum":1,"safeFinite":true,"overflowFactor":1.0747},"heldOut":[{"layer":0,"head":0,"max":13.6892,"min":-9.6774,"count":76800,"atOrOverFp16Overflow":56},{"layer":0,"head":1,"max":15.1935,"min":-11.7603,"count":76800,"atOrOverFp16Overflow":31},{"layer":1,"head":0,"max":23.4102,"min":-10.814,"count":76800,"atOrOverFp16Overflow":515},{"layer":1,"head":1,"max":21.203,"min":-21.6205,"count":76800,"atOrOverFp16Overflow":268}],"verdict":"The probe row's largest score is 10.319: e^10.32 = 30320 in fp16, below 65504, and the naive fp16 sum is 30320, finite; scaling those scores by 1.075 would overflow it. On 256 held-out rows the largest score is 23.41 and 870 of 307200 scores reach 11.09: each would be infinity in fp16 without the max subtracted, and its row's weights would become 0 or NaN. With the max subtracted the largest exp is 1."},"controls":{"about":"Controls. vector: the online softmax of eight numbers against the two-pass safe one (largest difference in the weights), how often its max rose, and the running sum of the broken variant against the right one. checker (N 256, d 16, tiles of 16): the tiled float32 output passes; the exact answer rounded to float32 passes; one output moved by 0.001 fails with exactly one mismatch; zero tolerance rejects the float32 output, so the bound does work.","vector":{"values":[0.5,-1,3,2.5,7,-4,7.25,1],"maxWeightDiff":1.11e-16,"maxRises":3,"sum":1.8051,"brokenSum":4.8316,"trace":[{"m":0.5,"l":1},{"m":0.5,"l":1.2231},{"m":3,"l":1.1004},{"m":3,"l":1.7069},{"m":7,"l":1.0313},{"m":7,"l":1.0313},{"m":7.25,"l":1.8032},{"m":7.25,"l":1.8051}],"brokenTrace":[{"m":0.5,"l":1},{"m":0.5,"l":1.2231},{"m":3,"l":2.2231},{"m":3,"l":2.8297},{"m":7,"l":3.8297},{"m":7,"l":3.8297},{"m":7.25,"l":4.8297},{"m":7.25,"l":4.8316}]},"checker":{"tiled":true,"roundedReference":true,"oneOff":1,"zeroTolerance":false}},"film":{"about":"What the films draw (the lighthouse and the cannery joined: the beam computes scores, the crew weighs and carries in one trip, and the unfused crew stacks the whole score table in the warehouse first). bars: global bytes per N for each path at d 16 and d 64, and the score table's share (16 N^2 bytes). table: the probe's full causal score table (rows are queries, keys up to each row) at the head with the largest score, from the model. sweep: row 10 at every layer and head walked in tiles of 4 keys, with the running max and running sum after each tile (the dial and the gauge); rises counts the tiles that raised the max (at the head with the largest score the max is found in the first tile and never moves, so the film should use a head whose dial turns). scoreBuffer: bytes of the N x N table at each N.","bars":{"d16":{"unfused":[{"N":64,"bytes":81920,"scoreBytes":65536},{"N":128,"bytes":294912,"scoreBytes":262144},{"N":256,"bytes":1146880,"scoreBytes":1048576},{"N":512,"bytes":4521984,"scoreBytes":4194304},{"N":1024,"bytes":17956864,"scoreBytes":16777216},{"N":2048,"bytes":71565312,"scoreBytes":67108864},{"N":4096,"bytes":285736960,"scoreBytes":268435456}],"fused":[{"N":64,"bytes":16384,"scoreBytes":0},{"N":128,"bytes":32768,"scoreBytes":0},{"N":256,"bytes":98304,"scoreBytes":0},{"N":512,"bytes":327680,"scoreBytes":0},{"N":1024,"bytes":1179648,"scoreBytes":0},{"N":2048,"bytes":4456448,"scoreBytes":0},{"N":4096,"bytes":17301504,"scoreBytes":0}]},"d64":{"unfused":[{"N":64,"bytes":163840,"scoreBytes":65536},{"N":128,"bytes":589824,"scoreBytes":262144},{"N":256,"bytes":2228224,"scoreBytes":1048576},{"N":512,"bytes":8650752,"scoreBytes":4194304},{"N":1024,"bytes":34078720,"scoreBytes":16777216},{"N":2048,"bytes":135266304,"scoreBytes":67108864},{"N":4096,"bytes":538968064,"scoreBytes":268435456}],"fused":[{"N":64,"bytes":98304,"scoreBytes":0},{"N":128,"bytes":327680,"scoreBytes":0},{"N":256,"bytes":1179648,"scoreBytes":0},{"N":512,"bytes":4456448,"scoreBytes":0},{"N":1024,"bytes":17301504,"scoreBytes":0},{"N":2048,"bytes":68157440,"scoreBytes":0},{"N":4096,"bytes":270532608,"scoreBytes":0}]}},"table":{"layer":1,"head":1,"words":["rerun","found","a","kettle",".","pylon","put","a","cone","on","the"],"scores":[[0.929],[-2.099,-0.279],[-4.902,-1.608,2.936],[-0.917,1.37,2.022,-7.489],[-0.897,0.455,1.178,-5.691,1.776],[1.619,2.028,0.488,-11.338,2.284,1.621],[3.069,1.862,-0.808,-5.608,0.175,1.6,1.667],[0.007,0.371,0.545,-5.438,1.496,0.265,-0.096,-0.399],[-0.77,0.715,1.294,-5.359,1.495,0.846,0.231,0.732,-1.719],[-0.119,0.319,0.311,-3.99,1.195,0.052,-0.026,-0.475,-1.785,0.255],[-2.757,-3.331,-0.555,10.319,-1.067,-3.44,-2.755,-5.325,1.05,-2.331,-3.755]]},"sweep":{"row":10,"tile":4,"heads":[{"layer":0,"head":0,"rises":1,"trace":[{"tile":0,"keys":[0,3],"m":0.9175,"l":1.0543},{"tile":1,"keys":[4,7],"m":0.9175,"l":1.1642},{"tile":2,"keys":[8,10],"m":2.1808,"l":1.3686}]},{"layer":0,"head":1,"rises":2,"trace":[{"tile":0,"keys":[0,3],"m":-4.6662,"l":1.8516},{"tile":1,"keys":[4,7],"m":-2.604,"l":2.7951},{"tile":2,"keys":[8,10],"m":0.9549,"l":1.4222}]},{"layer":1,"head":0,"rises":2,"trace":[{"tile":0,"keys":[0,3],"m":0.7124,"l":1.7285},{"tile":1,"keys":[4,7],"m":0.9106,"l":2.75},{"tile":2,"keys":[8,10],"m":1.3328,"l":3.5278}]},{"layer":1,"head":1,"rises":0,"trace":[{"tile":0,"keys":[0,3],"m":10.3194,"l":1},{"tile":1,"keys":[4,7],"m":10.3194,"l":1},{"tile":2,"keys":[8,10],"m":10.3194,"l":1.0001}]}]},"scoreBuffer":[{"N":64,"bytes":16384},{"N":128,"bytes":65536},{"N":256,"bytes":262144},{"N":512,"bytes":1048576},{"N":1024,"bytes":4194304},{"N":2048,"bytes":16777216},{"N":4096,"bytes":67108864}]},"lab":{"about":"The lab's measurement, run only in a browser on the reader's own GPU through WebGPU (system/school/fused-gpu.mjs, runAttention). For each variant and size it first checks O against the float64 reference within the bound in fused.mjs (every row up to N = 1024, the first 64 rows above that, and it says which), then times reps runs after one warm-up and reports the median wall time from submit to onSubmittedWorkDone (submission overhead included). Every result is labelled with the adapter's vendor, architecture and description. It is not run by the fill or the tests, and the GPU is not used while another job holds it.","sizes":[256,512,1024,2048,4096],"heads":[16,64],"kinds":["unfused","fused","broken"],"reps":9,"warmup":1,"rowsCheckedAbove1024":64,"measures":["medianMs per run (unfused: three dispatches in one submit)","fused over unfused speed ratio at each N and d","checker result for every run, including the broken kernel"],"predicts":"The emulator counts global traffic: the fused kernel saves exactly 4 N^2 floats (16 N^2 bytes), the score matrix written, read, written as P and read again. The traffic block gives that saving's share of the unfused traffic at each N and d. A measured speed ratio far from the traffic ratio says caches, launch overhead or arithmetic set the time, not global memory. The model gives counts, not times."},"speed":{"about":"Milliseconds in Node (V8), one process at below-normal priority, on the fill machine's CPU. The emulator runs every multiply-add and exp in float32, so it is far slower than a GPU. slice: milliseconds per work item at N 1024 (each kernel's items timed together, first item included) and how many fit a 40 ms lab slice. runs: wall time of each full emulation in the correctness block. totals: milliseconds spent counting, emulating and building references over the fill; fillSeconds is the wall time.","slice":[{"item":"scores workgroup","N":1024,"d":16,"B":128,"msPerItem":4.4454,"itemsPer40ms":8},{"item":"softmax row","N":1024,"d":16,"B":128,"msPerItem":0.0348,"itemsPer40ms":1150},{"item":"P V workgroup","N":1024,"d":16,"B":128,"msPerItem":4.5562,"itemsPer40ms":8},{"item":"fused workgroup","N":1024,"d":16,"B":128,"msPerItem":15.5126,"itemsPer40ms":2},{"item":"scores workgroup","N":1024,"d":64,"B":32,"msPerItem":5.7828,"itemsPer40ms":6},{"item":"softmax row","N":1024,"d":64,"B":32,"msPerItem":0.0397,"itemsPer40ms":1008},{"item":"P V workgroup","N":1024,"d":64,"B":32,"msPerItem":5.0348,"itemsPer40ms":7},{"item":"fused workgroup","N":1024,"d":64,"B":32,"msPerItem":14.4628,"itemsPer40ms":2}],"runs":[{"N":64,"d":16,"kind":"unfused","ms":10.1141},{"N":64,"d":16,"kind":"fused","ms":4.6218},{"N":64,"d":16,"kind":"broken","ms":4.12},{"N":256,"d":16,"kind":"unfused","ms":35.4048},{"N":256,"d":16,"kind":"fused","ms":17.237},{"N":256,"d":16,"kind":"broken","ms":9.2545},{"N":1024,"d":16,"kind":"unfused","ms":170.7379},{"N":1024,"d":16,"kind":"fused","ms":129.1097},{"N":1024,"d":16,"kind":"broken","ms":123.3205},{"N":64,"d":64,"kind":"unfused","ms":2.006},{"N":64,"d":64,"kind":"fused","ms":2.3058},{"N":64,"d":64,"kind":"broken","ms":2.2374},{"N":256,"d":64,"kind":"unfused","ms":27.2161},{"N":256,"d":64,"kind":"fused","ms":29.7907},{"N":256,"d":64,"kind":"broken","ms":31.7863},{"N":1024,"d":64,"kind":"unfused","ms":388.7106},{"N":1024,"d":64,"kind":"fused","ms":466.4781},{"N":1024,"d":64,"kind":"broken","ms":453.0145}],"totals":{"count":78.3173,"reference":572.4401,"emulate":1907.2747},"fillSeconds":3.314},"seconds":4.4}