{"about":"Lesson K22, matmul tiling. Computed by tools/school/fill_tiling.mjs from the emulator in system/school/tiling.mjs. Do not edit by hand. No GPU was used: every number is a count from the emulator or a claim of a cited source. Timings on a real GPU happen only in the lab, on the reader's device, labelled with their adapter.","sourceSha256":"be292084decc55489e0bbb6dfde46aedd3644c0b73257aa66d92e3b601076dda","sourceFile":"system/school/tiling.mjs","sources":{"about":"What the sources claim, read for this lesson. cudaGuide: the CUDA C++ Programming Guide 12.6 (https://docs.nvidia.com/cuda/archive/12.6.0/cuda-c-programming-guide/), section 3.2.4 Shared Memory, which gives a matrix multiply without and with shared memory, BLOCK_SIZE 16, matrix dimensions assumed to be multiples of BLOCK_SIZE, and two __syncthreads() calls (one so the sub-matrices are loaded before the computation starts, one so the computation is done before the next sub-matrices are loaded). The current guide (13.4.2) teaches shared memory with a matrix transpose instead (section 2.3.4.2.1), so the matmul example is cited from 12.6. volkovDemmel: Volkov and Demmel, Benchmarking GPUs to tune dense linear algebra, SC 2008 (UC Berkeley tech report EECS-2008-49), section 4.1 and table 2. The kernels here are the guide's scheme (T x T tiles in shared memory, one output per invocation); Volkov and Demmel's fast kernel holds a 64 x 16 block of C in registers and only B in shared memory, which this lesson does not implement.","cudaGuide":{"version":"12.6 (archived), section 3.2.4","withoutShared":"A is read B.width times from global memory and B is read A.height times","withShared":"A is only read (B.width / block_size) times from global memory and B is read (A.height / block_size) times","blockSize":16,"assumesMultiples":true,"barriers":2},"volkovDemmel":{"report":"UC Berkeley EECS-2008-49 (SC08)","bandwidthWords":"m n k (1/bn + 1/bm) words for bm x bn blocks of C, 2 / (1/bn + 1/bm) times less than with no blocking; independent of bk","flopToWord":"18 for a GeForce 8800 GTX aiming at 346 Gflop/s under a 75 GB/s cap (12 for 230 Gflop/s), so the smallest useful block is 18 x 18 (12 x 12)","blocking":"64 x 16 blocks of C in registers, B in shared (local) memory, bk = 4","measured8800GTX":{"peakGflops":230,"cublas11Gflops":128,"estimatedGflops":202,"observedGflops":206,"note":"C := C + AB, square matrices up to 4096, table 2; peak is for multiply-add with one operand in shared memory, per their benchmarks"},"abstract":"GEMM runs 60% faster than CUBLAS 1.1 and approaches the peak of hardware capabilities"}},"counts":{"about":"Counts from the emulator for C = A x B, square n x n float32, every kernel run in full (all n^3 multiply-adds). globalLoads and globalStores are 4-byte element requests to global memory; sharedLoads and sharedStores to workgroup memory; barriers are workgroupBarrier() calls per workgroup (barrierArrivals counts each invocation's arrival); fma is multiply-adds (flops = 2 fma). intensity = flops / global bytes (loads and stores). matchesClosedForm: every count equals naive 2 n^3 loads, n^2 stores; tiled 2 n^3 / T loads, 2 n^3 / T shared stores, 2 n^3 shared loads, 2 (n / T)^3 barriers. volkovWords is Volkov and Demmel's m n k (1/bn + 1/bm) for the same blocking. Counts are requests issued, not DRAM traffic: a GPU's caches can serve repeated naive reads.","rows":[{"n":64,"kind":"naive","T":16,"globalLoads":524288,"globalStores":4096,"sharedLoads":0,"sharedStores":0,"barriers":0,"fma":262144,"barrierArrivals":0,"workgroups":16,"globalBytes":2113536,"sharedBytes":0,"flops":524288,"intensity":0.2481,"globalLoadsVsNaive":1,"sharedPerWorkgroupBytes":0,"matchesClosedForm":true,"volkovWords":null},{"n":64,"kind":"tiled","T":8,"globalLoads":65536,"globalStores":4096,"sharedLoads":524288,"sharedStores":65536,"barriers":1024,"fma":262144,"barrierArrivals":65536,"workgroups":64,"globalBytes":278528,"sharedBytes":2359296,"flops":524288,"intensity":1.8824,"globalLoadsVsNaive":0.125,"sharedPerWorkgroupBytes":512,"matchesClosedForm":true,"volkovWords":65536},{"n":64,"kind":"tiled","T":16,"globalLoads":32768,"globalStores":4096,"sharedLoads":524288,"sharedStores":32768,"barriers":128,"fma":262144,"barrierArrivals":32768,"workgroups":16,"globalBytes":147456,"sharedBytes":2228224,"flops":524288,"intensity":3.5556,"globalLoadsVsNaive":0.0625,"sharedPerWorkgroupBytes":2048,"matchesClosedForm":true,"volkovWords":32768},{"n":128,"kind":"naive","T":16,"globalLoads":4194304,"globalStores":16384,"sharedLoads":0,"sharedStores":0,"barriers":0,"fma":2097152,"barrierArrivals":0,"workgroups":64,"globalBytes":16842752,"sharedBytes":0,"flops":4194304,"intensity":0.249,"globalLoadsVsNaive":1,"sharedPerWorkgroupBytes":0,"matchesClosedForm":true,"volkovWords":null},{"n":128,"kind":"tiled","T":8,"globalLoads":524288,"globalStores":16384,"sharedLoads":4194304,"sharedStores":524288,"barriers":8192,"fma":2097152,"barrierArrivals":524288,"workgroups":256,"globalBytes":2162688,"sharedBytes":18874368,"flops":4194304,"intensity":1.9394,"globalLoadsVsNaive":0.125,"sharedPerWorkgroupBytes":512,"matchesClosedForm":true,"volkovWords":524288},{"n":128,"kind":"tiled","T":16,"globalLoads":262144,"globalStores":16384,"sharedLoads":4194304,"sharedStores":262144,"barriers":1024,"fma":2097152,"barrierArrivals":262144,"workgroups":64,"globalBytes":1114112,"sharedBytes":17825792,"flops":4194304,"intensity":3.7647,"globalLoadsVsNaive":0.0625,"sharedPerWorkgroupBytes":2048,"matchesClosedForm":true,"volkovWords":262144},{"n":256,"kind":"naive","T":16,"globalLoads":33554432,"globalStores":65536,"sharedLoads":0,"sharedStores":0,"barriers":0,"fma":16777216,"barrierArrivals":0,"workgroups":256,"globalBytes":134479872,"sharedBytes":0,"flops":33554432,"intensity":0.2495,"globalLoadsVsNaive":1,"sharedPerWorkgroupBytes":0,"matchesClosedForm":true,"volkovWords":null},{"n":256,"kind":"tiled","T":8,"globalLoads":4194304,"globalStores":65536,"sharedLoads":33554432,"sharedStores":4194304,"barriers":65536,"fma":16777216,"barrierArrivals":4194304,"workgroups":1024,"globalBytes":17039360,"sharedBytes":150994944,"flops":33554432,"intensity":1.9692,"globalLoadsVsNaive":0.125,"sharedPerWorkgroupBytes":512,"matchesClosedForm":true,"volkovWords":4194304},{"n":256,"kind":"tiled","T":16,"globalLoads":2097152,"globalStores":65536,"sharedLoads":33554432,"sharedStores":2097152,"barriers":8192,"fma":16777216,"barrierArrivals":2097152,"workgroups":256,"globalBytes":8650752,"sharedBytes":142606336,"flops":33554432,"intensity":3.8788,"globalLoadsVsNaive":0.0625,"sharedPerWorkgroupBytes":2048,"matchesClosedForm":true,"volkovWords":2097152},{"n":512,"kind":"naive","T":16,"globalLoads":268435456,"globalStores":262144,"sharedLoads":0,"sharedStores":0,"barriers":0,"fma":134217728,"barrierArrivals":0,"workgroups":1024,"globalBytes":1074790400,"sharedBytes":0,"flops":268435456,"intensity":0.2498,"globalLoadsVsNaive":1,"sharedPerWorkgroupBytes":0,"matchesClosedForm":true,"volkovWords":null},{"n":512,"kind":"tiled","T":8,"globalLoads":33554432,"globalStores":262144,"sharedLoads":268435456,"sharedStores":33554432,"barriers":524288,"fma":134217728,"barrierArrivals":33554432,"workgroups":4096,"globalBytes":135266304,"sharedBytes":1207959552,"flops":268435456,"intensity":1.9845,"globalLoadsVsNaive":0.125,"sharedPerWorkgroupBytes":512,"matchesClosedForm":true,"volkovWords":33554432},{"n":512,"kind":"tiled","T":16,"globalLoads":16777216,"globalStores":262144,"sharedLoads":268435456,"sharedStores":16777216,"barriers":65536,"fma":134217728,"barrierArrivals":16777216,"workgroups":1024,"globalBytes":68157440,"sharedBytes":1140850688,"flops":268435456,"intensity":3.9385,"globalLoadsVsNaive":0.0625,"sharedPerWorkgroupBytes":2048,"matchesClosedForm":true,"volkovWords":16777216},{"n":1024,"kind":"naive","T":16,"globalLoads":2147483648,"globalStores":1048576,"sharedLoads":0,"sharedStores":0,"barriers":0,"fma":1073741824,"barrierArrivals":0,"workgroups":4096,"globalBytes":8594128896,"sharedBytes":0,"flops":2147483648,"intensity":0.2499,"globalLoadsVsNaive":1,"sharedPerWorkgroupBytes":0,"matchesClosedForm":true,"volkovWords":null},{"n":1024,"kind":"tiled","T":8,"globalLoads":268435456,"globalStores":1048576,"sharedLoads":2147483648,"sharedStores":268435456,"barriers":4194304,"fma":1073741824,"barrierArrivals":268435456,"workgroups":16384,"globalBytes":1077936128,"sharedBytes":9663676416,"flops":2147483648,"intensity":1.9922,"globalLoadsVsNaive":0.125,"sharedPerWorkgroupBytes":512,"matchesClosedForm":true,"volkovWords":268435456},{"n":1024,"kind":"tiled","T":16,"globalLoads":134217728,"globalStores":1048576,"sharedLoads":2147483648,"sharedStores":134217728,"barriers":524288,"fma":1073741824,"barrierArrivals":134217728,"workgroups":4096,"globalBytes":541065216,"sharedBytes":9126805504,"flops":2147483648,"intensity":3.969,"globalLoadsVsNaive":0.0625,"sharedPerWorkgroupBytes":2048,"matchesClosedForm":true,"volkovWords":134217728}],"verdict":["At n = 1,024: naive 2,147,483,648 global loads, tiled T 8 268,435,456, tiled T 16 134,217,728: 1/8 and 1/16 of naive. All 15 rows match the closed forms: true.","Arithmetic intensity (flops per global byte, stores included): naive 0.248, 0.249, 0.25, 0.25, 0.25; T 8 1.882, 1.939, 1.969, 1.984, 1.992; T 16 3.556, 3.765, 3.879, 3.938, 3.969 for n = 64, 128, 256, 512, 1024. The limits are 1/4, 2 and 4: each loaded float feeds 1, 8 or 16 multiply-adds, two loads per multiply-add.","The price: 2,147,483,648 shared loads (2 n^3, as many as the naive kernel's global loads), 524,288 barriers at T 16 and 4,194,304 at T 8, and 2048 bytes of workgroup memory per workgroup at T 16 (WebGPU guarantees 16384)."]},"correctness":{"about":"Correctness of the emulated float32 result against a float64 CPU reference on the same inputs (uniform in [-1, 1), seed 1). Tolerance, per element: gamma_n * sum_k |a_ik b_kj| with gamma_n = n u / (1 - n u), u = 2^-24, the worst-case error of a float32 inner product of length n in any summation order (Higham, Accuracy and Stability of Numerical Algorithms, equation 3.5), so a GPU that fuses multiply-adds or sums in another order is held to the same bound. maxRatio is the largest error over its bound. sameBitsAsNaive: the tiled kernel adds the n products in the same order as the naive one, so the float32 results are identical. stale is the tiled kernel without its first barrier, run under the worst schedule that allows (every read sees the previous phase's tile).","rows":[{"n":64,"kind":"naive","T":16,"ok":true,"mismatches":0,"maxErr":0.00000314,"maxRatio":0.0439,"gamma":0.00000381,"sameBitsAsNaive":true},{"n":64,"kind":"tiled","T":8,"ok":true,"mismatches":0,"maxErr":0.00000314,"maxRatio":0.0439,"gamma":0.00000381,"sameBitsAsNaive":true},{"n":64,"kind":"tiled","T":16,"ok":true,"mismatches":0,"maxErr":0.00000314,"maxRatio":0.0439,"gamma":0.00000381,"sameBitsAsNaive":true},{"n":64,"kind":"stale","T":8,"ok":false,"mismatches":4096,"maxErr":3.766,"maxRatio":54908.8714,"gamma":0.00000381,"sameBitsAsNaive":null},{"n":64,"kind":"stale","T":16,"ok":false,"mismatches":4096,"maxErr":4.8237,"maxRatio":77448.5558,"gamma":0.00000381,"sameBitsAsNaive":null},{"n":128,"kind":"naive","T":16,"ok":true,"mismatches":0,"maxErr":0.00000583,"maxRatio":0.0241,"gamma":0.00000763,"sameBitsAsNaive":true},{"n":128,"kind":"tiled","T":8,"ok":true,"mismatches":0,"maxErr":0.00000583,"maxRatio":0.0241,"gamma":0.00000763,"sameBitsAsNaive":true},{"n":128,"kind":"tiled","T":16,"ok":true,"mismatches":0,"maxErr":0.00000583,"maxRatio":0.0241,"gamma":0.00000763,"sameBitsAsNaive":true},{"n":128,"kind":"stale","T":8,"ok":false,"mismatches":16381,"maxErr":4.4341,"maxRatio":18764.83,"gamma":0.00000763,"sameBitsAsNaive":null},{"n":128,"kind":"stale","T":16,"ok":false,"mismatches":16384,"maxErr":6.1418,"maxRatio":25134.931,"gamma":0.00000763,"sameBitsAsNaive":null},{"n":256,"kind":"naive","T":16,"ok":true,"mismatches":0,"maxErr":0.0000124,"maxRatio":0.0128,"gamma":0.0000153,"sameBitsAsNaive":true},{"n":256,"kind":"tiled","T":8,"ok":true,"mismatches":0,"maxErr":0.0000124,"maxRatio":0.0128,"gamma":0.0000153,"sameBitsAsNaive":true},{"n":256,"kind":"tiled","T":16,"ok":true,"mismatches":0,"maxErr":0.0000124,"maxRatio":0.0128,"gamma":0.0000153,"sameBitsAsNaive":true},{"n":256,"kind":"stale","T":8,"ok":false,"mismatches":65481,"maxErr":4.6493,"maxRatio":4805.0701,"gamma":0.0000153,"sameBitsAsNaive":null},{"n":256,"kind":"stale","T":16,"ok":false,"mismatches":65498,"maxErr":5.7995,"maxRatio":5708.879,"gamma":0.0000153,"sameBitsAsNaive":null},{"n":512,"kind":"naive","T":16,"ok":true,"mismatches":0,"maxErr":0.0000336,"maxRatio":0.0083,"gamma":0.0000305,"sameBitsAsNaive":true},{"n":512,"kind":"tiled","T":8,"ok":true,"mismatches":0,"maxErr":0.0000336,"maxRatio":0.0083,"gamma":0.0000305,"sameBitsAsNaive":true},{"n":512,"kind":"tiled","T":16,"ok":true,"mismatches":0,"maxErr":0.0000336,"maxRatio":0.0083,"gamma":0.0000305,"sameBitsAsNaive":true},{"n":512,"kind":"stale","T":8,"ok":false,"mismatches":261229,"maxErr":4.1772,"maxRatio":1067.0437,"gamma":0.0000305,"sameBitsAsNaive":null},{"n":512,"kind":"stale","T":16,"ok":false,"mismatches":261546,"maxErr":6.2397,"maxRatio":1541.6402,"gamma":0.0000305,"sameBitsAsNaive":null},{"n":1024,"kind":"naive","T":16,"ok":true,"mismatches":0,"maxErr":0.0000818,"maxRatio":0.0055,"gamma":0.000061,"sameBitsAsNaive":true},{"n":1024,"kind":"tiled","T":8,"ok":true,"mismatches":0,"maxErr":0.0000818,"maxRatio":0.0055,"gamma":0.000061,"sameBitsAsNaive":true},{"n":1024,"kind":"tiled","T":16,"ok":true,"mismatches":0,"maxErr":0.0000818,"maxRatio":0.0055,"gamma":0.000061,"sameBitsAsNaive":true},{"n":1024,"kind":"stale","T":8,"ok":false,"mismatches":1034756,"maxErr":4.8112,"maxRatio":305.2013,"gamma":0.000061,"sameBitsAsNaive":null},{"n":1024,"kind":"stale","T":16,"ok":false,"mismatches":1038595,"maxErr":6.0818,"maxRatio":393.5309,"gamma":0.000061,"sameBitsAsNaive":null}],"verdict":"Naive and tiled pass at every size, with the largest error at most 0.0439 of the bound (the bound is a worst case; random signs cancel); tiled results equal naive bit for bit: true. The stale kernel fails at every size: n 64 T 8 4,096 of 4,096 wrong; n 64 T 16 4,096 of 4,096 wrong; n 128 T 8 16,381 of 16,384 wrong; n 128 T 16 16,384 of 16,384 wrong; n 256 T 8 65,481 of 65,536 wrong; n 256 T 16 65,498 of 65,536 wrong; n 512 T 8 261,229 of 262,144 wrong; n 512 T 16 261,546 of 262,144 wrong; n 1024 T 8 1,034,756 of 1,048,576 wrong; n 1024 T 16 1,038,595 of 1,048,576 wrong; its smallest error-over-bound is 305.2x at n 1024."},"controls":{"about":"Controls for the checker at n = 64 (seed 3): the tiled kernel passes; the exact answer rounded to float32 passes; one element moved by 0.001 fails with exactly one mismatch; the tiled kernel's own output fails when the tolerance is set to zero, so the bound is doing work and is not loose enough to pass anything.","tiled":true,"roundedReference":true,"oneElementOff":1,"zeroTolerance":false},"film":{"about":"What the films draw (the cannery: a crew carries a tray of cans to the bench instead of walking to the store for every can). trips: for one output element, reads from global memory: naive 2n, tiled 2n / T. perElement: how many times each element of A (and of B) is fetched from global memory: naive n, tiled n / T. grid: n = 64 with T = 16 is a 4 x 4 grid of workgroups; schedule is workgroup (x 1, y 2), phase by phase: the A tile and B tile it copies (rows and columns, inclusive), and the work each phase does. ratios: tiled over naive global loads for every n. intensity: flops per global byte at n = 64.","trips":[{"n":64,"naive":128,"tiled8":16,"tiled16":8},{"n":128,"naive":256,"tiled8":32,"tiled16":16},{"n":256,"naive":512,"tiled8":64,"tiled16":32},{"n":512,"naive":1024,"tiled8":128,"tiled16":64},{"n":1024,"naive":2048,"tiled8":256,"tiled16":128}],"perElement":[{"n":64,"naive":64,"tiled8":8,"tiled16":4},{"n":128,"naive":128,"tiled8":16,"tiled16":8},{"n":256,"naive":256,"tiled8":32,"tiled16":16},{"n":512,"naive":512,"tiled8":64,"tiled16":32},{"n":1024,"naive":1024,"tiled8":128,"tiled16":64}],"grid":{"n":64,"T":16,"workgroups":16,"invocationsPerWorkgroup":256},"schedule":[{"phase":0,"aTile":{"rows":[32,47],"cols":[0,15]},"bTile":{"rows":[0,15],"cols":[16,31]},"globalLoads":512,"sharedStores":512,"sharedLoads":8192,"fma":4096,"barriers":2},{"phase":1,"aTile":{"rows":[32,47],"cols":[16,31]},"bTile":{"rows":[16,31],"cols":[16,31]},"globalLoads":512,"sharedStores":512,"sharedLoads":8192,"fma":4096,"barriers":2},{"phase":2,"aTile":{"rows":[32,47],"cols":[32,47]},"bTile":{"rows":[32,47],"cols":[16,31]},"globalLoads":512,"sharedStores":512,"sharedLoads":8192,"fma":4096,"barriers":2},{"phase":3,"aTile":{"rows":[32,47],"cols":[48,63]},"bTile":{"rows":[48,63],"cols":[16,31]},"globalLoads":512,"sharedStores":512,"sharedLoads":8192,"fma":4096,"barriers":2}],"ratios":[{"n":64,"tiled8":0.125,"tiled16":0.0625},{"n":128,"tiled8":0.125,"tiled16":0.0625},{"n":256,"tiled8":0.125,"tiled16":0.0625},{"n":512,"tiled8":0.125,"tiled16":0.0625},{"n":1024,"tiled8":0.125,"tiled16":0.0625}],"intensity":{"naive":0.2481,"tiled8":1.8824,"tiled16":3.5556,"limits":{"naive":0.25,"tiled8":2,"tiled16":4}},"bytesPerFloat":4},"lab":{"about":"The lab's measurement, run only in a browser on the reader's own GPU through WebGPU (system/school/tiling-gpu.mjs, runKernel). For each kernel, tile and size it first checks C against the float64 CPU reference within gamma_n sum |a||b| (the whole matrix up to n = 512, the first 64 rows above that, and it says which), then times reps dispatches after one warm-up and reports the median wall time from submit to onSubmittedWorkDone (submission overhead included) and 2 n^3 / time in Gflop/s. Every result is labelled with the adapter's vendor, architecture and description. It is not run by the fill or the tests.","sizes":[64,128,256,512,1024],"kernels":[["naive",16],["tiled",8],["tiled",16]],"reps":9,"warmup":1,"rowsCheckedAbove512":64,"measures":["medianMs per dispatch","gflops = 2 n^3 / medianMs","tiled over naive speed ratio at each n","checker result for every run, including the stale kernel"],"predicts":"The emulator counts global loads: the tiled kernel issues 1/T of the naive kernel's (1/8 and 1/16). A measured speed ratio well below 8 or 16 says the naive kernel's repeated reads are served by caches, that the kernels are bound by something other than global loads, or both. The model gives counts, not times."},"speed":{"about":"Milliseconds in Node (V8), one process at below-normal priority, on the fill machine's CPU. The emulator runs every multiply-add in float32 and counts every access, so it is far slower than a GPU. A lab runs it in 40 ms slices of whole workgroups (slice: milliseconds per workgroup over 16 workgroups after 2 untimed ones, and how many fit in 40 ms; a naive workgroup does T^2 dot products of length n). runs: the wall time of each full emulation in this fill, and the float64 reference.","slice":[{"kind":"naive","T":16,"n":256,"msPerWorkgroup":0.4861,"workgroupsPer40ms":82},{"kind":"naive","T":16,"n":1024,"msPerWorkgroup":0.9834,"workgroupsPer40ms":40},{"kind":"tiled","T":8,"n":256,"msPerWorkgroup":0.0382,"workgroupsPer40ms":1047},{"kind":"tiled","T":8,"n":1024,"msPerWorkgroup":0.2551,"workgroupsPer40ms":156},{"kind":"tiled","T":16,"n":256,"msPerWorkgroup":0.165,"workgroupsPer40ms":242},{"kind":"tiled","T":16,"n":1024,"msPerWorkgroup":0.7921,"workgroupsPer40ms":50},{"kind":"stale","T":16,"n":256,"msPerWorkgroup":0.162,"workgroupsPer40ms":246},{"kind":"stale","T":16,"n":1024,"msPerWorkgroup":0.8141,"workgroupsPer40ms":49}],"runs":[{"n":64,"msReference":2.7801,"naive16":5.1613,"tiled8":4.3577,"tiled16":0.654,"stale8":1.6437,"stale16":0.7944},{"n":128,"msReference":6.0751,"naive16":6.4965,"tiled8":9.6758,"tiled16":5.1149,"stale8":5.4459,"stale16":5.3184},{"n":256,"msReference":30.6143,"naive16":58.0986,"tiled8":66.9297,"tiled16":46.395,"stale8":44.7886,"stale16":41.5364},{"n":512,"msReference":320.0079,"naive16":365.4973,"tiled8":373.1963,"tiled16":381.0128,"stale8":410.2912,"stale16":336.3471},{"n":1024,"msReference":2304.8227,"naive16":3724.5004,"tiled8":3587.1314,"tiled16":3042.8915,"stale8":3513.4956,"stale16":2895.2298}],"fillSeconds":21.801},"seconds":21.9}