Repository navigation
Conversation
A decode window's routed experts run on the same item plan the prompt path and the NVFP4/FP8 MoE path already use, instead of tf_exl3_group plus the maxu x member-tile grid: the 9-slot picks are grouped by the Router into items of at most 16 pairs and the grouped GEMVs launch in standard mode on grids maxItems x (n/128) x (mats*sk), so at 64 rows ~325 blocks do work instead of ~55k and the one-block O(experts x pairs) group scan becomes the ranked plan. No 9->8 compaction and no copy back: the down epilogue writes ey's routed slot rows in place through tf_exl3_down_combine. The grouped kernel and tf_exl3_down_combine already take item triples in standard mode and a caller-chosen slot-row base, so no kernel change is needed: the plan reaches the decode as the same Items the prompt path takes, with the family's Router and the plan buffer the block already holds. Bit-exact: the glm-exl3-items GPU check compares the item plan's ey against the plain grouped decode byte for byte over distinct and lockstep (16/32/48/64-row) windows, and calls the plain launch shape directly so both paths stay covered. A wrong plan (a members offset off by one, a bad item first/count, the shared item not skipped) routes a slot to the wrong expert, so ey's bytes differ. A plan is refused (error.TileNotBuilt) for a non-mcg codebook, a non-zero wts or out, a zero ey, or a null plan array: the item kernels read one codebook a launch and write the caller's slot rows. Files: - zig/src/cuda/exl3_experts.zig: Items, runWith on the plan (standard mode), and fusedBytes/fusedScratch for the 9-slot layout without y. - zig/src/families/glm/cuda_exl3.zig: the decode branch on decodeItems, and scratchBytes sizing a decode arena with decodeBytes, the larger of the plain 8-slot layout and the 9-slot one, so no pool changes at GLM's shapes. - zig/tests/cuda/glm_exl3_items_tests.zig: the items decode against the plain launch, case by case.
|
Thanks @BadAd84, this is a neat change: decode now reuses the item plan and grouped kernels the prompt path already I read it on current main. It applies cleanly, and the lean check is unchanged.
Your 0-of-3,984 test is the right check. Two small things:
We don't have a GLM EXL3 checkpoint on our GB10s yet, so your serve run against main's base is the device receipt for now: equal token hashes and decode speed at 1 to 8 streams with |
The staged check only ran dims 512 at widths 256/640. Adds dims 4096 at the two widths a rank really meets (768 and 1024), one distinct window and one lockstep window each, so the item plan is exercised at the shape the serve uses (reviewer ask on ashhart#2044). Dims is threaded through the weights/layer/pack helpers; the small cases are unchanged and the comparison count grows from 36 to 44.
|
Thanks @BadAd84 for adding the served-size case so quickly. Dims 4,096 at widths 768 and 1,024 is the shape a rank actually runs, so that ask is done. Three small things before it lands:
The engine code is unchanged from the version we read, and we're happy with it. |
…derived Reviewer on ashhart#2044: lock = rows >= 16 turned the 17-row window lockstep, and it is the only distinct window that spans two 16-pair items. Both shapes now carry an explicit case list (base_cases, served_cases) instead of deriving the flag from the row count, so 17 rows is distinct again and the served-size shape keeps 8 distinct / 64 lockstep. The RESULT line counts the NaN tallies as well as the byte comparisons, so it prints 66 for this set: 18 base cases x 3 + 4 served cases x 3.
|
Thanks, both are fixed.
|
|
Thanks @BadAd84, that's all three test points handled: the one-line comment brings lean back to its count, 17 rows is Two text fixes when you next push: the PR body's "0 of 3,984" should read 0 of 66, and commit 2's "36 to 44" is Then it's the serve run on main's base, plus the |
Dims 512 alone only asked the item plan about small matrices. The check now also runs dims 4096 at the two widths a rank meets, 768 and 1024, one distinct window and one lockstep window each, with dims threaded through the weights, layer and pack helpers. Each shape carries an explicit case list, so 17 rows stays distinct: it is the only distinct window that spans two 16-pair items. The RESULT line counts the NaN tallies beside the byte comparisons, so this set prints 66 checks: 18 base cases and 4 served cases at three checks each.
4cda259 to
06fdcc8
Compare
|
Squashed commits 2-4 into one, with the count corrected to 66. The only content change is two comment lines that drop the reviewer notes. |
|
Thanks @BadAd84, the squash is clean: the code is exactly what we reviewed, and only the two comment lines changed. Commit 2's message now gives the 66 and how it adds up. One text fix left: the body's "Proof it's the same bits" section still says 0 of 3,984; it should say 0 of 66. After that, we only need the serve run on main's base and the |
|
|
|
Thanks @BadAd84. The body now says 0 of 66, and the |
What's slow today
A decode window's routed experts are grouped by
tf_exl3_group(a 1024-thread block that scans every expert against every pair) and launched overmin(pairs, E) x (n/128) x (mats*sk*ceil(rows/16))blocks. At the widths a shared verify window reaches, most blocks have nothing to do: at 64 rows and the wide 768-column share that is 55,296 blocks, about 800 of them with work. The group kernel costs 0.054 ms a layer at 24 rows and 0.134 at 64, against 0.005-0.010 for a ranked plan.Measured on one GB10 with both engines' kernels verbatim over one synthetic EXL3 weight set, per layer at the wide share, every row routed alike (a batching workload):
44 distinct experts a window: 1.033 -> 0.958 ms at 24 rows and 1.221 -> 1.058 at 48 (a plan-based launch: 0.915 / 0.990).
What the change is
The prompt path already groups pairs into items of at most 16 through
grouped.Router, and the grouped kernels already read item triples in theirstandardmode (exl3_grouped.cuh:standard,skip, direct pair indices). Decode only has to use them, so this is plumbing, not a kernel change:exl3_experts.runWith(..., ey, it: ?Items)runs the same single implementation for both paths;items = it != nullis the only switch, andrunis a one-line delegation to it withnull. The plain branch inside is unchanged line for line.Routerthe prompt path uses and the same plan buffer - sodecodeItemsruns for every non-prompt window.decodeBytes, the larger of the two), so no pool changes at GLM's shapes.error.TileNotBuiltfor a non-mcg codebook, a non-zerowts/out, a zeroey, or a null plan array: the item kernels read one codebook a launch and write the caller's slot rows.Nothing under
zig/kernels/changes.Proof it's the same bits
tf-cuda-test glm-exl3-itemscompares the item decode's packedeyslot rows against the plain launch's own packed rows, over distinct windows (1, 2, 3, 8, 17 rows) and lockstep windows (16, 32, 48, 64 rows) at two widths, and again at GLM-5.3-Flash's served size (dims 4096 at widths 768 and 1024), with a plain-twice control and a NaN check: 0 of 66 checks differ on our tree, where a mutant that reads the plan's members one slot off fails exactly the item comparisons and none of the other cases.What it's worth
Our fork's serve, TP=3 on three GB10s, production's probe, ABBA, every reply digest equal, the pool at boot equal:
One-stream prose +1.2%, cold and next-turn prompt cells inside their floors. A run priced on main's own base is running now and will be posted with the numbers.
Files
zig/src/cuda/exl3_experts.zig(+105/-30):Items,runWith, the layout split, the standard-mode launches,down_combineoney.zig/src/families/glm/cuda_exl3.zig(+26):decodeItemscalled for every non-prompt window,decodeBytes.zig/tests/cuda/glm_exl3_items_tests.zig(new, 209) andmain.zig(+3): the check.Checks
fmt,check-native-cuda-host,compile-cuda-tests,test-cuda-host, every sm_121a fatbin through nvcc, and lean reports 0 new against main.glm-exl3-itemsruns on any GB10.