Skip to content

glm cuda: EXL3 mcg experts at any whole-bit width, mixed by projection - #2094

Open
MiaAI-Lab wants to merge 4 commits into
ashhart:mainfrom
MiaAI-Lab:main
Open

MiaAI-Lab wants to merge 4 commits into
ashhart:mainfrom
MiaAI-Lab:main

Conversation

@MiaAI-Lab

Copy link
Copy Markdown
Contributor

Refs #2093.

GLM's CUDA family admits only 4-bit EXL3 mcg routed experts; anything else is error.NotExl3Mcg4. This takes each tensor's half-bit width from its trellis shape (whole bits 1..8, mixed per projection inside one layer), templates the decode and prompt expert kernels on it, and keeps the 4-bit path exactly today's: the upstream 4-bit loader runs verbatim in a test and must upload equal bytes; the 4-bit entry points keep their SASS at nvcc 13.3.

Four commits, one concern each:

  1. glm cuda: EXL3 mcg experts at any whole-bit width, mixed by projection - admission keeps the codebook check and verifies each width's range; the loader reads widths from the trellises and holds them per projection; scratch stays width-independent.
  2. tests: GLM EXL3 at mixed widths, synthetic and a checkpoint's own layers - synthetic mixed-width rigs and glm-exl3-layer, one real checkpoint layer's experts against float64 unpacks.
  3. exl3 cuda: the common prompt kernels take 3-bit mcg gate/up and down, by projection group - tf_exl3_prompt_gu_k6 / tf_exl3_prompt_down_k6; contract records with 13.0 receipts pending and 13.3's measured usage.
  4. tests: exl3-prompt-k6, the 3- and 4-bit prompt kernels at GLM's shapes - the k6 pair against the 4-bit pair's references, edges and shapes.

Gates so far (host, no GPU claims): zig build install green with -Dnvcc -Dsm=80,86,89,120,121 (nvcc 13.3.1, Zig 0.17.0 per .zig-version); test-cuda-host 4705/4708 with 2 fails in drama_voice_tool.zig, untouched by this branch (pre-exist on main's tree here); tools/zig/lean_check.py zero findings over main's baseline count; zig fmt clean. Kernel-contract and per-SM launch receipts on sm_120 hardware post below when the run completes; the 3-bit launches' 13.0 resources stay pending records per the registry's convention, 13.3 measured.

Metal, XPU, other codebooks and other families: untouched.

MiaAI-Lab and others added 4 commits November 4, 2026 11:29
Admission reads EXL3 mcg checkpoints at 1 to 8 bits: "bits" may be absent
(a mixed checkpoint gives "avg_bits"), and a given one must lie in range.
The loader takes each trellis's width from its last dimension, slices TP
shares by that width's tile bytes, places each expert's trellis at its own
offset and uploads a width table a projection (one table, as before, when
the three agree). Layer gets uniform_k2_gu and uniform_k2_d; uniform_k2
stays nonzero only when every projection agrees, so a 4-bit layer uploads
upstream's bytes and keeps the prompt kernels.

Scratch is sized by the path each layer takes: the prompt kernels' when
they take it, else the grouped path's, the buffers taking the largest over
every layer and the MTP layer (a mixed layer's prompt chunks run grouped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015bV57ikYD5vn1va1KmaPjw
glm-exl3 runs Mia-AiLab's 4-bit experts, 3-bit gate and up over 4-bit
down, and 2/3/4-bit gate/up/down, each projection on its own width
table, decode and prompt chunks (1030 rows, two calls) against float64,
rows as alone, NaN kept to its row.

glm-exl3-layer <ckpt> <layer> <rank> <world> loads one checkpoint layer
through cuda_exl3.load at a rank's TP share and runs it (decode windows
and a prompt chunk) against float64 matrices decoded from the whole
trellises (core's unpack, cut, rotated, scaled), with a planted control
(gate and up swapped) that must fail the same tolerance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015bV57ikYD5vn1va1KmaPjw
… by projection group

tf_exl3_prompt_gu_k6 and tf_exl3_prompt_down_k6 run the transposed prompt
kernels on 96-byte (3-bit) mcg tiles, with tf_exl3_prompt_gu/_down's
arguments. The kernels are templated on the half-bit width; the 4-bit
entry points keep their SASS byte for byte on sm_80/86/89/120/121 (nvcc
13.3). The down kernel reads its item in the kernel itself so that its
launch bound still limits the thread arithmetic.

Layer gains uniform_k2_gu and uniform_k2_d (the loader's per-group
verified widths, uniform_k2 when zero). The prompt path takes a layer
whose gate/up width and down width are each 6 or 8 and launches each
group's own kernel; widths that disagree with uniform_k2 stay unverified
and keep the grouped path. Scratch is width-independent.

Contract: both launches registered pending nvcc 13.0's receipts, with
nvcc 13.3's five-SM usage measured by kernel-contract-limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015bV57ikYD5vn1va1KmaPjw
tf-cuda-test exl3-prompt-k6 [DIR|none] [BASE]: one synthetic GLM layer
(4096 x 1024 per rank, 288 experts, top-8) at every gate/up and down
width pair through the prompt kernels against runSlots and against
float64 (glm_exl3_tests' tolerance) at 1, 64, 1000 and 4096 rows; a
row's bytes in one call, in 1024-row calls and with a rotation a pick;
BASE, an earlier exl3_experts fatbin, must give the 4-bit kernels'
bytes; then one layer's time. DIR adds layer 4 of a GLMQ checkpoint
(rank 0 of 2) the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015bV57ikYD5vn1va1KmaPjw
@garvrao80

Copy link
Copy Markdown
Contributor

The #2031 native Linux sweep on main 7eae7dc00 found that test-exl3-prompt-host stops at the glm.experts.Kernels initializer: missing struct field: b32 (zig/exl3_prompt_host.zig:59). Current main's Kernels now includes the block-32 expert table.

Since this PR updates the same fixture, could you account for that field when bringing it onto current main? Draft CI-coverage PR #2095 keeps this check included and records the compile failure as a prerequisite. Its paired explicit-step/aggregate runs reproduce the same failure. I've kept the EXL3 fixture repair with this work rather than adding a competing patch.

@ashhart

ashhart commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Thanks @MiaAI-Lab, the host side reads well: widths come from the trellis shape, admission still refuses by name, and the real-layer float64 check has its own planted control. To land it on the terms we set on #2093, a few things are still owed:

  1. The sm_120 device evidence you mentioned: the CUDA contract SUMMARY with the two _k6 kernels registered, a planted control for them shown red, and their bounds at the served shapes.
  2. The 4-bit SASS byte for byte at nvcc 13.0 as well as 13.3, since the release fatbins build with 13.0, and the _k6 launches' 13.0 receipts.
  3. A served run of the width-mixed checkpoint at a pinned revision: drafted equal to plain, concurrent equal to alone.
  4. As @garvrao80 noted, exl3_prompt_host.zig's glm.experts.Kernels needs main's new b32 field, or the step doesn't compile.
  5. A CHANGELOG line.

On our side, we'll check on our GB10s that the pinned 4-bit checkpoint's replies equal main's, with decode level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants