Skip to content

Living Weights on Qwen4 dense, step 2: the native CUDA and Metal learner (experimental, --slide refused) - #2054

Draft
ashhart wants to merge 3 commits into
mainfrom
lead/1804-qwen4-lw-step2
Draft

ashhart wants to merge 3 commits into
mainfrom
lead/1804-qwen4-lw-step2

Conversation

@ashhart

@ashhart ashhart commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Wire the native Qwen4 dense pocket and learner on CUDA and Metal through the shared Living Weights family,
fitting policy, shard writer and save guard. Both native adapters feed a bounded native trainer from the serving
forward's recorded HC/GDN/QSA nodes and selected IDs. Empty and untaught pockets encode no additional GPU work;
appended checkpoint neurons are loaded separately while original packed rows and columns remain exact.

Experimental draft for #1804. Public --slide remains refused by name on qwen4 and qwen4_text
on CUDA and Metal until all six raw BF16 gradient cells pass and device qualification succeeds. Depends on the host selected-attention backward in #2021. The internal experimental writable layout is
MLX affine4/group64 MLPs, BF16 or affine4/group64 other training projections, 2 through 64 layers, and no PLE.
PLE backward and unsupported quantization/layouts refuse explicitly. Other family adapters retain their refusals.

Host validation includes actual native launch wrappers with recording devices, nonzero native forwards,
untaught-stream isolation, failed-completion retention, atomic pocket replacement, shared learner refusal,
shared save failure/retry/reopen, independent GDN/QSA and full-graph frozen-store finite differences, accumulator
witnesses and seven real source mutations. The final focused host/contract graph passes 293 tests; admission
checks pass 16, and the separate host/shared/golden/fuzz graph passes 166 with zero unexpected golden differences.
The plain Linux cross-build passes. Lean adds zero problems over the pinned 424 inherited findings.

The raw BF16 whole-graph rule-v2 panel failed five of six tiny synthetic cells, with unchanged selected IDs.
Those failures are retained and reproducible with zig build diagnose-qwen4-raw-gradient-host -j1. The passing frozen-store derivative checks do not resolve or replace them.
Successful real-model keep/recall/removal, raw gradient qualification, cross-backend saves, off-path tokens/speed,
CUDA compiler/device limits and complete Metal numeric/resource qualification are still owed. The full
device-bearing standard test graph is also owed on this tree. No support or performance qualification
is inferred from compilation, recording devices, or the ideal host attention formula.

The native four-kernel selected-attention chain now agrees with the independent host backward from #2021
at 17 causal rows, 24 query heads, four KV heads and width 256, including mixed sparse/dense selections
and early key/value gradients from answer-only loss. This comparison uses continuous FP32 softmax probabilities;
it does not qualify CUDA probability rounding. The off-path dense input store is gated on an active learner,
and the primitive contract callback table has static storage. The follow-up host graph passes 175 tests in 47 steps; three planted source controls fail and restore,
and lean adds zero findings. No device-bearing full-graph rerun applies to this follow-up tree.

Connect CUDA and Metal pockets, native causal pullbacks and shared persistence with bounded admission and completion-safe ownership. Add recording-device, lifecycle, save and precision contracts, with explicit raw directional diagnostics and pending device qualification.
Keep the public alias, record order and existing resource-refusal test unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant