Skip to content

A submission is exactly the residual, checked header first; miners never start from an untrained residual - #290

Merged
george-connito merged 7 commits into
feat/qwen-06-residual-trainfrom
feat/qwen-07-residual-shard
Oct 14, 2026
Merged

george-connito merged 7 commits into
feat/qwen-06-residual-trainfrom
feat/qwen-07-residual-shard

Conversation

@george-connito

Copy link
Copy Markdown
Collaborator

Into staging_v3, on top of PR 06 (#289). This is PR 07 of the Qwen3.6 stack, which merges in order.

Do not merge alone, and do not ship to validators without PR 08. At this PR, the validator already accepts
residual submissions, but two things are still missing until PR 08:

  • Its round base_shard is still the old expert-group shard, so a round's baseline loss can be measured with the
    previous miner's residual still loaded.
  • It doesn't reinstall the residual when a task switch changes K.

Nodes also still fail at boot on the missing expert_assignment.json (see PR 03) until PR 08 deletes the reader.

Why

Miners now train only residual experts (PRs 04 and 06). Until this PR, every residual submission failed validation:

  • The expert-group check read ...mlp.residual_experts.0... as no expert at all.
  • The save dropped every residual tensor as "non-expert", so at the PR 06 tip a miner wrote an empty shard.

A submission must be exactly the residual, in protocol shapes and dtypes, and a real one.

Real means trained. A residual whose down_proj is all zero (or whose gate and up are both zero) scores
exactly like the base, and nothing in it can ever learn. If such a shard won a round and was published, every
miner would restart from it and get no gradient. So nodes refuse an untrained residual both as a submission and as
a starting point.

Nothing is shipped as a starting point. Each node creates its own residual with PR 04's
install_residual_experts, and with no published winner a miner keeps that fresh draw, or its own last trained
checkpoint.

The change

  • residual_shapes(config) (residual.py):
    • It gives every residual key with its shape and dtype for the family and the task's K. Experts are bf16 and the
      router is fp32.
    • It reads the family's config.json (cached; the revision is a pinned sha), builds a meta-device skeleton, and
      attaches the residual through the same helper install_residual_experts uses.
    • Qwen K=1: 40 layers, 160 keys, 252 MB. DeepSeek-V2 K=1: 26 layers (layer 0 is dense), 104 keys, 450 MB.
    • It takes 0.02–0.2 s per call.
  • save_residual_shard(model, dest_dir, group_id) writes the model's residual experts and routers, each in its
    own dtype, as the group's shard (expert_group_shard_name). save_checkpoint takes group_id and uses it; its
    four expert-group parameters are gone.
  • verify_residual_shard(path, expected) returns the shard's tensors, or None:
    1. The header must equal the spec exactly: same keys, shapes and dtypes. It is checked before any tensor is
      read; a refusal takes about 1 ms.
    2. Every tensor must be finite.
    3. Every expert must be trained: its down_proj is not all zero, and its gate and up are not both all zero.
      This never trips on honest training, because the gate q is above 0 on every token, so every down_proj moves
      at the first step.
    • The header and tensors are read in one safe_open. A corrupt, truncated or missing file returns None and
      never raises.
    • A full pass on a real Qwen shard takes about 0.4 s.
  • ResidualCfg refuses threshold / temperature >= 80. From about 88, the fp32 gate underflows to exactly
    0, and honest training could leave a down_proj at zero. v1 is 16. A live switch to such a task is refused
    and rolled back.
  • Validation.
    • ModelCheckpoint.validate(expected) checks the signature, then the shard, then the hash.
    • Both the validator's scoring (validate_miner_submission) and the miner's fetch use it.
    • A refused shard is bad_residual_shard: eval status code 14. Codes 5 (incomplete_expert_set) and 13
      stay in the table as retired.
    • The worker now records validation failures on the miner's status gauge. Before, it only logged them, so
      signature and hash failures (codes 6, 3 and 4) now reach miners too.
    • The worker derives the spec from config every round, so a task switch to a different K is followed.
  • A miner's starting point (get_model_from_checkpoint):
    • It walks the local candidates newest first and loads the first one that verifies. If none does, it keeps its
      fresh draw, with a warning.
    • So a newer folder that is empty, untrained or for another K no longer outranks the miner's own trained
      checkpoint.
    • It still reports the newest candidate on disk, so the mid-round resync check doesn't loop.
  • Fetch leaves nothing behind on failure.
    • It creates the download folder only after its checks pass, and removes it when the download or the validation
      fails.
    • A refused published baseline is therefore downloaded again on each later fetch until a newer one appears. That
      costs 252 MB per fetch for Qwen K=1, and stops as soon as a newer baseline is published.
  • Deleted:
    • _verify_expert_group and the expert-group checkpoint flags and parameters;
    • CheckpointCfg.strict_sharding;
    • reload_model_inplace (no callers), and the fetch parameters only it used (allowed_hotkeys,
      download_timeout_sec);
    • the optimizer-resume block that never ran (resume = False), with load_checkpoint and load_optimizer;
    • the unused rank parameter of get_model_from_checkpoint, load_model, _build_eval_model and the
      validator's setup_training.
  • Left for PR 08: save_state_dict_by_expert_group, its last fp16 lines and test_checkpoint_sharding.py
    (its last caller, _write_pretrained_shard, is PR 08's), and train.py's ExpertManager.

Not on this stack

incumbent.fetch_round_incumbent (#282) is not in this stack. If #282 lands in staging_v3 first, the rebase changes
its expert_group_assignment argument to expected.

Outside this repo

The gateway joins on eval status codes, so it needs code 14 (bad_residual_shard). cycle-api's residual check
(C1) should apply the same threshold / temperature < 80 bound.

Tests

test_residual_shard.py (new; tiny Qwen and tiny DeepSeek-V2, CPU, offline) checks:

  • residual_shapes equals a hand-written spec, and the live installed model's residual parameters, for both
    families at K=1 and K=2.
  • The save round trip: the keys are exactly the spec, the router stays fp32, and reloading gives identical
    tensors.
  • Refused through ModelCheckpoint.validate:
    • an extra key, a missing key or a wrong shape;
    • a bf16 router;
    • a shard for another K;
    • NaN or Inf;
    • a .pt file, or a corrupt header;
    • an all-zero shard, or the untouched init (down_proj = 0);
    • gate and up both zero;
    • a read error.
  • Accepted: a shard where only gate_proj is zero.
  • No tensor is read when the header is off-spec, and a valid shard is read through the same open as its header.
  • The miner's starting point:
    • with an own trained globalver_1_inneropt_100 and a newer, empty or untrained globalver_5 download, the miner
      resumes its own checkpoint, and still reports version 5;
    • with both trained, the newer baseline is the start;
    • a K-mismatched, corrupt or untrained shard keeps the fresh draw, and one step then moves every down_proj.
  • Fetch: a failed download, a failed validation, or no HF coordinates leaves no folder behind.
  • The A/τ bound: 0.8 / 0.01 is refused and 0.79 / 0.01 is accepted.

test_validate_miner_submission.py (rewritten):

  • a real signed ChainCheckpoint on a written shard;
  • every malformed case gives bad_residual_shard, while signature and hash failures keep their own reasons;
  • a refused submission sets the miner's status gauge to 14, and counts it once.

Other tests follow the new signatures. The worker test checks that the spec is read from config every round and
passed to validation. test_eval_worker_group_assignment.py is deleted along with the setter it tested.

Mutation checks: 44 mutants, each killed by at least one test. They cover:

  • the header check, how tensors are read, and the values and untrained-expert checks;
  • the A/τ bound and code 14;
  • the start guard (newest candidate only, guard removed, break removed, the returned version);
  • the fetch cleanup;
  • where the shard is saved;
  • validate's reason order;
  • residual_shapes;
  • every place the spec is passed on.

Results

  • Full suite: 686 passed, 4 failed. The 4 failures already fail on staging_v3:
    test_reserve_owner_share_normal_case, test_reserve_owner_share_owner_already_present,
    test_checkpoint_cfg_restores_download_concurrency_for_compatibility and test_checkpoint_precision_preservation.
  • ruff check connito: 312 (was 329), with no new violations at any commit.
  • Python under connito/, tests excluded: +219 / −655.

How to verify: python -m pytest connito/test/test_residual_shard.py connito/test/test_validate_miner_submission.py → 47 passed

GPU test

titan, one RTX A6000 per run, this PR's code (acfee83). Each run loads the real base, installs a K=1
residual, trains it for 3 steps, then saves, verifies and reloads it:

Qwen3.6, fp8 experts Qwen3.6, fp8 all DeepSeek-V2-Lite, bf16
residual_shapes: keys / equals the live model 160 / yes 160 / yes 104 / yes
residual_shapes time 0.10 s 0.10 s 0.06 s
untouched init (down_proj = 0) refused yes yes yes
trained shard: size / save / verify 252.0 MB / 0.40 s / 0.09 s 252.0 MB / 0.28 s / 0.11 s 450.1 MB / 0.64 s / 0.13 s
dtypes in the file experts bf16, router fp32 same same
fresh residual + the verified tensors: hidden states vs the trained model bit-identical bit-identical bit-identical
refused: extra key, missing key, wrong shape, bf16 router, NaN, one untouched expert all 6 all 6 all 6

…ne helper

_attach hangs K experts, in the block's dtype, and an fp32 router on a MoE
block without touching their values. install_residual_experts calls it and
then draws gate, up and the router and zeroes down_proj, as before, now on the
modules already in place.

The next commit builds the same modules on a meta-device skeleton to read the
residual spec, so the keys, shapes and dtypes a shard is checked against come
from the code that makes them.
…type for the family and K

The spec a residual shard must match exactly. It reads only the family's
config.json at the pinned revision and builds the model as a meta-device
skeleton, which holds no weights (0.05 s for real-size Qwen), then attaches
the task's K residual modules through the install's own _attach and walks the
result. So it works where no model is loaded (the miner's download process,
or before the model is built), and the MoE layers come from the family's own
block type: 40 on Qwen, 26 on DeepSeek-V2 (layer 0 is dense), with no
per-family rule.

TensorSpec(shape, dtype) lives in checkpoint_helper, beside the shard I/O that
compares against it.
…its own dtype

save_residual_shard writes the model's `.residual_` parameters as the group's
model_expgroup_<g>.safetensors: the experts in bf16, the router in fp32. It
reads parameters, not state_dict(), so an fp8 base is never dequantized for a
save.

save_checkpoint takes the group id and always writes that shard. Its
expert-group switches go (save_model_by_expert_group, expert_manager,
strict_sharding, active_expert_group_id), together with the whole-model
model.pt branch nothing used, and the cast of every tensor to one dtype that
fell back to fp16. That cast would have rounded the fp32 router to bf16, and
the expert-group splitter it fed drops residual keys altogether, since
get_layer_expert_id reads `residual_experts.<j>` as no expert: at the parent
commit a miner's shard was empty.

ckpt.strict_sharding had no other reader and goes; old YAMLs that still carry
it load, as unknown keys are ignored.
The validator's peer resync from before the 2026-09-11 decision: the validator
keeps no model of its own now, and nothing calls this function. Its only test
coverage re-implemented it in test_validator_grad_safety.py, and that section
goes too. It also carried an ExpertManager into the chain fetch, which the
next commit changes, so it goes rather than being ported. The five imports
only it used go with it.

It was also the only caller to pass fetch_model_from_chain_validator's
allowed_hotkeys and download_timeout_sec. Both go, with the hotkey filter, the
timed-out branch and the wrapper that only threaded the timeout through; the
fetch calls download_checkpoint_from_hf_with_timeout without a timeout, as
every remaining caller already did.
…header first

verify_residual_shard reads the safetensors header and compares every key,
shape and dtype with residual_shapes before any tensor is read: an extra or
missing key, a wrong shape, a bf16 router or another task's K is refused
there. Then, from the same open, it reads the tensors whose header passed,
and requires every tensor to be finite and every expert trained. An expert
whose down_proj is all zero, or whose gate_proj and up_proj both are, adds
nothing to the base and, if two of its three projections are zero, never
gets a gradient. Published as a round baseline, an all-zero shard would stop
every miner training. A .pt name, an unreadable header or a tensor that
cannot be read is refused too, never raised.

Honest training never leaves such an expert, because q > 0 on every token
moves every down_proj at the first step. That holds only while A/τ is below
about 88: past that, sigmoid(−A/τ) is exactly 0 in fp32. ResidualCfg now
refuses a threshold / temperature of 80 or more. v1 is 16.

ModelCheckpoint.validate(expected) runs the signature, this check, then the
hash over the tensors it returned. It reads the file itself (a submission) or
the group's shard in a folder (a downloaded baseline). One shard_verified
flag replaces expert_group_check_required/expert_group_verified, and
_verify_expert_group goes. It accepted no residual key anyway:
get_layer_expert_id reads `residual_experts.<j>` as no expert.

Callers:
- fetch_model_from_chain_validator takes `expected`; load_model drops its
  ExpertManager and passes residual_shapes(config); the miner's download
  process derives the spec per job, so it follows a task switch.
- validate_miner_submission(expected) returns bad_residual_shard for any of
  the above. The worker now records each validation failure for the miner:
  bad_residual_shard is status code 14. Before this, no validation reason
  reached the gauge. incomplete_expert_set and expert_group_or_nan are no
  longer produced; codes 5 and 13 stay in EVAL_STATUS_CODES, retired.
  _submission_keys goes.
- BackgroundEvalWorker derives the spec from config at each round load, in
  place of the base shard's key set, so a task switch needs no hand-over.
  These go:
  - its expert_group_assignment argument and set_expert_group_assignment;
  - the eval_worker parameter _switch_task / _maybe_switch_task carried only
    for it;
  - run()'s now-unused expert_manager rebind.

The validator's own round base is never a submission and is loaded without
this check, so its down_proj = 0 needs no exception.
…lse keeps its own draw

get_model_from_checkpoint is the one place a miner's start is chosen, over
the residual install_residual_experts has just drawn. The candidates are the
published round baselines it downloaded and its own checkpoints, newest
(global_ver, inner_opt) first. It now loads the first one that passes
verify_residual_shard against residual_shapes. One that is off spec
(another task's K, say) or untrained (all zero, down_proj zero, or gate_proj
and up_proj zero) is passed over, so a refused baseline does not cost the
miner its own trained checkpoint below it. Only when none verifies does it
train from its fresh draw, which every step moves. That covers a null
residual from any source: the fetch already refuses one, and the validator
never scores one. The newest candidate is still what it returns, since the
resync check compares the next newest with it.

The fetch now leaves nothing behind when it fails. It creates the uid_*
folder only once it has something to download, and removes it when the
download or the validation fails. An empty folder there was the newest
checkpoint on disk.

local_checkpoints_best_first lists the candidates; select_best_checkpoint
is now its first entry. group_shard names the shard to read: the path itself
if it is a file, otherwise the group's shard in the folder.
ModelCheckpoint.validate uses it too.

get_model_from_checkpoint, load_model and the validator's setup_training and
_build_eval_model no longer take a rank: load_checkpoint was its last use.
…eckpoint

setup_training's resume block set `resume = False` and then required it, so
the optimizer, scheduler and dataloader state a checkpoint carries was never
read back. Since the previous commit, get_model_from_checkpoint loads the
residual shard itself, so that block was load_checkpoint's last caller, and
load_optimizer had no other. All three go, with two imports only they used.
save_checkpoint still writes those files; whether they should stay is a
separate question.
@isabella618033

Copy link
Copy Markdown
Contributor

The spec and the live model agree only by coincidence of dtypes.

The skeleton is forced to bf16, while the live residual takes block.gate.weight.dtype. They match today because no router is ever quantised to fp8.
If that ever changes, every honest submission fails with code 14 and nothing explains why.
Cheap guard: at miner boot, after install, assert that the live residual equals residual_shapes(config). The GPU table confirms they match now; the guard keeps it that way.

@george-connito
george-connito merged commit f65c24e into feat/qwen-06-residual-train Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants