Repository navigation
A submission is exactly the residual, checked header first; miners never start from an untrained residual - #290
Merged
george-connito merged 7 commits intoOct 14, 2026
Conversation
…ne helper _attach hangs K experts, in the block's dtype, and an fp32 router on a MoE block without touching their values. install_residual_experts calls it and then draws gate, up and the router and zeroes down_proj, as before, now on the modules already in place. The next commit builds the same modules on a meta-device skeleton to read the residual spec, so the keys, shapes and dtypes a shard is checked against come from the code that makes them.
…type for the family and K The spec a residual shard must match exactly. It reads only the family's config.json at the pinned revision and builds the model as a meta-device skeleton, which holds no weights (0.05 s for real-size Qwen), then attaches the task's K residual modules through the install's own _attach and walks the result. So it works where no model is loaded (the miner's download process, or before the model is built), and the MoE layers come from the family's own block type: 40 on Qwen, 26 on DeepSeek-V2 (layer 0 is dense), with no per-family rule. TensorSpec(shape, dtype) lives in checkpoint_helper, beside the shard I/O that compares against it.
…its own dtype save_residual_shard writes the model's `.residual_` parameters as the group's model_expgroup_<g>.safetensors: the experts in bf16, the router in fp32. It reads parameters, not state_dict(), so an fp8 base is never dequantized for a save. save_checkpoint takes the group id and always writes that shard. Its expert-group switches go (save_model_by_expert_group, expert_manager, strict_sharding, active_expert_group_id), together with the whole-model model.pt branch nothing used, and the cast of every tensor to one dtype that fell back to fp16. That cast would have rounded the fp32 router to bf16, and the expert-group splitter it fed drops residual keys altogether, since get_layer_expert_id reads `residual_experts.<j>` as no expert: at the parent commit a miner's shard was empty. ckpt.strict_sharding had no other reader and goes; old YAMLs that still carry it load, as unknown keys are ignored.
The validator's peer resync from before the 2026-09-11 decision: the validator keeps no model of its own now, and nothing calls this function. Its only test coverage re-implemented it in test_validator_grad_safety.py, and that section goes too. It also carried an ExpertManager into the chain fetch, which the next commit changes, so it goes rather than being ported. The five imports only it used go with it. It was also the only caller to pass fetch_model_from_chain_validator's allowed_hotkeys and download_timeout_sec. Both go, with the hotkey filter, the timed-out branch and the wrapper that only threaded the timeout through; the fetch calls download_checkpoint_from_hf_with_timeout without a timeout, as every remaining caller already did.
…header first
verify_residual_shard reads the safetensors header and compares every key,
shape and dtype with residual_shapes before any tensor is read: an extra or
missing key, a wrong shape, a bf16 router or another task's K is refused
there. Then, from the same open, it reads the tensors whose header passed,
and requires every tensor to be finite and every expert trained. An expert
whose down_proj is all zero, or whose gate_proj and up_proj both are, adds
nothing to the base and, if two of its three projections are zero, never
gets a gradient. Published as a round baseline, an all-zero shard would stop
every miner training. A .pt name, an unreadable header or a tensor that
cannot be read is refused too, never raised.
Honest training never leaves such an expert, because q > 0 on every token
moves every down_proj at the first step. That holds only while A/τ is below
about 88: past that, sigmoid(−A/τ) is exactly 0 in fp32. ResidualCfg now
refuses a threshold / temperature of 80 or more. v1 is 16.
ModelCheckpoint.validate(expected) runs the signature, this check, then the
hash over the tensors it returned. It reads the file itself (a submission) or
the group's shard in a folder (a downloaded baseline). One shard_verified
flag replaces expert_group_check_required/expert_group_verified, and
_verify_expert_group goes. It accepted no residual key anyway:
get_layer_expert_id reads `residual_experts.<j>` as no expert.
Callers:
- fetch_model_from_chain_validator takes `expected`; load_model drops its
ExpertManager and passes residual_shapes(config); the miner's download
process derives the spec per job, so it follows a task switch.
- validate_miner_submission(expected) returns bad_residual_shard for any of
the above. The worker now records each validation failure for the miner:
bad_residual_shard is status code 14. Before this, no validation reason
reached the gauge. incomplete_expert_set and expert_group_or_nan are no
longer produced; codes 5 and 13 stay in EVAL_STATUS_CODES, retired.
_submission_keys goes.
- BackgroundEvalWorker derives the spec from config at each round load, in
place of the base shard's key set, so a task switch needs no hand-over.
These go:
- its expert_group_assignment argument and set_expert_group_assignment;
- the eval_worker parameter _switch_task / _maybe_switch_task carried only
for it;
- run()'s now-unused expert_manager rebind.
The validator's own round base is never a submission and is loaded without
this check, so its down_proj = 0 needs no exception.
…lse keeps its own draw get_model_from_checkpoint is the one place a miner's start is chosen, over the residual install_residual_experts has just drawn. The candidates are the published round baselines it downloaded and its own checkpoints, newest (global_ver, inner_opt) first. It now loads the first one that passes verify_residual_shard against residual_shapes. One that is off spec (another task's K, say) or untrained (all zero, down_proj zero, or gate_proj and up_proj zero) is passed over, so a refused baseline does not cost the miner its own trained checkpoint below it. Only when none verifies does it train from its fresh draw, which every step moves. That covers a null residual from any source: the fetch already refuses one, and the validator never scores one. The newest candidate is still what it returns, since the resync check compares the next newest with it. The fetch now leaves nothing behind when it fails. It creates the uid_* folder only once it has something to download, and removes it when the download or the validation fails. An empty folder there was the newest checkpoint on disk. local_checkpoints_best_first lists the candidates; select_best_checkpoint is now its first entry. group_shard names the shard to read: the path itself if it is a file, otherwise the group's shard in the folder. ModelCheckpoint.validate uses it too. get_model_from_checkpoint, load_model and the validator's setup_training and _build_eval_model no longer take a rank: load_checkpoint was its last use.
…eckpoint setup_training's resume block set `resume = False` and then required it, so the optimizer, scheduler and dataloader state a checkpoint carries was never read back. Since the previous commit, get_model_from_checkpoint loads the residual shard itself, so that block was load_checkpoint's last caller, and load_optimizer had no other. All three go, with two imports only they used. save_checkpoint still writes those files; whether they should stay is a separate question.
Contributor
|
The spec and the live model agree only by coincidence of dtypes. The skeleton is forced to bf16, while the live residual takes block.gate.weight.dtype. They match today because no router is ever quantised to fp8. |
isabella618033
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Into
staging_v3, on top of PR 06 (#289). This is PR 07 of the Qwen3.6 stack, which merges in order.Do not merge alone, and do not ship to validators without PR 08. At this PR, the validator already accepts
residual submissions, but two things are still missing until PR 08:
base_shardis still the old expert-group shard, so a round's baseline loss can be measured with theprevious miner's residual still loaded.
Nodes also still fail at boot on the missing
expert_assignment.json(see PR 03) until PR 08 deletes the reader.Why
Miners now train only residual experts (PRs 04 and 06). Until this PR, every residual submission failed validation:
...mlp.residual_experts.0...as no expert at all.A submission must be exactly the residual, in protocol shapes and dtypes, and a real one.
Real means trained. A residual whose
down_projis all zero (or whose gate and up are both zero) scoresexactly like the base, and nothing in it can ever learn. If such a shard won a round and was published, every
miner would restart from it and get no gradient. So nodes refuse an untrained residual both as a submission and as
a starting point.
Nothing is shipped as a starting point. Each node creates its own residual with PR 04's
install_residual_experts, and with no published winner a miner keeps that fresh draw, or its own last trainedcheckpoint.
The change
residual_shapes(config)(residual.py):router is fp32.
config.json(cached; the revision is a pinned sha), builds a meta-device skeleton, andattaches the residual through the same helper
install_residual_expertsuses.save_residual_shard(model, dest_dir, group_id)writes the model's residual experts and routers, each in itsown dtype, as the group's shard (
expert_group_shard_name).save_checkpointtakesgroup_idand uses it; itsfour expert-group parameters are gone.
verify_residual_shard(path, expected)returns the shard's tensors, orNone:read; a refusal takes about 1 ms.
down_projis not all zero, and its gate and up are not both all zero.This never trips on honest training, because the gate q is above 0 on every token, so every
down_projmovesat the first step.
safe_open. A corrupt, truncated or missing file returnsNoneandnever raises.
ResidualCfgrefusesthreshold / temperature >= 80. From about 88, the fp32 gate underflows to exactly0, and honest training could leave a
down_projat zero. v1 is 16. A live switch to such a task is refusedand rolled back.
ModelCheckpoint.validate(expected)checks the signature, then the shard, then the hash.validate_miner_submission) and the miner's fetch use it.bad_residual_shard: eval status code 14. Codes 5 (incomplete_expert_set) and 13stay in the table as retired.
signature and hash failures (codes 6, 3 and 4) now reach miners too.
get_model_from_checkpoint):fresh draw, with a warning.
checkpoint.
fails.
costs 252 MB per fetch for Qwen K=1, and stops as soon as a newer baseline is published.
_verify_expert_groupand the expert-group checkpoint flags and parameters;CheckpointCfg.strict_sharding;reload_model_inplace(no callers), and the fetch parameters only it used (allowed_hotkeys,download_timeout_sec);resume = False), withload_checkpointandload_optimizer;rankparameter ofget_model_from_checkpoint,load_model,_build_eval_modeland thevalidator's
setup_training.save_state_dict_by_expert_group, its last fp16 lines andtest_checkpoint_sharding.py(its last caller,
_write_pretrained_shard, is PR 08's), and train.py'sExpertManager.Not on this stack
incumbent.fetch_round_incumbent(#282) is not in this stack. If #282 lands in staging_v3 first, the rebase changesits
expert_group_assignmentargument toexpected.Outside this repo
The gateway joins on eval status codes, so it needs code 14 (
bad_residual_shard). cycle-api'sresidualcheck(C1) should apply the same
threshold / temperature < 80bound.Tests
test_residual_shard.py(new; tiny Qwen and tiny DeepSeek-V2, CPU, offline) checks:residual_shapesequals a hand-written spec, and the live installed model's residual parameters, for bothfamilies at K=1 and K=2.
tensors.
ModelCheckpoint.validate:.ptfile, or a corrupt header;down_proj= 0);gate_projis zero.globalver_1_inneropt_100and a newer, empty or untrainedglobalver_5download, the minerresumes its own checkpoint, and still reports version 5;
down_proj.test_validate_miner_submission.py(rewritten):ChainCheckpointon a written shard;bad_residual_shard, while signature and hash failures keep their own reasons;Other tests follow the new signatures. The worker test checks that the spec is read from config every round and
passed to validation.
test_eval_worker_group_assignment.pyis deleted along with the setter it tested.Mutation checks: 44 mutants, each killed by at least one test. They cover:
breakremoved, the returned version);validate's reason order;residual_shapes;Results
test_reserve_owner_share_normal_case,test_reserve_owner_share_owner_already_present,test_checkpoint_cfg_restores_download_concurrency_for_compatibilityandtest_checkpoint_precision_preservation.ruff check connito: 312 (was 329), with no new violations at any commit.connito/, tests excluded: +219 / −655.How to verify:
python -m pytest connito/test/test_residual_shard.py connito/test/test_validate_miner_submission.py→47 passedGPU test
titan, one RTX A6000 per run, this PR's code (
acfee83). Each run loads the real base, installs a K=1residual, trains it for 3 steps, then saves, verifies and reloads it:
expertsallresidual_shapes: keys / equals the live modelresidual_shapestimedown_proj= 0) refused