Repository navigation
📈 feat(validator): measure the published baseline against the one it replaced - #282
Open
george-connito wants to merge 1 commit into
Open
george-connito wants to merge 1 commit into
george-connito wants to merge 1 commit into
Conversation
…replaced Measurement only: nothing here changes what is scored, paid or published. Each round, fetch the chain-advertised baseline at freeze (the file these submissions trained from, via the miner's own fetch so the majority-hash rule and hash check are the ones miners get), score it once on the round's cached batches, and at finalize log `baseline selection comparison`: the uid `select_baseline_uid` publishes, this round's best, the incumbent's loss, what a ratchet (best, only if it beats the incumbent; incumbent wins ties) would have published, and whether the published pick regressed. The question it answers: how often does the baseline every miner restarts from score worse than the one it replaced? The pick ranks proven miners by rolling average, not by this round's loss, and nothing compares it with the incumbent or pretrained. The fetch runs on a daemon thread with its own archive connection; the eval goes through none of the miner path (no per-uid score, queue slot or telemetry) and is skipped when the key set is not exactly the base shard's. A failed fetch or eval logs and leaves the round untouched. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Measures whether the baseline the validators publish is better or worse than
the one it replaces. Measurement only: nothing here changes what is scored,
paid or published, so it needs no coordinated rollout.
Each round the validator now also scores the incumbent, the chain-advertised
baseline at freeze, which is exactly the file that round's submissions trained
from. At finalize it logs one line comparing the published pick with the
incumbent and with this round's best submission.
Why
Every miner restarts from the published baseline at Distribute, so that choice
sets the direction of the whole lineage. Today it is made by
select_baseline_uid, which ranks the three proven miners by rollingaverage and uses this round's
val_lossonly to break ties. Nothing comparesthe pick with the model it replaces, or with pretrained.
So a proven miner's worse file can become everyone's starting point. Whether
that happens often enough to matter is an empirical question. This PR answers it
before anyone changes the rule.
What it logs
At finalize,
baseline selection comparison:legacy_uid,legacy_val_lossselect_baseline_uidpublishesbest_uid,best_val_lossval_lossincumbent_val_lossratchet_uidNonemeans keep the incumbent (ties go to it)legacy_is_bestlegacy_regressedNoneinincumbent_val_lossmeans the round had no incumbent to measure:nothing advertised, a failed download, or a round resumed after a restart.
How
validator/incumbent.py, new): a daemon thread started right afterfreeze calls the miner's own
fetch_model_from_chain_validator, so themajority-hash rule, the HF coordinates and the hash check are the ones miners
get. At freeze, "previous phase" resolves to the last ValidatorCommit pair,
which is the baseline those submissions started from. It opens its own archive
connection, since a
Subtensoris not safe to share across threads.background_eval_worker.py): once per round, between miners, assoon as the download lands. Same cached batches and resident model as the
miners, but none of the miner path, so no per-uid score, queue slot or
telemetry. Skipped if its key set is not exactly the base shard's, because a
partial overlay would measure a hybrid with the previous miner's experts.
round.py): a purebaseline_selection_reportnext toselect_baseline_uid, logged fromrun.pyright afterfinalize_round_scores.Any failure in the fetch or the eval is logged and leaves the round untouched.
Cost
not by draining the queue, so in a round that doesn't finish its roster this
displaces roughly one tail miner.
validator_checkpoint_path.checkpoint_topk(2) caps what stays on disk.Seen and not changed
publish_round_baselineand the podium upload are not gated by observer mode.An observer is kept from uploading only by having no HF token. Worth closing
separately.
Tests
connito/test/test_incumbent_measurement.py, 9 cases: the report (a provenminer's regression the ratchet avoids, the incumbent kept on a loss and on a tie,
no incumbent, an empty round); the worker (scores the incumbent's own weights
exactly once, skips a mismatched key set without touching the model, never
writes a finalized round); the fetch (points the round at the shard, and
survives nothing advertised or a failing chain).
Trial
Run on the L40 observer against production's clock. Read
legacy_regressedacross a day or more of rounds.🤖 Generated with Claude Code