Skip to content

📈 feat(validator): measure the published baseline against the one it replaced - #282

Open
george-connito wants to merge 1 commit into
staging_v3from
feat/measure-incumbent-baseline
Open

george-connito wants to merge 1 commit into
staging_v3from
feat/measure-incumbent-baseline

Conversation

@george-connito

Copy link
Copy Markdown
Collaborator

Summary

Measures whether the baseline the validators publish is better or worse than
the one it replaces. Measurement only: nothing here changes what is scored,
paid or published, so it needs no coordinated rollout.

Each round the validator now also scores the incumbent, the chain-advertised
baseline at freeze, which is exactly the file that round's submissions trained
from. At finalize it logs one line comparing the published pick with the
incumbent and with this round's best submission.

Why

Every miner restarts from the published baseline at Distribute, so that choice
sets the direction of the whole lineage. Today it is made by
select_baseline_uid, which ranks the three proven miners by rolling
average
and uses this round's val_loss only to break ties. Nothing compares
the pick with the model it replaces, or with pretrained.

So a proven miner's worse file can become everyone's starting point. Whether
that happens often enough to matter is an empirical question. This PR answers it
before anyone changes the rule.

What it logs

At finalize, baseline selection comparison:

field meaning
legacy_uid, legacy_val_loss what select_baseline_uid publishes
best_uid, best_val_loss this round's lowest val_loss
incumbent_val_loss the baseline these submissions trained from, same batches
ratchet_uid best, only if it beats the incumbent; None means keep the incumbent (ties go to it)
legacy_is_best whether the published pick was also this round's best
legacy_regressed the answer: did the published pick score worse than the incumbent

None in incumbent_val_loss means the round had no incumbent to measure:
nothing advertised, a failed download, or a round resumed after a restart.

How

  • Fetch (validator/incumbent.py, new): a daemon thread started right after
    freeze calls the miner's own fetch_model_from_chain_validator, so the
    majority-hash rule, the HF coordinates and the hash check are the ones miners
    get. At freeze, "previous phase" resolves to the last ValidatorCommit pair,
    which is the baseline those submissions started from. It opens its own archive
    connection, since a Subtensor is not safe to share across threads.
  • Score (background_eval_worker.py): once per round, between miners, as
    soon as the download lands. Same cached batches and resident model as the
    miners, but none of the miner path, so no per-uid score, queue slot or
    telemetry. Skipped if its key set is not exactly the base shard's, because a
    partial overlay would measure a hybrid with the previous miner's experts.
  • Report (round.py): a pure baseline_selection_report next to
    select_baseline_uid, logged from run.py right after finalize_round_scores.

Any failure in the fetch or the eval is logged and leaves the round untouched.

Cost

  • One extra eval per round. The eval window is bounded by a block deadline,
    not by draining the queue, so in a round that doesn't finish its roster this
    displaces roughly one tail miner.
  • One download per round of the active group's shard, about 2 to 3 GB, into
    validator_checkpoint_path. checkpoint_topk (2) caps what stays on disk.

Seen and not changed

publish_round_baseline and the podium upload are not gated by observer mode.
An observer is kept from uploading only by having no HF token. Worth closing
separately.

Tests

connito/test/test_incumbent_measurement.py, 9 cases: the report (a proven
miner's regression the ratchet avoids, the incumbent kept on a loss and on a tie,
no incumbent, an empty round); the worker (scores the incumbent's own weights
exactly once, skips a mismatched key set without touching the model, never
writes a finalized round); the fetch (points the round at the shard, and
survives nothing advertised or a failing chain).

test_incumbent_measurement + 8 neighbouring validator suites    123 passed
ruff: no new findings in the changed files (counts equal the base)

Trial

Run on the L40 observer against production's clock. Read
legacy_regressed across a day or more of rounds.

🤖 Generated with Claude Code

…replaced

Measurement only: nothing here changes what is scored, paid or published.

Each round, fetch the chain-advertised baseline at freeze (the file these
submissions trained from, via the miner's own fetch so the majority-hash rule
and hash check are the ones miners get), score it once on the round's cached
batches, and at finalize log `baseline selection comparison`: the uid
`select_baseline_uid` publishes, this round's best, the incumbent's loss, what
a ratchet (best, only if it beats the incumbent; incumbent wins ties) would
have published, and whether the published pick regressed.

The question it answers: how often does the baseline every miner restarts
from score worse than the one it replaced? The pick ranks proven miners by
rolling average, not by this round's loss, and nothing compares it with the
incumbent or pretrained.

The fetch runs on a daemon thread with its own archive connection; the eval
goes through none of the miner path (no per-uid score, queue slot or
telemetry) and is skipped when the key set is not exactly the base shard's.
A failed fetch or eval logs and leaves the round untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant