Define and enforce the vNext result and ranking contract

[isPANN]

Background

The current submission schema describes one whole-repository session with a shared
submit_limit, a flat submit_log, agent_mode: whole-repo, and token efficiency as a
tie-breaker. Those fields cannot prove that a valid Top50 was frozen or that 50 rules received
identical non-transferable budgets.

The new private submission artifact must carry the complete evaluation-owned triage and episode
ledgers so the backend can recompute rankability and score. The public leaderboard must continue
to expose aggregate data only; rule identities, hypotheses, certificates, and trajectories remain
private answer-key material.

Objective

Introduce a versioned Top50 submission and scoring contract, make the backend independently
validate its shortlist and all 50 episode ledgers, re-verify every accepted certificate, and rank
only clean Standardized Model Track results by distinct accepted rules.

Historical unlimited whole-repository and coding-agent results must remain viewable under their
original contract but must never be mixed into the Top50 ranking. Equal verified bug counts remain
ties; elapsed time, token usage, cost, and bugs/Ktok are informational only.

Interface (Input → Output)

• Input: I2 Top50 run record, evaluation-owned ledgers, model/provider and exposed inference
  parameters, pinned repository/pred/runner identifiers, and accepted certificates.
• Private output: schema-valid versioned submission containing the complete budget vector,
  ordered Top50, triage ledger, exactly 50 rule-bound episode ledgers, usage, errors, and
  certificates.
• Scored output: backend-derived VerifiedBugs@Top50[T,E_t,M,E,P,P_solve,S=2,O], Bugs@10/25/50,
  first-versus-second-attempt acceptance, calls per accepted bug, cap-hit diagnostics, and
  rankability verdict.
• Public output: aggregate leaderboard entry with model and contract provenance, headline
  verified bug count, and non-ranking efficiency fields; no answer-key data.

Technical recommendations

• Add an explicit benchmark-contract identifier instead of inferring semantics from optional
  legacy fields. Keep a deliberate legacy parser for historical data rather than making the new
  schema accept ambiguous mixtures.
• Validate that the shortlist has exactly 50 canonical unique IDs, episode order matches it,
  every configured limit is identical, each usage count is within its limit, P_solve <= P,
  S=2 per rule, and request IDs/attempt numbering are internally consistent.
• Derive result rows only from evaluation-owned accepted submit records for the matching current
  rule. Re-run verify() on every accepted certificate; ignore self-reported bugs_found.
• Treat missing episodes, infrastructure errors, CLI/custom-harness provenance, custom prompts,
  or inconsistent ledgers as unrankable. Preserve the private artifact for debugging.
• Version the public board by benchmark contract. Remove token efficiency from sort keys for the
  new contract while leaving old-board rendering intact.
• The operators are a trusted internal team. Do not add remote-run attestation, signatures, or
  external-user authentication to this issue.

Verification

Run the contract and scorer acceptance suites:

pytest -v benchmark/tests/test_top50_submission.py benchmark/tests/test_verify_submission.py benchmark/tests/test_backend_score.py

They must use a schema-valid synthetic Top50 artifact and demonstrate:

1. The valid artifact has exactly 50 frozen rules and 50 matching episode ledgers; the backend
   ignores a deliberately false self-reported score and returns the distinct count implied by
   accepted, re-verified certificates.
2. Two accepted certificates for one rule count once. Accepted rules at shortlist positions 7,
   18, and 41 produce Bugs@10 = 1, Bugs@25 = 2, and Bugs@50 = 3.
3. A certificate for a rule outside the shortlist, a certificate submitted from another rule's
   episode, a third attempt, a shared 100-attempt pool, duplicate/missing episode, over-limit
   counter, changed budget, or forged request ordering makes the run unrankable. These are the
   negative controls proving the scorer does not trust the envelope.
4. A pure run_error or unrecoverable infrastructure episode is preserved privately but excluded
   from ranking. A rejected model submission remains a clean charged attempt and does not make the
   run an infrastructure failure.
5. A codex/claude-code agent mode or custom prompt provenance is excluded from the Top50 board.
   A canonical API harness run is accepted.
6. Public output contains no rule ID, hypothesis, certificate, source instance, target config,
   submit reason, or trajectory. The aggregate guard fails if any such answer-key field is added.
7. For equal verified bug counts, changing tokens, cost, or elapsed time does not change rank
   order; the UI represents the entries as tied. A larger verified count ranks first.
8. One legacy v0.6 whole-repository fixture remains readable in its historical view but cannot
   enter or replace an entry in the Top50 board.

Also run:

python .github/scripts/check_aggregate.py site/results.json

against a generated development board. It must print no errors for the aggregate-only fixture;
injecting one rule or certificate field must make the guard exit nonzero.

Dependencies

Depends on #66 so the contract validates the real Top50Runner artifact rather than a parallel
invented format.

Out of scope

• cryptographic attestation or accepting untrusted external runs;
• multiple-seed aggregation, best-of-N, or confidence intervals;
• Fixed Top50 scores;
• ranking custom agent systems; and
• declaring unaccepted rules bug-free or reporting Top50 precision/recall.