Repository navigation
docs(verification): record the warning wording outcome - #102
Conversation
W cut stale runs with a user lesson from five of ten to one, but scored seven passes on memory-changed-fact against the baseline's nine, so it fails that predeclared bar and is parked. Both of W's misses there were runs never offered the lesson; that and the cost figures are reported, not decided on. Validation: counts checked against each run's jsonl and trajectories, the leak traced to a read of the session's own trajectory, guard edits scanned in both builds; cargo xtask qa exits 0.
jadenfix
left a comment
There was a problem hiding this comment.
Lane B review: OK to merge on green. I read the outcome against the predeclared reading on main, and it follows it:
- User task: baseline 5/10 stale (≥2, so there was harm to remove) and W 1/10. Four fewer clears the two-fewer bar. It is recorded as a direction, as fixed.
- Agent obsolete: 0 vs 0, so W passes.
- changed-fact: the bar counts passes, and 7 < 9, so W fails. As a share of scored runs it fails too (78% vs 90%), so the leak exclusion doesn't change the call. Dropping the leaked run follows the predeclared rule, and the cell still has nine scored runs, so no extra repeats were owed.
- Regressions: equal passes on both tasks, so W passes.
- W fails one bar, so it is parked and main is unchanged. That is the decided rule.
The figures conditional on the lesson add up: the baseline's 8/8 offered plus two not offered (one pass) gives 9/10, and W's 7/7 offered plus two not offered (both fail) gives 7/9. Keeping that under "reported, not decided on", and saying a warning-only bar would need its own predeclaration, is the right handling. I've nothing to add.
Generated by Claude Code
|
Independent exact-head scientific-integrity review — CLEAR for this documentation-only outcome record. Pinned identity:
I compared the single-file, +47-line diff with the predeclared decision rules on the base. The recorded calls are consistent with those rules:
The conditional counts also reconcile: baseline 8/8 offered plus one pass among two not-offered runs = 9/10; W 7/7 offered plus two not-offered failures = 7/9. Keeping that observation outside the predeclared decision and requiring a new predeclaration for any warning-only analysis is scientifically sound. Verification:
I could not independently rerun Cargo locally because this executor has no Rust toolchain; the exact-head hosted QA job is therefore the executable evidence relied upon. Scope boundary: this clears only the truthful recording of the predeclared wording experiment. It does not verify Rusty's portfolio core workflow, add a GIF/MP4 demo, earn current-main/demo coverage credit, or support a wording-effect claim. No merge action taken. |
The changed-fact bar now says how a run counts as offered the lesson, as a rule over the trajectory fixed before any run, needs at least five offered runs per build or stays undecided, and records #102's figures as the reason the bar changed. Validation: docs only; cargo xtask qa exits 0.
Baseline e7f0a8f against W 27fe97b on the seed-marker cohort. The user task decides as before; changed-fact compares failure shares among runs offered the lesson, with a floor of five offered runs per build. Validation: the offered rule reproduces #102's 8/0 and 7/0; docs only; cargo xtask qa exits 0.
Why
The wording test predeclared in #100 is done. This PR records it, read as decided.
Results
The baseline 4bc026f and W 63ac28d ran at the same time, each on its own key. One W run was a leak: it read its own session's
trajectory.json, a harness record. It is reported and not scored. No cell fell below seven scored runs, so nothing was rerun.Each cell shows passes out of scored runs, stale runs, the median requests to the first correct suite run, and the median wall time.
memory-changed-obsolete-usermemory-changed-obsoletememory-changed-factstale-anchorpoisoned-lessonReading
memory-changed-fact: W fails, with 7 passes against 9.Reported but not decided on:
memory-changed-factmisses were never offered the lesson, so they never saw the wording. On runs where the lesson was offered, both builds passed every scored run (8/8 and 7/7). The gap is in whether the agent lesson is offered late at all, which is the same code in both builds. The bar counted passes, as decided, so it stands.memory-changed-factbar that counts only runs shown the warning. That would be a new predeclaration.Validation
cargo xtask qaexits 0 on an idle host.