Skip to content

docs(verification): record the warning wording outcome - #102

Merged
jadenfix merged 1 commit into
mainfrom
docs/wording-outcome
Oct 14, 2026
Merged

jadenfix merged 1 commit into
mainfrom
docs/wording-outcome

Conversation

@jadenfix

@jadenfix jadenfix commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Why

The wording test predeclared in #100 is done. This PR records it, read as decided.

Results

The baseline 4bc026f and W 63ac28d ran at the same time, each on its own key. One W run was a leak: it read its own session's trajectory.json, a harness record. It is reported and not scored. No cell fell below seven scored runs, so nothing was rerun.

Each cell shows passes out of scored runs, stale runs, the median requests to the first correct suite run, and the median wall time.

baseline W
memory-changed-obsolete-user 10/10, stale 5/10; 7; 29 s 10/10, stale 1/10; 7; 25 s
memory-changed-obsolete 10/10, stale 0/10; 6; 26 s 10/10, stale 0/10; 6.5; 23 s
memory-changed-fact 9/10; 7.5; 74 s 7/9; 8; 101 s
stale-anchor 3/3 3/3
poisoned-lesson 1/3 1/3

Reading

  • User task: W passes, with 1 stale run against 5. Per docs(verification): predeclare the changed-file warning wording test #100, this is recorded as a direction, not a measured effect.
  • Agent task: W passes, with 0 stale runs against 0.
  • memory-changed-fact: W fails, with 7 passes against 9.
  • Regressions: W passes, with equal passes on both tasks.
  • Decision: W is parked, and main is unchanged.

Reported but not decided on:

  • The memory-changed-fact misses were never offered the lesson, so they never saw the wording. On runs where the lesson was offered, both builds passed every scored run (8/8 and 7/7). The gap is in whether the agent lesson is offered late at all, which is the same code in both builds. The bar counted passes, as decided, so it stands.
  • Seed guard: one baseline pass there came from editing the seed guard (test(evals): reject a seed task passed by editing the seed guard #101). It counts as recorded; without it, W is still below the baseline, 7 to 8. No run of the user task touched the guard in either build.
  • Next test: a later test of this wording would need a memory-changed-fact bar that counts only runs shown the warning. That would be a new predeclaration.

Validation

  • Counts: checked against each run's jsonl and trajectories. The leak was traced to a read of the session's own trajectory, and guard edits were scanned in both builds.
  • QA: cargo xtask qa exits 0 on an idle host.

W cut stale runs with a user lesson from five of ten to one, but scored
seven passes on memory-changed-fact against the baseline's nine, so it
fails that predeclared bar and is parked. Both of W's misses there were
runs never offered the lesson; that and the cost figures are reported,
not decided on.

Validation: counts checked against each run's jsonl and trajectories,
the leak traced to a read of the session's own trajectory, guard edits
scanned in both builds; cargo xtask qa exits 0.

@jadenfix jadenfix left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lane B review: OK to merge on green. I read the outcome against the predeclared reading on main, and it follows it:

  • User task: baseline 5/10 stale (≥2, so there was harm to remove) and W 1/10. Four fewer clears the two-fewer bar. It is recorded as a direction, as fixed.
  • Agent obsolete: 0 vs 0, so W passes.
  • changed-fact: the bar counts passes, and 7 < 9, so W fails. As a share of scored runs it fails too (78% vs 90%), so the leak exclusion doesn't change the call. Dropping the leaked run follows the predeclared rule, and the cell still has nine scored runs, so no extra repeats were owed.
  • Regressions: equal passes on both tasks, so W passes.
  • W fails one bar, so it is parked and main is unchanged. That is the decided rule.

The figures conditional on the lesson add up: the baseline's 8/8 offered plus two not offered (one pass) gives 9/10, and W's 7/7 offered plus two not offered (both fail) gives 7/9. Keeping that under "reported, not decided on", and saying a warning-only bar would need its own predeclaration, is the right handling. I've nothing to add.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Independent exact-head scientific-integrity review — CLEAR for this documentation-only outcome record.

Pinned identity:

  • base 5c8f06316022d080e3fa926ecb75c0fc0acc9d1b
  • head 9d392452fd3d7cac0a442ed99db4a6e5b06fae46
  • tree 54df6ccf3339f9c0cb94c1673dd23494f94ad68b

I compared the single-file, +47-line diff with the predeclared decision rules on the base. The recorded calls are consistent with those rules:

  • obsolete-user: stale 5/10 → 1/10, passing the ≥2-run reduction bar and correctly described as direction only;
  • obsolete-agent: 0/10 → 0/10 stale, pass;
  • changed-fact: 9/10 → 7/9 passes after the disclosed self-trajectory leak exclusion, fail;
  • regressions: 3/3 = 3/3 and 1/3 = 1/3, pass;
  • because all four bars were required, W is correctly parked and main remains unchanged.

The conditional counts also reconcile: baseline 8/8 offered plus one pass among two not-offered runs = 9/10; W 7/7 offered plus two not-offered failures = 7/9. Keeping that observation outside the predeclared decision and requiring a new predeclaration for any warning-only analysis is scientifically sound.

Verification:

  • local git diff --check 5c8f063..9d39245: PASS
  • local checkout after review: clean
  • exact-head hosted check job in run 38530247722: PASS — fmt, clippy with -D warnings, workspace tests, and memory/CLI/offline-provider checks all passed
  • initial messages failure was limited to PR-description policy; rerun 38530261805 passed title, description, and every-commit policy
  • GitGuardian: PASS

I could not independently rerun Cargo locally because this executor has no Rust toolchain; the exact-head hosted QA job is therefore the executable evidence relied upon.

Scope boundary: this clears only the truthful recording of the predeclared wording experiment. It does not verify Rusty's portfolio core workflow, add a GIF/MP4 demo, earn current-main/demo coverage credit, or support a wording-effect claim. No merge action taken.

@jadenfix
jadenfix merged commit e422e83 into main Oct 14, 2026
4 of 5 checks passed
jadenfix added a commit that referenced this pull request Oct 15, 2026
The changed-fact bar now says how a run counts as offered the lesson,
as a rule over the trajectory fixed before any run, needs at least five
offered runs per build or stays undecided, and records #102's figures as
the reason the bar changed.

Validation: docs only; cargo xtask qa exits 0.
jadenfix added a commit that referenced this pull request Oct 15, 2026
Baseline e7f0a8f against W 27fe97b on the seed-marker cohort. The user
task decides as before; changed-fact compares failure shares among runs
offered the lesson, with a floor of five offered runs per build.
Validation: the offered rule reproduces #102's 8/0 and 7/0; docs only;
cargo xtask qa exits 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant