Arc 3: commit_runtime — 52 verified envs + filter/leak/artifact fixes - #47
Merged
Merged
Conversation
P0 audit fixes from Arc 3: - Reject conventional-commit types that are not bugfixes (chore/docs/feat/refactor/style/test/ci/build/perf/revert). The existing `_CC_PREFIX_RE` only stripped the prefix; a feature/refactor commit would then walk into bootstrap + validation and waste cycles. - Require at least one bugfix-positive signal: a `fix:` prefix, a `Closes #N` / `Fixes #N` issue trailer, or a bugfix keyword in the subject (fix/bug/regression/crash/broken/incorrect/fail/...). Stops bare-subject feature commits with no type prefix from sailing through. - Strip solution leaks from the instruction. Commit subjects and bodies routinely name the function being fixed and link to companion fix-PRs / commit SHAs; `_strip_info_leak` (ported from pr_runtime) removes them. - Extend `_CLOSES_RE` to catch the markdown form (`fixes [#N](url)`) so these don't leak through after the subject strip.
…on parity Three P1 instruction-quality + metadata wins from Arc 3: - Issue-fetch fallback. When a commit has `Closes #N`, source the problem statement from the linked issue body (via github.fetch_issue) instead of the commit message. This is Arc 2's biggest leak-reduction lesson applied to commits: the bug *report* doesn't usually name the function being fixed; the commit *message* does. Commits without a linked issue fall back to the leak-stripped subject + body as before. - Reflow long bodies. Port `_reflow_pr_body` from pr_runtime so commit bodies get the same template-noise stripping (HTML comments, checklist headers, multi-blank collapse, length cap) as PR bodies do. Commit messages can be just as verbose as PRs in practice. - Stamp `reward_calibration`. Add the same `[metadata.repo2env. reward_calibration]` block pr_runtime writes (`f2p_count`, `p2p_count`, `source_files`, `loc_changed`, `difficulty`) and use the computed difficulty bucket on the HarborTask instead of the hard-coded "medium". This gives the launch-side manifest enricher (`eval_grade` = command- resolved AND p2p_count > 0) the fields it needs without re-parsing. 3 new tests cover issue-fetch override, reflow on template-noisy bodies, and the reward_calibration stamping (28 commit_runtime tests pass; full suite 723).
Audit on the first 56-env scale sweep surfaced three concrete issues:
- The emitted test.sh fell back to the binary exit-code reward because
build_eval_script was called without fail_to_pass/pass_to_pass — and so
reward.json was never written, tracked / command_resolved / the parser
breakdown were silently lost. Pass the lists in so the graded path
triggers.
- Tasks shipped test.sh only, missing tests/{verifier.py,f2p.json,p2p.json}.
Arc 2's plain-artifacts refactor wasn't inherited; add the same
aux_files=_runtime_aux_files(...) wiring pr_runtime uses, gated on f2p
presence so --skip-validation tasks still build.
- Extend _strip_info_leak (pr_runtime, shared by commit_runtime) to catch
two soft-leak patterns the audit found in ~5/56 tasks: trailing squash
trailers like `(#1234)` and cross-repo issue refs without a closes
keyword (`gorilla#739`). Existing pattern only caught the
closes/fixes-prefixed and the markdown-with-url forms.
commit_runtime is still experimental (instructions are thinner than pr_runtime's when no linked issue exists, and Go-subtest parsing on stretchr/testify still over-reports untracked failures). But we now have 52 oracle-verified envs published at AdithyaSK/repo2rlenv-commit-runtime — added to the HF "Verifiable RL Environments" collection. build_enriched_manifest also got the trial-id matcher hardened so harbor's SHA truncation (12-char commit SHA -> 9-char in trial dir names) no longer drops tasks from the count.
- Rewrote docs/pipelines/commit_runtime.md to match the current pipeline (issue-fetch fallback + leak strip + reward_calibration + plain artifacts; the old doc still claimed an LLM authors instruction text, which was never the case in shipped code). - Added docs/release_notes/v0.8.3/findings-commit_runtime.md following Arc 2's findings template: published dataset link, what changed, validation evidence (52/52 resolved, 47/52 command_resolved, 47/52 eval_grade), honest limitations (thin no-linked-issue instructions, Go-subtest parser over-counting, lower yield on PR-driven repos), and the critical Arc 2 inheritance bug we caught at oracle-gate time.
adithya-s-k
changed the title
adithya-s-k
marked this pull request as ready for review
May 28, 2026 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arc 3 of the v0.8.3 pipeline-by-pipeline optimization. Closes #46. Predecessors: Arc 1
pr_diff(#40), Arc 2pr_runtime(#45) — both merged.What shipped
Dataset:
AdithyaSK/repo2rlenv-commit-runtime— 52 oracle-verified envs across 12 repos (Python + Go). Added to the Verifiable RL Environments collection.resolved(tracked SWE-bench-style)command_resolved(clean test command)eval_grade(cmd_resolved + has P2P guard)Kept experimental — not promoting to stable yet. The structural limitation (commits without
Closes #Ngive thinner instructions thanpr_runtimedoes) hasn't fully gone away. README/at-a-glance table updated accordingly.Code changes
2549b0f) — non-bugfix conventional-commit-type rejection (chore:/docs:/feat:/refactor:/ …) + bugfix positive-signal requirement (fix:prefix ORCloses #NOR keyword in subject).5db9d2d) — issue-fetch fallback whenCloses #Nis present (sources problem statement from the issue body, not the leak-prone commit message), body reflow,reward_calibrationmetadata stamping for manifest parity withpr_runtime.758b56d) — commit_runtime was callingbuild_eval_scriptwithoutfail_to_pass/pass_to_pass, hitting the binary-reward fallback path;reward.jsonwas never written across the whole pipeline. Plus emitted tasks shipped onlytest.sh, missing the plaintests/{verifier.py,f2p.json,p2p.json}artifacts. Both fixed. Extended_strip_info_leakto catch trailing(#NNNN)squash trailers + cross-reporepo#Nrefs (shared withpr_runtime).e7e5613) — dataset link in the experimental list; hardened the trial-ID matcher inplans/build_enriched_manifest.pyso harbor's SHA truncation doesn't drop tasks from the count.07b0b2a) — rewrotedocs/pipelines/commit_runtime.mdto match the current pipeline (the old doc still claimed an LLM authors instruction text). Addeddocs/release_notes/v0.8.3/findings-commit_runtime.md.Audit / validation
psf/requests-6404f3,testify-15f682,cli-195aaf,cli-b4f42d— all single flaky P2P, 0.97-0.999 on gold).Out of scope (deliberately)
pr_runtimemine 30+ per repo. Forcommit_runtimethose same repos merge-commit their PRs and yield ~0 directly mineable bugfix commits (correctly filtered out). 52 is what the pipeline's natural fit produces on the bootstrap-cached repo set; the remaining ~50 would require new bootstraps which is out of Arc 3's scope.commit_stream/ continuous variant.pr_streamwas already removed as scope-creep; if needed it's flags oncommit_runtime, not a separate pipeline.