[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments

@Bounty-hunter

Current state: @Bounty-hunter remains the assignee. The four proposed slices #334, #335, #336 and #337 remain open in draft #338, now at fd0f396c2bd7574a97af013120d4d0fb1733699f. The formal-integration documentation and updated author index now record an engineering-benchmark-v1 correctness-artifact adapter in skill_eval_compare and skill_eval_local: freeze the common task and both skill arms, retain each actual implementation.py, execute the artifact through Linux bubblewrap, check arrays with the parent-process shared oracle and reverify retained files in report. This supersedes the earlier unimplemented formal-compare status, but does not add provider generation or skill_eval_improve. The starter is already correct and can PASS unchanged; this entry supplies no timing discrimination or evidence of skill/Agent benefit. Artifact replay isolation does not qualify producer permissions or make public inputs/returned-call records tamper-proof execution evidence. The published software-control record attributes 49 passing tests with zero skips, including five real sandbox cases, to source dfcc1855; its hosted run completed full checks, package checks, references and artifact controls successfully. The record reports 3,631 passed / 18 skipped in each full suite. The source-to-head delta contains only three documentation/evidence files; those prior results retain their tested identity. Current-head Ready checks finished at 2026-11-04 00:31 UTC: both full suites, both installed-package checks, references and artifact controls passed; draft KV qualification skipped. This synchronization verified job states and the documentation delta, not a full source review or independent JUnit replay. Local Windows new tests report 44 passed / five Linux skips; legacy batches remain non-green with failures reproduced on baseline. Prior compare exit-code regression evidence, v1 performance FAILs and published 300-sample v2 measurements remain historical and are not rebound to this adapter/head. Reliable performance ranking, real Agent/skill benefits, provider generation, improve integration and GPU/NPU/service qualification remain open. The earlier replay-only review at ac03ae23 is not full review of this head. Contributor-reported delegation does not replace maintainer scope acceptance; the original tracker authorization below is unchanged.
Decision needed: Whether to accept, revise or reject the proposed first CPU subset, shared contract and scoped delivery criteria, from requester/acceptance owner Dong1017. No such decision is recorded in the inspected parent/child discussions.
Next action: Existing assignee @Bounty-hunter coordinates the recorded scope decision and full current-head PR review, keeping v1/v2 evidence and consumer boundaries explicit and preserving failures and open acceptance items. The separate optimization-search-policy RFC remains separate under the earlier design review. This update records status, not new implementation, paid execution, hardware, merge or closure authorization.

Goal

Track a minimal, reusable engineering benchmark foundation for InferRSI, organized around SearchSpace and engineering objectives, not around individual skills.

Reuse and adapt existing benchmarks wherever possible. The shared task/environment/validation packages should support:

Workbench measurement of code/configuration candidates;
SkillEval controlled agent comparisons for skill/knowledge PRs;
overall harness experiments (task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248) and optimization-search experiments (task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249).

A benchmark measures the produced implementation. Attributing an improvement to a skill or search policy additionally requires a controlled experiment; these are separate responsibilities.

Candidate coverage, to narrow before implementation

Operators: representative shapes/dtypes, reference correctness and numerical tolerances, latency/throughput.
Communication: topology and message-size conditions, correctness, latency/bandwidth.
Memory: representative workloads, peak allocation/reservation definitions, capacity/OOM boundaries, with correctness and performance constraints.
Further SearchSpace slices only when justified by an actual campaign or accepted requirement.

These are coverage directions, not a commitment to build every category or hardware combination in the first delivery. No per-skill large question bank or universal aggregate score is required.

Minimal shared contract to discuss

For each selected task: pinned upstream source/license, workload and allowed search scope, environment/device requirements, baseline/reference implementation, independent validation commands, metric units and timing boundaries, repeatability checks, and explicit unsupported/missing coverage.

Keep task/grader versions fixed within a comparison. Reuse existing runners and evidence mechanisms rather than introduce another execution platform. CPU/software checks do not establish GPU/NPU correctness or performance; unavailable environments remain unverified.

Relationship to existing work

[RFC] Benchmark selection by task for InferRSI #60 (closed): task-specific benchmark selection research, especially model/serving workloads. Reuse its source report; this tracker concerns the shared minimal engineering benchmark delivery.
[RFC] Iteration Campaign Plan: one table for model, hardware, goal, search plan and test plan #30 and [RFC] Hardware-aware optimization goals throughout the RSI workflow #143: campaign plans, hardware-aware goals and constraints.
[Tracking] SearchSpace distillation coverage and knowledge consolidation #91: SearchSpace knowledge coverage, not benchmark execution coverage.
Skill Eval v1.1: automated evaluation for skill additions and updates #88: SkillEval comparison/reporting workflow; reference supplement. Benchmark construction remains separable from SkillEval implementation.
task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248: overall agent benefit experiment; task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249: parallel candidate-search scheduling. Neither alone supplies this shared benchmark foundation.

Initial reuse candidates include KernelBench, BackendBench, existing framework tests/benchmarks, and applicable task-package practices from SkillsBench. Listing a source is not adoption or evidence that it supports the target device.

Tracking and acceptance

Agree a small initial SearchSpace/task subset, upstream reuse choices and accountable implementer.
Agree the concrete task/environment/oracle interface and scoped delivery criteria before implementation.
Deliver and validate the selected subset, retaining reproducible measurements and explicit coverage limits.
Demonstrate that the selected shared package can be consumed by its agreed Workbench/evaluation use cases without duplicating task truth or runners.
Record evidence-backed acceptance of the agreed subset; do not close merely because a catalog/document exists or software CI is green.

Ownership and authorization

Requested by @Dong1017 as an umbrella tracking issue. Current authorization is to track and clarify this work, not to launch implementation, claim another contributor's work, run paid models or allocate hardware. Implementer and initial subset remain unassigned. This issue does not reopen completed scopes or alter #248/#249 ownership.

Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation.

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

Status: Proposed. Base revision a8215b3.
Creates no threshold, revision pin, campaign, promotion or execution authority.
Related: this comment on #252; depends on #252 for task/oracle packages, orthogonal to #249 (parallel lanes), extends row 4 of the campaign plan RFC #30.

Item	Value
Scope	Machine-readable search-space projection; candidate card schema; admission gates; cost tier → `RunSpec` mapping; phase-locked round recording; stop evaluator; negative-result ledger
Out of scope	Candidate synthesis, code editing by a model, autonomous execution, agent-side hook enforcement, `evaluation.py` statistics, parallel lanes (#249), benchmark task packages (#252)
Reuses	Campaign plan row 4 field set; `RunSpec` (`src/inferrsi/runspec.py`); AB/BA runner; `assessment()` short-circuit shape (`src/inferrsi/harness/improvement.py`); existing rejected-direction cases
Adds	No 🟢 SearchSpace cell, no measured coverage, no skill, no claim

1. Problem

optimization-search-policy already states a complete policy: four admission considerations, a seven-phase locked round, a six-tier cost ladder and four stop criteria. Every one of those is prose. Nothing in src/inferrsi/ reads it.

The gap is visible in three concrete places:

Row 4 of the campaign plan already specifies the candidate card — ID, plane, target stage and its measured exposed share of E2E, conditional headroom, diagnostic, hypothesis, mechanism class, risk class, cost tier, prior art or negative record, kill criterion, status, evidence link. It is a human-filled table cell. There is no schema, no validator and no way to ask "which cells are still admissible".
Prior-art and negative-record lookup is unimplemented and, as written, contradicted. The policy requires recall-complete free-text grep over full case bodies, while src/inferrsi/catalog.py:127 scores records by token-AND (every token must appear) and ranks by raw occurrence count. That is a precision-ordered substring index, not a recall-complete case search. The three existing rejected-direction cases are plain Markdown with no query interface.
The stop criteria have no inputs. S1 needs measured uncertainty and removable time; today evaluation.py:36 reports range_pct_of_median as descriptive only and asserts "statistical_significance": "not established". S3 needs a verdict history across rounds; nothing aggregates rounds.

The result: the policy is correct and unusable. An agent can read it and still re-enter a recorded dead end, because nothing makes the dead end queryable at the moment of candidate generation.

The narrow claim of this RFC: the admission, ordering and stopping decisions can be made deterministic over declared inputs and existing evidence, without synthesizing candidates and without spending compute. That is a contract problem, not a modelling problem.

2. Design principles

Row 4 is the candidate card. Do not introduce a second ontology. This RFC defines a machine-readable form of row 4 plus the evaluation over it, and defers to docs/searchspace-matrix.md as the authoritative coverage matrix. No third matrix.
Gates produce refusals, not rankings. A gate can kill a candidate, mark it unadmissible without a diagnostic, or raise its burden of justification. It never scores two candidates against each other. Ranking by expected value is a further decision that needs the measurement contract to exist first.
Absent input stays absent. Every gate that lacks an input returns unverified with the cheapest check that would resolve it. No default fraction, no assumed share, no fallback threshold. This matches the existing INCONCLUSIVE > FAIL > PASS precedence.
Immutability follows row 4's freeze semantics. A kill criterion and a cost tier are fixed once the candidate has run; changing either requires a new candidate ID, not an edit.
The policy governs records, not agent behaviour. vllm-evolve enforces phase order with a PreToolUse hook. InferRSI cannot and should not: it is not in the agent's tool path. The enforceable analog is that a round which skips a phase cannot be recorded as a qualifying round, and an unrecorded round cannot feed a promotion or a stop decision.

3. Objects

3.1 Search-space cell index

Generated from docs/searchspace-matrix.md by extending tools/build_skill_coverage.py (which already resolves SEARCHSPACE = Path("docs/searchspace-matrix.md")), never hand-authored. The matrix remains authoritative; the index is a derived artifact with a --check mode that fails on drift.

{
  "schema_version": 1,
  "id": "accelerator.kv.admission",
  "layer": "accelerator",
  "stages": ["diagnose", "configure", "verify"],
  "target_classes": ["serving", "kv-cache"],
  "skill_ids": ["diagnose-vllm-admission-pressure"],
  "evidence_pins": [{"source": "<repo>", "commit": "<40-hex>"}],
  "maturity": "source_candidate",
  "plane_hint": "model",
  "pruning_claims": [
    "Queue length and KV usage do not establish allocation pressure",
    "stats are gated and incomplete"
  ],
  "rejected_direction_refs": ["casebook/cases/qwen-ascend-rejected-directions.md"]
}

pruning_claims is the load-bearing field. The matrix's Cases / Experience column currently holds falsifying statements that only a human can read — "Shape agreement is not conversion", "CPU bypasses accelerator capacity and segmentation gates", "matching total tokens does not preserve peer rows". Those are pre-registered counter-claims. Surfacing them as queryable constraints is the cheapest available reduction of the candidate space, and it costs no measurement.

maturity carries the existing key unchanged (covered / partial / gap); source_candidate and not_run remain distinct states and neither becomes an admissibility signal.

3.2 Candidate card

The machine form of row 4. Field names are the campaign plan's; this adds typing, not vocabulary.

{
  "schema_version": 1,
  "id": "D-01",
  "plane": "model",
  "cell": "accelerator.kv.admission",
  "hypothesis": "If admission accounting charges <mechanism>, <metric> improves because <causal step>",
  "mechanism_class": "ownership",
  "risk_class": "lossless-exact",
  "cost_tier": "profile-run",
  "stage": {"name": "<stage>", "exposed_share_pct": null, "share_source": "unmeasured"},
  "headroom": {"removable_ms": null, "assumptions": [], "uncertainty_ms": null},
  "diagnostic": {"cost_tier": "profile-run", "kill_criterion": "<observable>"},
  "prior_art": [],
  "negative_record": [],
  "falsification": {
    "metric_key": "<dotted.positive.key>",
    "direction": "minimize",
    "criterion": "<the observation that would reject this hypothesis>"
  },
  "kill_criterion": "<immutable once run>",
  "status": "planned"
}

Validation rules (all reject, none warn):

falsification.criterion nonempty, and falsification.metric_key resolves to a positive finite number under the existing metric contract (model/protocols/metrics.py). A candidate that no single metric could reject is not a hypothesis; it is a wish.
kill_criterion nonempty and not equal to the falsification criterion.
plane ∈ harness | model | infra, one per candidate (AGENTS.md RSI contract).
cell must resolve in the generated index; mechanism_class must be one of the seven declared tags.
stage.exposed_share_pct may be null only when stage.share_source == "unmeasured" and a diagnostic is attached.

3.3 Admission gates

Ordered cheapest-first. Each returns admit | refuse | admit_conditional | unverified, with a reason and, where applicable, the cheapest resolving action.

G0 — Amdahl plausibility.

upper_bound = exposed_share(cell) × plausible_removal(mechanism_class)
refuse if upper_bound < residual_target

residual_target is campaign-declared, never a tree default. This is an upper bound on what the stage could yield, not a prediction, and it is the cheapest way to discard a candidate before any compute is spent. Missing share → unverified with required_action: "run the cheapest diagnostic in this cell" (typically a profile run, which is a measurement, not a comparison).

G1 — Mechanism class alignment. The declared mechanism_class must be reachable from the measured bottleneck class. A residency candidate against a compute-bound stage is refused with reason: class_mismatch. The five/six families the policy names — ownership, work elimination, residency, algorithm, configuration, fusion/operator, graph/launch, parallelism/communication, precision/approximation — are the declared tag set from row 4 and are not extended here.

G2 — Prior art and negative record. Two lookups, deliberately different:

Prior art (recall-oriented). Full-text grep across case bodies, per the policy's explicit requirement. Result is a list of leads.
Negative record (classification-oriented). Structured query over the ledger (§3.7), keyed by (cell, mechanism_class, backend, device, model, workload, evidence_level).

On a negative-record hit the gate returns admit_conditional, never refuse. The card must then name:

the rejection class — one of quality_rejection, no_demonstrated_gain, resource_failure, confounded_control, unintegrated_lead, analytical_rejection;
what changed since that record (revision, device, workload, evidence level);
the cheapest check that would distinguish the two conditions.

A keyword match is a lead, not a causal diagnosis and not an automatic veto. A different implementation failing does not ban a mechanism family.

G3 — Cost tier assignment. The cheapest tier that can separate the hypothesis from local noise, declared before measurement. A tier above the minimum needs a stated reason. Escalation is one-way and only on a positive result at the current tier.

3.4 Cost ladder → `RunSpec`

The policy's six tiers map onto RunSpec knobs. The mapping is shape, not numbers; tier presets are campaign-scoped in rows 4 and 6.

Tier	`RunSpec` shape	May produce a verdict?
`static/config`	no benchmark command; validation command only	correctness only — never a performance verdict
`microbench`	narrow `metric_key`, single scenario, minimal `repetitions`	screening only
`profile-run`	measurement run, no A/B pair	no verdict; supplies `exposed_share` for G0
`scenario@c1`	full paired `RunSpec`, one declared regime	screening PASS/FAIL; not acceptance
`multi-regime`	paired `RunSpec`, declared regime set	may support acceptance if rows 3/5 declare it
`full-cohort`	the frozen acceptance regime	the only tier that can feed a public claim

Two hard rules:

Escalate only on positive. A negative or inconclusive result at tier k does not buy a run at tier k+1.
Screening may reject early; it may never substitute for the acceptance regime. A scenario@c1 PASS is not an acceptance PASS, and no promotion rule may treat it as one. This mirrors the CRISP adaptation boundary already recorded in the policy: screening rejects, it does not conclude.

3.5 Phase-locked round

Seven phases, with the existing runner as P5's executor. The lock is enforced on the record:

flowchart LR
  P1["P1 Read context<br/>cell + prior art + negative record"] --> P2["P2 Design<br/>candidate card"]
  P2 --> P3["P3 Apply<br/>bounded, one plane"]
  P3 --> P4["P4 Verify correctness"]
  P4 --> P5["P5 Evaluate<br/>AB/BA at declared tier"]
  P5 --> P6["P6 Compare<br/>better / worse / inconclusive"]
  P6 --> P7["P7 Keep or discard<br/>ledger entry, unconditional"]
  P7 --> P1
  P4 -.->|"fails"| L["Ledger: correctness_rejection<br/>no benchmark budget spent"]
  L --> P1

A ledger entry is qualifying only if it records P1 inputs (the cell and the negative-record lookup result), a P2 card, a P4 outcome, and a P5/P6 result. An entry missing P4 is recorded as unqualified and cannot be cited by a stop decision or a promotion.

This is weaker than a hook by design: it cannot stop an agent from running a benchmark out of order, it can only refuse to treat the result as evidence. Given the repo's non-autonomous posture, that is the correct strength.

3.6 Stop evaluator

Deterministic over the ledger. Each criterion yields {decision: stop | continue | unverified, reason, required_action}.

ID	Criterion	Inputs	Behaviour when input missing
S1	Current-mechanism headroom exhausted	removable exposed time, `uncertainty`	`unverified` — never infer headroom from a roofline
S2	Amdahl ceiling reached	residual target, plausible removal over remaining stages	`unverified`
S3	Repeated inconclusive on the same hypothesis	count of consecutive `inconclusive` for one `hypothesis` hash	`continue` with `required_action: "repair the measurement protocol or change hypothesis"`
S4	Negative record saturated	every cell in the declared scope has a terminal ledger entry	`stop` with `required_action: "escalate abstraction level"`

S1 must not be satisfied by proximity to a roofline. The existing policy says this explicitly and it matters: changing the algorithm changes both work and traffic, so proximity to the old roof does not rule out improvement. The evaluator encodes that by requiring a measured removable quantity, and returning unverified otherwise.

S3's repetition count is campaign-declared. No tree default.

3.7 Negative-result ledger

One record per terminal round outcome, including worse and inconclusive. This is the compounding asset: the repository already keeps three rejected-direction cases by hand, and each one exists because someone paid to learn it.

{
  "schema_version": 1,
  "candidate_id": "D-01",
  "cell": "accelerator.kv.admission",
  "mechanism_class": "ownership",
  "context": {
    "repo": "<repo>", "revision": "<40-hex>", "backend": "ascend",
    "device": "<exact SKU>", "model": "<model>", "workload": "<id>"
  },
  "evidence_level": "scenario@c1",
  "hypothesis": "<as tested>",
  "metric_key": "<key>",
  "verdict": "FAIL",
  "rejection_class": "no_demonstrated_gain",
  "observed": "<measured values and uncertainty>",
  "constraint_derived": "<the search constraint this failure establishes>",
  "reopen_condition": "<what changed would justify retrying>"
}

constraint_derived is mandatory. A negative result that establishes no constraint has not narrowed the space and cannot satisfy S4.

4. Relationship to existing work

Work	Relationship
#252	Supplies the task package and oracle that P4/P5 consume. This RFC defines which candidate is admissible and what tier measures it; #252 defines what the measurement runs against. Neither duplicates the other's truth.
#249	Parallel lanes are an orthogonal scheduling change. A lane submits a candidate card; the round records it. Nothing here assumes one lane.
#30 (campaign plan RFC)	This RFC is the schema and evaluator for row 4, and defers to rows 1/2/3/5/6/7 unchanged. It proposes no new frozen row.
#248	Harness benefit experiment. The ledger feeds it with per-round outcomes; this RFC does not run it.
#91	SearchSpace knowledge coverage. The cell index is derived from that coverage, not a second copy of it.
#88	SkillEval. Its `assessment()` short-circuit ordering (`incomplete_evidence → unknown_usage → quality_floor → regression_limit → no_improvement`) is the precedent for §3.3's ordering, not a component to reuse: it scores skill-text edits, not inference candidates.
`optimization-search-policy` skill	Promoted from prose to contract. The skill remains the human-readable rationale and gains references to the generated schemas.

5. Dependency and ordering

The gates can land first; multi-round search cannot.

Two preconditions for using this contract across more than one round:

Measurement resolution. harness/evaluation.py:36 compares two independent medians and discards the pairing that the AB/BA schedule already established. Over repeated rounds, selecting the best of N unvalidated comparisons is selection bias, not tuning, and it will deposit noise into the ledger as if it were a constraint. The minimum repair — paired difference plus a resolvability gate that refuses PASS when baseline spread is on the order of min_improvement_pct — is a change to evaluation.py, not to this RFC, and is not proposed here as a slice.
Candidate-space indexing. Without the cell index and the negative-record query, G0/G1/G2 return unverified for everything and the gates reduce to a schema validator.

So the slices below are ordered accordingly, and no slice depends on statistical work.

I am not claiming that fixing resolution is sufficient. It is a precondition for round count > 1, and stating it here is cheaper than discovering it after a campaign.

6. Delivery slices

Slice	Content	Depends on	Hardware
A	Cell index generated from the matrix; `--check` drift gate	—	none
B	Candidate card schema + validator; rejects unfalsifiable cards	A	none
C	Negative-record ledger schema + query over the three existing cases	A	none
D	G0–G3 evaluation; `unverified` paths	A, B, C	none
E	Cost-tier → `RunSpec` mapping + escalation refusal	B	none
F	Round recorder: qualifying vs unqualified entries	B, E	none
G	Stop evaluator S1–S4 over a ledger fixture	C, F	none
H	One synthetic end-to-end fixture: cards in, gate decisions and stop decision out	D, G	none

Every slice is model-free and hardware-free. No slice spends compute, allocates a device, calls a paid model, or produces a measured claim.

7. Acceptance criteria for this RFC

A1. The generated index round-trips against the matrix, and --check fails on an unreconciled edit. The matrix remains the only authoritative source; the index cannot be edited by hand.
A2. The card validator refuses: no resolvable positive-finite metric; empty kill criterion; kill criterion equal to the falsification criterion; more than one plane; an unresolvable cell; a null stage share with no diagnostic attached.
A3. A negative-record hit yields admit_conditional, never refuse, and a card without what changed fails to validate.
A4. A card cannot be assigned a tier above the minimum without a stated reason, and a scenario@c1 result cannot satisfy a promotion rule.
A5. A round missing P4 is recorded unqualified and is rejected as input to S3/S4.
A6. S1 and S2 return unverified — not continue — when headroom inputs are absent.
A7. A negative entry with an empty constraint_derived cannot satisfy S4.
A8. One synthetic fixture demonstrates that the gates select the subset a human would, and that the fixture contains no measured performance value.
A9. No new 🟢 cell, no measured coverage, no skill promotion, no claim. Every new artifact is status: candidate or a schema.

8. What this RFC does not authorize

No execution, device allocation, paid model call or benchmark run.
No change to evaluation.py, the runner, the AB/BA schedule or any frozen row.
No numeric default enters the tree. Repetition counts, tier presets, residual_target and S3's N are campaign-scoped in rows 4 and 6.
No promotion of any knowledge, skill or SearchSpace cell.
No claim that the policy improves search outcomes. The policy's own benefit is unmeasured and this RFC does not establish it.

9. Open questions

Does the cell index belong in src/inferrsi/data/ as package data, or stay a tools/ artifact? Package data would let knowledge search join on cells; it would also create a second shipped copy of the matrix.
Should G2's prior-art lookup use the existing token-AND catalog or a separate grep path? The policy demands recall-complete free-text search, and catalog.py:127 is precision-ordered by construction. Separate paths risk the two disagreeing.
How is plausible_removal(mechanism_class) bounded without inventing numbers? Options: campaign-declared per class, derived only from recorded positive results in the ledger, or left unverified until a campaign declares it. The third is honest and the least useful; the second cannot bootstrap on an empty ledger.
Does an unqualified round still write a negative ledger entry? It carries information but fails the P4 precondition. Recording it pollutes S4; discarding it loses a real dead end.
Should the ledger be issue-scoped or in-tree? Campaign parameters and numeric gates stay issue-scoped per AGENTS.md; a ledger that only lives in an issue is not queryable by the gate that needs it.

Proposed for review. Comment on scope and slice boundaries before any implementation claim. No part of this proposes reopening #248, #249 or #252.

@Dong1017

Issue/design review of the body updated 2026-10-29T11:14:20Z and the optimization-search-policy RFC, against main 7d1d82f55cda89842bd23ee2da91b521ee60eb08. Source/document inspection only; no benchmark, model, device or workflow execution.

The shared task/environment/oracle foundation remains a sound bounded goal. The RFC proposes a separate search-policy contract; it should not silently become a prerequisite or expand this tracker's benchmark acceptance. Please revise these three design points before implementing that contract:

P2 — G0 rejects useful incremental candidates (§3.3). upper_bound < residual_target compares one candidate's ceiling with the entire remaining campaign target. In a hypothetical campaign targeting a 10% latency reduction, two independent changes could each remove 6% of the original E2E time and jointly meet the goal, yet G0 refuses both before either runs. Preserve candidate contribution/composition, and reserve the whole-target impossibility test for the combined remaining opportunity. The existing search policy applies its campaign ceiling to all remaining stages together; the campaign template also retains conditional headroom, overlap, queueing, capacity and uncertainty. An unknown or inapplicable bound should remain unverified.
P2 — Static/profile tiers cannot be represented by RunSpec presets alone (§3.4–3.5). static/config supplies only validation, and profile-run explicitly has no A/B pair. Current RunSpec.validate() requires nonempty baseline and candidate commands, and the runner executes paired measurements before optional validation. Define diagnostic/correctness records and their transitions/qualification separately, keeping RunSpec for actual comparisons. Do not fabricate a baseline or performance verdict to pass validation. Clarify how a no-verdict diagnostic can advance under the positive-only escalation rule without pretending it completed P5/P6 comparison.
P2 — S4's per-cell terminal-entry test does not establish search exhaustion (§3.6–3.7). One matrix cell can contain several mechanisms and candidates. One failed candidate in each cell still leaves other candidates untried; G2 itself correctly says a negative hit is conditional evidence, not a veto. Scope exhaustion to an explicitly declared, versioned candidate/hypothesis set and matching context, consistent with the existing candidate-based stopping rule. Retain all terminal records, including correctness failures and inconclusive runs, but keep retention separate from eligibility to exclude a candidate or satisfy S4; inconclusive evidence need not establish a causal search constraint.

Next: @Dong1017 should retain #252's small shared-task scope, identify an accountable owner for the proposed policy contract, and settle these admission/execution/stop semantics before delegating it. Implementation owner and initial benchmark subset remain unassigned; neither this review nor the RFC demonstrates search benefit, accepts a benchmark delivery, or authorizes execution.

共享工程评测任务在 Draft PR #338，最新提交 fd0f396c2bd7574a97af013120d4d0fb1733699f。用户继续接入的委托已落实为正式 SkillEval 正确性产物 adapter；本索引不代替维护者范围验收。

task: select initial optimization tasks and benchmark reuse #334 — 选题复用：论文/开源与首批任务决策
task: share task inputs and evaluation rules across RSI consumers #335 — 统一规则：共享任务、输入、oracle 与版本记录
task: ship executable Softmax and RMSNorm CPU tasks #336 — 交付题包：Softmax / RMSNorm CPU
task: connect shared tasks to RSI and verify their use #337 — 接入验证：Workbench、正式两臂产物与独立检查

完整导读 · 四项交付总览 · 正式接入与命令 · 执行证据。

skill_eval_compare / skill_eval_local 现在可冻结同题与两臂 skill，保留各自实际 implementation.py，用 Linux bubblewrap 运行候选，并在父进程共享 oracle 检查；报告再验留存文件。starter 已正确，原样提交也能 PASS；此入口不提供性能分辨或 skill 收益结论。

source head dfcc1855 的 Linux软件控制 49通过、0跳过，包括5项实际sandbox案例；两套hosted全量各 3631 passed、18 skipped，安装包检查通过。后续文档仅记录结果，不改写被测版本。没有模型调用。

本地Windows新回归44通过、5项Linux控制跳过。旧路径两批分别66通过/14失败、133通过/1失败，全部失败报告在未改基线复现；本地批次仍不绿。历史全量3479/51/56与旧v1/v2性能证据另行保留，旧300个样本不重新绑定新提交。

真实Agent/skill收益、可靠性能排序、provider generation、skill_eval_improve及GPU/NPU/服务验收仍未完成。父任务和四子项保持打开，PR保持Draft；范围/验收决定仍待维护者记录。当前账号无主仓库write，原生父子关系仍用引用和此索引追踪。

Intent: the tracker asks the project to agree and deliver a minimal shared engineering benchmark foundation; the new comment reports a first CPU subset implemented in draft PR #338 and asks maintainers to adopt #334–#337 as native sub-issues. Verdict: human decision required — no accepted scope is recorded and this Issue explicitly withholds implementation authorization, so no development task should start. I verified the PR head's hosted checks pass and the new test module parametrizes to exactly 26 cases, but the raw samples/profiles and the Windows baseline comparison remain local-only and every tracker acceptance item is still open. The earlier design review of the separate search-policy RFC is untouched by this update (the RFC is kept out of this delivery, and the PR's check/measure/profile split matches its point 2), so it is not re-raised. Labels: priority:P2 (tracking umbrella whose decision is pending; no recorded deadline or blocking dependency).

F1 — The proposed delivery precedes the tracker's own scope/interface decision, and its authorization claim is not recorded

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Ownership and authorization" and acceptance checklist items 1–2; AGENTS.md:117; docs/agent-collaboration.md:13-15.
Problem: The update reports a finished 33-file, +1,962-line draft PR (feat: add shared CPU engineering benchmark tasks #338, head d0ce9d785e1bead9124eed1cf3f6e9f2e0cf05db, base 663f3b327998526a13aa31e72ad729025c18983c, author fork) while [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has unchecked "Agree a small initial SearchSpace/task subset, upstream reuse choices…" and "Agree the concrete task/environment/oracle interface and scoped delivery criteria before implementation", with no recorded decision. The PR text states "the requester explicitly authorized local implementation and publication on 2026-11-03", but no Issue comment or label records an accepted scope, and the Issue body itself says that reported delegation "is not a recorded acceptance of this parent scope". Assignment identifies accountability, not authority (docs/agent-collaboration.md:13-15).
Conditions: Any decision to keep feat: add shared CPU engineering benchmark tasks #338 moving, to record acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or to turn this tracker into an implementation task.
Impact: The owner must review a pre-built design instead of deciding the subset/interface first; a rejection discards the work. All four slices still carry the open item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and the tracker cannot close on a draft PR plus green CI.
Fix: Have the named decision owner record one comment accepting, revising or rejecting the CPU subset and the task/oracle interface; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists, and treat the PR's authorization sentence as a proposal rather than recorded delegation.

F2 — The claimed validation is not reviewable from repository artifacts

Where: docs/plans/issue-252/validation.md and validation-summary.json in PR feat: add shared CPU engineering benchmark tasks #338 at d0ce9d785e1bead9124eed1cf3f6e9f2e0cf05db; repository evidence rules docs/contributing.md:105-107 and evidence/README.md.
Problem: The validation record says the full raw logs, commands, stderr, timing samples and CPU profiles stay in a local directory outside the repository, and that the committed summary cannot substitute for independent replay. The committed summary reports campaign_acceptance: INCONCLUSIVE, 6 trials and 180 measured calls per task (360 total), but the samples, profiles and the claim that 56 Windows failures all reproduce on the unmodified baseline are not in the repo. The acceptance checklist asks for raw samples, rejected changes and evidence paths (docs/contributing.md:105-107).
What I could verify: hosted checks at the PR head are green (full-check 3.10/3.11, installed-package 3.10/3.11, declarations, pr-references; kv-task-qualification skipped), the new test module's 13 test functions parametrize to exactly 26 cases (matching the reported count), and the reuse claim is plausible at import level (harness/benchmarks/cli.py imports harness.runner.run_experiment; consumers.py imports scenarios.external.execute_trial and skill_evaluation.valid_checks). None of the new files exist on main at 663f3b3: no src/inferrsi/harness/benchmarks/, docs/plans/issue-252/ or src/inferrsi/data/engineering_benchmarks/ (verified in the checkout).
Impact: An acceptance owner cannot check sample retention, the baseline comparison or the stated coverage limits from repository bytes, so the tracker criterion "Deliver and validate the selected subset, retaining reproducible measurements" is not yet met; green draft CI establishes CI only.
Fix: Publish the per-task evidence (samples, commands, stderr, profile, hashes) with the PR or make tools/engineering_benchmark_validate.py reproduce the summary from committed inputs, and put the Windows baseline comparison in the same reviewable form.

F3 — The four proposed slices are not children of the tracker, and one slice is outside the pending-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (sub_issues_summary.total = 0), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, PR feat: add shared CPU engineering benchmark tasks #338 body, and task: connect shared tasks to RSI and verify their use #337's labels.
Problem: The update's tracking exists only as cross-references: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links, and the request to add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 needs a maintainer with write access (the comment reports the sub-issue API returns 404 for a READ-only account). PR feat: add shared CPU engineering benchmark tasks #338 correctly uses non-closing Refs (docs/agent-collaboration.md:102), so merging would close nothing and progress will not roll up. task: connect shared tasks to RSI and verify their use #337 also lacks status:proposed and priority:P2 (and bot-reviewed) that task: select initial optimization tasks and benchmark reuse #334–task: ship executable Softmax and RMSNorm CPU tasks #336 carry, so it does not appear in the same status:proposed decision queue.
Conditions: Deciding whether to adopt this four-slice split and tracking its acceptance.
Impact: Five records (tracker, four slices, one draft PR) must be reconciled by hand; the umbrella cannot show which slice is proposed, accepted or complete, and task: connect shared tasks to RSI and verify their use #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings. If not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep only the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record which minimal shared engineering benchmark subset, contract and owner are agreed; this edit records that @Bounty-hunter is now the assignee and has proposed four slices delivered together in draft #338, while accept/revise/reject stays with Dong1017. Verdict: human decision required — no accepted scope is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). The update's checkable claims hold: hosted checks at d0ce9d7 are green (full-check 3.10/3.11, 3570 passed on 3.11, installed-package, pr-references), kv-task-qualification is skipped by its draft == false gate, and validation-summary.json matches the reported INCONCLUSIVE / 360-sample state. F1: open — delivery and assignment are reported, but no accept/revise/reject decision is recorded, and the PR's unlinked "explicitly authorized local implementation" claim is still absent from the record. F2: open — re-read the pinned validation record: raw logs, samples, profiles and the Windows baseline comparison still stay outside the repository, so "retaining reproducible measurements" remains unreviewable from repository bytes (the new focused tests do run in the green hosted Linux suite). F3: open — #252 still has zero native sub-issues and #337 still lacks status:proposed/priority:P2/bot-reviewed. No label value is suggested: every axis already carries a value.

F4 — medium, actionable — The new status block contradicts the unchanged ownership text, so the tracker no longer records one authorization state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the unchecked acceptance item "Agree a small initial SearchSpace/task subset, upstream reuse choices and accountable implementer", and the closing "Next step"; AGENTS.md:117; docs/agent-collaboration.md:13-17.
Problem: The edit adds an assignee, four proposed slices and a delivered draft PR, but leaves text asserting the opposite: "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new "superseding the earlier unassigned status below" wording covers only the first of these. The first acceptance item still reads as unmet on the implementer half although an accountable implementer is now recorded, and all four child Issues (task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337) remain unassigned while the parent is assigned.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether further benchmark work may start or be scheduled.
Impact: The governing record simultaneously answers "implementer assigned, work delivered" and "no implementer, implementation unauthorized and not yet scheduled". docs/agent-collaboration.md:13-17 requires accepted scope and delegation to be recorded or linked, and a self-contradicting record cannot serve as that link; the effect is the same ambiguity that made F1 possible, now inside the Issue itself rather than only in the PR.
Fix: Rewrite "Ownership and authorization" and the closing "Next step" to the recorded state (accountable parent/coordinator @Bounty-hunter; acceptance owner Dong1017; decision pending; child implementers unassigned) and mark or delete the superseded sentences, then split the first checklist item into its recorded half (implementer recorded, done by assignment) and its open half (subset and reuse choices agreed). help:wanted ("Needs an owner or contributor") also now sits next to a recorded assignee; whether to reconcile that label is the owner's call, not something this review applies.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, contract and owner, and it now also records that its first CPU delivery in draft #338 has an unresolved maintainer review finding. Verdict: human decision required — no accept/revise/reject decision is recorded, the body still withholds implementation authorization, and nobody should start a development task. F1: open — the edit documents the unresolved #338 finding but the only decision-shaped event is Dong1017's PR review comment, which accepts nothing on scope. F2: open — raw samples/profiles remain outside the repository and the new body sentence confirms it, so "retaining reproducible measurements" is still unreviewable, while the cited timing samples come from the loop flagged at numpy_task.py:125. F3: open — #252 still has zero native sub-issues and #337 still lacks status:proposed/priority:P2. F4: open — the new "Next action" now directs work on #338 while the unchanged "Ownership and authorization" and "Next step" say implementation is not authorized and must wait. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit records delivery and an unresolved review, but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; fork plan docs/plans/issue-252/README.md at d0ce9d7; docs/agent-collaboration.md:13-17; AGENTS.md:117.
Problem: The updated block adds the unresolved feat: add shared CPU engineering benchmark tasks #338 timing finding to the state, but no decision from the acceptance owner is recorded. Verified: the only decision-shaped event since the last review is Dong1017's PR review comment r4456416093 (2026-11-03T15:00:54Z) on src/inferrsi/harness/benchmarks/numpy_task.py:125, which reviews measurement code and does not accept the CPU subset, the task/oracle contract, or the four slices. The fork plan still rests on a "user's explicit delegation on 2026-11-03" that has no counterpart anywhere in this repository's record, and the body itself concedes that delegation "is not a recorded acceptance of this parent scope".
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merges, to mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or to treat this tracker as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards a 33-file, +1,962-line delivery.
Fix: Have the named decision owner post one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists. A review comment on the PR is not that decision.

F2 — high, human decision required — The validation evidence is still not reviewable, and the cited timing samples come from the path a maintainer review says can measure replay

Where: docs/plans/issue-252/validation.md at d0ce9d7 (raw run kept in an out-of-repository inferrsi-252-validation-final directory; states the public summary "不能代替独立重放"), the new body sentence about unpublished raw timing/profile logs, and the open finding at src/inferrsi/harness/benchmarks/numpy_task.py:125.
Problem: Acceptance item 3 asks to "validate the selected subset, retaining reproducible measurements". The 360 samples offered as the delivery state were produced by the loop r4456416093 flags: the precheck calls compute(x) and every warmup/sample then reuses the same x, so a process-local memoizing candidate can satisfy the value assertions while most timed calls only replay stored output. Because samples and profiles stay outside the repository, an acceptance owner cannot tell measured execution from replay, nor re-check the "56 Windows failures reproduce on baseline" claim.
Conditions: Any decision that rests on suite_latency_ms, on the reported profile, or on the Windows baseline comparison.
Impact: "Reproducible measurements" cannot be confirmed from repository bytes, and a timing-path correctness defect is now recorded as known but not tracked anywhere in the acceptance checklist; green hosted checks establish CI only (full-check 3.10/3.11, installed-package 3.10/3.11, pr-references pass; kv-task-qualification skipped).
Fix: Publish per-task samples, commands, stderr, profiles and hashes, or make tools/engineering_benchmark_validate.py rebuild the summary from committed inputs; fix the timing boundary (frozen varying input sequence shared by baseline and candidate, plus a stateful memoizing negative regression) and republish before citing any latency number.

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text, now including a direct work instruction

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; docs/agent-collaboration.md:13-17.
Problem: The new "Next action" tells the assignee to "address the unresolved feat: add shared CPU engineering benchmark tasks #338 review finding with current-head evidence", while the body still says "Current authorization is to track and clarify this work, not to launch implementation", "Implementer and initial subset remain unassigned", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The "superseding the earlier unassigned status below" wording still covers only the assignee half.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery work" and "no implementer, implementation unauthorized, not yet scheduled"; that ambiguity is what let the pre-built delivery arise in the first place.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 is still outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (sub_issues returns an empty list), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues; task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 carries only kind:task and area:workbench. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress would roll up nowhere. The new task: connect shared tasks to RSI and verify their use #337 preregistration comment (2026-11-03T15:29:12Z) — the newest evidence the body relies on — therefore lands on an Issue that does not appear in the same status:proposed decision queue as its siblings.
Conditions: Deciding whether to adopt the four-slice split and tracking its acceptance.
Impact: Five records must be reconciled by hand, and the tracker cannot show which slice is proposed, accepted or complete; task: connect shared tasks to RSI and verify their use #337 is easy to miss.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings. If not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria; this edit records Bounty-hunter as assignee, pins draft #338 at c06b725 and reports a new 720-sample usefulness supplement. Verdict: human decision required — no accept/revise/reject decision exists and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). The edit's checkable claims hold: usefulness-summary.json contains exactly 720 samples, two profiles are published, no-epsilon passes the ordinary smoke but fails six assertions, both comparisons are FAIL (-50.52% unchanged-vs-unchanged, -76.95% einsum), and run 41665826142 is green with kv-task-qualification skipped. F1: open. F2: partially addressed — samples, profiles and hashes are now in-tree, but command logs/stderr/source snapshots remain local and the samples predate the head's v2 input protocol. F3: open. F4: open (sub_issues is still empty; #337 still carries only kind:task and area:workbench). F5 (new): the state block pins a superseded PR head.

F1 — high, human decision required — The edit still records delivery and an unresolved review, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-17.
Problem: The update records an assignee, four proposed slices and draft feat: add shared CPU engineering benchmark tasks #338 at c06b7255, and states the decision is pending, but no decision from the acceptance owner exists. The only decision-shaped event since the last round remains Dong1017's PR review comment r4456416093 on src/inferrsi/harness/benchmarks/numpy_task.py:125, which reviews measurement code and accepts nothing about the CPU subset, the task/oracle contract or the four slices. The body itself concedes the reported local delegation "is not a recorded acceptance of this parent scope". Assignment identifies accountability, not authority (docs/agent-collaboration.md:14), and work is supposed to follow accepted scope or recorded delegation (docs/agent-collaboration.md:15).
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merge, mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the open item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards the delivery.
Fix: Have Dong1017 post one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists.

Status: open (unchanged this round).

F2 — high, human decision required — Evidence retention improved, but the 720 samples are v1 and the run commands still are not reviewable

Where: docs/plans/issue-252/usefulness.md and usefulness-summary.json at c06b7255; docs/plans/issue-252/validation.md; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3.
Problem: The edit's claim is accurate: usefulness-summary.json holds exactly 720 samples_ms values across 144 arrays, the two usefulness-profiles/*.txt files exist, and the no-epsilon/einsum results match. What remains unreviewable is the rest: the usefulness.md reproduction section still says "完整原始日志留在本地" and the validation.md initial run (26 focused tests, 360 samples, 56 Windows failures "all reproduced on the unmodified baseline") is still summary-only, so the baseline comparison and command/stderr evidence cannot be checked from repository bytes. In addition, the published samples were produced by the v1 precheck/warmup/measure loop the P2 review flagged, and the PR head has since replaced that protocol, so they do not establish reproducibility for the current runtime.
Conditions: Any acceptance decision that rests on the 720 samples, the profiles or the Windows baseline comparison.
Impact: Acceptance item 3 ("retaining reproducible measurements") is only partly met; a decision owner still cannot distinguish measured execution from replay for the initial 360-sample record, and the newly published numbers are tied to a superseded task version.
Fix: Publish the remaining commands/stderr/source-snapshot hashes (or make tools/engineering_benchmark_validate.py rebuild the summary from committed inputs), and label the 720-sample supplement explicitly as v1 so it is not read as evidence for the current head.

Status: partially addressed (samples, profiles and hashes are now in-tree; commands/logs and the current-runtime rerun are missing).

F5 — medium, actionable — The state block pins a superseded PR head and describes a fixed timing path as unresolved

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" ("at c06b72555d4643df133c59fe7cead5a91e7b86f5 (runtime/task definitions unchanged from d0ce9d7...)", "An unresolved current-head P2 review …", "Current-head hosted full-suite/install checks … passed").
Problem: The PR head is now ce84506e7106c4fc48fafdfab55d2ea2f9dffd50 ("fix: bind benchmark timing to distinct paired inputs", committed 2026-11-03T16:04:22Z, after the body edit at 16:03:29Z). That commit changes src/inferrsi/harness/benchmarks/numpy_task.py (+73/−30) plus both task.json/README files to the numpy-axis1-v2 protocol, adds replay-validation.md, replay-validation-summary.json and +123 lines of tests, and its own record says "任务、grader 身份和指标的五 case 聚合均已变化". At head, evaluate() now builds per-call inputs with inputs(case, np, sequence=index + 1) and rejects repeated input bytes, which is the repair the P2 finding asked for. So the three statements above no longer describe the PR: runtime/task definitions did change, the P2 is addressed by an unreviewed fix commit rather than open at head, and run 41665826142's green checks belong to c06b7255 — at ce84506e full-check (3.10/3.11) was still in progress when checked.
Conditions: Any decision owner reading the tracker to evaluate the delivery state or the evidence behind it.
Impact: The governing record would have acceptance decided against a superseded measurement protocol and stale CI status; the published 720 samples are v1 and do not automatically validate the v2 runtime that is now at the PR head.
Fix: Update the block to the actual head, say whether the v2 fix is reviewed and which evidence/suite status belongs to it, and keep the v1 supplement labeled as v1 (or wait for the head's checks and review before re-stamping the state).

F3 — medium, actionable — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; AGENTS.md:117.
Problem: The edit adds a new "Next action" telling the assignee to coordinate the scope decision and "address the unresolved feat: add shared CPU engineering benchmark tasks #338 review finding with current-head evidence", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation, claim another contributor's work, run paid models or allocate hardware", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The "superseding the earlier unassigned status below" wording still covers only the assignee half.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery work" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark or delete the superseded sentences, and split acceptance item 1 into its recorded half and its open half.

Status: open (the new "Next action" adds a work instruction but leaves the contradictory text in place).

F4 — medium, actionable — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns an empty array), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues or dependency links. task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 carries only kind:task and area:workbench, so the Issue that now holds the supplemental results comment (6608126902) does not appear in the same status:proposed decision queue as its siblings. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress would roll up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand and the tracker cannot show which slice is proposed, accepted or complete; task: connect shared tasks to RSI and verify their use #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and owner; this edit re-pins the delivery to draft #338 at ac03ae2, reports the numpy-axis1-v2 replay fix, and cites the v2 record (40 author-reported tests, 300 timing samples) plus green hosted checks. Verdict: human decision required — the edit's checkable claims hold, but no accept/revise/reject decision is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). F1: open. F2: partially addressed (v2 identity hashes and planned/executed match flags are in-tree, but no timing values, profiles or commands are, so the 300 samples remain local-only). F3: open. F4: open (sub_issues is still empty; #337 still carries only kind:task and area:workbench). F5: fixed — the body now pins ac03ae2, the actual draft head; run 41672237031 has that head SHA with all checks green/skipped, and GraphQL confirms the P2 thread isOutdated=true/isResolved=false, matching the new wording. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records delivery and an unresolved review, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; docs/agent-collaboration.md:13-17; AGENTS.md:117.
Problem: The status block was rewritten again and now points at ac03ae2, but no decision from the acceptance owner is recorded. The only decision-shaped events remain Dong1017's PR review comment r4456416093 (a P2 measurement finding on numpy_task.py) and the author's reply r4457228612, which itself says it does not mark the thread resolved and makes no maintainer-acceptance claim. The body concedes the reported local delegation "is not recorded parent scope acceptance".
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merge, to mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or to treat this tracker as an authorized implementation task.
Impact: Work continues ahead of the tracker's own decision; all four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope and a rejection discards the delivery.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists. A review comment on the PR is not that decision.

F2 — high, human decision required — The v2 record publishes identities and input hashes but still no measurements

Where: docs/plans/issue-252/replay-validation.md:42 and docs/plans/issue-252/replay-validation-summary.json at ac03ae2; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: Verified this round: the committed v2 summary carries identity hashes, correctness/verdict fields, measurement_trials: 6, measured_calls: 150 per task, all_workloads_match/all_input_sequences_match/all_valid_executed_calls_match: true and a 35-entry input_sequence with per-call SHA-256, but no samples_ms, median or latency field at all; replay-validation.md:42 states the raw timings, logs and profiles remain local. The superseded v1 usefulness supplement published 720 samples and two profiles in-tree, so the current-head record is less reviewable than the evidence it replaces, and the "40 focused tests" / "300 timing samples" figures are author-reported.
Conditions: Any acceptance decision that rests on the v2 record or on the word "validated".
Impact: Acceptance item 3 cannot be checked from repository bytes, and the decision owner cannot re-derive the summary or inspect the timing path the P2 thread questioned; green hosted checks establish CI only.
Fix: Publish per-call samples_ms (as usefulness-summary.json already does) or make tools/engineering_benchmark_validate.py regenerate the summary from committed inputs, and say which figures are v2 versus historical v1.

F6 — medium, human decision required — The new state block drops the recorded local Windows suite failure

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" (the sentence now reading "The v1 360-sample integration and separate 720-sample usefulness results remain historical evidence…" plus "Current-head hosted full-suite/install/reference checks passed"), against docs/plans/issue-252/validation.md:40-41 at ac03ae2.
Problem: The previous body recorded that the full Windows suite remains non-green, with the failures reproduced on the unmodified baseline. The edit keeps the v1 samples as "historical evidence" but drops the suite status, and its remaining-gaps sentence lists same-code noise, performance ordering, SkillEval two-arm, agent/skill benefit and hardware — not the suite. The v2 record itself reports only the 40 focused tests plus CLI/SkillEval regressions, so no full-suite run is recorded at ac03ae2; validation.md:40-41 still says "3479 passed, 51 skipped, 56 failed … 全部在未修改的原始基线复现 … 全量结果仍为非绿色".
Conditions: An acceptance decision that reads the green hosted run (run 41672237031) as suite-clean for the delivery.
Impact: Acceptance item 5 explicitly says not to close because software CI is green; removing the caveat from the governing record makes exactly that misreading easier and weakens the "preserve failures" commitment in the previous "Next action".
Fix: Add one sentence to the state block: the local Windows suite remains non-green (56 failures, all reproduced on the unmodified baseline), and v2 has no recorded full-suite run beyond hosted Linux CI.

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — new "Current state"/"Next action" block vs the unchanged "Ownership and authorization" section, the closing "Next step", and acceptance item 1; AGENTS.md:117.
Problem: The edit adds "Existing assignee @Bounty-hunter coordinates the recorded scope decision and current-head re-review", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". Nothing in the new block marks those sentences superseded, although an implementer is recorded and the work has been delivered.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code") or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery and re-review" and "no implementer, implementation unauthorized, not yet scheduled", which is the ambiguity that allowed the pre-built delivery.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark the superseded sentences, and split acceptance item 1 into its recorded half and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns an empty array), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues. task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 (updated 2026-11-03T16:16:09Z, i.e. after the v2 work) still carries only kind:task and area:workbench. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress would roll up nowhere, and the body's new sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand and the tracker cannot show which slice is proposed, accepted or complete; task: connect shared tasks to RSI and verify their use #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at ac03ae23, cites Dong1017's scoped source confirmation (r4459426716) that the cached-output replay finding is addressed, and asks for a full current-head PR review. Verdict: human decision required — the edit's checkable claims hold (PR head and run 41672237031 both at ac03ae23, thread still outdated/unresolved, v2 record still local-only timings), but no accept/revise/reject decision is recorded and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). F1: open. F2: open. F6: open. F3: open. F4: open (verified: #252 sub-issues still empty, #337 still lacks status:proposed/priority:P2). F5: remains fixed. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit records a scoped technical confirmation, still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-16.
Problem: The new sentence cites r4459426716 as confirming the replay fix, and that comment is accurate about its own limits — it was written by the acceptance owner Dong1017 at 2026-11-03T19:23:55Z and says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open". So the only new decision-shaped event is explicitly not the scope decision. Checks this round: no acceptance comment exists on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 (their newest comments are the reporter's on task: connect shared tasks to RSI and verify their use #337 and bot reviews on task: select initial optimization tasks and benchmark reuse #334–task: ship executable Softmax and RMSNorm CPU tasks #336), and the body itself concedes the reported local delegation "is not recorded parent scope acceptance". Assignment and a scoped review identify accountability, not accepted scope (docs/agent-collaboration.md:14-16).
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merge, to mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or to treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards the delivery.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists. A scoped PR review comment is not that decision.

Status: open (unchanged in substance; the 19:23 comment narrows one technical point only).

F2 — high, human decision required — The v2 record is still less reviewable than the v1 evidence it supersedes

Where: docs/plans/issue-252/replay-validation.md ("完整原始计时、日志和 profile 仍在本地") and docs/plans/issue-252/replay-validation-summary.json at ac03ae23; acceptance item 3 ("retaining reproducible measurements").
Problem: Re-verified at the pinned head: the committed v2 summary contains identity hashes and per-input digests but no samples_ms, median or latency field at all, and the record states raw timings, logs and profiles stay local. The superseded v1 usefulness supplement published 720 samples and two profiles in-tree, so the current-head record is the least reviewable artifact in the set; the "40 focused tests" and "300 timing samples" figures remain author-reported. The edit states this limitation honestly, but stating it does not meet the acceptance criterion.
Conditions: Any acceptance decision that rests on the v2 record or on the word "validated".
Impact: The decision owner cannot re-derive the summary or inspect the timing path from repository bytes; green hosted checks establish CI only.
Fix: Publish per-call samples_ms (as usefulness-summary.json already does) or make tools/engineering_benchmark_validate.py regenerate the summary from committed inputs, and label every figure as v2 or historical v1.

Status: open (no change this round).

F6 — medium, human decision required — The state block still omits the recorded non-green local suite

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" ("Current-head hosted full-suite/install/reference checks passed") versus docs/plans/issue-252/validation.md:40-41 at ac03ae23.
Problem: The record still reads "3479 passed、51 skipped、56 failed … 全部在未修改的原始基线复现 … 全量结果仍为非绿色，不能称为所有检查通过". The edit keeps the v1 samples as "historical evidence" and preserves "both v1 usefulness performance FAILs", but its remaining-gaps sentence lists only same-code noise, performance ordering, SkillEval two-arm, agent/skill benefit and hardware — not the suite. The green hosted run cited (41672237031) is Linux CI, not the local Windows suite.
Conditions: An acceptance decision that reads the green hosted run as suite-clean for the delivery.
Impact: Acceptance item 5 explicitly forbids closing because software CI is green; removing the caveat from the governing record makes exactly that misreading easier.
Fix: Add one sentence to the state block: the local Windows suite remains non-green (56 failures, all reproduced on the unmodified baseline), and v2 has no recorded full-suite run beyond hosted Linux CI.

Status: open (the edit did not restore the caveat).

F3 — medium, human decision required — The status block still contradicts the unchanged ownership and next-step text

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — "Current state"/"Next action" versus the unchanged "Ownership and authorization" section (AGENTS.md:117 routing rule).
Problem: The new "Next action" tells the assignee to coordinate "full current-head PR review", while the body below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The earlier "superseding the earlier unassigned status below" wording is now gone entirely, so nothing marks those sentences as superseded even though an assignee and a delivered draft are recorded.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code"), or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record simultaneously answers "assignee coordinating delivery and full review" and "no implementer, implementation unauthorized, not yet scheduled" — the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite "Ownership and authorization" and "Next step" to the recorded state (accountable coordinator Bounty-hunter, acceptance owner Dong1017, decision pending, child implementers unassigned), mark or delete the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded) and its open half (subset and reuse choices agreed).

Status: open (worse this round: the superseding sentence was removed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 still has no native sub-issues. task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 (last updated 2026-11-03T16:16:09Z) still carries only kind:task and area:workbench. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress would roll up nowhere, and the body's own sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect shared tasks to RSI and verify their use #337 — which holds the supplemental usefulness results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared CPU benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at 9454672f, reports the compare exit-status fix and now publishes the 300 timing samples plus two profiles. Verdict: human decision required — the edit's checkable claims hold (PR head equals the pin, run 41739933972 is green on 3.10/3.11 with qualification skipped, benchmarks/ grader digests recompute to 0b75b5bb… at ac03ae23 and 34928e69… at head with only cli.py differing, the export holds 12 trials/300 samples whose per-case medians sum exactly to the recorded suite latencies, and the benchmark test module expands to exactly 43 cases), but no accept/revise/reject decision is recorded, so no development task should start (the Issue is also assigned). F1: open. F2: fixed — per-call samples and profiles are now in-tree and internally consistent. F6: fixed — the state block again names the non-green historical Windows suite. F5: remains fixed. F3: open. F4: open (sub_issues still empty; #337 still lacks status:proposed/priority:P2/bot-reviewed). No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records delivery and a scoped confirmation, not an accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:13-16.
Problem: The block was rewritten again (exit-status fix, published samples) but still contains no decision from the acceptance owner. Checked this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the bot review; PR feat: add shared CPU engineering benchmark tasks #338 has exactly three reviews — r4456416093 (2026-11-03T15:00Z, the P2 measurement finding), the author reply r4457228612, and r4459426716 (19:23Z) whose own text says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open"; and task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 still carry no acceptance comment. The body itself concedes the reported local delegation "is not recorded parent scope acceptance", and assignment identifies accountability, not authority (docs/agent-collaboration.md:14).
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merge, mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked acceptance item, so nothing can merge as accepted scope and a rejection discards the delivery; the decision owner is being asked to judge a pre-built design instead of deciding subset and interface first.
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists. Neither a scoped PR review nor an assignee substitutes for it.

F7 — medium, human decision required — The published 300-sample record no longer binds to the head, because the grader digest covers the benchmark CLI

Where: src/inferrsi/harness/benchmarks/contract.py:33-34 and worker.py identity check at 9454672f; docs/plans/issue-252/replay-validation.md:17; [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: grader_digest() is tree_digest() of the whole harness/benchmarks/ directory, so the exit-status fix — a 2-line change in cli.py that only adds the retained status to the command output — rotated the grader identity. Recomputed from both trees: ac03ae23 → 0b75b5bb…, 9454672f → 34928e69…, and cli.py is the only differing file. worker.py rejects any identity mismatch, so the just-published replay-measurements.json (all 12 trials carry grader_sha256: 0b75b5bb…) and the two profiles are evidence for the superseded identity only; the current head has no measurement evidence bound to its own identity. The record states this boundary honestly and says the compute/oracle/timing files are unchanged, so this is not a concealment — but acceptance item 3 then rests on the owner accepting that stated equivalence rather than on a head-bound record.
Conditions: Any acceptance decision that cites the v2 record as current-head evidence, and every future change to benchmark CLI/reporting code.
Impact: Measurement identity is coarser than measurement semantics: any reporting or plumbing touch invalidates all prior timing evidence and forces either a re-run or a fresh equivalence argument, which will keep recurring as the CLI evolves.
Fix: Decide and record one of: (a) narrow grader_digest() to the measurement-bearing modules so reporting changes do not rotate the identity, or (b) re-run the v2 validation at the head (tools/engineering_benchmark_validate.py, CPU-only, minutes) so the accepted scope has head-bound samples. Either way, label the 0b75b5bb samples as pre-fix evidence in the acceptance record.

F3 — medium — "Ownership and authorization" and the closing "Next step" still contradict the recorded state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — the unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The section still says "Implementer and initial subset remain unassigned" although an assignee is recorded and an implementation has been delivered, and its closing "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation" contradicts a "Next action" that tasks the assignee with coordinating current-head review of that delivery. The new sentence "the original tracker authorization below is unchanged" now points at those sentences rather than reconciling them, so a reader or agent routing per AGENTS.md:117 still gets two answers for "is there an implementer and may this work be scheduled".
Conditions: Any reader or agent using this Issue as the governing record for benchmark work.
Impact: The tracker cannot be cited as one consistent authorization state, which is the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and the closing step to the recorded state (accountable assignee/coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not yet agreed), and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

F4 — medium — The four slices are still not children of the tracker, and #337 is still outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 (last updated 2026-11-03T16:16:09Z) carries only kind:task and area:workbench. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress rolls up nowhere, and the body's own sentence that child Issues mix v1 and current-head links has no rollup surface to correct.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect shared tasks to RSI and verify their use #337 is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared CPU engineering-benchmark subset, task/oracle contract and delivery criteria; this edit re-pins draft #338 at fd0f396c, adds the engineering-benchmark-v1 correctness-artifact adapter and a 49-test software-control record. Verdict: human decision required — no accept/revise/reject decision exists and the body still withholds implementation authorization, so no development task should start (the Issue is also assigned). Verified this round: the pin equals the PR head, runs 41745549558 and 41746656869 match their claims (the later 3.10 check has since finished green), the record's 49 tests/0 skips and 3,631 passed/18 skipped per suite match, and the tested-head-to-head delta is three docs files; F1 open, F7 open (the published samples still carry the pre-fix grader identity), F3/F4 open, F8 is new, and F2/F5/F6 remain fixed. No label value is suggested: every axis already carries a value.

F1 — high, human decision required — The edit still records a delivered adapter and a green software-control run, but no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117 ("confirm the governing Issue's accepted scope before changing code"); docs/agent-collaboration.md:13-16 ("Work follows accepted scope and explicit or standing delegation. Record or link that authority").
Problem: The block was rewritten again (new adapter, 49-test record, new pins), but still contains no decision from the acceptance owner. Checked this round: the only events since the previous review are this body edit and the author's index comment edit at 00:25:49Z (status only); PR feat: add shared CPU engineering benchmark tasks #338 still has exactly three reviews — r4456416093, the author reply r4457228612, and r4459426716, whose own text says the recorded parent scope decision remains open; and task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 still carry no acceptance comment. The body itself concedes that contributor-reported delegation "is not recorded parent scope acceptance".
Conditions: Any move to keep feat: add shared CPU engineering benchmark tasks #338 moving toward merge, to mark acceptance on task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337, or to treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: All four children still carry the unchecked item "维护者接受该子项的范围及交付", so nothing can merge as accepted scope, and a rejection discards a delivery that keeps growing (the head is now nine commits).
Fix: Dong1017 posts one comment accepting, revising or rejecting the CPU subset plus the task/oracle interface and delivery criteria; keep feat: add shared CPU engineering benchmark tasks #338 draft until that exists. A scoped PR review, an assignee or a green CI run does not substitute for it.

Status: open (unchanged in substance; the new adapter is technical progress, not a scope decision).

F7 — medium, human decision required — The published timing samples still belong to the pre-exit-fix grader identity, and neither offered fix is chosen

Where: docs/plans/issue-252/replay-measurements.json and replay-validation.md:43 at fd0f396c; src/inferrsi/harness/benchmarks/contract.py:33-34 (grader_digest() = tree_digest() of the whole harness/benchmarks/ directory); [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 acceptance item 3 ("retaining reproducible measurements").
Problem: Re-verified at the current head: all 12 published trials still carry grader_sha256: 0b75b5bb… (the ac03ae23 identity), while recomputing tree_digest() over src/inferrsi/harness/benchmarks/ at fd0f396c yields 34928e69… — unchanged since 9454672f, because the new adapter lives in src/inferrsi/harness/skill_eval_benchmark.py, outside the digested tree. The new adapter now freezes grader_sha256 = 34928e69… in every task descriptor, so the published samples and profiles are evidence for neither the current head nor the adapter's own frozen identity. The edit labels them "historical and not rebound to this adapter/head", which addresses the labeling half of the earlier finding, but neither remedy (narrow the digest or re-run at the head) is decided.
Conditions: Any acceptance decision that cites the v2 record as current-head evidence, and every future change to benchmark CLI/reporting code.
Impact: Acceptance item 3 still rests on the owner accepting a stated equivalence argument rather than on a head-bound record, and each reporting/plumbing change rotates the identity and invalidates prior timing evidence again.
Fix: Record one decision: (a) narrow grader_digest() to the measurement-bearing modules so reporting changes do not rotate identity, or (b) re-run tools/engineering_benchmark_validate.py (CPU-only, minutes) at fd0f396c and publish head-bound samples. Keep the 0b75b5bb export labelled as pre-exit-fix evidence either way.

Status: open (acknowledged and labelled; no fix chosen).

F8 — medium, human decision required — The same state block says formal SkillEval compare integration is both delivered and incomplete

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Current state" (first paragraph: the adapter in skill_eval_compare/skill_eval_local "supersedes the earlier unimplemented formal-compare status"; later gap sentence: "formal SkillEval compare/improve integration … remain incomplete"); docs/plans/issue-252/README.md ("第二条现已增加 skill_eval_compare 的正式产物交接"); docs/plans/issue-252/04-integration.md (status: 正式 SkillEval 正确性产物交接已实现); docs/plans/issue-252/replay-validation.md:52 ("正式 SkillEval 两臂产物交接 … 仍未完成").
Problem: The edit adds a delivered compare/local adapter and explicitly says it supersedes the earlier unimplemented formal-compare status, then lists "formal SkillEval compare/improve integration" as still incomplete. The plan docs support the first reading (compare implemented, skill_eval_improve not), while replay-validation.md:52 still says the two-arm artifact handoff is outstanding. Only skill_eval_improve and real agent-produced two-arm experiments are actually absent (formal-integration.md says provider generation, skill_eval_improve and real Agent runs are not added).
Conditions: The acceptance owner reading the tracker to decide acceptance item 4 ("Demonstrate that the selected shared package can be consumed by its agreed Workbench/evaluation use cases…").
Impact: The decision owner cannot tell whether the SkillEval half of acceptance item 4 is delivered at the software-control level or still missing, which invites either accepting without knowing what exists or re-requesting work already done.
Fix: Replace the compound sentence with the recorded split — engineering-benchmark-v1 compare/local adapter implemented and covered by the 49-test control; real two-arm agent runs, provider generation, skill_eval_improve and reliable ranking still absent — and update the stale sentence at replay-validation.md:52.

Status: new this round.

F3 — medium — "Ownership and authorization" and the closing "Next step" still contradict the recorded state

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section ("Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation") versus the new "Current state"/"Next action" block, which says "the original tracker authorization below is unchanged" and tasks the assignee with coordinating the scope decision and full current-head review of a delivered implementation; acceptance item 1; AGENTS.md:117.
Problem: The new sentence now explicitly points at the contradictory paragraphs instead of reconciling them. An implementer and a delivered draft are recorded, yet the governing text still says no implementer exists and implementation is unauthorized and unscheduled.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117, or deciding whether benchmark work may proceed or be scheduled.
Impact: One governing record gives two answers to "is there an implementer and may this work be scheduled" — the ambiguity that allowed the pre-built delivery to arise.
Fix: Rewrite the section and closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not yet agreed), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half.

Status: open (unchanged; the new "unchanged" sentence makes the conflict explicit).

F4 — medium — The four slices are still not children of the tracker, and #337 remains outside the proposed-decision queue

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []), task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 labels, PR feat: add shared CPU engineering benchmark tasks #338 body.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: select initial optimization tasks and benchmark reuse #334, task: share task inputs and evaluation rules across RSI consumers #335 and task: ship executable Softmax and RMSNorm CPU tasks #336 carry kind:task, area:workbench, priority:P2, status:proposed, bot-reviewed, while task: connect shared tasks to RSI and verify their use #337 (updated 2026-11-04T00:25:47Z) carries only kind:task and area:workbench. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress rolls up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; the tracker cannot show which slice is proposed, accepted or complete, and task: connect shared tasks to RSI and verify their use #337 — which holds the supplemental results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues of [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and align task: connect shared tasks to RSI and verify their use #337's status/priority labels with its siblings; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Status: open (verified unchanged this round).

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

@Bounty-hunter

Intent: the tracker should record an agreed minimal shared engineering benchmark subset, task/oracle contract and delivery criteria, with Dong1017 owning the accept/revise/reject decision. Verdict: human decision required — this edit's checkable claims all hold (pin at fd0f396, run 41746656869 green with only draft KV qualification skipped, the retained JUnit artifact has 49 tests/0 skips and SHA-256 ac2d2164… as recorded, both full-suite job logs show 3,631 passed/18 skipped, and the dfcc1855→head delta is the three documentation/evidence files it names), but no decision is recorded and the Issue still withholds implementation authorization, so no development task should start. Earlier findings: F2, F5 and F6 fixed; F7 answered by explicitly labelling the 300 v2 samples as historical (head grader digest is 34928e69… versus 0b75b5bb… in those samples); F1, F3 and F4 remain open; I found no new factual defect in this edit.

F1 — high, human decision required — The edit adds verified delivery evidence but still no accept/revise/reject decision

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body "Decision needed" and acceptance items 1–2; AGENTS.md:117; docs/agent-collaboration.md:14-15; PR feat: add shared CPU engineering benchmark tasks #338 review r4459426716.
Problem: Checked this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no comment newer than the 2026-11-04T00:07Z bot review; PR feat: add shared CPU engineering benchmark tasks #338 has exactly three reviews, the newest being Dong1017's own scoped confirmation (r4459426716, 2026-11-03T19:23Z), which says "Full-PR review, v2 timing/noise qualification and the recorded parent scope decision remain open"; task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 carry no acceptance comment. The edit pins the real head and adds new, verified correctness evidence (49 tests/0 skips bound to dfcc1855, JUnit SHA-256 ac2d2164… matching the retained artifact, docs-only delta to head), but that is implementation progress, not the decision.
Conditions: Any move to merge or advance feat: add shared CPU engineering benchmark tasks #338, mark child acceptance, or treat [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 as an authorized implementation task.
Impact: Accepting now would accept a correctness-only delivery: at head grader_digest() is 34928e69… while the published 300 v2 samples carry 0b75b5bb… (the body now labels them historical), no real two-arm skill experiment or performance ranking exists, and the full current-head source review the "Next action" asks for is still outstanding. A rejection discards the delivery.
Fix: Dong1017 records one comment accepting, revising or rejecting the CPU subset, the task/oracle interface and the delivery criteria — including whether historical samples plus the stated code-identity equivalence satisfy "retaining reproducible measurements" — after the full fd0f396 review; keep feat: add shared CPU engineering benchmark tasks #338 draft until then.

F3 — medium, human decision required — "Ownership and authorization" and the closing "Next step" still contradict the status block

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 body — unchanged "Ownership and authorization" section and closing "Next step" versus the new "Current state"/"Next action"; AGENTS.md:117.
Problem: The block records an assignee coordinating the scope decision and full current-head review plus a delivered implementation, while the section below still says "Implementer and initial subset remain unassigned", "Current authorization is to track and clarify this work, not to launch implementation…", and "Next step: retain this tracker, then finish the current SkillEval and Indexer discussions before scheduling benchmark implementation". The new sentence "the original tracker authorization below is unchanged" now explicitly preserves that contradiction instead of marking it superseded.
Conditions: Any reader or agent routing on this Issue per AGENTS.md:117, or deciding whether benchmark work may proceed or be scheduled.
Impact: docs/agent-collaboration.md:14-15 requires recorded or linked authority; a record that answers both "assignee delivered and coordinating" and "no implementer, unauthorized, unscheduled" cannot serve as that authority, which is the ambiguity that let the pre-built delivery arise.
Fix: Rewrite the section and the closing step to the recorded state (accountable coordinator @Bounty-hunter, acceptance owner Dong1017, scope decision pending, initial subset and reuse choices not agreed), delete or mark the superseded sentences, and split acceptance item 1 into its recorded half (implementer recorded by assignment) and its open half (subset and reuse choices agreed).

F4 — medium, human decision required — The four slices are still not children of the tracker, and #334–#336 still point at superseded snapshots

Where: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 (GET /issues/252/sub_issues returns []); task: select initial optimization tasks and benchmark reuse #334–task: ship executable Softmax and RMSNorm CPU tasks #336 bodies (links to ac03ae23/d0ce9d7); task: connect shared tasks to RSI and verify their use #337 labels (only kind:task, area:workbench, last updated 2026-11-04T00:25:47Z); PR feat: add shared CPU engineering benchmark tasks #338.
Problem: Verified this round: [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 has no native sub-issues or dependency links. task: connect shared tasks to RSI and verify their use #337 was refreshed to fd0f396 but still lacks status:proposed/priority:P2/bot-reviewed, so it does not appear in the same decision queue as its siblings, while task: select initial optimization tasks and benchmark reuse #334–task: ship executable Softmax and RMSNorm CPU tasks #336 still link the older ac03ae23/d0ce9d7 revisions that the tracker body has replaced. PR feat: add shared CPU engineering benchmark tasks #338 uses non-closing references, so merge progress rolls up nowhere.
Conditions: Deciding whether to adopt the four-slice split and track its acceptance.
Impact: Five records (tracker, four children, one draft PR) must be reconciled by hand; a decision owner opening a slice sees a superseded revision, and task: connect shared tasks to RSI and verify their use #337 — which holds the supplemental results — is easy to miss when the decision is made.
Fix: The maintainer decides whether to adopt the split; if yes, add task: select initial optimization tasks and benchmark reuse #334–task: connect shared tasks to RSI and verify their use #337 as native sub-issues, align task: connect shared tasks to RSI and verify their use #337's status/priority labels and refresh task: select initial optimization tasks and benchmark reuse #334–task: ship executable Softmax and RMSNorm CPU tasks #336's revision links to the tracker's pin; if not, fold the slices back into [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252 and keep the draft PR as the proposal.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Navigation Menu

Repository navigation

[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252

Goal

Candidate coverage, to narrow before implementation

Minimal shared contract to discuss

Relationship to existing work

Tracking and acceptance

Ownership and authorization

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

1. Problem

2. Design principles

3. Objects

3.1 Search-space cell index

3.2 Candidate card

3.3 Admission gates

3.4 Cost ladder → `RunSpec`

3.5 Phase-locked round

3.6 Stop evaluator

3.7 Negative-result ledger

4. Relationship to existing work

5. Dependency and ordering

6. Delivery slices

7. Acceptance criteria for this RFC

8. What this RFC does not authorize

9. Open questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252

Description

Goal

Candidate coverage, to narrow before implementation

Minimal shared contract to discuss

Relationship to existing work

Tracking and acceptance

Ownership and authorization

Activity

amy-why-3459 commented on Oct 29, 2026

[RFC] Optimization search policy: candidate admission, cost ladder and stop contract

1. Problem

2. Design principles

3. Objects

3.1 Search-space cell index

3.2 Candidate card

3.3 Admission gates

3.4 Cost ladder → RunSpec

3.5 Phase-locked round

3.6 Stop evaluator

3.7 Negative-result ledger

4. Relationship to existing work

5. Dependency and ordering

6. Delivery slices

7. Acceptance criteria for this RFC

8. What this RFC does not authorize

9. Open questions

Dong1017 commented on Oct 29, 2026

Bounty-hunter commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 4, 2026

agent-ci-infra-bot commented on Nov 4, 2026

agent-ci-infra-bot commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3.4 Cost ladder → `RunSpec`