task: pre-registered campaign on experience and context compaction for agents on the #75 task (Qwen3-TTS 0.6B, RTX 4090), with exploratory results #220

Current state: Exploratory, non-preregistered runs reported no candidate PASS; the body was revised on 2026-10-30, and that day's review confirmed the corrections, while the offered audit packet remains unpublished and uninspected.
Decision needed: Whether to open the proposed H1/H2 acceptance campaign (task, design, resources), which follow-ups become Issues, and where the audit packet goes, from the maintainers (goal-acceptance owner TBD).
Next action: Accountable parent ZX271848 presents a frozen H1-or-H2 design and the audit packet to a named maintainer; no campaign or new run is authorized.

Refs #75, #93, #40, #128

Revised 2026-10-30 after the proposal review: the #40 and #128 references are corrected; H1 and H2 are restated with matched controls, frozen controller settings and whole-workflow budgets; the replication unit, the first-PCM definition and the status of the 40-sentence figure are stated; the evidence section lists the packet offered for audit. Measurements and verdicts are unchanged; the results summary now says that the runs do not test H1 as restated.

Goal and acceptance

• Problem / desired outcome: [RFC] Skill Eval v1: revision-scoped discovery and task-efficacy evaluation #40's bounded N/E/R pilot on report-audit tasks observed no incremental correctness benefit. [RFC] Skill Indexer: better skill and context selection through collect #128 compared three retrieval/context conditions on report-audit tasks (legacy collection, relevance-selected complete resources, and bounded context over the same selected owners): its frozen no-observed-quality-loss rule was not met, and the bounded-context condition reported 25.74% more tokens than relevance-selected complete resources, a descriptive figure, not a library on/off comparison. Proposal: controlled task-local harness improvement on top of Skill Eval #93 proposed controlled harness improvement with a library on/off step. Whether experience helps an agent on a hardware-graded optimization task, and how it should enter the agent's context, is still unmeasured in this repository. This issue proposes a pre-registered acceptance campaign for two hypotheses, motivated by exploratory runs reported below:
  • H1 (Harness plane; model, infra and RunSpec fixed): with the machine probe, the first attempt's notes and the selector/briefing step held fixed, giving the selector access to the library improves the executor's verified result, compared with the same selector without library access.
  • H2 (Harness plane; model, infra and RunSpec fixed): when the context must be compacted mid-task, a compressor that checks the transcript's beliefs against evidence and consults the library yields better verified results than the harness's native compaction, with the trigger, prefix, controller settings and post-compaction limits frozen and compressor work counted in the budget.
• Completion criteria: maintainers decide (a) whether a pre-registered acceptance campaign for H1/H2, as sketched under "Proposed campaign", is wanted, and with which changes; (b) which of the follow-ups below, if any, should become their own issues. No PR is part of this issue.

Scope and constraints

• In scope: the business goal of acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75 (≥30% C1 end-to-end RTF reduction on one RTX 4090 without quality loss), executed by autonomous agents in isolated workspaces under several experience and compaction conditions.
• Out of scope: this is not a reproduction of the acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75 protocol. Our quality gates are ASR WER, total duration and first-PCM, without acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75's aligned log-mel fidelity gate; we measure default serving, without the batch-invariant control (acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75 baseline RTF ≈ 0.35, ours ≈ 0.10). Under acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75's fidelity gate, the sampling-changing candidates below would likely fail. No lesson here is transferred to another repo × revision × backend × device × model × scenario.
• Status of the runs below: they were not registered as an acceptance campaign beforehand. Gates and validators were frozen before the runs (hashes under "Evidence"); the results are offered as exploratory research input, not as campaign results.
• Limits: one task, one device type, agents on DeepSeek V4.1 Flash (the model that led acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75). The independent replicate is an agent run from the pristine checkout. Study 1 has one such run per condition (three for A). Study 2 has one shared prefix with three branches per arm: within-prefix repeats, so one independent replicate per arm. AB/BA pairs repeat the measurement of one final workspace, not the agent run.

RunSpec and gates (frozen before any agent run)

repo:      vllm-project/vllm-omni
revision:  baseline 6b35bca6ffff3c6bee4723984704fa39c9ac2af0 (pristine checkout);
           candidates = each agent's final workspace diff on that revision (not committed)
backend:   CUDA, vLLM 0.32.0 (CUDA 12.9 wheel), torch 2.13.0; default (non batch-invariant) serving
device:    1 × RTX 4090 24 GB, driver 590.48
model:     Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice @ 85e237c12c027371202489a0ec509ded67b5e4b5 (model bytes unchanged)
scenario:  C1 HTTP streaming PCM (/v1/audio/speech, CustomVoice "vivian", English), 2 warmups per run;
           RTF = request wall time to last byte / audio duration;
           first-PCM = time from sending the request to completion of the client's first 8192-byte read
           (≈171 ms of 24 kHz int16 mono audio), not to the first byte

• Levels: campaign-defined L4 end-to-end (C1 streaming RTF, first-PCM) and L3 quality (Whisper-large-v3 WER with the Whisper English normalizer; total audio duration). No L0–L2 claims.
• Gates: RTF reduction ≥ 30%; mean WER rise ≤ 0.02 absolute; total audio within ±5%; median first-PCM at most 20% worse.
• Comparison: each final workspace vs the pristine baseline, 3 alternating pairs (AB, BA, AB) on held-out sentences agents never saw (12 in study 1, 40 in study 2); agents tuned only on a separate 20-sentence dev set.
• Isolation: agents ran in filesystem/PID namespaces without earlier runs, held-out data, graders or the real API key (held by a local proxy). Runs from an earlier, weaker isolation, in which one agent read an archived solution, are excluded and listed in the evidence.

Results

Verdicts follow the contract: a candidate is PASS only if every gate holds. No candidate is PASS.
On this RunSpec, the 80-step arms show no library effect on the outcome (all stop at the same ≈22% configuration ceiling; at 160 steps the full-library run D2x went further but failed the duration gate, and one run per arm cannot separate that from chance), and H2 is not distinguishable (outcomes were decided by whether the executor tested a candidate at all). The runs do not test H1 as stated above: no pair of arms differed only in library access with the selector step matched.

Study 1: experience injection (minimal tool loop, DeepSeek V4.1 Flash, effort max, 80 steps, one run each)

Observed under this RunSpec:

• All 80-step arms ended on the same opt-in deploy flags (code_predictor_kv_cache, code_predictor_fused_sampling, code_predictor_fused, optionally decode_time_major_conv, decode_cudnn_benchmark): a ≈22% ceiling that the no-library agents reached as well. Library, selection and notes did not change RTF.
• D2x was the only code change: a weight-only int8 GEMV Triton kernel for the 15-step code predictor (≈2.4 GB/frame of BF16 weight reads). WER was unchanged on held-out; total duration +5.4% is just outside the gate, while the baseline's own total duration varied by ≈2.6% between runs on 12 sentences.
• Quantization was named as the next lever by 6 of 7 executor arms, including a no-library one; one no-library agent declined it because runtime quantization "reads as modifying the model" (our task wording was ambiguous there). With one run per arm, a library effect cannot be separated from chance.
• C and Cp differ in more than library access (a processed briefing from a selector that also probed the machine, versus raw notes), so they are not a library on/off control.
• Second attempts with the first attempt's notes (C, Cp, Cpp) fixed first-audio latency (first codec chunk ≥ 3 frames, or a chunk ramp, so the client's first 8192-byte read completes earlier). This does not change RTF.

Study 2: context compaction (Claude Code 2.1.312 on DeepSeek V4.1 Flash, effort max)

A shared prefix ran until the executor's context first reached 100k tokens (Claude Code's smallest auto-compact window). From that snapshot, three branches per arm ran 60 executor turns each, compacted whenever the context reached 100k again:

• N: native /compact;
• C: a separate read-only compressor session with the same skills checks the beliefs that steer the next step against evidence and writes the new context;
• H: as C, and the compressor also re-selects ≤ 15 mounted library skills and writes session skills (machine facts, belief ledger).

• A/A reference: a baseline-identical workspace measured −0.1% (pairs −0.2 / −0.0 / −0.0). The other baseline-identical workspaces are recorded as identical rather than re-measured.
• Improved branches: WER −0.002, total duration +3.1%, first-PCM −19% to −21%.
• No difference between arms: each arm reached the flag ceiling in one of three branches.
• The limiting factor was whether the executor tested a candidate at all. In all three C branches the compressor ranked the flag experiment first or second; one executor ran it. Five of nine branches never ran a benchmark; their turns went mostly to self-written analysis scripts (33–56 per branch). A benchmark takes 5–13 minutes, longer than Claude Code's 2-minute foreground limit, so every benchmark became a background job.

A third exploratory study (the same three arms on DeepSeek Harness, with dsh's own compaction as the native arm; one prefix, one run per arm) has finished. Like the two above, it was not pre-registered; it will be reported separately with its own artifact identities.

Observations offered for Skill Eval / harness work (scoped to this RunSpec)

1. On this task a no-library agent already reaches the configuration ceiling, so outcome alone cannot show a library effect. A library on/off comparison needs a task where the ceiling is below the target and the remaining step is code (here, predictor weight traffic).
2. Recording whether and when an agent tested a candidate separates harness effects from knowledge effects. Here the agents needed 24 to 52 steps to reach the ceiling.
3. We observed three selection failure modes:
   • knowledge is organized by source engine and modality rather than by bottleneck, so the opt-in approximate quantization lane (filed under a CUDA diffusion model) was not selected, although weight traffic was the bottleneck;
   • a wrong model of the metric (first-audio wait as an RTF lever) carried into the ranking;
   • procedures whose tool is missing on the machine (nsys) were dropped as a whole.
4. On this task, 12 held-out sentences did not make the duration gate decisive (the baseline's own total duration varied by ≈2.6% between runs), so study 2 used 40. Forty is a task-specific planning figure, not a calibrated minimum; a campaign would size the held-out set from an A/A calibration and a frozen decision rule. With AB/BA pairs, RTF A/A noise was about ±0.2%.
5. Isolation has to be enforced by the runner. Archives, /tmp and leftover processes were all reachable by agents until the namespace runner, which matches the runner requirement stated in the daily-learning prompt.

Proposed campaign (to be opened with the Campaign acceptance form only if wanted)

• Campaign ID (suggested): qwen3tts-06b-agent-context / cuda×1.
• Replication unit: an independent agent run from the pristine checkout (for H2, an independent prefix), with the number per arm frozen beforehand. Branches from one shared prefix and AB/BA pairs are reported as within-prefix and measurement repeats, never as independent replicates.
• H1 matched control: every arm runs the same machine probe, the same first attempt (or its frozen notes) and the same selector/briefing step: selector model, prompt, step and token budget. Only the selector's library access differs (without the library, or with the frozen library revision). The executor prompt, budget and tools are identical. Raw-notes or full-library arms, as in study 1, change more than library access and would be exploratory only.
• H2 frozen settings: harness and version; compaction trigger in tokens; the prefix procedure (run until the first trigger, snapshot, branch) and task text; native compaction settings (e.g. the retained tail); the compressor's step budget, its completion step for incomplete handovers and what the handover contains; executor steps after the fork and the wrap-up step; the foreground limit for benchmark commands.
• Whole-workflow budget: the matched and enforced budget is executor steps. Selector and compressor calls, tokens, wall clock and device time are counted per arm and reported beside the executor's, so no arm's result rests on unreported compressor work.
• Frozen before any run: the items above; H1/H2, one per iteration; the RunSpec; gates; held-out list (hash only), sized from an A/A calibration, and validators; agent model, effort and harness version; the follow-through measures (whether and at which step a candidate was first benchmarked).
• Design changes suggested by the exploratory runs: pick a task whose configuration ceiling is below the target, so a knowledge effect can show in the outcome; run several independent repetitions per arm rather than one; let benchmarks block in the foreground so that harness backgrounding does not decide the result; keep every arm in the same namespace isolation.
• Initial verdict: INCONCLUSIVE until evidence is complete.

Ownership and authorization

• Proposed by / actual requester: @ZX271848
• Status: Proposed
• Accountable parent: @ZX271848
• Acting contributors:
  • Claude Opus 5.5 (claude-opus-5-5, Claude Code; effort not recorded): experiment hosting, analysis, this text;
  • GPT-6-Astra (gpt-6-astra, effort xhigh): study-2 controller and sandbox code;
  • agents under test: DeepSeek V4.1 Flash (deepseek-flash), reasoning effort max.
• Goal-acceptance owner: TBD (maintainers)
• Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

• Relevant sources, revisions or linked issues: acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75, Proposal: controlled task-local harness improvement on top of Skill Eval #93, [RFC] Skill Eval v1: revision-scoped discovery and task-efficacy evaluation #40, [RFC] Skill Indexer: better skill and context selection through collect #128; InferRSI library at 97f77a9 (81 skills, 108 records); revisions in the RunSpec above.
• Evidence: held locally by the requester and not uploaded; no local path is presented as a public artifact. For independent assessment we offer a minimal redacted packet per run and arm: run and arm identity (arm, prefix, branch, harness and model versions); the exact candidate diff, including untracked files, with its SHA-256; the frozen task prompt, library revision, measurement and validator scripts and held-out list; per-pair measurements (baseline and candidate RTF, WER, total duration and first-PCM); and every gate result. It can go to a named maintainer for private audit, or be published as evidence/<campaign-id>/{plan,results,patch,validator}, as maintainers prefer. Artifact identities (SHA-256):
  • study-1 held-out list b885978f1775ca8765744565fde8d1582f47675051007dc5c5528371885f4fc5, grader 92654bbbcd686707bb9fc4fbaf5897e6ddc6e1dc9cca46e7dd1cfe305c84736d, measurement 8151c342bd06b62ef684742beba02e69f289225f6fa4ce52a3d956210074143c (run_bench), ceda0eb245286e09b562c7aba0f372231d9f6d383950a2952540d88a260057cd (client), 89f11f7cb3b40e3606f3f22e7b910f719fff52c02703250bd260362546192fd4 (WER);
  • study-1 grades: A 59ed5591…, C f96823a2…, D ce7a2732…, Cp 1c0f4a1b…, Cpp 416bef1d…, A2x 2101cbb2…, D2x 0e3414e2…;
  • study-2 held-out list 4dc95e1eadeeb3e37d685ebe0ae01d64c8837a06c04076d5498eda831bf5c14f, input manifest 8d2524636543066880009a6438f0c6cb244108d514387155b709b1d30bec0aab, grading summary e3701ec97e1a5739e63446cdf14c7884e1ab9bfc56473b913938fb6cec8b1ac4.
• Possible follow-ups, each only if wanted, each its own issue:
  • a hardware case record for Qwen3-TTS 0.6B on Ada (flag ceiling, first-audio chunking, predictor weight-traffic bound);
  • the int8 predictor GEMV as a candidate upstream change, after acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75-style fidelity checks;
  • a context-compaction comparison harness for agent runs.

Completion and closure

• Who accepts completion and may close: the goal-acceptance owner above.
• Remaining uncertainty / next decision:
  • whether to open the proposed acceptance campaign, and with which task and changes;
  • whether the evidence packet goes to private audit or public hosting.