[RFC] Add support for LoopCD and validate its correctness #85

Motivation.

LoopCD uses an earlier recurrent depth as a reference for the final readout. The research question is whether this improves the quality–cost tradeoff of looped inference: better quality at the same depth, or acceptable quality with less recurrent computation. Engine integration, fewer head projections and a passing correctness test do not by themselves answer that question.

This RFC covers the synchronous Ouro/Huginn prototype, the evidence collected so far, and a small contributor roadmap. It follows the scope/status/claim structure used in #69 and #43. It complements their model/feature roadmaps without assigning their existing work again.

Paper and architectural idea

This RFC follows Decoding Looped Transformers Better for (Almost) Free — arXiv:2610.02185v1 (method and architecture, Sections 2–3 / Appendix A.2). A recurrent pass already produces earlier and final states for the same prefix; LoopCD reuses that pair at readout. It needs no auxiliary model or extra recurrent iterations. The paper's quality/FLOP results motivate evaluation here; they are not measurements of this runtime.

1. Shared recurrence, two readout choices

One token's prefix
      |
      v
Model input / initial state
      |
      v
Shared recurrent block: h_0 -> ... -> h_b -> ... -> h_R
                                      |              |
                                      |              `-- final state
                                      `-- owned reference copy
                                                 |
                 +-------------------------------+----------------------+
                 |                                                      |
                 v                                                      v
LoopCD-Logits                                              LoopCD-Hidden
h_b -> output layers -> z_b                                h_b ----+
h_R -> output layers -> z_R                                h_R ----+--> h' = h_R + w*(h_R-h_b)
                 |                                                      |
                 v                                                      v
z' = z_R + w*(z_R-z_b)                                     coda -> norm / vocabulary head -> z'
                 |                                                      |
                 `------------------> sample next token <---------------'

b < R is the chosen reference depth and w is guidance strength; w=0 recovers the unguided readout. The paper introduces the first recurrent state as its reference and also studies later references. Our protocols freeze b separately for prefill and decode. Guidance changes the readout, while the original recurrent-state update remains intact.

2. Where the current Ouro/Huginn prototype inserts guidance

Ouro
  token embeddings
      |
      v
  shared decoder stack repeated R times (original normalized feedback)
      +-- normalized reference state at b -- vocabulary head --> z_b --+
      `-- normalized final state at R ----- vocabulary head --> z_R --+
                                                                       |
                                                       logit contrast -> sampler

Huginn
  tokens -> prelude -> fixed input injection ---------------------+
                         |                                       |
                         v                                       v
                    initial state -> shared recurrent core x R <-+
                                         +-- pre-coda h_b --+
                                         `-- pre-coda h_R --+--> hidden contrast
                                                                      |
                                                                      v
                                                        coda attention / KV
                                                                      |
                                                        final norm / LM head
                                                                      |
                                                                   sampler

Ouro uses two_head as the reference path; its optional linear_fused path combines the already-normalized head inputs and needs separate floating-point qualification. Huginn combines only the state portion before coda, preserving the fixed injection. Full-prompt coda KV and the first-token readout remain required. The separate KV-only candidate skips unused prompt norm/head outputs, retaining coda/KV work; its measured effect below was mixed and near zero.

3. Runtime ownership and the boundary tests this motivates

Request admission (fixed prefill P / decode D, explicit KV budget)
      |
      v
Synchronous scheduler -> recurrent runner -> original core / CUDA Graph path
      |                         |
      |                         +-- capture reference at configured b
      |                         |     owner = request object + token position
      |                         |             + output index + loop depth
      |                         |
      |                         `-- validate owner -> guided readout -> sample
      |                                                   |
      |                                                   v
      |                                             next token / finish
      |
      `-- finish or abort -> release reference + KV -> safe request-ID reuse

This is the bounded synchronous prototype: async scheduling, prefix reuse, preemption, P/D disaggregation and speculation remain rejected for nonzero guidance. These diagrams explain the current dataflow and why Tasks 1–4 focus on official-checkpoint boundaries and quality–cost evidence.

Proposed Change.

Scope and semantics

• Ouro: combine normalized-state readout logits from the final and reference recurrent depths. Keep two_head as the reference implementation; qualify linear_fused separately for floating-point effects.
• Huginn: extrapolate the state before coda as h_final + omega * (h_final - h_reference). Preserve original recurrent feedback and input injection, construct full-prompt coda KV, and retain the first-token readout.
• Keep guidance and the optional Huginn loopcd_context_kv_only optimization off by default. KV-only removes unused prompt norm/vocabulary projection while preserving coda/KV work; it is a separate implementation comparison from guidance-off versus guidance-on quality.
• Own reference state by request identity, token position and loop depth. Release it on completion or cancellation and reject stale references after ID reuse.
• Nonzero guidance currently rejects asynchronous scheduling, prefix reuse, preemption, P/D and speculation. Standalone support for one of those features does not establish compatibility with LoopCD.

Implementation and validation status

Task 0 is now available as draft PR #86 — Add initial LoopCD support for Ouro and correctness checks. This public Ouro-only increment is based directly on upstream main; it does not depend on Huginn PR #82.

• Current contributor base: 0z5a:feature/loopcd-ouro-task0, current head 48932b2360f6ee750c154b268dae71fef8701782, targeting upstream main.
• This revision keeps the bounded Ouro guidance/lifecycle contract, projects only real request rows, reuses the coda graph path and checks prefill/decode depth admission. The experimental benchmark/recipe/result files have been removed from the PR diff. Current-head CUDA coda parity and performance remain pending; preparation and CPU imports are not GPU qualification.
• The original contributor pin d700ecc, parent ecb1f8b and original readout/ownership contract remain historical provenance. Later public-loader, FP32 continuation and cost evidence is recorded at the archived result commit; it does not qualify new behavior at 48932b2. Follow-up branches should reconcile their historical base with the current Task0 head before claiming combined coda/feature behavior.
• PR Add initial LoopCD support for Ouro and correctness checks #86 remains draft and unmerged. The retained independent full-model BF16 mixed-depth failure, fused-head rejection and quality non-inferiority uncertainty remain unchanged. Historical sections below retain their original source pins and execution boundaries.

Current-head native GPU regressions — 2026-10-13

At the captured Task0/PR #86 head 48932b2360f6ee750c154b268dae71fef8701782, the complete original114-case native GPU suite returned 1:74 passed,33 failed,7 skipped. Another node checked every original archive payload, the140-file source manifest, JUnit and all114 actual outcomes. The complete original and CPU closure archives also passed a full compressed-SHA/member/fsync backup check. This is source-pinned native regression data, not an official-checkpoint, full-model quality or performance result.

The33 failures remain: 28 require missing official FlashAttention packages (22 FA3,6 FA4), 3 PD tests request device1 while the execution exposed one GPU, and 2 tiny Coda request-reuse tests fail the original fallbacks > 0 counter assertion. The two_head/linear_fused cases captured once, replayed36/18 times and reported0 fallbacks; the subsequent original eager/graph token-depth equality assertion was not executed. No assertion or tolerance was weakened. Dependencies/device visibility and the Coda counter failure need separate prepared follow-ups; current-head coda parity and performance remain unqualified. See complete114-case outcomes and archived independent checks.

Audited official-weight native Task0 evidence

Exact public head d700ecca664da2e8f3122f66ad9d45de4295e45a and all128 source files;
ByteDance/Ouro-1.4B@574fa66cb8bf5abdc979642d01cf2b79b16bfab1, all9 physical
input files/full hashes verified. RTX5090 32GB, Torch2.12.1+cu130/CUDA13.0,
Triton3.7.1, BF16/TF32 off, native synchronous eager execution and one CPU thread.
Existing runtime was read-only; no environment installation or source changes.

Independent FP32 oracle passes: 14 first-readout cases and18 teacher-forced
continuation tokens. Maximum state/logits/selected-log-probability absolute errors:
1.52587890625e-5 / 4.2438507080078125e-5 / 2.47955322265625e-5, exact argmax,
frozen atol/rtol 1e-4. Dense full-prefix continuation equivalence covers P=D;
mixed P/D smoke does not establish identical dense full-prefix context semantics.
40 BF16 short smoke trials ×2 requests ×3 tokens pass repeated token/depth/ID/
finish parity and full KV reclamation.

BF16 linear_fused is rejected by the reserved four-example development gate:
maximum vocabulary error 0.1875, selected-log-probability error
0.09750795364379883 exceeds the frozen 0.03 threshold, with1 top1 flip.
two_head remains the default; no threshold relaxation or favorable-value rerun.

Execution boundary: this bare node used a disclosed, independently CPU-qualified
private checkpoint transport plus frozen canonical token IDs and a qualified
ByteLevel decode-only transport. Native model/parameter mapping and mathematics
are unchanged. Official execution through the public safetensors/tokenizer
entry remains NOT_RUN. This evidence qualifies the bounded native mathematical
path; it does not close every Task0 loading/runtime qualification or any Task1 feature.

The quality pilot reuses the existing ARC-Challenge32/HumanEvalPlus16 cohort, not
a new independent held-out set. Greedy natural stopping, reference depth1,
MC strength0.5/generation strength0.3, max256 outputs, seed1729+index and2GiB KV
per engine. All candidate-token log-probabilities are scored. Every frozen1pp
noninferiority result remains UNCERTAIN; executable EvalPlus pass@k is
NOT_RUN under the no-child-termination constraint. Generation exports are not scores.

Two fresh cost starts per depth, with reversed depth order on start1. B1/B4,
five warmups plus five alternating pairs per arm/start, output16/ignoreEOS,
same2GiB KV per engine. First output uses P, then15 use D. Within each arm,
repeat outputs/depths/IDs/finishes and KV release match. Different guidance arms
can emit different tokens. Load/hash/warmup/scoring/HTTP are outside timings;
cross-depth comparisons use separate processes and do not imply equal quality.
Ratios are off elapsed / guided elapsed, above1 favors guidance. Rates are
means of trial rates; all10 measured pairs, ranges and elapsed CV are retained.

All eight median ratios are below1 (−0.313% to−1.190% ratio changes). P2D2/B1 has
a positive mean outlier while its median is negative. No consistent speedup
or default-on performance justification follows. The full independent audit
checks384 quality sample files/6,776 candidate-token LPs and320 cost trials
(160 warmups/160 measured;12,800 output tokens including warmup). Timed CUDA Graph
captures/replays/fallbacks are0 because this is eager Task0, not Graph qualification.

The broader exploratory Ouro/Huginn prototype remains on the local research/loopcd-thor-20261003 branch based on Huginn Graph PR #82. Its tested implementation 74547e4e2bddf636278bc5d764a9d0523242f50d, full-depth quality-report commit e89def411c20c8d2da920e33f6f1b31952646461 and reduced-depth report 0f51735 are evidence provenance. The public Task 0 PR excludes that separate Huginn implementation, concurrency candidate and exploratory reports; those results below do not qualify official Ouro.

Audited native fixed-depth Graph slice

A separate, frozen official-weight run on the same public head now qualifies the
BF16 synchronous last_exited combination: P4D4/P4D3/P4D2/P2D2 × B1/B2 ×
eager/original RecurrentGraphs × off/two_head strength0.3/reference1 × two
unequal-length token-ID fixtures with request-ID reuse. All nine retained official
inputs and all128 source files matched their full hashes. One fresh native child;
same read-only Torch2.12.1+cu130/Triton3.7.1 runtime, TF32 off. No production
source, model/default, environment or frozen gate changed.

All64 trials /96 requests /288 output tokens pass independent raw audit.
The32 eager/Graph comparisons have exactly equal state, reference and complete
vocabulary logits (torch.equal, maximum absolute error0), matching outputs,
depths, request IDs, finishes, engine steps and core work. All144 guided reference
readouts pass ownership checks; first-readout state is unchanged by guidance,
reused IDs are safe, and final KV/references are empty. Cold graph counters are
12 captures /192 replays /0 fallbacks; guided trials make0 new captures.
Maximum allocated/reserved storage:2,962,911,232 /3,168,796,672 bytes.

This uses the disclosed unchanged private checkpoint transport and synthetic
IDs; the public safetensors/Transformers entry still remains NOT_RUN. It covers
this fixed-depth native combination, not Graph cancellation/cache fallback,
adaptive exit, async, prefix reuse, preemption or a full Task1 closeout. Diagnostic
CPU tensor copies and synchronization make these times unsuitable for speedup
claims. No new quality, serving, training, high-concurrency or long-context result;
Task4 stays last after the claimed combinations qualify. The existing fused
rejection and two_head default remain unchanged.

Native child, controller, SSH parent and independent CPU audit naturally exited0;
no GPU rerun or tolerance relaxation. Full235-file/34,882,374-byte weight-free
archive SHA256 6b8d93a6622656f20eb0b134ba214258e1dbbd4f101ad065a0e5c56993f3eec4,
all234 manifest entries/all128 tested source files verified. Independent audit
SHA256 ae2f59109b0cfa4648974013058cb0a01b4689584ec1bf11a23ffc26eaae2dcf.
The finite resource window was explicitly returned after full reader/device/original
lock checks; own model inputs retain future consumers. No physical isolation or
capacity-envelope claim.

Task roadmap and existing PRs — status checked 2026-10-13

Task0 is the initial Ouro adaptation owned by @0z5a. Existing follow-ups and engine prerequisites are linked here so their work is not assigned again. Each PR's reported validation applies to its own source and support scope; an open PR is not accepted integration, and a merged general engine fix is not a LoopCD compatibility pass.

Task 0 — Initial Ouro adaptation (@0z5a)

• Implement the same-prefix early/final readout contract with fixed reference depth and strength; w=0 preserves the baseline. Do not change recurrent feedback while adding readout guidance.
• Check owned references, actual prefill/decode depths, first-token and continuation outputs, abort/completion and ID reuse. Unsupported feature combinations must fail before allocating or starting work.
• Deliver a minimal integration and regression suite. Verify official-checkpoint logits/likelihoods against a dense/eager oracle before claiming full official Ouro correctness. Qualify BF16 linear_fused separately, including low-margin top-1 flips; algebraic equivalence alone is insufficient.
• Draft PR #86 provides the public Ouro-only integration. The current 48932b2 coda path requires its own GPU qualification; earlier native/public-entry/FP32 continuation results remain attached to their tested historical commits. Huginn evidence and the retained full-model BF16 failure remain separate.

Task 1 — Existing feature integration

Existing preemption slice: @prettygirlisnotme has opened PR #101 for the single-worker synchronous eager work accepted here, tracked by #98. The PR reports repeated suspend/resume, blocked allocation, cancellation/ID reuse, official FP32 continuation and complete native cost evidence. Its scheduler prerequisite #113 is merged. Preserve those source pins and negative outcomes, reconcile #101 with current Task0 48932b2, and qualify preemption with the new coda path before claiming combined GPU behavior. The bounded historical B1/B2 Graph evidence above remains historical; other Graph boundaries and independent feature slices remain open.

Why this is actionable: the repository already has a CUDA Graph path, and tiny guided eager/Graph tests pass; official Huginn short Graph runs are also available. That establishes a starting point, not complete feature composability. The current admission guard explicitly rejects nonzero guidance with async scheduling, prefix reuse and preemption. Their support requires changes to the guidance-state contract, not simply removing the guard.

• First slice: validate official Ouro eager versus Graph under identical guidance, covering warm replay, changing batch shapes, refill/drain, request reuse and graph-cache fallback. Check state/KV/logits/selected log-probabilities and token/depth/finish behavior with predeclared dtype tolerances. Reuse existing tiny tests; do not repeat them as new evidence.
• Further slices: take one existing feature. For async, preserve reference lifetime and stream ordering; for prefix reuse, define what guided coda/reference state and policy identity are cached; for preemption, define what is saved/restored or recomputed. Completion/cancellation must release owned state without affecting surviving requests.
• Done means a focused compatibility patch, an eager/dense comparison, boundary regressions and a documented support matrix. If claiming performance, include two fresh starts and matched paired timing with capture, fallback and memory costs. No speedup is required to accept a correct integration.

Task 2 — More model support

Why this is actionable: the current prototype has different Ouro normalized-logit and Huginn pre-coda Hidden paths, while its type checks exclude other recurrent backends. The existing Nanbeige backend provides a concrete candidate to investigate; this is not evidence that Ouro's formula can be transplanted unchanged.

• Claim a named model and checkpoint. For Huginn, reuse the preliminary oracle/quality evidence below and address a specific integration or coverage gap. For Nanbeige or another model, first document its recurrent update, normalization, injection/coda and the same-prefix reference boundary. If no sound mapping exists, report that result before implementing a misleading generic switch.
• Preserve the model's original recurrence/RNG and define model-specific guidance semantics. Compare the integrated path with an independent eager/dense oracle, then a small frozen generation/MC cohort and matched cost. Keep new architecture support separate from checkpoint-only qualification.
• Done means the adapter/configuration/guards, model-specific regression tests, official-weight correctness evidence and an honest quality–cost report. Use a disjoint development/final split, a finite sample/GPU budget and frozen evaluator; retain regressions and uncertainty. The 32-question Huginn pilot below is not sufficient to establish a 1 pp bound.

Task 3 — P/D separation experiments

Why this is actionable: the existing pd/engine.py explicitly rejects LoopCD because reference transfer is not implemented. Huginn also requires full-prompt coda KV. Worker handoff needs a defined guidance-state contract before this feature can be enabled.

• Start with one model and a finite two-worker experiment. Define which KV, recurrent/injection state, reference state (if live), token position, depth/guidance settings and RNG metadata must transfer or be reconstructed at the chosen boundary.
• Compare with a monolithic guided reference under identical prompts/policy/initialization. Cover the first emitted token, the first decode after handoff, cancellation, partial transfer failure and reused IDs. Keep rejection for unqualified combinations.
• Deliver the experimental integration, reproducible correctness cases and native TTFT/TPOT, transfer bytes/time and peak-memory results including all staging overhead. A neutral or slower result is valid; P/D is not assumed to improve small-batch latency. This task's small integration checks precede the final scale validation below.

Task 4 — Final high-concurrency × long-context closeout

Order: this is the final validation stage, after the adaptations and feature/model/P-D combinations claimed in Tasks 0–3 have passed their focused correctness gates. Freeze the supported matrix at that point; explicitly excluded combinations remain excluded. It is not a request to start broad stress sweeps before those integrations are ready.

Why this is needed: existing high-concurrency measurements use short prompts, while the reference/KV lifetime and full-prompt coda work grow with context. A short C128 result does not validate long prompts at the same concurrency, and isolated long-context or single-request success does not validate their combination.

• Freeze a finite cross-product of scheduler batch B, outstanding requests C and input/output lengths, within each checkpoint's supported context and actual memory budget. Include chunk/block boundaries, unequal prompt lengths, refill/drain, partial final batches and EOS/cancellation/ID reuse. Cover monolithic and qualified P/D paths separately. Reuse completed short-context points as historical anchors; run new combined cases to answer the missing question.
• Use matched policy, input order, initialization and explicit KV budgets per comparison. Verify surviving request outputs/log-probabilities, actual depths, reference/KV ownership and reclamation. Freeze dtype tolerances and report low-margin flips. Do not assume Huginn is invariant to arbitrary batch/RNG reordering or silently extend positional limits.
• Run two fresh starts and paired measurements at the declared valid points, retaining all warmups, captures, fallbacks, incomplete attempts and capacity stops. Report native TPS/RPS, TTFT/TPOT/ITL and request-tail latency, memory/reference/KV high water, transfer costs where applicable, and quality regressions on a frozen cohort. HTTP SLO claims require a separate real serving measurement.
• Closeout artifact: a supported/unsupported model-feature-workload table plus raw evidence and the measured capacity/latency/quality envelope. Show plateaus, regressions and unrun capacity stops; do not crop the curve to successful points or equate a planning bound with an observed OOM. State which combinations qualify for release and which remain experimental. No favorable speedup is required for completion.

Claiming and acceptance

Comment with Task 1, Task 2, Task 3 or the final Task 4, the exact feature/model/experiment slice, available hardware, finite validation plan and expected artifact. Task 0 is already taken by @0z5a. Multiple contributors may take non-overlapping slices; update the tracker as ownership is agreed. Reuse existing source/evidence and avoid rerunning completed pilot points merely to obtain favorable values.

The roadmap is grounded in the inspected model type checks, feature admission guards, guided Graph tests, reference/KV lifecycle and explicit P/D rejection. It identifies implementation work and bounded experiments; it does not pre-judge quality, speed or compatibility outcomes.

Existing exploratory evidence (separate from task completion)

• Twelve actual CUDA guided/refill/replay cases passed. Existing CPU tests also cover abort/ID reuse, stale-reference rejection and unsupported admission; these are not new work to repeat for Task 1.
• Official Huginn checkpoint tomg-group-umd/huginn-0125@bb6621b65e90b6a4b9b29ef88dc83866d450470c, all 12 files verified. Independent FP32 guidance oracle: maximum state error 2.7418136596679688e-6, logits error 6.198883056640625e-6, exact argmax under the frozen 1e-4 tolerance. Twenty short BF16 smoke runs had exact token/depth/request-ID/finish parity.
• The first private smoke failed because 64 KV blocks were insufficient; it passed after a matched 256-block correction. The formal performance budget was unchanged and the failure is retained.
• Two fresh native performance starts completed eight B/C cases and all 320 trials, including 160 warmups and 160 measurements. Exact output/depth/ID/finish parity, zero timed CUDA captures and full KV release were independently checked.
• The KV-only optimization removed 51,200 prompt vocabulary-head rows at B128/C128 but left recurrent core work unchanged. Paired median performance changes ranged from −0.566% to +0.858%, with regressions at some points. This is not a consistent throughput improvement and does not justify default-on behavior.
• Official Huginn P32/D32 ARC-Challenge pilot: 9/32 → 10/32 correct (28.125% → 31.25%); one gain, no losses. The conservative paired 95% interval is [−13.1134, +18.5154] percentage points, so the frozen 1 pp noninferiority claim is uncertain.
• HumanEval+ generation completed 16 examples per arm. Executable EvalPlus pass@k is not run; JSONL generation is not a code-quality score. The current execution environment prohibits evaluator child-process termination.
• The same-cohort reduced-depth pilot is now audited: P32/D16 ARC 9/32 → 9/32 and P16/D16 9/32 → 10/32, both with uncertain 1 pp noninferiority. HumanEval+ exports completed 16 examples per arm at both depth settings; executable scoring remains unrun. Matched Huginn fixed-work cost, larger independent cohorts, long-context and guided training remain separate work; the bounded official Ouro pilot is recorded separately above. Archive SHA256 742371ec04644d104cf891222780d1d85c7a47615255bd07808ffb0ed8b20797; independent audit c03b5d52b15836a7d7084b8ad108fc676e5ee2cb26b5c393075203877e13c130.

Feedback Period.

Keep this RFC open for scope and acceptance feedback and contributor claims. Task 0 is the initial Ouro adaptation taken by @0z5a; update follow-up status only for the claimed, verified scope.

CC List.

No additional assignees are preselected. Existing model/feature ownership is tracked in #69; contributors can claim the bounded slices above.

Anything else.

The short native measurements use a shared RTX 5090 host; other GPUs had unknown busy utilization. They do not establish HTTP throughput, task accuracy, training convergence or physical isolation. Mean per-trial p95/p99 is not a pooled request percentile. Full failed/partial attempts, warmups and fallback costs are retained.

Performance evidence: 170-file weight-free archive SHA256 33d3e0734398845f07745019f4f68232aa719cef8eea025e5462419caac3abf4; independent audit 04c324e5e94f9be28eca3d1d40b879b2710c15406cc7110e13df216abd6fe717. Quality evidence: 257-file archive 3221a774b84a10452cf98dabf653e085fe6e863d88eb2f0f1c97d67ab007d1e9; independent audit d0bad93794266638ef8d06c63ea2cbfc881a0a5a7208687f3856a0cfdbe3e85f. Both include the 139 tracked tested source files. The quality supplement records datasets, tokenization, seeds, peaks, raw-result checks and limitations.

Related: module/state ownership #32, speculation scope #43, model/feature roadmap #69. The consumer audit in vLLM #56441 inspired the KV-only investigation; its speedups do not transfer to LoopCD.

Before submitting a new issue...

• Reviewed related architecture, model/feature and speculation issues; this updates the existing LoopCD RFC rather than creating duplicate task issues.