Repository navigation
Replace byte-period detection with token-based per-chunk detection #66
Description
Activity
- added a parent issue
on Sep 1, 2026 The #65 research tuned the detection defaults — fold these into this task:
limit= 3 (not 2)maxUnit= 32 chars (not 256)minRunBytes= 96- Run the suffix scan on a rolling 4096-char tail every 128 chars (Z-array exact-cycle, oh-my-pi trick)
Full heuristics + skip list live on Add obvious-code and tool-call skip heuristic #71.
Scope update from #73: max pattern length is config-driven, not hardcoded. Read the per-request effective value (model override → global
STREAM_REPETITION_MAX_PATTERN→ 8) and pass it into the detector alongside the limit. Everything else in this task is unchanged.Repo-conformant notes (from review):
- Mirror
internal/streaming/slowdown_stream.go:io.ReadCloserwrap,closeOnceon the source,ctx-aware cancellation. ReuseparseDataLine,nextEventBoundary,donePayload,lfEventBoundaryfromobserved_sse_stream.go(already exported in package). - Replace
bytes.Bufferoutput queue withstreamBuffer(fromstream_buffer.go) — already pooled, non-concurrent FIFO, the right shape. Minimal diff. - Drop the
queuedEvent/rewriteEventAtCut/releaseSafeEventsmachinery entirely — eager per-chunk means no holdback. - Detector: rolling token tail of last
limit * maxPatterntokens.limit=3,maxPattern=8defaults (from Legitimate-repetition heuristics: what counts as 'obvious code'? #65 research); both come from the per-request config (r.repetitionLimit,r.repetitionMaxPattern). - Skip heuristics come from Add obvious-code and tool-call skip heuristic #71 — apply them here.
Test style:
internal/streaming/observed_sse_stream_test.gouses table-driventesting(no testify) with arecordingReadCloserandstrings.NewReader. Use the same shape —TestRepetitionGuardStream_DisabledPassthrough,TestRepetitionGuardStream_TokenStutter,TestRepetitionGuardStream_ChainLoop,TestRepetitionGuardStream_CleanStream,TestDetectRuntable-driven. Brown-test-style producer moves to #72.Minimal-diff goal:
repetition_guard_stream.goshrinks vs the prototype;slowdown_stream.gountouched;observed_sse_stream.gountouched;stream_buffer.goreused as-is.This runs on 400+ machines in production — every code path (active, off, fallback, skip) covered by a unit test before merge.
- Mirror
Working on this. Order:
TokenCounterinterface + lazy loader first (#67 deliverable, ininternal/streaming/tokenizer.go), then the guard refactor to consume it (#66). Byte-period stays as the fallback implementation behind the same interface per #64. Committing incrementally onfeat/stream-repetition-canceller; this ticket closes when the guard is token-based and tests cover all paths.Resolved on feat/stream-repetition-canceller (commit 9707690).
What changed:
- internal/streaming/repetition_guard_stream.go rewritten: per-chunk token detection via a rolling token tail (detectTokenRun / tokenTailRepeats); byte-period detector kept only as fallback.
- Eager trigger, no holdback queue, no straddling-event rewrite: events pass through before inspection; on trigger the upstream is closed and a synthetic data: [DONE] is appended.
- internal/streaming/tokenizer.go + tokenizer_test.go: TokenCounter over github.com/ron2111/omnitoken v0.1.7 (pure-Go tiktoken port).
Evidence:
- go test ./internal/streaming/... green (TestRepetitionGuardStream_TokenStutter, _ChainLoop, _FallbackStutter, _TriggerBeforeUpstreamEOF, TestTokenTailRepeats, TestClampGuardParams).
- Brown e2e over real HTTP SSE (worktree .tmp/stream-repetition-brown): guard triggered on gpt-4o tokenizer, upstream cut after 17 events, client saw 4 loop phrases, output ends with [DONE].
🤖 Written by Kimi Code (AI agent) from the stream-repetition-canceller worktree.
- added a commit that references this issue
on Sep 3, 2026
Derived from: #54, #55, #57, #62, #64
Work:
internal/streaming/repetition_guard_stream.gowith token-based detection.queuedEvent,rewriteEventAtCut, andreleaseSafeEvents.detectRunhelper or replace it with a token equivalent.internal/streaming/repetition_guard_stream_test.go./tmp/stream-repetition-brown/.Done when:
go build ./...passes;go test ./internal/streaming/...passes including the new token-based cases; the brown test shows truncation + [DONE] from a real HTTP SSE loop source.