Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs-site/src/content/docs/ko/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@ HTTP 경계는 `server/index/serve-options.ts`가 맡고, Responses 데이터
7. `bridge/sse.ts` / `bridge/response-json.ts`가 Responses SSE 또는 JSON을 만듭니다. `server/request-log.ts`와 `usage/`는 응답을
건드리지 않은 채 종료 상태, 지연 시간, 프로바이더/모델, 최선 추정 토큰 사용량을 기록합니다.

요청 전 입력량 추정은 라우팅된 어댑터의 전송 방식에 맞춥니다. `openai-chat` 모델이
`preserveReasoningContentModels`에 없으면 전송하지 않는 이전 assistant thinking은 계산에서
제외하므로, 실제로 전송하지 않는 기록 때문에 로컬 컨텍스트 한도에서 잘못 거부되지 않습니다.
reasoning을 보존하는 모델과 다른 어댑터에서는 이전 thinking도 계속 계산합니다.

## 파서

`responses/parser.ts`는 들어오는 요청을 `responses/schema.ts`(Zod)로 검증한 다음
Expand Down
5 changes: 5 additions & 0 deletions docs-site/src/content/docs/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ the `server/responses.ts` facade and its `server/responses/*.ts` modules:
7. `bridge/sse.ts` / `bridge/response-json.ts` produces Responses SSE or JSON. `server/request-log.ts` and `usage/` collect terminal
status, latency, provider/model labels, and best-effort token usage without changing the response.

The pre-dispatch input estimate follows the routed adapter: for `openai-chat` models outside
`preserveReasoningContentModels`, it excludes replayed assistant thinking that the adapter does not
send. Models that preserve reasoning still count it. This prevents a local context-limit refusal
based on history that never reaches the provider.

## The parser

`responses/parser.ts` validates the incoming request with `responses/schema.ts` (Zod), then builds an
Expand Down
16 changes: 10 additions & 6 deletions src/server/responses/input-admission.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import { getModelMetadata } from "../../generated/model-metadata";
import { estimateTokens } from "../../lib/token-estimate";
import { isCanonicalOpenAiForwardProvider, OPENAI_CODEX_PROVIDER_ID } from "../../providers/openai-tiers";
import { modelRecordValue } from "../../reasoning-effort";
import type { OcxContentPart, OcxParsedRequest, OcxProviderConfig } from "../../types";
import { modelInList, type OcxContentPart, type OcxParsedRequest, type OcxProviderConfig } from "../../types";

/**
* Multiplier applied to the ceiling before refusing.
Expand Down Expand Up @@ -112,10 +112,12 @@ function contentTokens(content: string | readonly OcxContentPart[], modelId: str
* their content as `OcxAssistantContentPart[]` — text, thinking blocks, and tool calls whose
* JSON arguments are frequently the largest single item in an agent conversation. A walk
* that counted only `{type:"text"}` would undercount exactly the turns that trigger this
* gate.
* gate. The routed provider determines whether replayed thinking reaches the wire.
*/
export function estimateInputTokens(parsed: OcxParsedRequest, modelId: string): number {
export function estimateInputTokens(parsed: OcxParsedRequest, modelId: string, provider?: OcxProviderConfig): number {
const { context } = parsed;
const countThinking = provider?.adapter !== "openai-chat"
|| modelInList(provider.preserveReasoningContentModels, parsed.modelId);
let total = 0;

for (const prompt of context.systemPrompt ?? []) total += estimateTokens(prompt, modelId);
Expand All @@ -124,7 +126,9 @@ export function estimateInputTokens(parsed: OcxParsedRequest, modelId: string):
if (message.role === "assistant") {
for (const part of message.content) {
if (part.type === "text") total += estimateTokens(part.text, modelId);
else if (part.type === "thinking") total += estimateTokens(part.thinking, modelId);
else if (part.type === "thinking") {
if (countThinking) total += estimateTokens(part.thinking, modelId);
}
else total += estimateTokens(part.name, modelId) + estimateTokens(JSON.stringify(part.arguments), modelId);
}
// Opaque provider blob replayed verbatim upstream, so it costs real input tokens.
Expand Down Expand Up @@ -296,7 +300,7 @@ export function checkComboTargetInputAdmission(
const requiredOutputHeadroom = targetOutput === null
? requestedOutput
: Math.min(requestedOutput, targetOutput);
const estimatedTokens = estimateInputTokens(parsed, modelId);
const estimatedTokens = estimateInputTokens(parsed, modelId, provider);
return {
admitted: estimatedTokens <= ceiling && estimatedTokens + requiredOutputHeadroom <= window,
estimatedTokens,
Expand All @@ -319,6 +323,6 @@ export function checkInputAdmission(
): InputAdmissionResult {
const ceiling = resolveInputCeiling(provider, providerName, modelId, nativeContextCap);
if (ceiling === null) return { admitted: true, estimatedTokens: 0, ceiling: null };
const estimatedTokens = estimateInputTokens(parsed, modelId);
const estimatedTokens = estimateInputTokens(parsed, modelId, provider);
return { admitted: estimatedTokens <= ceiling * ADMISSION_TOLERANCE, estimatedTokens, ceiling };
}
2 changes: 2 additions & 0 deletions structure/transports/responses-failover.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,8 @@ output actually share. When the caller declared `max_output_tokens`,
`checkComboTargetInputAdmission` requires both `estimated input <= ceiling` and
`estimated input + min(declared output, target output ceiling) <= window`, so the output reserve
is counted once rather than charged twice against an already-tightened input budget.
Both direct and combo estimates omit replayed assistant thinking for `openai-chat` models outside
`preserveReasoningContentModels`, matching the adapter's wire omission; other targets still count it.

The refusal is local: HTTP 413 `input_admission_refused` before any upstream bytes are sent, which
existing combo policy already treats as a safe hop. That ordering is the whole point. A target whose
Expand Down
42 changes: 42 additions & 0 deletions tests/server/input-admission.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
resolveOutputCeiling,
} from "../../src/server/responses/input-admission";
import { modelRecordValue } from "../../src/reasoning-effort";
import { messagesToChatFormat } from "../../src/adapters/openai-chat/messages";
import type { OcxMessage, OcxParsedRequest, OcxProviderConfig, OcxTool } from "../../src/types";

const CANONICAL_NATIVE: OcxProviderConfig = {
Expand All @@ -29,6 +30,20 @@ function userText(text: string): OcxMessage {
return { role: "user", content: text, timestamp: 0 };
}

/** Create a replay whose thinking alone exceeds the test model's context window. */
function replayedThinking(): OcxParsedRequest {
return {
...request([
userText("Continue."),
{ role: "assistant", timestamp: 0, content: [
{ type: "thinking", thinking: asciiTokens(30_000) },
{ type: "text", text: "Done." },
] },
]),
modelId: "m",
};
}

/** Roughly `tokens` worth of plain ASCII at the default 4 chars/token ratio. */
function asciiTokens(tokens: number): string {
return "a".repeat(tokens * 4);
Expand Down Expand Up @@ -197,6 +212,22 @@ describe("checkInputAdmission", () => {
modelContextWindows: { "m": 10_000 },
};

/** Compare direct admission with the actual serialized reasoning payload. */
test("ignores reasoning that openai-chat drops but counts preserved and native reasoning", () => {
const parsed = replayedThinking();
const dropped = checkInputAdmission(parsed, provider, "custom", "m");
expect(dropped.admitted).toBe(true);
expect(dropped.estimatedTokens).toBeLessThan(100);
expect(JSON.stringify(messagesToChatFormat(parsed, provider))).not.toContain("reasoning_content");
expect(JSON.stringify(messagesToChatFormat(parsed, provider))).not.toContain(asciiTokens(30_000));

const preserved = { ...provider, preserveReasoningContentModels: ["m"] };
expect(JSON.stringify(messagesToChatFormat(parsed, preserved))).toContain("reasoning_content");
expect(JSON.stringify(messagesToChatFormat(parsed, preserved))).toContain(asciiTokens(30_000));
expect(checkInputAdmission(parsed, preserved, "custom", "m").admitted).toBe(false);
expect(checkInputAdmission(parsed, { ...provider, adapter: "openai-responses" }, "custom", "m").admitted).toBe(false);
});

test("admits input under the ceiling", () => {
const result = checkInputAdmission(request([userText(asciiTokens(5_000))]), provider, "custom", "m");
expect(result.admitted).toBe(true);
Expand Down Expand Up @@ -268,6 +299,17 @@ describe("combo target input admission", () => {
modelMaxOutputTokens: { m: 32_000 },
};

/** Verify each combo target counts only reasoning retained by its wire compiler. */
test("uses the target's reasoning replay policy before reserving output space", () => {
const parsed = { ...replayedThinking(), options: { maxOutputTokens: 1_000 } };
const target = { ...capped, modelContextWindows: { m: 10_000 }, modelMaxOutputTokens: { m: 1_000 } };
expect(checkComboTargetInputAdmission(parsed, target, "custom", "m").admitted).toBe(true);
expect(JSON.stringify(messagesToChatFormat(parsed, target))).not.toContain(asciiTokens(30_000));
const preserved = { ...target, preserveReasoningContentModels: ["m"] };
expect(checkComboTargetInputAdmission(parsed, preserved, "custom", "m").admitted).toBe(false);
expect(JSON.stringify(messagesToChatFormat(parsed, preserved))).toContain(asciiTokens(30_000));
});

const withMaxOutput = (inputTokens: number, maxOutputTokens = 64_000): OcxParsedRequest => ({
...request([userText(asciiTokens(inputTokens))]),
modelId: "m",
Expand Down
Loading