Skip to content

perf(gateway): parse the request body once instead of eight times - #8868

Merged
Wei-Shaw merged 4 commits into
Wei-Shaw:mainfrom
lishuangqiang:perf/gateway-request-single-scan
Nov 20, 2026
Merged

Wei-Shaw merged 4 commits into
Wei-Shaw:mainfrom
lishuangqiang:perf/gateway-request-single-scan

Conversation

@lishuangqiang

@lishuangqiang lishuangqiang commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

背景

ParseGatewayRequest 是转发前的公共解析步骤,/v1/responses、/v1/messages、chat_completions、Gemini 四条入口都走它,并且在账号选择之前同步执行。

parseGatewayRequestCurrentBody 用逐字段 gjson.Get 取 model、stream、metadata.user_id、thinking.type、output_config.effort、speed、max_tokens,再取 system / messages / input / contents / systemInstruction.parts,最后做一次 gjson.ValidBytes。问题在于 gjson.Get 找不存在的键时也要把整份 body 扫完:典型 Responses 请求里 metadata.user_id、thinking.type、output_config.effort、max_tokens、system、messages 六个键都不存在,于是每次解析要做 7~8 次全量查找,其中 6 次只为确认「没有」。

修改

两个 commit:

1. 0c571e9e 用一次根对象遍历取代 7~8 次查找

遍历实测约 220 µs,替代 6 次缺失键扫描(1 MiB 上约 852 µs)与 input 读取(约 150 µs)。

需要保留的语义:gjson 的嵌套路径会跨重复键重试,不是简单取第一个同名键。

body gjson.Get(body,"metadata.user_id")
{"metadata":"x","metadata":{"user_id":"u"}} "u"
{"metadata":null,"metadata":{"user_id":"u"}} "u"
{"metadata":{"other":1},"metadata":{"user_id":"u"}} "u"
{"metadata":{"user_id":null},"metadata":{"user_id":"u"}} null(第一个命中键的就停)

所以遍历里嵌套字段以「子键是否存在」为判据继续尝试后续同名键,单段字段(model、stream、max_tokens…)保持第一个同名键胜出。

Anthropic 的 [1m] 后缀规范化会用 sjson 改掉 body,改写后旧的值视图失效,因此在改写过的那条罕见路径上重新遍历一次。

2. 844459e7 已校验 body 跳过重复校验

/v1/responses 的 handler 在入口已经 gjson.ValidBytes 过同一份字节,之后只经 sjson 改写(applyOpenAIReasoningEffortPolicyForRequest)。新增 ParseGatewayRequestValidated 给它用,跳过这次 511 µs 的校验。

未校验入口不变;ReplaceBody / CloneForBody 继续校验——它们拿到的字节可能已被改写。这个 commit 可以单独丢弃,PR 的第一个 commit 已经能独立成立。

3. 8d962ba4 chat_completions 入口同样接入

形态与 responses 一致:入口已校验、之后只经 sjson 改写、解析错误本来就被忽略。两点之间没有别的写入者——parseOpenAICompatibleStream、checkSecurityAudit、tokenInflightEstimate 都只读。协议名只影响区间字段(这里取 system / messages,没有 input),标量查找是同一套,因此同尺寸 body 的收益与 responses 同量级。

行为保证

  • 差分测试把改动前的实现原样保留为 legacy*,逐字段比较 model / stream / metadata.user_id / thinking / effort / speed / max_tokens / system·messages·input range / protocol / 解析后的 body,覆盖 4 种协议 × 44 组语料(重复键、空值、错误类型、\u 转义键、[1m] 泄露、根不是对象/数组、非法 JSON)。
  • 3000 组确定性随机组合(键顺序、重复键、类型混杂)。
  • fuzz 目标对任意字节与协议做同样的对比,累计约 180 万次执行通过。
  • 已校验变体额外断言:合法 JSON 上结果与校验路径完全一致(fuzz 内也做同一断言)。
  • 分配数与改前一致(216–232 B / 3 allocs),没有引入额外拷贝。

收益

Apple M5 Pro,0 分配:

场景 改前 单次遍历 + 已校验变体
responses 128 KiB 208 µs 96 µs 32 µs
responses 1 MiB 1.59 ms 735 µs 239 µs
anthropic 1 MiB 1.43 ms 723 µs 207 µs(该入口无提前校验,未接入)

chat_completions 与 responses 走同一段解析代码,已接入变体。

仍未改动

  • /v1/messages 与 Gemini 入口没有提前校验,无法跳过第二次校验。给它们加一次前置校验只会把校验挪个位置,不会减少总成本。
  • 没有选择把「body 是否合法」缓存在 RequestBodyRef 上:那要求每次改写都正确失效,风险高于显式传参。

Fixes #8867

parseGatewayRequestCurrentBody looked up eight fields with separate
gjson.Get calls. A lookup for a key that is not present still has to scan the
whole body to prove absence, and for a typical Responses request six of those
keys are absent (metadata.user_id, thinking.type, output_config.effort,
max_tokens, system, messages). On a 1MiB body that is six full passes at about
142us each, while a single root walk over the same body costs about 220us.

Collect every field the parser needs in one ForEach over the root and derive
the raw ranges from the same results.

Nested paths keep gjson semantics: each path segment resolves to the first
duplicate key that actually yields the child, not simply the first key with
that name. {"metadata":"x","metadata":{"user_id":"u"}} resolves to "u", and the
walk reproduces that by retrying while the collected child does not exist.
Duplicate scalar keys keep first-occurrence-wins.

The Anthropic [1m] model normalization rewrites the body mid-parse, so the walk
is repeated after the rewrite. That path only runs when the client leaks the
suffix.

Benchmark (Apple M5 Pro, 3 allocs unchanged):
  responses 128KiB: 208us -> 96us
  responses 1MiB:   1.59ms -> 735us
  anthropic 1MiB:   1.43ms -> 723us

Equivalence tests keep a verbatim copy of the previous implementation as
legacy* and compare every field (model, stream, metadata user id, thinking,
effort, speed, max tokens, system/messages/input ranges, protocol, body after
parsing) across 4 protocols and 44 bodies, plus 3000 deterministic random
combinations. A fuzz target does the same for arbitrary input.
…es requests

The /v1/responses handler validates the request body with gjson.ValidBytes
before doing anything else, then hands the same bytes to ParseGatewayRequest,
which validates them a second time. After the single-root-walk change the
validation is the majority of what that call does: at 1MiB it is about 510us
of the remaining 730us.

Add ParseGatewayRequestValidated for callers that have already established the
body is valid JSON, and use it in the handler, where the only rewrite between
the two points is applyOpenAIReasoningEffortPolicyForRequest (sjson, which
preserves validity).

The validating entry point is unchanged, so invalid JSON keeps producing a
clean error everywhere else. ReplaceBody / CloneForBody keep validating as
well, because their bytes can differ from the ones a caller validated.

Equivalence tests assert the trusted variant produces identical results to the
validating one on valid JSON, and the fuzz target checks the same for every
input that parses.

Benchmark (Apple M5 Pro, 3 allocs unchanged):
  responses 1MiB: 735us -> 239us
  anthropic 1MiB: 723us -> 207us (not yet wired: that handler has no earlier
  validation, so it keeps the validating path)
Same shape as the responses handler: the body is validated at the top of
ChatCompletions, the only later rewrite is
applyOpenAIReasoningEffortPolicyForRequest (sjson), and the parse error was
already being ignored. Nothing between the two points mutates the bytes:
parseOpenAICompatibleStream, checkSecurityAudit and tokenInflightEstimate only
read them.
…mark

The benchmark body builder discarded the results of strings.Builder.WriteString,
which errcheck rejects (the repo enables errcheck with check-blank off, so a
plain call is flagged even though Builder never returns an error). Route the
writes through one helper that assigns the result.
@Wei-Shaw
Wei-Shaw merged commit a437224 into Wei-Shaw:main Nov 20, 2026
7 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Perf] ParseGatewayRequest 对每个请求做 7~8 次全量 JSON 查找,其中 6 次只为确认键不存在

2 participants