Repository navigation
perf(gateway): parse the request body once instead of eight times - #8868
Merged
Wei-Shaw merged 4 commits intoNov 20, 2026
Merged
Conversation
parseGatewayRequestCurrentBody looked up eight fields with separate
gjson.Get calls. A lookup for a key that is not present still has to scan the
whole body to prove absence, and for a typical Responses request six of those
keys are absent (metadata.user_id, thinking.type, output_config.effort,
max_tokens, system, messages). On a 1MiB body that is six full passes at about
142us each, while a single root walk over the same body costs about 220us.
Collect every field the parser needs in one ForEach over the root and derive
the raw ranges from the same results.
Nested paths keep gjson semantics: each path segment resolves to the first
duplicate key that actually yields the child, not simply the first key with
that name. {"metadata":"x","metadata":{"user_id":"u"}} resolves to "u", and the
walk reproduces that by retrying while the collected child does not exist.
Duplicate scalar keys keep first-occurrence-wins.
The Anthropic [1m] model normalization rewrites the body mid-parse, so the walk
is repeated after the rewrite. That path only runs when the client leaks the
suffix.
Benchmark (Apple M5 Pro, 3 allocs unchanged):
responses 128KiB: 208us -> 96us
responses 1MiB: 1.59ms -> 735us
anthropic 1MiB: 1.43ms -> 723us
Equivalence tests keep a verbatim copy of the previous implementation as
legacy* and compare every field (model, stream, metadata user id, thinking,
effort, speed, max tokens, system/messages/input ranges, protocol, body after
parsing) across 4 protocols and 44 bodies, plus 3000 deterministic random
combinations. A fuzz target does the same for arbitrary input.
…es requests The /v1/responses handler validates the request body with gjson.ValidBytes before doing anything else, then hands the same bytes to ParseGatewayRequest, which validates them a second time. After the single-root-walk change the validation is the majority of what that call does: at 1MiB it is about 510us of the remaining 730us. Add ParseGatewayRequestValidated for callers that have already established the body is valid JSON, and use it in the handler, where the only rewrite between the two points is applyOpenAIReasoningEffortPolicyForRequest (sjson, which preserves validity). The validating entry point is unchanged, so invalid JSON keeps producing a clean error everywhere else. ReplaceBody / CloneForBody keep validating as well, because their bytes can differ from the ones a caller validated. Equivalence tests assert the trusted variant produces identical results to the validating one on valid JSON, and the fuzz target checks the same for every input that parses. Benchmark (Apple M5 Pro, 3 allocs unchanged): responses 1MiB: 735us -> 239us anthropic 1MiB: 723us -> 207us (not yet wired: that handler has no earlier validation, so it keeps the validating path)
Same shape as the responses handler: the body is validated at the top of ChatCompletions, the only later rewrite is applyOpenAIReasoningEffortPolicyForRequest (sjson), and the parse error was already being ignored. Nothing between the two points mutates the bytes: parseOpenAICompatibleStream, checkSecurityAudit and tokenInflightEstimate only read them.
…mark The benchmark body builder discarded the results of strings.Builder.WriteString, which errcheck rejects (the repo enables errcheck with check-blank off, so a plain call is flagged even though Builder never returns an error). Route the writes through one helper that assigns the result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
ParseGatewayRequest是转发前的公共解析步骤,/v1/responses、/v1/messages、chat_completions、Gemini 四条入口都走它,并且在账号选择之前同步执行。parseGatewayRequestCurrentBody用逐字段gjson.Get取model、stream、metadata.user_id、thinking.type、output_config.effort、speed、max_tokens,再取system/messages/input/contents/systemInstruction.parts,最后做一次gjson.ValidBytes。问题在于gjson.Get找不存在的键时也要把整份 body 扫完:典型 Responses 请求里metadata.user_id、thinking.type、output_config.effort、max_tokens、system、messages六个键都不存在,于是每次解析要做 7~8 次全量查找,其中 6 次只为确认「没有」。修改
两个 commit:
1.
0c571e9e用一次根对象遍历取代 7~8 次查找遍历实测约 220 µs,替代 6 次缺失键扫描(1 MiB 上约 852 µs)与
input读取(约 150 µs)。需要保留的语义:gjson 的嵌套路径会跨重复键重试,不是简单取第一个同名键。
gjson.Get(body,"metadata.user_id"){"metadata":"x","metadata":{"user_id":"u"}}"u"{"metadata":null,"metadata":{"user_id":"u"}}"u"{"metadata":{"other":1},"metadata":{"user_id":"u"}}"u"{"metadata":{"user_id":null},"metadata":{"user_id":"u"}}null(第一个命中键的就停)所以遍历里嵌套字段以「子键是否存在」为判据继续尝试后续同名键,单段字段(
model、stream、max_tokens…)保持第一个同名键胜出。Anthropic 的
[1m]后缀规范化会用 sjson 改掉 body,改写后旧的值视图失效,因此在改写过的那条罕见路径上重新遍历一次。2.
844459e7已校验 body 跳过重复校验/v1/responses的 handler 在入口已经gjson.ValidBytes过同一份字节,之后只经 sjson 改写(applyOpenAIReasoningEffortPolicyForRequest)。新增ParseGatewayRequestValidated给它用,跳过这次 511 µs 的校验。未校验入口不变;
ReplaceBody/CloneForBody继续校验——它们拿到的字节可能已被改写。这个 commit 可以单独丢弃,PR 的第一个 commit 已经能独立成立。3.
8d962ba4chat_completions入口同样接入形态与 responses 一致:入口已校验、之后只经 sjson 改写、解析错误本来就被忽略。两点之间没有别的写入者——
parseOpenAICompatibleStream、checkSecurityAudit、tokenInflightEstimate都只读。协议名只影响区间字段(这里取system/messages,没有input),标量查找是同一套,因此同尺寸 body 的收益与 responses 同量级。行为保证
legacy*,逐字段比较model/stream/metadata.user_id/thinking/effort/speed/max_tokens/system·messages·inputrange /protocol/ 解析后的 body,覆盖 4 种协议 × 44 组语料(重复键、空值、错误类型、\u转义键、[1m]泄露、根不是对象/数组、非法 JSON)。收益
Apple M5 Pro,0 分配:
chat_completions与 responses 走同一段解析代码,已接入变体。仍未改动
/v1/messages与 Gemini 入口没有提前校验,无法跳过第二次校验。给它们加一次前置校验只会把校验挪个位置,不会减少总成本。RequestBodyRef上:那要求每次改写都正确失效,风险高于显式传参。Fixes #8867