Repository navigation
fix(openai): stop pinning large request bodies after the handler - #8875
Merged
Merged
Conversation
Clone the few request-view strings that outlive the handler instead of aliasing the request body, drop the converted Responses input once it is marshaled, and avoid pointing usage logs at the forward input. Async prompt audit checks its scope before copying the body and no longer clones twice, splits and trims prompts without full []rune copies, and skips redaction rewrites that cannot match. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
All contributors have signed the CLA. ✅ |
Contributor
Author
|
I have read the CLA Document and I hereby sign the CLA |
zhx8702
marked this pull request as ready for review
November 17, 2026 05:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
一台 2 GiB 主机上,有用户持续发送带 4 张 base64 图片的
/v1/responses请求,单个请求从 20MB 涨到 46.6MB,至少 3 个并发。网关被 OOM kill 后重启,客户端立即重试,又被杀,如此循环。逐阶段测量存活堆时,除了改写阶段本身约 6 倍请求体的峰值,还发现了两类跟预算无关的问题:
[]rune。本 PR 只修这两类问题,不增加任何拒绝条件,也不加配置项。处理内存准入(503/413)的改动放在 #8876,那个 PR 依赖本 PR。
改动
请求体不再被长期引用(
internal/service/)newOpenAIRequestView:parseRawJSONView是零拷贝读取,读出的Model、PromptCacheKey、PreviousResponseID、ServiceTier、ReasoningEffort与请求体共享底层内存。这些字段会进入 usage log 和异步任务,于是整块请求体被一起留住。现在改为strings.Clone,每个字段只有几十字节。RecordUsage:usageLog.UserAgent、IPAddress原来取的是&input.UserAgent、&input.IPAddress,会把整个input留到异步写库结束,input里有Result,Result又引用请求体。现在改为取局部副本的地址。json.Marshal(responsesReq)之后把responsesReq.Input置为 nil。responsesReq要一直活到流结束后的计费,但之后只读它的标量字段。这样流式期间的存活量从约 3 倍请求体降到约 2 倍。异步提示词审核(
internal/securityaudit/)Coordinator不再先Clone()再交给PromptService.Enqueue,因为Enqueue自己还会再复制一次。接口注释写明了借用约定:调用期间只读借用,需要保留就自己复制。PromptService.Enqueue在复制正文之前,先检查配置是否生效、分组是否在范围内、是否有启用的端点,不满足就直接返回。日志事件和 dropped 计数跟Enqueuer原有的判断保持一致;Enqueuer持久化前仍会再检查一次配置。SplitRunes按 UTF-8 边界切出不可变子串,不再把整段转成[]rune再逐块复制。遇到无效 UTF-8 时保持原有行为。TrimRunes只遍历要保留的前缀。RedactPreview在正则没有命中时跳过ReplaceAllString,因为它即使不做任何替换也会复制一份全文。替换顺序不变,仍然对完整输入做匹配。行为变化
没有。对合法输入,输出与原实现一致;没有新的拒绝条件、配置项或 SQL。
效果
单请求存活堆:
GOGC=1,数值是请求体的倍数,在530c5e3a4上测得。异步审核每次操作新增的 Go 分配量:
TotalAlloc差值,统计的是累计分配,不是驻留内存。调用的是真实的Coordinator.Check → PromptService.Enqueue → Enqueuer.Enqueue → ExtractPromptSnapshot链路,仓库使用 fake 实现。代价:脱敏正则大量命中的文本慢约 3%–4%(约 0.6ms),原因是命中时多做一次
MatchString。测试
securityaudit/prompt_memory_optimization_test.goTrimRunes、SplitRunes、RedactPreview与原实现做对照:覆盖 Unicode、无效 UTF-8、优先级分隔符、完整输入的替换顺序,以及固定种子的语料。service/openai_gateway_service_hotpath_test.go:在构造 view 之后改写请求体,读出的字段不受影响,证明它们不再与请求体共享内存。go test ./internal/service/ ./internal/securityaudit/ ./internal/handler/通过,go vet通过。附注
main。perf: 同一用户的扣费按用户通道合并结算,请求体热点路径改用零拷贝读取 #8816 已经把compaction_trigger和空 base64 图片检测改成了零拷贝视图,这两处原本也在本 PR 的改动里,现已去掉。