Skip to content

perf: optimize memory alloc - #4158

Closed
seefs001 wants to merge 4 commits into
QuantumNous:mainfrom
seefs001:perf/memory-alloc
Closed

seefs001 wants to merge 4 commits into
QuantumNous:mainfrom
seefs001:perf/memory-alloc

Conversation

@seefs001

@seefs001 seefs001 commented Apr 9, 2026 •

Copy link
Copy Markdown
Collaborator

AI Generated:

  • 分发前读取 model/group 的逻辑做了轻量化处理。原来仅为了拿顶层字段也会完整反序列化请求体,现在优先直接从 JSON body 提取顶层字段,只有不满足条件时才回退原逻辑,减少一次整包 JSON 解码。
  • legacy param override 的常见场景改成直接在原 JSON 上做顶层字段替换。对于 model、service_tier 这类简单 key,不再走整包 Unmarshal -> map -> Marshal,只在复杂 path 规则下保留旧逻辑,降低 override 的额外内存开销。
  • UnmarshalBodyReusable 在处理 JSON 时不再先把 body 复制成一份新的 []byte,而是直接从 BodyStorage 解码,再复位请求体,减少一次不必要的 body 拷贝。
  • CombineText 的构造方式从 []string + strings.Join 改成 builder 逐段拼接,在保持结果一致的前提下,去掉中间切片分配,降低 token 统计和敏感词检查前的瞬时内存占用。

Summary by CodeRabbit

  • Refactor

    • Improved request parsing efficiency for JSON payloads.
    • Streamlined token counting mechanism across multiple request types.
    • Optimized legacy override application with fast path support for simple configuration keys.
  • Tests

    • Added comprehensive test coverage for request parsing, token counting validation, and override handling.

@seefs001
seefs001 marked this pull request as draft April 9, 2026 06:30
@coderabbitai

coderabbitai Bot commented Apr 9, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

The PR refactors token text building across multiple DTO files to use a dedicated tokenTextBuilder pattern instead of string slicing and joining, optimizes UnmarshalBodyReusable for stream-based JSON decoding, adds a fast-path for JSON-based model/group extraction in the distributor middleware, and introduces a direct-apply fast-path for simple legacy parameter overrides using byte-level JSON manipulation.

Changes

Cohort / File(s) Summary
Token Text Builder Refactoring
dto/token_text_builder.go, dto/token_text_builder_test.go, dto/claude.go, dto/embedding.go, dto/gemini.go, dto/openai_request.go, dto/rerank.go
Introduced new tokenTextBuilder type for incremental text accumulation with newline separation. Replaced string slice accumulation + strings.Join(texts, "\n") pattern with textBuilder.Add() and textBuilder.String() across all token count meta generation methods. Removed strings import from affected files.
Stream-Based JSON Decoding
common/gin.go
Refactored UnmarshalBodyReusable to decode JSON directly from reusable BodyStorage stream via DecodeJson(storage, v) after seeking to start. Moved requestBody extraction into respective content-type branches for form-encoded and multipart data.
Distributor JSON Extraction
middleware/distributor.go, middleware/distributor_test.go
Added JSON-specific fast-path in getModelFromRequest using gjson to directly extract model and group fields from JSON bodies. Introduced helper functions extractModelRequestFromJSONBody and isStringOrMissingJSONField with validation logic. Falls back to reusable body unmarshal on JSON extraction failure.
Override Fast-Path Optimization
relay/common/override.go, relay/common/override_test.go
Introduced canApplyLegacyOverridesDirectly helper and regex validation for simple top-level keys. Added fast-path in applyOperationsLegacy to apply overrides directly via sjson.SetBytes when keys match allowed pattern, bypassing JSON unmarshal/marshal cycle. Preserved fallback to existing map-based path for complex keys.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Suggested reviewers

  • Calcium-Ion

Poem

🐰 A builder for tokens, precise and true,
Streams flow without lingering—old bytes bid adieu,
JSON fields extracted with swift, keen sight,
Overrides now bypass the marshal—so light!
whiskers twitch The refactoring hops just right! 🐇

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title 'perf: optimize memory alloc' is vague and generic, using non-descriptive terms that don't convey specific information about the actual changes in the changeset. Provide a more specific title that describes the actual optimization, such as 'perf: replace string.Join with tokenTextBuilder for token counting' or 'perf: add JSON fast path in distributor middleware'.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
dto/openai_request.go (1)

447-461: ⚠️ Potential issue | 🟡 Minor

Direct json.Unmarshal and json.Marshal usage violates coding guidelines.

Lines 452 and 459 use encoding/json directly instead of common.Unmarshal and common.Marshal. As per coding guidelines, all JSON marshal/unmarshal operations must use wrapper functions from common/json.go.

🔧 Proposed fix
 func (m *Message) ParseToolCalls() []ToolCallRequest {
 	if m.ToolCalls == nil {
 		return nil
 	}
 	var toolCalls []ToolCallRequest
-	if err := json.Unmarshal(m.ToolCalls, &toolCalls); err == nil {
+	if err := common.Unmarshal(m.ToolCalls, &toolCalls); err == nil {
 		return toolCalls
 	}
 	return toolCalls
 }

 func (m *Message) SetToolCalls(toolCalls any) {
-	toolCallsJson, _ := json.Marshal(toolCalls)
+	toolCallsJson, _ := common.Marshal(toolCalls)
 	m.ToolCalls = toolCallsJson
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@dto/openai_request.go` around lines 447 - 461, Replace direct encoding/json
calls in Message.ParseToolCalls and Message.SetToolCalls with the project's JSON
wrappers: use common.Unmarshal instead of json.Unmarshal when decoding
m.ToolCalls in ParseToolCalls and use common.Marshal instead of json.Marshal
when encoding in SetToolCalls; preserve current behavior (returning nil or the
decoded slice on error) but call the wrapper functions and handle their returned
(value, error) results consistently, referencing the Message.ParseToolCalls and
Message.SetToolCalls methods to locate the changes.
dto/claude.go (1)

410-417: ⚠️ Potential issue | 🟡 Minor

Direct json.Unmarshal usage violates coding guidelines.

Line 412 uses json.Unmarshal directly instead of common.Unmarshal. As per coding guidelines, all JSON marshal/unmarshal operations must use wrapper functions from common/json.go.

🔧 Proposed fix
 func (c *ClaudeRequest) GetEfforts() string {
 	var OutputConfig OutputConfigForEffort
-	if err := json.Unmarshal(c.OutputConfig, &OutputConfig); err == nil {
+	if err := common.Unmarshal(c.OutputConfig, &OutputConfig); err == nil {
 		effort := OutputConfig.Effort
 		return effort
 	}
 	return ""
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@dto/claude.go` around lines 410 - 417, The GetEfforts method in ClaudeRequest
uses json.Unmarshal directly; replace that call with the project's JSON wrapper
(common.Unmarshal) to comply with guidelines: call
common.Unmarshal(c.OutputConfig, &OutputConfig) inside ClaudeRequest.GetEfforts,
keep the existing error handling (i.e., only return effort when unmarshal
succeeds), and reference the OutputConfigForEffort struct and the GetEfforts
method when making the change.
🧹 Nitpick comments (2)
dto/gemini.go (1)

519-530: Inconsistent refactoring: GeminiEmbeddingRequest.GetTokenCountMeta() still uses the old pattern.

This method still uses []string + strings.Join while GeminiChatRequest.GetTokenCountMeta() was refactored to use tokenTextBuilder. Consider applying the same refactor for consistency.

♻️ Suggested refactor for consistency
 func (r *GeminiEmbeddingRequest) GetTokenCountMeta() *types.TokenCountMeta {
-	var inputTexts []string
+	var textBuilder tokenTextBuilder
 	for _, part := range r.Content.Parts {
 		if part.Text != "" {
-			inputTexts = append(inputTexts, part.Text)
+			textBuilder.Add(part.Text)
 		}
 	}
-	inputText := strings.Join(inputTexts, "\n")
 	return &types.TokenCountMeta{
-		CombineText: inputText,
+		CombineText: textBuilder.String(),
 	}
 }

The same applies to GeminiBatchEmbeddingRequest.GetTokenCountMeta() at lines 547-559.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@dto/gemini.go` around lines 519 - 530, Replace the old []string +
strings.Join pattern in GeminiEmbeddingRequest.GetTokenCountMeta() with the same
tokenTextBuilder utility used by GeminiChatRequest: instantiate a
tokenTextBuilder, iterate r.Content.Parts and call builder.Add(part.Text) for
non-empty texts, then call builder.String() (or builder.Build()) to produce
CombineText and return it in the TokenCountMeta; apply the identical change to
GeminiBatchEmbeddingRequest.GetTokenCountMeta() so both methods use
tokenTextBuilder consistently.
middleware/distributor_test.go (1)

41-65: Tests cover key scenarios appropriately.

The type mismatch and non-object tests validate the fallback/skip behavior correctly. Consider adding a test for group being non-string while model is valid to ensure both fields are validated.

💡 Additional test case suggestion
func TestExtractModelRequestFromJSONBodySkipsNonStringGroup(t *testing.T) {
	gin.SetMode(gin.TestMode)

	body := `{"model":"gpt-4.1","group":123}`
	ctx, _ := gin.CreateTestContext(httptest.NewRecorder())
	ctx.Request = httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBufferString(body))
	ctx.Request.Header.Set("Content-Type", "application/json")

	modelRequest, ok, err := extractModelRequestFromJSONBody(ctx)
	require.NoError(t, err)
	require.Nil(t, modelRequest)
	require.False(t, ok)
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@middleware/distributor_test.go` around lines 41 - 65, Add a unit test that
verifies extractModelRequestFromJSONBody rejects payloads where "group" is not a
string even if "model" is valid: create a Gin test context and request with body
`{"model":"gpt-4.1","group":123}`, call extractModelRequestFromJSONBody(ctx),
and assert no error, returned modelRequest is nil, and ok is false; place it
alongside TestExtractModelRequestFromJSONBodySkipsNonJSONObject and reference
extractModelRequestFromJSONBody to locate the logic to validate.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@common/gin.go`:
- Around line 119-130: The bug is variable shadowing: the inner `err` declared
with `:=` when calling `storage.Bytes()` in the branches for
`strings.Contains(contentType, gin.MIMEPOSTForm)` and
`strings.Contains(contentType, gin.MIMEMultipartPOSTForm)` hides the outer
`err`, so errors from `parseFormData` and `parseMultipartFormData` are lost; fix
by using assignment (`=`) instead of short-declare (`:=`) for `requestBody, err`
(or split into two statements: `requestBody, err = storage.Bytes()` or
`requestBody := ...; if err := ...;` avoided) and ensure the same `err` variable
is used so that returned errors from `parseFormData` and
`parseMultipartFormData` propagate correctly from the surrounding function
(update the branches around `storage.Bytes()`, `parseFormData`, and
`parseMultipartFormData` accordingly).

---

Outside diff comments:
In `@dto/claude.go`:
- Around line 410-417: The GetEfforts method in ClaudeRequest uses
json.Unmarshal directly; replace that call with the project's JSON wrapper
(common.Unmarshal) to comply with guidelines: call
common.Unmarshal(c.OutputConfig, &OutputConfig) inside ClaudeRequest.GetEfforts,
keep the existing error handling (i.e., only return effort when unmarshal
succeeds), and reference the OutputConfigForEffort struct and the GetEfforts
method when making the change.

In `@dto/openai_request.go`:
- Around line 447-461: Replace direct encoding/json calls in
Message.ParseToolCalls and Message.SetToolCalls with the project's JSON
wrappers: use common.Unmarshal instead of json.Unmarshal when decoding
m.ToolCalls in ParseToolCalls and use common.Marshal instead of json.Marshal
when encoding in SetToolCalls; preserve current behavior (returning nil or the
decoded slice on error) but call the wrapper functions and handle their returned
(value, error) results consistently, referencing the Message.ParseToolCalls and
Message.SetToolCalls methods to locate the changes.

---

Nitpick comments:
In `@dto/gemini.go`:
- Around line 519-530: Replace the old []string + strings.Join pattern in
GeminiEmbeddingRequest.GetTokenCountMeta() with the same tokenTextBuilder
utility used by GeminiChatRequest: instantiate a tokenTextBuilder, iterate
r.Content.Parts and call builder.Add(part.Text) for non-empty texts, then call
builder.String() (or builder.Build()) to produce CombineText and return it in
the TokenCountMeta; apply the identical change to
GeminiBatchEmbeddingRequest.GetTokenCountMeta() so both methods use
tokenTextBuilder consistently.

In `@middleware/distributor_test.go`:
- Around line 41-65: Add a unit test that verifies
extractModelRequestFromJSONBody rejects payloads where "group" is not a string
even if "model" is valid: create a Gin test context and request with body
`{"model":"gpt-4.1","group":123}`, call extractModelRequestFromJSONBody(ctx),
and assert no error, returned modelRequest is nil, and ok is false; place it
alongside TestExtractModelRequestFromJSONBodySkipsNonJSONObject and reference
extractModelRequestFromJSONBody to locate the logic to validate.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 469100c4-8bb6-48e0-b4cf-20a5baf5f380

📥 Commits

Reviewing files that changed from the base of the PR and between 78e4cb3 and 96b0a67.

📒 Files selected for processing (12)
  • common/gin.go
  • dto/claude.go
  • dto/embedding.go
  • dto/gemini.go
  • dto/openai_request.go
  • dto/rerank.go
  • dto/token_text_builder.go
  • dto/token_text_builder_test.go
  • middleware/distributor.go
  • middleware/distributor_test.go
  • relay/common/override.go
  • relay/common/override_test.go

Comment thread common/gin.go
Comment on lines 119 to 130
} else if strings.Contains(contentType, gin.MIMEPOSTForm) {
requestBody, err := storage.Bytes()
if err != nil {
return err
}
err = parseFormData(requestBody, v)
} else if strings.Contains(contentType, gin.MIMEMultipartPOSTForm) {
requestBody, err := storage.Bytes()
if err != nil {
return err
}
err = parseMultipartFormData(c, requestBody, v)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical bug: variable shadowing causes parsing errors to be silently ignored.

The := on lines 120 and 126 declares new err variables in the inner scope that shadow the outer err from line 109. When parseFormData (line 124) or parseMultipartFormData (line 130) returns an error, it's assigned to the inner err, which goes out of scope immediately. The check on line 135 then reads the outer err (which is nil), causing the function to return nil instead of the parsing error.

🐛 Proposed fix to prevent variable shadowing
 	} else if strings.Contains(contentType, gin.MIMEPOSTForm) {
-		requestBody, err := storage.Bytes()
-		if err != nil {
-			return err
+		requestBody, bytesErr := storage.Bytes()
+		if bytesErr != nil {
+			return bytesErr
 		}
 		err = parseFormData(requestBody, v)
 	} else if strings.Contains(contentType, gin.MIMEMultipartPOSTForm) {
-		requestBody, err := storage.Bytes()
-		if err != nil {
-			return err
+		requestBody, bytesErr := storage.Bytes()
+		if bytesErr != nil {
+			return bytesErr
 		}
 		err = parseMultipartFormData(c, requestBody, v)
 	} else {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
} else if strings.Contains(contentType, gin.MIMEPOSTForm) {
requestBody, err := storage.Bytes()
if err != nil {
return err
}
err = parseFormData(requestBody, v)
} else if strings.Contains(contentType, gin.MIMEMultipartPOSTForm) {
requestBody, err := storage.Bytes()
if err != nil {
return err
}
err = parseMultipartFormData(c, requestBody, v)
} else if strings.Contains(contentType, gin.MIMEPOSTForm) {
requestBody, bytesErr := storage.Bytes()
if bytesErr != nil {
return bytesErr
}
err = parseFormData(requestBody, v)
} else if strings.Contains(contentType, gin.MIMEMultipartPOSTForm) {
requestBody, bytesErr := storage.Bytes()
if bytesErr != nil {
return bytesErr
}
err = parseMultipartFormData(c, requestBody, v)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@common/gin.go` around lines 119 - 130, The bug is variable shadowing: the
inner `err` declared with `:=` when calling `storage.Bytes()` in the branches
for `strings.Contains(contentType, gin.MIMEPOSTForm)` and
`strings.Contains(contentType, gin.MIMEMultipartPOSTForm)` hides the outer
`err`, so errors from `parseFormData` and `parseMultipartFormData` are lost; fix
by using assignment (`=`) instead of short-declare (`:=`) for `requestBody, err`
(or split into two statements: `requestBody, err = storage.Bytes()` or
`requestBody := ...; if err := ...;` avoided) and ensure the same `err` variable
is used so that returned errors from `parseFormData` and
`parseMultipartFormData` propagate correctly from the surrounding function
(update the branches around `storage.Bytes()`, `parseFormData`, and
`parseMultipartFormData` accordingly).

@Calcium-Ion
Calcium-Ion force-pushed the main branch 2 times, most recently from 51fdfc5 to 2b6f1df Compare August 30, 2026 15:03
@seefs001 seefs001 closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant