Skip to content

feat(relay): add provider response-model diagnostics across Go, Rust and Web - #584

Merged
LIghtJUNction merged 6 commits into
mainfrom
codex/quality-response-model-20261003
Oct 3, 2026
Merged

LIghtJUNction merged 6 commits into
mainfrom
codex/quality-response-model-20261003

Conversation

@LIghtJUNction

@LIghtJUNction LIghtJUNction commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

LMM API Pull Request

变更说明 / Summary

用量日志以前只能显示请求和映射模型,无法看到供应商实际声明的模型。本次在转换前观察原始 OpenAI/Responses、Claude、Gemini 响应,保留有用的 requested/selected/returned 三个名称,并在列表和详情动态计算警告。真实不相容声明不会被后续匹配或空事件清除。

比较规则沿用上游最终语义:返回名称与 requested 或 selected 任一相容就不警告;A→B→A 保留三个名称但不报警。仅放行明确的日期/latest/preview 后缀,gpt-4o→mini 等不同模型仍报警。Go/Rust/Web 共用 24 条样例,不存 mismatch boolean。诊断不参与路由或计价,也保留 Claude 原有模型 setter。

Rust 在已有 OpenAI/Responses 消费日志写相同 metadata;其原生 Claude/Gemini 尚无该结算日志、WS 未配置的边界已写入文档。七语言文案经维护脚本生成,旧日志正常显示。

与上游的关系 / Upstream relationship

  • 同步或移植上游变更 / Upstream sync or backport

上游参考 / Upstream reference: QuantumNous/new-api@ed7c4e3 、 QuantumNous/new-api@972aed1 。按 LMM 双后端和现有日志/UI 适配,收窄宽松后缀比较。

关联 Issue / Related issue

Closes #378

依赖 #591 已合入。服务器验收抓到的 main 既有 WS 失败预留释放竞态已修复,此 immutable head 的所有检查已重新通过。

变更类型 / Type of change

  • 新功能 / Feature
  • 前端或交互 / Frontend or UX

验证 / Verification

最终 immutable head 1db2716764167cb80289abaa2c0fcd053a7f3cde,正常 merge 已验 9142600c 与 #591 dff6ae1ff,未 rebase/force。相对 914 仅两个 WS 文件,诊断、翻译、35 项计数及 negative guard 原样;合并后 relay tree 的 31 个有测试包 race 全绿,relay/service/model vet 通过。

  • Go relay/service 全套 tests/vet、独立 relaykit tests/vet 通过;中央再次运行六包模型诊断测试通过。真实 handler 同时断言响应字节、usage、转换模型和日志;禁用 observer 后新断言变红。
  • 完整整合 lib 暴露旧 Evidence 测试样例漏必填 Usage 字段,已补完整旧样例;未放宽生产 decoder。整合 lib 修后 938 passed/5 ignored,WS 35、observability 13、readonly 2 项均通过。
  • Rust 最终 production + 真实 HTTP/PostgreSQL settlement:35 passed、0 ignored,覆盖四种 Chat/Responses stream/nonstream 及恢复、不重放、费用和 token 不变。最终 lib/Clippy/workspace fmt 由中央及 PR CI 校验。
  • pinned Bun 1.3.14:最终合并 head 57 项专项测试、3 项 scoped generator、typecheck、1912 文件格式检查通过。七语言逐项保留两个 parent 的全部旧 key/value;新增 WS 四条文案的独立生成选项避免引入无关旧文案改动。此前 scoped lint/copyright、缺键和 full suite 通过;最终整合中央补 pinned full Web。
  • Rust 实际集成 gate 已同步精确名单与 35 项执行计数;新增故意漏掉诊断测试的负向检查,确保 gate 在执行前拒绝漏跑。脚本契约检查通过(8 项 harness + 3 项 inventory)。
  • 没有实际窄屏浏览器验收;已做 DOM 列表/详情/popover/copy、长名称和空 selected 测试。

截图或日志 / Screenshots or logs

真实 HTTP/PG 消费断言保持 quota 32、tokens 20/4,诊断只新增 metadata。源码中的共享 fixture 与组件测试提供 exact/alias/mismatch/旧日志证据。exact head 的 CI、完整服务器资格、Billing safety、browser-review 和 CodeQL 全部通过,25 checks 均已完成,无失败或待执行项。

提交检查 / Checklist

  • 我已搜索本仓库的 Issues 和 PRs,确认没有重复工作。
  • 此 PR 只包含与当前目标相关的改动,未混入格式化或其他无关变更。
  • 我已说明该变更对当前产品流程或兼容行为的影响,并保留必要的来源、版权和许可证信息。
  • 我已检查兼容性;如有破坏性变化,已在说明中明确列出迁移方式。
  • 我已补充或更新相关测试,并在上方记录实际验证结果;如未测试,已说明原因。
  • 我已更新受影响的文档、示例配置和多语言文案(如适用)。
  • 代码、日志、截图和提交记录中不包含 API Key、Cookie、Token、密码、DSN 或其他敏感信息。

Observe provider-declared models before conversion, retain useful three-name metadata with a sticky derived mismatch, and share a bounded compatibility contract across Go, Rust, and the usage-log UI. Keep existing response and billing behavior, including native Claude model handling. Refs #378.
Copilot AI balanced review requested due to automatic review settings October 3, 2026 12:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-10-03T12:18:17.225549Z 801c645 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Member Author

[P2] Go 的 selected model 还需要覆盖参数覆写后的实际出站模型

ObserveResponseModel 在 response_model.go:69-71 从 GetUpstreamModelName() 冻结 selected,但 TextHelper 会在模型映射/请求转换之后调用 ApplyParamOverrideWithRelayInfo,并直接发送覆写后的 JSON;这个函数只同步 reasoning/header/audit,没有同步这里的新诊断字段。

因此,请求 client-alias → 映射 gpt-4o → 渠道参数覆写 {"model":"gpt-4.1"},上游正确返回 gpt-4.1 时,日志仍是 requested=client-alias / selected=gpt-4o / returned=gpt-4.1,列表和详情都会误报 mismatch。现有 model 的 set/trim_suffix 覆写也会遇到这个问题。

建议在最终出站请求确定后单独冻结诊断用的 selected model,覆盖参数覆写路径,同时保持现有计价/路由字段不变;补一条经过真实请求构建的回归用例,断言上游收到的 model 与 response_model.upstream_model 一致、合法覆写不报警。

核对版本:801c645909d77ab4324ac4fc7f455f3b4e9fc3ec。相关路径:apps/api-go/relay/compatible_handler.go、apps/api-go/relay/common/override.go、apps/api-go/relay/common/response_model.go。

LIghtJUNction commented Oct 3, 2026 •

Copy link
Copy Markdown
Member Author

更新(head d598662):本评论记录的两项 CI 问题均已修复并验证。集成测试门禁已同步至 35 项;旧 Evidence fixture 补齐了原有 Usage 必填字段,生产 decoder 未放宽。当前 head 的 CI 全部通过,原失败的 missing_empty_or_exact_provider_model_keeps_legacy_evidence_and_logs_unchanged 用例也已通过。

CI:https://github.com/TokenNotIncluded/api.lmm.best/actions/runs/37123734458

以下保留初次失败的定位记录:

CI 已确认一个独立阻塞点:新增的两条 PostgreSQL 回归测试尚未加入真实集成测试门禁清单。

当前 Workflow and resource-safety contracts 在 Validate real integration gate contracts 失败,日志为:relay_openai_settlement_pg.rs ignored-test set changed: expected 33, found 35。

需要一起同步:

  • apps/api-rust/tests/scripts/check-real-integration-gates.sh 的 relay_openai_settlement_pg.rs 测试名称清单,加入本 PR 的两条 provider_response_model_diagnostics_* 用例
  • apps/api-rust/tests/scripts/run-real-integration-gates.sh 中 require_api_ignored_test_count 和 run_counted_api_integration_tests 的 33 → 35
  • apps/api-rust/tests/scripts/test-new-integration-suites.py 的 relay-settlement listing/execution 数量断言 33 → 35

只改第一处仍会在真实集成 runner 的数量校验处失败。核对版本:801c645。失败 job:https://github.com/TokenNotIncluded/api.lmm.best/actions/runs/37121618855/job/111198708863

补充:Rust workspace 测试也在同一 head 失败(928 passed / 1 failed / 5 ignored)。missing_empty_or_exact_provider_model_keeps_legacy_evidence_and_logs_unchanged 在 billing/tests.rs:639 解码旧证据 fixture 时报 missing field cached。嵌套的 Usage 没有 #[serde(default)];当前 fixture 只给了 input/output,缺少既有必填字段 cached/cache_write/image,尚未执行到 response_model 的兼容断言。请用完整的旧版序列化 usage fixture(补齐这三个字段为 0,或从旧版等价 Usage 值序列化)验证缺少 response_model 的兼容性,再重跑该用例和 workspace 测试。

Rust 失败 job:https://github.com/TokenNotIncluded/api.lmm.best/actions/runs/37121618855/job/111198708918

Bind the settlement integration gate to all 35 source and compiled tests, including both response model diagnostics regressions. Keep exact test names and execution-count guards, and verify that dropping the new regressions fails before execution. Refs #378.
Retain WebSocket capability, authorization and reader isolation alongside provider model observations. Preserve both translation sets and provide a scoped WebSocket copy generator with exact-key regression coverage. Keep all documentation index links and the 35-test settlement gate.
@LIghtJUNction
LIghtJUNction merged commit ed4eb57 into main Oct 3, 2026
25 checks passed
@LIghtJUNction
LIghtJUNction deleted the codex/quality-response-model-20261003 branch October 11, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(log): track upstream response-model mismatches as diagnostics

2 participants