Skip to content

fix: correct cache rate calculation by excluding cache_creation_tokens from denominator - #8018

Open
yikang-shen wants to merge 1 commit into
Wei-Shaw:mainfrom
yikang-shen:fix/channel-monitor-cache-rate-calculation
Open

yikang-shen wants to merge 1 commit into
Wei-Shaw:mainfrom
yikang-shen:fix/channel-monitor-cache-rate-calculation

Conversation

@yikang-shen

Copy link
Copy Markdown

Problem

Current channel monitor v2 calculates cache rate incorrectly:

cache_read / (input + cache_creation + cache_read)

This penalizes requests that write new cache entries. Cache creation represents tokens being indexed for future reuse, not cache misses. A request can simultaneously read from existing cache AND write new cache entries.

Example (production data, 2026-10-12)

Request with:

  • cache_read: 132,627 tokens
  • input: 50 tokens
  • cache_creation: 47,869 tokens

Current (incorrect) calculation:

132,627 / (50 + 47,869 + 132,627) = 58.4%

❌ Misleading - this request actually had 99.96% cache coverage!

Correct calculation:

132,627 / (50 + 132,627) = 99.96%

✅ Accurately represents that 99.96% of required input came from cache

Solution

Use standard cache hit rate definition:

cache_read / (cache_read + input)

This correctly measures what percentage of required input tokens came from cache.

Changes

  • Remove cache_creation from denominator in metricAccumulator.metric()
  • Denominator now: input + cache_read (tokens that needed to be processed)
  • Cache creation tokens still tracked separately for cost analysis

Impact

  • ✅ Cache rates will increase significantly for workloads with active cache writing
  • ✅ Metrics now align with Anthropic's usage model and standard cache semantics
  • ✅ No breaking changes to data collection or API schema
  • ✅ CacheRateNumerator and CacheRateDenominator fields updated consistently

Validation

Tested with production Response API data (2026-10-12, 19 requests):

Metric Before After
Displayed rate 82.87% 99.65%
Matches actual cache coverage ❌ No ✅ Yes

Aggregated totals:

  • cache_read: 1,951,088
  • input: 6,942
  • cache_creation: 396,240
  • Before: 1,951,088 / (6,942 + 396,240 + 1,951,088) = 82.87%
  • After: 1,951,088 / (6,942 + 1,951,088) = 99.65%

Reference

Anthropic Prompt Caching tracks cache_creation_input_tokens and cache_read_input_tokens separately - creation is not a "miss".

The current formula incorrectly treats cache creation as a penalty rather than recognizing it as building the cache for future requests.

…s from denominator

Current channel monitor v2 calculates cache rate as:
  cache_read / (input + cache_creation + cache_read)

This incorrectly penalizes requests that write new cache entries. Cache creation
represents tokens being indexed for future reuse, not cache misses. A request
can simultaneously read from existing cache AND write new cache entries.

Example with current (incorrect) formula:
  cache_read: 132,627 | input: 50 | cache_creation: 47,869
  Displayed rate: 132,627 / (50 + 47,869 + 132,627) = 58.4%
  ❌ Misleading - this request actually had 99.96% cache coverage

Standard cache hit rate definition:
  cache_read / (cache_read + input)

This correctly measures what percentage of required input came from cache.

Changes:
- Remove cache_creation from denominator in metricAccumulator.metric()
- Denominator now: input + cache_read (tokens that needed to be processed)
- Cache creation tokens still tracked separately for cost analysis

Impact:
- Cache rates will increase significantly for workloads with active cache writing
- Metrics now align with Anthropic's usage model and standard cache semantics
- No breaking changes to data collection or API schema

Tested with production data (Response API, 2026-10-12):
  Before: 82.87% displayed | After: 99.65% (matches actual cache coverage)

Reference: Anthropic Prompt Caching tracks cache_creation_input_tokens and
cache_read_input_tokens separately - creation is not a "miss".
riofutab added a commit to riofutab/sub2api that referenced this pull request Oct 14, 2026
riofutab added a commit to riofutab/sub2api that referenced this pull request Oct 14, 2026
- OpenAI/Codex 稳定性:OAuth 缺过期时间时按 JWT 判定并在 WS 上恢复 401(Wei-Shaw#8034,替换 fork 原有的 Wei-Shaw#7826 实现)、模型拒绝不计入调度错误率(Wei-Shaw#8013)、首包前预算耗尽换号(Wei-Shaw#8005)、Sol 别名误判修正(Wei-Shaw#8063)
- 流与协议:原生 Claude 转 Chat 转发心跳(Wei-Shaw#8052)、Codex 流式出图兼容 JSON 响应(Wei-Shaw#8028)、Antigravity 工具结果图片(Wei-Shaw#8045)
- 用量显示:5h 换窗保留未到期周额度(Wei-Shaw#7990)、缓存命中率口径修正与单次命中率显示(Wei-Shaw#8018、Wei-Shaw#7995)

测试:go test -tags=unit ./... 全部通过,golangci-lint 0 issues;前端 make test-frontend 通过;集成测试交 CI。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant