Skip to content

feat: Add support for counting cache-hit tokens in llama.cpp - #3357

Merged
seefs001 merged 3 commits into
QuantumNous:mainfrom
wenyifancc:cache_llama_cpp
Mar 21, 2026
Merged

seefs001 merged 3 commits into
QuantumNous:mainfrom
wenyifancc:cache_llama_cpp

Conversation

@wenyifancc

@wenyifancc wenyifancc commented Mar 20, 2026 •

Copy link
Copy Markdown

The current new-api cannot accurately count the number of tokens from llama.cpp or llama-swap backends that hit the cache, which prevents the application of caching billing rules. This PR adds the functionality to identify such cases.

Summary by CodeRabbit

  • Bug Fixes
    • Enhanced usage reporting to correctly capture cached token counts from responses, improving accuracy of API token metrics.
    • Improved handling when cached-token data is missing or not present in responses, making usage displays more complete and reliable.

@coderabbitai

coderabbitai Bot commented Mar 20, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8e2aff73-fbaa-46f4-afbe-b49961239a46

📥 Commits

Reviewing files that changed from the base of the PR and between 498199b and 2c3ae32.

📒 Files selected for processing (1)
  • relay/channel/openai/relay-openai.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • relay/channel/openai/relay-openai.go

Walkthrough

Adds extraction of cached prompt-token counts from OpenAI/Llama-style responses by adding extractLlamaCachedTokensFromBody(body []byte) (int, bool) and using it in applyUsagePostProcessing for constant.ChannelTypeOpenAI when usage.PromptTokensDetails.CachedTokens is zero.

Changes

Cohort / File(s) Summary
Cached Token Extraction
relay/channel/openai/relay-openai.go
Added extractLlamaCachedTokensFromBody(body []byte) (int, bool) to unmarshal response bodies and read timings.cache_n. Updated applyUsagePostProcessing to call this helper and set usage.PromptTokensDetails.CachedTokens for constant.ChannelTypeOpenAI when currently zero.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested reviewers

  • seefs001

Poem

🐰 I nibble through the JSON night,

sniff out numbers hidden tight,
Llama's timings whisper "cache",
I hop, parse, set the counted stash,
tiny paws, metrics bright.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'feat: Add support for counting cache-hit tokens in llama.cpp' directly summarizes the main change: adding functionality to detect and count cache-hit tokens for llama.cpp backends.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@relay/channel/openai/relay-openai.go`:
- Around line 705-715: The code currently dereferences
payload.Usage.CachedTokens without a nil check (the struct in this function uses
payload.Usage.CachedTokens), which can panic if cache_n is missing; update the
return path to check that payload.Usage.CachedTokens != nil before dereferencing
and if nil return 0, false (or an appropriate zero value/error), ensuring the
Unmarshal error handling remains unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 452ef2e8-e370-4f42-94fc-c4a6cf74773e

📥 Commits

Reviewing files that changed from the base of the PR and between ed6ff0f and ff29900.

📒 Files selected for processing (1)
  • relay/channel/openai/relay-openai.go

Comment thread relay/channel/openai/relay-openai.go Resolved
@seefs001
seefs001 merged commit 4d675b4 into QuantumNous:main Mar 21, 2026
1 check passed
RoodraNambisa added a commit to RoodraNambisa/new-api that referenced this pull request Mar 24, 2026
commit dbf900a
Author: CaIon <i@caion.me>
Date:   Wed Mar 25 00:04:01 2026 +0800

    fix: restore doubao coding plan deprecation and regex ignored models lost during conflict resolution

commit 7399e47
Author: CaIon <i@caion.me>
Date:   Tue Mar 24 23:56:10 2026 +0800

    feat: add slide-in animations and update translations for new UI elements

    # Conflicts:
    #	web/src/components/table/channels/modals/EditChannelModal.jsx

commit a5e2026
Author: CaIon <i@caion.me>
Date:   Tue Mar 24 23:53:50 2026 +0800

    security: harden Docker and release CI workflows

    - Pin all GitHub Actions to commit SHA to prevent supply chain attacks
    - Enable SLSA provenance attestation (mode=max) and SBOM generation
    - Add cosign keyless signing for Docker images via GitHub OIDC
    - Capture and output image digests to GitHub Job Summary
    - Pin Dockerfile base images to digest (bun:1, golang:1.26.1-alpine, debian:bookworm-slim)
    - Add SHA256 checksum generation for binary releases (Linux/macOS/Windows)
    - Update actions/checkout v3->v4, actions/setup-go v3->v5 in release.yml

commit 9ae9040
Merge: 0191a68 ded4a12
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:04:06 2026 +0800

    Merge pull request QuantumNous#3401 from seefs001/fix/convert-openai-detail-field

    fix: the "detail" field is empty, an empty field was sent to upstream

commit 0191a68
Merge: 16221f8 9ecad90
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:03:57 2026 +0800

    Merge pull request QuantumNous#3400 from seefs001/fix/openai-usage

    refactor: optimize billing flow for OpenAI-to-Anthropic convert

commit 16221f8
Merge: 763c3ff 929b506
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:03:47 2026 +0800

    Merge pull request QuantumNous#3399 from seefs001/refactor/codex-usage

    Refactor/codex usage

commit 763c3ff
Merge: c667e47 e520977
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:03:36 2026 +0800

    Merge pull request QuantumNous#3331 from seefs001/fix/claude-beta-query

    fix: apply forced beta query at final upstream URL stage

commit c667e47
Merge: 216b94d b09337e
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:03:23 2026 +0800

    Merge pull request QuantumNous#3333 from seefs001/fix/channel-affinity-disable

    fix: honor channel affinity skip-retry when channel is disabled

commit 216b94d
Merge: 49eb533 e9f8f62
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:03:01 2026 +0800

    Merge pull request QuantumNous#3335 from seefs001/chore/adjuct-default-settings

    adjuct default settings

commit 49eb533
Merge: 7693eda 45f65c2
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:02:44 2026 +0800

    Merge pull request QuantumNous#3381 from seefs001/feature/regex-ignored-upstream-models

    feat: support regex-prefixed ignored upstream models

commit 7693eda
Merge: d6982c8 f40eb4e
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 15:02:34 2026 +0800

    Merge pull request QuantumNous#3393 from seefs001/fix/oauth-bind

    fix: oauth bind callback handling

commit ded4a12
Author: Seefs <i@seefs.me>
Date:   Mon Mar 23 15:00:20 2026 +0800

    fix: the "detail" field is empty, an empty field was sent to the upstream system.

commit d6982c8
Merge: deff59a 6c074ef
Author: Calcium-Ion <i@caion.me>
Date:   Mon Mar 23 14:53:13 2026 +0800

    Merge pull request QuantumNous#3379 from seefs001/refactor/rm-coding-plan

    fix: disable doubao coding plan selection

commit 9ecad90
Author: Seefs <i@seefs.me>
Date:   Mon Mar 23 14:22:12 2026 +0800

    refactor: optimize billing flow for OpenAI-to-Anthropic convert

commit 929b506
Author: Seefs <i@seefs.me>
Date:   Mon Mar 23 13:54:54 2026 +0800

    refactor: simplify codex account modal and collapse raw json by default

commit 755ece2
Author: Seefs <i@seefs.me>
Date:   Mon Mar 23 00:58:59 2026 +0800

    refactor: simplify codex account modal and collapse raw json by default

commit f40eb4e
Author: Seefs <i@seefs.me>
Date:   Mon Mar 23 00:48:55 2026 +0800

    fix: oauth bind callback handling

commit 45f65c2
Author: Seefs <i@seefs.me>
Date:   Sun Mar 22 15:43:03 2026 +0800

    feat: support regex-prefixed ignored upstream models

commit 6c074ef
Author: Seefs <i@seefs.me>
Date:   Sun Mar 22 15:01:09 2026 +0800

    fix: disable doubao coding plan selection

commit deff59a
Author: CaIon <i@caion.me>
Date:   Sun Mar 22 13:55:03 2026 +0800

    fix: increase StreamScannerMaxBufferMB limit and add handling for gpt-5.4-nano prefix

commit 3c51608
Merge: 4d675b4 e80d867
Author: Seefs <40468931+seefs001@users.noreply.github.com>
Date:   Sun Mar 22 00:43:13 2026 +0800

    Merge pull request QuantumNous#3360 from lcq225/docs/improve-bt-installation-guide

    docs: 完善宝塔面板部署教程并修复链接错误

commit 4d675b4
Merge: 87b426f 2c3ae32
Author: Seefs <40468931+seefs001@users.noreply.github.com>
Date:   Sun Mar 22 00:39:49 2026 +0800

    Merge pull request QuantumNous#3357 from wenyifancc/cache_llama_cpp

    feat: Add support for counting cache-hit tokens in llama.cpp

commit 87b426f
Merge: 42846c6 49db514
Author: Seefs <40468931+seefs001@users.noreply.github.com>
Date:   Sun Mar 22 00:32:01 2026 +0800

    Merge pull request QuantumNous#3369 from RedwindA/feat/logsManagement

    feat: add server log file management to performance settings

commit 49db514
Author: RedwindA <austinaosid@gmail.com>
Date:   Sat Mar 21 21:48:31 2026 +0800

    fix: align log cleanup button with other controls in the row

commit 13122aa
Author: RedwindA <austinaosid@gmail.com>
Date:   Sat Mar 21 21:11:52 2026 +0800

    fix: refresh log info on partial delete failure

commit dcd0911
Author: RedwindA <austinaosid@gmail.com>
Date:   Sat Mar 21 20:40:39 2026 +0800

    fix: log management race condition, partial delete reporting, and UX issues

    - Fix data race on gin.DefaultWriter during log rotation by adding LogWriterMu
    - Report partial failure when some log files fail to delete instead of always returning success
    - Fix misleading "logging disabled" banner shown before API responds
    - Fix en.json translation for numeric validation message

commit e904579
Author: RedwindA <austinaosid@gmail.com>
Date:   Sat Mar 21 20:06:49 2026 +0800

    feat: add server log file management to performance settings

    Add API endpoints (GET/DELETE /api/performance/logs) to list and clean up
    server log files by count or by age. Track the active log file path in
    the logger to prevent deleting the currently open log. Add a management
    UI section in the performance settings page with log directory info,
    file statistics, and cleanup controls. Includes i18n translations for
    all supported languages (en, fr, ja, ru, vi, zh-CN, zh-TW).

commit e80d867
Author: mm413 <lcq225@163.com>
Date:   Fri Mar 20 20:13:30 2026 +0800

    Update docs/installation/BT.md

    Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

commit cf86fe5
Author: lcq225 <lcq225@163.com>
Date:   Fri Mar 20 20:06:09 2026 +0800

    docs: 完善宝塔面板部署教程并修复链接错误

    - 完善 docs/installation/BT.md,从 2 行扩展为完整教程
    - 包含前置要求、安装步骤、配置说明、常见问题
    - 修复 README.zh_CN.md 中的链接错误
    - 所有内容基于官方文档 https://docs.newapi.pro 编写

commit 2c3ae32
Author: wenyifan <yifan.wen@eisgroup.com>
Date:   Fri Mar 20 16:48:04 2026 +0800

    fix map

commit 498199b
Author: wenyifan <yifan.wen@eisgroup.com>
Date:   Fri Mar 20 16:38:48 2026 +0800

    fix code quality

commit ff29900
Author: wenyifan <yifan.wen@eisgroup.com>
Date:   Fri Mar 20 16:10:18 2026 +0800

    feat: Add support for counting cache-hit tokens in llama.cpp OpenAI-Compatible API

commit eff5185
Author: Seefs <i@seefs.me>
Date:   Fri Mar 20 16:00:36 2026 +0800

    refactor: show codex account info tag and highlight plan type in usage modal

commit e9f8f62
Author: Seefs <i@seefs.me>
Date:   Thu Mar 19 16:58:13 2026 +0800

    fix: raise default overload disk threshold to 95%

commit 5fe8e98
Author: Seefs <i@seefs.me>
Date:   Thu Mar 19 16:56:28 2026 +0800

    fix: default codex and claude channel affinity templates to skip retry on failure

commit e520977
Author: Seefs <i@seefs.me>
Date:   Thu Mar 19 15:49:50 2026 +0800

    fix: apply forced beta query at final upstream URL stage

commit b09337e
Author: Seefs <i@seefs.me>
Date:   Wed Mar 18 16:08:31 2026 +0800

    fix: honor channel affinity skip-retry when preferred channel is disabled
VeryGoodUser1 pushed a commit to xmz-ai/new-api that referenced this pull request Apr 3, 2026
feat: Add support for counting cache-hit tokens in llama.cpp
salem-2007 added a commit to salem-2007/new-api that referenced this pull request Sep 10, 2026
feat: Add support for counting cache-hit tokens in llama.cpp
junjundesk pushed a commit to junjundesk/new-api that referenced this pull request Oct 3, 2026
feat: Add support for counting cache-hit tokens in llama.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants