Skip to content

fix(doctor): check the complete Hermes SOUL read budget - #1331

Merged
SaulMoro merged 2 commits into
Tencent:mainfrom
nateEc:fix/1221-hermes-soul-budget
Oct 24, 2026
Merged

SaulMoro merged 2 commits into
Tencent:mainfrom
nateEc:fix/1221-hermes-soul-budget

Conversation

@nateEc

@nateEc nateEc commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A current managed rules block is insufficient proof of delivery when member-authored text makes the complete SOUL.md exceed Hermes' context-file cap. Doctor now measures the whole managed file using Python Unicode code points after trimming, and reads context_file_max_chars from Hermes' resolved config.yaml.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change
  • Documentation only
  • Refactor / internal cleanup

Test Plan

  • npx tsc --noEmit passes

  • npm run lint passes

  • npx vitest run passes completely (two unchanged local baseline failures; see below)

  • Added/updated tests for the change

  • RED: an authored prefix plus a small managed block had no read-budget diagnostic.

  • 276 focused tests pass, covering exact boundaries, astral Unicode, missing caps, malformed YAML and unmanaged files.

  • TypeScript, lint and build pass.

  • Real CLI: doctor-hermes-budget.test.ts runs doctor --json on an oversized file, verifies the budget failure and unchanged contents, then shortens the file and verifies that check passes.

Real-CLI command: npx vitest run --config vitest.e2e.config.ts src/__tests__/e2e/doctor-hermes-budget.test.ts.

Repository-wide regression: the five-patch combined run on origin/main 6c57c3b7 completed with 12,268 passed, 36 skipped, 2 failed across 503 files. Both failures are the existing submodule-checkout cases for preserving empty old Claude/Codex hook files (git config --get hook.teamai-post-checkout.command). An independent pristine checkout of the same base reproduces exactly those 2 failures (24 other submodule tests pass). No new full-suite failure was introduced. The final ZCode exclusion follow-up is additionally covered by 267 uninstall/lifecycle tests and a post-uninstall pull through the real CLI.

The final combined build, TypeScript and lint checks also pass. A final real-CLI run with all five patches passes 6 critical cases across all 5 affected E2E files, including the post-uninstall pull.

Related Issues

Related to #1221: Hermes standing-instruction truncation.

Notes for Reviewers

An explicit configured overflow fails the check. Without an explicit cap, Hermes derives a model-dependent limit with a 20,000-character floor, so an overflow of that floor is an informational warning rather than a claim of definite truncation. Malformed YAML reports an unknown budget. The fix hint and bilingual guide explain shortening the file or reviewing the cap; no content is truncated automatically.

Source: Hermes SOUL guide, prompt assembly, and the prompt-builder implementation referenced by #1221. This tests the CLI diagnostic, not a live Hermes/model session.

Local runtime tests use the Git provider and isolated filesystem fixtures; additional providers and live agent applications are deferred to CI/manual verification.

Follow-up verification (2026-10-25)

  • Rebased onto current main, retaining OpenClaw's budget guidance and the new resource-location API.
  • Fixed the Python str.strip boundary mismatch: strip C0 separators and NEL, preserve BOM. The new boundary regression failed before the change and passed afterwards.
  • Hermes budget/config unit tests: 22 passed. Built-CLI doctor-hermes-budget.test.ts: 1 passed. Build, type checking and lint passed.
  • Without an explicit cap, the model-dependent default warning remains informational in pull; standalone doctor still counts a failed informational check as a failure under its existing policy. Malformed YAML reports an unknown budget. Live Hermes was not run.
  • Rebased again onto main a26f74d after fix(hooks): launch OpenCode through the installed Node entry #1336 merged; retained both sides of the documentation conflict. Final rebase verification: 22 focused unit tests, 1 built-CLI E2E case, build, types and lint passed.

Updated head: 290662f7f2ff5d36c8542e2fb8dbb3ff126f896a. Review base: a26f74d5bef51c498cc465b90599faa0130dc666.

@github-actions

Copy link
Copy Markdown
  • [P3 nit] src/doctor-delivery.ts:2239 — JavaScript .trim() does not match Python .strip(). For example, Python strips leading U+0085 whitespace, but JavaScript counts it. A managed file whose remaining text exactly meets the explicit cap therefore incorrectly fails this check. Match Python’s whitespace handling alongside its code-point counting.

The PR description provides sufficient testing documentation, including build verification and a representative real-CLI run covering overflow, unchanged contents, and recovery after shortening. No missing-testing finding. I did not run, build, or install anything.

@SaulMoro SaulMoro left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

@nateEc Thanks, approving. This covers the Hermes SOUL.md part of the #1221 shared-budget item.

We built this head and its merge-base and ran teamai doctor --json against a fixture Hermes home. With an explicit context_file_max_chars, a SOUL.md one character over fails the new check, and one at the limit passes. Base has no such check. The new unit and e2e tests fail on base and pass here, and the existing doctor and Hermes tests pass. tsc and lint pass.

Suggestions:

  • [P2 non-blocking] Without an explicit cap, the check is informational, but that flag only affects pull's summary (src/pull.ts). teamai doctor still exits 1 with "Some checks failed" (allPassed = results.every(r => r.ok) in src/doctor.ts). Going by Hermes's source, the default cap is max(20000, min(context_window × 4 × 0.06, 500000)), so with a 128k-context model a 25,000-character SOUL.md is read whole, yet doctor fails. The existing informational checks behave the same, so this follows the repo. Passing the check when no cap is set, or saying in the fix text that doctor counts this as a failure, would avoid a red doctor for a file Hermes reads whole.
  • [P2 non-blocking] A malformed config.yaml fails the check as "budget unknown". Going by Hermes's source, its loader falls back to the last good config or the defaults, so nothing is truncated. We did not run Hermes.
  • [P3 nit] .trim() is not Python's .strip(): Python also strips U+0085 and U+001C to U+001F. It only matters for a file at the exact boundary.
  • [P3 nit] The unmanaged-file test has an unused await budget('').

#1332 adds a sibling check and touches the same paragraphs in docs/guide/diagnostics.md, so whichever lands second needs a rebase there.

CI review: its one finding ([P3 nit] .trim() vs Python .strip()) is the third suggestion above, and we agree. It also accepted the testing record, which reproduces here. It did not raise the two [P2 non-blocking] points.

@SaulMoro
SaulMoro self-requested a review October 23, 2026 12:52

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

@nateEc The runtime review remains positive, but main moved when #1332 merged. This review supersedes the approval until the documentation conflicts are resolved.

  • [P1 blocking] This head no longer merges cleanly with main. A merge against main 12b80e7f conflicts in docs/guide/diagnostics.md and docs/guide/zh-CN/diagnostics.md. Please merge or rebase current main, preserving both this PR's Hermes guidance and #1332's OpenClaw guidance. The production doctor changes merge cleanly; full content re-review is deferred to the updated head.

CI review reported one [P3 nit] about JavaScript .trim() differing from Python .strip(), and accepted the representative real-CLI testing record. We agree that the whitespace point is non-blocking and the record is sufficient. Existing head CI passed; the new blocker is the merge conflict caused by main advancing, not a failing CI job.

@SaulMoro

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

@nateEc the conflict with main has grown since my last review. Besides docs/guide/diagnostics.md and its Chinese copy, it now also hits src/doctor-delivery.ts, where #1265 changed writesOf() to return objects. Please rebase onto current main. Apart from that, the runtime review was positive, and only the CI review's .trim() vs .strip() nit is open.

@nateEc
nateEc force-pushed the fix/1221-hermes-soul-budget branch from c07733d to e7a53ed Compare October 24, 2026 17:15
@nateEc

nateEc commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review follow-up at e7a53ed492e1a132325f386adffa744c2e132e04 on current main.

  • Rebased onto current main, retaining OpenClaw's budget guidance and the new resource-location API.
  • Fixed the Python str.strip boundary mismatch: strip C0 separators and NEL, preserve BOM. The new boundary regression failed before the change and passed afterwards.
  • Hermes budget/config unit tests: 22 passed. Built-CLI doctor-hermes-budget.test.ts: 1 passed. Build, type checking and lint passed.
  • Without an explicit cap, the model-dependent default warning remains informational in pull; standalone doctor still counts a failed informational check as a failure under its existing policy. Malformed YAML reports an unknown budget. Live Hermes was not run.

The PR description contains the updated verification record. Ready for another review.

@github-actions

Copy link
Copy Markdown

No new findings in the scoped PR diff.

  • Resolved: The earlier Python str.strip() mismatch is fixed: the whitespace set includes C0 separators and NEL, preserves BOM, and has a boundary regression test.
  • Testing documentation: Sufficient. The description records build verification and a representative built-CLI run covering overflow, unchanged contents, and recovery after shortening. Additional provider/live-agent coverage is explicitly deferred.

I only read the diff and trusted base review rules; I did not run, build, or install anything from the PR.

- Measure member-authored text and every managed block together, using Python Unicode code points after trimming whitespace as Hermes does.
- Honor an explicit context_file_max_chars cap and report malformed YAML; treat the model-dependent 20,000-character default floor as an informational warning.
- Keep instructions intact and explain how to shorten the complete file or review the configured cap, with matching English and Chinese diagnostic guidance.
- Verify 276 focused tests, TypeScript, lint and build; the real doctor CLI rejects an oversized fixture, preserves its contents and passes the budget check after shortening.
- 对齐 Python str.strip 的 Unicode 空白集合,正确剥离 C0 分隔符和 NEL,同时保留 BOM。
- 添加精确上限用例,修复前复现失败,修复后通过;保留主分支 OpenClaw 诊断说明和定位接口。
- 22 个相关单测和真实 CLI 预算诊断用例通过,构建、类型检查和 lint 通过。
@nateEc
nateEc force-pushed the fix/1221-hermes-soul-budget branch from e7a53ed to 290662f Compare October 24, 2026 18:06
@nateEc

nateEc commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review follow-up at 290662f7f2ff5d36c8542e2fb8dbb3ff126f896a on current main.

  • Rebased onto current main, retaining OpenClaw's budget guidance and the new resource-location API.
  • Fixed the Python str.strip boundary mismatch: strip C0 separators and NEL, preserve BOM. The new boundary regression failed before the change and passed afterwards.
  • Hermes budget/config unit tests: 22 passed. Built-CLI doctor-hermes-budget.test.ts: 1 passed. Build, type checking and lint passed.
  • Without an explicit cap, the model-dependent default warning remains informational in pull; standalone doctor still counts a failed informational check as a failure under its existing policy. Malformed YAML reports an unknown budget. Live Hermes was not run.
  • Rebased again onto main a26f74d after fix(hooks): launch OpenCode through the installed Node entry #1336 merged; retained both sides of the documentation conflict. Final rebase verification: 22 focused unit tests, 1 built-CLI E2E case, build, types and lint passed.

The PR description contains the updated verification record. Ready for another review.

@github-actions

Copy link
Copy Markdown

No new findings in the scoped PR diff.

  • Resolved: The earlier Python str.strip() mismatch is fixed. The whitespace set includes C0 separators and NEL, preserves BOM, and has a boundary regression test.
  • Testing documentation: Sufficient. The description records build verification and a representative built-CLI run covering overflow, unchanged contents, and recovery after shortening. Additional providers and live-agent coverage are explicitly deferred.

I only read the scoped diff and trusted base review rules; I did not run, build, or install anything from the PR.

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

@nateEc Thanks, approving at 290662f. The latest rebase keeps #1336's OpenCode paragraph next to this PR's Hermes paragraph in both diagnostics guides, so the conflict is resolved. This is also the full content review I deferred last time. The whitespace handling now matches Python's strip(): the regex covers exactly the 29 code points isspace() accepts. Build, tsc, lint, the unit files (568 tests), the Hermes e2e and the doctor e2e pass, also with a symlinked TMPDIR.

  • [P2 non-blocking] Hermes reads SOUL.md with utf-8-sig, so it never sees a BOM, but this check counts it. A file with a BOM exactly at the cap is reported as over: the built CLI counts 20,001 where Hermes loads 20,000. The keeps BOM test pins that behavior.
  • [P3 nit] The trailing-whitespace regex is quadratic. 50k trailing spaces take 8.9 s and 100k take 47.6 s, which is unlikely in a real SOUL.md. The lint directive is eslint-disable where src uses oxlint-disable. The unmanaged-file test still has an unused await budget('').
  • The earlier notes stand as accepted: a default-cap overrun makes doctor exit 1, per the existing policy, and malformed YAML reports an unknown budget.

The CI review reports no findings. We agree that nothing blocks. CI at this head is still running.

@SaulMoro
SaulMoro merged commit 1266d4d into Tencent:main Oct 24, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants