Skip to content

fix(recall): cap the confidence feedback ratio - #1256

Merged
SaulMoro merged 1 commit into
Tencent:mainfrom
ydflow:fix/bound-learning-confidence
Oct 21, 2026
Merged

SaulMoro merged 1 commit into
Tencent:mainfrom
ydflow:fix/bound-learning-confidence

Conversation

@ydflow

@ydflow ydflow commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1255.

Manual positive feedback can exceed a learning's recall count, making the confidence ratio exceed its 30% weight and the complete score exceed 1. Cap that ratio before weighting, preserving the existing base, recency, feedback counters, and promotion thresholds.

Add regressions for excess feedback with recent, absent, and expired recalls; clarify the score's limits in the bilingual knowledge guide and agent troubleshooting reference. This addresses one existing arithmetic defect related to #408, without adding an evaluation system.

Validation:

  • New regression tests failed before the fix (2.2 / 1.9 / 1.9), then passed: npx vitest run src/__tests__/confidence.test.ts (8 passed).
  • npm run build, npx tsc --noEmit, npm run lint passed.
  • Real built-CLI verification in isolated Windows/local-Git fixtures: five recall feedback --positive lesson invocations followed by recall maintenance --confidence-writeback, with the generated votes staged on the fixture reports branch. Before: confidence: 2.2; after: confidence: 1. No remote service or model was used.
  • Focused confidence/promotion/prune/read-only-votes suite: 23 passed, 1 failed. The failure is the unchanged Windows assertion in maintenance-promote.test.ts:137 expecting skills/candidate.md inside a native Windows path. The same test also fails on clean upstream fa6a361.
  • The full Windows unit-suite attempt was interrupted after failures in unmodified modules (including symlink/permission failures). Its log is retained; those failures have not all been baseline-audited. No complete Windows-suite pass is claimed.

Signed-off-by: ydflow <314143294+ydflow@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

No findings in the specified diff.

The ratio cap fixes the reported overflow while preserving the existing base, recency calculation, counters, and promotion thresholds. Regression tests cover excess feedback with recent, absent, and expired recalls; documentation is consistent across both languages.

The PR description satisfies the testing requirement with a representative built-CLI before/after verification. The reported unchanged Windows test failure is not a blocking issue for this diff.

I did not run, build, or install anything from the PR.

@SaulMoro
SaulMoro self-requested a review October 21, 2026 07:54

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

This PR caps the upvote-to-recall ratio in computeConfidence at 1 before the 0.4 / 0.3 / 0.3 weights, so manual recall feedback --positive can no longer push confidence above 1 or stand in for recency (fixes #1255). The one-line formula change is correct, and the new tests pin the three cases from the issue. Every reader of the score now gets values in [0, 1]: frontmatter writeback, prune, promote, and search-index hot/cold. The only behavior change is the intended one: a learning that reached the 0.90 promotion bar, or the hot threshold, only through ratio overflow no longer does. The earlier codex-review-on-assign pass reported no findings, and there are no other review threads.

Local checks at head 3ce7e78c, and on its merge into current origin/main 0da2be76 (the PR base fa6a361b is behind; the merge is clean):

  • npx tsc --noEmit: pass on head and on the merge.
  • npx vitest run src/__tests__/confidence.test.ts src/__tests__/maintenance-promote.test.ts: 14 passed on head. On the merge, adding maintenance-prune, maintenance-votes-readonly, search-index, votes, and commands-reference: 7 files, 132 passed.
  • Red/green: with origin/main's confidence.ts restored on the merge, the 3 new tests fail (expected 2.2 to be 1, expected 1.9 to be 0.7 twice). With the PR's version they pass.
  • npm run lint (exit 0) and npm run build: pass on the merge.
  • Real CLI under a throwaway HOME with a local bare git team repo. I ran teamai recall feedback --positive lesson 5 times (Upvoted: lesson each time), staged the resulting totals (recalled_count: 1, upvoted_count: 5, recent last_recalled_at) on teamai-reports, then ran teamai recall maintenance --confidence-writeback. The published teamai-learnings:learnings/lesson.md had confidence: 2.2 with the origin/main build and confidence: 1 with the merge build. Both exited 0 and printed Published maintenance changes to the learnings branch.

Findings

  • [P1 blocking] None.
  • [P3 nit] docs/guide/knowledge.md:100 (and docs/guide/zh-CN/knowledge.md:100): "cannot exceed that factor's 30% weight or replace the recency factor" refers to weights and factors the guide never defines, so a reader cannot use it. A plainer sentence is enough: "Confidence is an activity score from 0 to 1, not a measure of whether the knowledge is correct. Extra positive feedback cannot push it above 1 or make up for a learning that has not been recalled recently." Both languages are present and say the same thing.
  • [P3 nit] skill-data/core/references/troubleshooting.md:359-362: this paragraph sits under "A recalled doc got no upvote" but describes the score formula, which is not that symptom, and it repeats the guide. AGENTS.md asks to change docs rarely and to treat skill-data/ like docs. Consider keeping only the first sentence (manual feedback adds an upvote without a recall), which fits the section, or dropping the paragraph.

Not covered

  • I did not get the pending feedback deltas onto teamai-reports through the CLI itself. teamai pull in this fixture left them pending in ~/.teamai/user-votes/, so I staged the totals by hand, as the issue's repro did. Vote sync is unchanged by this PR.
  • No e2e file covers confidence writeback, so I ran none. I did not run the full unit or E2E suites, or any other provider or agent. The change is a pure function, so they are not needed here.

Verdict: Approve.

@SaulMoro
SaulMoro merged commit 2aa8a8a into Tencent:main Oct 21, 2026
17 checks passed
legend91019 added a commit to legend91019/teamai-cli that referenced this pull request Oct 21, 2026
Walternater pushed a commit to Walternater/teamai-cli that referenced this pull request Nov 2, 2026
Signed-off-by: ydflow <314143294+ydflow@users.noreply.github.com>
Co-authored-by: ydflow <314143294+ydflow@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Manual feedback can raise learning confidence above 100%

3 participants