Repository navigation
fix(memory): carry the reindex failure reason out of the watch loop - #2767
Conversation
`on_event` receives only an event name, so a failing reindex could print "Reindex failed; change kept pending, will retry next poll." while the exception message that explains why was discarded at watch.py:126 and again at watch.py:174. Add an opt-in `on_error(event, exc)` that poll_once calls with the exception it already holds, and print the reason from `wren memory watch`. `on_event` keeps its one-argument contract unchanged. Consequence 1 of Canner#2724 (never retrying a permanent failure) is left alone: choosing the terminal exception set is a design call for maintainers.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. 🧰 Additional context used📚 Code guidelines (1)No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review. WalkthroughThe watcher now accepts an optional ChangesWatcher Reindex Error Reporting
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix · Severity of issue fixed: Medium Suggested reviewers: Merge Risk: ⚪ Minimal · up to The watcher now reports reindex failure details while continuing to retry pending changes. No actionable merge-blocking risk is established. Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The change remains within the local memory command and preserves pending-change retries. No security defect was established, but the suitability of all backend exception messages for captured error output remains unconfirmed. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. A rabbit hears the watcher sigh, Comment |
goldmedal
left a comment
There was a problem hiding this comment.
Thanks — the split from #2724 is well-reasoned, and keeping on_event's one-argument contract while adding a keyword-only on_error is the right shape. The three tests would genuinely fail on a broken implementation, and the once-per-attempt test pins the no-duplicate property nicely.
One issue in the CLI branch, reproduced against 5ae576f:
_reindex can fail by echoing its own error and raising typer.Exit(1) — e.g. _load_manifest on a half-written mdl.json, which is the canonical transient case the watcher's retry exists for. typer.Exit subclasses RuntimeError, so poll_once catches it, and str(typer.Exit(1)) == "1". With a truncated target/mdl.json the output is:
Error: invalid JSON in .../target/mdl.json: Unterminated string starting at: line 1 column 2 (char 1)
Reindex failed: 1; change kept pending, will retry next poll.
Reindex failed: 1 reads like an error code and adds nothing, since the real reason was already printed on the line above. Suggest skipping the reason for typer.Exit:
def _on_error(event: str, exc: BaseException) -> None:
if event != "reindex-error":
return
# typer.Exit means _reindex already echoed its own error.
reason = "" if isinstance(exc, typer.Exit) else f": {str(exc) or type(exc).__name__}"
typer.echo(
f"Reindex failed{reason}; change kept pending, will retry next poll.",
err=True,
)This path is also cheap to cover at the CLI level: _load_manifest fails before _get_store, so a CliRunner test only needs resolve_backend monkeypatched to a non-grep value plus an invalid mdl.json. It doesn't need the memory extra or any backend mocks, and it would pin both the typer.Exit case and the message format.
|
Fixed in c6870ec — |
c456877 to
c6870ec
Compare
goldmedal
left a comment
There was a problem hiding this comment.
Thanks @lihongyuan99 👍
Summary
wren memory watchis the one memory command that cannot tell you why it stopped working. When a reindex fails, the user sees a fixed sentence and the exception that explains the problem is thrown away:Meanwhile the object it just discarded said, in full (the
OnnxBackendErrorfrom the issue's repro):This adds the opt-in
on_error(event, exc)callback that #2724 suggests as the smaller of its two fixes, and prints the reason fromwatch.Closes #2724 — consequence 2 only. See Scope below for why consequence 1 is not in this PR.
What changes for a user
wren memory watchnow reports why a reindex failed, on every attempt, instead of an unexplained retry loop:The retry behaviour, the fingerprint semantics and
WatchState.errorsare untouched.How it works
The exception is available at the point where it is dropped, so this is a plumbing change:
poll_once(src/wren/memory/watch.py) binds the exception it already catches and callson_error("reindex-error", exc)before re-raising. Re-raising is unchanged, sowatch_loopstill keeps the change pending and retries.watch_loopaccepts and forwardson_error. It deliberately does not call it again in its ownexceptclause: the same exception arrives there as a re-raise frompoll_once, so reporting at both sites would print two messages per attempt. The docstring states this.memory/cli.pymoves the "Reindex failed" line out of_on_eventinto a new_on_error(event, exc)that interpolatesstr(exc) or type(exc).__name__(an exception with an empty message still names its class).on_eventkeeps itsCallable[[str], None]contract, which is why this is additive rather than a widening: the existing tests passon_event=events.append, a one-argument callable, and #2724 records that both of its proposed shapes "alterwatch_loop's contract". A new keyword-only parameter with aNonedefault is the one form that does not.Tests
Three tests in
tests/unit/test_memory_watch.py, all callingpoll_once/watch_loopand asserting on what the callbacks actually receive — not on source text:test_poll_once_reports_reindex_reason_to_on_error(event, exception)pair reacheson_error, andpoll_oncestill re-raisestest_watch_loop_reports_reindex_reason_once_per_attemptstate.errors == 2and the rendered messages are["reindex-error: attempt 1: …", "reindex-error: attempt 2: …"]— i.e. one reason per attempt, no duplicatetest_watch_loop_on_event_still_works_without_on_erroron_event=events.append(one positional arg) still drives a failing loop: the new parameter is opt-inRed before the fix, with the source changes reverted and only the tests present:
Green after:
16 passed in 0.25s. The third test is expected to pass both before and after — it pins the compatibility property, so it can only fail ifon_event's contract is broken.These run in CI.
test_memory_watch.pycarries nopytestmark, so a-m unitselection silently skips it; theunit testsjob in.github/workflows/wren-ci.ymlruns the whole tree instead and says so in a comment ("the marker is opt-in, so any unmarked file … was silently skipped"). Measured with that job's own command:core/wren, Python 3.11 venv)pytest tests/unit/ --ignore=tests/unit/test_memory.py --ignore=tests/unit/test_mcp_server.pyond26ab6aThe one failure is
test_cloud.py::test_create_refuses_without_a_git_identity_before_creating_anythingin both runs — pre-existing onmain, unrelated to this change (no git identity in a detached checkout). Delta is exactly+3, the three tests added here.Lint, with the
lintjob's own commands (uvx ruff format --check src/,uvx ruff check src/):71 files already formatted,All checks passed!.Not run, and I am not claiming it: the end-to-end repro from #2724 (
WREN_EMBEDDING_BACKEND=onnx+BAAI/bge-small-en-v1.5). It needs thememoryextra and an ONNX model download, neither available here. The CLI branch is three lines oftyper.echofed by the callback the unit tests cover; I did not add a CLI-level test because reaching_reindexrequires stubbing the backend guard and project discovery, which would assert on my own mocks rather than on the behaviour.Scope
#2724 lists two consequences. This PR fixes consequence 2 (the dropped reason), which the issue itself calls "the smaller change" and "the one that costs a user their afternoon".
Consequence 1 — retrying a permanent failure forever — is not here on purpose. It requires choosing a terminal exception set (
OnnxBackendErrorspecifically, or something broader), and the issue says that is a deliberate decision because anything wider "risks killing a watcher over a genuinely transient error, which is the failure mode the current code was written to avoid". That is a maintainer call, not a doc-level fix, and the two compose independently.I also did not touch
#2624's recorded decision that this is not an injection surface, and no claim in this PR depends on one.Duplicate check
Retrieved the changed-file list for all 73 open PRs (
GET /pulls?state=open, pages 1-2, cross-checked against theLink: rel="last"header showing 73 is the complete count, so this is not agh pr list --limittruncation). No open PR touchescore/wren/src/wren/memory/watch.py,core/wren/src/wren/memory/cli.pyorcore/wren/tests/unit/test_memory_watch.py— in fact none touches any file undercore/wren/src/wren/memory/.#2707 is referenced by #2724 as where this was split out from; that PR is merged and covers routing the error for
index/store/recall/fetch, which is the behaviourwatchis now being brought in line with.Summary by CodeRabbit