Skip to content

Fix tracing for thrown JSRPC method errors - #7644

Closed
danlapid wants to merge 14 commits into
mainfrom
workerd-experiment/dlapid/jsrpc-thrown-error-logging-fix
Closed

danlapid wants to merge 14 commits into
mainfrom
workerd-experiment/dlapid/jsrpc-thrown-error-logging-fix

Conversation

@danlapid

@danlapid danlapid commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Record delivered JSRPC method exceptions on the callee invocation while preserving cancellation semantics for calls that never deliver an exception.

EdgeWorker integration MR: https://gitlab.cfdata.org/cloudflare/ew/edgeworker/-/merge_requests/15291

Final workerd SHA: 731a02d8aa4a988ada35b5fb9e8ea0f04732b038
Rebased EdgeWorker SHA: 7ff2d455a3d56bae485845b68344e0a5eca86bfa

Root cause

When a JSRPC method threw and its exception was delivered to the caller, the callee invocation still followed the broader session lifetime. Graceful session teardown could therefore finalize the callee trace as canceled without recording the application exception, even though the caller received and caught that exception.

Exception reporting also needed to remain scoped to a live invocation. A rejection arriving after session cancellation must not report a request failure or mutate the canceled invocation.

Changes

  • Add a delivered-exception observer shared by request metrics and tracing.
  • Mark a delivered JSRPC exception, report the request failure exactly once, and add the exception to the callee invocation trace.
  • Clear observer liveness during session teardown so late rejections are ignored.
  • Keep trace extraction best-effort so a tracing failure cannot replace the exception being delivered.
  • Add direct C++ controls for exact-once failure reporting and post-cancellation late rejection.
  • Add streaming-tail tests for synchronous and asynchronous thrown method errors.
  • Add buffered and streaming production-harness coverage in the linked EdgeWorker MR.
  • Preserve genuine pending-call cancellation as canceled with no delivered application exception.

Behavior

Before:

  • The caller received and caught the remote method error.
  • The callee trace could finish with outcome = canceled and no exception.

After:

  • Caller-visible behavior is unchanged.
  • The callee invocation records the delivered error and finishes with outcome = exception.
  • The intentional error is reported exactly once.
  • A pending method canceled before delivering an exception remains canceled and does not report a failure.

Tests

The final GitHub checks cover:

  • //src/workerd/api:jsrpc-call-plan-test@
  • //src/workerd/api/tests:jsrpc-error-trace-test@
  • generated all-autogates and all-compat-flags variants
  • Linux, Linux ARM release/debug, ASan, TSan, macOS, Windows, coverage, lint, and snapshot jobs

The EdgeWorker MR adds production-harness buffered and streaming coverage through //src/edgeworker/api-tests:js-rpc-exception-logging.ew-test@ and its generated variants.

Out of scope

Raw EdgeWorker streaming traces contain one entrypoint jsRpcCall span and one method call. Any duplicate rpcMethods or rpcCallCount produced later by WOBS aggregation is a separate downstream issue and is not changed here.

@danlapid
danlapid requested review from a team as code owners October 6, 2026 21:36
@ask-bonk

ask-bonk Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Since last review: 0 resolved, 0 still open, 0 new.
LGTM!

Not re-run: api-compat, tests, docs, compat-flags, design-simplicity, jsg-gc, kj-style, memory-safety (no author changes in their files since the last review)


Reviewed commit: 731a02d8 · github run

Comment thread src/workerd/api/jsrpc-call-plan-test.c++
Comment thread src/workerd/api/tests/jsrpc-error-trace-test.js Outdated

danlapid commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in ca861d7: NoopDeliveredJsRpcExceptionObserver and EntrypointDeliveredJsRpcExceptionObserver were consolidated into one final DeliveredJsRpcExceptionObserver with optional metrics/tracer/invocation context. The default transient instance still marks delivery; the entrypoint instance additionally performs session-owned failure/trace reporting.

Comment thread src/workerd/api/tests/jsrpc-error-trace-test-tail.js Outdated
@jasnell

jasnell commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

@/tmp/opencode/workerd-pr-body.txt

What's up with the PR description?

@codecov-commenter

codecov-commenter commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 39.60396% with 183 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.46%. Comparing base (eb940ea) to head (731a02d).

Files with missing lines Patch % Lines
...c/workerd/api/tests/jsrpc-error-trace-test-tail.js 0.00% 127 Missing ⚠️
src/workerd/api/tests/jsrpc-error-trace-test.js 0.00% 36 Missing ⚠️
src/workerd/api/worker-rpc.c++ 85.22% 4 Missing and 9 partials ⚠️
src/workerd/api/jsrpc-call-plan-test.c++ 85.10% 0 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7644      +/-   ##
==========================================
- Coverage   38.46%   38.46%   -0.01%     
==========================================
  Files         864      866       +2     
  Lines      267594   267874     +280     
  Branches    25265    25286      +21     
==========================================
+ Hits       102928   103030     +102     
- Misses     150539   150702     +163     
- Partials    14127    14142      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread src/workerd/api/worker-rpc.c++
@danlapid
danlapid force-pushed the workerd-experiment/dlapid/jsrpc-thrown-error-logging-fix branch 2 times, most recently from 883f94c to cbbc329 Compare October 7, 2026 00:33
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

Record delivered callee-side JSRPC method exceptions when the method body throws or its returned promise rejects. Keep session teardown cancellation distinct and cover both paths with a streaming-tail regression test.
Bind delivered method failures to the originating session observer and tracer, release observation state on canceled calls, and strengthen cancellation and synchronous-error coverage.
Use the direct no-call session test to assert no failure reporting, and keep the streaming test focused on a delivered pending method canceled during teardown.
Keep the streaming regression focused on delivered exceptions while asserting that pure workerd capability release remains successful and exception-free.
Use separate service bindings and assert one callee invocation per method so sync, async, and pending-session traces cannot contaminate each other.
Directly verify exact-once failure reporting and post-cancellation rejection suppression, consolidate delivered-exception observation, and remove the release-only hanging trace fixture.
Exercise exact-once failure reporting and late-rejection suppression directly, consolidate exception observers, and remove the release-only hanging trace fixture.
Assert synchronous and asynchronous method failures on their observed separate callee invocations.
Keep exact-once assertions for the intentional sync and async failures while tolerating runtime-generated exceptions on the invocation.
Account for the runtime cancellation event explicitly and document why trace extraction failures do not replace delivered exceptions.
Accept the exact sync exception sequence with or without the known cleanup cancellation while rejecting every other exception shape.
Clone the current shared tracer claim for delivered-exception reporting after mapAddRef was removed.
Select Maybe's mutable map overload so the shared tracer reference can be cloned.
@danlapid
danlapid force-pushed the workerd-experiment/dlapid/jsrpc-thrown-error-logging-fix branch from dc777bf to 731a02d Compare October 7, 2026 01:29
: [
{
name: 'Error',
message: HUNG_REQUEST_MESSAGE,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the bug we want to get rid of and this asserts that it's expected behaviour (...sometimes).

I'm not at all convinced this is directionally the right fix. I think we need to add a Rejected state in worker-rpc.h: kj::OneOf<Pending, Resolved, Disposed> state -> kj::OneOf<Pending, Resolved, Disposed, Rejected> state where Rejected drops the PendingPipeline reference that Pending is currently holding onto even after receiving a rejection. That is what's keeping the RPC session and the callee's request context alive leading to the hang detector kicking in on the callee.

@danlapid danlapid closed this Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants