fix: create root span per prompt for OTEL trace isolation - #36179
fix: create root span per prompt for OTEL trace isolation#36179josephwangrb wants to merge 3 commits into
Conversation
When OTEL_EXPORTER_OTLP_ENDPOINT is set, all spans across all prompts
in a session inherit the server's boot-time trace context, producing
one giant trace per session instead of one trace per prompt.
This wraps SessionPrompt.loop in Effect.withSpan with { root: true }
so each prompt starts a new trace. All child spans (SessionProcessor.process,
LLM.run, ai.streamText, tool execution) then inherit the new trace ID,
making traces queryable per-prompt in backends like Jaeger/Tempo.
The session.id attribute is added to the root span for correlation.
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
The following comment was made by an LLM, it may be inaccurate: Found a potentially related PR: Related PR:
The other result (#5245 — "feat: integrate OpenTelemetry") appears to be an older PR and may not be directly relevant to the current span isolation fix. |
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
Per the OpenTelemetry GenAI semantic conventions, the root span for an agent invocation should be named 'invoke_agent' with gen_ai.operation.name and gen_ai.conversation.id attributes. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
Ported from upstream anomalyco#36179.
Ported from upstream anomalyco#36179.
) * fix(opencode): create root span per prompt for OTEL trace isolation Ported from upstream anomalyco#36179. * fix(opencode): attach invoke_agent root span to forked work Addresses Opus review of PR #31: the root span wrapped the ensureRunning caller, but ensureRunning may store work and start it later from a prior run's fiber, so queued/steered prompts got child spans mis-parented under the previous prompt's trace and the span measured queue-wait instead of the agent turn. Wrap runLoop (the forked work) instead, and keep the named SessionPrompt.loop fn.
|
Could I get a review on this bug fix? It is unblocking the observability for OpenCode, as the current trace structure isn't usable. |
|
I think most effort is going into v2 for otel, check out this PR and other commits. in v2 branch
|
|
@jesse-schein this is pretty cool! Can't wait to see it merged! |
Issue for this PR
Closes #32920
Type of change
What does this PR do?
When
OTEL_EXPORTER_OTLP_ENDPOINTis set, all spans across all prompts in a session inherit the server's boot-time trace context, producing one giant trace per session instead of one trace per prompt. This also causes the<trace-without-root-span>issue reported in #32920 — there's no root span boundary per prompt, so backends like Jaeger can't group spans into meaningful traces.Root cause:
SessionPrompt.loopwas defined asEffect.fn("SessionPrompt.loop"), which creates a child span of the ambient context — not a new root span. The Effect runtime'sAsyncLocalStorageContextManagerpropagates the trace context from server boot, so everyEffect.fn()call inherits the same trace ID.Fix: Wrap
SessionPrompt.loopinEffect.withSpan("SessionPrompt.loop", { root: true })so each prompt starts a new trace. All child spans (SessionProcessor.process,LLM.run,ai.streamText, tool execution) then inherit the new trace ID.The
{ root: true }option tells Effect's tracer to create a new root span (new trace ID) instead of parenting to the ambient context.How did you verify your code works?
Built opencode with the fix and tested via
opencode serve+ SDK with 2 prompts (4 turns total) against a local OTel collector:Before — 1 trace for entire session (1429 spans, 30.4s), all turns in one trace.
After — 1 trace per prompt:
195697cf...(488 spans) with 2SessionProcessor.processturns andai.streamText.doStreamcarryinggen_ai.request.model=glm-5.2, token usage, and finish reason82b2b0cd...(450 spans) with 2 turns, same GenAI attributesEach
SessionPrompt.loopspan hasparent=(none)confirming it's a root span. Typecheck passes (bun turbo typecheck— 30/30 packages).Checklist