Repository navigation
Observability: OTel spans from the orchestrator + failure-mode clustering across threads #280
Description
Inspired by LangSmith Insights and the OTel GenAI conventions.
Solenta is the one place a cross-provider trace tree exists (it sees claude/codex/kimi from the outside). Emit OpenTelemetry GenAI spans per thread/fork/tool call (session.id = thread, gen_ai.agent.id = provider+profile), exportable over OTLP, and ingest Claude Code's native OTel metrics directly. On top of the event log, run a periodic automation that clusters errored/stalled/retried threads into ranked recurring failure modes with links to the offenders.
Activity
currentbits commentedon Aug 17, 2026
Landed on
coder/observability-otel-spans-from-the-orches-63d5e2.Spans.
electron/otel.js— OTLP/HTTP with a JSON body, no@opentelemetry/*dependency: OTLP/JSON is a stable documented encoding, so the exporter is a batchedfetchof a plain object (~270 lines, batching mirrorssession-record.js). Ids are derived, never stored —traceIdfrom the ROOT thread (walkinghandoffFrom), so an orchestrator and its whole forked crew share one trace, and a restart mid-thread keeps it. Attributes follow the GenAI conventions:session.id= thread,gen_ai.agent.id= provider+profile,gen_ai.provider.name,gen_ai.request.model,gen_ai.usage.*, plussolenta.*for cost/project/orch-worker.Emitted from
electron/runner.jsat the choke points that already existed, so all five provider paths are covered by one call site each:startRunopens,notifyRunTerminalcloses with status+usage, and tool spans are bracketed byappendMessage(the only place every adapter mints a tool message, hence the only place a tool start is visible) and closed at the four adapter result sites. The orchestrator fork gets its own span parenting the worker's run — that's the cross-provider tree.Claude Code metrics. Redirected rather than received:
otel.claudeEnv()returnsCLAUDE_CODE_ENABLE_TELEMETRY=1+OTEL_EXPORTER_OTLP_*pointed at the same collector, spread into the spawn via a newclaude.jsenvExtra(same shapecodex.jsalready had). Since it's env-only it joins the warm-CLI reuse key, otherwise a process spawned before the toggle would mask it. This is ~10 lines instead of standing up an in-process OTLP receiver; if you want the metrics displayed in-app rather than in your collector, that's the follow-up.Failure modes.
electron/failuremodes.jsgroups errored/stalled/retried threads by a normalized signature (paths, ids, numbers, quoted strings, timestamps redacted; HTTP statuses kept distinct, exit codes collapsed), filters one-offs, ranks by count then recency. Surfaced atinsights:failureModesand in a new Insights view with click-through to each offender.Deliberate deviation: the issue asks for a periodic automation. This computes on demand instead — it's deterministic and runs in milliseconds over the whole store, so a scheduler would add stored state and a retention policy for no extra information. The gap that buys: you only see a mode when you open the view, rather than being told proactively that something has now failed six times.
automations.jsalready has the minute-ticker if that proactive notification is worth it — say so and it's a small follow-up.Everything is inert while
settings.otel.endpointis null: nothing buffered, no request made, no exception path into a run.Tests (+38 new):
otel.test.js8 (exporter against injected deps),failuremodes.test.js10,otel-runner.test.js4 (a real run reaching a stubbed collector, tool span parented to the run span in one trace, ERROR span carrying stderr, nothing posted with no endpoint, telemetry env arriving in the spawned CLI),insightsView.test.tsx+settingsModal.test.tsx+otelHeaders.test.ts16.Full suite: core 38/38, renderer 789/789, memory 122/122, electron 913/914 — the one failure is the pre-existing
wireClient × webBridge > reconnectbaseline that also fails onmain.Merge origin/main into the observability branch (#280)