Repository navigation
telemetry: per-call and per-run correlation IDs via donkey.run() #195
Description
Summary
Piece 6 of 6, first half. Every request gets a per-call ID and, optionally, a
per-run ID the developer sets once.
async with donkey.run(id=ticket.id, enduser_id=user.id):
await triage_agent.run(ticket)Both are injected as headers, attached to every exception, every span, and
logged — so client-side and gateway-side records join.
Why it matters — Scenario C
Compliance asks "prove the HR bot's answer to user X on date Y went through
content-safety." The correlation ID on the Slack log line joins to the gateway
record. That is what an EU AI Act Article 12 log request looks like in practice.
Acceptance criteria
-
DonkeyError.correlation_idalways equals the header that was sent. - The per-run ID is attached to every span emitted inside the block.
- Nested
donkey.run()blocks behave predictably and are documented. - Works without OTel installed — correlation is not gated on telemetry extras.
Propagation through graph nodes
A per-run ID the developer has to thread into every node is a per-run ID that
gets dropped. Twelve LangGraph nodes call the model across one ticket, and the
audit story only works if all twelve carry the same ID.
- A run ID set at
donkey.run()reaches every LangGraph node via
contextvar, without the developer threading it through state. - Concurrent runs in the same process do not leak IDs into each other.
- Propagation survives
asynciotask boundaries created by the framework. - The conformance suite (1.5) includes "propagates correlation ID".
Cross-surface impact
- core/telemetry + core/errors + area:adapters: the change.
- Known limitation to record: frameworks where LiteLLM owns the transport
cannot propagate a per-run ID; assert the exemption rather than skipping. - Verification: the header name the gateway reads is unverified — use the
Unverified(...)placeholder with a config override.
Out of scope
Cost-attribution tags.
Activity
tommasobolis commentedon Sep 10, 2026
Agreed design (brainstormed 2026-09-10)
Two decisions settled before implementation:
enduser_iddeferred to telemetry: validated cost-attribution tags (team, project, env, enduser.id) #196 — this issue shipsfabric.run(id=…)only.fabric.run()is designed so telemetry: validated cost-attribution tags (team, project, env, enduser.id) #196 can addenduser_id/team/project/envwithout a breaking change.
The two ids
fabric.run(id=…), shared by every request in the block; falls back to the auto-bound context id (a "run of one") outside a blockX-Correlation-IdFabricError.correlation_id, spanfabric.correlation_id, log line — the client↔gateway join keyX-Fabric-Request-IdFabricError.call_id, log line — pinpoints one request within a run; exists even on a transport error with no responseFabricError.correlation_idstays = the run/correlation header (preservingtoday's semantics + the verified join), so "correlation_id always equals the
header that was sent" holds — that header is always emitted.
Public API
fabric.run(id: str | None = None)— a dual sync/async context manager(both
withandasync with), binding the run id to the existingtelemetry._correlation_idcontextvar.fabric.run_context()becomes a thinback-compat alias. Nested blocks rebind and restore on exit (contextvar token).
Verification (§0.3)
docs/verified-apis.mdrecordsx-correlation-idas VERIFIED (LIVE) — aresponse echo; whether the gateway reads an inbound correlation header is
UNVERIFIED. So both request-header names move into
core/_verify.pyasUnverified(...)placeholders (verified=False), with a genuine configoverride: two new
FabricConfigfields (correlation_header,call_id_header; envAGENT_FABRIC_CORRELATION_HEADER/AGENT_FABRIC_CALL_ID_HEADER). The transport resolves each name once and usesit both to inject and to read back in
classify().Propagation + conformance
Every LangGraph node calls the model through the one shared
FabricAsyncClient(
http_async_client), so the contextvar-bound run id already reaches everynode, and
contextvarscopy acrossasynciotask boundaries. Fill theexisting
correlation_id_propagatedconformance scenario body (one run idacross N node calls; no cross-run leakage); keep the
adk/crewaiLiteLLMexemption asserted, not skipped.
Works without OTel
Correlation is pure contextvar + headers; spans only decorate when OTel is
installed. Covered by a base-only test.
File-by-file
core/_verify.py— twoUnverifiedheader placeholders.core/config.py—correlation_header/call_id_headeroverride fields.core/telemetry.py— call-id helper alongside the run-id contextvar.core/transport.py— inject both headers (call id once, before the retryloop); resolve names from config→_verify; read both back.
core/errors.py—FabricError.call_id;classify(..., call_id=…).fabric.py—fabric.run(id=…)dual CM; keeprun_contextalias.tests/conformance/suite.py(+ a LangGraph propagation test) — fill thescenario; keep the exemption.
observability.mdx,errors.mdx, a short correlation concept,docs/verified-apis.md(correlation request-header row = UNVERIFIED withthe response-echo note), README ergonomics,
CHANGELOG.md.fabric.run()+ the two headers— recorded here, no TS code yet.
Out of scope
Cost-attribution tags (
enduser_id, team/project/env) — #196.feat(telemetry): per-run and per-call correlation IDs via fabric.run() (
tommasobolis commentedon Sep 10, 2026
Fixed by #326 (merged as ca5dfaa).
tommasobolis commentedon Sep 10, 2026
Cross-surface lockstep — TypeScript parity (Phase 5, BG §3.5).
Recording the obligation for when the TS port ships (no TS code exists yet, so nothing to change now). The TypeScript SDK must reach parity with what #326 landed on the Python side:
fabric.run(id=…)as a run-scoping construct (the TS-idiomatic equivalent of the dual sync/async context manager).X-Correlation-Idrequest header (→ spanfabric.correlation_id), and a fresh per-request call id →X-Fabric-Request-Id, stable across a request's retries.FabricError.correlation_idandFabricError.call_id, withclassify()/its TS equivalent deriving both client-sent ids from the response's request.Fold this into the Phase-5 TS work item when it's cut.
[-]telemetry: per-call and per-run correlation IDs via fabric.run()[/-][+]telemetry: per-call and per-run correlation IDs via donkey.run()[/+]