verify: whether the gateway echoes the client x-correlation-id or mints its own #300

Activity

1. amirkhan-ak-sf

   added this to the Verification milestone on Sep 9, 2026
2. amirkhan-ak-sf

   added

   blocked-on-verificationCannot be closed by writing code (build plan §0.3)Cannot be closed by writing code (build plan §0.3)

   size/s~half day, one file~half day, one file

   area:telemetryOTel GenAI spans, correlation IDs, cost tags (1.6, 1.7)OTel GenAI spans, correlation IDs, cost tags (1.6, 1.7)

   priority/p0Critical: blocks the phaseCritical: blocks the phase

   on Sep 9, 2026
3. amirkhan-ak-sf commented on Sep 11, 2026

   

   amirkhan-ak-sf

   on Sep 11, 2026

   CollaboratorAuthor

   More actions

   Partial evidence from a live probe, plus a concrete way to finish the verification.

   A live 200 through a governed proxy returned x-correlation-id: 4733fc2e-… (a UUID)
   on a request sent from Postman that carried no correlation header. All three
   committed captures in python/tests/fixtures/anypoint/llm_proxy/ likewise show a
   x-correlation-id UUID on the response, and policy_list.openai-sdk.json lists
   x-correlation-id in the API's CORS exposedHeaders.

   So: the gateway mints one when the client sends none, and x-correlation-id is
   confirmed to be the name it speaks on the response. That does not answer the question in
   this issue, because a minted value is indistinguishable from an echoed one when nothing
   was sent.

   The remaining test is cheap and needs no SDK: send X-Correlation-Id: test-12345 and
   inspect the response.

   • response carries x-correlation-id: test-12345 → the gateway echoes, and
     _verify.CORRELATION_ID_HEADER can be flipped to VERIFIED
   • response carries a fresh UUID → the gateway always mints, the client-sent header is
     ignored, and this becomes an upstream gap rather than a config default

   Worth doing under the same probe as #352/#354 since it is the same request. Related:
   the UnverifiedValueWarning pair this currently produces is noisy enough that it
   appears on every demo run before any output.
4. amirkhan-ak-sf commented on Sep 11, 2026

   

   amirkhan-ak-sf

   on Sep 11, 2026

   CollaboratorAuthor

   More actions

   Two of the premises here have gone stale since filing, and there is a piece of
   evidence in the committed capture that narrows the question. Recording both so
   whoever picks this up does not re-investigate.

   Stale: the header name is already behind Unverified

   The description says:

   core/transport.py declares CORRELATION_HEADER = "X-Correlation-Id" as a
   hard-coded string literal — it is not behind a _verify.Unverified(...)
   guard, so it emits no UnverifiedValueWarning and has no row in the ledger.

   That is no longer the case. It is now _verify.CORRELATION_ID_HEADER, an
   Unverified(...) with a docs/verified-apis.md §3 ref, alongside a second one
   for the per-call id:

   https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/python/src/donkey_kit/core/_verify.py#L84-L101

   It warns on read, resolved once per client in _resolve_header_names(), and
   both names have §3 ledger rows (the two "#195" rows). The warning is live — a
   default Donkey.from_env() run prints both:

   UnverifiedValueWarning: Using UNVERIFIED placeholder for 'correlation.request_header'
     ('X-Correlation-Id'). This has NOT been confirmed against a real Anypoint sandbox …
   UnverifiedValueWarning: Using UNVERIFIED placeholder for 'correlation.call_id_header'
     ('X-Donkey-Request-Id'). …
   

   So the fourth Done when box — "if the name is not confirmed,
   CORRELATION_HEADER is moved behind _verify.Unverified(...)" — is already
   satisfied and can be ticked independently of the live round trip. #195 has also
   since closed, so "should not ship before it" is moot; what remains is confirming
   the semantics for a feature already shipped on a placeholder.

   Still accurate: nothing adopts an inbound correlation id. That is #308.

   Evidence: the captured value is not in the client's format

   new_correlation_id() returns uuid.uuid4().hex — 32 hex chars, no dashes.
   The captured gateway response carries a dash-formatted UUID:

   https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/python/tests/fixtures/anypoint/llm_proxy/responses.success.headers.txt#L30

   x-correlation-id: d34a1c5a-a3af-4bcc-8fe1-217682d7d0e0
   

   A gateway echoing a client-sent value verbatim would return the dashless form.
   This is consistent with Envoy minting its own id — but it is not conclusive,
   because we do not know whether that particular probe sent a correlation header
   at all. If it did not, Envoy generating one is the expected behaviour and tells
   us nothing about the echo case. Worth checking against however the capture was
   taken before treating it as an answer.

   Repro recipe for the live round trip

   The first Done when box needs only this, and it does not need a special
   harness — with_raw_response exposes both directions of one call:

   import asyncio
   from donkey_kit import Donkey
   
   async def main() -> None:
       async with Donkey.from_env() as donkey:
           client = donkey.openai()
           async with donkey.run(id="echo-probe-0001") as run_id:
               raw = await client.responses.with_raw_response.create(
                   model="gpt-4o",
                   input="ping",
               )
           sent = raw.http_request.headers.get("X-Correlation-Id")
           echoed = raw.headers.get("x-correlation-id")
           print("bound   :", run_id)
           print("sent    :", sent)
           print("echoed  :", echoed)
           print("ECHOED  :", sent == echoed)
   
   asyncio.run(main())

   Three outcomes and what each means:

   Result	Conclusion
   echoed == sent	the gateway reads X-Correlation-Id; §3 row flips to VERIFIED and verified=True
   echoed is a fresh UUID each run	the gateway mints its own; the client-sent id correlates nothing gateway-side, and BG §1.7's "client-side and gateway-side records join" needs rewording
   echoed absent	the echo itself is policy-dependent, not unconditional as the §3 row implies

   Worth running the same probe with a deliberately wrong name
   (DONKEY_CORRELATION_HEADER=X-Nonsense-Id) in the same session. If the response
   x-correlation-id is a fresh UUID in that case but matches in the default case,
   that isolates the read to the specific header name rather than to any inbound
   header.

   Knock-on already filed

   Separately from the verification: the config override that exists because this
   is unverified currently breaks DonkeyError.correlation_id read-back, since
   _sent_ids() looks up the placeholder name rather than the configured one.
   Filed as #363 — independent of the outcome here, but it means the documented
   workaround for this issue is not currently safe to recommend.
5. tommasobolis

   added a commit that references this issue on Sep 15, 2026

   feat(skeleton): define the last-call record once at donkey.last_call (#…

   ...

   f367c37
6. tommasobolis

   self-assigned this

   on Oct 8, 2026
7. tommasobolis

   added a commit that references this issue on Oct 8, 2026

   docs(verify): record x-correlation-id echo semantics on §3 (#1037)

   ...

   This issue will close once commit 5008e34 is merged into the 'develop' branch.

   5008e34
8. tommasobolis

   closed this as completedin #1037on Oct 8, 2026
9. tommasobolis

   added a commit that references this issue on Oct 8, 2026

   docs(verify): record x-correlation-id echo semantics on §3 (#1037)

   ...

   This issue will close once commit 43607ea is merged into the 'develop' branch.

   43607ea