taitelee · Oct 13, 2026 · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -41,7 +41,7 @@ Twenty internal packages under `internal/` (plus `internal/testutil/` for shared
 - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output; over a `Sharded` queue, `claims.go`'s `ClaimShards` narrows the worker to the units this process is assigned — membership leases, capped rendezvous, halt-drain-then-release handover and stop, reset at takeover from a dead owner, each unit's share of a 10,000-row budget of unsettled rows). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, format, columns, row}` and nothing else — `row` is one positional `JSONCompactEachRow` line and `columns` names its slots, the table's **insertable** columns (a `MATERIALIZED`/`ALIAS` column cannot be named in an `INSERT`); the worker batches per (tenant, table, column list), the tenant read off each message's `mq.Topic`, and inserts each batch into its tenant's own ClickHouse (`chconn.Pools.Target`); the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/ops/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/ops/*`, so non-admin callers never reach the proxy) or through an operator-authored pipe that writes, gated only by its `allowed_roles` (#386). A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`)
 - **`keyenc/`** — the one escaping composite keys are built from: `Escape` keeps `[A-Za-z0-9_-]` (exactly the tenant-id grammar, so a tenant id is its own escaped form) and writes every other byte as `%XX`, `Unescape` is `url.PathUnescape` (lenient: either hex case, and a byte left unescaped reads as itself, so a `%2D` an earlier build wrote still reads), `Join`/`AppendJoin` escape each field and put a separator between them (they panic on no fields, and on a separator the escaping could write or one outside ASCII) and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to and no field reaches a key unescaped: NATS subjects (`Join`/`Split` after the verbatim tenant) and the cache's keys — the version index and the shared backend's Redis keys (`internal/cache`) — and the dedupe keys (`<tenant>/<table>/<id>`) use it; changing what it keeps orphans every stored key (an orphaned dedupe key lets a seen id through again), and on the shared backend, whose keys every process builds for itself, splits them between builds for the length of a rolling upgrade (a bump one build makes misses the entries the other filed, served until their TTL)
 - **`mq/`** — the message-queue boundary: the **only** package that imports NATS/JetStream (Key Design Decision #20), and the only one that knows how the broker works. Everything else addresses events by `Topic{Tenant, Table, Scope}` (a validated tenant id and raw names — the tenant leads every subject, `ingest.<tenant>.<table>`, so one wildcard selects a tenant's traffic, and a topic without one is refused) and states intent through the interfaces — `Publisher` (`ErrQueueFull` is the backpressure signal, `ErrUnavailable` a broker that cannot be reached — both a `503`, with `Retry-After` `30` and `5`; `WithIdempotencyKey` makes a republish inside the queue's duplicate window a no-op), `Subscriber`, `ConsumerManager`/`Consumer`/`ConsumerConfig` (the ingest worker's durable consumer), `DeadLetterer` and `DeadLetterStats` (park a message, count what is parked), `Purger` (drop what is both acked and older than a cutoff — the sweeper), `Replayer` (SSE gap-fill) — composed into `Broker`, which adds each tenant's byte budget (`SetMaxBytes`/`MaxBytes`: the `mq.max_bytes_gb` reload, which opens a tenant's queue the first time) and `Stats` (the system gauges' source). A broker whose ingest queue is split into units one consumer at a time owns implements `Sharded` too (`IngestUnits`, `ResetOrphaned`, `Unowned`; `ConsumerConfig.Units` narrows a consumer to some of them, and its consumer is a `Releaser` and a `Halter`, capped per unit by `ConsumerConfig.MaxHeld`, with `ErrConsumerMismatch` when an operator durable no longer fits), and `Message.OnSettled` runs a hook once, at the first ack or nak attempt, confirmed or not. Every interface speaks per tenant, never per stream: the embedded implementation gives each tenant a queue of its own (a stream pair, `INGEST_<tenant>`/`DLQ_<tenant>`), and nothing outside the package may assume that layout — an external implementation may keep one shared stream. Subjects, prefixes, wildcards, stream names, sequences, and ack floors are private to the implementations, whose subject tokens are escaped by the shared `internal/keyenc`: `EmbeddedNATS` (`embedded.go`, `subject.go`, `purge.go`, `deadletter.go`), which `internal/app` constructs and hands everything else as a `mq.Broker`, and `ExternalNATS` (`external.go`, `subject_nats.go`, `nats_topology.go`: an operator-owned cluster whose streams, durables and lease bucket it never creates, changes, purges or deletes; `lease.go` holds `coord.backend: nats`'s leases in that bucket), which `internal/app` constructs from the `mq.nats` block when `mq.backend` is `nats` ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). Every implementation passes the conformance suite in `internal/mq/mqtest` (`mqtest.Run`), which states the `Broker` contract as behavior; a new backend runs it from its own test, with `mqtest.Caps` only where its semantics legitimately differ
-- **`observability/`** — OpenTelemetry pipeline: `InitProvider` wires trace/metric/log providers via OTLP gRPC (each signal independently gated). A top-level `Prometheus` config block drives an optional `/metrics` scrape endpoint that runs independently of OTLP push — standalone (Alloy/Mimir scrape, no collector), alongside OTLP, or off. `NewLogger` produces a slog handler that fans out to stdout AND OTLP (stdout always 100%, OTLP sample-rate-aware). `TraceHandler` injects trace_id/span_id from active spans. `tracer.go` provides W3C trace context propagation over message headers (`InjectHeaders`/`ExtractHeaders` on a plain header map; `internal/mq` injects on every publish and extracts on the `Subscribe` path, so this package never sees a NATS type).
+- **`observability/`** — OpenTelemetry pipeline: `InitProvider` wires trace/metric/log providers via OTLP gRPC (each signal independently gated). A top-level `Prometheus` config block drives an optional `/metrics` scrape endpoint that runs independently of OTLP push — standalone (Alloy/Mimir scrape, no collector), alongside OTLP, or off. `NewLogger` produces a slog handler that fans out to stdout AND OTLP (stdout always 100%, OTLP sample-rate-aware). `TraceHandler` injects trace_id/span_id from active spans, and `request_id`/`tenant` from the request scope (`request.go`: `WithRequest`, filled by `api`'s `requestID` middleware and its tenant resolution; on log records and span attributes, never a metric label). `tracer.go` provides W3C trace context propagation over message headers (`InjectHeaders`/`ExtractHeaders` on a plain header map; `internal/mq` injects on every publish and extracts on the `Subscribe` path, so this package never sees a NATS type).
 - **`pipes/`** — Named query pipes: `NamedQuery` type + `BindParams` + `Source` (read per request; `settings.Store` in production, `Static(q...)` in tests)
 - **`policy/`** — Hasura-style access control, **role-first**: `TablePolicy` is `map[string]RolePermissions`, and a role's grant splits by operation into `SelectPermissions` (columns, row `filter`, aggregations, the `max_*` limits) and `InsertPermissions` (columns, `check`) — so a field only one side honors does not exist on the other. `Evaluate()` resolves ONE operation and leaves the other side **nil** (`Select *ResolvedSelect` / `Insert *ResolvedInsert`), which every accessor fails closed on — nil is "not resolved", distinct from an empty side, which is "unrestricted" (what the admin return builds). Claim templating (`{{ jwt.claim.path }}`) resolves during that call. Policies come from `Source`, a `func() *Policy` read per call (`settings.Store.Policy` in production, `Static(p)` in tests)
 - **`query/`** — Structured query AST types + SQL builder with schema validation, structural policy predicate/limit emission, timestamp bucketing

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## Unreleased
 
+### Added
+
+- **Every request's log records and server span say which request and which tenant they belong to** (`internal/observability/{request,logger}.go` (+ tests), `internal/api/correlation.go` (new, + tests), `internal/api/{router,tenant,dlq}.go`, `cmd/wavehouse/main.go`, a comment in `internal/auth/auth.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `{configuration,reverse-proxy}.mdx`, `AGENTS.md`): closes [#771](https://github.com/Wave-RF/WaveHouse/issues/771), part of the observability epic [#777](https://github.com/Wave-RF/WaveHouse/issues/777), and carries the request-id item of [#333](https://github.com/Wave-RF/WaveHouse/issues/333) that [#25](https://github.com/Wave-RF/WaveHouse/issues/25) asked for. chi's `RequestID` ran on every request, but nothing read the id, so no log line, span or response carried it. A request now gets an id when it arrives: the `X-Request-Id` it was sent with, when it is sent once and is 1 to 128 printable ASCII characters without spaces, otherwise a random 32-hex-character one, which unlike chi's names no host. The response carries it back in `X-Request-Id`, and `TraceHandler` stamps it as `request_id` on every record logged through the request's context. Once the tenant is resolved (`TenantMW`'s `X-Tenant-ID`, or an ops route's `?tenant=`, read ahead of authentication by `opsRequestTenant`; tenant `0`, from the same point, for any other ops route that names none, reads, raw SQL and the schema refresh alike; a settings reload that names none acts on every tenant and carries none) the records carry `tenant` as well, filled in place so the panic recoverer's record, logged through a context taken before resolution, has it too; a record that names its own tenant keeps it. The otelhttp server span gets both as attributes, `request_id` and `tenant`, so the authorization-denied `WARN`, the operator-key audit line and a request's span can be grouped by tenant and joined to a proxy's access log. Neither is a label on the HTTP request metrics. `cmd/wavehouse`'s boot logger is wrapped in `TraceHandler` too, so stdout carries these fields, and `trace_id`/`span_id`, with `otel.logs.enabled` off, where it used to lose the trace ids. `internal/api/ingest.go` is untouched, since [#712](https://github.com/Wave-RF/WaveHouse/pull/712) rewrites it: its `batch ingested` and per-event lines log through the request's context and carry both fields with no change there, and its `ingest` child span keeps only `table`, with the tenant on the server span above it.
+
 ### Changed
 
 - **`Wave-RF/cloudflare-preview` is pinned at v1.1.2** (`.github/workflows/ci.yml`, `.github/workflows/docs-preview-cleanup.yml`). The docs preview comment no longer turns an `@name` in a commit subject into a mention that notifies that person: the old `&#64;` escape was decoded by GitHub, and live testing measured a real mention. URLs in commit subjects no longer autolink. A Cloudflare authentication error is no longer retried three times. ([Wave-RF/cloudflare-preview#2](https://github.com/Wave-RF/cloudflare-preview/issues/2))

diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go
@@ -13,6 +13,7 @@ import (
 
 	"github.com/Wave-RF/WaveHouse/internal/app"
 	"github.com/Wave-RF/WaveHouse/internal/config"
+	"github.com/Wave-RF/WaveHouse/internal/observability"
 )
 
 // Pre-populated build info variables, set via ldflags by the Makefile and by
@@ -160,7 +161,10 @@ Run 'wavehouse <command> -h' for command-specific help.
 func run(ctx context.Context) int {
 	logLevel := &slog.LevelVar{}
 	logLevel.Set(logLevelFromEnv())
-	logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: logLevel}))
+	// TraceHandler, so a request's lines carry request_id and tenant (and its
+	// trace ids when tracing is on) even when the OTLP logger never replaces
+	// this one.
+	logger := slog.New(&observability.TraceHandler{Handler: slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: logLevel})})
 	slog.SetDefault(logger)
 
 	logger.Info("starting WaveHouse", "version", Version, "build_time", BuildTime, "git_commit", GitCommit)

diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md
@@ -80,6 +80,8 @@ This contract holds for:
 
 Historically some error paths defaulted to `text/plain` because they were emitted via `http.Error` or chi's default handlers; those paths now route through a shared `writeJSONError` helper so strict clients can branch on `Content-Type` consistently.
 
+Every response, success or failure, also carries `X-Request-Id`: the id the request is logged and traced under. A request that sends one `X-Request-Id` of 1 to 128 printable ASCII characters, with no spaces, gets its own back; any other gets a fresh random 32-hex-character id. Quote it when reporting a failure ([Following one request or one tenant](/deployment#following-one-request-or-one-tenant)).
+
 The per-endpoint error tables below list the bodies you can expect for each status code; the `Content-Type` and `X-Content-Type-Options` headers above apply uniformly and are not repeated.
 
 :::caution[Streaming / partial-write responses]