feat(observability): identify the process in telemetry

CHANGELOG.md

@@ -41,6 +41,8 @@ Upgrading from 0.1.0 is a conversion, not a binary swap: every behavioural setti
 
 ### Added
 
+- **Every span, metric series and log record names the process that sent it** (`internal/observability/identity.go` (new, + tests), `internal/observability/provider.go` (+ tests), `internal/app/wire_identity.go` (new, + test), `internal/app/{wire,app}.go`, `cmd/wavehouse/main.go`, `internal/config/config.go`, `internal/testutil/otlp.go`, `tests/integration/otel_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`; closes [#772](https://github.com/Wave-RF/WaveHouse/issues/772)): the OpenTelemetry resource carried only `service.name`, so series and spans pushed by two replicas of one role were indistinguishable unless the collector added an instance, and which build each process ran was visible only on `/version`. The resource now also carries `service.version`, `service.instance.id` (the boot config's `instance_id`), `wavehouse.roles` (sorted), `wavehouse.git_commit` and `wavehouse.build_time`, and the standard `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` variables override or extend it (a malformed `OTEL_RESOURCE_ATTRIBUTES` is logged, never fatal). A new `wavehouse_build_info` gauge, always `1`, is labeled `version`, `git_commit`, `build_time` and `roles`, so a mixed-version rollout shows on a dashboard. Every log line after the boot config loads carries `instance_id`, stdout included, so the `process roles` boot line no longer repeats it.
+
 - **`/readyz` consults the external queue, not just ClickHouse** (`internal/api/health.go` (+ tests), `internal/app/{app,wire,wire_ops,wire_nats}.go` (+ tests), `internal/mq/external.go` (+ test), `internal/mq/natstest/natstest.go`, `tests/integration/boot_resilience_test.go`, `docs/src/content/docs/{api,deployment,architecture}.md`, `reverse-proxy.mdx`, `AGENTS.md`): closes [#775](https://github.com/Wave-RF/WaveHouse/issues/775), the dependency checks [#144](https://github.com/Wave-RF/WaveHouse/issues/144) asked for and [#213](https://github.com/Wave-RF/WaveHouse/pull/213) deferred. The readiness handler runs a list of named `Check`s instead of one ClickHouse ping, every one on every call, and its `503` body's `error` names each that failed, one per line: `clickhouse` (`chconn.Pools.Ping`, as before, now under that prefix) and `mq` under `mq.backend: nats` (`ExternalNATS.Ready`: not connected, or the topology failed its last check — the states of the `wavehouse_mq_connected` and `wavehouse_mq_topology_ok` gauges; the lease bucket of `coord.backend: nats` rides that connection and is part of that check, so it is covered; the check also runs again as soon as the client reconnects, so a cluster that came back without a stream fails readiness without waiting for the periodic recheck). A process whose NATS connection has closed for good is now taken out of rotation instead of answering `503` on every request, and a process without the `api` role consults the queue too, where it used to be ready once booted without pools. Deliberately not checks: the shared cache (a tripped breaker falls back to ClickHouse) and the dedupe stores, Pebble and DynamoDB alike (a store that is not answering fails that tenant's ingest closed per request, and is only needed to dedupe, so it should not take the whole process out of rotation; parking those events on NATS until it answers is [#800](https://github.com/Wave-RF/WaveHouse/issues/800)). The probe bodies keep their diagnostic detail, the failing pool's ClickHouse address included: the bare probe paths are documented as internal-only, and the routes meant for a reverse proxy (`/v1/health`) carry none.
 - **Each tenant's streams are kept alive at its own `stream.keepalive_interval`** (`internal/app/keepalives.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/stream.go` (+ tests), comments in `internal/stream/{heartbeat,hub}.go`, `internal/chconn/chconn.go`, `internal/config/config.go` and `internal/settings/settings.go`, `docs/src/content/docs/{architecture,deployment}.md`, `{settings-directory,configuration,reverse-proxy}.mdx`, `AGENTS.md`): a story 2 follow-up of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), closing [#597](https://github.com/Wave-RF/WaveHouse/issues/597). `stream.keepalive_interval` and `stream.keepalive_buckets` are per tenant like every other key of a folder's `config.json`, but one keepalive wheel nudged every tenant's `GET /v1/stream` connections, at the shortest interval among the tenants being served: a tenant asking for one second set every tenant's cadence, and a reload of one tenant's settings moved another's. Each served tenant now has a wheel of its own, in the shape of its own pair. A connection is nudged at its tenant's interval whatever the others ask for; a reload of a tenant's pair rebuilds that tenant's ring in place under its open streams, as before, and leaves every other tenant's wheel untouched; and a tenant a reload removes or rejects has its wheel stopped as its streams are ended, and starts over on a fresh one when it is served again. A wheel exists from the reload that adopts its tenant, whether or not a stream is open: idle, it is one parked goroutine, about 5 KB, and a wake of about 3 µs every `keepalive_interval ÷ keepalive_buckets` (ten seconds at the seed's `30` and `3`), so a thousand idle tenants cost some 5 MB and 0.03% of one core, less than starting a wheel with a tenant's first stream and stopping it after its last would be worth. A flat directory's one tenant is kept alive exactly as before. Since the pair is a tenant's own, `stream.keepalive_buckets` is now bounded at `100` (`settings.MaxKeepaliveBuckets`; it had no upper bound): a wheel ticks once per bucket per interval and holds a set per bucket, so the cap bounds each tenant's tick rate and ring, and a hundred buckets already spreads the writes to 1% of its streams per tick; a `config.json` asking for more is refused at boot and on reload like any other out-of-range key.
 - **Codegen reads any tenant of a nested settings directory** (`clients/ts/src/cli/codegen.ts` (+ tests), `clients/ts/README.md`, `docs/src/content/docs/sdk/reference.md`): a story 6 follow-up of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), closing [#640](https://github.com/Wave-RF/WaveHouse/issues/640). `wavehouse-codegen` takes `--tenant <id>` (`-t`), sent as the `?tenant=` the `/v1/ops/schema` route reads, and `--operator-key <key>` (`-k`), sent as `X-Operator-Key` — the one credential a nested directory's `/v1/ops/*` routes admit, which `--auth`'s bearer token is not. The flags are independent: `--auth` stays the flat-directory credential, with or without `--tenant`, and the operator key without `--tenant` reads tenant `0` on either shape. Every flag that takes a value, `--url`, `--out` and `--auth` included, is refused client-side when its value is missing, empty, or another flag (what an unset shell variable leaves behind), so the next flag is never sent as the tenant or the credential; a malformed tenant id is the server's `400`. A request carrying either credential does not follow a redirect, where across origins `fetch` would drop the bearer token and re-send `X-Operator-Key` to the redirect's target: codegen stops and names the target instead.

cmd/wavehouse/main.go

@@ -178,6 +178,10 @@ func run(ctx context.Context) int {
 		logger.Error("load config", "error", err)
 		return 1
 	}
+	// Every later line names the process; the OTLP-aware logger app.New may
+	// swap in carries it too (observability.Identity.LogAttrs).
+	logger = logger.With("instance_id", cfg.InstanceID)
+	slog.SetDefault(logger)
 
 	// data_dir, when a selected backend keeps state there, must be writable
 	// before anything dials out, so the refusal (and, for the typical cause —

config.yaml

@@ -13,8 +13,8 @@ data_dir: ./data
 # coord.backend: nats (there is no sweeper under nats); separate api and ingest processes also need
 # cache.backend: redis. Boot refuses a split the backends cannot serve.
 roles: [api, ingest, sweeper]
-# Names this process: logged at boot, and a lease's holder under
-# coord.backend: nats. Empty means <hostname>-<8 hex>, fresh at every boot.
+# Names this process: on every log line after boot config loads, as service.instance.id in its
+# telemetry, and a lease's holder under coord.backend: nats. Empty means <hostname>-<8 hex>, fresh at every boot.
 instance_id: ""
 
 server:

docs/src/content/docs/architecture.md

@@ -133,7 +133,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/
 - **backends.go** — the `<layer>.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. One rule spans two layers: while `mq.backend` is `embedded`, `dedupe.lease` plus its own ceiling to the next whole second (`ceilSecond`) plus one more second must fit the embedded MQ's 2m duplicate window (`embeddedDuplicateWindow`), a cap of 59s (`maxEmbeddedLease`), because a client obeying the in-flight `503`'s `Retry-After` can republish as late as the lease plus that ceiling plus a second after the claim, since DynamoDB rounds a claim's expiry up to the second. A backend's own settings go in a `<layer>.<backend>` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are harmless or correct for one replica only (a shared MQ over a local cache or Pebble dedupe; under `nats`, `mq.max_bytes_gb` not applied; an `mq.nats` or `coord.nats` block its layer's backend ignores), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. `mq.backend` has two values, `embedded` and `nats` (`MQNATS`), and `nats` reads the `mq.nats` sub-block, which `MQ.validate` checks only when it is selected. `coord.backend` takes `nats` (`CoordNATS`) too, whose `coord.nats` block holds only the bucket name: the leases ride `mq.nats`'s connection.
 - **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address (exactly one in `standalone` mode, which dials only the first), each `host:port` with a port from 1 to 65535 (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `timeout` and `dial_timeout` of at most 1 s each (boot and `Close` each wait out a dial: a connect and a handshake bounded by `dial_timeout`, and a cluster's topology read bounded by the larger of the two), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection.
 - **mq_nats.go** — the two nats blocks and their rules, apart from backends.go for the e2e coverage gate as `cache_redis.go` is: `MQNATSConfig` (URLs, file-path-only credentials, TLS, and the topology to expect; its defaults, `defaultMQNATS`, are in `defaults()`) and its checks, `CoordNATSConfig` (the bucket name), the nats lines of `Warnings` (`natsWarnings`) and the two cross-layer rules `validateTopology` delegates (`validateNATSTopology`: `coord.backend=nats` needs `mq.backend=nats`, a process running `ingest` on a shared queue needs `coord.backend=nats`, and a process running only `sweeper` there is refused).
-- **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`<hostname>-<8 hex>` when empty, resolved in `Load`; logged at boot, and the holder a NATS lease's value names). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, `coord.backend=nats` without `mq.backend=nats` (the leases ride its connection), `mq.backend=nats` with `coord.backend=local` in a process running `ingest` (the ingest processes share the queue through leases), a process running only `sweeper` under `mq.backend=nats` (it would run nothing), and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and the cache and dedupe warnings are skipped without `api`, since only that role opens a cache it reads or a dedupe store.
+- **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`<hostname>-<8 hex>` when empty, resolved in `Load`; a field on every log line once loaded, `service.instance.id` in the telemetry, and the holder a NATS lease's value names). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, `coord.backend=nats` without `mq.backend=nats` (the leases ride its connection), `mq.backend=nats` with `coord.backend=local` in a process running `ingest` (the ingest processes share the queue through leases), a process running only `sweeper` under `mq.backend=nats` (it would run nothing), and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and the cache and dedupe warnings are skipped without `api`, since only that role opens a cache it reads or a dedupe store.
 - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory.
 - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses.
 
@@ -188,7 +188,8 @@ The **only** package that imports NATS/JetStream — a `depguard` rule in `.gola
 
 ### `observability/` — OpenTelemetry Pipeline
 
-- **provider.go** — `InitProvider(ctx, serviceName, ProviderConfig)` wires the OTel pipeline. Each output is independently gated; the W3C TraceContext + Baggage propagator is always installed (cheap, harmless when traces are off). Returns `(shutdown, promHandler http.Handler, err)` — `promHandler` is non-nil only when `PrometheusEnabled` is true and reads from a *private* `prometheus.Registry` to avoid leaking the process/Go collectors that `prometheus.DefaultRegisterer` auto-registers. OTLP-metrics push (`MetricsEnabled`) and Prometheus exposition (`PrometheusEnabled`) are independent: either, both, or neither may be set, and any combination produces a single MeterProvider feeding the active readers. The OTLP exporters (traces / metrics-OTLP / logs) take their endpoint from the `OTEL_EXPORTER_OTLP_*` env vars, not from `ProviderConfig`, and are only constructed when their signal is enabled, so Prometheus-only operation dials nothing even with those vars set. Provider init in `internal/app` runs whenever `otel.enabled` OR `prometheus.enabled` is true, so Prometheus-only operation (Alloy/scrape, no collector) is a first-class mode.
+- **provider.go** — `InitProvider(ctx, Identity, ProviderConfig)` wires the OTel pipeline, every signal over one resource built from the `Identity`. Each output is independently gated; the W3C TraceContext + Baggage propagator is always installed (cheap, harmless when traces are off). Returns `(shutdown, promHandler http.Handler, err)` — `promHandler` is non-nil only when `PrometheusEnabled` is true and reads from a *private* `prometheus.Registry` to avoid leaking the process/Go collectors that `prometheus.DefaultRegisterer` auto-registers. OTLP-metrics push (`MetricsEnabled`) and Prometheus exposition (`PrometheusEnabled`) are independent: either, both, or neither may be set, and any combination produces a single MeterProvider feeding the active readers. The OTLP exporters (traces / metrics-OTLP / logs) take their endpoint from the `OTEL_EXPORTER_OTLP_*` env vars, not from `ProviderConfig`, and are only constructed when their signal is enabled, so Prometheus-only operation dials nothing even with those vars set. Provider init in `internal/app` runs whenever `otel.enabled` OR `prometheus.enabled` is true, so Prometheus-only operation (Alloy/scrape, no collector) is a first-class mode.
+- **identity.go** — `Identity`, the process's name in its telemetry ([#772](https://github.com/Wave-RF/WaveHouse/issues/772)): service name, version, commit, build time, `instance_id` and roles, built by `internal/app`'s `identity` (`wire_identity.go`). `newResource` turns it into the resource (`service.name`, `service.version`, `service.instance.id`, `wavehouse.roles` (sorted by `identity`), `wavehouse.git_commit`, `wavehouse.build_time`, each left out when empty), then merges `resource.WithFromEnv`, so `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` override it key by key; a malformed `OTEL_RESOURCE_ATTRIBUTES` is logged, not fatal. `registerBuildInfo` exports `wavehouse_build_info` (always `1`; `version`, `git_commit`, `build_time`, `roles`) on the meter provider, with no instance label, since the resource carries it. `LogAttrs` are the build fields and `instance_id` the OTLP-aware logger carries on every record; `cmd/wavehouse` puts `instance_id` on the boot logger once the config is loaded.
 - **logger.go** — `NewLogger(component, level, isJSON, otlpSampleRate)` produces a slog logger that fans out to stdout (always 100%) and the OTLP log exporter (DEBUG/INFO sampled at `otlpSampleRate`, WARN/ERROR always 100% as a non-configurable safety floor). `TraceHandler` injects `trace_id`/`span_id` from the active span when one exists, and `request_id` and `tenant` from the request scope; `cmd/wavehouse` wraps its boot logger in it too, so stdout carries them whether or not OTLP logs are on. `otlpSamplerFn` is exposed (lowercase) for unit testing the per-level rate logic without driving through the slogmulti middleware.
 - **request.go** — The request scope: `WithRequest(ctx, id)` puts it in the context, `SetRequestTenant` fills in the tenant once it is resolved (in place, so a record logged through a context taken earlier, the panic recoverer's, carries it too) and sets it on the span in the context, and `AnnotateSpan` copies both onto the server span. `request_id` and `tenant` are the keys on records and spans alike; a record that names its own tenant keeps it. They are never metric labels.
 - **metrics.go** — `RegisterSystemMetrics(mqStats, pebbleStats)` registers observable gauges for the MQ's NATS connections and in-msgs (the embedded server's, or under `nats` this process's client connection), and Pebble dedupe storage stats. Both are functions read on every scrape — `mqStats` a `func() (MQStats, error)` (`mq.Broker.Stats` in production; nil skips the MQ gauges), `pebbleStats` a `func() map[string]int64` (`dedupe.Embedded.Stats`, the one instance's figures; nil, or a nil map while it is closed, skips the Pebble gauges) — this package never holds the NATS server or a store. Wired in `internal/app` after the providers are up.