Skip to content

feat(typelayer)!: validate, coerce and filter through chtypes - #712

Draft
EricAndrechek wants to merge 145 commits into
mainfrom
chtypes-v2
Draft

feat(typelayer)!: validate, coerce and filter through chtypes#712
EricAndrechek wants to merge 145 commits into
mainfrom
chtypes-v2

Conversation

@EricAndrechek

@EricAndrechek EricAndrechek commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

WaveHouse stops modelling ClickHouse itself. Validation, coercion, DEFAULT substitution, row-level security and insert checks now run through chtypes (github.com/wave-rf/chtypes/go v1.1.0, ABI v1). chtypes is a cgo dlopen of a per-ClickHouse-line library that runs the server's own parser and analyzer in-process. Ingest hands the request body to ClickHouse's own reader and refuses it the way an INSERT would. /v1/query and pipes ask ClickHouse to render their results. The Go that approximated ClickHouse is deleted.

This redoes #589 on current main, which is now multi-tenant. #589, on the old main, is closed as superseded by this PR.

Updated since the last revision

For reviewers who read the previous revision (head 30b9a07, chtypes Go v1.0.2):

Fixes #387 (timestamp filters and time_range shifted on non-UTC columns), fixes #516 (the TS SDK's client-side stream filters compared timestamps as text), fixes #547 (the dedupe walkthrough posted a column clicks lacks), fixes #558 (discovery now warns on tables without DDL; the second half is moot because the type layer builds its table declarations from system.columns and never reads create_table_query), fixes #722 (NaN/Inf rendered as null on /v1/query but as strings on the stream).

Breaking, by design:

  • cgo is unconditional; there is no pure-Go build.
  • Release platforms narrow to linux/amd64, linux/arm64 and darwin/arm64, and Linux binaries need glibc ≥ 2.34.
  • Autofetch and the network. The images carry no artifact. With autofetch on (the default), the first bind of a tenant on a line fetches about 45 MB from registry.wavehouse.dev (or your mirror), so a deployment now depends on that registry or a mirror until each line is cached, and the first tenant binds a few seconds late. Air-gapped deployments set WH_CHTYPES_AUTOFETCH=false and install the artifact into the cache beforehand. Mount a volume at /var/cache/chtypes or every new container fetches again.
  • A process with the api role refuses to start when autofetch is off and no artifact for its platform is installed, when an explicitly set WH_CHTYPES_CACHE does not exist, or when the cache exists but cannot be read.
  • A line is never updated once cached. Moving to a newer build of a line means fetching it with the bundled CLI and restarting. chtypes.lock is removed.
  • Supported ClickHouse lines are what chtypes 1.0 publishes: 26.3, 26.7, 26.8 and 26.9. A tenant on any other line is unavailable.
  • A malformed record refuses the whole ingest request (400 clickhouse.rejected, record N: …, nothing published, earlier records included), where main reported it per record. Policy checks stay per-record (403).
  • NaN/±Inf are the strings "nan"/"inf"/"-inf" on /v1/query and pipes, where main failed the request. Cached /v1/query and pipe entries from before the upgrade are not reused (new rendering marker).
  • TS codegen types Float columns as number | "nan" | "inf" | "-inf", where it emitted number. Regenerate and fix call sites that do arithmetic on them.
  • /v1/query and pipe results come out in SELECT order with Decimal as a number, and responses over 64 MiB fail (502); SSE/wire timestamps are rendered by ClickHouse at the column's scale (table below).
  • The ingest worker's INSERT now carries wait_for_async_insert=1 and leaves async_insert at the server's setting; only isolation retries are synchronous.
  • Several documented behaviours become "whatever ClickHouse does". They are listed below.

What changed

Type layer (internal/typelayer). One process-wide engine holds one lazily opened chtypes registry, set up once with the image zone before the first open; the zone is recorded only after that open succeeds. Each tenant gets its own table set, bound from that tenant's schema refresh before the registry reports loaded and forgotten when the tenant is removed or moves.

  • Handles: each base table compiles one handle, shared by every request. Row filters for a role's claims compile against the table and are cached per table in a bounded cache, compiled outside the cache's lock.
  • Refusing a tenant: a tenant is refused while every other tenant keeps working when its ClickHouse line has no loadable artifact (not published, fetch failed, or not installed with autofetch off), when its server time zone is a name WaveHouse does not recognise, or when it would be the first tenant served and chtypes cannot open a library in its zone. Since chtypes 1.0.3 that last case, and an artifact that fails to load or fetch on the first open, make that tenant alone unavailable; the next servable tenant sets the image zone.
  • Zones: the image zone is the server zone of the first tenant actually served. There is no UTC fallback, and a restart can change which tenant sets it (WaveHouse typelayer: the process image zone depends on which tenant binds first #726). A tenant whose zone differs gets its zone passed on every compile and call. Only a table with a zone-less DateTime and a DEFAULT, MATERIALIZED, ALIAS or EPHEMERAL expression is unavailable for such a tenant (Per-schema server profile: the server's zone and input-affecting settings, honoured by functions, DEFAULTs, literals and parsing (1.0.x, high priority) chtypes#419); a role's _eq check on a zone-less DateTime loses its auto-inject.
  • Artifacts: internal/typelayer asks for a tenant's line on its first bind. The fetch is signature-verified by chtypes on every download: a signed statement whose subject is the layer digest, checked against the SDK's embedded release key. A mirror can withhold or delay a build but cannot substitute bytes chtypes did not sign. Sharing a cache between processes is safe (rename-first installs); they must run as the same user.
  • Roles: only api-role processes load chtypes. Ingest-worker and sweeper processes need no artifact.
  • Compile profile: carries ClickHouse's type gates, so tables with LowCardinality(<integer>) and Variant columns ingest and filter. Tables compile from a generated CREATE TABLE … ENGINE = MergeTree ORDER BY tuple(), because chtypes models only some engines.

Ingest. The request body goes to chtypes as-is: one Rows call per body, exporting JSONCompactEachRow, with the role's check clauses as a row filter. ClickHouse's error-tolerant mode is off.

  • Whole-body refusal: the first record ClickHouse cannot read fails the request with 400, code: "clickhouse.rejected", ClickHouse's exception_code and an error record N: <message> (1-based, a header line not counted). Nothing is published. This is what an INSERT does.
  • Why not per-record verdicts: the previous revision kept a per-record answer on top of the server's error recovery and guarded the cases it could detect. The recovery resumes at the next line, which can sit inside the same record or past the next one. Probing 76 bodies through the real handler found that the guard did not prevent silent loss: 25 got a wrong answer. 13 lost a real record while the client got ok at its index (a quoted multi-line CSV field, a nested object on its own NDJSON line, a raw line feed inside a JSON string), 19 published a row that no submitted record holds (a fragment of a record parsed as a row), and the rest misattributed an error to a valid record, so that a retry duplicates it. Every published row was exactly what a ClickHouse server stores from the same body with error tolerance on, so these are ClickHouse's behaviour, not a WaveHouse bug (measured, 76 of 76). Removing the recovery removes the class.
  • Policy stays per-record: a record failing a role check clause, or lacking a required dedupe id, is reported in results (403/400 per record) and does not block the rest, once the body has parsed.
  • Bodies chtypes cannot judge: a body in which a record omits a generateSnowflakeID(), rowNumberInAllBlocks() or timezone() column, or a tenant whose zone chtypes cannot load after another tenant set the image zone, gets one 422 for the request and nothing is published.
  • Published bytes: the published row is ClickHouse's own writer output, carrying the inserting role's wire columns, with output_format_json_quote_denormals=1 pinned.
  • Formats: Content-Type declares the format: the JSON family, text/csv and text/tab-separated-values, with RFC 4180's header parameter mapped three ways: present → CSVWithNames/TSVWithNames, absent → positional with header detection off, no parameter → ClickHouse's default detection.
  • Dedupe: main's windowed reserve/publish/commit is unchanged. A null or empty id cell counts as missing.
  • Removed: the record-count guard and the UUID-shortness limitation, which existed only to work around error recovery.

Ingest worker. The worker's bulk INSERT leaves async_insert at the server's setting and sets wait_for_async_insert=1, so a message is acked only after its rows are stored even if a profile sets it to 0, and the server can merge many small worker batches into fewer parts. Isolating a rejected batch row by row stays synchronous (async_insert=0): an async flush merges concurrent INSERTs of the same statement, so a CHECK violation by one row fails every INSERT in the flush. Measured on 26.8 with two workers on one table, async isolation also parked the good row isolated in step with the bad one; synchronous isolation stored every good row and parked only the bad one, and a 500-row batch with one bad row took 29 s to isolate asynchronously against under 1 s synchronously.

Errors. Bodies keep main's string code class. ClickHouse's numeric code travels as exception_code, on a whole-request parser refusal (code: "clickhouse.rejected").

Query path. /v1/query and pipes go over HTTP to the tenant's target with FORMAT JSONEachRow. Failures are typed chconn.HTTPErrors, so the per-class error mapping is unchanged.

  • Fixed settings on every read:
    • wait_end_of_query=1
    • http_write_exception_in_output_format=0
    • a server-side max_execution_time
    • cancel_http_readonly_queries_on_client_close=1
    • readonly=2
    • pinned JSON rendering settings, including date_time_output_format=iso so DateTime is RFC 3339 UTC on every surface, and output_format_json_quote_denormals=1 so Float NaN and ±Inf are the strings "nan", "inf" and "-inf" as on ingest and the stream
  • Write pipes: keep IsMutation routing; they are never cached.
  • Cache key: carries a rendering marker, now JSONEachRow/4, so old and new builds never share an entry.
  • Connections: each tenant's HTTP connections are capped at its max_open_conns.

Timestamp filters. Before this change, an RFC 3339 filter value and the time_range bounds were rewritten in Go to zone-less UTC text, and ClickHouse read that text in the column's zone. On any non-UTC column, results were off by the offset. Measured with the server in Europe/Berlin: a DateTime('Asia/Tokyo') column matched nothing, and a DateTime64(3,'America/New_York') column matched the row 4 hours late.

  • Fix: ClickHouse now parses the value itself: col OP parseDateTime64BestEffort({p:String}, 8[, '<column zone>']).
  • Verified on 26.8: every column returns the right rows, the primary key is still used, and an unparseable value is a 400.

in lists. They travel as ClickHouse external-data tables in a multipart body instead of as URL parameters, which ClickHouse caps at about 128 KiB per value. A 1.26 MiB, 60,000-element list goes through, and the primary key is still used through IN (SELECT …).

Integer claims. A claim bound as {p:String} and compared with an integer column wraps modulo 2^64 (at their own width for 128/256-bit columns), on the server and in chtypes alike. Row filters, insert checks and the /v1/query policy predicate now compare integer columns against a round-trip strict cast:

if(toString(accurateCastOrNull({p:String}, 'T')) = {p:String}, accurateCastOrNull({p:String}, 'T'), NULL)
  • In range: a canonical claim answers exactly as before and keeps the primary key in use.
  • Out of range or non-canonical: a claim like 007, +5, 1.0 or ≥ 2^64 matches no row and fails an insert check with 403.

Stream. Row filters run through chtypes over the published bytes.

  • Narrower rows: a row published by a column-restricted role is evaluated with its own column list.
  • Absent columns: a filter over a column the event doesn't carry withholds the row for that reader. The server computes that column again at insert, so judging the stream's copy could let through a row /v1/query excludes.
  • Withheld-row metric: labelled by reason (filter, error, decline, unavailable, drift).

TypeScript SDK and codegen. Generated types for Float32 and Float64 columns are number | "nan" | "inf" | "-inf", matching what the wire carries. main's --tenant and --operator-key codegen flags (#729) are merged.

Build, CI and release.

  • CGO_ENABLED=1 everywhere, go 1.27, golangci-lint v2.13.2.
  • glibc builder plus distroless/cc runtime. The images carry the chtypes CLI at /app/chtypes and an empty cache at /var/cache/chtypes, and no artifact; they declare no VOLUME for the cache.
  • Native-runner release builds per target.
  • CI, dev and test ClickHouse move from 26.6.3.62 to 26.8.15.10 (internal/chversion). CI fetches the latest build of that line (scripts/fetch-chtypes.sh) and caches it; bumping the pin moves the artifact line with it.
  • chtypes.lock is deleted.

Deleted:

  • Go-side validation and timestamp canonicalization (internal/discovery/{validation,timestamp}.go).
  • The Go row-filter and numeric code (internal/policy/{rowfilter,numeric}.go, LiteralValue, CanonicalNumericLiteral, RowVisible).
  • The compact encoder (internal/ingest/compact.go).
  • The native-driver result transformer.
  • chtypes.lock and the baked-in artifact layer.
  • Every CGO_ENABLED=0 path.

Size, measured. Production Go (non-test .go outside tests/) goes from 35,889 to 39,318. The migration trades hand-written ClickHouse modelling for a wrapper around the real thing, and adds per-tenant lifecycle, artifact fetching, the HTTP reader and external-data in lists; it does not shrink the codebase lines.

Ingest benchmark, main vs this PR (measured)

AWS Graviton4 (arm64), ClickHouse 26.8.15.10, concurrency 1 to 32, every point 3 × 60 s after a 10 s warm-up with the build order rotated, 988,113 requests, 0 non-200 and 0 partial batches (this PR measured at head 30b9a07; the later commits change the worker's insert settings and the error path, not the per-request parse). amd64 and stream fan-out were not run.

  • Default durable setup (embedded NATS on disk, SyncAlways): both builds are fsync-bound at about 8.5k rows/s from concurrency 8. At concurrency 1 this PR does 48 req/s against main's 62. The extra WaveHouse CPU per row (1.27 to 1.40×) shows up as throughput or latency only at concurrency ≤ 4.
  • Queue on tmpfs (fsync free, isolating the API stage): this PR reaches 0.62 to 0.86× of main's throughput (0.62× at concurrency 1, 0.75× at 8, 0.86× at 32) at 1.36 to 1.63× the CPU per row.
  • Where the CPU goes: the whole gap is inside the chtypes library call (profiled, tmpfs, concurrency 8 and 32). Everything outside the call (publish, NATS, worker, GC, scheduler) is within ±1 µs/row between the builds, and Go allocates less than main. Per call that is about 0.9 ms fixed plus about 52 µs per row (Per-row cost of a batch preview: ~52 µs/row inside the library on a realistic ingest shape (measured) chtypes#504).
  • Shared-handle scaling: when this benchmark ran, one handle shared by every request scaled to 2.82× at 8 goroutines on arm64, against 6.90× with a handle per goroutine (upstream's figure on Go SDK 1.0.2). Since chtypes library build 20261006.170903 it scales about like a handle per goroutine (upstream measured 7.6× against 7.6× on arm64; One shared handle should scale across threads like N separate handles chtypes#480, closed).
  • Re-run on that build (measured, tmpfs, one 60 s run per point on the same instance type, this PR at fedcb20): 0.62× / 0.84× / 0.88× of main's throughput at concurrency 1 / 8 / 32, up from 0.62× / 0.75× / 0.86×. Only part of that gain is the library build (about −5 to −6 % CPU per row at concurrency ≥ 8, isolated on one binary); the rest is the worker's async-insert change.
  • Memory and threads: RSS is 120 to 180 MiB higher on the default setup, and the process runs up to 35 OS threads against main's 15 (inferred: concurrent cgo calls each hold a thread).
  • Async inserts: a first cut of this PR forced synchronous inserts, which cost ClickHouse 2 to 3× the CPU once ingest passed about 10k rows/s because the worker cut a part every ~500 rows. With the server's async inserts restored (the build measured in the benchmark's third column), ClickHouse CPU, parts and merges match main's at every concurrency. On the default durable setup the forced-synchronous insert cost nothing, so only the tmpfs figures moved.

Documented contract changes (please review these first)

surface before after
denied insert column 403 column "x" not allowed for insert 400, exception_code 117 (does not reveal whether the column exists)
unknown field / ALIAS / MATERIALIZED supplied WaveHouse wording 400, exception_code 117
EPHEMERAL column supplied accepted accepted where the format names columns (JSON, *WithNames) and a DEFAULT reads it; otherwise 400/117 (positional CSV/TSV carry wire columns only)
_eq check value supplied for a column the role may not write 403 accepted when it equals the claim; injected when omitted
role whose column policy cannot be compiled for the table n/a 500 retryable:false, no Retry-After; cause logged
a malformed record in a batch (type or parse error) reported per record, the rest published the whole request is refused: 400 clickhouse.rejected, record N: <ClickHouse message>, ClickHouse's exception_code, nothing published (per-record answers may return through Wave-RF/chtypes#497)
parse error and check failure on one record 403 the 400 with ClickHouse's code, for the whole request
JSON array body with content after the closing ] the tail was ignored 400 (code 27, no record N: prefix)
JSON array with an empty element ([a,,b], leading or trailing comma) accepted 400, code 27, named as the record it stands in for
a record whose check clause fails, or that lacks a required dedupe id per record unchanged: per record in results, the rest of the body published
a body chtypes cannot judge (a record omits a generateSnowflakeID(), rowNumberInAllBlocks() or timezone() column; tenant zone chtypes cannot load) n/a one 422 validation engine declined: … for the request, nothing published
out-of-range or non-canonical integer claim on an integer column wrapped, or matched matches no row on a read; 403 on an insert check
text/csv / text/tab-separated-values ingest 415 accepted, with the three-way header mapping
unparseable timestamp on ingest accepted at the edge, failed at the worker 400 with ClickHouse's code at the edge
bare JSON number in a DateTime64 column read in the column's units whatever the server does; on 26.8 that is epoch seconds (epoch milliseconds clamp to 9999-12-31)
unavailable tenant (or, per the row below, table) on ingest n/a 503, Retry-After: 5, generic body; cause in logs only
/v1/query and pipe rendering Go json.Marshal: alphabetical keys, Decimal as string, NaN/Inf fail the request ClickHouse JSONEachRow: SELECT order, Decimal as number, NaN/Inf as the strings "nan"/"inf"/"-inf" (as on ingest and the stream); DateTime stays RFC 3339 UTC (date_time_output_format=iso), now at the column's scale (.000Z, not trimmed)
cached /v1/query and pipe results keyed by the previous rendering marker keyed by JSONEachRow/4; entries from other builds are never read
TS codegen type of Float32/Float64 number number | "nan" | "inf" | "-inf"
/v1/query and pipe response over 64 MiB uncapped 502 clickhouse.response_too_large
/v1/query null filter value silent empty result 400
/v1/query timestamp filter / time_range on a non-UTC column off by the zone offset the exact instant
/v1/query in element that isn't a value of the column's type 400 matches no row
row-filtered SSE reader whose filter uses a column the inserting role can't write Go evaluated the padded row withheld (decline)
SSE / wire timestamps RFC 3339 UTC, trailing zeros trimmed RFC 3339 UTC rendered by ClickHouse at the column's scale; events published before the upgrade replay in the old spelling
table with a zone-less DateTime and a DEFAULT/MATERIALIZED/ALIAS/EPHEMERAL expression, on a tenant whose zone differs from the process's image zone n/a that table is unavailable (503, generic body); other tables and tenants are unaffected

Verification

  • make ci passed on this exact tree (d06d30c): Go unit 3,753, integration 536 + 101, e2e 96, TS SDK 270, Go total coverage 94.8%, and every coverage gate passed. No threshold was lowered. The tree before the image cache-path rename (43be200) also passed the full make ci on linux-arm64 against chtypes library build 20261007.044112.
  • Both images build and boot as their non-root user against ClickHouse 26.8.15.10 (the goreleaser image with --read-only), fetch the artifact into the cache on the first tenant bind, answer /readyz 200, accept a typed record and refuse a malformed one with 400 (code 27). (Verified at the previous revision's head; the artifact model changed after it, and the autofetch path is covered by the suites below.)
  • Stream-vs-query differential: 8,838 cells, comparing stream row-filter verdicts with the production /v1/query answer, covering every column shape × operator × claim, including hostile spellings and integer claims at and beyond every width. 0 disagreements (run at the previous revision's head).
  • Live filter-value test against ClickHouse 26.8.15.10 covering timestamp columns in three zones, time_range, and an external-data in list. It fails if the zone or the cast is removed.
  • The masked-swallow probe: 76 bodies through the real ingest handler, and each body also through clickhouse-local 26.8.15.10, confirming every published row is exactly what the server stores from the same body.
  • Tenancy tests: a missing line affects only that tenant; Forget never blocks a reload hook; binds racing tenant removal never resurrect a removed tenant.
  • Image-zone tests run in unset subprocesses through the real bind: Tokyo first, missing artifact first, unrecognised zone first, a zone chtypes refuses, a first open whose artifact fails to load (now committing nothing), and concurrent first binds.
  • An ingest-only process boots with no artifact installed. An api process with autofetch off refuses to boot without one, and chtypes' strict mode refuses an unusable cache (CHTYPES_CACHE_UNUSABLE); an explicit cache directory that does not exist is refused by WaveHouse.
  • The repo's pre-push reviewers ran on every round in fresh context. The latest code review (da5e72b) found the code correct and confirmed, by removing each piece of code, that every fix from the round before is pinned by a test. Its remaining findings and the docs review's were wording, applied in 4339d2e. Neither review had a blocking finding.

Known limitations

Several of these close when chtypes v2 ships. chtypes locks its library interface per major version, so the server profile (Wave-RF/chtypes#419), verbatim CREATE TABLE (#477) and defer-to-server defaults (#479) arrive in v2: first as opt-in 2.0.0-dev SDKs, then as stable 2.0.0. This PR stays a draft until then. The v2 adoption is prepared separately and tested against the v2 dev SDK: per-tenant server handles replace the process image zone, the per-call zone and the per-table zone refusal, and timezone() defaults are answered (#726 and #717 close with it). It moves into this PR in one step when chtypes 2.0.0 ships, so this PR's docs describe one stable download contract. Bug fixes ship on 1.x: shared-handle scaling (#480) is in the production library build, and strict cache (#486) is in SDK 1.1.0.

Each is tied to an issue; none is a regression against main unless it says so.

First-run risks on GitHub

  • registry.wavehouse.dev is a CI dependency on a cold artifact cache. Each fetch verifies a signature, so the registry cannot change what runs, but an outage fails a cold CI run (unverified).
  • First use of ubuntu-24.04-arm and macos-latest (advisory release-validation job).
  • CodeQL's default setup builds go1.27 with cgo (not a required check).

Follow-ups (not in this PR)

Closes #826 (the gate its follow-ups in #827 are blocked by).

🤖 Generated with Claude Code

EricAndrechek and others added 30 commits October 1, 2026 05:21
@EricAndrechek @claude
build(chtypes): build, package and release against the chtypes SDK
d177545 
Pin the revision-6 26.6 artifact in chtypes.lock and fetch it with
scripts/fetch-chtypes.sh. Every build is cgo: a glibc builder and a
distroless/cc runtime that bakes the artifact, native-runner release
builds for linux/amd64, linux/arm64 and darwin/arm64, and the setup-env
action caches the artifact for the unit, integration and e2e jobs.
golangci-lint moves to v2.13.2 (the next go directive needs it), which
retires four nolint directives it no longer needs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
docs: describe the chtypes type layer on the multi-tenant text
e58d17f 
Ports the reference migration's documentation onto main's current
multi-tenant, multi-role text. Ingest validation and row-level security run
through one process-wide chtypes engine with a per-tenant table set, loaded
only by api-role processes; a tenant with no artifact for its ClickHouse
line, or a server time zone that differs from the zone that line was opened
with, is refused on its own. Error bodies keep the string code class and add
exception_code. /v1/query and pipes are rendered by ClickHouse. Platforms
narrow to linux/amd64, linux/arm64 and darwin/arm64 on glibc.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge docs for the chtypes migration into chtypes-v2
b4779b2
@EricAndrechek @claude
feat(policy): resolved row-filter predicates and a typed WHERE renderer
e071158 
Export the resolved row-filter predicate (Predicate) and hand it out
through ResolvedPermissions.Predicates, the same resolution the query
path renders, so the stream can evaluate it with ClickHouse's own engine.
ResolvedSelect.WhereSQL(colType) renders the clause with an integer
column's claims bound through the strict round-trip cast, so a claim that
does not fit the column matches nothing instead of wrapping. WhereClause,
WhereParams and RowVisible stay until their callers move.

chsql gains the shared {p:String} encoding (EscapeStringParam), the
strict cast (IntegerType, StrictInt, IntParam), and QuoteIdent now
escapes NUL and the control characters exactly as ClickHouse's
backQuote does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
feat(discovery): refresh hooks before loaded, and the server zone
6e1fce2 
A SchemaRegistry now runs OnRefresh hooks after each successful Refresh
publishes its schemas and before it marks the registry loaded, so a
reader that sees Loaded() also sees what the hooks built from the first
refresh. Overlapping refreshes run their publish and hooks as one step,
in publish order. The server's default zone name is stored with the
version and exposed as ServerTimezone. A refresh that finds tables
without a CREATE statement (the two system scans are not one snapshot)
warns once, naming them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
feat(typelayer): tenant-keyed chtypes engine with lazy handle pools
3f27439 
Add internal/typelayer, the one package that calls the chtypes SDK
(go/v0.4.0, which needs go 1.27). One process-wide Engine holds one lazy
registry and every tenant's compiled tables: Bind(tenant, version, zone,
tables) compiles a tenant's set, Table and RoleTable hand out read-locked
handles, and Forget drops a tenant with its handles closed in the
background so a caller holding a reload lock never waits on a request.

A missing artifact for a tenant's server line, a server zone that differs
from the zone its line was opened with in this process, or a table that
does not compile makes that tenant (or that table) Unavailable; every
other tenant keeps answering. A table compiles one handle at Bind and
grows its pool only when every handle is busy, up to min(GOMAXPROCS, 8);
role shapes keep one. A rebind closes the old role projections after
releasing the base table, so a request holding a projection can still
look the base table up.

SkipWithoutArtifact lets any package's tests skip without the artifact,
or fail under WAVEHOUSE_TEST_REQUIRE_CHTYPES=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge the Go foundation for the chtypes migration into chtypes-v2
443b19a
@EricAndrechek @claude
feat(api)!: serve /v1/query and pipes through ClickHouse's HTTP inter…
4995331 
…face

The structured query and named pipes now read through ClickHouse's HTTP
interface and serve ClickHouse's own FORMAT JSONEachRow rendering, framed
as a JSON array, instead of scanning rows through the native driver and
re-marshalling them in Go.

- Per-tenant Target and the TLS-aware client cache the ops proxy uses; a
  refusal is a chconn.HTTPError and a failure on the way wraps with %w, so
  the error classes keep working. An oversized response is typed and
  answers clickhouse.response_too_large.
- Every read pins the rendering settings and sends wait_end_of_query,
  http_write_exception_in_output_format=0, max_execution_time (the tighter
  of the role's cap and query_timeout), cancellation on client close and
  readonly=2. A READONLY refusal of a read is clickhouse.misconfigured.
- Write pipes run without readonly=2 and keep BYPASS, no-store and the
  never-retryable error answer. IsMutation stays, in sql_classify.go.
- Reads share one connection cap per tenant (MaxConns, default 100).
- The builder binds named {pN:String} parameters, an in list as one
  Array(String), refuses a null filter value and a value too large for
  the HTTP interface with a 400, and rewrites an RFC 3339 filter value
  only on a DateTime or DateTime64 column.
- The cache key opens with a rendering marker, so builds that render
  differently never share a Redis entry.

BREAKING CHANGE: /v1/query and pipe responses carry ClickHouse's
rendering: keys in SELECT order, DateTime as `YYYY-MM-DD hh:mm:ss[.fff]`
in the column's zone, decimals as numbers, NaN and Inf as null. A null
filter value is a 400.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w3
e000751
@EricAndrechek @claude
feat(typelayer): parse a row under the INSERT column list it was publ…
6dd65fb 
…ished with

ParseRow accepted only the generation's full wire column list, so a row
published by a role that may write some columns could never be read. It
now accepts any duplicate-free subset of the wire columns, in any order,
and names that list on the parse (chtypes.WithColumns), so every unlisted
column holds the DEFAULT the server would store for it, computed with the
listed values in scope. A list naming an unknown or repeated column is
still ErrColumnsDrift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
feat(query): bind integer policy claims through the strict cast
35a008e 
The structured query's row filter now comes from WhereSQL with the
discovered column types: a policy claim on an integer column (any width,
Nullable or LowCardinality) binds as one {pN:String} parameter compared
through the strict round-trip cast, so a value that is not the canonical
spelling of an in-range integer matches nothing instead of wrapping.
Every other value keeps the plain {pN:String} binding, encoded by
chsql.EscapeStringParam, the same encoding the type layer's row filters
use.

The e2e suite pins the rendering of a Decimal and a DateTime64, RFC 3339
filter values on DateTime and DateTime64 columns, a timestamp read back
from /v1/query filtering as it is, and the 400 for a null filter value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
feat(config,ingest): chtypes registry key and typelayer insert settings
2aa6f7d 
clickhouse.chtypes_registry (WH_CHTYPES_REGISTRY) names the chtypes
artifact directory the API role's type layer searches first; empty keeps
the SDK's own search path. It is a bound key, so boot's refusal of unbound
WH_* variables lets it through.

The ingest worker takes its parsing settings from typelayer.InsertSettings,
the map the API judges rows under, and pins async_insert=0 so a message is
acked only once its row is stored. Nothing else in the worker changes; its
test fixture builds rows by hand instead of through EncodeCompactRow.

The dev and quickstart ClickHouse move to 26.8.15.10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge the HTTP read path for the chtypes migration into chtypes-v2
e87c9ce
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w5
b02bdd2
@EricAndrechek @claude
feat(stream): judge row-level security with the type layer, per tenant
852d5f8 
The hub's row filter is now decided by ClickHouse's own parser and
expression engine through the tenant's compiled schema, instead of a Go
re-implementation of ClickHouse's comparisons over a name-keyed decode.

- RowEvaluator.Prepare(tenant, table, columns, row) parses each event once;
  the returned view answers per subscriber. NewRowEvaluator(engine) is the
  production evaluator; an unwired hub or a nil engine withholds every row
  of a row-filtered role instead of delivering it.
- Rows published with the inserting role's narrower column list are
  parsed under that list and evaluated, in any column order. A filter on a
  column the event does not carry withholds the row for that subscriber,
  since the stored row's DEFAULT is computed again at insert.
- wavehouse_sse_rows_withheld_total gains a reason label: filter, error,
  decline, unavailable or drift.
- The policy read stays per event during replay; the schema frame, Prune
  and tenant topics are unchanged.
- The SSE end-to-end test expects ClickHouse's DateTime64 rendering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge the chtypes-backed stream filter into chtypes-v2
91e86dc
@EricAndrechek @claude
feat(ingest)!: judge every body with ClickHouse's own parser
0537e89 
Ingest hands the request body to the tenant's chtypes type layer in one
parse (Table.IngestWith through the role's projection) instead of decoding,
validating, checking, injecting and re-encoding records in Go. The verdicts
feed the existing windowed dedupe (reserve, publish, commit, release) and
the envelope carries ClickHouse's exported row with the role's wire columns.

- content_type.go (was record_reader.go): text/csv and
  text/tab-separated-values, with RFC 4180's header parameter three ways
  (present, absent, auto-detect); any other header value is a 415.
- ingest_framing.go: depth-1 comma reframing of a compact JSON array so one
  bad record does not cost its siblings, and the dedupe id read by position
  from the exported row. A null id cell is missing, like an absent one.
- Error bodies keep the string code class. ClickHouse's numeric code
  travels as exception_code: per record, and on a whole-request header
  refusal alongside code clickhouse.rejected.
- A tenant or table the type layer cannot judge is a 503 with
  Retry-After: 5, decided before the body is read, with a generic body and
  the cause in the log.
- Column policy is the role's compiled schema, so a denied column is
  ClickHouse's 117 (400) instead of a gateway 403; parse errors now precede
  check errors.
- The RecordValidator/InsertChecker seams are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w5
3ba7c6e
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w2
6fec76e
@EricAndrechek @claude
test(e2e)!: pin ClickHouse's own ingest verdicts and exception_code
1ceb21d 
The SDK's InsertRecordResult gains exception_code, ClickHouse's numeric
error code on a per-record parser refusal; the string code class is
unchanged. The ingest, NDJSON, DLQ and batching suites now assert the
type layer's behaviour: CSV/TSV with the header parameter, a compact JSON
array that keeps its good records, a denied column as 117, a header
refusal as clickhouse.rejected with exception_code 117, an _in check
judged on the table default, and an unparseable row refused at ingest so
the DLQ never sees it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge the chtypes-judged ingest path into chtypes-v2
0499111
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w5
8dcb496
@EricAndrechek @claude
feat(app): open the type layer in API processes and bind it per tenant
a776b2c 
An API-role process opens one typelayer.Engine at boot (the
clickhouse.chtypes_registry directory, else the SDK's search path) and
refuses to start without an artifact; an ingest-only or sweeper-only
process never opens it and boots with none installed. Each tenant's
schema registry binds the tenant from every successful refresh, attached
before its first one so a loaded registry is a bound one, and a registry a
reload retires forgets it without waiting on anything. Only the tenant's
current registry binds it: a refresh still in flight when its registry was
retired is dropped, and one already binding forgets again afterwards, so
a tenant a reload moved never keeps the previous database's tables and a
removed tenant's do not come back. The engine is released after schema
discovery, after the HTTP drain.

The ingest handler judges with the engine, the stream hub evaluates row
filters with stream.NewRowEvaluator over it, and pipes and structured
queries cap their HTTP connections at the tenant's max_open_conns.

Tests that boot the api role skip without the artifact, or fail under
WAVEHOUSE_TEST_REQUIRE_CHTYPES=1; the test settings close the HTTP port
too, so a ClickHouse on the developer's :8123 cannot answer them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
build(chtypes): move to SDK go/v0.5.1 and the ClickHouse 26.8 line
df6f93d 
The 26.6 line gets no further chtypes builds, so the lock moves to the
26.8 LTS line: 26.8.15.10-lts, build b1790845279, on darwin-arm64,
linux-amd64 and linux-arm64, written by the v0.5.1 CLI as lock schema 2
(every consumer of the lock is on v0.5.1). scripts/fetch-chtypes.sh
fetches line 26.8 with the same SDK version as go.mod, and CI and the
e2e orchestrator pin clickhouse/clickhouse-server:26.8.15.10, the patch
the artifact is built from. ABI revision 6 and the abi6 cache path are
unchanged.

The 26.8 build fills a skipped row's VerdictCode/VerdictErr, reports
RowsPassed 0 for a rejected batch and sets ExportDeclined on a zero-row
refusal, where every 26.6 build does not. The type layer already gates
on Outcome and reads ErrCode/ErrMsg, which are right on both; the test
that pinned the old RowsPassed now pins only what Ingest must do, and
the comments say which builds behave which way.

TestNewEngine_OpensNoLibraryAtConstruction now shadows the test line
itself and releases a table it did not expect to get, so a passing
lookup fails the test instead of deadlocking Close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
fix(typelayer): set each library's zone through the SDK default
3694f5e 
chtypes.Timezone is superseded in SDK v0.5.1, and a direct write races
the SDK's own read when another library opens. openLine now sets the
zone with SetDefaultTimezone under the lock every open takes, so the
zone an open reads is the one set for it. WithTimezone does not fit one
registry built at boot, before any server has reported its zone, that
then opens each line in the zone of its first tenant.

The zone record is keyed on the opened library's path, the unit the SDK
initialises once per process. A second registry that asks for a line
already open in another zone gets the SDK's own, untyped refusal; it is
recognised from the record and reported as the same per-tenant cause,
with the SDK's words kept.

The registry is asked for the server's line, as v0.4.0 resolved it, so
every tenant on a line shares one library whichever patch its server
runs. Which artifact answers a tenant is logged once each time the
tenant's library changes, as a warning when the server runs another
patch. Anything the SDK would print on stderr goes to the process log
through its Progress writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek @claude
fix(typelayer): compile with ClickHouse's type gates
cd1ddd7 
A table with a LowCardinality(UInt64) column exists on a server only
because its CREATE passed allow_suspicious_low_cardinality_types, but
the type layer compiled it without that gate, so chtypes refused every
record with code 455 and every filter over the table declined. A
FixedString wider than 256 and a Variant of similar types failed the
same way with code 44.

The compile profile now carries the ten type gates that the 25.8, 26.6
and 26.8 artifacts all accept. Every table compiled here already exists
on the server, so admitting its types changes no verdict a server would
give.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w1c
e1e3404
@EricAndrechek @claude
test(integration): chtypes on the app harness, with policies and tokens
92c6c6e 
Policy-driven tests now run against the wired app the way an operator
drives it: withPolicy writes roles.json and policies.json into the
settings directory and reloads it through the ops route, and bearer mints
a token for the role under the app's JWT secret. default_role stays the
admin role, so the rest of the suite is unchanged.

Ported onto it: the stream-vs-query row-filter differential, every query
now through the production /v1/query as its own role and every stream
verdict from the hub's evaluator over the app's own type layer, with a
check that the query side admits something; CSV, TSV and their WithNames
forms landing in ClickHouse; a compact array with one bad record; an
unknown header refused as 400 with exception_code 117; the published row
being the stored row; filter values round-tripping both bindings; the
type-rendering pin; a denied column refused per record with 117; an
injected check column; an _in check judged on the table default; an
integer claim that does not fit refused. The resource-cap test runs
through the same harness, and a role's time cap is pinned through
/v1/query against a slow view instead of through the native driver,
which no read path uses now.

Dropped with their subject: the Go row-filter narrowing and timestamp
canonicalization differentials. The identifier fixtures quote names the
way ClickHouse's backQuote does, control characters included. The suite's
ClickHouse moves to 26.8.15.10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB
@EricAndrechek
Merge SDK go/v0.5.1 and the ClickHouse 26.8 line into chtypes-v2
9f78c0b
@EricAndrechek
Merge branch 'chtypes-v2' into v2-w5
2f230f8
EricAndrechek and others added 6 commits October 6, 2026 17:45
@EricAndrechek @claude
Merge origin/main into chtypes-v2
aafbce0 
Brings in #734 (make ci's unit and integration time limits) and #736.
Makefile: keep this branch's chtypes-artifact target beside main's
UNIT_TIMEOUT / CI_UNIT_TIMEOUT / INTEGRATION_TIMEOUT variables.
CHANGELOG: keep both sides' entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek @claude
refactor(typelayer): take chtypes cache paths from the SDK and force …
04a3826 
…fetch failures offline

chtypes go v1.1.0 exposes no cache-root API, so the layout knowledge is
isolated in typelayer.cachePaths. Fetch-failure tests use Config.Offline
(FetchOptions.Offline) with an empty or default cache and assert
CHTYPES_ARTIFACT_MISSING. fetch-chtypes.sh gains --where and the setup-env
cache step uses it instead of a hard-coded path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek @claude
fix(typelayer): pin cachePaths to the SDK and keep the autofetch pin …
9d72d8b 
…in the offline tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek @claude
test(typelayer): keep TestCachePaths_MatchesTheSDK out of the real cache
47cb96f 
Every case now points HOME at a temp dir, so a cachePaths that falls
through to the default root plants its mode-000 entry in a temp dir,
not in the developer's real chtypes cache (where strict mode would then
refuse boot).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek @claude
test(typelayer): fail TestCachePaths_MatchesTheSDK before it writes o…
43be200 
…utside the temp dirs

A cachePaths regression that ignored $HOME would otherwise plant the
case's unreadable entry in the real cache; require the resolved root to
sit under os.TempDir() before anything is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek @claude
build(deployments): mount the chtypes cache at /var/cache/chtypes
d06d30c 
The cache path was introduced by this unreleased change. On chtypes go v1.1.0
CHTYPES_CACHE names the layout directory itself, and a later SDK appends its
own subroot under any cache root, so the version-neutral /var/cache/chtypes
replaces /var/cache/chtypes/v1 in the images, compose file and docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
Open
Closed
Open
Open
Merged
EricAndrechek added a commit that referenced this pull request Oct 12, 2026
@EricAndrechek @claude
ci: pull Docker Hub test images through the mirror.gcr.io cache (#808)
5317487 
## Summary

CI's container jobs no longer pull Docker Hub images under Docker Hub's
anonymous limit, and no repository secret is needed. The `integration`
and `e2e` jobs now run `scripts/ci/docker-hub-mirror.sh` right after
checkout. The script adds `https://mirror.gcr.io`, Google's cache of
Docker Hub, to the runner daemon's `registry-mirrors` in
`/etc/docker/daemon.json` and keeps the keys already there. Then it
sends dockerd a `SIGHUP`, because `registry-mirrors` is one of the
settings dockerd reloads without a restart. From then on, dockerd tries
the mirror first for every Docker Hub pull on that runner and falls back
to Docker Hub when the mirror lacks an image or refuses the pull. That
covers testcontainers (Ryuk included), the orchestrator and the
ClickHouse prefetch.

- **No image name changes.** The Go tests, the orchestrator, the compose
files and the prefetch keep their Docker Hub names, and the mirror is
named in the script alone.
- **A broken mirror costs nothing.** The step is `continue-on-error`: if
it fails, the daemon keeps pulling from Docker Hub, as before.
- **Local runs are unchanged.** The development guide says how to set
the same mirror, or `docker login`, after a local `toomanyrequests`.

### Images the suites pull

| Image | Pulled by | Registry |
|---|---|---|
| `clickhouse/clickhouse-server:26.6.3.62` | `tests/integration`,
`scripts/orchestrator`, CI prefetch | Docker Hub |
| `redis:8.10.2-alpine` | `internal/cache`,
`tests/integration/shared_cache_test.go`, `scripts/orchestrator` |
Docker Hub |
| `valkey/valkey:8.1.10-alpine` | `internal/cache` | Docker Hub |
| `nats:2.14.6-alpine` | `tests/integration` (`startNATS`) | Docker Hub
|
| `amazon/dynamodb-local:3.3.1` | `tests/integration` | Docker Hub |
| `testcontainers/ryuk:0.14.0` | testcontainers itself
(`internal/config.ReaperDefaultImage` in v0.44.0), once per test process
| Docker Hub |
| `docker.dragonflydb.io/dragonflydb/dragonfly:v2.0.0` |
`internal/cache` | Dragonfly's own registry, so the Docker Hub limit
doesn't apply |

The compose files (`deployments/compose/*.yaml`) back `make dev`, not
the tests, and `scripts/otel/*.sh` run only on a developer's machine.

### Why this option (measured 2026-10-10 unless labelled)

- **Rewriting image names to the mirror** (testcontainers' own
`TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX`, or a helper of ours): rejected.
- `mirror.gcr.io` serves official images only under `library/`.
`mirror.gcr.io/library/nats:2.14.6-alpine` resolves, but
`mirror.gcr.io/nats:2.14.6-alpine` is `no such manifest`, and
testcontainers' prefix is a plain `path.Join(prefix, image)`.
- A prefixed pull has no fallback, and `mirror.gcr.io` can refuse a miss
on its own upstream quota. The first lookup of
`mirror.gcr.io/clickhouse/clickhouse-server:26.6.3.62` returned
`toomanyrequests: Quota exceeded for quota metric 'Read requests to
external upstream hosts per organization per host in the US
multi-region'`, and a retry minutes later succeeded.
- Through `registry-mirrors`, dockerd normalizes `library/` itself and
falls back to Docker Hub on such an error. moby's `continueOnError`
falls back for any registry error code from a mirror endpoint (from the
source, not tested).
- **ECR Public** (`public.ecr.aws/docker/library/<image>`): rejected.
- It mirrors only the Docker Official Images. `redis` and `nats` resolve
there, but ClickHouse, Valkey, dynamodb-local and Ryuk don't exist under
`public.ecr.aws/docker/`, so each image would need its own path mapping.
- It isn't a drop-in daemon mirror either. Its repositories are
`docker/library/<image>`, not `library/<image>`, and dockerd puts a
mirror URL's path before `/v2/`, so no mirror URL maps one onto the
other (inferred).
- **Caching the images in the Actions cache**: rejected.
- The six Docker Hub images are 559 MiB compressed for linux/amd64
(ClickHouse 264, dynamodb-local 227, Redis 37, Valkey 16, NATS 10, Ryuk
2), and about 1.7 GB unpacked.
- The cache sizing policy in `.github/workflows/README.md` already
budgets past the 10 GB cap in its worst case, and the first fill would
still pull from Docker Hub.
- **`docker login` with a token**: not built, because it needs a
repository secret, which is the maintainer's call. It would raise the
fallback path from 100 to 200 pulls per six hours, and it composes with
the mirror.

Documented limits:

- **Docker Hub:** 100 pulls per six hours per IPv4 address or IPv6 /64
when anonymous, and 200 for an authenticated Personal account. A
multi-arch pull counts once per architecture.
- **`mirror.gcr.io`:** Google documents no quota. "Pulling cached images
does not count against Docker Hub rate limits", and it caches only
frequently requested images. The upstream quota above is what a miss
met.
- **ECR Public:** 1 unauthenticated pull per second, and 500 GB a month
of data for unauthenticated customers.

### Not covered

`publish-dev.yml`, `release.yml` and `goreleaser-validate.yml` build
images whose stages start from `alpine:3` and `golang:1.26.9-alpine` on
Docker Hub. They build through BuildKit's `docker-container` driver,
which doesn't read the daemon's `registry-mirrors` (inferred). That is
about one pull per push to `main`, and it hasn't been seen failing.
Moving those base images is a change to the shipped build recipe, so it
is left for a follow-up if wanted.

### Overlap with #712

#712 moves the ClickHouse pin into `internal/chversion` and bumps it to
26.8.15.10 (both versions exist on the mirror). This PR changes no image
reference. Against #712's head, `git merge-tree` conflicts in the same
files (`.github/actions/setup-env/action.yml`,
`.github/workflows/{README.md,ci.yml}`) and with the same number of
conflict regions as #712 already has against `main`; one line of this
PR's README addition sits inside one of those README regions, so
whichever lands second resolves it there. None of this PR's `ci.yml`
lines fall inside a conflict region.

## Test plan

- [x] **The script against a real dockerd** (measured): a throwaway
`docker:28.0.4-dind`, the same engine version as the `ubuntu-24.04`
runner image (20261004.327.1), on overlay2.
  - Run as root, and through `sudo` as uid 1000.
- An existing `daemon.json` (`exec-opts`, `max-concurrent-downloads`)
kept, and a second run is a no-op.
  - `/etc/docker` created when it is missing.
  - `docker info` then lists `https://mirror.gcr.io/`.
- [x] **The suites inside that daemon** (measured): test binaries built
for linux/arm64 with `-tags integration` (no `-race`) all pass.
- `internal/cache` `TestRedis_Conformance` (Redis, Valkey, Dragonfly,
Redis Cluster).
- In `tests/integration`: `TestCoordNATS_ReplicasShareTheShards`,
`TestSharedCache_RedisDownQueriesBypass`,
`TestIngest_FlowsToClickHouseWithoutDLQ` and
`TestDynamoDBDedupe_TwoInstancesShareSeenIDs`, with `TestMain`'s
ClickHouse and dynamodb-local.
- The dockerd debug log shows `Trying to pull <image> from
https://mirror.gcr.io/` for each of the six Docker Hub images, with no
`Attempting next endpoint` and no `registry-1.docker.io`. Dragonfly came
straight from `docker.dragonflydb.io`.
- [x] **Fallback** (measured): with the mirror set to an unresolvable
host, the daemon logged `Attempting next endpoint for pull after error`
and pulled `nats:2.14.6-alpine` from `registry-1.docker.io`.
- [x] `make lint-sh lint-gha lint-md lint-prose`
- [x] `make ci`
- [ ] **CI** (inferred until a run shows it): each container job's "Pull
Docker Hub images through a mirror" step logs `docker-hub-mirror: Docker
Hub pulls try https://mirror.gcr.io first`, and `toomanyrequests` stops
appearing in the integration and e2e logs.

## Related Issues

Closes #805. Part of #740.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@EricAndrechek EricAndrechek mentioned this pull request Oct 12, 2026
Open
This was referenced Oct 12, 2026
Merged
Merged
This was referenced Oct 13, 2026
Open
Open
Open
taitelee added a commit that referenced this pull request Oct 13, 2026
@taitelee @claude
feat(observability): tenant and request id on every log record and sp…
38cec1b 
…an (#819)

<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4TPFxAvhYcEYmx1tNsqBHmDP)_

## Summary

Before: chi's `RequestID` ran on every request but nothing read the id,
so no log line, span or response carried it, and an incoming
`X-Request-Id` was dropped. The tenant was on no span and on few
request-path log lines, so the authz-denied WARN, the operator-key audit
line and `batch ingested` could not be grouped by tenant.

After: every record logged through a request's context carries
`request_id`, and `tenant` once the tenant is resolved. The otelhttp
server span carries the same two attributes. Every response returns the
id in `X-Request-Id`. Neither field is a metric label.

How:
- `internal/observability/request.go` (new) adds a request scope in the
context. `TraceHandler` stamps it on records; a record that already
names a tenant keeps its own. The tenant is filled in place, so the
panic recoverer's record, logged through an earlier context, has it too.
- `internal/api/correlation.go` (new) replaces chi's `RequestID`. It
keeps an incoming `X-Request-Id` sent once with 1 to 128 printable ASCII
characters and no spaces, otherwise it mints a random 32-hex id that,
unlike chi's, names no host. `annotateSpan` runs after otelhttp and
copies the scope onto the server span.
- `TenantMW` stamps the `X-Tenant-ID` tenant. On `/v1/ops`,
`opsRequestTenant` runs ahead of authentication, so the operator-key
audit line carries the tenant: a valid `?tenant=`, or tenant `0` for any
ops route that names none (reads, raw SQL, the schema refresh). A
settings reload naming none acts on every tenant and stamps nothing.
- `cmd/wavehouse`'s boot logger is wrapped in `TraceHandler`, so stdout
keeps these fields, and `trace_id`/`span_id`, with `otel.logs.enabled`
off.

Left for later: `internal/api/ingest.go` is untouched because #712
rewrites it. Its `batch ingested` and per-event lines already log
through the request context, so they get both fields with no edit there.
Its `ingest` child span still carries only `table`; the tenant is on the
server span above it. As asked, no wiring went into
`internal/app/{wire,app}.go`, `internal/chconn/chconn.go` or
`internal/ingest/worker.go`.

The docs are updated in `api.md`, `architecture.md`, `deployment.md` (a
new "Following one request or one tenant" section), `configuration.mdx`,
`reverse-proxy.mdx` and `AGENTS.md`, with a CHANGELOG entry under
Unreleased.

## Test plan

- [x] `make build`, `go vet ./...`
- [x] `go test -race ./internal/api/ ./internal/observability/
./internal/auth/ ./cmd/...`
- [x] New tests cover: request id parsing (kept, too long, space,
control, non-ASCII, repeated), the echo on every response, span
annotation in both orders, a tenant set after the context was taken, no
duplicate `tenant` key, tenant-route and ops-route lines (both routers)
carrying the tenant from authentication on, and the ops and DLQ default
tenant.
- [x] Pre-push code and docs reviewers: `ship_it` on f70346f
- [ ] `make ci`: not run in the cloud session, which has no Docker and
cannot download golangci-lint; left for the owner to run locally

## Related Issues

Closes #771

Part of #777. Carries #333's request-id item. Related: #25.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01RzXgYrkbx4W91MUu18T2G7

---------

Co-authored-by: Claude <noreply@anthropic.com>
taitelee added a commit that referenced this pull request Oct 13, 2026
@taitelee @claude
feat(observability): identify the process in telemetry (#820)
538274d 
<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4TMNd2wVPEpUpcYkDWKyvEGU)_

## Summary

Before: the OpenTelemetry resource carried only `service.name` (the
constant `wavehouse`), `OTEL_SERVICE_NAME` and
`OTEL_RESOURCE_ATTRIBUTES` were ignored, `instance_id` was logged once
at boot and nowhere else, and the build was visible only on `/version`.
Two replicas of one role pushing over OTLP were indistinguishable.

After: every span, metric series and OTLP log record carries
`service.version`, `service.instance.id` (the boot config's
`instance_id`), `wavehouse.roles` (sorted), `wavehouse.git_commit` and
`wavehouse.build_time` alongside `service.name`, and the standard env
vars override or extend them (a malformed `OTEL_RESOURCE_ATTRIBUTES` is
logged, never fatal). A new `wavehouse_build_info` gauge (always 1;
`version`, `git_commit`, `build_time`, `roles`; no instance or tenant
label, since the resource carries the instance) shows a mixed-version
rollout. Every log line after the boot config loads carries
`instance_id`, stdout included.

How: `observability.Identity` (new `identity.go`) builds the resource
with `resource.WithFromEnv()` merged last and registers the build-info
gauge on the meter provider; `InitProvider` takes an `Identity` instead
of a service name. `internal/app/wire_identity.go` (new) builds it from
the boot config and build info. Because #712 and #770 edit `wire.go` and
`app.go`, those touches are minimal: `wire.go` changes two lines inside
`wireObservability` (the `InitProvider` argument and the OTLP logger's
`With`), neither in a region those PRs edit; `app.go` drops the
now-duplicate `instance_id` key from the `process roles` line (line
184), separated by an unchanged line from #712's nearest hunk. A `git
merge-tree` against both PR heads showed no new conflicts.

## Test plan

- [x] `make build`, `go vet ./...` (and with `-tags integration`)
- [x] `go test` for `internal/observability`, `internal/app`,
`internal/config`, `cmd/wavehouse`: resource attributes, env override
key by key, empty fields omitted, malformed env non-fatal, Prometheus
scrape shows `wavehouse_build_info` and `target_info` with
`service_instance_id`
- [x] Pre-push code and docs reviewers: `ship_it`
- [ ] `make ci` (not run here: needs Docker; owner runs it locally).
`golangci-lint` could not be downloaded in this environment either.
`tests/integration/otel_test.go` gains build_info/target_info assertions
that run there.

## Related Issues

Closes #772
Part of #777

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0174TPmnzAJSxK7ej3Q9q1i8

---
_Generated by [Claude
Code](https://claude.ai/code/session_0174TPmnzAJSxK7ej3Q9q1i8)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
This was referenced Oct 13, 2026
Merged
Merged
Merged
Merged
taitelee added a commit that referenced this pull request Oct 14, 2026
@taitelee @claude
feat(observability): bound per-table metric label cardinality (#841)
2a9fa5f 
<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4T77ZnX42McjAEHbrkWKKt95)_

## Summary

Before: seven metrics carry a `table` label (the issue lists two; the
others are `wavehouse_ingest_retries_total`,
`wavehouse_ingest_poison_total`,
`wavehouse_ingest_dedupe_commit_failed_total`,
`wavehouse_ingest_dedupe_disabled_total` and
`wavehouse_dedupe_hashed_id_total`), so a schema with thousands of
tables becomes thousands of series per metric, and nothing tells the
operator.

After: `prometheus.per_table_metrics: false`
(`WH_METRICS_PER_TABLE=false`) drops the label from all of them, and
each reports one series per remaining label set, summed over every
table. The default stays `true`. With the label on, the process logs one
`WARN` naming the setting once the metrics have carried more than 500
distinct tables. Configuration § Metric labels states which labels a
metric may carry.

How: one view on the meter provider
(`internal/observability/cardinality.go`) matches every `wavehouse_*`
instrument, so no call site changes and it applies to the Prometheus
endpoint and OTLP push alike, which is why the key also shapes OTLP
output. With the knob off the view's attribute filter denies `table`;
with it on, the filter keeps everything and counts distinct table names
until it warns, then drops the set. The convention limits `table` to
counters, up-down counters and histograms: dropping the label from a
gauge would leave one table's value, not a sum, which matters for the
per-table size gauges #417 plans.

`internal/app/wire.go` (also edited by #712 and #770) gets one line:
`AggregateTables: !cfg.Prometheus.PerTableMetrics` in the
`ProviderConfig` literal.

## Test plan

- [x] `make build`, `go vet ./...`
- [x] `go test -race` for `internal/observability`, `internal/config`,
`internal/app`, `internal/stream`, `internal/api`
- [x] New tests: label kept, label dropped and summed, library
instruments untouched, observable up-down counter summed, single warning
past the threshold, end-to-end Prometheus scrape through `InitProvider`;
defaults test covers the `true` default and the env override
- [x] golangci-lint on the changed packages, markdownlint, misspell
- [ ] `make ci` (owner runs it locally). In this sandbox `make verify`
cannot finish: govulncheck gets a 403 from vuln.go.dev, and
golangci-lint flags `jwksRetryMin` at `internal/auth/auth.go:177`, which
fails on main too.

## Related Issues

Closes #538

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BPFAg9iJ3uHnZdsjncjgKf

---
_Generated by [Claude
Code](https://claude.ai/code/session_01BPFAg9iJ3uHnZdsjncjgKf)_

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
taitelee added a commit that referenced this pull request Oct 14, 2026
@taitelee
feat(observability): expose net/http/pprof endpoints on an admin port (…
731d54e 
…#843)

<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4TCHrtVnnNfv8VAPYnyiCiq2)_

## Summary

Before: to diagnose memory growth, a goroutine leak or a CPU hotspot in
a running process, you had to rebuild it with debug instrumentation.
WaveHouse exposed no `net/http/pprof` endpoints.

After: with `profiling.enabled: true` (`WH_PROFILING_ENABLED`, default
`false`), every process serves Go's pprof endpoints under
`/debug/pprof/` on `profiling.addr` (`WH_PROFILING_ADDR`, default
`127.0.0.1:6060`). This applies to every process whatever its roles, and
the listener is separate from the API port. Covered: heap, CPU,
goroutine, allocs, threadcreate, block, mutex, the execution trace,
cmdline and symbol. The endpoints are unauthenticated, so the default
address is loopback. `kubectl port-forward` still reaches it. Boot logs
a `WARN` for any address that is not loopback. `make dev` turns
profiling on unless `WH_PROFILING_ENABLED` is already set.

How:
- `observability.PprofHandler` mounts pprof's handlers on a mux of its
own. It is never added to the API router, and nothing serves
`http.DefaultServeMux`, where the package's import also registers them.
- `internal/app/wire_profiling.go` (new) runs the listener through the
existing `serve`. A bind failure is logged and not fatal, the same as
the metrics sidecar.
- The listener has no `WriteTimeout`, because pprof refuses a
`?seconds=` longer than one.
- Its `BaseContext` is canceled as the drain begins, so a CPU profile or
trace in flight ends then instead of holding the stop for the whole
`server.shutdown_timeout`.
- `config.Validate` refuses an address that is not `host:port`, and a
port that is `server.port` or the enabled Prometheus sidecar's port.

**Call site in a shared file:** `internal/app/app.go` gains one line,
`a.wireProfiling()`, at the end of `New`. `wire.go` is untouched, to
stay clear of #712 and #770.

**Not included:** block and mutex profiles return empty, because
WaveHouse sets no sampling rate for either. The docs say so. Setting a
rate would add a knob and runtime cost, so I left it for a follow-up if
someone wants it.

Docs: configuration.mdx (new Profiling section, plus the YAML and env
examples), deployment.md (new Profiling section with the commands),
development.md (`make dev`), architecture.md, AGENTS.md, config.yaml,
CHANGELOG.

## Test plan

- [x] `make build`, `go vet ./...`
- [x] `go test -race` for `internal/config`, `internal/observability`
and `internal/app`: config validation and warning cases, the default and
zero-value cases, every pprof endpoint served, no other path served, the
listener wired only when enabled, pprof returning 404 on the API router
whether profiling is on or off, a live listener serving, a 60s CPU
profile ending at the stop (this test fails without the shutdown
cancel), and a bind failure that is not fatal
- [x] golangci-lint v2.11.4 on the touched packages, and markdownlint on
the changed docs
- [x] Pre-push reviewers: code `ship_it`, docs `ship_it`
- [x] GitHub CI green on the head commit

## Related Issues

Closes #143
Part of #777

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Q9hHKxkKwmj22yP4Uu8fJR
taitelee added a commit that referenced this pull request Oct 14, 2026
@taitelee @claude
fix(observability): continue ingest traces into the worker and insert (…
5d20684 
…#842)

<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4T3xRphsBikbNzJVvACgXsvm)_

## Summary

Before: an ingest request's trace ended at the queue publish. The
publisher wrote the request's trace context into every row's queue
headers, but the worker's consumer path never read it back (only the
`Subscribe` path did), and the worker started no span, so neither the
flush nor the ClickHouse `INSERT` could be tied to the request that
produced the rows.

After: the worker's consumer path extracts the trace context onto each
delivered message, in both the embedded and the external NATS broker.
Each flush of a batch opens `IngestWorker.Flush`, a root span linked to
the span of every request whose rows the batch holds (one batch holds
many requests' rows, so links rather than a parent; at most 128 linked,
`producers` counting them all), with `tenant`, `table` and `rows` as
attributes. Each `INSERT`, row-by-row isolation included, runs under a
`clickhouse.insert` child span with the standard database attributes,
and the error is recorded when it fails.

- New code lives in `internal/ingest/trace.go`;
`internal/ingest/worker.go` gets only the two call sites (`flushGroup`
and `insertToClickHouse`, the latter now with a named `err` return so
the deferred span end sees it), since open PRs #712 and #770 edit that
file. `internal/chconn/chconn.go` is untouched.
- `internal/mq/embedded.go` and `internal/mq/external_consumer.go`
change one line each; the broker conformance suite gains
`ConsumeCarriesTheTraceContext` beside the existing `Subscribe` case.
- `tenant` rides as a span attribute, never a metric label. The docs
note that a flush is a trace of its own, sampled at
`otel.traces.sample_rate` independently of the requests it links to.
- Query, pipe and ops ClickHouse spans mentioned in the issue are out of
scope here.

## Test plan

- [x] `internal/ingest/trace_test.go`: link per distinct producer, the
128 cap, the insert span as the flush span's child with its attributes,
error recording, and an end-to-end test through the real embedded queue
(publish under a span → consumer → flush links → insert child)
- [x] `mqtest` conformance: the consumer path carries the trace context
on `CreateConsumer`'s ctx (embedded broker run here; external broker
runs under the integration tag)
- [x] `tests/integration/otel_test.go` `TestOTel_IngestTraceContinuity`:
through the wired app with a caller's `traceparent`, the handler span is
in the caller's trace, the flush links to it and the insert is the
flush's child
- [x] `make build`, `go vet ./...`, gofumpt, goimports, markdownlint,
misspell; `-race` unit tests for `internal/ingest`, `internal/mq/...`,
`internal/observability`, `internal/api`
- [x] GitHub CI green on the head commit: Lint, unit, integration (app
and backends), e2e, coverage, docs build

## Related Issues

Closes #774

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011Da6nRMNGVuCzVmpf2G2Pi

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
taitelee added a commit that referenced this pull request Oct 15, 2026
@taitelee @claude
feat(observability): metrics for settings reloads and tenants served (#…
71de22b 
…851)

<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4T8x6JCiRc55QtYXgY1JMBUb)_

## Summary

Before: settings reload outcomes were log lines only. When a tenant's
folder was rejected on one process of several, the only way to find it
was a log grep. A reload also counted as adopted when the work after it
failed, because `AfterAdopt` hooks return nothing: the ClickHouse pool
hook only logged its error, and the Pebble dedupe and embedded
queue-budget hooks discarded theirs (`_ = reconcile()`).

After: every process exports these metrics, none with a tenant label
(#538):

- `wavehouse_settings_reloads_total{trigger, result}`, where `result` is
`adopted`, `partial` or `rejected`
- `wavehouse_settings_last_reload_successful`
- `wavehouse_settings_last_reload_success_timestamp_seconds`
- `wavehouse_settings_tenants{state}` (`served`, `rejected`)
- `wavehouse_settings_tenants_removed_total`
- `wavehouse_settings_reconcile_failures_total{component}`
(`clickhouse`, `dedupe`, `mq`)

How it works:

- **Hook failures now count.** The three hooks report through the new
`settings.Registry.Reconciled(component, err)`, and a failure makes the
reload `partial`. A failure while boot applies the settings counts
against the boot reload instead.
- **Where it lives.** `internal/settings/stats.go` keeps the counts.
`internal/app/wire_settings_metrics.go` (new) exports them, and is
registered last in `New`, so no collection can see the boot reload's
count go down.
- **Unchanged.** The response of `POST /v1/ops/settings/reload` is the
same as before.

**Files shared with #712 and #770.** `internal/app/wire.go` gets one
`Reconciled` call per hook (pools, Pebble dedupe, embedded MQ), and
`internal/app/app.go` gets one call at the end of `New`. I test-merged
both PRs: this change adds no conflicted files beyond the ones they
already conflict on.

**Skipped local checks.** Taite agreed to skip the git hooks for this
PR, and GitHub CI is the gate. In this sandbox `make verify` cannot run
vulncheck (a 403 from vuln.go.dev), and golangci-lint was built from
source because the official download was refused. That build flags
`jwksRetryMin` in `internal/auth/auth.go`, which CI's lint does not
flag. `make ci` was not run. The pre-push code and docs reviewers both
returned ship_it on 1dc1b93.

## Test plan

- [x] `make build`, `go vet ./...`
- [x] `go test -race ./internal/settings/ ./internal/app/` (new
`stats_test.go` covers flat, nested, hook-failure, boot-apply-failure
and per-tenant reloads; `wire_settings_metrics_test.go` covers every
series and unregistering)
- [x] markdownlint, Biome, misspell, gofumpt, astro check
- [ ] GitHub CI

## Related Issues

Closes #776. Part of #777.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01FiBgjWxuMqeWqYTi6hqADH

---
_Generated by [Claude
Code](https://claude.ai/code/session_01FiBgjWxuMqeWqYTi6hqADH)_

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@taitelee taitelee mentioned this pull request Oct 15, 2026
Merged
6 tasks done
taitelee added a commit that referenced this pull request Oct 20, 2026
@taitelee @claude
feat(api): trusted-proxy client address, and gate traceparent on it (#…
6179c67 
…855)

<!-- claude-projects-attribution: {"github_login":"taitelee"} -->
_Requested by **Taite** · [project
thread](https://claude.ai/code/project/chan_019fe1KaM5pLaH8s2bVcLf4T?thread=cmsg_019fe1KaM5pLaH8s2bVcLf4TEM2YcBKk4yXuaNGSEiYbKx)_

## Summary

Before: behind a proxy, WaveHouse's logs only had the proxy's address,
and the server span's `client.address` was whatever the client wrote
first in `X-Forwarded-For` (otelhttp reads the leftmost entry
unconditionally). Any client's `traceparent` was adopted as the parent
of the request's span.

After: `server.trusted_proxies` (`WH_SERVER_TRUSTED_PROXIES`, CIDRs or
bare addresses, default empty) lists the proxies to believe. From a
listed peer, `X-Forwarded-For` is walked right to left and the first hop
that isn't a listed proxy is the client; from anyone else the client is
the connection's own address and the header is ignored. The result is
`client.address` on every request-scoped log record (operator-key audit
line included) and on the server span, replacing otelhttp's value. A
`traceparent` is joined only from a listed proxy; from anyone else
otelhttp starts a new root linked to it (`WithPublicEndpointFn`).
`r.RemoteAddr` is never rewritten.

How: `internal/config/proxies.go` parses and validates the list (bad
entry refuses boot, a `/0` entry is a boot warning).
`internal/api/clientip.go` holds the walk and a `clientAddress`
middleware right after `requestID`; `observability.SetRequestClient`
stores it on the existing request scope from #771, and
`AnnotateSpan`/`TraceHandler` emit it. The otelhttp middleware is now
built once per router instead of once per request.

**Behavior change:** with the default empty list, a tracing proxy's
`traceparent` is now linked rather than continued until the proxy is
listed. It has its own `### Changed` CHANGELOG entry. The integration
suite now lists loopback as trusted so `TestOTel_IngestTraceContinuity`
still joins the caller's trace.

Call sites in files open PRs #712/#770 also edit: one line in
`internal/app/wire.go` (`deps.TrustedProxies = …`). `wire_ops.go` gets
the same line.

## Test plan

- [x] `make build`, `go vet ./...` (and `-tags integration
./tests/integration/`)
- [x] `go test -race ./internal/api ./internal/config
./internal/observability`; `go test ./internal/app ./internal/auth`
- [x] Table tests for the walk: spoofed leftmost entries,
trusted-looking spoof left of the client, chained proxies across header
lines, all-trusted chain, empty and unreadable hops, ports, bracketed
IPv6, IPv4-mapped and zoned peers, non-IP peer
- [x] Router test: one `client.address` on the span with the resolved
value; traceparent joined from a trusted peer, linked (new root) from an
untrusted one
- [x] Code and docs pre-push reviewers: ship_it
- [x] GitHub CI green: lint, unit, integration (app and backends), e2e,
coverage (new files at 100%)

## Related Issues

Closes #333

Part of #777. Reviewer noted a possible follow-up outside this scope:
`baggage` is still extracted from any peer (pre-existing; nothing reads
it today).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UbPNcxn6XzeNyU3qBMohvK

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/api HTTP handlers, routing, middleware area/app Process wiring (internal/app): component build, run, release area/docs Documentation, site/, README area/infra CI, build, deploy, Docker, release area/ingest Ingest pipeline (Bento, batching, DLQ) area/observability Metrics, logs, traces, health, profiling area/pipes Named query pipes area/policy Access control policies (Hasura-style) area/query Structured query AST, SQL builder area/sdk TypeScript SDK (clients/ts/) dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation github_actions Pull requests that update GitHub Actions code go Pull requests that update go code

Projects

WaveHouse Task Board
Status: Backlog

Milestone

No milestone

1 participant

@EricAndrechek