docs(api,deployment): examples and BYOS notes for the positional wire

AGENTS.md

@@ -43,7 +43,7 @@ Sixteen internal packages under `internal/` (plus `internal/testutil/` for share
 - **`policy/`** — Hasura-style access control, **role-first**: `TablePolicy` is `map[string]RolePermissions`, and a role's grant splits by operation into `SelectPermissions` (columns, row `filter`, aggregations, the `max_*` limits) and `InsertPermissions` (columns, `check`) — so a field only one side honors does not exist on the other. `Evaluate()` resolves ONE operation and leaves the other side **nil** (`Select *ResolvedSelect` / `Insert *ResolvedInsert`), which every accessor fails closed on — nil is "not resolved", distinct from an empty side, which is "unrestricted" (what the admin return builds). Claim templating (`{{ jwt.claim.path }}`) resolves during that call. Policies come from `Source`, a `func() *Policy` read per call (`settings.Store.Policy` in production, `Static(p)` in tests)
 - **`query/`** — Structured query AST types + SQL builder with schema validation, structural policy predicate/limit emission, timestamp bucketing
 - **`settings/`** — the settings directory: `Validate` (strict JSON, per-file rules, cross-file role references), `Store` (the adopted snapshot + serialized `Reload`, typed accessors read per call, `AfterAdopt` hooks), the fsnotify `Watch`, and the `go:embed`ded seed `wavehouse bootstrap` writes
-- **`stream/`** — SSE fan-out: the event `Hub` (registers subscribers by `(topic, role)`; `Broadcast` projects + serializes each event once per role, the #294 delivery hot path — a role carrying a row-level `filter` keeps the shared projection but delivers per subscriber, each subscriber's claims evaluated against the row, #319), `Subscriber` (per-connection outbound `Frame` queue, `Send`/`Frames`; claims fixed at construction, immutable), the `Bucket` fan-out set (`subscriberSet`, one per `(topic, role)`), the `Heartbeater` keepalive wheel, and `Metrics` (the `wavehouse_sse_*` stream instruments)
+- **`stream/`** — SSE fan-out: rows travel POSITIONALLY, so each connection is told its projected column list in an `event: schema` frame before its first row and again on drift — **not** guaranteed after a gap-fill across a column change, which can leave a connection reading live rows against a stale list until it reconnects ([#543](https://github.com/Wave-RF/WaveHouse/issues/543)) — (tracked per connection; replay tracks its own). The event `Hub` (registers subscribers by `(topic, role)`; `Broadcast` projects + serializes each event once per role, the #294 delivery hot path — a role carrying a row-level `filter` keeps the shared projection but delivers per subscriber, each subscriber's claims evaluated against the row, #319), `Subscriber` (per-connection outbound `Frame` queue, `Send`/`Frames`; claims fixed at construction, immutable), the `Bucket` fan-out set (`subscriberSet`, one per `(topic, role)`), the `Heartbeater` keepalive wheel, and `Metrics` (the `wavehouse_sse_*` stream instruments)
 
 ## Key Design Decisions
 
@@ -53,8 +53,8 @@ The invariant index — what must stay true. Full narrative and rationale live i
 2. **Bring Your Own Schema** — users create ClickHouse tables; WaveHouse discovers them via `system.columns` and never auto-migrates.
 3. **Schema-driven ingest** — `POST /v1/ingest?table={table}` takes flat JSON, validated against the discovered schema (unknown fields rejected, types/nullability enforced). No envelope. The **declared `Content-Type` chooses the format and the bytes never do** (arity within the JSON family is still the body's): no declaration, one whose **media type** is unsupported or unparseable, a comma-bearing value that, as a whole, does not parse as one media type, or repeated lines that **disagree**, is a `415` decided *before* the body is read. A malformed *parameter* on a comma-free line never costs the request (`; charset=a; charset=b` still reads as its media type), and repeated lines are accepted only when they all resolve to the same **supported** format — two agreeing `text/csv` lines are still a `415`. A body declared NDJSON stays NDJSON whatever its bytes, so a bad line is a per-record error rather than a silent re-framing; the reverse (NDJSON sent as `application/json`) is deliberately **not** caught — record one, `200`, the rest ignored ([#561](https://github.com/Wave-RF/WaveHouse/issues/561)). Fail-closed — preserve it when touching `internal/api`.
 4. **Async ingestion** — ingest returns 200 after optional dedup + MQ publish; ClickHouse writes happen later via `StartIngestWorker`. NATS full → 503 + Retry-After.
-5. **Per-table batching** — the worker groups events by table and bulk-INSERTs in schema column order; each table's batch is independent.
-6. **Dead Letter Queue** — failed batch inserts publish to `WAVEHOUSE_DLQ` (`dlq.<table>`), gated per table by `dlq.enabled` in the settings directory's `config.json` (hot-reloadable; off = leave the row unacked for redelivery). No silent data loss.
+5. **Per-table batching** — the worker groups events by table, then splits each batch by column list (`groupByColumns`), emitting one `INSERT INTO … (cols) FORMAT JSONCompactEachRow` per distinct list so a schema change mid-stream can't corrupt a statement. Each table's batch is independent.
+6. **Dead Letter Queue** — failed batch inserts publish to `WAVEHOUSE_DLQ` (`dlq.<table>`), gated per table by `dlq.enabled` in the settings directory's `config.json` (hot-reloadable; off = leave the row unacked for redelivery). No silent data loss on the insert path. The one drop is an envelope the worker cannot READ (malformed JSON, an unknown **or absent** `format` — a pre-v2 envelope carries none — or columns and row that don't pair): it is poison by construction, so with the DLQ off it is acked-and-dropped rather than redelivered forever — logged at `ERROR` and counted by `wavehouse_ingest_poison_total` under `disposition="dropped"`. With the DLQ on it is parked like any other failure, and counted under `disposition="parked"`.
 7. **Auth: always on, fail-loud, decoupled from authz (security)** — the JWT middleware always runs (no `auth.enabled`/`dev_mode` flag); it verifies with HMAC **or** JWKS (not both), with accepted `alg` pinned to the active verifier and checked before any key is used (rejects `alg:none` and cross-family confusion). No/invalid/expired token → empty role → policy `default_role`, with the bad-token reason stashed so a denying gate returns a loud `401`, not a bare `403`. Elevated access needs a valid granted role. **Sanctioned exception:** a configured non-JWT operator key (`auth.operator_key`; presented via `Authorization: Operator <key>` or the `X-Operator-Key` alias) deliberately couples authN+authZ — a constant-time match authorizes a full-access platform operator (stamps the admin role plus an operator bit) independent of the verifier (see #11). Detail: architecture.md § `api/` + `internal/auth`; see also #11, §Security Considerations.
 8. **Optional dedup** — opt-in via `dedupe.enabled` in the settings directory's `config.json` (hot-reloadable: a reload opens or closes the Pebble store via `dedupe.Managed`); `dedupe.id_field` there selects the JSON key, overridable per table.
 9. **Singleflight** — `TieredCache` coalesces concurrent misses (`x/sync/singleflight`) to prevent cache stampede.

CHANGELOG.md

@@ -27,7 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 - **A policy `check` on a column the table cannot accept is now refused instead of silently unenforced** (BREAKING; `internal/api/ingest.go`, `internal/discovery/discovery.go`, `docs/src/content/docs/{api.md,access-control.mdx}`): a `check` clause naming a column the table does not have, one it computes (`MATERIALIZED`/`ALIAS`), or an `EPHEMERAL` one can never be enforced — the published row carries one slot per insertable column, so an auto-injected value for anything outside that set is dropped on the way out, and an ephemeral column is never stored even though the row does carry it. The record inserted **without** the value the policy required and answered `200 {"ok":true}`. Demonstrated on this branch: a `check` of `tenant _eq {{ jwt.tenant }}` against a `MATERIALIZED tenant` column published `columns:["page"], row:["/a"]` — the tenant constraint absent from the row entirely. It is now refused, naming every offending column and the reason, on **every** insert by that role until the policy or the table is corrected: a single-object request answers `403`, while a batch answers `200` with the same message against each record in `results` — the batch is still read to the end and reports per record, as it does for any other rejection. Policy validation cannot catch this — it never sees the ClickHouse schema — so **audit your `check` blocks against their tables before upgrading**; `wavehouse validate` will not tell you.
 
-- **The ingest envelope carries only insertable columns** (`internal/discovery/discovery.go`, `internal/api/ingest.go`, `internal/stream/hub.go`, `internal/testutil/testutil.go`, `tests/integration/ingest_test.go`, `clients/ts/src/types.ts`): naming columns explicitly in the `INSERT` — the change above — makes a computed column fatal, so the envelope, the compact encoder and the SSE connect-time announcement now use the table's **insertable** subset. Verified against ClickHouse 26.6.3: a `MATERIALIZED` column in an `INSERT` column list is `Cannot insert column …, because it is MATERIALIZED column` (code 44, and `insert_allow_materialized_columns` defaults to `0`); an `ALIAS` column is `No such column …` (code 16). Schema discovery reads every row of `system.columns` with no `default_kind` filter, so without this both would land in the envelope and then in the statement, and **a table carrying either could ingest under the previous column-less `FORMAT JSONEachRow` and could not ingest at all** — every row to the DLQ, or redelivered forever where the DLQ is off. `Column` gains `DefaultKind`; `TableSchema` gains `IsInsertable` / `InsertableColumns` / `InsertableColumnNames`, memoized per table since the ingest path would otherwise rebuild them once per record. `EPHEMERAL` stays insertable — it is insert-only by construction, never stored, confirmed on the same server rather than assumed. `GET /v1/ops/schema` still reports the whole table, now including `default_kind`: a computed column stays queryable, it just cannot be written. No fixture in the suite declared a computed column, which is why every gate was green while this was broken; `tests/integration` now creates one and drives HTTP ingest → NATS → the worker's `INSERT` end to end.
+- **The ingest envelope carries only insertable columns** (BREAKING; `internal/discovery/discovery.go`, `internal/api/ingest.go`, `internal/stream/hub.go`, `internal/testutil/testutil.go`, `tests/integration/ingest_test.go`, `clients/ts/src/types.ts`): naming columns explicitly in the `INSERT` — the change above — makes a computed column fatal, so the envelope, the compact encoder and the SSE connect-time announcement now use the table's **insertable** subset. Verified against ClickHouse 26.6.3: a `MATERIALIZED` column in an `INSERT` column list is `Cannot insert column …, because it is MATERIALIZED column` (code 44, and `insert_allow_materialized_columns` defaults to `0`); an `ALIAS` column is `No such column …` (code 16). Schema discovery reads every row of `system.columns` with no `default_kind` filter, so without this both would land in the envelope and then in the statement, and **a table carrying either could ingest under the previous column-less `FORMAT JSONEachRow` and could not ingest at all** — every row to the DLQ, or redelivered forever where the DLQ is off. `Column` gains `DefaultKind`; `TableSchema` gains `IsInsertable` / `InsertableColumns` / `InsertableColumnNames`, memoized per table since the ingest path would otherwise rebuild them once per record. `EPHEMERAL` stays insertable — it is insert-only by construction, never stored, confirmed on the same server rather than assumed. `GET /v1/ops/schema` still reports the whole table, now including `default_kind`, `default_expression` and `position`: a computed column stays queryable, it just cannot be written. No fixture in the suite declared a computed column, which is why every gate was green while this was broken; `tests/integration` now creates one and drives HTTP ingest → NATS → the worker's `INSERT` end to end. **BREAKING:** a record that *supplies* a value for a `MATERIALIZED`/`ALIAS` column is now rejected (`400 … cannot be inserted`) where it was previously accepted and silently dropped by the positional encoder.
 
 - **Ingest reads the request body up front, and the per-record decisions sit behind interfaces** (`internal/api/{ingest,ingest_seams,bufpool,record_reader}.go`, `internal/stream/hub.go`, `internal/ingest/compact.go`): responses are unchanged except at the body cap and one new read-failure body (`400 {"error":"invalid request body"}`, when the body cannot be read at all — a malformed transfer encoding or a truncated upload, which previously surfaced through the decoder as `invalid json`), and at the cap the `413` is now decided before any record is processed: an over-cap batch no longer ingests the prefix it had already decoded, and an over-cap single-object body whose first object was followed by an oversized tail — which used to answer `200` after ingesting that one object — now answers `413`. Both are improvements, since a client retrying a `413` can no longer double-insert a prefix, but they are behavior changes and the memory profile changes too (see below); this is the seam work the native type layer lands against. The handler now reads the whole (already `MaxBytesReader`-capped) body into a pooled `*bytes.Buffer` and runs the record readers over those bytes rather than the live connection — so the `413` surfaces at that read instead of mid-iteration (same status, same message), and the `415` is decided from the header before a single byte is read. Three decision points became interfaces with default implementations that delegate to today's code unchanged: `RecordValidator` (schema validation + timestamp canonicalization — the two calls stay where they are, with the check-clause block between them, since merging them would move checks onto canonicalized values), `InsertChecker` (the `_eq` and `_in` comparisons), and `stream.RowEvaluator` (row visibility, reached by both the live fan-out and replay through the one shared admission step). **The memory profile is not unchanged, and that is the deliberate part.** Streaming meant peak resident bytes on the order of one record: the NDJSON path scanned line by line and the array path let `json.Decoder` compact after each element. Peak is now O(body) per in-flight request — and `bytes.Buffer` grows by doubling, so the peak allocation can exceed the body cap before `MaxBytesReader` errors. `maxPooledBufferBytes` (1 MiB) caps what a request hands *back* to the pool, not its peak, and nothing in `internal/api` bounds total in-flight bytes, so the ceiling is concurrency × the 16 MiB data-plane cap — which has no operator knob (`maxRequestBytes` is test-only), so the outer limit is the reverse proxy's, which the reverse-proxy guide already advises setting. Kept because it is the shape the native type layer lands against, which needs the body addressable rather than consumed; a bound on total in-flight ingest bytes is tracked in [#544](https://github.com/Wave-RF/WaveHouse/issues/544). Operators fronting large batches at high concurrency should size for it or cap body size at the proxy. All three are nil-safe: an un-wired handler or `Hub` uses the default rather than panicking past the check. Also new: `ingest.EncodeCompactRow`, which renders a record as one `JSONCompactEachRow` line — inert in this commit, and the encoder every published row goes through by the end of the release.
 

clients/ts/src/types.ts

@@ -264,15 +264,16 @@ export interface Column {
   name: string;
   type: string;
   is_nullable: boolean;
-  /** Whether the column declares any default at all. */
+  /** Whether the column declares any default at all — see `default_kind`. */
   has_default: boolean;
   /**
    * How the default is declared, when there is one: `DEFAULT`, `MATERIALIZED`,
    * `ALIAS` or `EPHEMERAL`. Omitted when the column declares none.
    *
    * `MATERIALIZED` and `ALIAS` columns are computed by ClickHouse and cannot be
-   * inserted, so they never appear in an ingest envelope or an SSE `schema`
-   * frame — though they are still reported here and are still queryable.
+   * inserted — a record naming one is rejected — so they never appear in an
+   * ingest envelope or an SSE `schema` frame, though they are still reported
+   * here and are still queryable.
    */
   default_kind?: string;
   /** 1-based ordinal in the table's declaration order — the order `columns` itself is in. Always present. */