Repository navigation
Conversation
Makisuo
added a commit
that referenced
this pull request
Jul 21, 2026
…#194) * feat(cli): pass chDB config file to local server * feat(cli): add local chDB checkpoint command * feat(cli): restore dirty local store from checkpoint * fix(cli): harden chDB checkpoint recovery * fix(cli): make checkpoints immutable and durable * fix(cli): reconcile interrupted checkpoint restores * fix(cli): preserve checkpoints across recovery * fix(cli): make retirement cleanup restart-safe * fix(cli): journal destructive store resets * feat(cli): checkpoint pins, maintenance lock, and archive foundation Export acquireCheckpointPin / releaseCheckpointPin and withMaintenanceLock from the checkpoint module so the dependent archive branch can pin an immutable checkpoint against retention and serialize its work against checkpoint creation, restore, and reset. Pin acquisition resolves and validates the checkpoint and writes a UUID-named durable pin record under backups/pins/<id>/; release removes only the exact owned record and fails closed on identity mismatch. The maintenance lock wraps the existing sibling ownership lock with the same dead-PID reconciliation and live-PID refusal semantics. Add the archive module skeleton: ArchiveError typed error, the six raw-telemetry signal map with event-time columns, and the archive path model with strict ID and range-date validation, recursive symlink rejection, root containment, and a live-data-directory separation guard. Twelve pin and lock tests cover acquire/release, retention protection, identity mismatch, escape refusal, and concurrent-owner rejection. * feat(cli): centralized archive tuning configuration Add a validated, documented tuning model for the seven machine-sensitive archive knobs: Parquet writer threads, row-group rows, maximum shard rows, maximum estimated shard bytes, target chunk bytes, minimum free-space reserve, and the archive and scratch roots. Defaults are the measured research baselines (max_threads=1, 10,000-row groups, ~500k rows / ~256 MiB per shard). The parser rejects non-positive or fractional values, a row group larger than a shard, a shard byte budget too small for one row group, a free-space reserve larger than the chunk target, an implausible thread count, and missing roots. A tuning record captures the effective values for every generation manifest so a generation is reproducible and deployment drift is visible. * feat(cli): archive generation write path with durable promotion Add the versioned, strict archive generation manifest and active-pointer formats with location-bound parsing that rejects unknown versions, signal/range/ generation mismatch, negative counts, malformed shard names, and missing tuning. A read helper binds a manifest to its on-disk (signal, range, generation) directory. Add the Parquet shard exporter: one bounded UTC-hour slice per shard, written via SELECT ... INTO OUTFILE FORMAT Parquet directly on the restored scratch chDB (result consumed as a write side effect, never returned into JavaScript). Each shard is validated for row count, byte bound, and SHA-256 before the generation is sealed. Add the generation write lifecycle: acquire maintenance lock, resolve and pin the checkpoint, restore to sacrificial scratch, export bounded shards, validate row counts against the source, durably move the owned building generation into place, atomically select it through the active pointer (reporting supersession while retaining the old generation), append the catalog, release the pin, and remove only owned building output. Free-space preflight runs at operation time. Twenty manifest, pointer, path-model, promotion, supersession, and catalog tests cover the pure and filesystem-level state machine; the full chDB export path is exercised by the native smoke. * feat(cli): archive listing, active parquet paths, and catalog rebuild Add the archive read-side. listActiveGenerations reports the active generation per signal/range with archived row count, shard count, total shard bytes, and the active generation's Parquet shard paths in order. Superseded generations are retained on disk but never appear in listings or active-path output, so late-arrival history cannot be double-counted. A malformed active pointer for one range is skipped without hiding the others, and the pointer file is left untouched. activeParquetPaths resolves the machine-readable DuckDB-ready shard paths for one signal across all sealed ranges in ascending order, excluding superseded generations. rebuildCatalog regenerates the per-signal catalog.jsonl from the authoritative generation manifests, so a truncated or missing catalog is recoverable without rescanning Parquet bytes. All retained generations (active and superseded) appear once; a generation directory missing its manifest is skipped. Seven listing and catalog-rebuild tests cover active selection, supersession exclusion, cross-range ordering, malformed-pointer tolerance, and catalog rebuild including superseded and manifest-less generations. * feat(cli): maple archive command group Register a 'maple archive' command group with three subcommands following the server.ts Effect-CLI house style: - 'maple archive create <range-date> <signal>': seal one UTC day of one signal into a validated Parquet generation from the current or a specified checkpoint, using the centralized tuning config and reporting supersession. - 'maple archive list [--format summary|paths|json] [--signal <name>]': report active generations with sizes and paths, or emit machine-readable active Parquet shard paths (excluding superseded generations) for DuckDB. - 'maple archive rebuild <signal>': rebuild a signal's catalog from manifests. All commands are local-only by construction (filesystem/checkpoint work, never WarehouseExecutor), map failures to ArchiveError for non-zero exit, and accept the inherited shared flags without acting on them. * docs: local telemetry archives architecture and operator guide Add docs/local-telemetry-archives.md covering the hot store, immutable checkpoint source, pinning, shared scratch lifecycle, Parquet generations, catalog, calibration, and the independent DuckDB query path with memory-limit and spill guidance. Document the happy path from checkpoint through DuckDB, every major off-happy-path outcome (unavailable/incompatible checkpoint, stale pin, interrupted restore, partial shard, validation mismatch, full volume, pointer or catalog corruption, late arrivals, interrupted GC, insufficient budget, failed calibration), which failures leave the live store untouched vs. require action, the capacity/resource model with its one-machine caveat, and the v1 non-goals. Add the archive create/list/rebuild commands to the local-mode CLI reference, cross-linking to the architecture document. * feat(cli): native archive smoke and DuckDB acceptance Add native-archive-smoke.sh: ingest markers into all six raw tables, create two checkpoints, archive a sealed UTC day per signal, list active generations, query each archive with DuckDB for exact source counts (2), prove the live store is unchanged after archiving, and rebuild every signal catalog. Two consecutive runs pass. Fix two real defects the smoke exposed: 1. JSONEachRow count parsing. count() results from chDB's FFI path are newline-delimited JSON objects, not a JSON array. The day-count, hour-count, and time-bounds helpers were JSON.parse-ing the whole string as an array and indexing [0], yielding undefined -> 0 rows. Now split on newlines and read the count() column by name, matching the checkpoint module's readJsonRows. 2. Date predicate correctness. chDB's bundled ClickHouse miscounts aggregate count() over a toDateTime64-vs-DateTime predicate (per-row comparison is correct but the aggregate returns zero). Switched all range predicates to toDate()/toHour() equality, which normalizes both second-precision DateTime and nanosecond DateTime64(9) event-time columns. Also rename the archive list format flag from --format to --output to avoid collision with the global --format (json|table) shared flag, and update the docs accordingly. * feat(cli): archive calibration with candidate matrix and generated config Add 'maple archive calibrate' and its internal 'calibrate-run' worker. The calibrator runs a bounded matrix of Parquet writer/row-group/shard candidates against a pinned checkpoint restored into sacrificial scratch, measuring peak RSS, wall time, and output bytes per candidate in a fresh child process (so peak RSS is measured per-candidate, not accumulated in-process). It selects the lowest-RSS candidate within the operator's memory and time budgets, reports high/low confidence, and writes a versioned configuration document only when --write-config is passed (never auto-applies). A native calibration smoke verified the full round-trip: four candidates ran at ~320MB RSS each, the matrix selected writerThreads=1/rowGroupRows=20000 within a 1GB budget with high confidence, and the generated config recorded the measured peak RSS, output bytes, and effective tuning. An impossible budget fails closed with low confidence and no config mutation. Per D-017, true external-volume deployment-scale calibration under the deployment chDB/user/filesystem is a Phase 3 dependency; Phase 2 proves the mechanism and generated-config round-trip on a local scratch volume. * fix(archives): close archive-root symlink escapes and bind state to location (C-1, H-7) The prior approval missed a reproducible external-write vulnerability: symlinked archive descendants (signal root, catalog, manifest, pointer) could be followed by mkdir/write, placing state outside the configured archive root. Path safety (C-1): - ensurePrivateDirectory now walks each ancestor with lstat BEFORE creating, refusing to cross a symlink at any depth (mkdir -p + final lstat was unsafe). - A sync assertNoSymlinkSync variant is added for the synchronous read-side. - promoteGeneration, appendCatalog, and rebuildCatalog now assert no-symlink on every path they create or overwrite, immediately before use. - assertSafeRealPath verifies type + containment + no-symlink at read/write sites. Strict state binding and fail-closed malformed handling (H-7): - parseArchiveActivePointer binds signal/range to its directory, rejecting a pointer copied or moved to the wrong range. - listActiveGenerations surfaces malformed pointers/manifests as errors instead of silently skipping them, while still listing unaffected ranges. - Catalog entries now carry formatVersion 1. Eight external-sentinel tests prove the escapes are closed: symlinked signal root, catalog, manifest, pointer, and ancestor; outside targets are verified untouched. Malformed-pointer surfacing and pointer-directory binding are tested. * fix(archives): close read-side symlink escapes (HIGH-1/HIGH-2 from R1 review) The R1 fix closed the C-1 write side but left the read side open: a non-racing attacker who plants one symlinked descendant (generation dir, shard file, or range dir) could make listActiveGenerations/activeParquetPaths/rebuildCatalog follow it out of the archive root, feeding attacker-controlled Parquet to DuckDB and attacker-controlled manifest fields to the catalog. The fix's own guard primitives were written but never wired into read paths. This closes the asymmetry: - readArchiveGenerationManifest (the single chokepoint for manifest reads) now asserts no-symlink + real-file before readFileSync, so a symlinked signal/range/generation/manifest chain is refused. - listActiveGenerations asserts no-symlink on the active-pointer path before reading it and on every shard path before returning it to DuckDB. - The dead assertSafeRealPath helper (never called) is removed. Three read-side sentinel tests prove the escapes are closed: a symlinked generation dir is refused by readArchiveGenerationManifest and rebuildCatalog (the attacker manifest is not trusted); a symlinked shard file is refused by listActiveGenerations and surfaced as an error rather than returned to DuckDB. Outside targets are verified unread. * fix(archives): fail-closed listing, rebuild preflight, mandatory root, fresh-root creation (H-7/R7-R8) Close four defects from the Gate 1 conditional review: 1. activeParquetPaths now THROWS when any relevant range for the requested signal has a malformed pointer/manifest/shard, rather than returning a partial path list that would silently feed DuckDB incomplete data. Errors for other signals do not block a clean signal's paths. 2. rebuildCatalog now PREFLIGHTS every manifest before writing. If any generation is missing its manifest, is malformed, or is on a symlinked path, the existing catalog is PRESERVED untouched and the call throws. The old code silently skipped corrupt generations and wrote a partial catalog. 3. Pointer reads now require a real regular file (assertRealFileSync), matching the manifest-read check, so a non-file entry cannot feed undefined content. 4. ensurePrivateDirectory's root parameter is now MANDATORY (was optional and degenerate when omitted). Fresh archive-root creation is repaired: the root is created first before walking children, fixing the ENOENT on a fresh root. assertRealFileSync is moved to paths.ts as a shared sync primitive. Three new tests cover the fail-closed activeParquetPaths (malformed same-signal range throws; different-signal error does not block), rebuild catalog preservation on missing manifest, and fresh-root nested directory creation. * fix(archives): bounded sharding and pre-promotion Parquet reopen validation (H-1, H-2) R3 — bounded sharding: a sealed UTC day is now partitioned by UTC-hour windows, then within each hour by LIMIT/OFFSET sub-splits when the hour exceeds maxShardRows or the estimated uncompressed-byte budget (maxShardBytes). The prior code aborted if a single hour exceeded the bound instead of splitting. Shard names encode their slice: HH-NNNN.parquet (hour + sequence). R2 — pre-promotion Parquet validation: after writing each shard, it is REOPENED via chDB file(path, Parquet) and its row count, min/max event time, and column list are READ BACK from the Parquet file. A 19-byte invalid 'Parquet' file fails this reopen because file() cannot parse it. The shard record now carries the REOPENED counts and columns, not the source counts — closing the tautology where the record always matched the source query. An empty (0-row) reopen also fails. The ArchiveShardRecord gains a columns field (schema round-trip proof); the manifest parser strictly validates it. Test fixtures updated to the new shard name format and columns field. * fix(archives): deterministic sharding, schema/hour/byte validation, path escaping (CR-1, H-A, H-B, H-C, M-1, M-2) Close the adversarial R2/R3 review's release blockers: CR-1 — sub-sharding now uses ORDER BY (_part, _part_offset) + LIMIT/OFFSET, making each hour's sub-shards an EXACT partition (no overlaps, no gaps). The prior LIMIT/OFFSET without ORDER BY was nondeterministic and could silently duplicate or drop rows. The last shard gets the remainder via min(rowsPerShard, hourRows - offset), and the expected row count is computed and passed to validation. H-A — DESCRIBE file() failures are no longer swallowed. A schema that did not round-trip fails the shard (columns.length === 0 throws), so the manifest's columns field is a real guarantee, not best-effort. H-B — validateShard now checks the reopened row count against the intended slice size AND verifies all rows fall within the expected hour window (toHour(min) === toHour(max) === hour), so a shard containing rows from the wrong hour is rejected. H-C — the byte bound is now enforced on UNCOMPRESSED bytes via parquet_metadata().uncompressed_size, not compressed on-disk size. Compression could previously keep an oversized shard under the compressed ceiling. M-1 — operator paths containing single quotes or backslashes are rejected before export (assertSafePath); both INTO OUTFILE and file() literals use the same sqlLiteral escaping. M-2 — the promoteGeneration test no longer passes a stale 7th argument. * test(archives): pin export path-safety invariants (single-quote/backslash rejection) The adversarial R2/R3 review flagged that no test exercised the path-rejection (M-1) primitives. Add two pure-logic tests proving exportSignalShards rejects a shardsDir containing a single quote or backslash before any chDB query runs. The full Parquet write+reopen+validation path is exercised by the native smoke. * fix(archives): repair runtime incompatibilities and strictness gaps (Gate 2 NO-GO) Fix two confirmed chDB runtime incompatibilities the native smoke exposed: - formatRow(RowBinary, *) -> formatRow('RowBinary', *): the bare token RowBinary is an unknown identifier in chDB (code 47); it must be a quoted string literal. - parquet_metadata() -> file(path, ParquetMetadata).total_uncompressed_size: parquet_metadata() is DuckDB syntax, not ClickHouse; bundled chDB exposes uncompressed size via the file() ParquetMetadata interface. Strengthen validation to compare against source intent (not just 'parseable'): - captureSourceSchema captures name+type before export; validateShard compares the reopened Parquet schema against it (base type match), so a schema that did not round-trip fails the shard. - validateShard verifies the UTC DATE of all rows (toDate min==max==rangeDate) in addition to the hour window. - estimateBytesPerRow and validateShardBytes now use the corrected chDB APIs. Strengthen manifest strictness (H-7): - sha256 must be 64 hex chars; counts must be safe non-negative integers. - Cross-field: unique shard names, shard-row sum == archivedRowCount, sourceRowCount == archivedRowCount, rangeEndExclusive == next-midnight. - parseShardRecord requires nonempty columns; min <= max event time. Other repairs: - Strict calendar date validation (reject 2026-02-31; JS Date normalizes it). - rangeEndExclusive is now the next midnight (exclusive), not 23:59:59 inclusive. - Free-space preflight checks the actual destination volume (climbs to the closest existing ancestor when the root is missing) and includes working bytes. - rebuildCatalog is async and uses durableWrite (atomic temp+fsync+rename) so an interruption cannot destroy the prior catalog. - Active shard paths are asserted to be real existing regular files. - Pin-release failures are surfaced to stderr, not silently swallowed. 133/133 tests pass; typecheck, lint, format clean. * fix(archives): timezone, schema normalization, and smoke determinism The native smoke exposed three runtime issues after the API fixes: 1. Timezone: toDate()/toHour() without an explicit 'UTC' argument used the host session timezone, causing the export query and the validation query to shard by different hours (12 UTC stored, 08 EDT computed). All toDate/toHour calls now pass 'UTC' explicitly in export.ts and generation.ts. 2. Schema normalization: Parquet export strips LowCardinality(...) wrappers and widens DateTime to DateTime64. The schema comparison now unwraps LowCardinality/Nullable and normalizes DateTime to DateTime64 before comparing base types, so legitimate Parquet type transformations pass while real schema loss still fails. 3. Smoke determinism: the insert markers now use toDateTime64(..., 9, 'UTC') at a fixed UTC noon timestamp within RANGE_DATE, so the archive's toDate/toHour predicates find the rows regardless of the host timezone. The prior now()/ now64(9) inserts landed on a different UTC day when local and UTC diverged. Two consecutive native archive smokes PASS: 6 signals archived, DuckDB exact counts (2 each), live store unchanged, catalog rebuilt. 133/133 unit tests, typecheck, lint, format clean. * fix(archives): merge-safe static-snapshot part-interval sharding (CRITICAL cursor defect) Replace the broken ORDER BY (_part, _part_offset) LIMIT/OFFSET pagination with a true static-snapshot physical plan that is merge-safe: 1. For each UTC hour, enumerate the active parts and their _part_offset ranges (freezing the layout into a shard plan). 2. Split each part's offset range into bounded intervals (one Parquet file each). 3. Export each shard via WHERE _part = '<part>' AND _part_offset BETWEEN <lo> AND <hi> — a physical predicate targeting exact immutable rows, not a relative OFFSET that drifts when parts merge. 4. After all shards for the hour, re-enumerate and verify every planned part still exists with the same row count. If a merge removed or changed any part, the export fails closed. The prior OFFSET approach was confirmed corrupting data: an injected merge between shard queries produced [5,6,7,8,5,6,7,8], omitting [1,2,3,4] while all local validations and the aggregate count passed. The part-interval plan targets exact rows by immutable location, so a merge of a different part cannot affect this shard's rows, and if THIS part merges away, the post-export verifyPartsUnchanged detects it. A multi-part adversarial native probe (native-archive-merge-probe.sh) reproduces the cross-check's corruption scenario and confirms the fix: 2 parts → 2 shards, exact ID match [t0..t7], no duplicates. The dead estimateBytesPerRow is removed; the byte bound is enforced post-write via parquet_metadata uncompressed size. * fix(archives): verify shard SHA-256 and byte size against manifest (tamper detection) listActiveGenerations now hashes each shard file and compares its SHA-256 and byte size against the manifest before returning the path to DuckDB. A tampered regular file whose actual hash or size disagrees with the manifest is rejected (D-015's checksum-mismatch fail-closed rule). The manifest is authoritative; a mismatched file is surfaced as a listing error, not silently returned. Test fixtures updated to compute real SHA-256 and byte size from placeholder content, and to use the new HH-NNNN.parquet shard name format. 133/133 pass. * fix(archives): merge freeze, source-interval validation, exact schema comparison Three remaining cross-check blockers closed: 1. Merge freeze: exportSignalShards now runs SYSTEM STOP MERGES before enumerating parts and SYSTEM START MERGES in a finally. Without this, a background merge on the restored scratch store changed part names between enumeration, export, and source-interval validation, causing stale-part false failures (and on the old OFFSET approach, silent corruption). The freeze makes the physical layout stable for the entire export. 2. Source-interval validation: validateShard now re-queries the SOURCE table for the same (_part, _part_offset) interval and compares count + nanosecond time bounds (via toUnixTimestamp64Nano(toDateTime64(...))) with the reopened Parquet. This proves each shard contains exactly the source rows for its physical interval — a wrong-but-valid row set can no longer pass. Time bounds use nanosecond comparison to avoid session-timezone string mismatches. 3. Exact schema comparison: compareSchema now enforces exact column count, name, and order (positional match), plus base-type compatibility (LowCardinality/ Nullable unwrapped, DateTime normalized to DateTime64). Extra/dropped/ reordered columns fail closed. Nested type parameters (Map key/value types) are still normalized to the base token — documented as a compatibility limitation. Also: ExportSettings gains an afterShardValidated callback for the adversarial merge-injection probe. The sha256FileSync comment is corrected (reads whole file, not streaming). verifyPartsUnchanged is replaced by verifyHourTotalUnchanged (robust to part-name churn from merges, detects data loss/gain). Two native smokes PASS: six-signal (with source-interval validation on all six tables) and merge-safety (2 parts → 2 shards, exact ID match). 133/133 unit tests. * test(archives): adversarial merge-injection probe (Gate 2 cross-check #5) A standalone probe that reproduces the exact cross-check corruption scenario: two parts for the same UTC hour, with an OPTIMIZE TABLE ... FINAL injected between shard exports via the afterShardValidated hook. SYSTEM STOP MERGES blocks the OPTIMIZE (Code 236, Cancelled merging parts), the physical layout is frozen, and the exported shards contain the exact source row set [t0..t7] with no duplicates or omissions. This is the definitive proof that the static-snapshot part-interval plan is merge-safe: the prior OFFSET approach produced [5,6,7,8,5,6,7,8] under the same scenario. The probe uses the real bundled chDB via FFI (MAPLE_LIBCHDB). * feat(archives): repair Gate 2 round-4 (non-contiguous offsets, schema, complex, bytes, range, merge-freeze) Close the seven Gate 2 round-3 blockers, all grounded in measured chDB behavior (gate2-round4-probes.md) and verified by a native adversarial probe suite plus the six-signal smoke: - Sharding (#1): keep the fixed UTC date+hour predicate in every export AND source-validation query, paging within an hour with ORDER BY (_part, _part_offset) LIMIT N OFFSET M. Deterministic under SYSTEM STOP MERGES; handles non-contiguous matching offsets (the round-3 offsets 0/9 layout) naturally. Source re-query binds by PAGE POSITION, not by _part_offset value (offsets repeat per part). - Merge-freeze scope (#2): try begins immediately after a successful STOP MERGES so any later failure restarts merges. - Schema compare (#4): recursive normalize-then-compare using the measured chDB->Parquet transforms (LowCardinality unwrap, DateTime->DateTime64(3), DateTime64(N) UTC add, recurse Map/Array/Nullable). Parameterized inner types survive: Array(UInt64) != Array(String) is now caught. - Complex-value digest (#3): per-shard NULL-safe, DateTime-stable digest over the source slice vs reopened Parquet. Catches a changed map/array/NULL value with identical count and time extrema. Robust to cityHash64 NULL propagation (histogram Min/Max) and bare-DateTime widening. - Byte-aware planning (#5): estimate bytes/row from a probe shard, pass maxShardBytes to the planner so a wide-row hour splits by bytes even under the row limit. - Sealed-range timestamps (#6): bind shard min/max event time to [rangeStart 00:00 UTC, next midnight); reject out-of-range (e.g. 2027 for a 2026 range) and add the complexDigest field to the shard record. 149/149 unit tests; 7/7 native adversarial scenarios; six-signal smoke PASS (6 signals archived, DuckDB exact counts, live store unchanged, catalog rebuilt). Typecheck, lint, format, git diff --check clean. * fix(archives): non-commutative digest, authoritative byte refinement, UTC-nano bounds (Gate 2 round 5) Close the four Gate 2 round-4 defects, each grounded in measured chDB behavior and verified by a hermetic adversarial probe suite plus an independent DuckDB oracle: - Complex-value digest (CRITICAL): replace the commutative per-column sum (which could not detect a same-typed column swap or a cross-row value reassociation) with a per-row POSITION-BOUND hash aggregated as an order-independent multiset: per row h = cityHash64(isNull(c1), norm(c1), ...); shard digest = cityHash64(groupArray(h ORDER BY h)). cityHash64 of a tuple is order-sensitive (measured), so a column exchange changes the row hash; the sorted multiset preserves row identity + multiplicity. NULL-safe, DateTime-normalized, 41ms/+15MiB at 500k rows (measured). Manifest gains a complexDigestAlgorithm field. - Byte bounds: measure each candidate shard's actual total_uncompressed_size and recursively bisect the physical range when it exceeds maxShardBytes, until every accepted shard meets both bounds. The only impassable case — a single matching row whose size alone exceeds the bound — fails distinctly. Sampling no longer determines correctness. - UTC time evidence: replace timezone-less ISO strings parsed with Date.parse (host-timezone-dependent: a valid 23:30 UTC bound was rejected as next-day under America/New_York) with min/max event time as UTC epoch-NANOSECOND decimal strings parsed with BigInt. Manifest format version bumped 1 -> 2; a v1 manifest fails closed with a distinct message (preserved, not migrated). - Export query: no ORDER BY (D-016). Page each MergeTree part's _part_offset domain into half-open ranges; offset holes are excluded by the UTC date/hour predicate, and counts derive from actual matching rows. Each defect was first reproduced by a failing probe (red baseline), then the fix turned it green. 147/147 unit tests; 15/15 adversarial probes (incl. the independent DuckDB oracle confirming read-back fidelity of counts, nanosecond timestamps, NULLs, and arrays); six-signal native smoke, merge-injection, and merge-safety probes pass. Typecheck/lint/format/git diff --check clean. * test(archives): permanent adversarial validation matrix (Gate 2 round 5) Add apps/cli/test/archive-adversarial-matrix.md as a permanent gate, not a one-time review. It exists because four consecutive Gate 2 rounds failed for the same root cause: implementation and tests shared one mental model, so the tests confirmed the author's intent while independent review attacked the author's assumptions. For each invariant (exact row identity, physical sharding, complex values, byte bounds, UTC time bounds, cleanup, malformed state, recovery), the matrix records the invariant, the counterexample transformation that realizes it, the named hermetic probe that must catch it, the independent oracle, and the required result. It also clarifies that D-016 forbids ORDER BY in the export query itself (the round-5 implementation honors this). The working rule every future archive-export change must answer before being called complete: "How could an incorrect archive preserve every metric I currently check?" * fix(archives): NULL-safe multiset digest + fail-closed digest algorithm (Gate 2 round 5 repair) Close the round-5 independent review's blocking defect and its two additional findings, each grounded in a measured chDB behavior and covered by a red→green probe: - CRITICAL — NULL-bearing rows collapsed the digest. The round-5 per-row hash passed `isNull(c), c` into cityHash64, and chDB returns NULL if ANY argument is NULL — so a row with NULL Min/Max (histogram tables) lost value sensitivity for ALL its other columns (verified: two datasets with NULL Min/Max but different Count/Sum produced the identical digest). A bare nullable value is now NEVER passed: per column `i, if(isNull(c), '\x00NULL', toString(norm(c)))`. Time columns normalize to a NUMERIC epoch (tz-stable source↔Parquet): bare DateTime -> toUnixTimestamp(c,'UTC'); DateTime64(N) -> toUnixTimestamp64Nano(toDateTime64(c,9)); Array(DateTime*) -> arrayMap over elements (a raw toString(Array(DateTime64)) diverges on the inner timezone — measured). - complexDigestAlgorithm is now fail-closed: the manifest parser accepts only cityhash64-multiset-v2 (a known set); unknown algorithms throw distinctly (digest semantics are versioned, so a v1/v3/bogus value must not be silently re-interpreted). Algorithm bumped v1 -> v2. - Removed the unused `duck` helper flagged by oxlint in the DuckDB oracle probe. - Aligned the adversarial matrix's probe names with the committed probes. Added `archive-probe-null-value-sensitivity.ts`: two NULL-Min datasets with different non-null values must produce different digests (red at 747a70c6, green after the fix). 16/16 adversarial probes green (incl. the independent DuckDB oracle), six-signal smoke + merge probes pass, 148/148 unit tests, typecheck/lint(0)/format/`git diff --check` clean. * fix(archives): explicit NULL flag in digest (invariant alignment, Gate 2 round-5 repair) The round-5-repair digest used a sentinel string for NULL, but a real Nullable(String) value equal to the sentinel would collide with an actual NULL (reproduced: Note=NULL and Note='\x00NULL' produced the same digest). The documented invariant was an EXPLICIT NULL flag, not a sentinel. Not production-exploitable today (the six archive schemas' nullable columns are numeric/time), but the algorithm claims general NULL-safe semantics. Each column now contributes `columnIndex, isNull(c), if(isNull(c), '', toString(norm(c)))` — the explicit isNull flag binds NULL-ness, so NULL can never be conflated with a value. Algorithm bumped v2 -> v3 (digest semantics changed); the parser accepts only cityhash64-multiset-v3. Also fixes the two prefer-string-starts-ends-with lint warnings in normalizeValueForHash (regex tests -> startsWith). Adds archive-probe-null-flag-binding.ts (NULL vs sentinel string -> different digests) and a matrix row. 17/17 adversarial probes green (incl. the independent DuckDB oracle), six-signal smoke + merge probes pass, 148/148 unit tests, typecheck/lint(0)/format/ git diff --check clean. * feat(checkpoints): additive seams for crash-safe archive generation (Gate 3a) Allow a caller-supplied deterministic pinId in acquireCheckpointPin (refuses overwrite so post-crash ownership is unambiguous), a deterministic scratchSubdir plus a beforeRestore seam in withRestoredCheckpoint, so the archive generation journal can record the exact pin/scratch it owns BEFORE allocation and reconcile an interrupted operation by exact identity. Purely additive; existing callers pass no new args and behave identically. * feat(archives): crash-safe generation journal + automatic reconciliation (Gate 3a) Port the checkpoint subsystem's proven crash-safety pattern to archive generation: a versioned operation intent journal written BEFORE pin acquisition, recording the deterministic pinId/scratchSubdir/generationId up front (closing the orphan-pin window), with reconcileArchiveGeneration driving an interrupted operation to its exact intended state on the next run. Lifecycle boundaries (each a recorded durable phase, journal never ahead of the filesystem): intent -> pin -> scratch -> restore -> building -> shards -> manifest -> promote -> pointer (CAS-guarded) -> catalog -> pin-release -> scratch-remove -> complete. promoteGeneration now only moves+writes the manifest; selectActiveGeneration flips the pointer only if it equals the recorded base or already selects the intended generation (no clobbering of concurrent activity). Reconciliation: at most one active op; pre-publication ops quarantine incomplete building (retained, D-004), remove only owned scratch, release only the owned pin; post-promotion ops finish pointer/catalog then clean up. Idempotent (a second reconcile converges to the same state). The finally handles only thrown errors; the journal is authoritative for SIGKILL recovery. selectActiveGeneration is CAS-guarded and idempotent. Tests cover promotion split, CAS refusal, idempotence, and the journal's fail-closed parsing (unknown format version, invalid phase, scratch-subdir escape, identity mismatch, multi-op refusal, malformed intent). * test(archives): authoritative SIGKILL crash-recovery harness (Gate 3a) native-archive-crash-recovery-probe.sh injects a REAL SIGKILL at each of 14 lifecycle boundaries via committed child workers paused at fault seams, then reconciles WITHOUT a fresh export and verifies exact convergence: active pointer, catalog (no duplicate), no orphan pin/journal/scratch/building debris, DuckDB exact counts for published generations, live store unchanged, and idempotent re-reconciliation. Plus a cross-cutting check that a fresh 'archive create' reconciles an interrupted op as its first locked step before allocating. Hook-throw results cannot substitute for this oracle: a thrown error runs the finally (releasing the pin/removing debris) while SIGKILL does not, so only a real kill proves the journal — not the finally — is the crash-recovery authority. The adversarial matrix gains a 'Crash recovery' section (invariant 9) recording every kill-point, the SIGKILL-vs-hook fidelity classification, the oracle, and the required recovery result, with the working rule answered per row. * fix(archives): bind crash recovery to durable owned state * test(archives): make SIGKILL recovery oracle exact * fix(archives): tighten aborted journal and crash oracles * fix(archives): preserve legitimate restored-store symlinks * fix(archives): verify observed crash-recovery invariants * style(archives): format crash-recovery matrix * feat(archives): v3 operation journal with kind discriminator + v2 migration (Gate 3b) Raise the operation journal format v2 → v3 with a discriminated kind field (create | gc) so reconcile can dispatch on operation type. The strict parser now validates kind first, then create-vs-gc fields; create intents carry the same fields as before plus kind:'create'. GC intents record the frozen deletion set, keep, manifest+shard evidence per target, and progress. A v2 (pre-kind) create intent left by a Gate 3a binary is migrated to v3 under the maintenance lock by migrateActiveIntentIfLegacy before reconcile reads it, so a stranded 3a intent is reconciled rather than blocking all future archive work. Migration is a mechanical, re-validated field addition; a corrupt v2 record fails closed rather than being silently lifted. The v3 parser rejects raw v2. * feat(archives): conservative journaled GC with tombstone-based collection (Gate 3b) reconcileArchiveGeneration now dispatches on operation kind: create ops run the existing 3a lifecycle recovery; gc ops delegate to reconcileGcOperation in gc.ts. A crashed GC shares the single operations/active/ slot, so it MUST reconcile or it blocks all future archive work. planArchiveGc enumerates superseded generations under the maintenance lock and builds a frozen deletion set: --keep N (default 1) retains the newest N superseded per range; the active generation is never selected; a signal whose catalog cannot be authoritatively reconstructed is excluded ENTIRELY before any mutation (never delete a range then discover reconstruction is impossible); malformed/symlinked/ambiguous state fails toward over-retention. Collection is crash-safe via tombstone rename — never in-place recursive delete: each target is atomically renamed source→tombstone (manifest intact, no half-deleted window), then only the journal-owned tombstone is removed. A SIGKILL mid-collection leaves whole owned state that reconcile proves it owns. Each target revalidates the pointer (CAS — must equal the recorded active, not merely 'some other gen') and the source (manifest SHA + per-shard SHA) before rename; a pointer that came back onto a target stops collection and preserves it. Reconciliation resumes from the recorded progress cursor and NEVER expands the frozen set. * feat(cli): archive reconcile + archive gc subcommands (Gate 3b) maple archive reconcile [--dry-run]: converge an interrupted create or gc op to its intended state without a fresh export, inside the maintenance lock. Idempotent; ambiguous/malformed state exits nonzero without mutation; no-op when nothing is active. maple archive gc [--keep N] [--dry-run]: reclaim superseded generations via the tombstone collector, retaining the newest N per range (default 1; 0 reclaims all). --dry-run reports the exact delete/retain sets WITHOUT mutating anything (no journal, no deletion, no catalog change). Rejects negative/fractional --keep. Both acquire the maintenance lock and reconcile any prior op before planning new work. Adds a red() style helper for the over-retention/excluded warning lines. * test(archives): interrupted-GC SIGKILL oracle + hostile GC tests + matrix (Gate 3b) native-archive-gc-probe.sh injects a REAL SIGKILL at each of 5 GC lifecycle boundaries (intent-durable, first-rename, during-removal, after-all-removals, after-catalog) via a committed gc worker paused at fault seams, then converges via the real maple archive reconcile CLI and verifies: only frozen targets removed, active generation intact + DuckDB-queryable, pointer unchanged, catalog exactly matches manifests, no tombstone retains a generation, completed journal retained, second reconcile is a no-op, and a subsequent archive create succeeds (a crashed GC never blocks future work). Plus a dry-run-mutates-nothing check. archive-gc.test.ts covers keep-N/keep-0 ordering, never-active selection, fail-closed on malformed/symlinked/ambiguous, signal-level exclusion, and dry-run filesystem state unchanged. archive-journal.test.ts adds v2→v3 migration tests (clean lift, corrupt-v2 fail-closed, stranded-on-disk lift, no-op). The adversarial matrix gains a 'Garbage collection (Gate 3b)' section (invariant 10) recording every boundary, the tombstone topology, the CAS pointer guard, and the over-retention working rule. * fix(archives): GC state machine, locked reconcile, nonmutation, missing-pointer (Gate 3b repair) Repair the seven NO-GO blockers from the Gate 3b round-1 review: 1. GC state machine: introduce a nonterminal gc-collecting phase (allows a full cursor — the legitimate post-final-deletion state). Progress is persisted as gc-collecting per target, NEVER complete. complete is written only after all targets are absent + every affected catalog passes assertCatalogExact. Fixes the premature-complete defect where reconcile archived a journal mid-collection. 2. Terminal invariant proof: a complete GC journal is verified before archival (completedTargets === targets.length, every target source + tombstone absent, every affected pointer unchanged, every catalog exact). A failed invariant preserves the active journal (fail closed). reconcileGcOperation re-reads the durable record (not the stale in-memory cursor) before verifying. 3. Phase-strict parser: the v3 union rejects kind-incompatible phases (a GC intent at pin-acquired), aborted for GC, and inconsistent phase/cursor combinations (complete requires a full cursor; intent requires cursor 0). Duplicate-target detection. 4. Locked explicit reconcile: runArchiveReconciliation acquires the maintenance lock before migrating/reconciling; the CLI calls only this (no racing create/GC planning, v2 migration, or pointer/catalog repair). 5. Nonmutating dry-runs: a shared planArchiveReconciliation planner is consumed by both dry-run and apply. gc --dry-run never reconciles (the reconcile-first call is apply-only); if an active op is present, dry-run reports the blocker rather than predicting a deletion set from unstable state. planArchiveReconciliation inspects journals WITHOUT migrating (migration is an apply action). 6. Missing pointers fail closed: a range with no active pointer is uncertain — excluded entirely, nothing targeted, no empty-string sentinel that strands the journal. 7. Faithful crash boundary: a dedicated awaited afterTargetProgress(index, total) seam replaces the duplicate during-removal label; the nonfinal-progress boundary (index < total-1) now exposes the premature-complete defect. The oracle verifies EXACT active/pointer/frozen-target/journal/catalog state after CLI reconcile. * fix(archives): strict locked reconcile wrapper — v2 action, fail-closed, shared plan (Gate 3b repair r3) Repair the five round-2 NO-GO blockers, all in the explicit reconciliation wrapper (no GC redesign requested): 1. Apply ALWAYS acquires the maintenance lock FIRST, then re-plans under it — never returns success for unsafe state without locking. 2. Valid v2 migration is an ACTION (migrate + reconcile), not a blocker. The planner reports needsMigration=true; apply migrates under the lock. A corrupt v2 intent is fail-closed (will not migrate). 3. Malformed/ambiguous/unreadable/unknown-debris state FAILS CLOSED (throws) — never reported as successful reconciliation. listActiveOperationIdsSafe (which silently filtered debris into absence) is replaced by a strict enumerator that surfaces unknown entries and read errors. 4. dry-run and apply share ONE strict nonmutating planner using the authoritative journal reader (parseArchiveOperationIntent), so the plan apply consumes is the same validated plan dry-run reports — no shallow raw reads. 5. Oracle tightened: a missing catalog fails (not skipped); the completed GC journal's target IDs are compared with the pre-crash frozen set (never re-expanded). Adds archive-reconcile.test.ts (8 tests) covering the locked-wrapper contract: v2-through-wrapper, malformed/multi/debris fail-closed, corrupt-v2 fail-closed, dry-run nonmutation with a v2 intent, apply-throws-on-unsafe, no-op success. * fix(archives): reconciliation as one locked protocol with concrete actions (Gate 3b r4) Replace the parallel weaker reader + metadata-only planner with one authoritative protocol per the round-3 review's explicit guidance: 1. ONE authoritative V2/V3 inspector (inspectActiveOperation in journal.ts): reuses listActiveOperationIds (root containment + no-symlink + per-entry debris/prefix/ validateArchiveId) + the guarded readIntent path (assertNoSymlinkSync + assertRealFileSync before readFileSync) + directory/record operationId binding. Reads v2 in-memory (no write); returns no-op | fail-closed | a validated snapshot. DELETE the parallel listActiveOperationIdsStrict + shallow planArchiveReconciliation reader from generation.ts. 2. migrateActiveIntentIfLegacy consumes the inspector: all pre-write checks (no-symlink, real-file, directory/record binding, clean v2 lift) run BEFORE any durableJson write. A symlinked intent, a mismatched ID, or a corrupt v2 record is detected before mutation. Closes the v2-mutates-before-detecting-identity/symlink blockers at the mutation site (defense in depth). 3. Concrete discriminated-union plan (ReconciliationPlan = no-op | fail-closed | create | gc) with EXACT ordered actions (CreateAction/GcAction) and preconditions. No opaque reconcile-create/reconcile-gc rediscovery: buildCreatePlan/buildGcPlan make the branch decision ONCE from observed topology; the executor dispatches per-action via named helpers, revalidating each precondition before mutation. 4. One locked protocol: both dry-run and apply acquire the maintenance lock before inspection; dry-run performs NO mutation (no migration, no reconcile) and a fail-closed plan returns NONZERO; apply executes only the planned actions and captures the terminal postcondition WHILE HOLDING THE LOCK (no post-unlock re-plan → no reporting TOCTOU). Adds state-space tests: v2/v3 × valid/mismatch/symlink/debris × dry-run/apply, lock contention (live owner blocks dry-run), symlinked-intent outside-target survival through both modes, v2-mismatch-never-rewritten-before-rejection. * feat(archives): pure reconciliation decision engine — the single transition table (Gate 3b r5) Introduce the ONE pure transition-table function (decideReconciliation) that is the sole branch logic for archive reconciliation. No I/O, no mutation. Both dry-run and apply will consume the SAME ReconciliationDecision; there is no second if-phase implementation to diverge (the defect that recurred across rounds 2–4). inspectReconciliationState (to follow in the wiring commit) will produce a ReconciliationInspection = null | FailClosed | ValidSnapshot, so validation failures (root/pin/scratch/pointer-CAS/owned-path/identity) enter the SAME decision function as valid state — every unsafe state is decided uniformly. The decision union: NoOp | FailClosed | CreateVerifyComplete | CreateAbortPrepublication | CreateFinishPublication | GcVerifyComplete | GcResume. Each carries exact identities (operationId, journalDigest), observed preconditions, affected signals/targets, and migrationRequired. Uses the canonical PHASE_ORDER/phaseAtLeast from journal.ts (no duplicated sequence). Includes the building+final impossible-topology gate. Preserves the lifted v2 record's exact phase (a v2 record can hold any create-eligible phase, not just intent). Exports PHASE_ORDER from journal.ts for reuse. 21 exhaustive transition-table tests cover every (kind, phase, topology) row and every FailClosed gate, asserting the exact decision. Post-promotion is always CreateFinishPublication (unconditional repair — selectActiveGeneration + rebuildCatalog — NOT phase-conditional verify). Complete phases are verify-only. Multi-signal GC carries explicit affectedSignals. * fix(archives): single reconciliation engine — one inspector, one pure decision, one executor (Gate 3b r5) Wire the pure decideReconciliation as the SOLE branch logic, deleting the parallel r4 action engine entirely. All reconciliation now flows through one path: inspectReconciliationState → decideReconciliation → execute branch via proven helpers - inspectReconciliationState: runs inspectActiveOperation (symlink-safe V2/V3 inspector) + the full proven validation (assertReconciliationRoots, validateReconciliationTopology, validateOwnedPinState). Any failure → FailClosed (zero mutation). For v2, lifts in-memory preserving the EXACT phase. Returns a complete validated ReconciliationSnapshot. - decideReconciliation: the ONE pure transition table (no I/O). Maps the snapshot to NoOp | FailClosed | CreateVerifyComplete | CreateAbortPrepublication | CreateFinishPublication | GcVerifyComplete | GcResume. - reconcileArchiveGenerationUnderLock: inspect → decide → (dry-run: return; apply: migrate-v2 if needed, then execute via the proven reconcileArchiveGeneration, capture postcondition under lock). - runArchiveReconciliation (CLI): acquires the lock, calls the under-lock function. FailClosed → nonzero in both modes. DELETE: buildCreatePlan, buildGcPlan, executeCreateAction, executeGcPlan, observeCreateTopology, quarantineBuildingStep, planFromInspection, the CreateAction/GcAction unions, the old ReconciliationPlan/ReconcileTopology types. The parallel engine is gone; there is one decision function. 238/238 tests pass (21 pure decision + 13 protocol + 204 existing). 0 lint. Typecheck, format, diff-check clean. * fix(archives): make the pure decision operative — executor switches on it directly (Gate 3b r6) The core fix: reconcileArchiveGeneration no longer independently re-branches. It inspects → decides → switches on the computed ReconciliationDecision, executing ONLY the branch-specific helpers for that decision. The decision IS the operative state machine. All three entry points (CLI, automatic create, automatic GC) call this same function — there is one call graph, one decision function, one executor. The inspection now includes branch-significant read-only preconditions: - create post-promotion: verifyPublishedGeneration (manifest SHA + shards); - create terminal: verifyCompletedOperationInvariants; - GC: assertReconciliationRoots + verifyCompletedGcInvariants for complete. So dry-run returns FailClosed for the same state apply rejects (only for valid topology; impossible topologies are gated by the decision function from snapshot fields before any branch-specific verification runs). 238/238 tests pass. 0 lint. Typecheck, format, diff-check clean. * fix(archives): complete preflight, split GC helpers, parity test (Gate 3b r7) Close the four round-6 bounded defects (no redesign): 1. Create post-promotion inspection now includes assertPointerConsistent — a conflicting pointer is FailClosed in dry-run AND apply (parity). 2. GC resume preflight: preflightGcTargets validates EVERY remaining target's source/tombstone topology, manifest/shard evidence, pointer CAS, and both-present detection read-only before the decision authorizes any mutation. Prevents partial deletion where target 1 succeeds but target 2 fails. 3. GC executor split: GcVerifyComplete → verifyCompleteAndArchiveGc (verify + archive, no repair); GcResume → resumeFrozenTargetsAndCompleteGc (resume + complete + verify + archive). The executor dispatches directly to each — no phase re-branch. The decision IS operative for GC as well as create. 4. Destination collision preflight: completed-op dir + quarantine dir existence checked in inspection before the decision authorizes earlier actions. Adds a dry-run/apply parity test: conflicting pointer → FailClosed in both modes, zero mutation, state preserved. 239/239 tests. 0 lint. Typecheck, format, diff-check clean. * fix(archives): exhaustive GC preflight + symlink-aware collision checks + parity tests (Gate 3b r8) Complete the round-7 GC preflight gaps: 1. preflightGcTargets now validates the ENTIRE frozen set: - prefix (index < cursor): source absent, tombstone absent, pointer CAS; - current/suffix (index >= cursor): all documented crash topologies with evidence validation. Prevents partial deletion when a prefix target is corrupted but suffix targets are valid. 2. revalidateTombstone: for source-absent/tombstone-present targets, proves the tombstone is a real non-symlinked directory with manifest/shard evidence matching the frozen set. A symlinked or tampered tombstone → FailClosed before any mutation. 3. Collision probes replaced with lstatSync-based pathExistsIncludingSymlinks: catches broken/dangling symlinks at deterministic destinations that existsSync misses. Adds two GC multi-target hostile parity tests: - cursor-ahead (completedTargets=1 but target 0 source exists): dry-run FailClosed, apply nonzero, zero mutation (byte-identical snapshot). - source-absent/tombstone-present with symlinked tombstone: dry-run FailClosed, apply nonzero. 241/241 tests. 0 lint. Typecheck, format, diff-check clean. * fix(archives): lstat-aware topology classifier + strict suffix/prefix states + ENOENT-only collision (Gate 3b r9) Close the three round-8 bounded defects: 1. Replace ALL existsSync source/tombstone topology checks in gc.ts with a shared lstat-aware tri-state classifier (classifyPath): absent only on ENOENT; real-directory/real-file validated; symlink or non-ENOENT error → fail closed. Used in preflightGcTargets, collectOneTarget, and verifyCompletedGcInvariants. Dangling symlinks are no longer treated as absent. 2. Enforce distinct prefix/current/suffix topology in preflightGcTargets: - prefix (i < cursor): source absent, tombstone absent, pointer CAS; - current (i === cursor): documented crash topologies (source/tomb states); - suffix (i > cursor): source-present + tombstone-absent + evidence + CAS only. A suffix target that is already absent or tombstoned → FailClosed (impossible out-of-order mutation). 3. pathExistsIncludingSymlinks: returns false ONLY on ENOENT; every other lstat error propagates as fail-closed (permission, I/O, malformed path). Adds three hostile parity tests with complete durable-state snapshots: - dangling source symlink → FailClosed, zero mutation (byte-identical); - dangling tombstone symlink → FailClosed, zero mutation; - impossible suffix (target 2 absent while cursor=0) → FailClosed, zero mutation. 244/244 tests. 0 lint. Typecheck, format, diff-check clean. * fix(archives): enforce root-aware reconciliation topology * feat(archive): harden calibration and config loading * docs(archive): operator/architecture guide and draft PR narrative (R9) Expand docs/local-telemetry-archives.md from a user overview into the full operator and architecture reference required by the plan: the six signals and their event-time columns, the complete directory layout, the manifest (v3), active pointer (v1), and catalog.jsonl formats, the full tuning field reference with defaults and validation constraints, the calibration workflow (candidate matrix, worst-case aggregation, margin-inside-ceiling, disjoint held-out validation, no-recommendation cases), recovery and the single reconciliation decision function, the GC lifecycle, and the complete off-happy-path catalog with the untouched/debris/reconcile/intervention classification. Add docs/pr-local-telemetry-archives.md as the fresh long-form draft PR description (what/why, dependency on PR #129, resource and adoption implications, happy path, failure outcomes, validation evidence, known limits, follow-up). This is written against the branch head and does not reuse prior stale draft PR materials. * fix(archive): strict manifests, session calibration, semantic config, volume binding Architectural repair for the R9/R10 fresh-review NO-GO (D-024), in one commit atop 5d891e83. Manifest v3 strictness (finding 1). parseArchiveGenerationManifest is now exact-key (assertExactOwnKeys over MANIFEST_KEYS) and rejects unknown top-level fields. The tuning block is parsed by parseTuningRecord, which enforces positive integers and the same cross-field invariants as the writer. Omitting tuningConfig now fails closed instead of normalizing to null; zero/invalid tuning is rejected. tuningConfig accepts the two known config format identities (legacy v1 + verified v2) since the manifest stores an opaque hash-bound identity; only the loader refuses v1 for new writes. Calibration intent-retention wedge (finding 2). reconcileCalibration catches a resolveCheckpoint failure and, when the record is a provably inert intent (phase intent, pinPath null, every exact derived checkpoint/pin/scratch/ sample resource absent via classifyArchivePathSync), retires the record without resolving the retired source. Any ambiguity preserves the record. Calibration/config acceptance contract (finding 3). Calibration is now a parent-owned session: the parent resolves ONE checkpoint, acquires ONE pin, and threads the exact checkpointId+fingerprint to every child. Children (assertCalibrationSession) refuse current/previous, bind to the parent's operation/checkpoint/fingerprint, verify the parent pin is live, and clean only their own sample (cleanupCalibrationSample); the parent owns full reconcileCalibration in a finally. Held-out results, worstCase, comparisons, tolerances, and a heldOutAttempts log are retained in the recommendation and written to the config. Tolerances are one canonical exported constant (HELD_OUT_TOLERANCES) and the loader recomputes the comparison, rejecting a forged withinTolerance. The two non-candidate knobs are DERIVED (deriveTargetChunkBytes = max(4*maxShardBytes, reserve+maxShardBytes); minFreeSpaceReserve = budget.freeSpaceReserve) and the derivation formula + recomputed effective block are re-verified by the loader. Config format bumped to 2; confidence must be high to load. A new calibrate-session command (open/close) makes the parent session lifecycle explicit so a single child or SIGKILL probe can run against an open session and an operator can retire a wedged one. Environment + volume identity at create (findings 5,7). archive create --config enforces the recorded mapleVersion/chdbVersion/schemaFingerprint/ executionUser/platform/arch/cpuModel/cpuCount/totalMemoryBytes and the archive-volume fsid/type + canonical path against the live host (assertCalibrationEnvironment), and re-checks the volume identity immediately before publication (beforePublicationVolumeRecheck seam). archiveVolumeIdentity no longer climbs ancestors: it requires an existing real non-symlink canonical directory. R9 documentation corrections. Recovery, quarantine, catalog, calibration cleanup, operation paths, free-space, list output, volume, and capacity claims are corrected to match actual behavior; both docs pass oxfmt. Tests/probes: 303/303 unit (+2 env/volume negative tests; manifest and semantic-rewrite coverage). The calibration probes are rewritten to the session model: the crash probe covers a SIGKILLed sampling child and the retired-intent wedge; the calibrate probe adds a forged-volume rejection check. * fix(archive): canonical tolerances, strict pin identity, larger disjoint held-out scope Closes the four R9/R10 repair-review blockers (D-024, round 2) in one commit atop 3a7ac948. Findings 1, 2, and 4 plus the core of finding 3. Canonical tolerances + triggers (finding 1). HELD_OUT_TOLERANCES and RECALIBRATION_TRIGGERS now live once in config.ts (CALIBRATION_HELD_OUT_TOLERANCES / CALIBRATION_RECALIBRATION_TRIGGERS) and the loader requires the document's heldOut.tolerances and recalibrationTriggers to exactly match them (exactJson), so a document can no longer redefine its own comparison policy to pass a 1000x wall regression. Tolerances tightened so every metric is < 1.0. Strict parent-pin identity (finding 2). assertCheckpointPinIdentity is one shared strict validator (exact keys, formatVersion, pinId, checkpointId, purpose regex, parseable createdAt) used by both releaseCheckpointPin and assertCalibrationSession. assertCalibrationSession now parses the derived pin and requires exact id/checkpoint/purpose, so a substituted same-path regular file (e.g. '{}') is no longer accepted as the live parent pin. Larger, persisted, auditable held-out scope (finding 3). The held-out window is now STRICTLY LARGER than training and disjoint: training covers ordered rows [0, sampleRows); held-out covers [sampleRows, sampleRows + 2*sampleRows). Every training and held-out result carries a persisted sample scope ({checkpointId, checkpointManifestFingerprint, rangeDate, role, startRow, requestedRows, rowCount}) emitted by the child (the authoritative source) and attached by the parent. The config document records a samplePolicy (trainingRows, heldOutMultiplier=2, heldOutRows, training/heldOut windows), and the loader re-verifies: all scopes bind to ONE checkpoint/range; training is role training, startRow 0, requestedRows=budget.sampleRows; held-out is role held-out, startRow=sampleRows, requestedRows=heldOutRows; rowCount matches metrics.rowCount; failed results record no scope. Representative-data confidence now matches all four candidate fields (isSameCalibrationCandidate), not just writerThreads. Metric-coherence validation rejects forged compressionRatio/throughput; positive freeSpaceReserve enforced; recalibration triggers exact. R9 documentation (finding 4). Both docs corrected to config v2, 306 tests, the canonical <1.0 tolerances, null-selected throws (no config written), the new samplePolicy/scope fields, and the larger disjoint held-out window. The stale 'separate validation report' and 'low-confidence config may be written' claims are removed. Native calibrate probe asserts the larger persisted held-out scope and samplePolicy for every result. Unit tests add five hostile scope-forgery cases (forged checkpoint, role, non-disjoint startRow, rowCount mismatch, samplePolicy multiplier) and a held-out-larger-and-disjoint property. 306/306 unit; typecheck/lint/format/diff-check clean. * fix(archive): larger persisted held-out scope + hybrid size-scaled comparison Closes finding 3 (held-out contract) on top of the round-2 repair. The held-out sample is now strictly LARGER than training and disjoint: training covers ordered rows [0, sampleRows); held-out covers [sampleRows, sampleRows + 2*sampleRows) (heldOutSampleRows = HELD_OUT_SAMPLE_MULTIPLIER * sampleRows). Every training and held-out result carries a persisted sample scope ({checkpointId, checkpointManifestFingerprint, rangeDate, role, startRow, requestedRows, rowCount}) emitted by the child and attached by the parent; the config records a samplePolicy and the loader proves every scope binds to ONE checkpoint/range and that the two windows are disjoint and correctly sized. Hybrid predicted-vs-observed comparison (per the calibrator contract): because a 2x-larger held-out naturally takes ~2x wall time and bytes, wallMs and physicalBytes predictions are rescaled by heldOut.logicalBytes / training.logicalBytes before the two-sided check; throughput and compressionRatio (size-invariant rates) compare directly; peak RSS and peak temp disk compare as absolute peaks (NOT scaled by row count — they are process peaks, not per-row costs). The raw values, scale ratio, training/held-out logical bytes, adjusted predictions, and comparison results are all persisted; the loader recomputes the ratio and every comparison from the recorded evidence using the fixed canonical tolerances only (no run-specific widening). comparePredictedObserved gains an optional sizeScaling param mirrored by the loader's expectedComparisons. Native calibrate probe asserts the larger persisted held-out scope and samplePolicy for every result (training=10 held-out=20, disjoint, single source) and the loop closes through the hybrid comparison. Unit tests add hostile forged-scale-ratio and forged-scope cases plus a held-out-larger-and-disjoint property. 306/306 unit; typecheck/lint/format/diff-check clean. * fix(archive): enforce held-out cardinality and release fault seam * fix(archive): per-signal like-for-like held-out hybrid comparison Closes the final D-026 contract: the held-out comparison is now PER-SIGNAL and like-for-like, replacing the cross-signal aggregate ratio that could mask a single-signal regression. Each signal's held-out result is paired with the same candidate's TRAINING result for that signal (compareHeldOutPerSignal). wallMs/physicalBytes are scaled by THAT signal's own heldOut.logicalBytes/training.logicalBytes ratio; throughput and compressionRatio are compared directly (size-invariant rates); peak RSS and peak temp disk are compared as absolute peaks. The attempt passes only when ALL SIX signals pass; cross-signal a…
This was referenced Sep 28, 2026
JeremyFunk
added a commit
that referenced
this pull request
Sep 28, 2026
…ind (#3) Follow-up to registering the OpenInference integration for framework vendor ids. Once smolagents and agno spans decoded `input.value` / `output.value` as message fields, an agent run's arguments became the turn's user row: smolagents' `CodeAgent.run` anchor rendered `{"task": "Hi! Briefly introduce yourself.", "stream": false, ...}` as the user message, and agno's plain-text agent input came before the system row and suppressed the model call's own user row. `input.value` and `output.value` are whatever the span's function took and returned. They leave the source lists and are read in the refine: on a model call (or a span naming no kind) they are the request and reply as before; on any other kind they are messages only when they unwrap to a role-carrying message list (LangChain's `{messages}` chain input); on a TOOL span they fill the call's arguments and result when the GenAI attributes are absent. The schema-as-arguments replacement moves under the TOOL branch. The envelope normalisers move to `ai-messages.ts` so the vendor refine can unwrap the same way the default one does. The integration's doc comment no longer claims the gateway already stamps langchain / llamaindex for OpenInference scopes; that takes effect with the ingest detection batch. Seen in: smolagents (docs_smolagents_a/b `assistant.run` a92540159081b72d), agno (docs_agno_a).
JeremyFunk
added a commit
that referenced
this pull request
Sep 29, 2026
* fix(agent-sessions): keep plain-text values of JSON-typed GenAI attributes
Symptom: tool results that are plain strings or numbers (OpenAI Agents
`deleted /tmp/scratch-notes.txt`, `391`, a 503 error text, a handoff
summary) read "result: not captured", and Mastra's plain-text
`gen_ai.system_instructions` never reached the transcript.
Cause: the mapper decoded every `json` catalog field with JSON.parse and
kept only objects and arrays, so anything else was dropped. The
convention types `gen_ai.tool.call.arguments` / `.result` as any value.
Fix: JSON null still means not captured; objects, arrays and JSON strings
decode as before; everything else (plain text, numbers, booleans,
truncated JSON) is kept as the text it arrived as.
Seen in: openai-agents (docs_openai-agents_a, spans bf977b7555424c18,
bda436b865ce7528, 189ec7a1efd8b75b) and mastra (docs_mastra_a).
* fix(agent-sessions): read OpenAI-format tool_calls and tool messages
Symptom: a model call whose reply only calls a tool showed no output row
and no tool-call request (LiteLLM a1 turn-3 `chat` span 7072bdfe10b317a7);
in the proxy capture the assistant tool-call message was missing from the
input history (6 of 7 messages shown). OpenInference dual-write histories
lose the same messages.
Cause: `messageParts` (span-detail.ts) read only `parts` / `content`, so an
OpenAI chat message with `content: null` and a `tool_calls` array had no
parts and was skipped.
Fix: read `tool_calls` (`{ id, function: { name, arguments } }`) as
tool_call parts beside the content, and a `{ role: "tool", tool_call_id }`
message as the tool_result for that call id, which also lets the session
result index match it.
Seen in: litellm (docs_litellm_a, docs_litellm_proxy), OpenInference TS
(docs_openai-agents_ts input.value).
* fix(agent-sessions): unwrap the OpenRouter Broadcast message envelopes
Symptom: every OpenRouter Broadcast `LLM Generation` span showed two raw
JSON messages (span a04a15db8c490dbb: input/user = the whole
`{"messages":[...]}` blob, output/assistant = the `{"completion":"",
"reasoning":...}` blob); the 16 turns read "Segment 1..16" with no turn
label and the session had no title.
Cause: Broadcast sends `gen_ai.prompt` as `{ messages }` and
`gen_ai.completion` as `{ completion, reasoning, rawRequest }`. The mapper
passed both through, and every reader (turn label, transcript, span
detail) expects the documented message array.
Fix: the default integration's refine unwraps a `{ messages: [...] }`
envelope on either message field, and turns `{ completion, reasoning }`
into one assistant message with a reasoning part and a text part. Shape
keyed, so OpenInference's `input.value` request body benefits too.
Seen in: openrouter (capture `openrouter`, 1595 prompts / 1594
completions in these shapes).
* fix(agent-sessions): decode chat.completion output and llm.finish_reason
Symptom: no model reply and no tool-call request on any OpenInference TS
model span (docs_openai-agents_ts generations fa84aae0ec2fa65d,
9e250b5c18100b86), so the last reply of each turn was absent; the
reply-length check was skipped because no finish reason was read.
Cause: the OpenInference integration reads `llm.output_messages`, a key
OpenInference never writes (it flattens to `llm.output_messages.N.*`),
then `output.value`, which OpenInference TS sets to the raw
`[{"object":"chat.completion","choices":[...]}]`; the message walker
skips entries without role/parts/content. `llm.finish_reason` was not
mapped.
Fix: the default refine turns a `chat.completion` object (alone or in an
array) into its choices' messages, carrying each choice's finish reason
onto the message; the OpenInference integration reads
`llm.finish_reason` as the response finish reason. The flattened keys are
left unread: `output.value` carries the same reply, and reading them would
need a prefix projection on the span read.
Seen in: openai-agents TS (docs_openai-agents_ts, vendor
unknown:openinference); OpenRouter's test generation uses the same shape.
* fix(agent-sessions): read finish reasons off the output messages
Symptom: the reply-length check was always skipped for Strands sessions
("No model call recorded a finish reason"), and call meta lines showed no
stop reason, although every `chat` span recorded one.
Cause: the mapper reads finish reasons only from
`gen_ai.response.finish_reasons` (and aliases). Strands (Python and TS)
writes them only where the convention also allows them, as
`finish_reason` on each `gen_ai.output.messages` entry.
Fix: when the span carries no finish reason attribute, the default refine
collects the non-empty `finish_reason` values of its output messages. The
attribute still wins when present; a streamed chunk's empty reason is
ignored. This also picks up the reasons the chat.completion unwrap
carries onto each message.
Seen in: strands (docs_strands_a, docs_strands_g, docs_strands_ts).
* fix(agent-sessions): read a streamed Google ADK reply as its aggregate
Symptom: a streamed Google ADK turn rendered its reply as 8 assistant
messages, 7 one-token chunks followed by the full text (trace
1feecb5b2b06551a5c1ce516083635cc, span 32be028f453b9eb6).
Cause: ADK writes every streamed chunk into `gen_ai.output.messages` with
an empty `finish_reason`, then the aggregate with the real one, and the
message walker rendered each entry.
Fix: output messages whose `finish_reason` is the empty string are
dropped when a finished message is present to stand for them. Chunks with
no aggregate behind them are kept.
Seen in: google-adk (docs_google-adk_a, `generate_content` span
8f487cfcae3893f5 in the capture).
* fix(agent-sessions): unwrap the LangChain ToolMessage tool result
Symptom: every LangChain tool result rendered as the serialised
ToolMessage (`{"type": "tool", "data": {"content": "391",
"additional_kwargs": {}, ..., "status": "success"}}`) instead of what the
tool returned.
Cause: OpenInference's LangChain instrumentor dual-writes the whole
ToolMessage into `gen_ai.tool.call.result`, and the mapper passed it
through.
Fix: the default refine replaces a result of exactly that shape
(`type: "tool"` with a `data` object of `type: "tool"` carrying
`content`) by `data.content`. Any other object stays whole.
Seen in: langchain (docs_langchain_a, docs_langchain_b: every TOOL span).
* fix(agent-sessions): decode OpenInference keys for framework vendor ids
Symptom: the session detail page showed no messages, usage or cost for
spans the gateway stamps as dspy, agno, smolagents, openai_agents_sdk or
crewai unless the app turned on OpenInference's GenAI dual-write
(`TraceConfig(enable_genai_semconv=True)`, the workaround in all five
guides). The list read the same spans' `llm.*` keys, so list and detail
disagreed (agno: list $0.0042, detail none).
Cause: the OpenInference integration was registered only under
`openinference-openai` and `unknown:openinference`
(ai-vendors.ts AI_VENDOR_INTEGRATIONS). The gateway stamps the framework
id for `openinference.instrumentation.<framework>` scopes, and those ids
fell through to the default integration, which reads only `gen_ai.*`.
Fix: register the OpenInference integration under agno, crewai, dspy,
openai_agents_sdk and smolagents, plus langchain and llamaindex, whose
OpenInference scopes the gateway may fingerprint as those ids. Their
native spans carry no OpenInference keys, and the canonical `gen_ai.*`
keys keep priority, so nothing a native span decodes changes.
Seen in: agno, crewai, dspy, openai-agents, smolagents (docs_* captures).
* fix(agent-sessions): prefer real tool arguments over a dual-written schema
Symptom: crewai and llamaindex tool spans showed the tool's JSON schema
(`{"properties": {"city": {...}}, "required": ["city"], "type":
"object"}`) as the call's arguments instead of `{"city": "Berlin"}`.
Cause: OpenInference's GenAI dual-write (`enable_genai_semconv`) copies
`tool.parameters` into `gen_ai.tool.call.arguments` for these
instrumentors (an upstream bug), and the canonical key wins over the
dialect's keys.
Fix: the OpenInference integration's refine replaces
`gen_ai.tool.call.arguments` by `input.value` when its raw value is
exactly the span's `tool.parameters`. Exact equality, so arguments that
merely resemble a schema are never touched. The refine context gains
`read(field, key)`, the mapper's own decoder, so a refine can read another
key the way the source lists do.
Seen in: crewai (docs_crewai_a `get_weather.run` 0a327721d936828a),
llamaindex (docs_llamaindex_a `FunctionTool.acall` 8dc64cacfbc534dc).
* fix(agent-sessions): read a bare OpenAI message output as the reply (#14)
Follow-up to the OpenAI tool_calls fix. smolagents' model spans write
`output.value` as one OpenAI message object (`{role, content: null,
tool_calls: [...]}`), not an array, so the walker rendered it as raw JSON
and the tool-call request was lost.
The default refine now wraps a single record carrying a `role` into a
one-element message array; the tool_calls reading then applies.
Seen in: smolagents (docs_smolagents_a `OpenAIModel.generate`
db63f17aa5554acf).
* fix(agent-sessions): read OpenInference input/output values by span kind (#3)
Follow-up to registering the OpenInference integration for framework
vendor ids. Once smolagents and agno spans decoded `input.value` /
`output.value` as message fields, an agent run's arguments became the
turn's user row: smolagents' `CodeAgent.run` anchor rendered
`{"task": "Hi! Briefly introduce yourself.", "stream": false, ...}` as the
user message, and agno's plain-text agent input came before the system
row and suppressed the model call's own user row.
`input.value` and `output.value` are whatever the span's function took and
returned. They leave the source lists and are read in the refine: on a
model call (or a span naming no kind) they are the request and reply as
before; on any other kind they are messages only when they unwrap to a
role-carrying message list (LangChain's `{messages}` chain input); on a
TOOL span they fill the call's arguments and result when the GenAI
attributes are absent. The schema-as-arguments replacement moves under
the TOOL branch.
The envelope normalisers move to `ai-messages.ts` so the vendor refine can
unwrap the same way the default one does. The integration's doc comment
no longer claims the gateway already stamps langchain / llamaindex for
OpenInference scopes; that takes effect with the ingest detection batch.
Seen in: smolagents (docs_smolagents_a/b `assistant.run` a92540159081b72d),
agno (docs_agno_a).
* fix(agent-sessions): read OpenInference's flattened llm.*_messages keys (#30)
Follow-up to the chat.completion decoding. Every Python OpenInference
instrumentor writes model messages only as flattened keys
(`llm.input_messages.N.message.role`, `….contents.M.message_content.text`,
`….tool_calls.M.tool_call.function.name`, `….tool_call_id`); the
unflattened `llm.input_messages` / `llm.output_messages` keys the
integration listed never appear. Without the GenAI dual-write, dspy,
smolagents and llamaindex model calls fell back to `input.value`, a Python
repr (`ChatMessage(role=<MessageRole.USER: 'user'>, ...)`), and agno and
crewai sessions had no transcript and no turn labels at all.
The span read now also projects key prefixes an integration declares
(`refinePrefixes`, beside the existing prompt-variable family), and the
OpenInference refine rebuilds the flattened families as OpenAI chat
messages. Preference per message field: the GenAI dual-write, then
`input.value` / `output.value` when it is a message list (the exact
capture), then the flattened keys, then the bare value on a model call.
With tool calls now decoded from OpenInference outputs, a call whose tool
span recorded no call id (OpenInference never stamps one) rendered twice:
from the span and from the message. The transcript's name fallback for
unmatched `tool_call` parts now also covers parts whose id no tool span
carries.
Seen in: dspy, smolagents, llamaindex, agno, crewai, langchain
(docs_* captures, with and without the GenAI dual-write), openai-agents TS.
* test(agent-sessions): refresh the integration SQL baseline for the span projection
The session span read now projects llm.finish_reason, tool.parameters and
the OpenInference llm.*_messages prefixes; the recorded SQL baseline
follows.
* fix(agent-sessions): drop streamed chunks only beside a finished aggregate (#20)
Review follow-up: the chunk filter dropped empty-finish-reason messages
whenever any other message was present, including one that names no
finish reason at all, so an unfinished stream beside an ordinary message
lost its chunks. Chunks are now dropped only when a message with a
non-empty finish reason is there to stand for them.
* fix(agent-sessions): keep a completion whose choices carry no message (#30)
Review follow-up: a chat.completion whose choices hold no `message` (a
streamed chunk's `delta`, an empty `choices`) was unwrapped to an empty
list, which erased the captured output. The capture is now left as it
was unless the choices yield at least one message.
* fix(agent-sessions): show a failed tool span's error as its result (#30)
Follow-up to the tool-row dedupe. Once an OpenInference tool call renders
once, from its span, a failed span that recorded no result attribute and
no call id lost the only error text the session had (the history echo
under the message's call id): crewai_b, langchain_b and llamaindex_b
rendered result "not captured ... Whether it succeeded is unknown" for a
call that failed with "transport data service unavailable (503)".
A failed tool span with no captured or echoed result now falls back to
its failure text (`rawFailureText`: the status message unless generic).
The transcript never tells a reader a failed call's outcome is unknown:
when even that is absent the note says the span failed without recording
a result or an error message.
* fix(agent-sessions): drop empty flattened content parts (#30)
llamaindex writes `llm.output_messages.N.message.contents.0.message_content.text = ""`
beside its tool calls. The empty value is skipped, which left a bare
`{type: "text"}` part that rendered as raw assistant text (7 rows in
docs_llamaindex_a/b without the GenAI dual-write). A content part carrying
nothing but its type is now dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.