diff --git a/.agents/skills/maple-telemetry-conventions/SKILL.md b/.agents/skills/maple-telemetry-conventions/SKILL.md index 1cea2bd305..762f41bff4 100644 --- a/.agents/skills/maple-telemetry-conventions/SKILL.md +++ b/.agents/skills/maple-telemetry-conventions/SKILL.md @@ -1,12 +1,12 @@ --- name: maple-telemetry-conventions -description: Maple's OpenTelemetry conventions — custom span attribute keys (`maple.*` vendor namespace, `query.context`, `db.query.*`, `result.*`, `cache.*`, `tenant.*`), Title Case status codes (`Ok`/`Error`/`Unset`), resource attribute dual-emit (`deployment.environment` + `deployment.environment.name`), span kinds, Tinybird MV pre-extracted columns, loop-prevention filters, and sampling. Use whenever writing or reviewing instrumentation code in any language (TypeScript, Rust, Python) in this repo — adding `setAttribute`/`setAttributes`/`record`/`#[instrument(fields(...))]` calls, setting span status, configuring an OTLP exporter, defining a new resource attribute, or wiring a new query through `WarehouseQueryService.compiledQuery()`. +description: "Maple's OpenTelemetry conventions: custom span attribute keys (`maple.*` vendor namespace, `query.context`, `db.query.*`, `result.*`, `cache.*`, `tenant.*`), Title Case status codes (`Ok`/`Error`/`Unset`), resource attribute dual-emit (`deployment.environment` + `deployment.environment.name`), span kinds, Tinybird MV pre-extracted columns, loop-prevention filters, and sampling. Use whenever writing or reviewing instrumentation code in any language (TypeScript, Rust, Python) in this repo: adding `setAttribute`/`setAttributes`/`record`/`#[instrument(fields(...))]` calls, setting span status, configuring an OTLP exporter, defining a new resource attribute, or wiring a new query through `WarehouseQueryService.compiledQuery()`." version: "1.0.0" --- # Maple Telemetry Conventions -Reference for the **language-agnostic** OpenTelemetry conventions Maple uses across TypeScript (`apps/api`, Cloudflare workers in `packages/effect-sdk/`), Rust (`apps/ingest`), and future Python services. These conventions are load-bearing — Tinybird materialized views pre-extract certain attribute keys into columns, dashboards filter on Title Case status strings, and sampling-aware throughput math relies on the `SampleRate` column. Use the exact attribute spellings here in every language. +Reference for the language-agnostic OpenTelemetry conventions Maple uses across TypeScript (`apps/api` and the other Workers, via `@maple-dev/effect-sdk` in `packages/effect-sdk/`), Rust (`apps/ingest`), and any future Python service. These conventions are load-bearing. Tinybird materialized views pre-extract some attribute keys into columns, dashboards filter on Title Case status strings, and throughput math depends on the `SampleRate` column. Use the exact attribute spellings here in every language. ## When to apply @@ -15,36 +15,36 @@ Reference for the **language-agnostic** OpenTelemetry conventions Maple uses acr - Wiring a new query through `WarehouseQueryService.compiledQuery()` (the `context` and `profile` options become span attributes) - Configuring an OTLP exporter, tracer provider, or resource builder - Introducing a new pre-extracted MV column or a new vendor attribute under `maple.*` -- Reviewing a PR that touches `apps/api/src/services/WarehouseQueryService.ts`, `apps/ingest/src/main.rs`, `apps/api/src/app.ts`, `packages/effect-sdk/src/cloudflare/`, or `packages/domain/src/tinybird/materializations.ts` +- Reviewing a PR that touches `packages/query-engine/src/execution/executor.ts`, `apps/ingest/src/main.rs`, `apps/ingest/src/otel.rs`, `apps/api/src/http/api-observability.ts`, `packages/effect-sdk/src/cloudflare/`, `packages/infra/src/cloudflare/worker-telemetry.ts`, or `packages/domain/src/tinybird/materializations.ts` ## Index -- `rules/span-attributes.md` — Master reference of every custom attribute key Maple emits, grouped by namespace, with file:line citations. -- `rules/status-and-kind.md` — Title Case status code rule (`Ok`/`Error`/`Unset`) and span kind conventions (`Server` / `Client` / `Internal`). -- `rules/resource-attributes.md` — `service.*` identity, `deployment.environment.name` resolution order, the deprecated `deployment.environment` dual-emit and read-side coalesce, and `maple_org_id`. -- `rules/language-bindings.md` — Parallel TypeScript / Rust / Python snippets that emit the same attribute keys. -- `rules/mv-first-class-columns.md` — Which span and resource attributes Tinybird MVs pre-extract into columns (and the rule for adding new ones). -- `rules/service-map-attribution.md` — Required span and resource attributes for the service map to render edges, runtime icons, and platform badges. Includes the canonical `peer.service` registry. -- `rules/loop-prevention.md` — The three guards that prevent Maple's self-traffic from creating a feedback loop: API `TracerDisabledWhen`, ingest loopback guard, sampling. +- `rules/span-attributes.md`: the main custom attribute keys Maple emits, grouped by namespace, with the file that sets each. +- `rules/status-and-kind.md`: Title Case status codes (`Ok`/`Error`/`Unset`), the server-span 4xx rule, and span kinds (`Server` / `Client` / `Internal`). +- `rules/resource-attributes.md`: `service.*` identity, `deployment.environment.name` resolution order, the deprecated `deployment.environment` dual-emit and read-side coalesce, and `maple_org_id`. +- `rules/language-bindings.md`: parallel TypeScript / Rust / Python snippets that emit the same attribute keys. +- `rules/mv-first-class-columns.md`: which span and resource attributes Tinybird MVs pre-extract into columns, and the rule for adding new ones. +- `rules/service-map-attribution.md`: what the service map needs to draw service edges, database nodes, runtime icons, and platform badges, plus the `peer.service` naming registry. +- `rules/loop-prevention.md`: the guards that keep Maple's self-traffic from feeding back on itself: the API `TracerDisabledWhen` filter, the ingest loopback guard, and sampling. ## Quick reference | Topic | Rule | |---|---| | Status codes | Always Title Case: `"Ok"`, `"Error"`, `"Unset"`. Never `OK`, `ERROR`, `SUCCESS`, `FAILED`. | -| Vendor namespace | Custom attributes go under `maple.*`. Sub-namespaces: `maple.ingest.*`, `maple.cloudflare.*`. | +| Vendor namespace | Custom attributes go under `maple.*`. Sub-namespaces include `maple.ingest.*`, `maple.cloudflare.*`, `maple.query.*`. | | Standard semconv | Use OTel semconv keys verbatim: `service.name`, `http.request.method`, `db.system.name`, `error.type`. | | Org identity | `orgId` (camelCase) in TypeScript spans, `maple.org_id` (dotted) in Rust spans. Don't unify until MVs migrate. | -| Deployment env | Emit `deployment.environment.name` (we dual-emit the deprecated `deployment.environment` too). **Read** both via `DEPLOYMENT_ENV_SQL` / `deploymentEnvExpr` — never a bare map lookup. | -| Warehouse SQL spans | Every span from `WarehouseQueryService.executeSql` carries `db.system.name`, `peer.service`, `db.query.text`, `db.query.fingerprint`, `db.duration_ms`, `result.rowCount`, `orgId`, `query.context`, `query.profile`. Legacy spans (pre 2026-06) use `db.statement*`/`db.system`; warehouse readers coalesce both. | -| Service map | Outbound spans need `peer.service` (HTTP/RPC) or `db.system.name` (DB) on a `Client`/`Producer` span. Resource attrs need `process.runtime.name`, `cloud.platform`, `maple.sdk.type` for runtime icon + platform badge. See `rules/service-map-attribution.md`. | -| Loop prevention | Never remove `HttpMiddleware.TracerDisabledWhen` (apps/api/src/app.ts:169-175) or the ingest loopback guard (apps/ingest/src/main.rs:499-514). | +| Deployment env | Emit `deployment.environment.name` (our SDKs dual-emit the deprecated `deployment.environment` too). **Read** both via `DEPLOYMENT_ENV_SQL` / `deploymentEnvExpr`, never a bare map lookup. | +| Warehouse SQL spans | Every `WarehouseQueryService.executeSql` span (a `Client` span) carries `db.system.name`, `peer.service`, `db.query.text`, `db.query.fingerprint`, `db.duration_ms`, `result.rowCount`, `orgId`, `query.context`, and `query.profile` when set. Legacy spans (pre 2026-06) use `db.statement*`/`db.system`; warehouse readers coalesce both. | +| Service map | Service-to-service edges come from joining a `Client`/`Producer` span to its child `Server`/`Consumer` span in another service, so propagate trace context and set span kinds. Database nodes need `db.system.name` (plus `db.namespace`) on a `Client`/`Producer` span. Runtime icon and platform badge need `process.runtime.name`, `cloud.platform`, `maple.sdk.type` on the resource. See `rules/service-map-attribution.md`. | +| Loop prevention | Never remove `HttpMiddleware.TracerDisabledWhen` (`apps/api/src/http/api-observability.ts`) or the ingest loopback guard (`init_tracing` in `apps/ingest/src/main.rs`). | ## Canonical references (do not modify from this skill) -- `apps/api/src/services/WarehouseQueryService.ts:441-510` — `executeSql` span emission (the canonical example for TS). -- `apps/ingest/src/otel.rs` — Resource builder + platform detection + forward-span helper for Rust. The canonical example for Rust resource and outbound-span attribution. -- `apps/ingest/src/main.rs` — `handle_signal` and `handle_cloudflare_logpush` (search for `tracing::info_span!` with `otel.kind = "server"`) — Server-kind span macros for OTLP inbound. -- `apps/api/src/app.ts:169-175` — `TracerDisabledWhen` filter. -- `packages/effect-sdk/src/cloudflare/index.ts` — `MapleCloudflareSDK` tracer setup. -- `packages/domain/src/tinybird/materializations.ts` — MV `SELECT` lists that pre-extract attribute keys into columns. +- `packages/query-engine/src/execution/executor.ts`: `WarehouseQueryService.executeSql` span emission, the canonical TS example. +- `apps/ingest/src/otel.rs`: resource builder (`build_resource`), platform detection, and the client-span helpers (`forward_client_span`, `export_client_span`). The canonical Rust example for resource and outbound-span attribution. +- `apps/ingest/src/main.rs`: `handle_signal` and `handle_cloudflare_logpush` open the Server-kind `tracing::info_span!` for inbound OTLP and Logpush. +- `apps/api/src/http/api-observability.ts`: the `TracerDisabledWhen` filter and header redaction list. +- `packages/effect-sdk/src/cloudflare/index.ts`: `MapleCloudflareSDK` tracer setup. Maple's own Workers wrap it with `WorkerTelemetry` in `packages/infra/src/cloudflare/worker-telemetry.ts`. +- `packages/domain/src/tinybird/materializations.ts`: MV `SELECT` lists that pre-extract attribute keys into columns. diff --git a/.agents/skills/maple-telemetry-conventions/rules/language-bindings.md b/.agents/skills/maple-telemetry-conventions/rules/language-bindings.md index 191406669a..d2d2c851e9 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/language-bindings.md +++ b/.agents/skills/maple-telemetry-conventions/rules/language-bindings.md @@ -1,14 +1,14 @@ -# Language bindings — emit the same attributes everywhere +# Language bindings: emit the same attributes everywhere -The whole point of this skill is that **attribute keys are identical across languages**. Snippets below show how to emit the canonical `executeSql`-style annotation block in each language so any reader can copy the pattern into TS, Rust, or (forthcoming) Python. +Attribute keys are identical across languages. The snippets below emit the canonical `executeSql`-style annotation block in TS, Rust, and (forward-looking) Python. --- -## TypeScript — Effect + `@effect/opentelemetry` +## TypeScript: Effect + `@maple-dev/effect-sdk` -Maple's TS code wraps spans through Effect. Two patterns: +Maple's TS code creates spans through Effect's tracer; `@maple-dev/effect-sdk` (`packages/effect-sdk`) installs the tracer and OTLP exporter. Two patterns: -### Pattern A: `Effect.fn(name)` — declarative span on a function +### Pattern A: `Effect.fn(name)`, a declarative span on a function ```typescript import { Effect } from "effect" @@ -17,7 +17,7 @@ const executeQuery = Effect.fn("MyService.executeQuery")(function* ( tenant: TenantContext, sql: string, ) { - // Span attributes — annotate the current span (created by Effect.fn). + // Annotate the current span (created by Effect.fn). yield* Effect.annotateCurrentSpan("orgId", tenant.orgId) yield* Effect.annotateCurrentSpan("tenant.userId", tenant.userId) yield* Effect.annotateCurrentSpan("db.system.name", "clickhouse") @@ -30,9 +30,9 @@ const executeQuery = Effect.fn("MyService.executeQuery")(function* ( }) ``` -Status is set automatically: if the effect fails, the tracer records `Error`; on success, `Ok`. **Do not call `setStatus` manually.** +Status is set automatically: a failed effect records `Error`, a successful one `Ok`. Do not set status manually. Pass `{ kind: "client" }` as the second argument to `Effect.fn` when the function makes a network call. -### Pattern B: `Effect.withSpan(name, { attributes })` — wrap an inline effect +### Pattern B: `Effect.withSpan(name, { attributes })`, wrapping an inline effect ```typescript yield* doWork.pipe( @@ -46,21 +46,21 @@ yield* doWork.pipe( ) ``` -Use this when you need attributes set at span-open time rather than via `annotateCurrentSpan` calls inside. +Use this when attributes are known at span-open time instead of via `annotateCurrentSpan` calls inside. -### Cloudflare Workers — `MapleCloudflareSDK` +### Cloudflare Workers: `MapleCloudflareSDK` -Workers get their tracer from `MapleCloudflareSDK` in `packages/effect-sdk/src/cloudflare/index.ts`. The setup configures the OTLP exporter and resource — once it's installed via `telemetry.layer`, you use the same `Effect.fn` / `Effect.annotateCurrentSpan` / `Effect.withSpan` API as elsewhere. +Workers get their tracer from `MapleCloudflareSDK` in `packages/effect-sdk/src/cloudflare/index.ts`, which configures the OTLP exporter and resource. Maple's own Workers install it through `WorkerTelemetry` (`packages/infra/src/cloudflare/worker-telemetry.ts`). After that, use the same `Effect.fn` / `Effect.annotateCurrentSpan` / `Effect.withSpan` API as elsewhere. ### Canonical TS example -The reference implementation is `apps/api/src/services/WarehouseQueryService.ts:441-510` (the `executeSql` function). Read it whenever you're unsure how to structure a new query span. +The reference implementation is `executeSql` in `packages/query-engine/src/execution/executor.ts`. Read it when unsure how to structure a new query span. --- -## Rust — `tracing` + `tracing-opentelemetry` +## Rust: `tracing` + `tracing-opentelemetry` -Rust code uses the `tracing` macro to declare spans and field names. Special field names (`otel.name`, `otel.kind`, `otel.status_code`) drive OTel semantics; the rest become custom attributes. +Rust code declares spans and fields with `tracing` macros. The reserved fields (`otel.name`, `otel.kind`, `otel.status_code`, `otel.status_description`) drive OTel semantics; the rest become attributes. ### Pattern A: declarative span via `tracing::info_span!` @@ -92,15 +92,16 @@ match result { Err(err) => { span_handle.record("http.response.status_code", err.status_code()); span_handle.record("error.type", err.kind()); + // Inbound handlers: use otel_status_for_rejection (see status-and-kind.md). span_handle.record("otel.status_code", "Error"); } } ``` Key idioms: -- **Quote field names with dots** (`"http.request.method"`) — bare identifiers can't contain dots in Rust syntax. -- **Declare empty fields up front** with `tracing::field::Empty` and `record` them later. This keeps the field list visible at the span declaration site. -- **Use `%expr`** to record the `Display` impl, `?expr` for `Debug`. Use plain `field = value` for primitive types. +- **Quote field names that contain dots** (`"http.request.method"`). The reserved `otel.*` fields are the exception: the macro accepts them bare. +- **Declare empty fields up front** with `tracing::field::Empty` and `record` them later. `record` silently ignores a field the span did not declare, and the declaration site then lists every field. +- **Use `%expr`** to record the `Display` impl and `?expr` for `Debug`. Use plain `field = value` for primitives. ### Pattern B: `#[instrument]` attribute macro @@ -125,13 +126,13 @@ async fn resolve_connector(state: &AppState, connector_id: &str) -> Result - request.url === "/health" || - request.method === "OPTIONS" || - /\.(png|ico|jpg|jpeg|gif|css|js|svg|webp|woff2?)(\?.*)?$/i.test(request.url), +export const ApiObservabilityLive = Layer.mergeAll( + Layer.succeed( + HttpMiddleware.TracerDisabledWhen, + (request: { url: string; method: string }) => + request.url === "/health" || + request.method === "OPTIONS" || + OAUTH_CALLBACK_PATH.test(request.url) || + /\.(png|ico|jpg|jpeg|gif|css|js|svg|webp|woff2?)(\?.*)?$/i.test(request.url), + ), + Layer.succeed(Headers.CurrentRedactedNames, [/* credential and webhook-signature headers */]), ) ``` -Disables span creation for: +It disables the automatic server span for: | Request kind | Why | |---|---| | `/health` exact path | Pinged constantly by orchestrators; would dominate trace volume | -| `OPTIONS` (any path) | CORS preflights — high-volume, low-value | -| Static asset extensions (`.png`, `.ico`, `.jpg`, `.jpeg`, `.gif`, `.css`, `.js`, `.svg`, `.webp`, `.woff`, `.woff2`) | Asset fetches — produce no useful trace data | +| `OPTIONS` (any path) | CORS preflights: high volume, low value | +| OAuth callbacks (`/api/integrations/*/callback`, `/oauth/chat/*/callback`) | Their query string carries a live authorization `code`, which the server span would record in `url.full`. Each handler opens its own span with safe attributes instead. | +| Static asset extensions (`.png`, `.ico`, `.jpg`, `.jpeg`, `.gif`, `.css`, `.js`, `.svg`, `.webp`, `.woff`, `.woff2`) | Asset fetches produce no useful trace data | + +The same layer sets `Headers.CurrentRedactedNames`, so credential headers and webhook signatures never land on the span as `http.request.header.*`. ### Rules -- **Do not extend this filter** to skip paths that you find noisy — silence is debugging-hostile. Investigate the volume source first. +- **Do not extend this filter** to hide paths that look noisy. Silence is debugging-hostile. Find the volume source first. The OAuth callbacks are the exception because tracing them would leak a credential. - **Do not remove paths from this filter** without a replacement. Health-check spans alone can double Maple's daily trace count. -- **Be careful adding spans to internal high-frequency paths** like auth token validation that runs on every request. CLAUDE.md calls this out. +- **Don't add spans to high-frequency internal paths** such as token validation that runs on every request (CLAUDE.md calls this out). --- -## Guard 2: Rust ingest — OTLP loopback guard +## Guard 2: Rust ingest, OTLP loopback guard -**Where:** `apps/ingest/src/main.rs:499-514` +**Where:** `init_tracing` in `apps/ingest/src/main.rs`. `init_metrics` and `init_usage_metrics` apply the same check. ```rust +let deployment_env = resolve_deployment_env(); let forward_explicit = std::env::var("INGEST_FORWARD_OTLP_ENDPOINT").is_ok(); let skip_dev = deployment_env == "development" && !forward_explicit; let loopback = endpoint_loopback_to_self(forward_endpoint, bind_port); @@ -48,8 +56,7 @@ let loopback = endpoint_loopback_to_self(forward_endpoint, bind_port); if skip_dev || loopback { if loopback { eprintln!( - "INGEST_FORWARD_OTLP_ENDPOINT={forward_endpoint} resolves to this server's bind port {bind_port}; \ - skipping OTel exporter to avoid recursion" + "INGEST_FORWARD_OTLP_ENDPOINT={forward_endpoint} resolves to this server's bind port {bind_port}; skipping OTel exporter to avoid recursion" ); } // Init tracing-subscriber WITHOUT the OTel layer. @@ -60,51 +67,53 @@ if skip_dev || loopback { Two refusals: -1. **Loopback detection** — if `INGEST_FORWARD_OTLP_ENDPOINT` resolves to ingest's own bind port (via hostname/IP comparison in `endpoint_loopback_to_self()` at `main.rs:588+`), the OTel exporter setup is skipped. Otherwise ingest would forward its own spans to itself recursively. -2. **Dev no-op** — in `development` deployment with no explicit `INGEST_FORWARD_OTLP_ENDPOINT` set, the OTel exporter is skipped (logs-only). Prevents accidental local trace floods. +1. **Loopback detection:** if `INGEST_FORWARD_OTLP_ENDPOINT` resolves to ingest's own bind port (hostname/IP comparison in `endpoint_loopback_to_self()`), the OTel exporter is skipped. Otherwise ingest would forward its own spans to itself recursively. +2. **Dev no-op:** in the `development` environment with no explicit `INGEST_FORWARD_OTLP_ENDPOINT`, the OTel exporter is skipped (stdout logs only). This prevents accidental local trace floods. + +The gateway's own telemetry goes straight to the downstream collector, not back through its ingest routes. ### Rule -If you add another OTLP-emitting service that lives at the same hostname as the ingest gateway, write a similar loopback guard. Don't rely on configuration alone to prevent loops. +If you add another OTLP-emitting service that shares a host with the ingest gateway, give it a similar loopback guard. Don't rely on configuration alone to prevent loops. --- ## Guard 3: Sampling -For high-QPS deployments where unsampled tracing would saturate ingest, use OTel's parent-based ratio sampler: +The ingest gateway applies a per-org trace sampling policy (`trace_sample_ratio`, with options to always keep error and slow spans). It records the ratio as `maple.ingest.sampling_ratio` and the dropped count as `maple.ingest.sampled_dropped`. + +A service that exports through a standard OTel SDK can also head-sample with the parent-based ratio sampler: ```bash OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 ``` -This keeps 10% of root traces and inherits the sampling decision through children — so a trace is either fully kept or fully dropped (no broken trace trees). +This keeps 10% of root traces and children inherit the decision, so a trace is either fully kept or fully dropped. `@maple-dev/effect-sdk` does not read these variables. ### `SampleRate` column -Maple's traces datasource has a first-class `SampleRate` column (computed from `SpanAttributes['SampleRate']` or W3C TraceState `th:` threshold). Sampling-aware aggregations should multiply or sum by `SampleRate` to recover unbiased counts: +The `traces` datasource has a first-class `SampleRate` column (the `SAMPLE_RATE_EXPR` default in `packages/domain/src/tinybird/datasources.ts`). It comes from `SpanAttributes['SampleRate']` when that is `>= 1`, else the W3C TraceState `th:` threshold, else `1.0`. Sampling-aware aggregations sum `SampleRate` to recover unbiased counts: ```sql -- WRONG: under-counts when sampled -SELECT count() FROM spans WHERE … +SELECT count() FROM traces WHERE … -- RIGHT: scales each sampled span back up -SELECT sum(SampleRate) FROM spans WHERE … +SELECT sum(SampleRate) FROM traces WHERE … ``` See `docs/sampling-throughput.md` for the full pattern. ### Rule -If you're adding a new throughput / count widget on the traces datasource, use `sum(SampleRate)` unless you have a specific reason to undercount. Span-count aggregates without `SampleRate` weighting will silently drift downward as sampling ratio decreases. +When adding a throughput or count widget on the traces datasource, use `sum(SampleRate)` unless you have a specific reason to under-count. Unweighted span counts drift downward as the sampling ratio drops. --- -## What is **not** a loop-prevention guard - -To avoid confusion: +## What is not a loop-prevention guard -- **OTLP batch export** is async but is **not** a loop prevention measure — it just keeps the export off the request path. Removing batch export would slow requests, not cause a loop. -- **The OTLP export itself does not go through the API** — it ships directly to the ingest gateway from each service. CLAUDE.md confirms this: "The OTLP export itself does NOT go through the API (it goes directly to the ingest gateway), so it won't create recursive traces." +- **OTLP batch export** keeps export off the request path. Removing it would slow requests; it would not cause a loop. +- **OTLP export does not go through the API.** Each service ships directly to the ingest gateway, so exporting does not create API spans. -So the three guards above are the entirety of Maple's loop-prevention strategy. Touching any of them requires a thought-through replacement. +The guards above are the whole of Maple's loop-prevention strategy. Changing any of them requires a thought-through replacement. diff --git a/.agents/skills/maple-telemetry-conventions/rules/mv-first-class-columns.md b/.agents/skills/maple-telemetry-conventions/rules/mv-first-class-columns.md index 8da2f701c4..c290eafa08 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/mv-first-class-columns.md +++ b/.agents/skills/maple-telemetry-conventions/rules/mv-first-class-columns.md @@ -1,113 +1,149 @@ # Tinybird MV pre-extracted columns -Some span and resource attributes get **extracted into typed columns** at write time by Tinybird materialized views. These columns are first-class: they're indexed, can be used in `WHERE` / `GROUP BY` without a Map lookup, and drive dashboard widgets directly. Everything else stays in the `SpanAttributes` / `ResourceAttributes` Map columns. +Some span and resource attributes are extracted into typed columns at write time by Tinybird materialized views. These columns are first-class: they sit in sorting keys and projections, work in `WHERE` / `GROUP BY` without a Map lookup, and drive dashboard widgets directly. Everything else stays in the `SpanAttributes` / `ResourceAttributes` Map columns. -This file is the inventory — when you write a query that filters on one of these keys, prefer the column name. When you add a new MV column, follow the source-attribute consistency rule at the bottom. +This file is the inventory. When a query filters on one of these keys, use the column. When you add an MV column, follow the source-spelling rule at the bottom. -Source: `packages/domain/src/tinybird/materializations.ts` +Source: `packages/domain/src/tinybird/materializations.ts` (search for the MV name). Read `docs/warehouse-rollups.md` before adding an MV. --- ## `service_map_spans_mv` -Lightweight projection of trace spans for the service dependency map. +Projection of `Client`/`Producer`/`Server`/`Consumer` spans for the service map. The parent side of the edge join. -| Column | Extracted from | Line | -|---|---|---| -| `PeerService` | `SpanAttributes['peer.service']` | `materializations.ts:246` | -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:247` | +| Column | Extracted from | +|---|---| +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | -## `service_map_children_mv` +No attribute decides the edge. The downstream service comes from the join (see `service_map_edges_hourly` below). -Spans with a parent, for child-of-edge analysis in the service map. +## `service_map_children_mv` -| Column | Extracted from | Line | -|---|---|---| -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:315` | +`Server`/`Consumer` spans that have a parent. The child side of the edge join. -## `service_map_edges_hourly_mv` +| Column | Extracted from | +|---|---| +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | -Pre-aggregated client-to-peer edges. Pre-filters to spans with `peer.service != ''`. +## `service_map_edges_hourly` (not an MV extraction) -| Column | Extracted from | Line | -|---|---|---| -| `TargetService` | `SpanAttributes['peer.service']` | `materializations.ts:341` | -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:342` | +Hourly service-to-service edges. `ServiceMapRollupService` (`packages/backend/src/services/dashboards/ServiceMapRollupService.ts`) fills it on a schedule by joining each `Client`/`Producer` span to its child `Server`/`Consumer` span (`serviceMapEdgeJoinQuery` in `packages/query-engine/src/ch/queries/service-map.ts`). An incremental MV cannot express that cross-span join. `service_map_edges_hourly_ingest_mv` only forwards the rollup's rows from the Null ingest source into the aggregate target. -## `service_overview_spans_mv` +| Column | Derived from | +|---|---| +| `SourceService` | parent span's `ServiceName` | +| `TargetService` | child span's `ServiceName` | -Hourly service-overview rollup. +## `service_map_db_edges_hourly_mv` -| Column | Extracted from | Line | -|---|---|---| -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:277` | -| `CommitSha` | `ResourceAttributes['vcs.ref.head.revision']` | `materializations.ts:278` (see file for the field) | +Hourly service-to-database edges from `Client`/`Producer` spans with a database system set. -## `service_platforms_hourly_mv` +| Column | Extracted from | +|---|---| +| `DbSystem` | `DB_SYSTEM_ATTR_SQL`: `db.system.name`, fallback `db.system` | +| `DbNamespace` | `DB_NAMESPACE_ATTR_SQL`: `db.namespace` -> `db.name` -> `server.address` -> `net.peer.name` ('' when none) | +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | -Per-service hosting-platform attributes for the service map's runtime-icon resolver. Pre-aggregates as `max()` of each attribute. +Both fragments live in `packages/domain/src/tinybird/db-query-shape-sql.ts`. `service_map_db_query_shapes_hourly_mv` uses the same `DbSystem` / `DbNamespace` expressions. -| Column | Extracted from | Line | -|---|---|---| -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:414` | -| `K8sCluster` | `max(ResourceAttributes['k8s.cluster.name'])` | `materializations.ts:415` | -| `K8sPodName` | `max(ResourceAttributes['k8s.pod.name'])` | `materializations.ts:416` | -| `K8sDeploymentName` | `max(ResourceAttributes['k8s.deployment.name'])` | `materializations.ts:417` | -| `CloudPlatform` | `max(ResourceAttributes['cloud.platform'])` | `materializations.ts:418` | -| `CloudProvider` | `max(ResourceAttributes['cloud.provider'])` | `materializations.ts:419` | +## `service_external_edges_hourly_mv` -Additionally extracts (see file): `faas.name`, `sdk.type`, `process.runtime.name`. +Hourly edges from `Client`/`Producer` spans without `db.system.name` to external targets (the service-detail Dependencies tab). -## `error_events_mv` +| Column | Extracted from | +|---|---| +| `TargetType` | `messaging` if a messaging destination or `messaging.system` is set, else `rpc` if `rpc.service`/`rpc.system` is set, else `http` | +| `TargetSystem` | `messaging.system` or `rpc.system` ('' for http) | +| `TargetName` | messaging destination (`MESSAGING_DESTINATION_SQL`), `rpc.service`, or for http `server.address` -> `http.host` -> `url.authority` | +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | -Unwraps the first OTel `exception` event from `EventsName` / `EventsAttributes` Maps. +## `service_overview_spans_mv`, `service_overview_hourly_mv`, `service_overview_minutely_mv` -| Column | Extracted from | Line | -|---|---|---| -| `ExceptionType` | first `exception` event's `exception.type` (fallback `StatusMessage`) | `materializations.ts:496-498` | -| `ExceptionMessage` | first `exception` event's `exception.message` | `materializations.ts:498` | -| `ExceptionStacktrace` | first `exception` event's `exception.stacktrace` | `materializations.ts:499` | -| `TopFrame` | (computed from stacktrace) | `materializations.ts` | -| `FingerprintHash` | cityHash64 of grouping keys | `materializations.ts:489` | -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:525` | +Service-overview projection and rollups over entry-point spans. -## `trace_list_mv` +| Column | Extracted from | +|---|---| +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | +| `CommitSha` | `ResourceAttributes['vcs.ref.head.revision']` | +| `ServiceNamespace` | `ResourceAttributes['service.namespace']` | -Trace list optimized for the trace search UI. Pre-filters to entry-point spans. +## `service_platforms_hourly_mv` -| Column | Extracted from | Line | -|---|---|---| -| `HttpMethod` | `SpanAttributes['http.method']` (fallback `http.request.method`) | `materializations.ts:596` | -| `HttpRoute` | `SpanAttributes['http.route']` (fallbacks `url.path`, `http.target`) | `materializations.ts:597` | -| `HttpStatusCode` | `SpanAttributes['http.status_code']` (fallback `http.response.status_code`) | `materializations.ts:598` | -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:599` | +Per-service hosting-platform attributes for the service map's runtime-icon and platform-badge resolver. Each column is `max()` of the attribute, so any non-empty value in the hour wins. + +| Column | Extracted from | +|---|---| +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | +| `K8sCluster` | `ResourceAttributes['k8s.cluster.name']` | +| `K8sPodName` | `ResourceAttributes['k8s.pod.name']` | +| `K8sDeploymentName` | `ResourceAttributes['k8s.deployment.name']` | +| `K8sStatefulSetName` | `ResourceAttributes['k8s.statefulset.name']` | +| `K8sDaemonSetName` | `ResourceAttributes['k8s.daemonset.name']` | +| `K8sNamespaceName` | `ResourceAttributes['k8s.namespace.name']` | +| `CloudPlatform` | `ResourceAttributes['cloud.platform']` | +| `CloudProvider` | `ResourceAttributes['cloud.provider']` | +| `FaasName` | `ResourceAttributes['faas.name']` | +| `MapleSdkType` | `ResourceAttributes['maple.sdk.type']` | +| `ProcessRuntimeName` | `ResourceAttributes['process.runtime.name']` | + +## `error_events_mv` / `error_events_by_time_mv` + +Built from `traces WHERE StatusCode = 'Error'` with one shared `SELECT`. Unwraps the first OTel `exception` event from the `EventsName` / `EventsAttributes` arrays. + +| Column | Extracted from | +|---|---| +| `ExceptionType` | first `exception` event's `exception.type`, with fallbacks (see the file) | +| `ExceptionMessage` | first `exception` event's `exception.message` | +| `ExceptionStacktrace` | first `exception` event's `exception.stacktrace` | +| `TopFrame` | computed from the stacktrace | +| `FingerprintHash` | `cityHash64` of org, service, type, top-3 normalized frames, and message signature | +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | + +The fingerprint normalization is mirrored in `packages/domain/src/tinybird/fingerprint.ts`. Read `docs/error-issue-lifecycle.md` before changing it. + +## `trace_list_mv_mv` (target `trace_list_mv`) + +Trace list for the trace search UI, pre-filtered to entry-point spans. + +| Column | Extracted from | +|---|---| +| `HttpMethod` | `SpanAttributes['http.method']`, fallback `http.request.method` | +| `HttpRoute` | `SpanAttributes['http.route']`, fallbacks `url.path`, `http.target` | +| `HttpStatusCode` | `SpanAttributes['http.status_code']`, fallback `http.response.status_code` | +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | +| `ServiceNamespace` | `ResourceAttributes['service.namespace']` | ## `traces_aggregates_hourly_mv` Hourly trace-shape rollup. -| Column | Extracted from | Line | -|---|---|---| -| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | `materializations.ts:778` | +| Column | Extracted from | +|---|---| +| `DeploymentEnv` | `DEPLOYMENT_ENV_SQL` | + +It also groups by `ServiceName`, `SpanName`, `SpanKind`, `StatusCode`, and `IsEntryPoint`. + +## `logs_aggregates_hourly_mv` -Plus dimension keys pre-aggregated on the trace itself: `ServiceName`, `SpanName`, `SpanKind`, `StatusCode`, `IsEntryPoint`. +Extracts `DeploymentEnv` through `DEPLOYMENT_ENV_SQL` and `ServiceNamespace` from `ResourceAttributes['service.namespace']`. -## `logs_aggregates_hourly_mv` (referenced) +## Other MVs -Same pattern — extracts `DeploymentEnv` through `DEPLOYMENT_ENV_SQL`. See `materializations.ts:809`. +`service_operations_*`, `ai_trace_index_mv`, `span_metrics_calls_hourly_mv`, and the attribute key/value catalogs also extract columns. Read their `SELECT` in `materializations.ts` before filtering on them. --- -## Cardinal rule — consistent source spellings +## Cardinal rule: consistent source spellings -**If you add a new pre-extracted MV column, also emit the source attribute with that exact spelling.** Don't introduce a parallel spelling that the MV won't match. +**If you add a pre-extracted MV column, emit the source attribute with exactly that spelling.** Don't introduce a parallel spelling the MV won't match. -Example: if you add a new MV column `HttpUserAgent` that extracts `SpanAttributes['http.user_agent']`, every service emitting spans **must** use `http.user_agent` as the attribute key. Don't have one service emit `http.user_agent` and another `userAgent` — only one will populate the column. +Example: if a new MV column `HttpUserAgent` extracts `SpanAttributes['user_agent.original']`, every service must emit `user_agent.original`. If one service emits `user_agent.original` and another `userAgent`, only one populates the column. -Corollary: every MV here extracts `DeploymentEnv` through the shared `DEPLOYMENT_ENV_SQL` fragment (`packages/domain/src/tinybird/semconv-renames.ts`), which coalesces `deployment.environment.name` over the deprecated `deployment.environment` — as `MESSAGING_DESTINATION_SQL` does for `messaging.destination(.name)` in the external-edge rollup. A bare lookup on either key alone silently materializes an empty environment for half the instrumentation in the wild. See `rules/resource-attributes.md`. +Corollary: every MV extracts `DeploymentEnv` through the shared `DEPLOYMENT_ENV_SQL` fragment (`packages/domain/src/tinybird/semconv-renames.ts`), which coalesces `deployment.environment.name` over the deprecated `deployment.environment`. `MESSAGING_DESTINATION_SQL` does the same for `messaging.destination(.name)` in the external-edge MV. A bare lookup on either key alone materializes an empty environment for half the instrumentation in the wild. See `rules/resource-attributes.md`. ## When NOT to extract into a column -- **Low cardinality keys you'll never group by** — just leave them in the Map. -- **Per-request user data that changes per span** — leave it in `SpanAttributes`. The Map column is queryable; extraction is for fields the dashboard hits on every request. -- **Anything still being designed** — pre-extraction is a one-way door once you've backfilled. Wait until the attribute name is stable. +- **Keys you will never filter or group by.** Leave them in the Map. +- **Per-request data that varies per span.** Leave it in `SpanAttributes`. The Map column is queryable; extraction is for fields dashboards hit on every request. +- **Anything still being designed.** Pre-extraction is a one-way door once backfilled. Wait until the attribute name is stable. diff --git a/.agents/skills/maple-telemetry-conventions/rules/resource-attributes.md b/.agents/skills/maple-telemetry-conventions/rules/resource-attributes.md index e674ec0136..e34c07bd8f 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/resource-attributes.md +++ b/.agents/skills/maple-telemetry-conventions/rules/resource-attributes.md @@ -1,26 +1,28 @@ # Resource attributes -Resource attributes are set **once per process** on the OTel `Resource` and apply to every span and log that process emits. They identify the service and its deployment environment. The canonical reference implementation is the Rust ingest gateway's `init_tracing()` in `apps/ingest/src/main.rs:516-540`. +Resource attributes are set once per process on the OTel `Resource` and apply to every span, log, and metric the process emits. They identify the service and its deployment environment. The canonical Rust implementation is `build_resource` in `apps/ingest/src/otel.rs`. The TypeScript equivalent is `resolveResourceFromEnv` in `packages/effect-sdk/src/server/resource.ts`, shared by the server and Cloudflare SDKs. ## Required identity attributes | Key | Type | Source | Example | Notes | |---|---|---|---|---| -| `service.name` | string | static (per service) | `"ingest"`, `"api"`, `"web"` | Canonical name. For the Rust gateway this is hard-coded to `"ingest"` and replaces the legacy Prometheus `ingest-proxy` label (CLAUDE.md: "**canonical — replaces the legacy Prometheus-scrape `ingest-proxy` label**"). | -| `service.namespace` | string | static (per service) | `"core"` | Optional logical group for `service.name`. Extracted to the `ServiceNamespace` projection column (`service_overview_spans`, `trace_list_mv`, `logs_aggregates_hourly`) and surfaced in the services table + trace/log filters. Maple's own services all use **`core`** (`maple-api`, `maple-agent`, `alerting`, `maple-cli`, the Rust ingest gateway, `maple-web`, `maple-landing`, `scraper`, `electric-sync`). Set via the SDK `serviceNamespace` config field (TS) or `ResourceConfig.service_namespace` (Rust); **not** defaulted — external apps choose their own. | -| `service.version` | string | build-time | `env!("CARGO_PKG_VERSION")` in Rust, package version in TS | Semantic version; used for release-correlation. | -| `service.instance.id` | string | runtime | `uuid::Uuid::new_v4().to_string()` per process | Per-process UUID generated at startup. Lets dashboards distinguish replicas. | +| `service.name` | string | static (per service) | `"ingest"`, `"maple-api"`, `"alerting"` | Canonical name. The Rust gateway hard-codes `"ingest"`. | +| `service.namespace` | string | static (per service) | `"core"` | Optional logical group for `service.name`. Extracted to the `ServiceNamespace` column (`service_overview_spans`, `service_overview_hourly`/`_minutely`, `trace_list_mv`, `logs_aggregates_hourly`) and surfaced in the services table and trace/log filters. Maple's own services all use `core`: Workers get it from `workerTelemetryConfig` (`packages/infra/src/cloudflare/worker-telemetry.ts`), the Rust gateway from `ResourceConfig.service_namespace`. Set it through the SDK `serviceNamespace` config field in TS. It is not defaulted; external apps choose their own. | +| `service.version` | string | build time | `env!("CARGO_PKG_VERSION")` in Rust, package version in TS | Used for release correlation. | +| `service.instance.id` | string | runtime | one UUID per process | Generated at startup. Lets dashboards distinguish replicas. | -## Deployment environment — read both, emit both +Both SDKs also stamp `vcs.repository.url.full` and, when the deploy platform provides a commit SHA, `vcs.ref.head.revision` (extracted to the `CommitSha` column). -OTel renamed this attribute; the registry lists `deployment.environment.name` as stable and plain `deployment.environment` as deprecated ("Replaced by `deployment.environment.name`"). +## Deployment environment: read both, emit both + +OTel renamed this attribute. The registry lists `deployment.environment.name` as stable and plain `deployment.environment` as deprecated ("Replaced by `deployment.environment.name`"). | Key | Status | |---|---| -| `deployment.environment.name` | OTel-canonical — emit this one from new code | -| `deployment.environment` | Deprecated — Maple still emits it, and still **reads** it | +| `deployment.environment.name` | OTel-canonical. Emit this one from new code. | +| `deployment.environment` | Deprecated. Maple still emits it, and still **reads** it. | -**Reading is the rule that matters.** Anything that pulls the environment out of `ResourceAttributes` — an MV body, a query-engine filter, a facet — uses the shared coalesce, never a bare map lookup: +**Reading is the rule that matters.** Anything that pulls the environment out of `ResourceAttributes` (an MV body, a query-engine filter, a facet) uses the shared coalesce, never a bare map lookup: ```ts import { DEPLOYMENT_ENV_SQL, deploymentEnvExpr } from "@maple/domain/tinybird/semconv-renames" @@ -31,66 +33,63 @@ coalesce(nullIf(ResourceAttributes['deployment.environment.name'], ''), ResourceAttributes['deployment.environment']) ``` -A bare lookup on either key alone drops the environment for half the fleet: our own SDKs dual-emit, a current OTel SDK sends only `.name`, an older one only the legacy key. `packages/domain/src/tinybird/semconv-renames.test.ts` fails if any MV extracts `DeploymentEnv` some other way. The same module carries `MESSAGING_DESTINATION_SQL` for the `messaging.destination` -> `messaging.destination.name` rename; any future rename Maple *keys on* belongs there too. +A bare lookup on either key drops the environment for half the fleet. Our own SDKs dual-emit, a current OTel SDK sends only `.name`, and an older one sends only the legacy key. `packages/domain/src/tinybird/semconv-renames.test.ts` fails if any MV extracts `DeploymentEnv` another way. The same module carries `MESSAGING_DESTINATION_SQL` for the `messaging.destination` -> `messaging.destination.name` rename. Any future rename Maple *keys on* belongs there too. -Source: `apps/ingest/src/main.rs:526-538` +Source: `build_resource` in `apps/ingest/src/otel.rs` ```rust -.with_attribute(OtelKeyValue::new( - "deployment.environment.name", - deployment_env.clone(), -)) -// Dual-emit the deprecated `deployment.environment` key: the MVs -// coalesce both spellings since migration 0020, but rows materialized -// before it — and pre-0020 BYO-ClickHouse schemas — still read the -// legacy key alone. -.with_attribute(OtelKeyValue::new("deployment.environment", deployment_env)) +KeyValue::new("deployment.environment.name", cfg.deployment_env.clone()), +// Dual-emit the deprecated `deployment.environment` key (see below). +KeyValue::new("deployment.environment", cfg.deployment_env), ``` +The TS SDK dual-emits the same pair whenever an environment resolves (`resolveResourceFromEnv`). + ### Resolution order (priority) -The value comes from the first env var that resolves, in this order: +The value comes from the first env var that resolves: -1. `MAPLE_ENVIRONMENT` — set by alchemy via `resolveDeploymentEnvironment(stage)` (`selfObservabilityEnv` in `packages/infra/src/env.ts`) -2. `RAILWAY_ENVIRONMENT_NAME` — Railway's free runtime label -3. `DEPLOYMENT_ENV` — manual override of last resort +1. `MAPLE_ENVIRONMENT`: set by alchemy from `resolveDeploymentEnvironment(stage)` (`selfObservabilityEnv` in `packages/infra/src/env.ts`) +2. `RAILWAY_ENVIRONMENT_NAME`: Railway's runtime label +3. `DEPLOYMENT_ENV`: manual override of last resort 4. Default: `"development"` -See `apps/ingest/src/main.rs:492-495` for the exact precedence: +`resolve_deployment_env` in `apps/ingest/src/main.rs` has the exact precedence: ```rust -let deployment_env = std::env::var("MAPLE_ENVIRONMENT") +std::env::var("MAPLE_ENVIRONMENT") .or_else(|_| std::env::var("RAILWAY_ENVIRONMENT_NAME")) .or_else(|_| std::env::var("DEPLOYMENT_ENV")) - .unwrap_or_else(|_| "development".to_string()); + .unwrap_or_else(|_| "development".to_owned()) ``` -Any new service must follow the same priority order. Don't read `NODE_ENV` or `ENV` — they're not part of Maple's convention. +The TS SDK resolves the same chain, with the same default, in `environment` (`packages/effect-sdk/src/server/config.ts`). An explicit `environment` config field wins over it. Any new service must follow the same order. Don't read `NODE_ENV` or `ENV`. ### When can the dual-emit be dropped? -ClickHouse migration 0020 moved every MV onto the coalesce, so our own rollups no longer depend on the legacy emit. Two things still do: rows those MVs materialized **before** 0020 (they carry whatever the legacy key held and age out with the target TTL), and BYO-ClickHouse orgs whose schema is still pre-0020. Dropping the legacy emit is a follow-up gated on both, not a free cleanup. +ClickHouse migration 0020 (`packages/domain/src/clickhouse/migrations/0020_semconv_key_renames.ts`) moved every MV onto the coalesce, so our own rollups no longer depend on the legacy emit. Two things still do: rows those MVs materialized before 0020 (they age out with the target TTL), and BYO-ClickHouse orgs whose schema is still pre-0020. Dropping the legacy emit is a follow-up gated on both. -## `maple_org_id` — internal service identity +## `maple_org_id`: the owning org | Key | Type | Source | Default | Notes | |---|---|---|---|---| -| `maple_org_id` | string | env `MAPLE_INTERNAL_ORG_ID` | **none — required** | Tags Maple's own services (so their self-traces don't pollute customer trace lists). Set per process. | +| `maple_org_id` | string | the resolved ingest key, or env `MAPLE_INTERNAL_ORG_ID` for the gateway itself | none (required) | The `OrgId` every row is scoped to. | -There used to be a `"internal"` fallback here. It was removed (2026-08-23) because it never disabled anything: an unset value shipped a full stream of traces, logs and metrics into the warehouse under an `OrgId` no org owns and no UI can read — which is how the AWS ingest fleet looked like it had lost its self-telemetry for a day. The gateway now refuses to boot without it (`AppConfig::from_env`). +For telemetry that arrives through the ingest gateway, the gateway strips any client-supplied `org_id` / `maple_org_id` resource attribute and sets `maple_org_id` from the resolved ingest key (`apps/ingest/src/main.rs`). No SDK or Worker sets it by hand; Maple's own Workers get their org from the self-observability ingest key. -Source: `apps/ingest/src/main.rs` — `AppConfig::from_env` (parse) and `build_resource` call sites in `init_tracing` / `init_metrics` / `init_usage_metrics`. +The gateway's own telemetry skips that path: it exports straight to the downstream collector. So `build_resource` stamps `maple_org_id` from `MAPLE_INTERNAL_ORG_ID`, and `AppConfig::from_env` refuses to boot without it. There used to be an `"internal"` fallback. It was removed (2026-08-23) because an unset value shipped the gateway's traces, logs, and metrics under an `OrgId` no org owns and no UI can read. -This is intentionally **not** `maple.org_id` (the vendor-namespaced span attribute used for the *customer's* org). Resource-level `maple_org_id` is the org running this Maple service; span-level `maple.org_id` is the org sending data through it. The two underscore-vs-dot spellings prevent confusion in trace search. +This is intentionally not `maple.org_id`. Resource-level `maple_org_id` is the org that owns the row. The span-level `maple.org_id` on the gateway's own spans is the customer org whose request the gateway is handling. The underscore-vs-dot spellings keep them apart in trace search. -## Effect SDK — Cloudflare workers +## Effect SDK: Cloudflare Workers -For TypeScript services running in Cloudflare Workers, resource attributes are set when constructing `MapleCloudflareSDK` in `packages/effect-sdk/src/cloudflare/index.ts`. The config object accepts `serviceName`, `serviceVersion`, and `attributes` for additional resource keys. +TypeScript Workers get resource attributes from `MapleCloudflareSDK.make(config)` in `packages/effect-sdk/src/cloudflare/index.ts`. The config accepts `serviceName`, `serviceVersion`, `serviceNamespace`, `environment`, `repositoryUrl`, and `attributes` for extra resource keys. Unset fields fall back to env (`OTEL_SERVICE_NAME`, `MAPLE_ENVIRONMENT` chain, `OTEL_RESOURCE_ATTRIBUTES`). -When wiring a new worker: +When wiring a new Maple Worker: -- Set `serviceName` to the service identifier (matching what dashboards filter on). -- Pass `attributes: { "deployment.environment.name": env, "deployment.environment": env, "maple_org_id": "internal" }` — same dual-emit rule applies. +- Use `WorkerTelemetry({ serviceName })` (or `eventTelemetry` for background work) from `packages/infra/src/cloudflare/worker-telemetry.ts`. It sets `serviceNamespace: "core"`, the repo URL, and the anticipated-4xx list. +- Deploy it with `selfObservabilityEnv(stage)` in its env so `MAPLE_ENVIRONMENT`, the ingest key, and `OTEL_RESOURCE_ATTRIBUTES=maple.region=` resolve. The SDK then dual-emits `deployment.environment(.name)` itself. +- Do not pass `deployment.environment*` or `maple_org_id` through `attributes`. ## What goes here vs. on the span @@ -101,4 +100,4 @@ When wiring a new worker: | Per-request data (org, user, route, status) | Span attribute (`orgId`, `tenant.userId`, `http.route`, etc.) | | Per-request data that came in over the network | Span attribute (`maple.org_id`, `maple.signal`) | -**Rule of thumb:** if it's the same for every span this process emits, it's a resource attribute. If it varies per request, it's a span attribute. +**Rule of thumb:** if it's the same for every span the process emits, it's a resource attribute. If it varies per request, it's a span attribute. diff --git a/.agents/skills/maple-telemetry-conventions/rules/service-map-attribution.md b/.agents/skills/maple-telemetry-conventions/rules/service-map-attribution.md index a5ae6e3d9b..25d7369832 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/service-map-attribution.md +++ b/.agents/skills/maple-telemetry-conventions/rules/service-map-attribution.md @@ -1,41 +1,44 @@ # Service-map attribution -The service map at `apps/web/src/routes/service-map.tsx` is built entirely from attributes Tinybird MVs pre-extract at write time. If the MV's `SELECT` doesn't see the attribute key spelled the way it expects, the corresponding edge / node / badge **silently disappears** — there's no fallback, no warning. Use the exact keys and values in this file. +The service map (`apps/web/src/routes/service-map.tsx`) is built from span kinds, trace context, and a few attributes that the warehouse rollups read at write time. If a span has the wrong kind or an attribute is spelled differently, the edge, node, or badge silently disappears. There is no fallback and no warning. Read `docs/service-map-architecture.md` before changing the map itself. ## What the map renders, and what feeds each part -| Rendered element | Source query | Required attribute | Lives on | +| Rendered element | Source | Required on the spans | Lives on | |---|---|---|---| -| Service-to-service edge | `service_map_edges_hourly_mv` | `peer.service` | Span (`Client` or `Producer` kind) | -| Service-to-DB edge | `service_map_db_edges_hourly_mv` | `db.system.name` (legacy fallback: `db.system`) and `peer.service` | Span | +| Service-to-service edge | `service_map_edges_hourly`, filled by `ServiceMapRollupService` | A `Client`/`Producer` span in service A whose child is a `Server`/`Consumer` span in service B, in the same trace | Span kind + propagated trace context | +| Database node + edge | `service_map_db_edges_hourly_mv` | `db.system.name` (legacy fallback `db.system`) on a `Client`/`Producer` span; `db.namespace` splits databases of one system into separate nodes | Span | +| External dependency (service detail) | `service_external_edges_hourly_mv` | `server.address` (HTTP), `messaging.system` + `messaging.destination.name`, or `rpc.system` + `rpc.service` on a `Client`/`Producer` span without `db.system.name` | Span | | Platform badge (Cloudflare / AWS / Railway / k8s) | `service_platforms_hourly_mv` | `cloud.platform`, `cloud.provider`, optional `faas.name`, optional `k8s.*` | Resource | | Runtime icon (node / bun / deno / workerd / rust) | `service_platforms_hourly_mv` | `process.runtime.name` | Resource | | SDK badge | `service_platforms_hourly_mv` | `maple.sdk.type` | Resource | -| Region (set but not yet rendered) | reserved | `cloud.region` | Resource | -Source: [packages/query-engine/src/ch/queries/service-map.ts](../../../../packages/query-engine/src/ch/queries/service-map.ts) and [packages/domain/src/tinybird/materializations.ts:415-422](../../../../packages/domain/src/tinybird/materializations.ts). +Sources: `packages/query-engine/src/ch/queries/service-map.ts` and `packages/domain/src/tinybird/materializations.ts`. -## Resource attributes — set once at startup +**`peer.service` does not draw edges.** Modern OTel instrumentation rarely emits it, so the map recovers the downstream service by joining the `Client` span to its child `Server` span (`serviceMapEdgeJoinQuery`). ClickHouse migration 0002 dropped the old `peer.service` MV and the `PeerService` column. An edge needs three things: the caller's outbound span is `Client`/`Producer`, the callee's inbound span is `Server`/`Consumer`, and trace context (W3C `traceparent`) propagates between them. Both services must also export to the same Maple org. -Every service must publish these on its OTel `Resource`. Skip any of them and the service appears on the map with empty badges or doesn't appear at all. +## Resource attributes: set once at startup + +Every service must publish these on its OTel `Resource`. Skip any of them and the service shows empty badges. ### TypeScript (Effect server) -Use `MapleServerSDK` from [packages/effect-sdk/src/server/index.ts](../../../../packages/effect-sdk/src/server/index.ts). Detection lives in [packages/effect-sdk/src/server/platform.ts](../../../../packages/effect-sdk/src/server/platform.ts) — it auto-resolves `cloud.*` / `faas.*` / `process.runtime.name` from `std-env` plus per-platform env vars. No manual setup needed beyond passing `serviceName` and `serviceVersion`. +Use `Maple.layer({ serviceName, serviceVersion })` from `packages/effect-sdk/src/server/index.ts`. Platform detection lives in `packages/effect-sdk/src/server/platform.ts`: it resolves `cloud.*` / `faas.*` / `process.runtime.name` from `std-env` plus per-platform env vars. Nothing else is needed. ### TypeScript (Cloudflare Workers) -Use `MapleCloudflareSDK` from [packages/effect-sdk/src/cloudflare/index.ts](../../../../packages/effect-sdk/src/cloudflare/index.ts). Hard-codes `cloud.provider="cloudflare"`, `cloud.platform="cloudflare.workers"`, `process.runtime.name="workerd"`, `maple.sdk.type="cloudflare"`. +Use `MapleCloudflareSDK` from `packages/effect-sdk/src/cloudflare/index.ts` (Maple's own Workers go through `WorkerTelemetry` in `packages/infra/src/cloudflare/worker-telemetry.ts`). On workerd it resolves `cloud.provider="cloudflare"`, `cloud.platform="cloudflare.workers"`, `process.runtime.name="workerd"`, and sets `maple.sdk.type="cloudflare"`. ### Rust -Use `build_resource` from [apps/ingest/src/otel.rs](../../../../apps/ingest/src/otel.rs): +Use `build_resource` from `apps/ingest/src/otel.rs`: ```rust use maple_ingest::otel::{build_resource, ResourceConfig}; let resource = build_resource(ResourceConfig { service_name: "ingest", + service_namespace: "core", service_version: env!("CARGO_PKG_VERSION"), service_instance_id: uuid::Uuid::new_v4().to_string(), deployment_env, @@ -43,28 +46,34 @@ let resource = build_resource(ResourceConfig { }); ``` -The helper sets `process.runtime.name="rust"`, `maple.sdk.type="server"`, dual-emits `deployment.environment(.name)`, and runs the same platform-detection cascade as the TS SDK (Cloudflare → AWS Lambda → Railway → Vercel → Cloud Run → Render → Fly → k8s). Adding a new platform: extend `detect_platform()`, mirror the env-var sources used in `packages/effect-sdk/src/server/platform.ts`. +The helper sets `process.runtime.name="rust"` and `maple.sdk.type="server"`, dual-emits `deployment.environment(.name)`, and runs the same platform-detection cascade as the TS SDK (Cloudflare, AWS Lambda, Railway, Vercel, Cloud Run, Render, Fly, k8s). To add a platform, extend `detect_platform()` and mirror the env-var sources in `packages/effect-sdk/src/server/platform.ts`. ### Python (forward-looking) -Mirror the TS detection. Set the same keys with the same values — never invent Python-specific spellings. +Mirror the TS detection. Set the same keys with the same values. Never invent Python-specific spellings. -## Span attributes — set per outbound call +## Span attributes: set per outbound call | Call type | Span kind | Required attributes | Notes | |---|---|---|---| -| Outbound HTTP / RPC | `Client` or `Producer` | `peer.service`; optional `server.address`, `http.request.method` | `peer.service` is the **logical** destination, not the host | -| Database call | `Client` or `Producer` | `db.system.name` (e.g. `clickhouse`, `tinybird`, `postgresql`) and `peer.service` | `db.system.name` powers the DB edge; `peer.service` is the logical destination and may differ (for example chDB uses `clickhouse` + `chdb`) | +| Outbound HTTP / RPC to another instrumented service | `Client` | `server.address`, `http.request.method` (or `rpc.system` / `rpc.service`) | Propagate `traceparent` so the callee's `Server` span becomes the child; that join draws the edge | +| Outbound HTTP to an uninstrumented third party | `Client` | `server.address` | Shows up as an external dependency, not a service node | +| Database call | `Client` | `db.system.name` (e.g. `clickhouse`, `tinybird`, `postgresql`), `db.namespace` | `db.system.name` makes the DB node; `db.namespace` (else `server.address`) separates databases of the same system | | Message bus produce | `Producer` | `messaging.system`, `messaging.destination.name` | | +Maple's own code also sets `peer.service` on outbound spans (registry below). It keeps trace search consistent; the map does not read it. + ### TypeScript ```typescript -yield* Effect.annotateCurrentSpan("peer.service", "tinybird") -yield* Effect.annotateCurrentSpan("db.system.name", "tinybird") +const executeSql = Effect.fn("WarehouseQueryService.executeSql", { kind: "client" })(function* (/* ... */) { + yield* Effect.annotateCurrentSpan("db.system.name", "tinybird") + yield* Effect.annotateCurrentSpan("peer.service", "tinybird") + // ... +}) ``` -Canonical example: [packages/query-engine/src/execution/executor.ts](../../../../packages/query-engine/src/execution/executor.ts) `executeSql` — sets both `db.system.name` and `peer.service` from the resolved warehouse backend. +Canonical example: `executeSql` in `packages/query-engine/src/execution/executor.ts`. It sets `db.system.name`, `peer.service`, and (via `warehouseTargetAttributes`) `db.namespace` / `server.address` from the resolved warehouse backend, on a `Client` span. ### Rust @@ -77,45 +86,48 @@ let span = forward_client_span("collector", body_size, signal.path()); forward_to_collector(...).instrument(span).await ``` -The helper bakes in `peer.service`, `otel.kind="client"`, and the `http.*` field set. `url.full` and `server.address` get recorded later inside `forward_to_collector` once the URL is resolved. +The helper bakes in `peer.service`, `otel.kind="client"`, and the `http.*` field set. `url.full` and `server.address` are recorded later inside `forward_to_collector`, once the URL is resolved. Warehouse writes use `export_client_span`, which also declares `db.system.name`, `db.namespace`, and `db.collection.name`. Without `db.system.name` a warehouse write never becomes a database node. -For new outbound peers, follow the same pattern: pick a name from the registry below (or extend it), and use `tracing::info_span!` with `"peer.service" = peer_name` plus `otel.kind = "client"`. +For a new outbound peer, use `tracing::info_span!` with `otel.kind = "client"`, `"server.address"`, and `"peer.service" = `. -## Canonical `peer.service` registry +## `peer.service` naming registry -Keep these names consistent across services to avoid edge fragmentation on the map. If two services call the same peer with different names, the map shows two nodes. +Maple's code emits these values. Reuse them so trace search stays consistent, and add new peers here. | Peer name | What it is | |---|---| | `tinybird` | Tinybird hosted warehouse | -| `clickhouse` | Self-managed ClickHouse warehouse | +| `clickhouse` | ClickHouse warehouse (BYO or Tinybird's ClickHouse gateway) | | `chdb` | Embedded chDB behind the local Maple binary | | `collector` | Maple ingest's downstream OTLP collector | -| `cloudflare-logpush` | Cloudflare logpush source | +| `ingest` | Maple ingest gateway (as called by the scraper and the Cloudflare analytics poller) | +| `maple-api` | Maple API (as called by the web app) | +| `planetscale-postgres` | PlanetScale Postgres | +| `electric` | Electric sync | | `clerk` | Clerk auth | | `autumn` | Autumn metering | -| `github` | GitHub REST API (VCS sync) | -| `planetscale-metrics` | PlanetScale per-branch `/metrics` data plane, via the scrape proxy | -| `scrape-target` | Any other user-configured Prometheus scrape target, via the scrape proxy | - -The last two are deliberately generic. Scrape targets are user-configured and unbounded, so -naming a peer per target would grow the service map a node at a time. Bucket by target type. +| `github` | GitHub REST API | +| `slack`, `discord` | Chat platform APIs | +| `apns` | Apple Push Notification service | +| `planetscale-metrics` | PlanetScale per-branch `/metrics` endpoint (scraper) | +| `scrape-target` | Any other user-configured Prometheus scrape target (scraper) | -Add new peers to this list as they're introduced — single source of truth. +The last two are deliberately generic. Scrape targets are user-configured and unbounded, so bucket them by target type. ## Anti-patterns -- **Don't use `server.address` instead of `peer.service`.** `server.address` is the host (e.g. `api.tinybird.co`); `peer.service` is the logical service identity. The map keys off the latter. -- **Don't set `peer.service` on `Server`-kind spans.** The MV filters them out. Server spans describe inbound work, which doesn't produce an outbound edge. -- **Don't invent platform values.** Use the OTel semconv-defined `cloud.platform` strings (`aws_lambda`, `cloudflare.workers`, `gcp_cloud_run`, etc.) so badges match across services. -- **Don't pick a different `process.runtime.name` than what TS emits for the same runtime.** TS uses `nodejs`, `bun`, `deno`, `workerd`. New Rust services use `rust`. New Python services use `python`. Mismatched values produce duplicate runtime icons. -- **Don't set `peer.service` only on success.** Set it at span declaration so the edge appears for failed calls too — that's how the map shows error rates per edge. +- **Don't leave an outbound call as an `Internal` span.** The edge join and the DB and external MVs only read `Client`/`Producer` spans. +- **Don't break trace context across a hop.** A `Client` span whose request does not carry `traceparent` has no child `Server` span, so no edge. +- **Don't wrap a DB call's `Client` span in a second `Client` span** (e.g. an HTTP client span under `executeSql`). Maple's warehouse drivers disable the `HttpClient` tracer for this reason. +- **Don't invent platform values.** Use the OTel semconv `cloud.platform` strings (`aws_lambda`, `cloudflare.workers`, `gcp_cloud_run`, etc.) so badges match across services. +- **Don't pick a different `process.runtime.name` than TS emits for the same runtime.** TS uses `nodejs`, `bun`, `deno`, `workerd`. Rust uses `rust`. New Python services use `cpython` (the OTel semconv value). Mismatched values produce duplicate runtime icons. +- **Set attributes at span declaration, not only on success.** Failed calls must still land on the edge so the map shows per-edge error rates. ## Verification After deploying a new service or peer: -1. Generate a representative request that exercises the outbound call. -2. Wait for the MV to hourly-aggregate (or check the in-progress hour via the raw fallback branch of [packages/query-engine/src/ch/queries/service-map.ts:92-112](../../../../packages/query-engine/src/ch/queries/service-map.ts:92)). -3. Open the service map and confirm the expected node + edges appear with the right badges. -4. If anything's missing, use `mcp__maple__inspect_trace` on a recent span to see the actual emitted attributes and compare to this rule. +1. Send a representative request that exercises the outbound call. +2. The service-to-service rollup runs on a schedule, and the DB/external MVs bucket hourly. The read path unions in the current partial hour from raw spans (see the live branch in `packages/query-engine/src/ch/queries/service-map.ts`), so recent traffic should appear. +3. Open the service map and confirm the expected nodes, edges, and badges. +4. If anything is missing, run the Maple MCP `inspect_trace` tool on a recent trace and compare the span kinds, parent/child links, and attributes against this rule. diff --git a/.agents/skills/maple-telemetry-conventions/rules/span-attributes.md b/.agents/skills/maple-telemetry-conventions/rules/span-attributes.md index be7ddecec2..c71dae4038 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/span-attributes.md +++ b/.agents/skills/maple-telemetry-conventions/rules/span-attributes.md @@ -1,53 +1,55 @@ # Span attribute reference -The canonical list of custom span attribute keys Maple emits, grouped by namespace. **Use these exact spellings in every language** — Title Case status, dotted-lowercase keys (with the documented camelCase exceptions for the tenant block). +The main custom span attribute keys Maple emits, grouped by namespace. Use these exact spellings in every language: Title Case status, dotted-lowercase keys, and the documented camelCase exceptions in the tenant block. The list is not exhaustive (`maple.*` alone has well over a hundred keys), so grep for an existing key before inventing one. Columns: -- **Key** — exact attribute name to emit -- **Type** — JSON type (`string`, `int`, `bool`) -- **Set at** — file:line that proves the convention (canonical example) -- **Meaning** — one-line description +- **Key**: exact attribute name to emit +- **Type**: JSON type (`string`, `int`, `bool`) +- **Set at**: file that sets it (the canonical example) +- **Meaning**: one-line description --- -## ⭐ `WarehouseQueryService.executeSql` — canonical span +## `WarehouseQueryService.executeSql`: the canonical span -Every SQL execution against Tinybird or ClickHouse emits these. When you add a new attribute to a query path, prefer extending this block over inventing a parallel set. +Every SQL execution against Tinybird or ClickHouse emits these. When you add an attribute to a query path, extend this block instead of inventing a parallel set. -Source: `packages/query-engine/src/execution/executor.ts` (`executeSql`) +Source: `packages/query-engine/src/execution/executor.ts` (`executeSql`), wired into `WarehouseQueryService` in `packages/backend/src/services/warehouse/WarehouseQueryService.ts`. | Key | Type | Set at | Meaning | |---|---|---|---| -| `orgId` | string | `executor.ts` | Tenant org UUID (camelCase — historical, do not rename) | +| `orgId` | string | `executor.ts` | Tenant org UUID (camelCase, historical; do not rename) | | `tenant.userId` | string | `executor.ts` | User ID within the tenant | | `tenant.authMode` | string | `executor.ts` | `"api_key"` / `"user_login"` / etc. | | `clientSource` | string | `executor.ts` | `"org_override"` or `"managed"` (which config resolved) | | `db.client` | string | `executor.ts` | `"clickhouse"` or `"tinybird-sdk"` | | `db.system.name` | string | `executor.ts` | `"clickhouse"` or `"tinybird"` (legacy spans: `db.system`) | +| `peer.service` | string | `executor.ts` | `"tinybird"`, `"clickhouse"`, or `"chdb"` (from `BackendDialect` in `backend.ts`) | +| `db.namespace` | string | `backend.ts` (`warehouseTargetAttributes`) | ClickHouse database name; omitted when empty | +| `server.address` | string | `backend.ts` (`warehouseTargetAttributes`) | Warehouse host; omitted when empty | | `db.query.text` | string | `executor.ts` | Full compiled SQL, truncated to 16 KB (legacy spans: `db.statement`) | -| `db.query.length` | int | `executor.ts` | Pre-truncation byte length (legacy: `db.statement.length`) | +| `db.query.length` | int | `executor.ts` | Pre-truncation length (legacy: `db.statement.length`) | | `db.query.truncated` | bool | `executor.ts` | Whether SQL was capped at 16 KB (legacy: `db.statement.truncated`) | -| `db.query.fingerprint` | string | `executor.ts` | 32-bit FNV-1a hash with literals + numbers normalized (legacy: `db.statement.fingerprint`) | -| `db.duration_ms` | int | `executor.ts` | Execution time in ms (emitted on both success and error tap) | -| `db.total_duration_ms` | int | `executor.ts` | Total execution-span duration including config resolution and client setup | +| `db.query.fingerprint` | string | `executor.ts` | 32-bit FNV-1a hash with literals and numbers normalized (legacy: `db.statement.fingerprint`) | +| `db.duration_ms` | int | `executor.ts` | Warehouse execution time in ms (success and error paths) | +| `db.total_duration_ms` | int | `executor.ts` | Whole span duration, including config resolution and client setup | | `db.retry.attempts` | int | `executor.ts` | Retries actually performed (not total attempts) | -| `query.pipe` | string | `executor.ts` | Original pipe name resolved by `query()` | -| `query.context` | string | `executor.ts` | Semantic call-site label (e.g. `"errorsByType"`, `"spanHierarchy"`). Set via `SqlQueryOptions.context`. | -| `query.profile` | string | `executor.ts` | Execution profile (e.g. `"list"`, `"analytics"`). Set via `SqlQueryOptions.profile`. | +| `query.pipe` | string | `executor.ts` | Pipe name passed to the executor | +| `query.context` | string | `executor.ts` | Semantic call-site label (e.g. `"errorsByType"`, `"spanHierarchy"`). Set via `SqlQueryOptions.context`; falls back to the pipe name. | +| `query.profile` | string | `executor.ts` | Execution profile (e.g. `"list"`, `"analytics"`). Set via `SqlQueryOptions.profile`; only emitted when set. | +| `query.routing` | string | `executor.ts` | Legacy `"ingest"` marker, dual-emitted with `warehouse.route` | | `ch.settings` | string (JSON) | `executor.ts` | JSON-encoded ClickHouse settings applied to the query | | `result.rowCount` | int | `executor.ts` | Number of rows returned | | `db.response.returned_rows` | int | `executor.ts` | OTel spelling of `result.rowCount`, success only | | `db.operation.name` | string | `executor.ts` | Leading SQL verb (`SELECT`, `INSERT`) from `summarizeSql` | -| `db.collection.name` | string | `executor.ts` | First table named by the statement; the datasource for `ingest` | -| `db.query.summary` | string | `executor.ts` | `{operation} {collection}` — identical to what the shape rollup derives when absent | -| `db.operation.batch.size` | int | `executor.ts` | Rows in an `ingest` batch | -| `error.type` | string | `executor.ts` | Failure only: ClickHouse exception type (`UNKNOWN_TABLE`), else its code, else the error tag | -| `db.response.status_code` | string | `executor.ts` | Failure only: the ClickHouse error code, else the upstream HTTP status | +| `db.collection.name` | string | `executor.ts` | First table named by the statement | +| `db.query.summary` | string | `executor.ts` | `{operation} {collection}`, identical to what the shape rollup derives when absent | +| `error.type` | string | `errors.ts` (`warehouseFailureAttributes`) | Failure only: ClickHouse exception type (`UNKNOWN_TABLE`), else its code, else the error tag | +| `db.response.status_code` | string | `errors.ts` (`warehouseFailureAttributes`) | Failure only: the ClickHouse error code, else the upstream HTTP status | -`executeSql` is the **one** `Client` span per logical warehouse operation, retries included, as the -OTel database conventions ask. The drivers run on `warehouseHttpClient(...)`, which disables the -Effect `HttpClient` tracer for their round-trips — do not let an `http.client POST` span reappear -under it. +`executeSql` is the one `Client` span per logical warehouse operation, retries included, as the OTel database conventions ask. The `Client` kind is load-bearing: the service-map DB-edge MV only counts `Client`/`Producer` spans. The drivers run on `warehouseHttpClient(...)` (`packages/query-engine/src/execution/driver-http.ts`), which disables the Effect `HttpClient` tracer for their round-trips. Do not let an `http.client POST` span reappear under it. + +The sibling `WarehouseQueryService.ingest` span (same file) sets `datasource`, `orgId`, `rowCount`, `db.operation.name = "INSERT"`, `db.collection.name`, `db.query.summary`, and `db.operation.batch.size`. ## `warehouse.*` group @@ -59,24 +61,21 @@ Routing metadata emitted by the shared executor for API and local-CLI queries. | `warehouse.route` | string | Route purpose: `read`, `raw`, or `ingest` | | `warehouse.config_source` | string | Config source: `managed`, `org-byo`, or `org-jwt` | -The historical `clientSource` and `db.client` attributes remain on the same -canonical span. `clientSource` maps `org-byo` to `org_override`; other sources -map to `managed`. `db.client` records the concrete driver family -(`tinybird-sdk` or `clickhouse`). +The historical `clientSource` and `db.client` attributes remain on the same span. `clientSource` maps `org-byo` to `org_override` and every other source to `managed`. `db.client` records the driver family (`tinybird-sdk` or `clickhouse`). -**Rule:** When adding a new query, always pass a `context` string to `SqlQueryOptions` — it becomes filterable as `query.context` in trace search. Don't invent new keys when one of `query.*` fits. +**Rule:** When adding a query, always pass a `context` string in `SqlQueryOptions`. It becomes filterable as `query.context` in trace search. Don't invent new keys when a `query.*` key fits. --- ## `db.*` group (general) -Beyond `executeSql`, the same `db.system.name` / `db.duration_ms` keys appear wherever Maple talks to a warehouse. Add any new DB-related attrs under `db.*` — never under `database.*` or `clickhouse.*`. +Beyond `executeSql`, the same `db.system.name` / `db.duration_ms` keys appear wherever Maple talks to a warehouse. Put new DB-related attributes under `db.*`, never under `database.*` or `clickhouse.*`. --- ## `query.*` group (DSL query metadata) -Used by `packages/query-engine` and `apps/api/src/services/QueryEngineService.ts` for the higher-level DSL surface (not raw SQL). +Used by `packages/query-engine/src/runtime/query-engine.ts` for the higher-level DSL surface (not raw SQL). | Key | Type | Meaning | |---|---|---| @@ -98,9 +97,9 @@ Used by `packages/query-engine` and `apps/api/src/services/QueryEngineService.ts | Key | Type | Set at | Meaning | |---|---|---|---| -| `result.rowCount` | int | `WarehouseQueryService.ts:507` | Rows returned by a query | -| `result.groupCount` | int | QueryEngineService | Distinct groups in a breakdown | -| `result.renderedSpanCount` | int | `inspect-trace.ts` | Spans retained after trace-overview rendering limits | +| `result.rowCount` | int | `executor.ts` | Rows returned by a query | +| `result.groupCount` | int | `query-engine.ts` | Distinct groups in a breakdown | +| `result.renderedSpanCount` | int | `apps/ai/src/mcp/tools/inspect-trace.ts` | Spans retained after trace-overview rendering limits | --- @@ -108,31 +107,33 @@ Used by `packages/query-engine` and `apps/api/src/services/QueryEngineService.ts | Key | Type | Set at | Meaning | |---|---|---|---| -| `tenant.userId` | string | `WarehouseQueryService.ts:449` | User ID within the tenant | -| `tenant.authMode` | string | `WarehouseQueryService.ts:450` | Authentication mode | +| `tenant.userId` | string | `executor.ts` | User ID within the tenant | +| `tenant.authMode` | string | `executor.ts` | Authentication mode | -**Naming inconsistency to preserve:** `orgId` is camelCase, `tenant.userId` is dotted-lowercase. This is historical — `orgId` predates the `tenant.*` namespace and renaming it would break existing trace search filters and dashboard queries. Do not unify them. +**Naming inconsistency to preserve:** `orgId` is camelCase, `tenant.userId` is dotted. `orgId` predates the `tenant.*` namespace, and renaming it would break existing trace search filters and dashboard queries. Do not unify them. --- ## `cache.*` group -Emitted by `BucketCacheService` (timeseries cache) and the EdgeCache wrappers. Used to debug cache hit ratios and pinpoint dedup behavior. +Emitted by the bucket cache and the query-engine cache wrappers. Use them to debug hit ratios and segment behavior. -Source: `apps/api/src/services/BucketCacheService.ts`, `apps/api/src/services/QueryEngineService.ts` +Source: `packages/query-engine/src/caching/bucket-cache.ts`, `packages/backend/src/services/warehouse/QueryEngineService.ts` | Key | Type | Set at | Meaning | |---|---|---|---| -| `cache.fingerprint` | string | `BucketCacheService.ts:314` | First 12 chars of cache key fingerprint | -| `cache.hit` | bool | `QueryEngineService.ts:1694, 1806, 1830` | Cache hit or miss | -| `cache.ttlSeconds` | int | `QueryEngineService.ts:1695` | TTL of cached entry | -| `cache.bucketsHit` | int | `QueryEngineService.ts:1745` | Matching buckets from cache (bucket cache) | -| `cache.bucketsMissed` | int | `QueryEngineService.ts:1746` | Missing buckets (bucket cache) | -| `cache.missingRangeCount` | int | `BucketCacheService.ts:456`, `QueryEngineService.ts:1747` | Count of contiguous missing time ranges | -| `cache.existingBucketCount` | int | `BucketCacheService.ts:457` | Pre-existing buckets in cache | -| `cache.bucketSeconds` | int | `BucketCacheService.ts:477` | Bucket size for this cache request | -| `cache.rangeMs` | int | `BucketCacheService.ts:478` | Request time-range span in ms | -| `cache.dedup.waited` | bool | `BucketCacheService.ts:366` | Whether this request waited on an in-flight peer | +| `cache.fingerprint` | string | `bucket-cache.ts` | First 12 chars of the cache key fingerprint | +| `cache.bucketSeconds` | int | `bucket-cache.ts` | Bucket size for this cache request | +| `cache.rangeMs` | int | `bucket-cache.ts` | Request time range in ms | +| `cache.requestedBuckets` | int | `bucket-cache.ts` | Buckets the request covers | +| `cache.bucketsHit` | int | `bucket-cache.ts`, `QueryEngineService.ts` | Buckets served from cache | +| `cache.bucketsMissed` | int | `bucket-cache.ts`, `QueryEngineService.ts` | Buckets that had to be queried | +| `cache.missingRangeCount` | int | `bucket-cache.ts`, `QueryEngineService.ts` | Count of contiguous missing time ranges | +| `cache.warehouse_query_count` | int | `bucket-cache.ts` | Warehouse queries issued to fill misses | +| `cache.segments_hit` / `_missed` / `_timed_out` / `_skipped` / `_errored` | int | `bucket-cache.ts` | Per-segment outcomes | +| `cache.hit` | bool | `QueryEngineService.ts` | Cache hit or miss | +| `cache.ttlSeconds` | int | `QueryEngineService.ts` | TTL of the cached entry | +| `cache.path` | string | `QueryEngineService.ts` | `"blob"` or `"bucket"` | --- @@ -140,99 +141,100 @@ Source: `apps/api/src/services/BucketCacheService.ts`, `apps/api/src/services/Qu Emitted by `EmailService` for outbound transactional email. -Source: `apps/api/src/lib/EmailService.ts` +Source: `packages/backend/src/platform/EmailService.ts` -| Key | Type | Set at | Meaning | -|---|---|---|---| -| `email.subject` | string | `EmailService.ts:59` | Subject line | -| `email.provider` | string | `EmailService.ts:60` | `"cloudflare"` (Cloudflare Email Service Workers binding) | -| `email.message_id` | string | `EmailService.ts:103` | Provider message id returned after a successful send | +| Key | Type | Meaning | +|---|---|---| +| `email.subject` | string | Subject line | +| `email.provider` | string | `"cloudflare"` (Cloudflare Email Service Workers binding) | +| `email.message_id` | string | Provider message id returned after a successful send | --- ## `maple.*` vendor namespace -All `maple.*` keys are reserved for Maple-specific metadata that has no OTel semconv equivalent. +`maple.*` is reserved for Maple-specific metadata with no OTel semconv equivalent. Entity ids follow `maple..id` (`maple.dashboard.id`, `maple.alert.rule_id`, `maple.investigation.id`, `maple.share.id`). Other live sub-namespaces include `maple.query.*`, `maple.chat.*`, `maple.pr_review.*`, `maple.mcp.*`, `maple.session.*`, and `maple.webhook.*`. ### API entity attributes | Key | Type | Set at | Meaning | |---|---|---|---| -| `maple.api_key.id` | string | `ApiKeysService.ts` | API key entity involved in an operation. | -| `maple.api_key.last_used_memo_hit` | bool | `ApiKeysService.ts` | Whether a last-used write was skipped by the per-isolate memo. | -| `maple.dashboard.id` | string | `DashboardPersistenceService.ts` | Dashboard entity involved in an operation. | -| `maple.dashboard.version_id` | string | `DashboardPersistenceService.ts` | Dashboard history version involved in an operation. | -| `maple.ingest_attribute_mapping.id` | string | `IngestAttributeMappingService.ts` | Ingest attribute mapping involved in an operation. | -| `maple.organization.member.requested_count` | int | `OrgMembersService.ts` | Number of member ids requested for resolution. | +| `maple.api_key.id` | string | `ApiKeysService.ts` | API key entity involved in an operation | +| `maple.api_key.last_used_memo_hit` | bool | `ApiKeysService.ts` | Whether a last-used write was skipped by the per-isolate memo | +| `maple.dashboard.id` | string | `DashboardPersistenceService.ts` | Dashboard entity involved in an operation | +| `maple.dashboard.version_id` | string | `DashboardPersistenceService.ts` | Dashboard history version involved in an operation | +| `maple.ingest_attribute_mapping.id` | string | `IngestAttributeMappingService.ts` | Ingest attribute mapping involved in an operation | +| `maple.organization.member.requested_count` | int | `OrgMembersService.ts` | Number of member ids requested for resolution | -### Ingest gateway +These services live under `packages/backend/src/services/`. -Custom domain attributes for the Rust ingest gateway. +### Ingest gateway -Source: `apps/ingest/src/main.rs:843-861` (inbound signal span), `:920-937` (Cloudflare logpush), `:1132-1145` (downstream forward). +Custom attributes on the Rust ingest gateway's spans. Sources: `apps/ingest/src/main.rs` (`handle_signal` inbound span, `handle_cloudflare_logpush` inbound span) and `apps/ingest/src/otel.rs` (`forward_client_span`, `export_client_span`, `grpc_server_span`). -### `maple.signal` +#### `maple.signal` - **Type:** string (`"traces"`, `"logs"`, `"metrics"`) -- **Set at:** `apps/ingest/src/main.rs:853, 930, 1143, 1308` -- **Meaning:** Which OTel signal this request carries. +- **Meaning:** Which OTel signal the request carries. -### `maple.org_id` +#### `maple.org_id` - **Type:** string -- **Set at:** `apps/ingest/src/main.rs:854, 931, 1203` -- **Meaning:** Resolved organization ID from the ingest key. -- **⚠ Note:** In Rust this is `maple.org_id` (vendor-namespaced). In TypeScript (`WarehouseQueryService.ts:448`) it's `orgId` (camelCase, no namespace). Both are intentional — trace search filters in dashboards already expect both spellings. +- **Meaning:** Organization ID resolved from the ingest key. +- **Note:** Rust uses `maple.org_id` (vendor-namespaced). TypeScript uses `orgId` (camelCase, no namespace). Both are intentional; keep both. -### `maple.ingest.*` sub-namespace +#### `maple.ingest.*` sub-namespace -| Key | Type | Set at | Meaning | -|---|---|---|---| -| `maple.ingest.key_type` | string | `main.rs:855` | `"public"` / `"private"` / `"sentinel"` / `"connector"` | -| `maple.ingest.self_managed` | bool | `main.rs:856, 935, 1204` | Org using self-managed Tinybird | -| `maple.ingest.payload_format` | string | `main.rs:857` | `"otlp_json"` / `"otlp_protobuf"` / `"cloudflare_json"` | -| `maple.ingest.content_encoding` | string | `main.rs:858` | `"gzip"` / `"deflate"` / `""` | -| `maple.ingest.decoded_bytes` | int | `main.rs:859` | Size after decompression | -| `maple.ingest.item_count` | int | `main.rs:860, 936, 950` | Spans / logs / metrics in the payload | -| `maple.ingest.upstream_pool` | string | `main.rs:1144, 1309` | `"shared"` / `"self_managed"` (downstream collector pool) | - -### `maple.cloudflare.*` sub-namespace +| Key | Type | Meaning | +|---|---|---| +| `maple.ingest.key_type` | string | `"public"` / `"private"` / `"connector"` / `"sentinel"` | +| `maple.ingest.self_managed` | bool | Org has a connected BYO ClickHouse (`org_clickhouse_settings.sync_status = 'connected'`) | +| `maple.ingest.destination` | string | Export destination: `"tinybird"` or `"clickhouse"` | +| `maple.ingest.payload_format` | string | `"json"` / `"protobuf"` | +| `maple.ingest.content_encoding` | string | Request `Content-Encoding`, or `"identity"` when absent | +| `maple.ingest.decoded_bytes` | int | Size after decompression | +| `maple.ingest.item_count` | int | Spans / logs / metrics in the payload | +| `maple.ingest.reject_reason` | string | Why a request or stage was rejected (always recorded, even when status stays `Ok`) | +| `maple.ingest.upstream_pool` | string | Downstream collector pool; always `"shared"` today | +| `maple.ingest.sampling_ratio` | float | Per-org trace sample ratio applied at ingest | + +#### `maple.cloudflare.*` sub-namespace -| Key | Type | Set at | Meaning | -|---|---|---|---| -| `maple.cloudflare.connector_id` | string | `main.rs:932` | Cloudflare Logpush connector UUID | -| `maple.cloudflare.dataset` | string | `main.rs:933` | `"http_requests"` (only value today) | -| `maple.cloudflare.is_validation` | bool | `main.rs:934, 951` | Whether this is a Cloudflare validation ping | +| Key | Type | Meaning | +|---|---|---| +| `maple.cloudflare.connector_id` | string | Cloudflare Logpush connector UUID | +| `maple.cloudflare.dataset` | string | `"http_requests"` (only value today) | +| `maple.cloudflare.is_validation` | bool | Whether this is a Cloudflare validation ping | --- ## HTTP semconv (ingest gateway) -The Rust ingest gateway emits the canonical OTel HTTP semconv keys on its Server-kind and Client-kind spans. **Use these exact keys; do not invent `http.method` (legacy) or `http.url` in new code.** +The Rust ingest gateway emits the stable OTel HTTP semconv keys on its Server-kind and Client-kind spans. Use these exact keys. Do not write `http.method` (legacy) or `http.url` in new code. -| Key | Direction | Set at | Meaning | -|---|---|---|---| -| `http.request.method` | both | `main.rs:848, 925, 1137, 1302` | Always `"POST"` for ingest | -| `http.route` | server | `main.rs:849, 926` | Logical route (`"/v1/traces"`, `"/v1/logpush/cloudflare/http_requests/{connector_id}"`) | -| `http.request.body.size` | both | `main.rs:850, 927, 1138, 1303` | Request body size in bytes | -| `http.response.status_code` | both | `main.rs:851, 873, 893, 928, 948, 976, 1139, 1304` | Final HTTP status code | -| `error.type` | server, client | `main.rs:852, 894, 929, 977, 1142, 1307` | Error category: `"validation"` / `"auth"` / `"upstream"` / `"decode"` / `"forward"` etc. | -| `url.full` | client | `main.rs:1140, 1305` | Full downstream collector URL | -| `server.address` | client | `main.rs:1141, 1306` | Downstream collector host | +| Key | Direction | Meaning | +|---|---|---| +| `http.request.method` | both | Always `"POST"` for ingest | +| `http.route` | server | Logical route (`"/v1/traces"`, `"/v1/logpush/cloudflare/http_requests/{connector_id}"`) | +| `http.request.body.size` | both | Request body size in bytes | +| `http.response.status_code` | both | Final HTTP status code | +| `error.type` | both | Error category: `"auth"`, `"billing"`, `"throttle"`, `"payload_too_large"`, `"decode"`, `"enrich"`, etc. | +| `url.full` | client | Full downstream collector URL | +| `server.address` | client | Downstream collector host | -The Rust `tracing` macro reserves field names with dots only when quoted: `"http.request.method" = "POST"`. The Rust ingest gateway also uses `otel.name`, `otel.kind`, `otel.status_code` field names — see `rules/status-and-kind.md`. +In a Rust `tracing` macro, field names that contain dots must be quoted: `"http.request.method" = "POST"`. The gateway also uses the reserved `otel.name`, `otel.kind`, `otel.status_code`, and `otel.status_description` fields. See `rules/status-and-kind.md`. --- ## Misc attributes (call-site-specific) -These appear on individual span functions but are not part of a reusable namespace. Inventoried here so you don't accidentally invent parallel keys. +These appear on individual spans but do not form a reusable namespace. They are listed so you don't invent parallel keys. | Key | Type | Where | Meaning | |---|---|---|---| -| `datasource` | string | `WarehouseQueryService.ts:571` | Datasource being ingested into | -| `rowCount` | int | `WarehouseQueryService.ts:573` | Rows in ingest payload | -| `pipe` | string | `WarehouseQueryService.ts:517` | Pipe name (legacy `query()` method) | +| `datasource` | string | `executor.ts` (`ingest`) | Datasource being ingested into | +| `rowCount` | int | `executor.ts` (`ingest`) | Rows in the ingest payload | +| `pipe` | string | `executor.ts` (`query`) | Pipe name on the legacy `query()` method | | `attributeKey` | string | attribute-explore routes | Attribute key being browsed | -| `service` | string | many routes | Service name from query params (note: not `service.name` — that's a resource attribute) | +| `service` | string | many routes | Service name from query params (not `service.name`, which is a resource attribute) | | `spanId` | string | trace detail | Span being inspected | | `traceId` | string | trace detail | Trace being inspected | | `userId` | string | various | User ID context | @@ -240,4 +242,4 @@ These appear on individual span functions but are not part of a reusable namespa | `rootOnly` | bool | trace queries | Filter to root spans only | | `incidentCount`, `issueCount`, `errorCount`, `serviceCount`, `orgCount`, `eventCount`, `sentCount`, `totalRequests`, `totalErrors`, `resultCount` | int | various | Result-shape counters on enclosing routes | -**Rule:** Prefer extending an existing namespace (`query.*`, `result.*`, `cache.*`) over adding a new bare key. New bare keys make trace search harder. +**Rule:** Extend an existing namespace (`query.*`, `result.*`, `cache.*`, `maple.*`) instead of adding a new bare key. Bare keys make trace search harder. diff --git a/.agents/skills/maple-telemetry-conventions/rules/status-and-kind.md b/.agents/skills/maple-telemetry-conventions/rules/status-and-kind.md index f3b0e0535d..a228c59d72 100644 --- a/.agents/skills/maple-telemetry-conventions/rules/status-and-kind.md +++ b/.agents/skills/maple-telemetry-conventions/rules/status-and-kind.md @@ -1,24 +1,24 @@ # Span status codes and span kinds -## Status codes — always Title Case +## Status codes: always Title Case -Maple stores span status as the **rendered string**, in Title Case. The values are: +Maple stores span status as the rendered string, in Title Case: | Value | When | Note | |---|---|---| | `"Ok"` | Successful operation | Not `OK`, `SUCCESS`, `ok`, or `Success` | | `"Error"` | Failed operation | Not `ERROR`, `FAILED`, `error`, or `Failed` | -| `"Unset"` | Status not explicitly set (rare) | OTel default; usually only on Internal-kind spans that don't represent a discrete success/fail outcome | +| `"Unset"` | Status not explicitly set | OTel default; usually only on Internal-kind spans with no discrete success/fail outcome | -### Why this matters — it's load-bearing +### Why it matters -Tinybird MVs and dashboard widgets filter on the literal string. The error analytics path uses `WHERE StatusCode = 'Error'`. Uppercase or lowercase variants silently produce zero rows on the error-rate widget and zero traces in the errors list. CLAUDE.md states this explicitly: "**Span Status Codes:** Use title case (`"Ok"`, `"Error"`, `"Unset"`), not uppercase". +Tinybird MVs and dashboard widgets filter on the literal string. The error path uses `WHERE StatusCode = 'Error'` (`error_events_mv` and the error-rate widgets). Uppercase or lowercase variants silently produce zero rows on the error-rate widget and zero traces in the errors list. CLAUDE.md: "Span status codes: Title case: `"Ok"`, `"Error"`, `"Unset"`." ### How status is set -**TypeScript (Effect):** The Effect tracer maps an effect's failure to span status. When you call `Effect.fail(...)` or the effect dies, the tracer records `Error`. On success, the tracer records `Ok`. Do not set the string manually in TS — let the runtime do it. +**TypeScript (Effect):** The tracer maps an effect's outcome to span status. `Effect.fail(...)` or a defect records `Error`; success records `Ok`. Do not set the string manually. One exception is handled for you: `@maple-dev/effect-sdk` exports a span as `Ok` (no `exception` event) when its failure is entirely an anticipated 4xx error. Maple's Workers pass `ANTICIPATED_ERROR_IDENTIFIERS` (`packages/domain/src/anticipated-errors.ts`, generated from every domain error annotated with a 4xx `httpApiStatus`) through `workerTelemetryConfig` in `packages/infra/src/cloudflare/worker-telemetry.ts`. A new 4xx domain error needs a 4xx `httpApiStatus` and a regenerated identifier list (a drift test in `anticipated-errors.test.ts` catches a stale one). -**Rust (ingest gateway):** Status is set explicitly via the `otel.status_code` field on the span: +**Rust (ingest gateway):** Status is set explicitly through the `otel.status_code` field: ```rust let span = tracing::info_span!( @@ -30,33 +30,41 @@ let span = tracing::info_span!( ); // on success: span_handle.record("otel.status_code", "Ok"); -// on error — for inbound request handlers, follow the HTTP semconv rule below: -span_handle.record("otel.status_code", otel_status_for_rejection(status)); +// on error, for inbound request handlers, follow the rejection rule below: +span_handle.record("otel.status_code", otel_status_for_rejection(status, error_kind)); ``` -**HTTP SERVER spans — only 5xx is `Error`.** Per OTEL HTTP semconv, an inbound request handler must set `Error` *only* for 5xx responses; **4xx client rejections are `Ok`** (the caller is at fault). The ingest gateway centralizes this in `otel_status_for_rejection(status: u16)` (`apps/ingest/src/main.rs`): `>= 500 → "Error"`, else `"Ok"`. This stops expected rejections — missing/invalid ingest key (401), billing-limit (402), throttle (429), oversized/undecodable payload — from flooding the `WHERE StatusCode = 'Error'` dashboards, while the genuine server fault (auth resolver unavailable → 503) still reads as `Error`. The status code, `error.type`, and request metrics are recorded for all rejections regardless, so 4xx stays observable. Non-HTTP spans (forward `Client` spans, internal effects) are unaffected — they record `Error` on any failure as before. +**HTTP SERVER spans: 5xx is `Error`; a 4xx is `Error` only when it drops a healthy sender's data.** Per OTel HTTP semconv, a server span sets `Error` for 5xx and leaves caller faults alone. The ingest gateway centralizes this in `otel_status_for_rejection(status, error_kind)` (`apps/ingest/src/main.rs`), which returns `Error` when `status >= 500` or `rejection_loses_data(error_kind)` (`apps/ingest/src/otel.rs`), else `Ok`. -See `apps/ingest/src/main.rs:874, 895, 949, 978, 1201, 1306`. +- **Caller faults stay `Ok`:** `auth` (401), `billing` (402), `throttle` (429), `payload_too_large`, `unsupported_media`. These are expected and high-volume, and must not flood `StatusCode = 'Error'` dashboards. +- **Data loss is `Error`:** `decode`, `enrich`, `bad_request`. The sender believes it is shipping telemetry and none of it is stored. A bug that rejected 99.2% of one org's logs for over 24h hid behind a 0% error rate under the old 5xx-only rule. +- **Server faults are `Error`:** any 5xx, e.g. auth resolver unavailable (503). -**Python (forward-looking):** `span.set_status(Status(StatusCode.ERROR))` — Python SDK maps `StatusCode.ERROR` to the wire-level `Error` string before export, so no manual conversion is needed. But never call `span.set_attribute("otel.status_code", "ERROR")` (uppercase) — it bypasses the SDK conversion. +The status code, `error.type`, and `maple.ingest.reject_reason` are recorded for every rejection, so 4xx stays observable. Stage spans use the same rule through `record_stage_error` (`otel.rs`). Non-HTTP spans (forward and export `Client` spans, internal stages) record `Error` on any failure. + +Record the reason in `maple.ingest.reject_reason`, never only in `otel.status_description`. `tracing-opentelemetry` turns a description into `Status::error(...)`, and a later `Ok` replaces it, so on an `Ok` rejection the reason would be lost. + +**Python (forward-looking):** `span.set_status(Status(StatusCode.ERROR))`. The exporter encodes the enum as the OTLP status code, which Maple stores as `Error`. Never call `span.set_attribute("otel.status_code", "ERROR")`: it is a plain attribute, not the span status. --- ## Span kinds -The OTel spec defines five kinds. Maple uses three actively: +The OTel spec defines five kinds: | Kind | Use for | Example | |---|---|---| | `Server` | Inbound network request handlers | Ingest gateway `POST /v1/traces`, API HTTP handlers | -| `Client` | Outbound network calls | Ingest forward to downstream collector, fetch to ClickHouse, Cloudflare Email send | +| `Client` | Outbound network calls | Ingest forward to the downstream collector, warehouse queries (`executeSql`), Postgres queries, Cloudflare Email send | | `Internal` | Everything in-process | Query compilation, cache lookups, validation, DSL evaluation | -| `Producer` | (not currently used) | Reserved for future queue producers | -| `Consumer` | (not currently used) | Reserved for future queue consumers | +| `Producer` | Enqueueing a message | Queue producers | +| `Consumer` | Handling a dequeued message | Queue consumers | + +The service map treats `Producer` like `Client` and `Consumer` like `Server`. ### How span kind is set -**TypeScript (Effect):** `Effect.withSpan(name, { kind: "server" })`. The Effect SDK uses lowercase string kinds. Most Effect spans default to `Internal`; set `server` / `client` explicitly when they cross a network boundary. +**TypeScript (Effect):** `Effect.fn(name, { kind: "client" })` or `Effect.withSpan(name, { kind: "server" })`. Effect uses lowercase kind strings. Spans default to `internal`; set `server` / `client` explicitly when they cross a network boundary. Example: `Effect.fn("WarehouseQueryService.executeSql", { kind: "client" })` in `packages/query-engine/src/execution/executor.ts`. **Rust (tracing crate):** Use the `otel.kind` field: @@ -64,24 +72,25 @@ The OTel spec defines five kinds. Maple uses three actively: let span = tracing::info_span!("ingest", otel.kind = "server", /* ... */); ``` -See `apps/ingest/src/main.rs:846` (Server inbound), `:923` (Server inbound for logpush), `:1135` (Client downstream forward), `:1300` (Client logpush downstream). +See `handle_signal` and `handle_cloudflare_logpush` in `apps/ingest/src/main.rs` (Server inbound), and `forward_client_span` / `export_client_span` in `apps/ingest/src/otel.rs` (Client outbound). **Python:** `tracer.start_as_current_span(name, kind=trace.SpanKind.SERVER)`. ### Rule: always set Server / Client at boundaries -If a span handles an inbound request, set `Server`. If it makes an outbound network call, set `Client`. Dashboards and the service map rely on this for correct attribution. Leaving the default `Internal` on a network handler causes the service map to miss the edge. +If a span handles an inbound request, set `Server`. If it makes an outbound network call, set `Client`. The service map draws a service edge only by joining a `Client`/`Producer` span to its child `Server`/`Consumer` span, and counts a database call only on a `Client`/`Producer` span. Leaving the default `Internal` on either side drops the edge. --- ## Reserved OTel fields (Rust ingest) -The `tracing` crate field names below are special — they are interpreted by `tracing-opentelemetry` to populate corresponding OTel span fields rather than custom attributes: +`tracing-opentelemetry` interprets these `tracing` field names as OTel span fields rather than custom attributes: | Field | Maps to | Example value | |---|---|---| | `otel.name` | Span name | `"POST /v1/traces"` | -| `otel.kind` | Span kind | `"server"`, `"client"`, `"internal"` (lowercase in `tracing`; rendered as Title Case `Server`/`Client`/`Internal` on the wire) | -| `otel.status_code` | Span status | `"Ok"` / `"Error"` / `"Unset"` (Title Case — these go directly to the wire string) | +| `otel.kind` | Span kind | `"server"`, `"client"`, `"internal"` (lowercase in `tracing`; stored as `Server`/`Client`/`Internal`) | +| `otel.status_code` | Span status | `"Ok"` / `"Error"` (write Title Case to match the rest of the codebase) | +| `otel.status_description` | Span status message | Sets an `Error` status with this description | -Do not invent custom span attributes named `otel.*` — those slots are reserved. +Do not invent custom span attributes named `otel.*`. Those slots are reserved. diff --git a/CLAUDE.md b/CLAUDE.md index b6315542cb..6587cde3e9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ with `packageManager`. `/mcp`, `/api/chat/*`, `/internal/chat/*` to it over a service binding; OAuth stays in `apps/api`. - An investigation is **one agent turn** ending in `submit_diagnosis`. Do not split it across planner/lanes/validator. Sub-agents are fine elsewhere (e.g. review fan-out) via `SubagentHost`. -- LLM code (Effect AI + `@effect-agent/*`) is only in `apps/ai`; Maple-specific wiring lives in +- LLM code (Effect AI + `effect-agent`) is only in `apps/ai`; Maple-specific wiring lives in `apps/ai/src/platform/Llm.ts`. - Shared services live in `packages/backend`, imported via `@maple/backend/*` subpaths. diff --git a/apps/ai/scripts/generate-dashboard-skill.ts b/apps/ai/scripts/generate-dashboard-skill.ts index 52e172455b..19ea06b54d 100644 --- a/apps/ai/scripts/generate-dashboard-skill.ts +++ b/apps/ai/scripts/generate-dashboard-skill.ts @@ -7,7 +7,7 @@ * descriptions and a hand-maintained `maple://instructions`, which is three * copies of one truth and is exactly how they came to disagree: the skill still * shipped a v2 `{ endpoint, params }` widget template long after the decoders - * moved to v3, and its gauge example paired `unit: "percent"` with a 0–100 arc — + * moved to v3, and its gauge example paired `unit: "percent"` with a 0–100 arc, * the very mistake the write path now warns about. * * bun run --cwd apps/ai mcp:docs # write @@ -29,8 +29,8 @@ const INTRO = ` # Maple dashboard widgets via MCP Everything below is generated from the live widget schema by -\`bun run --cwd apps/ai mcp:docs\`. **Do not edit this file by hand** — edit -\`apps/api/src/mcp/lib/dashboard-schema-doc.ts\` and regenerate. The same module backs the +\`bun run --cwd apps/ai mcp:docs\`. **Do not edit this file by hand.** Edit +\`apps/ai/src/mcp/lib/dashboard-schema-doc.ts\` and regenerate. The same module backs the \`describe_dashboard_schema\` MCP tool, so an agent at runtime and a reader here see one truth. ## When to use this skill @@ -40,7 +40,7 @@ When constructing widget JSON for \`mcp__maple__create_dashboard\`, \`mcp__maple__replace_dashboard_widgets\`. For a brand-new dashboard, prefer the simplified \`widgets\` array on \`create_dashboard\` -(\`{ title, source, metric, group_by?, service_name?, unit? }\`) — it fills in the traps below. +(\`{ title, source, metric, group_by?, service_name?, unit? }\`). It fills in the traps below. Reach for raw JSON when the simplified spec can't express what you need: multi-query charts, formulas, hidden series, non-default transforms. diff --git a/apps/ai/src/chat/review-failure.test.ts b/apps/ai/src/chat/review-failure.test.ts index ff4ba1dbc6..8103cd8e74 100644 --- a/apps/ai/src/chat/review-failure.test.ts +++ b/apps/ai/src/chat/review-failure.test.ts @@ -1,7 +1,7 @@ /** * The words a failed review's pull request is shown come from the run's typed cause, never its text. */ -import { AgentPolicyError, ContextBudgetError, ModelProtocolError } from "@effect-agent/core/AgentError" +import { AgentPolicyError, ContextBudgetError, ModelProtocolError } from "effect-agent/agent-error" import { prReviewFailureReason } from "@maple/domain/http" import { Cause } from "effect" import { assert, describe, it } from "vitest" diff --git a/apps/ai/src/chat/review-failure.ts b/apps/ai/src/chat/review-failure.ts index cf267d27fe..09e1bb800c 100644 --- a/apps/ai/src/chat/review-failure.ts +++ b/apps/ai/src/chat/review-failure.ts @@ -9,7 +9,7 @@ import { ContextBudgetError, ContextOverflowError, ModelProtocolError, -} from "@effect-agent/core/AgentError" +} from "effect-agent/agent-error" import { PR_REVIEW_FAILURE_COPY, type PrReviewFailureReason } from "@maple/domain/http" import { Cause } from "effect" import { isAiError } from "effect/unstable/ai/AiError" diff --git a/apps/ai/src/mcp/lib/dashboard-schema-doc.ts b/apps/ai/src/mcp/lib/dashboard-schema-doc.ts index f1e0d7c594..cf396e49f7 100644 --- a/apps/ai/src/mcp/lib/dashboard-schema-doc.ts +++ b/apps/ai/src/mcp/lib/dashboard-schema-doc.ts @@ -14,9 +14,9 @@ import { makeQueryDraft } from "@maple/backend/dashboard-templates/helpers" /** * The agent-facing description of what a dashboard widget can be. * - * Every table below is *derived* from the definitions the runtime enforces — - * `PANEL_TYPES`, `WIDGET_UNITS`, `AGGREGATIONS_BY_SOURCE`, `GROUP_BY_TOKENS`, - * `WIDGET_DATA_SOURCE_KINDS` — and every JSON example is produced by CALLING the + * Every table below is *derived* from the definitions the runtime enforces + * (`PANEL_TYPES`, `WIDGET_UNITS`, `AGGREGATIONS_BY_SOURCE`, `GROUP_BY_TOKENS`, + * `WIDGET_DATA_SOURCE_KINDS`), and every JSON example is produced by CALLING the * v3 constructors in `@maple/widgets/dashboard` rather than being typed out. * * That is the whole point. The previous documentation was hand-written prose in @@ -27,7 +27,7 @@ import { makeQueryDraft } from "@maple/backend/dashboard-templates/helpers" * schema rejects, and `dashboard-schema-doc.test.ts` decodes each one to prove * it. * - * The prose that remains — the raw-SQL SELECT shapes, the whereClause grammar — + * The prose that remains (the raw-SQL SELECT shapes, the whereClause grammar) * is genuine hand-written knowledge with no machine-readable source, and is * marked as such where it appears. */ @@ -59,10 +59,10 @@ const panelTypesSection = (): string => { if (meta.isScalar) notes.push("needs `transform.reduceToValue`") if (meta.rawSqlDisplayType === undefined) notes.push("**no raw-SQL support**") return `| \`${meta.panelType}\` | ${meta.label} | \`${meta.visualization}\` | ${ - meta.chartId ? `\`${meta.chartId}\`` : "—" - } | ${meta.rawSqlDisplayType ? `\`${meta.rawSqlDisplayType}\`` : "—"} | ${ + meta.chartId ? `\`${meta.chartId}\`` : "-" + } | ${meta.rawSqlDisplayType ? `\`${meta.rawSqlDisplayType}\`` : "-"} | ${ meta.defaultLayout.w - }×${meta.defaultLayout.h} | ${notes.join("; ") || "—"} |` + }×${meta.defaultLayout.h} | ${notes.join("; ") || "-"} |` }) return [ @@ -74,7 +74,7 @@ const panelTypesSection = (): string => { "line/bar/area into `chart` and then needs a `display.chartId` to tell them apart. The two", "columns below are what `panel_type` resolves to, and are what you write directly when", "authoring an assembled widget rather than calling `add_dashboard_widget`. A panel whose", - "`chartId` is `—` takes none: the `visualization` alone identifies it.", + "`chartId` is `-` takes none: the `visualization` alone identifies it.", "", "| panel_type | Label | `visualization` | `display.chartId` | Raw-SQL type | Default w×h | Requirements |", "|---|---|---|---|---|---|---|", @@ -82,20 +82,20 @@ const panelTypesSection = (): string => { "", "### Choosing one", "", - "- **line / area / bar** — a value over time. `area` and `bar` accept `display.stacked`; `line` does not.", - "- **hbar** — a ranked “top N by volume”. Each row is labelled with its share of the **total**.", - "- **funnel** — sequential stages with a drop-off. Labels each bar as a share of the *largest*,", + "- **line / area / bar**: a value over time. `area` and `bar` accept `display.stacked`; `line` does not.", + "- **hbar**: a ranked “top N by volume”. Each row is labelled with its share of the **total**.", + "- **funnel**: sequential stages with a drop-off. Labels each bar as a share of the *largest*,", " so an unranked breakdown of four equal things reads “100%” four times. Use `hbar` for that.", ' `display.funnel.variant: "dropoff"` draws the same funnel as one column per step with the', " loss between steps, the median time between them, and where the leavers went.", - "- **paths** — what people do in the steps after (or before) one event or page, as a flow.", + "- **paths**: what people do in the steps after (or before) one event or page, as a flow.", " Defined by `display.paths` alone; no query set.", - "- **pie** — composition, few slices. Collapses a long tail into “Other”.", - "- **stat / gauge** — one number. A gauge adds an arc; set `display.gauge.min`/`max` to match the unit.", - "- **table** — rows and columns; set `display.columns` for headers and per-column units.", - "- **list** — recent traces/logs. Configured by `display.listDataSource`, never by SQL.", - "- **heatmap / histogram** — a distribution. A histogram over traces can bucket raw values client-side.", - "- **markdown** — a static note. Takes no query at all.", + "- **pie**: composition, few slices. Collapses a long tail into “Other”.", + "- **stat / gauge**: one number. A gauge adds an arc; set `display.gauge.min`/`max` to match the unit.", + "- **table**: rows and columns; set `display.columns` for headers and per-column units.", + "- **list**: recent traces/logs. Configured by `display.listDataSource`, never by SQL.", + "- **heatmap / histogram**: a distribution. A histogram over traces can bucket raw values client-side.", + "- **markdown**: a static note. Takes no query at all.", ].join("\n") } @@ -176,7 +176,7 @@ const exampleFunnelWidget = () => { * sections describe `add_dashboard_widget`'s *parameters*, but * `update_dashboard_widget`, `replace_dashboard_widgets` and `dashboard_json` all * take a whole widget object, and nothing showed one. Each agent inferred the - * envelope — `visualization` + `display.chartId` + `layout` — correctly but said + * envelope (`visualization` + `display.chartId` + `layout`) correctly but said * it was guessing. */ const exampleWidget = () => ({ @@ -200,11 +200,11 @@ const dataSourcesSection = (): string => (kind) => `\`${kind}\``, ).join(", ")}. Every arm requires its \`kind\`.`, "", - '> **If you have seen `{ "endpoint": …, "params": … }` anywhere — that is the retired v2 shape', + '> **If you have seen `{ "endpoint": …, "params": … }` anywhere, that is the retired v2 shape', "> and it will not decode.** A `query` source spreads `queries`/`formulas` at the TOP LEVEL,", "> not under `params`, and requires `resultShape`.", "", - '### `kind: "query"` — the query builder', + '### `kind: "query"`: the query builder', "", "`resultShape` is required and is one of `timeseries` (a value over time), `breakdown`", "(one row per group) or `list` (raw rows). Optional: `formulas`, `comparison`, `limit`,", @@ -212,7 +212,7 @@ const dataSourcesSection = (): string => "", json(exampleQuerySource()), "", - '### `kind: "raw_sql"` — your own ClickHouse SQL', + '### `kind: "raw_sql"`: your own ClickHouse SQL', "", json( makeRawSqlDataSource({ @@ -221,11 +221,11 @@ const dataSourcesSection = (): string => }), ), "", - '### `kind: "static"` — a markdown note, no request', + '### `kind: "static"`: a markdown note, no request', "", json(makeStaticDataSource()), "", - '### `kind: "route"` — a curated built-in panel', + '### `kind: "route"`: a curated built-in panel', "", '`{ "kind": "route", "endpoint": "service_overview", "params": { … } }`. These back the', "prebuilt panels; you rarely author one by hand.", @@ -235,7 +235,7 @@ const dataSourcesSection = (): string => "A `stat` or `gauge` reads `data[0].value`. Without `transform.reduceToValue` it renders", '`[object Object]`. `add_dashboard_widget` injects `{ field: "value", aggregate: "first" }`', "when you omit it; set it explicitly to choose a different reducer. Valid aggregates:", - "`sum`, `first`, `count`, `avg`, `max`, `min` — **there is no `last`**.", + "`sum`, `first`, `count`, `avg`, `max`, `min`: **there is no `last`**.", "", "Which one depends on what the query returns, because the query is bucketed over time and", "the reducer collapses those buckets into one number:", @@ -243,14 +243,14 @@ const dataSourcesSection = (): string => "- **A rate or a count** (`count`, a metrics `rate`) → `sum`, for a window total.", "- **A latency percentile or an average** (`p95_duration`, `avg_duration`, a gauge metric)", " → `avg` for the typical value over the window, or `max` for the worst bucket. **Not**", - " `sum` — adding percentiles together is meaningless, and it is the common wrong choice.", + " `sum`: adding percentiles together is meaningless, and it is the common wrong choice.", "- **A current reading**, where only the newest bucket matters → `first`.", "", json(exampleScalarSource()), "", "### The breakdown shape", "", - '`resultShape: "breakdown"` returns one row per group instead of a series over time — the', + '`resultShape: "breakdown"` returns one row per group instead of a series over time, the', "shape `pie`, `hbar`, `funnel` and `heatmap` need. It requires a group-by, and `limit` caps", "the rows (honoured for 1–100).", "", @@ -260,23 +260,23 @@ const dataSourcesSection = (): string => "", "A funnel widget has two modes. Without `display.funnel.steps` it draws a group-by breakdown", "as descending stages (the shape above). With them it is a **conversion funnel over product", - "events** — page views, `track()` events and server-side events, stitched per person — and", + "events** (page views, `track()` events and server-side events, stitched per person), and", "the query set is not used at all. Set the definition on `display_json.funnel` and", '`add_dashboard_widget` derives the data source (`kind: "route"`,', '`endpoint: "product_events_funnel"`) for you; do not pass `data_source_json`.', "", - '- `steps` — 1–10, in order. `{ kind: "event", eventName, attributeEquals? }`,', - ' `{ kind: "page", pagePath, host? }`, or — **step 1 only** —', + '- `steps`: 1–10, in order. `{ kind: "event", eventName, attributeEquals? }`,', + ' `{ kind: "page", pagePath, host? }`, or (**step 1 only**)', ' `{ kind: "session", dimension, value }` with `dimension` one of `referrerHost`,', " `utmSource`, `utmMedium`, `utmCampaign`, `country`, `host`.", - "- `keyBy` — `person` (default; user id, else the visitor's linked user, else the visitor),", + "- `keyBy`: `person` (default; user id, else the visitor's linked user, else the visitor),", " `visitor`, `user`, or `session`.", - "- `windowSeconds` — the whole chain must complete within this many seconds of step 1", + "- `windowSeconds`: the whole chain must complete within this many seconds of step 1", " (default 86400). Must be positive.", - "- `breakdownBy` — split the funnel by a session dimension (`referrerHost`, `utmSource`,", + "- `breakdownBy`: split the funnel by a session dimension (`referrerHost`, `utmSource`,", " `utmMedium`, `utmCampaign`, `country`, `host`) or `attribute:`; the widget draws one", " bar per group per step for the top 6 groups by step-1 count.", - "- `filters` — the population filter: only persons with a session matching these", + "- `filters`: the population filter: only persons with a session matching these", " dimensions take part. `{ host?, pagePath?, referrerHost?, country?, deviceType?,", " browserName?, osName?, language?, utmSource?, utmMedium?, utmCampaign?,", ' visitorType?: "new" | "returning" }`. They are spread FLAT into the derived route params.', @@ -297,27 +297,27 @@ const dataSourcesSection = (): string => "", '### Paths (`panel_type: "paths"` + `display.paths`)', "", - 'A flow of what people did in the hops after — or, with `direction: "before"`, before —', + 'A flow of what people did in the hops after (or, with `direction: "before"`, before)', "one anchor. Product events only; there is no query set. Set the definition on", '`display_json.paths` and `add_dashboard_widget` derives the data source (`kind: "route"`,', '`endpoint: "product_events_paths"`); do not pass `data_source_json`.', "", - '- `anchor` — required. `{ kind: "event", eventName, attributeEquals? }` or', + '- `anchor`: required. `{ kind: "event", eventName, attributeEquals? }` or', ' `{ kind: "page", pagePath, host? }`. A session step cannot anchor a sequence.', - '- `direction` — `"after"` (default) or `"before"`.', - "- `depth` — hops to draw, 1–5 (default 3). `branches` — nodes named per step, 1–10", + '- `direction`: `"after"` (default) or `"before"`.', + "- `depth`: hops to draw, 1–5 (default 3). `branches`: nodes named per step, 1–10", " (default 4); the rest fold into an *Other* node. A sequence that stops short ends in", " an *Ended* node.", "- `keyBy`, `windowSeconds` (how far from the anchor a hop may be, default 86400) and", - " `filters` — as for funnels.", - '- `include` — `"all"` (default), `"events"` or `"pages"`. `exclude` — names (event names', + " `filters`: as for funnels.", + '- `include`: `"all"` (default), `"events"` or `"pages"`. `exclude`: names (event names', ' or page paths) dropped before sequencing, e.g. `["heartbeat", "/"]`.', "", "### A complete widget", "", "The sections above describe `add_dashboard_widget`'s parameters, which it assembles into a", "widget for you. `update_dashboard_widget`, `replace_dashboard_widgets` and `dashboard_json`", - "take the assembled object instead — this is its shape. `timeRange` and `sectionId`/`tabId`", + "take the assembled object instead. This is its shape. `timeRange` and `sectionId`/`tabId`", "are the only other top-level keys, both optional.", "", json(exampleWidget()), @@ -375,15 +375,15 @@ const queriesSection = (): string => { "metric-only fields belong solely to `metrics` queries; do not add them to trace or log", "queries:", "", - "- `metricName` — required; discover real names with `list_metrics`.", - `- \`metricType\` — required, one of ${QUERY_BUILDER_METRIC_TYPES.map((type) => `\`${type}\``).join(", ")}. Anything else fails to decode.`, - `- \`signalSource\` — optional, one of ${QUERY_BUILDER_SIGNAL_SOURCES.map((source) => `\`${source}\``).join(", ")}. Omit it unless you know you need \`meter\`.`, - "- `isMonotonic` — optional; `false` marks a Sum as an UpDownCounter, which changes the", + "- `metricName`: required; discover real names with `list_metrics`.", + `- \`metricType\`: required, one of ${QUERY_BUILDER_METRIC_TYPES.map((type) => `\`${type}\``).join(", ")}. Anything else fails to decode.`, + `- \`signalSource\`: optional, one of ${QUERY_BUILDER_SIGNAL_SOURCES.map((source) => `\`${source}\``).join(", ")}. Omit it unless you know you need \`meter\`.`, + "- `isMonotonic`: optional; `false` marks a Sum as an UpDownCounter, which changes the", " aggregations that make sense (`rate`/`increase` assume a monotonic counter).", "", "### `addOns` is required, and all five keys must be present", "", - "`addOns: { groupBy, having, orderBy, limit, legend }` — every key, every time. A missing", + "`addOns: { groupBy, having, orderBy, limit, legend }`: every key, every time. A missing", "one fails to decode. Each flag gates whether the matching field is read at all, which is", "why `groupBy` without `addOns.groupBy: true` silently does nothing.", "", @@ -395,7 +395,7 @@ const queriesSection = (): string => { "", `On traces only, setting \`valueField: "attr."\` switches the query to numeric-attribute`, `mode, where the aggregation is one of ${TRACES_NUMERIC_AGGREGATIONS.map((fn) => `\`${fn}\``).join(", ")}.`, - "This is the **only** place a bare `p50`/`p95`/`p99` is valid — latency percentiles are", + "This is the **only** place a bare `p50`/`p95`/`p99` is valid; latency percentiles are", "spelled `p95_duration`. Metrics never accept percentiles.", "", "### Group-by tokens", @@ -412,9 +412,9 @@ const queriesSection = (): string => { "", "### `whereClause` is a custom grammar, not SQL", "", - "Operators — the only ones: `=`, `!=`, `>`, `<`, `>=`, `<=`, `contains`, `!contains`,", + "Operators (the only ones): `=`, `!=`, `>`, `<`, `>=`, `<=`, `contains`, `!contains`,", "`exists`, `!exists`. Clauses join with ` AND `; there is no `OR` and no parentheses.", - "Values use double quotes. **There is no `IS NULL` / `IS NOT NULL`** — write ` exists`", + "Values use double quotes. **There is no `IS NULL` / `IS NOT NULL`**: write ` exists`", "or ` !exists`. `exists` means present *and* non-empty, because attributes live in", "ClickHouse `Map` columns where a missing key reads back as `''`.", "", @@ -427,7 +427,7 @@ const queriesSection = (): string => { "", "`formulas: [{ id, name, expression, legend }]` references queries by `name` (`A / B`), and", "is valid on the **timeseries** shape only. Marking a query `hidden: true` is UI-only in raw", - 'JSON — also add `transform.hideSeries.baseNames: ["A"]` or the auxiliary series renders at', + 'JSON, so also add `transform.hideSeries.baseNames: ["A"]` or the auxiliary series renders at', "full scale and flattens the axis.", ].join("\n") } @@ -441,7 +441,7 @@ const displaySection = (): string => "| Key | Applies to | Notes |", "|---|---|---|", "| `title`, `description` | all | |", - "| `unit` | all | See the `units` section — read it before choosing a percent token. |", + "| `unit` | all | See the `units` section: read it before choosing a percent token. |", "| `thresholds` | stat, gauge, charts | `[{ value, color, label? }]`; highest matching value wins. |", "| `prefix`, `suffix` | stat, gauge | Wrap the formatted value. |", "| `chartPresentation.legend` | charts | `visible` \\| `hidden` \\| `right`. |", @@ -454,12 +454,12 @@ const displaySection = (): string => "| `columns` | table, list | `[{ field, header, unit?, width?, align?, hidden? }]`. |", "| `listDataSource`, `listWhereClause`, `listLimit`, `listRootOnly` | list | |", "| `pie` | pie | `{ donut, innerRadius, showLabels, showPercent }`. |", - "| `gauge` | gauge | `{ min, max }` — defaults to 0–100, which is wrong for a `percent` unit. |", + "| `gauge` | gauge | `{ min, max }`: defaults to 0–100, which is wrong for a `percent` unit. |", "| `histogram` | histogram | `{ bucketCount, bucketWidth, logScaleY }`. |", "| `heatmap` | heatmap | `{ colorScale, scaleType }`. |", - '| `funnel` | funnel | `{ showStepPercent, variant?, steps?, keyBy?, windowSeconds?, breakdownBy?, filters? }` — with `steps` it is a product-event funnel (see Data sources); `variant: "dropoff"` is the drop-off view. |', - "| `paths` | paths | `{ anchor, direction?, depth?, branches?, keyBy?, windowSeconds?, include?, exclude?, filters? }` — the whole definition (see Data sources). |", - "| `markdown` | markdown | `{ content }` — the note body. |", + '| `funnel` | funnel | `{ showStepPercent, variant?, steps?, keyBy?, windowSeconds?, breakdownBy?, filters? }`: with `steps` it is a product-event funnel (see Data sources); `variant: "dropoff"` is the drop-off view. |', + "| `paths` | paths | `{ anchor, direction?, depth?, branches?, keyBy?, windowSeconds?, include?, exclude?, filters? }`: the whole definition (see Data sources). |", + "| `markdown` | markdown | `{ content }`: the note body. |", "| `sparkline` | stat | `{ enabled, dataSource? }`; embeds a full nested data source. |", "", "### Stored but not rendered", @@ -486,7 +486,7 @@ const rawSqlSection = (): string => "## Raw SQL widgets", "", "Pass `sql` to `add_dashboard_widget` and the tool builds the data source for you.", - "**Call `describe_warehouse_tables` first** — a hallucinated table or column silently", + "**Call `describe_warehouse_tables` first**: a hallucinated table or column silently", "produces an empty chart.", "", "### Macros", @@ -498,27 +498,27 @@ const rawSqlSection = (): string => "", "### Conventions that catch everyone", "", - "- Columns are PascalCase (`ServiceName`, `Timestamp`) — never snake_case.", + "- Columns are PascalCase (`ServiceName`, `Timestamp`), never snake_case.", "- `StatusCode` / `SeverityText` / `SpanKind` values are **Title Case** (`'Error'`, not `'ERROR'`).", " Wrong casing runs fine and matches zero rows.", "- Span `Duration` is **nanoseconds**. Divide by `1e6` for ms.", - "- `SpanAttributes['key']` — square brackets. A missing key returns `''`, not NULL.", + "- `SpanAttributes['key']`: square brackets. A missing key returns `''`, not NULL.", "- One statement only; writes are rejected; every query is wrapped in `LIMIT 1001`.", "", "### What to SELECT, per panel type", "", "The renderer is opinionated. Wrong aliases give an empty chart or `[object Object]`.", "", - "- **line / area / bar** — a DateTime bucket as the FIRST column (alias `bucket`) plus one or", + "- **line / area / bar**: a DateTime bucket as the FIRST column (alias `bucket`) plus one or", " more **numeric** columns; each becomes a series named after the column. **String columns", - " are dropped**, so multi-series must be pivoted in SQL with `countIf(...)` — tall form", + " are dropped**, so multi-series must be pivoted in SQL with `countIf(...)`; tall form", " (`bucket, ServiceName, count()`) collapses to one aggregate line.", - "- **stat / gauge** — one scalar aliased `value`.", - "- **pie / funnel / hbar** — a string column aliased `name` plus a numeric column. Cap at ~8–10 rows.", - "- **heatmap** — three columns aliased `x`, `y`, `value`; string-cast numeric `x`/`y`.", - "- **histogram** — one numeric column aliased `value`, one row per observation; add `LIMIT 5000`.", - "- **table** — any rows; columns render in order, so use `AS` for readable headers.", - "- **list** — not supported. A list is configured by `display.listDataSource`.", + "- **stat / gauge**: one scalar aliased `value`.", + "- **pie / funnel / hbar**: a string column aliased `name` plus a numeric column. Cap at ~8–10 rows.", + "- **heatmap**: three columns aliased `x`, `y`, `value`; string-cast numeric `x`/`y`.", + "- **histogram**: one numeric column aliased `value`, one row per observation; add `LIMIT 5000`.", + "- **table**: any rows; columns render in order, so use `AS` for readable headers.", + "- **list**: not supported. A list is configured by `display.listDataSource`.", "", "```sql", "SELECT toStartOfInterval(Timestamp, INTERVAL $__interval_s SECOND) AS bucket,", @@ -532,7 +532,7 @@ const rawSqlSection = (): string => "", "`granularity_seconds` only matters if the SQL references `$__interval_s`. Either use", "`toStartOfInterval(…, INTERVAL $__interval_s SECOND)` with it, or a fixed `toStartOf*`", - "without it — mixing them means the setting silently does nothing.", + "without it; mixing them means the setting silently does nothing.", ].join("\n") const SECTION_RENDERERS = { @@ -547,7 +547,7 @@ const SECTION_RENDERERS = { const SECTION_SUMMARIES = { panel_types: "The 14 panel types, what each persists as, and which need a group-by or a reduction.", data_sources: "The four `kind` arms of a widget data source, with a decodable example of each.", - units: "The unit vocabulary — and the percent-vs-percent_100 scale rule.", + units: "The unit vocabulary, and the percent-vs-percent_100 scale rule.", queries: "Aggregations and group-by tokens per source, the whereClause grammar, formulas.", display: "Display config keys per panel type, including the ones that are stored but inert.", raw_sql: "Macros, ClickHouse conventions, and the SELECT shape each panel type expects.", @@ -563,7 +563,7 @@ export const renderDashboardSchemaIndex = (): string => "", 'Call this tool again with `section: ""` for any of:', "", - ...DASHBOARD_SCHEMA_SECTIONS.map((section) => `- \`${section}\` — ${SECTION_SUMMARIES[section]}`), + ...DASHBOARD_SCHEMA_SECTIONS.map((section) => `- \`${section}\`: ${SECTION_SUMMARIES[section]}`), "", "Two things to read before authoring anything:", "", diff --git a/docs/api-cpu-investigation-2026-09-12.md b/docs/api-cpu-investigation-2026-09-12.md index f702965ebf..824d3532e5 100644 --- a/docs/api-cpu-investigation-2026-09-12.md +++ b/docs/api-cpu-investigation-2026-09-12.md @@ -1,4 +1,4 @@ -# API CPU improvements — 2026-09-12 +# API CPU improvements (2026-09-12) Implemented locally against revision `79dfb6da9d`, with measurements after each logical change. No production deployment or production CPU profile was taken. diff --git a/docs/api-v1-retirement-audit.md b/docs/api-v1-retirement-audit.md index 4257e103b8..1ae766ebf1 100644 --- a/docs/api-v1-retirement-audit.md +++ b/docs/api-v1-retirement-audit.md @@ -24,7 +24,7 @@ The executable inventory comes from [MapleApi](../packages/domain/src/http/api.t Already migrated: dashboard CRUD, alerts, API/ingest keys, attribute mappings, scrape targets, investigations, anomalies, audit, chat and PlanetScale controls. CLI remote telemetry uses [MapleApiV2](../apps/cli/src/core/v2-client.ts). The Alchemy provider's resource operations use `/v2` too; its service named `MapleApi` is not the legacy contract. Query-engine reads, authenticated billing, demo, digest, AI triage, replay facets/trace summaries, and AI-session dashboard operations already use `MapleInternalApi`. -MCP issue tools call backend services directly. For example, [claim-error-issue.ts](../apps/api/src/mcp/tools/claim-error-issue.ts) calls `ErrorsService`, with identity resolved by [resolve-actor.ts](../apps/api/src/mcp/lib/resolve-actor.ts). Retiring an HTTP route does not authorize deleting the service or narrowing MCP authentication. +MCP issue tools call backend services directly. For example, [claim-error-issue.ts](../apps/ai/src/mcp/tools/claim-error-issue.ts) calls `ErrorsService`, with identity resolved by [resolve-actor.ts](../apps/ai/src/mcp/lib/resolve-actor.ts). Retiring an HTTP route does not authorize deleting the service or narrowing MCP authentication. ## Findings that change the migration @@ -63,7 +63,7 @@ These are separate reviewable changes; Hazel and organization deletion do not de ### 4. Split errors by consumer intent -- Add stable v2 issue subresources for events, comments, transitions, severity, linked PRs, and verification results—the operations the dashboard currently uses. Add assignment/fix proposals/incident listing and notification-policy resources if these remain supported HTTP capabilities; otherwise explicitly retire their unused HTTP routes after the gate. Add assignee/archive list filters before claiming full list parity. +- Add stable v2 issue subresources for events, comments, transitions, severity, linked PRs, and verification results: the operations the dashboard currently uses. Add assignment/fix proposals/incident listing and notification-policy resources if these remain supported HTTP capabilities; otherwise explicitly retire their unused HTTP routes after the gate. Add assignee/archive list filters before claiming full list parity. - Move dashboard claim/heartbeat/release and escalation-policy evaluation/history to dedicated `/internal` groups. Decide agent registration/listing and policy-management destinations from supported callers rather than copying all 26 operations wholesale. Existing MCP automation keeps using the shared services and its existing credential/actor resolution. - Preserve the lifecycle in [error-issue-lifecycle.md](error-issue-lifecycle.md): machine-owned states, lease ownership, human versus agent severity precedence, PR linking, and post-merge verification. New v2 mutation handlers must resolve the correct actor for both sessions and scoped API keys; blindly copying v1's `ensureUserActor` calls would misattribute agent actions. - Keep semantic tagged failures through the v2 envelope. Verify tenant isolation, role/scope refusals, conflicting leases, invalid transitions, duplicate mutation/retry behavior, activity pagination (the UI currently requests 200 events, above v2's 100-row cap), cache invalidation, and the issue's detail/bulk-action UI. diff --git a/docs/api-v2.md b/docs/api-v2.md index 21da819984..1283df3e0e 100644 --- a/docs/api-v2.md +++ b/docs/api-v2.md @@ -1,8 +1,8 @@ # Maple v2 Public API -The Maple v2 API is the public, documented, stability-committed HTTP surface for customer-stable Maple resources and workflows. It follows Stripe's API design philosophy — resource-oriented URLs, prefixed object IDs, uniform list/error envelopes, scoped keys — modernized where Stripe's v1 mechanics are legacy (JSON PATCH updates instead of form-encoded POST, ISO-8601 timestamps instead of epoch seconds). +The Maple v2 API is the public, documented, stability-committed HTTP surface for customer-stable Maple resources and workflows. It follows Stripe's API design: resource-oriented URLs, prefixed object IDs, uniform list/error envelopes, and scoped keys. It departs from Stripe's v1 mechanics where those are legacy (JSON PATCH updates instead of form-encoded POST, ISO-8601 timestamps instead of epoch seconds). -The **executable contract is the spec**: `MapleApiV2` in `packages/domain/src/http/v2/` (an Effect `HttpApi`). OpenAPI is derived from it automatically and served as an interactive reference at **`/v2/docs`**. This document is the design-guideline layer every v2 contract file must conform to, plus the roadmap for the full surface. +The **executable contract is the spec**: `MapleApiV2` in `packages/domain/src/http/v2/` (an Effect `HttpApi`). OpenAPI is derived from it automatically and served as an interactive reference at **`/v2/docs`**. This document holds the design guidelines every v2 contract file must follow, plus the roadmap for the full surface. ## Architecture: two tiers @@ -11,23 +11,23 @@ The **executable contract is the spec**: `MapleApiV2` in `packages/domain/src/ht | **Public API** | `MapleApiV2` HttpApi at `/v2/...` | Customers, agents/MCP, the dashboard | `/v2/docs` (OpenAPI/Scalar) | Committed; changes are additive or versioned | | **Internal RPC** | Effect RPC (`effect/unstable/rpc`) `RpcGroup`s served at `/rpc` | The dashboard only | none (private) | None; changes freely | -Dashboard-only operations — billing checkout/portal, onboarding state, demo seeding, AI chat apply, digest subscription, AI-triage settings, raw warehouse queries, and the error-agent claim/heartbeat/release loop — belong in the internal RPC tier. They use the same tenant resolution and org scoping but are **not** HTTP API groups and never appear in the public OpenAPI. Everything else is public API, and the dashboard consumes the same `/v2` endpoints customers do. +Dashboard-only operations belong in the internal RPC tier: billing checkout/portal, onboarding state, demo seeding, AI chat apply, digest subscription, AI-triage settings, raw warehouse queries, and the error-agent claim/heartbeat/release loop. They use the same tenant resolution and org scoping but are **not** HTTP API groups and never appear in the public OpenAPI. Everything else is public API, and the dashboard consumes the same `/v2` endpoints customers do. -The v1 API (`/api/...`) stays mounted while the dashboard migrates group-by-group; each v1 group is deleted once nothing consumes it. The audited group-by-group destination and removal gate live in [`http-api-migration.md`](http-api-migration.md). **The RPC tier is Phase 3 and not built yet** — `packages/domain/src/internal-rpc.ts` holds service-to-service schemas, not `RpcGroup`s — so until it exists the only two real homes for a new operation are v2 or the legacy v1 group it would extend. New surface goes to v2; v1 only grows where an existing v1 group already owns the resource. +The v1 API (`/api/...`) stays mounted while the dashboard migrates group-by-group; each v1 group is deleted once nothing consumes it. The audited group-by-group destination and removal gate live in [`http-api-migration.md`](http-api-migration.md). **The RPC tier is Phase 3 and not built yet**: no `RpcGroup` exists in the repo. Until it does, a new operation goes either to v2 or to the legacy v1 group it would extend. New surface goes to v2; v1 only grows where an existing v1 group already owns the resource. ### Integration endpoints: which tier -Integrations are the one family split across tiers, so the rule is explicit. **The OAuth handshake itself is never an API group in either tier**: the provider redirects a _browser_, so the callback is a raw `HttpRouter` route in `apps/api/src/routes/` that ends in a 302 back to the web app (see `routes/v1/chat-integration.http.ts`). What lands in an API group is the surrounding control surface — status, begin-install, uninstall, and provider resource lookups. +Integrations are the one family split across tiers, so the rule is explicit. **The OAuth handshake itself is never an API group in either tier**: the provider redirects a _browser_, so the callback is a raw `HttpRouter` route in `apps/api/src/routes/` that ends in a 302 back to the web app (see `apps/api/src/routes/v1/chat-integration.http.ts`). What lands in an API group is the surrounding control surface: status, begin-install, uninstall, and provider resource lookups. -That control surface is public v2 when a **public v2 resource depends on it**, and internal otherwise. Chat connectors qualify: `/v2/alerts/destinations` accepts `type: "chat"` with a `workspace_id` and a `channel_id`, and the connector's credential never leaves the server, so `GET /v2/integrations/chat_workspaces/{id}/destinations` is the only way any caller — customer, agent, or the dashboard — can discover a valid channel id. Withholding it would ship a public destination type nobody outside the dashboard could construct. The rest of the chat group comes along because splitting one provider across two tiers costs more than it buys; `install` is documented as browser-oriented since a headless caller cannot finish the redirect. +That control surface is public v2 when a **public v2 resource depends on it**, and internal otherwise. Chat connectors qualify: `/v2/alerts/destinations` accepts `type: "chat"` with a `workspace_id` and a `channel_id`, and the connector's credential never leaves the server, so `GET /v2/integrations/chat_workspaces/{id}/destinations` is the only way any caller (customer, agent, or the dashboard) can discover a valid channel id. Withholding it would ship a public destination type nobody outside the dashboard could construct. The rest of the chat group comes along because splitting one provider across two tiers costs more than it buys; `install` is documented as browser-oriented since a headless caller cannot finish the redirect. Within that group the role requirement is not uniform. Install, settings, unlink and the destinations listing require the org-admin role; the connector list does not, because the dashboard renders link state for every member. The destinations listing is gated because it lists channels the bot can reach, which is workspace membership information rather than Maple state. -PlanetScale was the first promotion under that rule, and it shows what "a public dependency" means in practice: infrastructure-as-code setups drive `POST /v2/integrations/planetscale/metrics_token` — the one step the OAuth flow cannot cover, because PlanetScale's metrics endpoints authenticate with service tokens rather than OAuth bearers — and a scripted caller needs a scoped API key, which only v2 has. Once one operation of a provider is public the rest follows: splitting one provider across two tiers costs more than it buys, so the whole surface moved (status, connect, organization binding, disconnect, inventory, webhook config, query insights, events). The role split matches the chat group's — `status`, `databases`, `query_insights`, and `events` are ungated because the dashboard and service map render them for every member; everything that writes, plus `organizations` and `webhook_config` (it carries the signing secret), requires org-admin. +PlanetScale was the first promotion under that rule, and it shows what "a public dependency" means in practice: infrastructure-as-code setups drive `POST /v2/integrations/planetscale/metrics_token`, and a scripted caller needs a scoped API key, which only v2 has. That is the one step the OAuth flow cannot cover, because PlanetScale's metrics endpoints authenticate with service tokens instead of OAuth bearers. Once one operation of a provider is public the rest follows: splitting one provider across two tiers costs more than it buys, so the whole surface moved (status, connect, organization binding, disconnect, inventory, webhook config, query insights, events). The role split matches the chat group's: `status`, `databases`, `query_insights`, and `events` are ungated because the dashboard and service map render them for every member; everything that writes, plus `organizations` and `webhook_config` (it carries the signing secret), requires org-admin. -Promotion does not imply deletion. PlanetScale's v1 endpoints stay mounted and are marked `deprecated` in the v1 OpenAPI: the dashboard is off them, so nothing in this repo would notice if they broke, but customers may still be calling them. The general rule in the tier table above — "each v1 group is deleted once nothing consumes it" — means _nothing_, including callers outside this repo, so a promoted provider's v1 surface is removed only once its access logs go quiet. Until then both surfaces are live over the same services, and only v2 gets new work. +Promotion does not imply deletion. PlanetScale's v1 endpoints stay mounted and are marked `deprecated` in the v1 OpenAPI: the dashboard is off them, so nothing in this repo would notice if they broke, but customers may still be calling them. The general rule above ("each v1 group is deleted once nothing consumes it") means _nothing_, including callers outside this repo, so a promoted provider's v1 surface is removed only once its access logs go quiet. Until then both surfaces are live over the same services, and only v2 gets new work. -Cloudflare, GitHub, and Hazel stay on the v1 `integrations` group (`/api/integrations`) with no v2 counterpart — they predate v2, nothing public depends on them, and per the rule above they get promoted individually if and when something does. Scope families are derived mechanically from the first path segment under `/v2`, so every provider mounted at `/v2/integrations/` shares the `integrations:read` / `integrations:write` family. +Cloudflare, GitHub, and Hazel stay on the v1 `integrations` group (`/api/integrations`) with no v2 counterpart. They predate v2 and nothing public depends on them. Per the rule above, each gets promoted individually if something does. Scope families are derived mechanically from the first path segment under `/v2`, so every provider mounted at `/v2/integrations/` shares the `integrations:read` / `integrations:write` family. Two things never move with a provider, no matter how much of it is promoted: the OAuth **callback** and any webhook **receiver**. Both are raw `HttpRouter` routes under `/api/…` serving a browser redirect or a provider POST, and a receiver URL is already registered in the provider's settings. So `POST /v2/integrations/planetscale/connect` still mints a `/api/integrations/planetscale/callback` URL, and `webhook_config` still reports a `/api/…` receiver. That is the intended end state, not leftover v1. @@ -48,9 +48,9 @@ POST /v2/traces/search complex reads are POST .../search ### Object IDs -Every v2 object has a prefixed public ID (`key_4CzLmR…`, `dash_…`, `alrt_…`). Public IDs are opaque; internally they are a reversible base58 encoding of the internal ID, computed at the API boundary (`packages/domain/src/http/v2/public-id.ts` — the prefix registry lives there and is the single source of truth). No database migration: rows keep their raw UUIDs / internal strings. +Every v2 object has a prefixed public ID (`key_4CzLmR…`, `dash_…`, `alrt_…`). Public IDs are opaque; internally they are a reversible base58 encoding of the internal ID, computed at the API boundary in `packages/domain/src/http/v2/public-id.ts`, which also holds the prefix registry (`PublicIdPrefixes`, the source of truth). No database migration: rows keep their raw UUIDs / internal strings. -Prefixes: `key` (API key), `ingk` (ingest key), `dash` (dashboard), `dbv` (dashboard version), `dtpl` (dashboard template), `alrt` (alert rule), `dest` (alert destination), `inc` (alert incident), `evt` (alert delivery event), `einc` (error incident), `iss` (error issue), `inv` (investigation), `anom` (anomaly incident), `scrp` (scrape target), `rec` (recommendation), `amap` (attribute mapping), `srep` (session replay), and `log` (synthetic log identity); `we` is reserved for webhooks. +Prefixes: `key` (API key), `ingk` (ingest key), `dash` (dashboard), `dbv` (dashboard version), `dshr` (dashboard share), `dtpl` (dashboard template), `alrt` (alert rule), `dest` (alert destination), `inc` (alert incident), `evt` (alert delivery event), `actor` (actor), `alog` (audit log entry), `einc` (error incident), `iss` (error issue), `inv` (investigation), `anom` (anomaly incident), `scrp` (scrape target), `rec` (recommendation), `amap` (attribute mapping), `srep` (session replay), `mdev` (mobile device), `chatw` (chat workspace), and `log` (synthetic log identity); `we` is reserved for webhooks. Exception: Clerk-issued `org_…` / `user_…` IDs are already prefixed public IDs and pass through unchanged. @@ -76,7 +76,7 @@ Every list endpoint accepts `limit` (1–100, default 20) and an opaque `cursor` } ``` -`next_cursor` is `null` on the last page. Cursors are opaque — clients must not parse them. (Endpoints backed by keyset pagination and endpoints backed by materialized arrays use different cursor payloads; the wire contract is identical.) +`next_cursor` is `null` on the last page. Cursors are opaque: clients must not parse them. (Endpoints backed by keyset pagination and endpoints backed by materialized arrays use different cursor payloads; the wire contract is identical.) Offset-backed endpoints fetch `limit + 1` rows from their backing store to determine `has_more`; they do not load a fixed history window and paginate it in memory. Every ordering has a deterministic tie-breaker. @@ -122,10 +122,10 @@ v2 accepts the same credentials as v1: API keys (`maple_ak_…`) and dashboard s - Grammar: `:read`, `:write`, or `*`. The family is the first path segment under `/v2` (`api_keys`, `dashboards`, `alerts`, `error_issues`, `traces`, …). - Enforcement is mechanical: `GET`/`HEAD` and explicitly declared read-only query POSTs (such as session-replay search, trace lookup, and alert preview) require `:read`; mutations require `:write`. `write` implies `read`. -- Keys with no scopes (all pre-v2 keys) have full access. Session tokens are never scope-checked — the dashboard's authorization comes from org roles, like Stripe's own dashboard. +- Keys with no scopes (all pre-v2 keys) have full access. Session tokens are never scope-checked. The dashboard's authorization comes from org roles, like Stripe's own dashboard. - Failing the check returns `permission_error` / `insufficient_scope`. -Implementation: `packages/domain/src/http/v2/auth.ts` + `apps/api/src/services/ApiAuthorizationV2Layer.ts`; scopes are stored on `api_keys.scopes` (jsonb). +Implementation: `packages/domain/src/http/v2/auth.ts` + `packages/backend/src/services/auth/ApiAuthorizationV2Layer.ts`; scopes are stored on `api_keys.scopes` (jsonb). ### Versioning @@ -133,7 +133,7 @@ Implementation: `packages/domain/src/http/v2/auth.ts` + `apps/api/src/services/A - Within v2, resource shapes evolve additively. Error `_tag` values are the compatibility identity; `code`, title, message, recovery hints, and the broad HTTP category are presentation and may be corrected without minting a new API version. - A `Maple-Version: YYYY-MM-DD` header is reserved for future in-v2 evolution; until multiple versions exist, it is accepted and ignored. -### Idempotency (Phase 4 — reserved) +### Idempotency (Phase 4, reserved) Mutating endpoints will accept an `Idempotency-Key` header. Replays within the retention window return the original response. Backed by a Postgres `idempotency_keys` table keyed by `(org_id, key)`. @@ -143,9 +143,9 @@ API-key-authenticated requests share one budget per key across the entire `/v2` Exceeding the budget returns `429` with the standard error envelope (`type: "rate_limit_error"`, `code: "rate_limited"`) and `Retry-After: 60`. No remaining/reset headers are emitted because the backing Cloudflare binding returns only an allow/deny result. -The limiter fails open: if its binding is missing or errors, Maple logs and traces `maple.rate_limit.outcome=failed_open` and continues the request. Cloudflare counters are local to the location serving the Worker and eventually consistent, so this is abuse protection rather than exact global quota accounting. +The limiter (`packages/backend/src/services/auth/ApiV2RateLimiter.ts`) fails open: if its binding is missing or errors, Maple logs and traces `maple.rate_limit.outcome=failed_open` and continues the request. Cloudflare counters are local to the location serving the Worker and eventually consistent, so this is abuse protection rather than exact global quota accounting. -### Expansion — not supported +### Expansion: not supported Stripe-style `expand[]` is deliberately omitted: responses embed the small, always-wanted sub-objects directly. May be revisited once real client demand exists. @@ -162,7 +162,7 @@ Implemented in phases; the pilot (`api_keys`) ships first and proves every conve | `alerts/destinations` ✅ | CRUD + `test` | `AlertsService` | | `alerts/incidents` ✅ | list/retrieve | `AlertsService` | | `alerts/deliveries` ✅ | list delivery attempts | `AlertsService` | -| `error_issues` 🟡 | list/retrieve ✅; `events`, `comments`, `transitions`, `assignee`, `severity` deferred | `errors` | +| `error_issues` 🟡 | list/retrieve/`service_counts` ✅; `events`, `comments`, `transitions`, `assignee`, `severity` deferred | `errors` | | `investigations` ✅ | list/retrieve/create/status | `InvestigationService` | | `anomalies` ✅ | incidents list/retrieve/timeseries/resolve/link-issue, `service_counts` aggregate + `PATCH` settings | `anomalies` | | `instrumentation/recommendations` ✅ | list + dismiss/reopen | `RecommendationIssueService` | @@ -181,6 +181,8 @@ Implemented in phases; the pilot (`api_keys`) ships first and proves every conve | `services` ✅ | `GET /v2/services`, `GET /v2/services/{name}`, `GET /v2/services/{name}/overview` (summary + all-signal series + operations, one round-trip) | `queryEngine` | | `service_map` ✅ | `GET /v2/service_map` | `queryEngine` | +Also mounted, outside this table: `audit_log` (list), `environments`, `instrumentation/signals` (retrieve), `widget_credentials` (mint/revoke), `widget_summary` (retrieve), and `share` (share-link reads). The contracts are in `packages/domain/src/http/v2/`. + The long tail of ~40 query-engine endpoints (facets, infra hosts/pods/nodes/workloads, Cloudflare infra, the PlanetScale infra timeseries) lives in the internal tier at `/internal/query-engine`, session-only and undocumented, and is promoted into `/v2` individually as shapes stabilize. ### Telemetry reads @@ -203,18 +205,18 @@ The dashboard can reconcile optimistic writes against ElectricSQL synced shapes ## Rollout phases -- **Phase 0 (this change)** — conventions doc, v2 primitives (`public-id`, `envelopes`, `errors`, `auth`), scoped API keys (schema + service + enforcement), `MapleApiV2` shell mounted at `/v2` with Scalar docs at `/v2/docs`, pilot resource `api_keys` end-to-end with tests. -- **Phase 1 — core resources**: dashboards, alerts, error issues, scrape targets, ingest keys, attribute mappings, investigations, anomalies, recommendations, organization, session replays. Thin handler adapters over existing services; `txid` preserved. -- **Phase 2 ✅ — telemetry reads**: signal-scoped traces/logs/metrics query operations plus services/service_map over `QueryEngineService`. -- **Phase 3 — internal RPC tier + dashboard migration**: `RpcGroup` contracts in `packages/domain/src/rpc/` served at `/rpc`; dashboard gets a `MapleApiV2AtomClient` (same wiring as `apps/web/src/lib/services/common/atom-client.ts`, pointed at `MapleApiV2`) plus an `RpcClient`; migrate group-by-group, deleting v1 groups as they empty. (Note: the billing-scoped 401 retry in `atom-client.ts` must follow billing to its RPC home.) -- **Phase 4 — hardening**: `Idempotency-Key`, `Maple-Version` header enforcement. Per-key rate limiting is implemented. -- **Phase 5 — events & webhooks**: `evt_` event objects, `GET /v2/events`, `/v2/webhook_endpoints` CRUD, HMAC-signed deliveries (`Maple-Signature`) via an outbox drained by the alerting worker. +- **Phase 0 ✅**: conventions doc, v2 primitives (`public-id`, `envelopes`, `errors`, `auth`), scoped API keys (schema + service + enforcement), `MapleApiV2` shell mounted at `/v2` with Scalar docs at `/v2/docs`, pilot resource `api_keys` end-to-end with tests. +- **Phase 1, core resources**: dashboards, alerts, error issues, scrape targets, ingest keys, attribute mappings, investigations, anomalies, recommendations, organization, session replays. Thin handler adapters over existing services; `txid` preserved. +- **Phase 2 ✅, telemetry reads**: signal-scoped traces/logs/metrics query operations plus services/service_map over `QueryEngineService`. +- **Phase 3, internal RPC tier + dashboard migration**: `RpcGroup` contracts (planned home `packages/domain/src/rpc/`, not created yet) served at `/rpc`; dashboard gets a `MapleApiV2AtomClient` (same wiring as `apps/web/src/lib/services/common/atom-client.ts`, pointed at `MapleApiV2`) plus an `RpcClient`; migrate group-by-group, deleting v1 groups as they empty. (Note: the billing-scoped 401 retry in `atom-client.ts` must follow billing to its RPC home.) +- **Phase 4, hardening**: `Idempotency-Key`, `Maple-Version` header enforcement. Per-key rate limiting is implemented. +- **Phase 5, events & webhooks**: `evt_` event objects, `GET /v2/events`, `/v2/webhook_endpoints` CRUD, HMAC-signed deliveries (`Maple-Signature`) via an outbox drained by the alerting worker. ## Adding a v2 resource (checklist) 1. Contract in `packages/domain/src/http/v2/.ts`: snake_case wire schemas with an `object` literal and validated `Timestamp` fields; public IDs via `PublicId(prefix, InternalId)` (register the prefix in `public-id.ts`); lists use `ListQuery` + `ListOf`; every endpoint lists its exact error schemas with `publicError(ErrorClass)` and/or explicit boundary definitions (shared exhaustive sets such as `V2WarehouseErrors` are fine); group `.prefix("/v2/")` + `.middleware(AuthorizationV2)`. Request-validation, authorization, and unexpected-error middleware contribute their own exact tags API-wide; do not attach them per group. -2. Add the group to `MapleApiV2` in `v2/api.ts` and export from `v2/index.ts`. +2. Add the group to `MapleApiV2` in `packages/domain/src/http/v2/api.ts` and export it from `v2/index.ts`. 3. Define expected failures with `HttpTaggedError` in the domain contract. Put their public status, code, safe-copy policy, retry behavior, and recovery action on the class. -4. Handlers in `apps/api/src/routes/v2/.http.ts`: thin adapters over the existing service — map camelCase/epoch-ms service responses to the wire model and let expected tagged errors pass through unchanged. Register the layer in `ApiV2Routes` (`apps/api/src/runtime/http-graph.ts`). +4. Handlers in `apps/api/src/routes/v2/.http.ts`: thin adapters over the existing service. Map camelCase/epoch-ms service responses to the wire model and let expected tagged errors pass through unchanged. Register the layer in `ApiV2Routes` (`apps/api/src/runtime/http-graph.ts`). 5. Tests: wire-shape encode (snake_case, public ID, envelope), public error serialization/redaction, and a PGlite service test if the service changed. 6. Confirm the resource renders correctly at `/v2/docs`. diff --git a/docs/attributes.md b/docs/attributes.md index 3d9b8fc325..fbfd8a5da0 100644 --- a/docs/attributes.md +++ b/docs/attributes.md @@ -1,25 +1,25 @@ # Attributes Maple recognizes -Maple stores every OTel attribute you send verbatim, but a curated set get special treatment — pre-extracted into fast columns at ingest, exposed as short filter aliases, rendered as colored badges, used to draw the service map, or scored higher in the attribute chip strip. This page is organized by **feature you want to unlock**, so you can pick the attributes worth instrumenting and skip the rest. +Maple stores every OTel attribute you send verbatim. A curated set gets special treatment: pre-extracted into fast columns at ingest, exposed as short filter aliases, rendered as colored badges, used to draw the service map, or scored higher in the log attribute chips. This page is organized by **the feature you want to unlock**, so you can pick the attributes worth instrumenting and skip the rest. -Most of these follow the [OTel semantic conventions](https://opentelemetry.io/docs/specs/semconv/) — if your SDK emits standard attributes you usually don't need to do anything extra. +Most of these follow the [OTel semantic conventions](https://opentelemetry.io/docs/specs/semconv/). If your SDK emits standard attributes you usually don't need to do anything extra. -The [`maple-audit` skill](../skills/maple-audit/SKILL.md) encodes this registry as an auditable checklist (`checks.md`) — keep the two in sync when adding attributes here. +The [`maple-audit` skill](../skills/maple-audit/SKILL.md) encodes this registry as an auditable checklist (`checks.md`). Keep the two in sync when adding attributes here. ## Service identity -The bare minimum every span needs. `service.name` is the primary axis Maple groups by — without it spans go to a synthetic `unknown_service` bucket. +The bare minimum every span needs. `service.name` is the primary axis Maple groups by. Without it, OTel SDKs fall back to `unknown_service:`, which Maple's setup audit flags. | Attribute | Example | What Maple does with it | | --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------- | | `service.name` | `api`, `ingest`, `web` | **Required.** Primary grouping for services list, service map, dashboards, alerts. Filter alias: `service`. | -| `service.version` | `1.4.2`, `c0b92f68` | Per-version slices on service overview. Auto-skipped from chip strip but always queryable. | -| `service.namespace` | `payments` | Logical grouping above `service.name`. Auto-skipped from chip strip. | -| `service.instance.id` | UUID per process | Distinguishes replicas of the same service. Auto-skipped from chip strip. | +| `service.version` | `1.4.2`, `c0b92f68` | Per-version slices on service overview. Skipped from log chips but always queryable. | +| `service.namespace` | `payments` | Logical grouping above `service.name`. Skipped from log chips. | +| `service.instance.id` | UUID per process | Distinguishes replicas of the same service. Skipped from log chips. | ## Deployment & version tracking -Tag every span with these and you get per-environment and per-version slices for free across the services table, service map, and per-service overview. +Tag every span with these to get per-environment and per-version slices across the services table, service map, and per-service overview. | Attribute | Example | What Maple does with it | | ----------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -27,13 +27,13 @@ Tag every span with these and you get per-environment and per-version slices for | `deployment.environment` | `production` | The deprecated spelling of the same attribute. Read everywhere the `.name` key is, so older instrumentation keeps working. | | `vcs.ref.head.revision` | `c0b92f68…` | The commit a service was built from. Release markers, the deploy list and per-version metrics in service overview; exposed as the `commit_sha` discovery facet so you can pivot by deploy. (`deployment.commit_sha`, Maple's former vendor key, is retired and no longer read.) | -OpenTelemetry renamed this attribute (`deployment.environment` → `deployment.environment.name`); Maple reads whichever one your spans carry, preferring `.name` when both are present, so emitting either is fine. Maple's own SDKs dual-emit both. The same holds for `messaging.destination` → `messaging.destination.name` on messaging spans. In the search bar, `env`, `environment`, and `commit_sha` are short aliases — see [Filter aliases](#filter-aliases) below. +OpenTelemetry renamed this attribute (`deployment.environment` → `deployment.environment.name`). Maple reads whichever one your spans carry and prefers `.name` when both are present, so emitting either is fine. Maple's own SDKs dual-emit both. The same holds for `messaging.destination` → `messaging.destination.name` on messaging spans. The shared expressions live in [semconv-renames.ts](../packages/domain/src/tinybird/semconv-renames.ts). In the search bar, `env`, `environment`, and `commit_sha` are short aliases; see [Filter aliases](#filter-aliases) below. ## Span status & kind -Two span-level fields (not attributes) that Maple reads directly. Both spellings are load-bearing — typos silently drop spans out of dashboards. +Two span-level fields (not attributes) that Maple reads directly. Both spellings are load-bearing: typos silently drop spans out of dashboards. -### Status code — **Title Case is required** +### Status code: **Title Case is required** | Value | When to set it | | ------- | --------------------------------------- | @@ -41,9 +41,9 @@ Two span-level fields (not attributes) that Maple reads directly. Both spellings | `Error` | Failed operation | | `Unset` | OTel default; status not explicitly set | -Use the strings `"Ok"`, `"Error"`, `"Unset"` exactly. The error-rate widget filters via `WHERE StatusCode = 'Error'` — uppercase (`ERROR`) or lowercase (`error`) variants silently produce zero rows. Most OTel SDKs serialize the status enum correctly; just don't hand-stamp the wire value. +Use the strings `"Ok"`, `"Error"`, `"Unset"` exactly. The error-rate widget filters on `StatusCode = 'Error'`, so uppercase (`ERROR`) or lowercase (`error`) variants silently produce zero rows. Most OTel SDKs serialize the status enum correctly. Don't hand-stamp the wire value. -The trace UI maps these to colored badges via `statusStyles` in [span-detail-panel.tsx:43-47](../apps/web/src/components/traces/span-detail-panel.tsx). +The trace UI maps these to colored badges via `getSpanStatusBadgeClass` in [span-kind.ts](../packages/ui/src/lib/span-kind.ts). ### Span kind @@ -55,15 +55,15 @@ The trace UI maps these to colored badges via `statusStyles` in [span-detail-pan | `Consumer` | Queue consumers | | `Internal` | In-process work (default; cache lookups, validation, query compilation) | -`Client` spans get a small outgoing-arrow icon in HTTP labels and render their route as `host+path` (so the destination service is visible); `Server` spans render path-only. The [service map](#service-map) only draws an edge for `Client` / `Producer` spans — leaving a network call on `Internal` makes it invisible in the map. +`Client` spans get a small outgoing-arrow icon in HTTP labels and render their route as `host+path`, so the destination is visible. `Server` spans render path-only. The [service map](#service-map) only draws edges from `Client` / `Producer` spans. A network call left on `Internal` is invisible in the map. ## HTTP requests -The most heavily instrumented namespace. Maple extracts three fields into fast materialized-view columns at write time, then renders method, route, and status code in trace rows. +The most heavily instrumented namespace. Maple extracts three fields into columns of `trace_list_mv` (root spans) at write time, then renders method, route, and status code in trace rows. ### Fast columns -Filtering on these is a scan over a small column instead of a per-row map lookup. Both the legacy and post-1.21 OTel semconv names map to the same column — the first non-empty source wins, so you don't need to migrate just for fast filtering. +Filtering on these scans a small column instead of doing a per-row map lookup. The legacy and current OTel semconv names map to the same column, and the first non-empty source wins, so you don't need to migrate just for fast filtering. | Attribute(s) | MV column | | ----------------------------------------------- | ---------------- | @@ -73,7 +73,7 @@ Filtering on these is a scan over a small column instead of a per-row map lookup ### Method color pills -`http.method` / `http.request.method` drives a colored pill on the span row. Colors from `HTTP_METHOD_COLORS` in [http.ts:131-139](../packages/ui/src/lib/http.ts): +`http.method` / `http.request.method` drives a colored pill on the span row. Colors come from `HTTP_METHOD_COLORS` in [http.ts](../packages/ui/src/lib/http.ts): | Method | Color | | ------- | --------- | @@ -87,20 +87,20 @@ Filtering on these is a scan over a small column instead of a per-row map lookup ### Status badge tiers -`http.status_code` / `http.response.status_code` is rendered as a colored badge in the trace list ([traces-table.tsx:57-74](../apps/web/src/components/traces/traces-table.tsx)): +`http.status_code` / `http.response.status_code` renders as a colored badge in the trace list (`HttpStatusBadge` in [traces-table.tsx](../apps/web/src/components/traces/traces-table.tsx)): -| Status range | Tone | -| ------------ | ------------------- | -| 5xx | Error (red) | -| 4xx | Warn (amber) | -| 3xx | Info (chart purple) | -| 1xx–2xx | Info (blue) | +| Status range | Tone | +| ------------ | ----------------------------- | +| 5xx | Error (red) | +| 4xx | Warn (amber) | +| 3xx | Chart color (`chart-p50`) | +| 1xx–2xx | Info (blue) | -In log chips, the same value is also scored 95 (top of the chip strip, just below `exception.*`) — see [Attribute prominence](#appendix-attribute-prominence-scoring). +In log chips the same value scores 95 (top of the chips, just below `exception.*`). See [Attribute prominence](#appendix-attribute-prominence-scoring). ### Route extraction & fallback chain -For full HTTP info ([http.ts:61-129](../packages/ui/src/lib/http.ts)), Maple tries each source in order until one matches: +For full HTTP info (`getHttpInfo` in [http.ts](../packages/ui/src/lib/http.ts)), Maple tries each source in order until one matches: **Method:** `http.method` → `http.request.method` → span name (e.g. `http.server GET /path`, or bare `GET /path`). @@ -110,51 +110,54 @@ For full HTTP info ([http.ts:61-129](../packages/ui/src/lib/http.ts)), Maple tri **Status:** `http.status_code` → `http.response.status_code`. -So `url.full` (e.g. `https://api.tinybird.co/v0/sql`) on a `Client` span lights up route rendering automatically — but emitting `http.route` is preferred because it's a semantic path (`/api/users/:id`) instead of a high-cardinality URL. +So `url.full` (e.g. `https://api.tinybird.co/v0/sql`) on a `Client` span lights up route rendering automatically. Prefer `http.route`: it is a semantic path (`/api/users/:id`) instead of a high-cardinality URL. ## Service map -The map renders nodes for services and edges for the calls between them. Three rules to make sure your spans show up correctly. +The map renders nodes for services and edges for the calls between them. Three layers feed it; see [service-map-architecture.md](service-map-architecture.md) for the data path. ### 1. Service-to-service edges -Emit `peer.service` on every `Client` or `Producer` span. The [`service_map_edges_hourly_mv`](../packages/domain/src/tinybird/materializations.ts) materialized view groups on `(SourceService, TargetService, DeploymentEnv)` per hour where `peer.service` is non-empty. +An edge is drawn when a `Client` or `Producer` span has a child `Server` or `Consumer` span in a different service. Maple joins the two on `TraceId` and `ParentSpanId`, and the target is the child span's `service.name`. The scheduled `ServiceMapRollupService` rolls these joins into `service_map_edges_hourly`. ``` -GET /v1/users (service.name=api, span.kind=Client, peer.service=users-service) +GET /v1/users (service.name=api, span.kind=Client) + └─ child: GET /v1/users (service.name=users-service, span.kind=Server) └──> draws an edge api → users-service ``` -If you put `peer.service` on a `Server` span, or leave it off entirely, no edge is drawn — the call won't appear in the map. +Two things must hold. The caller must propagate trace context (`traceparent`) so the callee's span is a child of the client span. The callee must be instrumented and export to the same org. A network call left on `Internal`, or a callee that starts a new trace, draws no edge. `peer.service` is not read for these edges, because modern instrumentation no longer emits it. ### 2. Database nodes -Pair `db.system` with `peer.service` on the same `Client` span — both are required. The map aggregates by `db.system`, so multiple services calling the same database land on a single shared node ([service-map-utils.ts:184-203](../apps/web/src/components/service-map/service-map-utils.ts)). +Emit `db.system.name` (or the legacy `db.system`) on the `Client` span. The `service_map_db_edges_hourly_mv` materialized view ([materializations.ts](../packages/domain/src/tinybird/materializations.ts)) groups those spans by system and namespace. The namespace resolves `db.namespace` → `db.name` → `server.address` → `net.peer.name`, so two databases of the same system become two nodes. Services calling the same database share one node. ``` -SELECT * FROM users (span.kind=Client, db.system=postgresql, peer.service=postgresql) - └──> draws an edge api → postgresql DB node +SELECT * FROM users (span.kind=Client, db.system.name=postgresql, db.namespace=app) + └──> draws an edge api → postgresql (app) DB node ``` -### 3. Pick canonical names +### 3. External targets -Keep `peer.service` spelling consistent across services so edges don't fragment. If one service emits `peer.service=tinybird` and another emits `peer.service=Tinybird` or `peer.service=tb`, they become separate nodes. Pick one canonical name per peer and stick with it. +`Client` and `Producer` spans without `db.system.name` land in `service_external_edges_hourly` and show on the service's Dependencies tab. The target is the messaging destination (or `messaging.system`), then `rpc.service` (or `rpc.system`), then `server.address` → `http.host` → `url.authority`. + +Keep `service.name` spelling consistent across deployments of the same service. `users`, `Users` and `users-svc` become three separate nodes. ## Database queries -In addition to powering the service map, these drive the chip strip and the AI error-debug prompt context. +Besides the service map, these drive the log chips and the AI error-debug prompt context. The current semconv name and its legacy spelling score the same. -| Attribute | What Maple does with it | -| -------------- | ----------------------------------------------------------------------------------------------------------- | -| `db.system` | Scored 70 in the chip strip; pairs with `peer.service` for service map DB nodes; toned `info` in log chips. | -| `db.statement` | Scored 70; rendered in span detail; included as context in the error-debug prompt. | -| `db.operation` | Scored 70 (e.g. `"SELECT"`, `"INSERT"`). | +| Attribute | What Maple does with it | +| ------------------------------------ | ----------------------------------------------------------------------------------------------------- | +| `db.system.name`, `db.system` | Scored 70 in log chips; toned `info`; database identity for service map DB nodes. | +| `db.query.text`, `db.statement` | Scored 70; rendered in span detail; included as context in the error-debug prompt. | +| `db.operation.name`, `db.operation` | Scored 70 (e.g. `"SELECT"`, `"INSERT"`). | -Source: [log-attributes.ts:37](../apps/web/src/lib/log-attributes.ts). +Source: `scoreKey` in [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). ## Caching -Maple detects a cache span when **any** `cache.*` attribute is present. When detected, the trace UI renders a hit/miss badge and an operation pill (GET / SET / DELETE) on the span row. +Maple treats a span as a cache span when it carries `cache.system` or `cache.result`. The trace UI then renders a hit/miss badge and an operation pill (GET / SET / DELETE) on the span row. | Attribute | Example | What Maple does with it | | ------------------------ | ---------------------- | ----------------------------------------------------------------------------- | @@ -164,11 +167,11 @@ Maple detects a cache span when **any** `cache.*` attribute is present. When det | `cache.operation` | `GET`, `SET`, `DELETE` | Drives the operation pill color. | | `cache.lookup_performed` | `true` \| `false` | Whether a lookup was actually executed (string, not bool). | -Source: [cache.ts](../apps/web/src/lib/cache.ts). +Source: `getCacheInfo` in [cache.ts](../packages/ui/src/lib/cache.ts). ## Errors & exceptions -Drives the destructive-tone error banner shown on log rows ([log-error-banner.tsx:12-22](../apps/web/src/components/logs/log-error-banner.tsx)) and the highest-priority chip on every row. +Drives the error banner in the log detail view ([log-error-banner.tsx](../apps/web/src/components/logs/log-error-banner.tsx)) and the highest-priority log chip. | Attribute | What Maple does with it | | ------------------- | --------------------------------------------------------------------------------------------------------------- | @@ -177,50 +180,50 @@ Drives the destructive-tone error banner shown on log rows ([log-error-banner.ts | `error.message` | Same as `exception.message` (legacy fallback). | | `error.type` | Same as `exception.type` (legacy fallback). Also used by Maple-internal services to categorize failure reasons. | -Any attribute matching `exception.*` is scored 100 (top of the chip strip) and auto-toned `error` (red) in log chips — see [log-attributes.ts:33,60](../apps/web/src/lib/log-attributes.ts). +Any attribute matching `exception.*` scores 100 (top of the chips) and is toned `error` (red) in log chips. See `scoreKey` and `getChipTone` in [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). -If `exception.message` is longer than 120 characters or contains a newline, the banner collapses by default and shows a "Show full error" toggle. +If the message runs past 3 lines or 160 characters, the banner collapses by default and shows a "Show more" toggle. ## RPC For gRPC and other RPC frameworks. -| Attribute | What Maple does with it | -| ---------------------- | ----------------------------------------------------------------------------- | -| `rpc.service` | Scored 68 in chip strip; toned `info`. | -| `rpc.method` | Scored 68; toned `info`. | -| `rpc.grpc.status_code` | Scored 90 (just below HTTP status). Non-zero values auto-toned `error` (red). | +| Attribute | What Maple does with it | +| ---------------------- | ------------------------------------------------------------------------ | +| `rpc.service` | Scored 68 in log chips; toned `info`. | +| `rpc.method` | Scored 68; toned `info`. | +| `rpc.grpc.status_code` | Scored 90 (just below HTTP status). Non-zero values are toned `error`. | -Source: [log-attributes.ts:35,38,77-83](../apps/web/src/lib/log-attributes.ts). +Source: [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). ## User identity -Promotes user/customer context to the chip strip so it's visible at a glance on every log row. +Promotes user and customer context into the log chips so it's visible at a glance on every log row. -| Attribute | What Maple does with it | -| ----------------------------------------------------- | -------------------------------------------------- | -| `user.id`, `enduser.id`, `customer.id`, `customer_id` | All scored 66 in chip strip (equivalent priority). | +| Attribute | What Maple does with it | +| ----------------------------------------------------- | ------------------------------------------------ | +| `user.id`, `enduser.id`, `customer.id`, `customer_id` | All scored 66 in log chips (equal priority). | -Source: [log-attributes.ts:39](../apps/web/src/lib/log-attributes.ts). +Source: [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). ## Log severity -For log records (not spans). Drives the per-row text color in the log list and the trace detail timeline. +For log records (not spans). Drives the per-row color in the log list and the trace detail timeline. | `log.severityText` value | Color theme | | ------------------------ | -------------- | | `TRACE` | severity-trace | | `DEBUG` | severity-debug | | `INFO` | severity-info | -| `WARN` | severity-warn | +| `WARN`, `WARNING` | severity-warn | | `ERROR` | severity-error | | `FATAL` | severity-fatal | -`ERROR` and `FATAL` severities additionally cause the error banner to render at the top of the log detail panel. Source: [span-detail-panel.tsx:57-64](../apps/web/src/components/traces/span-detail-panel.tsx). +Matching is case-insensitive. Source: `SEVERITY_COLORS` in [severity.ts](../packages/ui/src/lib/severity.ts). `ERROR` and `FATAL` also render the error banner at the top of the log detail view ([log-detail-sheet.tsx](../apps/web/src/components/logs/log-detail-sheet.tsx)). ## Kubernetes & infrastructure -Tag spans with `k8s.*` and they light up the service map's pod-count badges and Infrastructure tab. The `maple-k8s-infra` Helm chart sets most of these for you via the OTel operator + `k8sattributes` processor — see [service-map-infrastructure.md](service-map-infrastructure.md) for the full enrichment lifecycle. +Tag spans with `k8s.*` to light up the service map's pod-count badges and the Infrastructure tab. The `maple-k8s-infra` Helm chart ([deploy/k8s-infra](../deploy/k8s-infra/README.md)) sets most of these through the OTel operator and the `k8sattributes` processor. See [service-map-infrastructure.md](service-map-infrastructure.md) for the full enrichment lifecycle. ### Workload identity (joins to `service.name`) @@ -233,7 +236,7 @@ Tag spans with `k8s.*` and they light up the service map's pod-count badges and ### Promoted to log chips -Always shown in the chip strip when present. +Resource attributes that appear in the log chips when present. | Attribute | What Maple does with it | | -------------------- | ----------------------------------------- | @@ -254,27 +257,25 @@ Always shown in the chip strip when present. ### Containers (Docker) -The Maple Docker agent's `docker_stats` receiver stamps these on container metrics; `/infra/containers` -groups on `(container.name, host.name)` — Docker names are only unique per host. Rows carrying a -`k8s.pod.name` are treated as Kubernetes, not Docker. +The Maple Docker agent's `docker_stats` receiver ([deploy/docker-agent](../deploy/docker-agent/README.md)) stamps these on container metrics. `/infra/containers` groups on `(container.name, host.name)`, because Docker names are only unique per host. Rows carrying a `k8s.pod.name` are treated as Kubernetes, not Docker. | Attribute | What Maple does with it | | ---------------------- | --------------------------------------------------------------------------------------- | | `container.name` | Container identity: list/detail views and the span/log Infrastructure tab detect key. | | `container.id` | Facet cardinality (recreated containers keep their name, not their id) and correlation. | -| `container.image.name` | Image facet + display chip. | +| `container.image.name` | Image facet and display chip. | | `compose.project` | From the `com.docker.compose.project` label, promoted to resource by the agent; facet. | | `compose.service` | From the `com.docker.compose.service` label, promoted to resource by the agent; facet. | To correlate **app telemetry** with containers, the app's spans must also carry `container.id` -(or at least `container.name`). `@maple/effect-sdk` auto-detects Docker identity best-effort -(mountinfo/cgroup/hostname); the reliable path is explicit: +(or at least `container.name`). `@maple-dev/effect-sdk` detects Docker identity best-effort +(mountinfo/cgroup/hostname). The reliable path is explicit: `OTEL_RESOURCE_ATTRIBUTES=container.id=$(hostname),container.name=myservice` in the compose file (Docker's default hostname is the short container id). ## Cloud & platform badges -These set the platform badge and runtime icon next to a service on the service map. SDKs running on common platforms auto-detect most of them — document the keys so self-instrumenters can match. +These set the platform badge and runtime icon next to a service on the service map. SDKs on common platforms auto-detect most of them. The keys are listed here so self-instrumenters can match. | Attribute | Example values | What Maple does with it | | ---------------------- | -------------------------------------------------------- | ----------------------------------------------------- | @@ -286,34 +287,34 @@ These set the platform badge and runtime icon next to a service on the service m | `faas.version` | Function version / revision | Per-version slicing on FaaS. | | `faas.instance` | Function execution / instance ID | Replica identifier on FaaS. | -Keep `process.runtime.name` values consistent across services running the same runtime — don't have one service emit `nodejs` and another `node`, or you'll get two runtime icons for the same fleet. +Keep `process.runtime.name` values consistent across services on the same runtime. If one service emits `nodejs` and another `node`, you get two runtime icons for the same fleet. ## Sampling-aware throughput -Maple reads the W3C `tracestate: ot=th:` header to extrapolate throughput counts when you're sampling traces. See [sampling-throughput.md](sampling-throughput.md) for the math and SDK setup. You don't emit anything as an attribute — this is a header on the trace context. +Maple reads the W3C `tracestate: ot=th:` header to extrapolate throughput counts when you sample traces. See [sampling-throughput.md](sampling-throughput.md) for the math and SDK setup. There is no attribute to emit: this is a header on the trace context. ## Filter aliases -In Maple's WHERE-clause search bar (trace list, log search, dashboard widgets), you can type a short alias and it resolves to the canonical attribute. Source of truth: `normalizeKey` in [packages/domain/src/where-clause.ts](../packages/domain/src/where-clause.ts). +In Maple's WHERE-clause search bar (trace list, log search, dashboard widgets), a short alias resolves to the canonical attribute. Source of truth: `normalizeKey` in [packages/domain/src/where-clause.ts](../packages/domain/src/where-clause.ts). -| Alias | Resolves to | -| ------------------------------------- | -------------------------------------------------- | -| `service` | `service.name` | -| `span` | `span.name` | -| `environment`, `env` | `deployment.environment` | -| `commit_sha`, `deployment.commit_sha` | `vcs.ref.head.revision` | -| `root.only` | `root_only` (synthetic boolean — root spans only) | -| `errors_only` | `has_error` (synthetic boolean — error spans only) | +| Alias | Resolves to | +| ------------------------------------- | ------------------------------------------------- | +| `service` | `service.name` | +| `span` | `span.name` | +| `environment`, `env` | `deployment.environment` | +| `commit_sha`, `deployment.commit_sha` | `vcs.ref.head.revision` | +| `root.only` | `root_only` (synthetic boolean: root spans only) | +| `errors_only` | `has_error` (synthetic boolean: error spans only) | -So `env = "production"` and `deployment.environment = "production"` mean the same thing; pick whichever is shorter. +So `env = "production"` and `deployment.environment = "production"` mean the same thing. ## Reserved namespace -`maple_*` is reserved for Maple platform internals (org routing, ingest auth keys). Do not use this prefix for your own attributes — the UI hides anything starting with `maple_` from log and span attribute chips. +`maple_*` is reserved for Maple platform internals (org routing, ingest auth keys). Do not use this prefix for your own attributes. The UI hides anything starting with `maple_` from the log attribute chips. -## Attributes Maple hides in the UI chip strip +## Attributes Maple hides from log chips -These are stored on the row but skipped from the log/span attribute chip strip because they're noisy or already shown elsewhere (service column, etc.). Source: [apps/web/src/lib/log-attributes.ts](../apps/web/src/lib/log-attributes.ts). +These are stored on the row but skipped from the log attribute chips because they're noisy or already shown elsewhere (service column, etc.). Source: `SKIP_KEYS` / `SKIP_PREFIXES` in [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). - `service.name`, `service.namespace`, `service.instance.id`, `service.version` - `telemetry.sdk.*` @@ -322,35 +323,36 @@ These are stored on the row but skipped from the log/span attribute chip strip b - `host.arch`, `host.name` - `maple_*` -The data is still queryable — you can filter or group by these in the search bar — they're just not auto-promoted into the row's attribute chips. +The data is still queryable. You can filter or group by these in the search bar; they're just not promoted into the row's chips. ## Appendix: Attribute prominence scoring -The chip strip on each log/span row shows the top 4 attributes by score. Higher score = more prominent. Source: `scoreKey` in [log-attributes.ts:32-49](../apps/web/src/lib/log-attributes.ts). - -| Score | Attributes | -| ----- | ----------------------------------------------------- | -| 100 | `error`, `exception`, `exception.*` (anything) | -| 95 | `http.status_code`, `http.response.status_code` | -| 90 | `rpc.grpc.status_code` | -| 80 | `http.method`, `http.request.method` | -| 70 | `db.system`, `db.statement`, `db.operation` | -| 68 | `rpc.service`, `rpc.method` | -| 66 | `user.id`, `enduser.id`, `customer.id`, `customer_id` | -| 60 | `duration_ms`, `latency_ms`, `http.duration` | -| 55 | `http.url`, `http.route`, `url.path` | -| 40 | Other `http.*`, `url.*` | -| 38 | Other `db.*` | -| 36 | Other `rpc.*` | -| 34 | `messaging.*` | -| 32 | Other `user.*`, `enduser.*` | -| 25 | Anything with a dot (`namespace.key`) | -| 20 | Bare keys (no namespace) | - -Resource attributes only appear in chips if they're in the [`PROMOTED_RESOURCE_KEYS` set](../apps/web/src/lib/log-attributes.ts) (deployment env, k8s pod/namespace, cloud region); other resource attrs get their score decremented by 10 even if promoted. +Each log row shows its non-pinned attributes as chips, sorted by score (ties break alphabetically). Higher score = more prominent. Source: `scoreKey` in [log-attributes.ts](../packages/ui/src/lib/log-attributes.ts). + +| Score | Attributes | +| ----- | -------------------------------------------------------------------------------- | +| 100 | `error`, `exception`, `exception.*` (anything) | +| 95 | `http.status_code`, `http.response.status_code` | +| 90 | `rpc.grpc.status_code` | +| 80 | `http.method`, `http.request.method` | +| 70 | `db.system(.name)`, `db.statement`, `db.query.text`, `db.operation(.name)` | +| 68 | `rpc.service`, `rpc.method` | +| 66 | `user.id`, `enduser.id`, `customer.id`, `customer_id` | +| 60 | `duration_ms`, `latency_ms`, `http.duration` | +| 55 | `http.url`, `http.route`, `url.path` | +| 40 | Other `http.*`, `url.*` | +| 38 | Other `db.*` | +| 36 | Other `rpc.*` | +| 34 | `messaging.*` | +| 32 | Other `user.*`, `enduser.*` | +| 25 | Anything with a dot (`namespace.key`) | +| 20 | Bare keys (no namespace) | + +Resource attributes appear in chips only if they're in the `PROMOTED_RESOURCE_KEYS` set (deployment env in both spellings, k8s pod/namespace, cloud region), and they score 10 lower than the same key as a log attribute. ## See also -- [sampling-throughput.md](sampling-throughput.md) — sampling-aware throughput math and `tracestate: ot=th:` reading. -- [service-map-infrastructure.md](service-map-infrastructure.md) — full lifecycle for `k8s.*` enrichment. -- [self-hosted-clickhouse.md](self-hosted-clickhouse.md) — schema and column layout for self-hosters. +- [sampling-throughput.md](sampling-throughput.md): sampling-aware throughput math and `tracestate: ot=th:` reading. +- [service-map-architecture.md](service-map-architecture.md): the service map's data path. +- [service-map-infrastructure.md](service-map-infrastructure.md): full lifecycle for `k8s.*` enrichment. +- [self-hosted-clickhouse.md](self-hosted-clickhouse.md): schema and column layout for self-hosters. diff --git a/docs/backup-and-recovery.md b/docs/backup-and-recovery.md index 5924c3eef7..6294960072 100644 --- a/docs/backup-and-recovery.md +++ b/docs/backup-and-recovery.md @@ -1,8 +1,8 @@ # Backup and recovery What Maple backs up, how, and how recovery is proven. This is the operator guide behind the -"backups and tested restoration" control (SOC 2 A1.2/A1.3, ISO 27001 A.8.13/A.8.14): the system -inventory, the recovery mechanism for each, and the evidence each one produces. +"backups and tested restoration" control (SOC 2 A1.2/A1.3, ISO 27001 A.8.13/A.8.14). It lists +each system, its recovery mechanism, and the evidence it produces. ## Inventory @@ -15,8 +15,8 @@ inventory, the recovery mechanism for each, and the evidence each one produces. | **Clerk** | Users, organisations, memberships | Vendor-managed; exportable through the Clerk API | Vendor report | | **Infisical** | Secrets | Vendor-managed | Vendor report | -The application database is the only system whose loss is not either vendor-durable by contract -or regenerable, so it is the one with a first-party restore drill. +The application database is the only system that is neither vendor-durable by contract nor +regenerable, so it is the one with a first-party restore drill. ## The restore drill @@ -29,71 +29,75 @@ PLANETSCALE_ORG= bun run backup:restore-test It records, in order: -1. **Backup configuration as observed** — the source branch (cluster size, region, replica count) - and the recent backups with state, size, completion and expiry, from which the actual cadence - and retention are derived. This is the "backup settings" half of the evidence. -2. **The restore** — `pscale branch create restore-test- --restore - --cluster-size PS_DEV`, then waits for the branch to be ready. The source branch is only ever read (`branch show`, - `backup list`); the restore lands in a new branch on its own cluster. +1. **Backup configuration as observed.** The source branch (cluster size, region, replica count) + and the recent backups with state, size, completion and expiry. The actual cadence and + retention are derived from these. This is the "backup settings" half of the evidence. +2. **The restore.** `pscale branch create restore-test- --restore + --cluster-size PS_DEV --wait`. The source branch is only ever read (`branch show`, + `backup list`). The restore lands in a new branch on its own cluster. 3. **Verification** on the restored branch, each a pass/fail row in the report: - - the Drizzle migrations journal is present; + - the Drizzle migrations table (`drizzle.__drizzle_migrations`) is present; - every critical table exists (`org_ingest_keys`, `api_keys`, dashboards and their versions, alert rules/destinations/incidents, error issues and events, integrations), and the ones that are never empty in production hold rows; - - the newest record falls inside the backup's own window — no later than the backup completed, - no older than a day before it started — which is what distinguishes "the backup's snapshot" - from "some data"; + - the newest record falls inside the backup's own window: no later than the backup completed, + no older than a day before it started. This distinguishes "the backup's snapshot" from + "some data"; - a parent/child join (`dashboard_versions → dashboards`) has no orphans; - one record reads back with its full history. - A query that errors is a failed row, not a crash: the report is always written. -4. **Cleanup** — the only destructive call is `branch delete`, and it passes through a gate that - refuses any name not starting with `restore-test-` or equal to the source branch. The report - is written before the delete is requested, and a process `exit` hook re-requests the delete - if anything exits early, because a leaked restore branch bills until someone notices. - `RESTORE_TEST_KEEP_BRANCH=1` keeps it for inspection. - -The report lands in `packages/db/.restore-test/restore-test-report.{json,md}` (gitignored) and, in -CI, in the job summary and the `backup-restore-test-report` artifact. The repository is public and -so are its run logs and artifacts, so the report carries no org slug, row counts as orders of -magnitude rather than exact figures, and no record identifiers. A non-zero exit means a check -failed: a scheduled red run is a recovery incident, not noise. + + A query that errors is a failed row, not a crash. The report is always written. +4. **Cleanup.** The only destructive call is `branch delete`. It passes through a gate + (`assertThrowaway`) that refuses any name without the `restore-test-` prefix, the source + branch, and `main`. The report is written before the delete is requested. A process `exit` + hook re-requests the delete if anything exits early, because a leaked restore branch bills + until someone notices. `RESTORE_TEST_KEEP_BRANCH=1` keeps it for inspection. + +Locally the report lands in `packages/db/.restore-test/restore-test-report.{json,md}` +(gitignored; `RESTORE_TEST_REPORT_DIR` overrides it). In CI it goes to the job summary and the +`backup-restore-test-report` artifact. The repository is public, and so are its run logs and +artifacts. The report therefore carries no org slug and no record identifiers, and gives row +counts as orders of magnitude. A non-zero exit means a check failed. Treat a red scheduled run as +a recovery incident. ### Testing the drill -- **Locally, for real:** `PLANETSCALE_ORG= bun run backup:restore-test` with a `pscale auth -login` session. Takes 10–20 minutes, most of it PlanetScale provisioning the restored cluster; - the branch is billed for that window (restores default to the PS-10 size). Inspect +- **Locally, for real:** `PLANETSCALE_ORG= bun run backup:restore-test` with a + `pscale auth login` session. It takes 10 to 20 minutes, most of it PlanetScale provisioning the + restored cluster. The PS_DEV branch is billed for that window. Inspect `packages/db/.restore-test/`. This is the run that produces the first restore evidence. - **Keep the branch:** add `RESTORE_TEST_KEEP_BRANCH=1` to poke at the restored data by hand, then `pscale branch delete restore-test- --force`. - **Failure path without a restore:** `RESTORE_TEST_SOURCE_BRANCH=does-not-exist` fails at `branch show`, before anything is created. -- **In CI:** the workflow only runs on its schedule or `gh workflow run backup-restore-test.yml` - (after merge — a PR's CI does not run it). The first dispatch validates the service token. On - the database it needs `read_branch`, `create_branch`, `delete_branch`, `connect_branch`, - `delete_branch_password`, `read_backups` and `restore_production_branch_backup` — the - development-branch accesses only, never `delete_production_branch`. A token missing - `read_backups` fails at `backup list`, one missing the restore access fails at `branch -create`, both before anything is created; one missing the delete accesses restores fine and - then leaves the branch behind, which the drill reports as a failed check. Only the token's - creator or an organization administrator can change its accesses. +- **In CI:** the workflow runs only on its schedule or via + `gh workflow run backup-restore-test.yml` after merge. A PR's CI does not run it. The first + dispatch validates the service token. On the database it needs `read_branch`, + `create_branch`, `delete_branch`, `connect_branch`, `delete_branch_password`, `read_backups` + and `restore_production_branch_backup`. Beyond the restore access these are + development-branch accesses; it never needs `delete_production_branch`. A token missing + `read_backups` fails at `backup list`. One missing the restore access fails at + `branch create`. Both fail before anything is created. One missing the delete accesses + restores fine and then leaves the branch behind, which the drill reports as a failed check. + Only the token's creator or an organization administrator can change its accesses. - **Why the restore is `--cluster-size PS_DEV`:** a restore without an explicit size becomes a - PS-10 branch flagged `production: true`, which only `delete_production_branch` can remove — - and PlanetScale cannot demote a Postgres restore branch afterwards. A PS_DEV restore is a + PS-10 branch flagged `production: true`, which only `delete_production_branch` can remove. + PlanetScale cannot demote a Postgres restore branch afterwards. A PS_DEV restore is a development branch, so the CI token never needs an access that could reach `main`. ### Schedule -[`backup-restore-test.yml`](../.github/workflows/backup-restore-test.yml) runs the drill on the -first day of every quarter and on `workflow_dispatch`. It authenticates the `pscale` CLI with the -PlanetScale service token from Infisical (`PLANETSCALE_SERVICE_TOKEN_ID` / `_TOKEN` / -`PLANETSCALE_ORG`) — the same one the preview-orphan sweep uses, which already has branch create +[`backup-restore-test.yml`](../.github/workflows/backup-restore-test.yml) runs the drill at 06:23 +UTC on the first day of every quarter and on `workflow_dispatch`. It authenticates the `pscale` +CLI with the PlanetScale service token from Infisical (`PLANETSCALE_SERVICE_TOKEN_ID` / `_TOKEN` / +`PLANETSCALE_ORG`). The preview-orphan sweep uses the same token, which already has branch create and delete on the `maple` database. -Point-in-time recovery is the other restore path PlanetScale offers (any timestamp inside the -retention window, dashboard or `pscale branch create --from main --restore-point ` on current -CLI releases). The drill restores a named backup because that is what proves the _backups_ are -recoverable; PITR uses the same base backups plus WAL, so it is covered by the same evidence. +Point-in-time recovery is the other restore path PlanetScale offers: any timestamp inside the +retention window, from the dashboard or `pscale branch create --from main --restore-point ` +on current CLI releases. The drill restores a named backup because that is what proves the +_backups_ are recoverable. PITR uses the same base backups plus WAL, so the same evidence covers +it. ## Compliance-platform automations @@ -114,11 +118,11 @@ without anyone uploading files. - Download that run's `backup-restore-test-report` artifact and return `restore-test-report.json`. - Pass when the run's `updated_at` is inside the last 365 days and the report's `status` is - `pass`. Evidence: the report — backup restored, branch, every verification row — plus the run + `pass`. Evidence: the report (backup restored, branch, every verification row) plus the run URL. -Storing the report in the compliance platform on each run is what gives it retention beyond -GitHub's 90-day artifact cap. +Storing the report in the compliance platform on each run gives it retention beyond GitHub's +90-day artifact cap. For a dashboard screenshot as a supplement, the PlanetScale **Backups** page of `maple/main` (schedule, retention, recent backups) is the one to capture. diff --git a/docs/browser-sdk.md b/docs/browser-sdk.md index 1847d8e8ea..da73b7345d 100644 --- a/docs/browser-sdk.md +++ b/docs/browser-sdk.md @@ -1,6 +1,6 @@ # Browser SDK -`@maple-dev/browser` instruments a website with OpenTelemetry tracing **and** rrweb session replay in a single package. Every span and every replay event is tagged with the same `session.id`, so a trace can link straight to the replay that produced it — and vice versa — with no clock-skew guessing. +`@maple-dev/browser` instruments a website with OpenTelemetry tracing **and** rrweb session replay in one package. Every span and every replay event carries the same `session.id`, so a trace links straight to the replay that produced it (and the reverse) with no clock-skew guessing. > Session Replay is currently in **Beta**. @@ -23,43 +23,49 @@ MapleBrowser.init({ }) ``` -That single call: +That call: -- starts OTel browser tracing, auto-instrumenting `fetch`, exporting to Maple's ingest (`POST /v1/traces`); -- records the session with rrweb, chunking events (~5s / 100KB windows), gzipping them with the native `CompressionStream`, and uploading to `POST /v1/sessionReplays/blob`; -- writes session metadata at start (`active`) and on page hide (`ended`), including the trace ids observed during the session. +- starts OTel browser tracing, auto-instrumenting `fetch` and exporting to Maple's ingest (`POST /v1/traces`); +- captures uncaught errors and unhandled promise rejections as error spans (see [Errors](#errors)); +- records the session with rrweb, chunks events into ~5s / 100KB windows, gzips them with the native `CompressionStream`, and uploads them to `POST /v1/sessionReplays/blob`; +- writes session metadata to `POST /v1/sessionReplays/meta`: an `active` row at start, a heartbeat every 60s, and an `ended` row on page hide, which includes the trace ids observed during the session. The SDK is **best-effort**: network failures in telemetry never throw into your app. -`init()` returns a handle — `{ sessionId, shutdown }` — for reading the active session id and tearing telemetry down. See [Sessions](#sessions). +`init()` returns a handle, `{ sessionId, shutdown }`, for reading the active session id and tearing telemetry down. See [Sessions](#sessions). ## Configuration Every field accepted by `MapleBrowser.init`: -| Option | Type | Default | Description | -| ------------------------------ | --------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `ingestKey` | `string` | — | Public ingest key (`maple_pk_...`), used only as the `Authorization` header. Omit it behind a proxy that adds auth; see [Auth via a proxy](#auth-via-a-proxy). | -| `serviceName` | `string` | — | **Required.** Service name reported on traces and stored on replay sessions. | -| `endpoint` | `string` | `https://ingest.maple.dev` | Maple ingest base URL. Override for self-hosted / regional ingest. | -| `serviceNamespace` | `string` | — | Logical group this service belongs to, emitted as the OTel `service.namespace` resource attribute on traces. | -| `serviceVersion` | `string` | — | Service version or commit SHA, attached to traces. | -| `environment` | `string` | — | Deployment environment, e.g. `"production"`. | -| `userId` | `string` | — | User id attached to the replay session for correlation. See [Identifying users](#identifying-users). | -| `tracing.enabled` | `boolean` | `true` | Enable OTel browser tracing. | -| `tracing.instrumentFetch` | `boolean` | `true` | Auto-instrument `fetch()` to create network spans. Set `false` when another tracer (e.g. the Effect client SDK) already instruments requests — those spans feed the session via the published sink, and disabling this avoids duplicate network spans. | -| `replay.enabled` | `boolean` | `true` | Enable rrweb session recording. | -| `replay.sampleRate` | `number` | `1` | Fraction of sessions to record, `0`–`1`. See [Sampling](#sampling). | -| `privacy.maskAllInputs` | `boolean` | `true` | Mask all `` values in the recording. | -| `privacy.maskAllText` | `boolean` | `false` | Mask all text in the rrweb recording and omit captured click-target text from session events. | -| `privacy.persistVisitorId` | `boolean` | `true` | Store a persistent visitor id (localStorage + cookie) so unique visitors and new-vs-returning are measurable. Turning it off also purges any id already stored. | -| `privacy.crossSubdomainCookie` | `boolean` | `true` | Scope the visitor-id cookie to the registered domain so sibling subdomains share it. See [Linking a marketing site to your app](#linking-a-marketing-site-to-your-app). | -| `privacy.cookieDomain` | `string` | probed | Explicit cookie `Domain=` (no leading dot). `""` forces a host-only cookie. | -| `privacy.requireConsent` | `boolean` | `false` | Capture nothing until `MapleBrowser.setConsent(true)`. See [Consent](#consent). | -| `privacy.captureUserEmail` | `boolean` | `true` | Send `identify()`'s email through to the warehouse. | -| `privacy.respectDoNotTrack` | `boolean` | `false` | Treat `navigator.doNotTrack` like Global Privacy Control (suppresses the persistent visitor id). | - -A fully-specified call: +| Option | Type | Default | Description | +| -------------------------------------- | -------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ingestKey` | `string` | none | Public ingest key (`maple_pk_...`), used only as the `Authorization` header. Omit it behind a proxy that adds auth; see [Auth via a proxy](#auth-via-a-proxy). | +| `serviceName` | `string` | none | **Required.** Service name reported on traces and stored on replay sessions. | +| `region` | `"us" \| "eu"` | `"us"` | Region your Maple organization lives in. Ignored when `endpoint` is set. See [Regions](#regions). | +| `endpoint` | `string` | `https://ingest.maple.dev` | Maple ingest base URL. Overrides `region`. Use it for a proxy or self-hosted ingest. | +| `serviceNamespace` | `string` | none | Logical group this service belongs to, emitted as the OTel `service.namespace` resource attribute on traces. | +| `serviceVersion` | `string` | none | Service version or commit SHA, attached to traces. | +| `environment` | `string` | none | Deployment environment, e.g. `"production"`. | +| `user` | `MapleIdentity` | none | End-user identity attached to sessions and browser spans. Same shape as the `identify()` object. See [Identifying users](#identifying-users). | +| `userId` | `string` | none | **Deprecated**, use `user`. User id attached to the replay session and future browser spans. | +| `tracing.enabled` | `boolean` | `true` | Enable OTel browser tracing. | +| `tracing.instrumentFetch` | `boolean` | `true` | Auto-instrument `fetch()` to create network spans. Set `false` when another tracer (e.g. the Effect client SDK) already instruments requests. Its spans feed the session through the published sink, and turning this off avoids duplicate network spans. | +| `tracing.captureErrors` | `boolean` | `true` | Record uncaught errors and unhandled rejections as error spans. See [Errors](#errors). | +| `tracing.propagateTraceHeaderCorsUrls` | `Array` | `[]` | Cross-origin URLs whose `fetch()` requests carry the `traceparent` header. See [Tracing across origins](#tracing-across-origins). | +| `replay.enabled` | `boolean` | `true` | Enable rrweb session recording. | +| `replay.sampleRate` | `number` | `1` | Fraction of sessions to record, `0` to `1`. Out-of-range values are clamped with a warning. See [Sampling](#sampling). | +| `privacy.maskAllInputs` | `boolean` | `true` | Mask all `` values in the recording. | +| `privacy.maskAllText` | `boolean` | `false` | Mask all text in the rrweb recording and omit captured click-target text from session events. | +| `privacy.persistVisitorId` | `boolean` | `true` | Store a persistent visitor id (localStorage + cookie) so unique visitors and new-vs-returning are measurable. Turning it off also purges any id already stored. | +| `privacy.crossSubdomainCookie` | `boolean` | `true` | Scope the visitor-id cookie to the registered domain so sibling subdomains share it. See [Linking a marketing site to your app](#linking-a-marketing-site-to-your-app). | +| `privacy.cookieDomain` | `string` | probed | Explicit cookie `Domain=` (no leading dot). `""` forces a host-only cookie. | +| `privacy.requireConsent` | `boolean` | `false` | Capture nothing until `MapleBrowser.setConsent(true)`. See [Consent](#consent). | +| `privacy.captureUserEmail` | `boolean` | `true` | Send `identify()`'s email through to the warehouse. | +| `privacy.respectDoNotTrack` | `boolean` | `false` | Treat `navigator.doNotTrack` like Global Privacy Control (suppresses the persistent visitor id). | +| `privacy.sanitizeUrl` | `(url: string) => string` | none | Rewrite every URL before it leaves the page. Runs after the built-in redaction. See [Privacy & masking](#privacy--masking). | + +A fully specified call: ```ts MapleBrowser.init({ @@ -67,7 +73,7 @@ MapleBrowser.init({ serviceName: "acme-web", environment: "production", serviceVersion: "1.4.2", - userId: currentUser?.id, + user: { id: currentUser.id, email: currentUser.email }, tracing: { enabled: true, instrumentFetch: true }, replay: { enabled: true, sampleRate: 1.0 }, privacy: { maskAllInputs: true, maskAllText: false }, @@ -77,26 +83,32 @@ MapleBrowser.init({ ### Auth via a proxy `ingestKey` only sets the `Authorization: Bearer` header. Without it, tracing and replay still run and -send without the header, so a first-party proxy (to get past ad blockers, or to keep the key out of -the bundle) can attach the key server-side: set `endpoint` to the proxy and leave `ingestKey` out. -Keyless against Maple's hosted ingest cannot work (it answers 401), so that combination logs a +send without the header. A first-party proxy (to get past ad blockers, or to keep the key out of +the bundle) can then attach the key server-side: set `endpoint` to the proxy and leave `ingestKey` out. +Keyless requests to Maple's hosted ingest cannot work (it answers 401), so that combination logs a warning. +### Regions + +Maple runs separate US and EU instances, and an ingest key only works in the region it was created +in. Set `region: "eu"` for an organization on `app.eu.maple.dev`. The SDK then sends to +`https://ingest.eu.maple.dev`. An explicit `endpoint` always wins over `region`. + ## Sessions -Every span and replay event the SDK emits is tagged with one **`session.id`** (a +Every span and replay event the SDK emits carries one **`session.id`** (a `crypto.randomUUID()` v4), minted on the first `MapleBrowser.init` call. That shared id is -what lets a trace jump to the replay that produced it, and vice versa. +what lets a trace jump to the replay that produced it, and the reverse. ### Storage & continuity The session is persisted in `sessionStorage` under the key `maple.session`, so it **survives -reloads within a tab**. Because `sessionStorage` is per-tab, **each tab or window gets its own -session** — sessions are never shared across them. When `sessionStorage` is unavailable (e.g. +reloads within a tab**. `sessionStorage` is per-tab, so **each tab or window gets its own +session**. Sessions are never shared across them. When `sessionStorage` is unavailable (e.g. some private-browsing modes), the SDK falls back to an in-memory record for the life of the page. -SPA route changes do **not** start a new session — the SDK tracks no router events, so +SPA route changes do **not** start a new session. The SDK tracks no router events, so client-side navigation stays within the same session. Session boundaries are purely time-based (see below). @@ -104,31 +116,33 @@ time-based (see below). A fresh `session.id` is minted when either limit is crossed, whichever comes first: -- **30 minutes idle** — no recorded activity for half an hour rotates the session (the same +- **30 minutes idle**: no recorded activity for half an hour rotates the session (the same activity-window model PostHog uses). -- **24 hours old** — a hard cap on a single session's lifetime regardless of activity, so a +- **24 hours old**: a hard cap on a single session's lifetime regardless of activity, so a tab left open for days doesn't collapse into one giant replay. -While replay is recording, each flushed chunk marks the session active, pushing back the idle -deadline — so a continuously-recording session stays whole. +While replay is recording, each flushed chunk marks the session active and pushes back the idle +deadline, so a continuously recording session stays whole. ### Start & end metadata -The SDK writes a small session-metadata row at two points: +The SDK writes a small session-metadata row at these points: -- an **`active`** row when recording starts (and again on each reload), and -- an **`ended`** row on page hide / unload — fired on `visibilitychange → hidden` (the - reliable "leaving" signal on mobile) and `pagehide` (desktop tab close / navigation). +- an **`active`** row when recording starts (and again on each reload); +- a heartbeat row every 60s, so exit page, page views and duration survive a tab killed without an + unload event; +- an **`ended`** row on page hide or unload, fired on `visibilitychange → hidden` (the + reliable "leaving" signal on mobile) and `pagehide` (desktop tab close or navigation). The `ended` row carries the session duration, the click count, and the **trace ids observed -during the session**, which is what powers trace↔replay correlation and the user/session +during the session**. Those ids power trace↔replay correlation and the user/session columns in Maple's session list and detail views. The unload write uses `keepalive`, so it survives the page going away. ### Accessing the session id -`init()` returns a handle whose `sessionId` is the active session's id — useful for -correlating Maple sessions with your own backend logs: +`init()` returns a handle whose `sessionId` is the active session's id. Use it to +correlate Maple sessions with your own backend logs: ```ts const { sessionId } = MapleBrowser.init({ @@ -140,14 +154,14 @@ const { sessionId } = MapleBrowser.init({ fetch("/api/checkout", { headers: { "x-maple-session": sessionId } }) ``` -`init()` is idempotent — calling it again returns the same live handle. On the server (SSR / +`init()` is idempotent: calling it again returns the same live handle. On the server (SSR, or no `window`) it returns a no-op handle with an empty `sessionId`. ### Teardown -Call `shutdown()` to flush the final replay chunk and tear down tracing + replay. After it -resolves, telemetry is fully stopped and a later `init()` may start a new session — handy when -a single-page app unmounts its telemetry client: +Call `shutdown()` to flush the final replay chunk and tear down tracing and replay. After it +resolves, telemetry is fully stopped and a later `init()` may start a new session. This is useful +when a single-page app unmounts its telemetry client: ```ts const maple = MapleBrowser.init({ ingestKey: "maple_pk_...", serviceName: "acme-web" }) @@ -158,9 +172,9 @@ await maple.shutdown() ## Identifying users -Pass `userId` so replays and traces are correlated to a known user — it populates the user column in the Maple session list and detail views, and browser-created spans include `user.id`. +Pass `user` (or the deprecated `userId`) so replays and traces are tied to a known user. It fills the user column in the Maple session list and detail views, and browser-created spans include `user.id`. -If you don't know the user at init time (e.g. the SDK starts before login resolves), omit it; the session begins anonymous. Once you know who the user is, call `MapleBrowser.identify(userId)` to attach (or replace) the id on the active session. Future session rows read the latest id when they post, and future spans read it when they start. +If you don't know the user at init time (e.g. the SDK starts before login resolves), omit it and the session begins anonymous. Once you know who the user is, call `MapleBrowser.identify(userId)` to attach (or replace) the id on the active session. `identify()` is also safe to call before `init()`; the latest call is applied when `init()` runs. Future session rows read the latest id when they post, and future spans read it when they start. ```ts // after the user signs in @@ -170,8 +184,8 @@ MapleBrowser.identify(user.id) MapleBrowser.identify(null) ``` -`identify()` also takes an object, which is what fills the rest of the session's identity columns — -the email and name shown on the session, and the company/team the Sessions UI can group by: +`identify()` also takes an object. It fills the rest of the session's identity columns: the email +and name shown on the session, and the company or team the Sessions UI can group by: ```ts MapleBrowser.identify({ @@ -184,7 +198,7 @@ MapleBrowser.identify({ }) ``` -Each call **replaces** the identity rather than merging it — merging would leak a signed-out user's +Each call **replaces** the identity instead of merging it. Merging would leak a signed-out user's email into whoever signs in next on a shared device. Traits are capped (24 keys, 64-char keys, 256-char values) and the identity is never persisted to storage. @@ -192,7 +206,7 @@ email into whoever signs in next on a shared device. Traits are capped (24 keys, `track(name, props)` records a product event against the current session. It lands as a `session_events` row with `Type='custom'`, so it appears inline in the session transcript next to the -clicks and network calls around it, rather than in a separate analytics silo. +clicks and network calls around it instead of in a separate analytics silo. ```ts MapleBrowser.track("checkout_completed", { plan: "pro", seats: 12 }) @@ -202,11 +216,49 @@ Names are capped at 128 chars; props at 32 keys / 64-char keys / 1024-char value Values are coerced to strings (`Date` → ISO, objects → JSON; `null`/`undefined`/functions are dropped). Calls before `init()` finishes are queued, and `track()` never throws. +## Errors + +Every uncaught error and unhandled rejection becomes a span with status `Error` and an `exception` +event, the shape Maple fingerprints. Browser crashes group beside your server-side errors. + +Errors your app catches never reach the global handlers. Report those with `captureException` +(a framework error boundary is the typical caller). The same error object is recorded once, even if +it is rethrown afterwards: + +```ts +try { + render() +} catch (error) { + MapleBrowser.captureException(error, { name: "browser.render_error" }) +} +``` + +The span name defaults to `"exception"`; `attributes` adds extra span attributes. Turn the global +handlers off with `tracing: { captureErrors: false }` only when another tracker already owns them +and the same crash would be recorded twice. + +Cross-origin scripts report a bare `"Script error."` with no stack or filename. The SDK drops those, +since they all fingerprint to one empty issue. Add `crossorigin` to the script tag to get the real +error. + +## Tracing across origins + +`fetch` spans send the W3C `traceparent` header to same-origin requests only. When your API lives +on another origin, list it so browser and backend spans join one trace, and allow the `traceparent` +header in the API's CORS policy: + +```ts +MapleBrowser.init({ + // ... + tracing: { propagateTraceHeaderCorsUrls: [/^https:\/\/api\.example\.com\//] }, +}) +``` + ## Linking a marketing site to your app The visitor id is stored in **both** localStorage and a cookie scoped to your registered domain, so `example.com` and `app.example.com` resolve to the same `VisitorId`. Initialize the SDK on both and -an anonymous visit links to the signed-in sessions it later becomes — filter the Sessions list by +an anonymous visit links to the signed-in sessions it later becomes. Filter the Sessions list by visitor id to see the whole journey. The **session** id is deliberately not shared: each origin keeps its own session, and `VisitorId` is @@ -238,20 +290,20 @@ MapleBrowser.setConsent(true) ``` Revoking stops capture without flushing, and a later grant starts a fresh session. Global Privacy -Control is honored regardless of `requireConsent` — it suppresses the persistent visitor id (the one -cross-session identifier the SDK stores) while leaving session-scoped capture alone. `doNotTrack` is +Control is honored regardless of `requireConsent`. It suppresses the persistent visitor id (the one +cross-session identifier the SDK stores) and leaves session-scoped capture alone. `doNotTrack` is ignored unless you set `privacy.respectDoNotTrack`. `privacy.persistVisitorId: false` turns the -visitor id off entirely and purges any already stored; `privacy.captureUserEmail: false` keeps +visitor id off entirely and purges any id already stored. `privacy.captureUserEmail: false` keeps `identify()`'s email out of the warehouse. ## Privacy & masking -`maskAllInputs` is **on by default**, so every `` value is masked before it leaves the browser. Set `maskAllText: true` to additionally mask all rendered text. +`maskAllInputs` is **on by default**, so every `` value is masked before it leaves the browser. Set `maskAllText: true` to also mask all rendered text. For finer control, use rrweb's attribute hooks to block specific elements or subtrees from capture: -- `data-rr-block` attribute, or the `.rr-block` class — block an element and its subtree (rendered as a placeholder). -- `.rr-ignore` class — ignore input events on an element. +- `data-rr-block` attribute, or the `.rr-block` class: block an element and its subtree (rendered as a placeholder). +- `.rr-ignore` class: ignore input events on an element. ```html
@@ -260,6 +312,17 @@ For finer control, use rrweb's attribute hooks to block specific elements or sub
``` +URLs are redacted before they leave the page. The values of credential-shaped query and fragment +parameters (`token`, `code`, `access_token`, `password`, ...) become `REDACTED` in session rows, +events, network events, replay meta events and span attributes. Add your own rewriting with +`privacy.sanitizeUrl`, e.g. to collapse ids in paths: + +```ts +privacy: { + sanitizeUrl: (url) => url.replace(/\/users\/\d+/, "/users/:id") +} +``` + ## Sampling To record only a fraction of sessions, set `replay.sampleRate` between `0` and `1`. For example, `0.1` records ~10% of sessions. Tracing is unaffected by this setting. @@ -315,5 +378,5 @@ In Next.js, run the import from a client component mounted high in the tree (e.g ## Notes -- Replay event blobs live in object storage; only small, queryable metadata is indexed — playback streams blobs directly via signed URLs. +- Replay event blobs live in object storage. Only small, queryable metadata is indexed, and playback streams blobs directly via signed URLs. - The SDK is browser-only and best-effort: telemetry network failures never surface to your application. diff --git a/docs/ci-performance.md b/docs/ci-performance.md index ed13fc8079..c7e90527a3 100644 --- a/docs/ci-performance.md +++ b/docs/ci-performance.md @@ -49,7 +49,7 @@ with Vitest flags, and keep their original arguments; Rust retains its dedicated The runtime budget still applies to the Bun lanes. If those suites grow beyond it, they need runner-specific splitting rather than Vitest flags. -The file-count projection is a capacity model; the measured fivefold suite run +The file-count projection is a capacity model. The measured fivefold suite run below validates the test setup separately. More shards consume more runner setup time and require available organization concurrency; [GitHub schedules matrix jobs according to runner availability](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstrategymatrix). @@ -78,7 +78,8 @@ The isolation tests verify that one database's DDL and session settings do not leak into another, and that a cache hit does not rewrite the fixture. UI defaults to Node, with explicit `@vitest-environment jsdom` directives on the -DOM suites. Pure calculations no longer initialize a browser environment. +DOM suites. Pure calculations no longer initialize a browser environment. (Superseded: +DOM suites now run in Chromium, see [Native browser tests](#native-browser-tests).) The measurements below use the same worker count before and after. They exclude installation/build time and GitHub queueing, and are single paired runs rather @@ -137,8 +138,8 @@ the existing abort-signal guard. Vitest 5's default mock cleanup is retained. Execution evals now install the shared PGlite global setup too. Collection was validated with a placeholder credential and an unmatched test-name filter: -all 44 cases collected without running model calls. Browser tests still use -jsdom; Browser Mode is a separate migration to a real browser provider. +all 44 cases collected without running model calls. Browser tests still used +jsdom at this point; the move to Browser Mode is described below. The optimization comparisons above were measured on Vitest 4.1.10. They are not measurements of the version upgrade itself. diff --git a/docs/database-query-observability.md b/docs/database-query-observability.md index 4dd94f8521..e00dfed25f 100644 --- a/docs/database-query-observability.md +++ b/docs/database-query-observability.md @@ -1,21 +1,25 @@ # Database query observability -Maple gives every database call first-class treatment using the standard +Maple treats database calls as first-class using the standard [OpenTelemetry database semantic conventions](https://opentelemetry.io/docs/specs/semconv/db/database-spans/). -If your services are instrumented with an OTel-aware database client — which most -SDKs enable automatically — you get, with **no extra configuration**: +If your services use an OTel-aware database client (most SDKs enable one +automatically), you get the following with **no extra configuration**: - **Query timing inline in every trace.** A database call's client span shows up - in the waterfall like any other span, and its detail panel renders a database - summary block (system, namespace, table, operation, rows returned, server) - derived from the `db.*` attributes. -- **A cross-service Queries surface.** Every distinct query _shape_ (the query - with literals normalized to `?`) is aggregated across services with call - volume, error rate, and p50/p95/p99 latency, so you can find your slowest and - busiest queries and drill straight to sample traces. + in the waterfall like any other span. Its detail panel renders a database + summary block (system, namespace, table, operation, rows returned, batch size, + server, error outcome) derived from the `db.*` attributes + (`packages/ui/src/lib/cloud-platforms/database.ts`). +- **Query shapes per database.** Selecting a database node on the service map + opens a detail panel with "Query Activity" and "Top Query Shapes". Every + distinct query _shape_ (the query with literals normalized to `?`) is + aggregated across the calling services with call volume, error rate, and + p50/p95 latency, plus a sample statement. The shapes come from the + `service_map_db_query_shapes_hourly` rollup, with the in-progress hour read + from raw traces. -This works for **any** database — PostgreSQL, MySQL, ClickHouse, Redis, MongoDB, -and more — because it reads only the vendor-neutral semantic conventions. +This works for **any** database (PostgreSQL, MySQL, ClickHouse, Redis, MongoDB, +and more) because it reads only the vendor-neutral semantic conventions. ## Attributes Maple reads @@ -33,37 +37,38 @@ and more — because it reads only the vendor-neutral semantic conventions. Query text is grouped by _shape_: literals are stripped to `?` and `IN (...)` lists are collapsed, so `WHERE id = 1` and `WHERE id = 2` are the same shape. +The shared SQL lives in `packages/domain/src/tinybird/db-query-shape-sql.ts`. Prefer emitting parameterized `db.query.text` (the OTel spec says parameterized text should **not** be sanitized) so shapes stay clean. ## Correlating server-side query logs with traces (SQLCommenter) -The client span above captures the query _as the caller sees it_ — duration and -the query text — but it cannot see server-side detail such as memory used or -rows/bytes scanned. To bridge that gap, tag your queries with **SQLCommenter**, -the OpenTelemetry-standard way to propagate trace context into the database by -appending a comment to the query: +The client span captures the query _as the caller sees it_: duration and query +text. It cannot see server-side detail such as memory used or rows/bytes +scanned. **SQLCommenter** is the OpenTelemetry-standard way to bridge that gap. +It propagates trace context into the database by appending a comment to the +query: ```sql SELECT * FROM events WHERE ts > ? /*traceparent='00---01'*/ ``` -Most OTel database instrumentations can inject this for you (it is opt-in — see -your SDK's SQLCommenter / "DB statement comment" option). Because the database -records the full query text — comment included — in its query log, Maple can -read that log back (see the ClickHouse integration) and stitch each server-side -query to the exact client span that issued it, nesting it as a child in the -trace. +Most OTel database instrumentations can inject this comment (it is opt-in; see +your SDK's SQLCommenter or "DB statement comment" option). The database then +records the full query text, comment included, in its query log, so a reader of +that log can match each server-side query to the client span that issued it. -> Note: SQLCommenter comments are low-cardinality-unfriendly for MySQL prepared -> statements, Oracle, and SQL Server; consult the OTel guidance before enabling +Maple does not read database query logs today. An earlier SQLCommenter parser in +`@maple/domain` was removed in the repo restructure (#328) because nothing used it. + +> Note: SQLCommenter comments can defeat statement caching for MySQL prepared +> statements, Oracle, and SQL Server. Consult the OTel guidance before enabling > it broadly on those engines. -## Resource allocation (ClickHouse) +## Resource allocation (not built) -Server-side **resource allocation** — peak memory, rows/bytes read, CPU time, -ProfileEvents — is not available from client spans. For ClickHouse, connect your -cluster via the ClickHouse integration: Maple polls `system.query_log`, forwards -each sampled query into Maple as a span (nested under your app's trace via the -SQLCommenter `traceparent` above) plus aggregate metrics, so query timing and -resource allocation land alongside your existing traces and dashboards. +Server-side **resource allocation** (peak memory, rows/bytes read, CPU time, +ProfileEvents) is not available from client spans. The original plan was a +ClickHouse integration that polls `system.query_log`, forwards each sampled +query as a span nested under the app's trace via the SQLCommenter `traceparent`, +and emits aggregate metrics. That poller does not exist yet. diff --git a/docs/docker-container-monitoring.md b/docs/docker-container-monitoring.md index 6832b96315..929a31e403 100644 --- a/docs/docker-container-monitoring.md +++ b/docs/docker-container-monitoring.md @@ -1,14 +1,14 @@ # Docker container monitoring How a plain-Docker host's containers become rows on `/infra/containers`, charts on the detail -page, and an Infrastructure tab on spans/logs — and the design decisions that will bite if +page, and an Infrastructure tab on spans/logs, plus the design decisions that will bite if re-derived differently. The k8s sibling of this lifecycle is [service-map-infrastructure.md](service-map-infrastructure.md); the end-user install guide is `apps/landing/src/content/docs/guides/docker-infrastructure.md`. ## End-to-end flow -1. **Emit.** The Maple Docker agent (one container per host — see +1. **Emit.** The Maple Docker agent (one container per host, see [`deploy/docker-agent/README.md`](../deploy/docker-agent/README.md)) runs the contrib `docker_stats` receiver against the host's Docker socket. Identity rides on ResourceAttributes: `container.name`, `container.id` (64-hex), `container.image.name`, @@ -19,46 +19,47 @@ re-derived differently. The k8s sibling of this lifecycle is (parsed from the file path). 2. **Ship & store.** OTLP HTTP → ingest gateway → warehouse, untouched: no gateway changes, no warehouse schema changes. Gauge-typed metrics land in `metrics_gauge`, sum-typed in - `metrics_sum`, identity in the raw `ResourceAttributes` map — read at query time exactly like - the pod queries. + `metrics_sum`, identity in the raw `ResourceAttributes` map. Identity is read at query time, + exactly like the pod queries. 3. **Query.** `packages/query-engine/src/ch/queries/containers.ts` mirrors the pod slice: list/summary/detail/counters/timeseries/facets, fixtured in the SQL catalog gate. -4. **Serve.** Five container endpoints under `/internal/query-engine` — `list-containers`, - `containers-summary`, `container-detail-summary`, `container-infra-timeseries`, - `container-facets` (`apps/api/src/routes/internal/query-engine.http.ts`), contracts in - `packages/domain/src/http/query-engine.ts`. +4. **Serve.** Five container endpoints under `/internal/query-engine`: `list-containers`, + `containers-summary`, `container-detail-summary`, `container-infra-timeseries`, and + `container-facets`. Handlers are in `apps/api/src/routes/internal/query-engine.http.ts`, + contracts in `packages/domain/src/http/query-engine.ts`. 5. **Render.** `/infra/containers` list + `/infra/containers/$containerName?host=` detail, the `container.name`-keyed correlation group on span/log Infrastructure tabs, and the - `docker-containers` dashboard template. + `docker-containers` dashboard template + (`packages/backend/src/dashboard-templates/infrastructure/docker-containers.ts`). ## Load-bearing decisions -- **Identity is `(container.name, host.name)`.** Docker names are unique per host only — `redis` +- **Identity is `(container.name, host.name)`.** Docker names are unique per host only: `redis` on five hosts is five rows. `container.id` is the `uniq()` facet-cardinality and correlation key (a recreated container keeps its name, not its id) and is displayed, never in the URL. - **Percent scale is normalized in the queries.** docker_stats percents are 0..100; every projected percentage (list, summary, detail, and the cpu/memory_percent timeseries via `divideBy: 100`) divides by 100 so the 0.9/0.6 saturation thresholds and severity toning match - the pod pages. CPU can legitimately exceed 1.0 on multi-core containers — documented, - unclamped. The dashboard template charts the raw 0..100 gauge and says so in the widget title. + the pod pages. CPU can legitimately exceed 1.0 on multi-core containers. This is documented and + left unclamped. The dashboard template charts the raw 0..100 gauge and says so in the widget title. - **kubeletstats rows are excluded everywhere.** k8s per-container metrics also carry `container.name` (on the 0..1 scale); every container query, the correlation detector, and the dashboard template's widgets require `k8s.pod.name = ''`. - **Sum-family aggregation is per-metric.** Cumulative counters (network, block I/O) surface - bucketed sums (relative shape, same caveat as the host network chart). Sampled sums — - `container.memory.usage.total` — must AVERAGE a bucket's samples (`average: true`); summing + bucketed sums (relative shape, same caveat as the host network chart). Sampled sums such as + `container.memory.usage.total` must AVERAGE a bucket's samples (`average: true`); summing inflates the chart by samples-per-bucket. - **Counters detail groups per host before differencing.** `restartsDelta = maxIf − minIf` over two hosts' independent cumulative counters would fabricate a delta from their offset; the query differences per `host.name`, then sums. - **Compose labels are promoted from datapoint to resource.** `container_labels_to_metric_labels` lands `compose.project`/`compose.service` as _datapoint_ attributes, but every container query - reads them from `ResourceAttributes` — so `transform/compose` in the agent config moves them. + reads them from `ResourceAttributes`, so `transform/compose` in the agent config moves them. docker_stats emits one Resource per container, so a plain transform is safe here; the Fargate prometheus path in the k8s chart needs `groupbyattrs` first because one Resource covers many - pods. The query tests assert SQL strings only, so they cannot catch a regression here — it + pods. The query tests assert SQL strings only, so they cannot catch a regression here. It shows up as permanently empty project/service facets. -- **Scopes are `saturated | elevated | stale` — deliberately no `unbounded`.** Running without +- **Scopes are `saturated | elevated | stale`, deliberately with no `unbounded`.** Running without limits is the norm in plain Docker, so the pod "burning CPU with nothing capping it" bucket doesn't transfer. @@ -69,9 +70,9 @@ Docker has no k8sattributes/operator analog to inject identity into app telemetr - The reliable path is documented in the guide: `OTEL_RESOURCE_ATTRIBUTES` with `container.id` and `container.name` (Docker's default hostname _is_ the short id). - `@maple/effect-sdk` auto-detects `container.id`/`container.runtime` (mountinfo → cgroup → - short-id hostname, via `process.getBuiltinModule` so edge bundles stay safe) — but not - `container.name`, so auto-detection alone does not light the Infrastructure tab. -- Agent-shipped container logs carry only `container.id` — they correlate to Host, not to the + short-id hostname, via `process.getBuiltinModule` so edge bundles stay safe). It does not + detect `container.name`, so auto-detection alone does not light the Infrastructure tab. +- Agent-shipped container logs carry only `container.id`. They correlate to Host, not to the container group. - The correlation group is suppressed on k8s records and when a non-docker `container.runtime` is declared; records with `container.name` and no runtime attribute (e.g. some ECS setups) @@ -80,23 +81,24 @@ Docker has no k8sattributes/operator analog to inject identity into app telemetr ## Release coupling The install one-liner, docs guide, and baked `/etc/otel/docker-config.yaml` all name -`otel-collector-maple:0.2.0`, published from the `otel-collector-maple-v0.2.0` tag (tag-driven -workflow — merging to main publishes nothing). +`otel-collector-maple:0.2.0`, published from the `otel-collector-maple-v0.2.0` tag by +`.github/workflows/publish-otel-collector-maple.yml`. The workflow is tag-driven: merging to main +publishes nothing. Two things bite here. The image name is derived from `github.repository_owner`, so since the move to the MapleTechLabs org everything publishes to `ghcr.io/mapletechlabs/...`; the pre-move `ghcr.io/makisuo/...` packages still exist, are still public, and are frozen at 0.1.5. And a GHCR -package created by a workflow starts **private** — visibility is per-package and has to be flipped +package created by a workflow starts **private**. Visibility is per-package and has to be flipped once by hand, so a freshly published image is not pullable by users until someone does. -Before announcing: confirm the package is public with an anonymous pull, then smoke one live agent -— `container.cpu.utilization` rows arriving through the hosted gateway, and the compose -project/service facets populating (the query tests assert SQL strings only, so they cannot prove -that path). The BYO-ClickHouse `renderCollectorYaml()` tracks the latest _published_ tag. +Before announcing, confirm the package is public with an anonymous pull. Then smoke-test one live +agent: `container.cpu.utilization` rows should arrive through the hosted gateway, and the compose +project/service facets should populate. The query tests assert SQL strings only, so they cannot +prove that path. The BYO-ClickHouse `renderCollectorYaml()` tracks the latest _published_ tag. ## Not built yet, and why `service_platforms_hourly` ContainerId/ContainerName columns + the `"docker"` service-map -platform badge (warehouse migration — read [warehouse-rollups.md](warehouse-rollups.md) first), +platform badge (a warehouse migration; read [warehouse-rollups.md](warehouse-rollups.md) first), a container table on host detail, trimming the receiver's unused default-enabled metrics, rootless Docker/podman socket paths, and the landing marketing page. diff --git a/docs/drizzle-v1-effect-upgrade-plan.md b/docs/drizzle-v1-effect-upgrade-plan.md index 2a71868398..5f9030b43d 100644 --- a/docs/drizzle-v1-effect-upgrade-plan.md +++ b/docs/drizzle-v1-effect-upgrade-plan.md @@ -1,5 +1,10 @@ # Drizzle 1.0 + Effect driver upgrade plan +Status (2026-09-25): shipped in #904 (`drizzle-orm@1.0.0-rc.5-5935859`, `@effect/sql-pg`). The +client now uses `PgClient.make` (`packages/db/src/client.ts`), prd migrations apply in the alchemy +deploy (#939, which also removed `migrate:prod`), and #970 removed `ps:migrations-preflight`. The plan below +is kept as written. + Written 2026-09-15 against `main` (`0bfb54137c`). Maple is on `drizzle-orm@0.45.2` / `drizzle-kit@0.31.10` with the `postgres-js` driver on Workers and `pglite` in vitest, on `effect@4.0.0-rc.112`. @@ -37,7 +42,7 @@ Split the work so the ORM major and the driver swap never land in the same deplo 3. **Phase B, the Effect driver (4 to 6 days).** Rebuild `DatabaseLive` and the connection scope over `PgClient` + `PgDrizzle`, then sweep the call sites to `yield*`. Needs drizzle `>= 1.0.0-rc.5`: either the tagged rc.5 once it is cut, or the `1.0.0-rc.5-5935859` snapshot - now. Pinning a commit-suffixed snapshot in prod is the owner's call; the kit in that snapshot + now. Pinning a commit-suffixed snapshot in prod is the owner's call. The kit in that snapshot reads the converted folder unchanged, so bumping from rc.4 is a manifest change only. Gate B on A having soaked in prod for at least a week, because A already moves the migration diff --git a/docs/electric-sync.md b/docs/electric-sync.md index 95ed2658f8..ecf324cf2b 100644 --- a/docs/electric-sync.md +++ b/docs/electric-sync.md @@ -2,28 +2,28 @@ Maple syncs a small set of **relational control-plane tables** to the web app in real time using [ElectricSQL](https://electric.ax) shapes fronted by -[TanStack DB](https://tanstack.com/db) collections. Warehouse/analytics data -(traces, logs, metrics via `@maple/query-engine`) is **not** synced — it stays on -the effect-atom + `WarehouseQueryService` path. +[TanStack DB](https://tanstack.com/db) collections. Warehouse data (traces, logs, +metrics via `@maple/query-engine`) is **not** synced. It stays on the effect-atom + +`WarehouseQueryService` path. Electric sync is the primary read path for the dashboards and alerts verticals. Only `/dashboards` has a fallback: it degrades to a plain-HTTP snapshot with -writes disabled (`SyncDegradedBanner`). The alerts lists have none — with the +writes disabled (`SyncDegradedBanner`). The alerts lists have none. With the sync worker or its upstream unreachable they show `SyncUnavailable` and a retry. -So an Electric outage is visible, and a deploy of the singleton service is a -short one. +An Electric outage is therefore visible, and a deploy of the singleton service is +a short one. The reusable machinery lives in two workspace libraries: -- `@maple/effect-db/electric` — `createEffectCollection` wraps +- `@maple/effect-db/electric` (`lib/effect-db`): `createEffectCollection` wraps `@tanstack/electric-db-collection` with Effect Schema rows, Effect write handlers run on a `ManagedRuntime`, exponential backoff, and typed - `awaitTxIdEffect`. The backoff `onError` also dispatches the - `auth:session-expired` (401) and `collection:schema-error` (post-deploy schema - drift) window events. -- `@maple/unitflow/db` — collection subscriptions feed model-scoped Stores used - by the dashboards and alerts lists. Mutations use the typed API write paths; - alert toggles run through `Mutation.make`. + `awaitTxIdEffect`. The backoff `onError` also dispatches window events: + `collection:auth-error` (401), `collection:schema-error` (post-deploy schema + drift) and `collection:sync-failed` (retries exhausted). +- `@maple/unitflow/db` (`lib/unitflow`): collection subscriptions feed + model-scoped Stores used by the dashboards and alerts lists. Mutations use the + typed API write paths; alert toggles run through `Mutation.make`. ## How it fits together @@ -31,14 +31,16 @@ The reusable machinery lives in two workspace libraries: Browser (apps/web) TanStack DB collections, one set per active org read: ShapeStream → GET {VITE_ELECTRIC_SYNC_URL}/api/sync/shape?shape=&offset=…&handle=… - (mapleShapeFetch injects the Clerk / self-hosted bearer) + (mapleSyncFetch in lib/collections/shape-fetch.ts injects the Clerk / self-hosted bearer) write: typed HTTP endpoints on apps/api (dashboards use public `/v2`; Electric is read-path only) -apps/electric-sync Worker — /api/sync/shape (src/routes/shape.http.ts, a raw HttpRouter) +apps/electric-sync Worker: /api/sync/shape (src/routes/shape.http.ts, a raw HttpRouter) a standalone, DB-free worker (deploys independently of apps/api) - auth: Clerk/self-hosted tenant resolution ONLY (makeResolveTenant, shared from - @maple/api/electric-sync) — no API-key path, since it has no database - pins: table + `"org_id" = $1` (+ per-shape extra WHERE), params[1]=orgId, secret + auth: Clerk/self-hosted tenant resolution ONLY (makeResolveTenant from @maple/auth, + wrapped in src/auth/TenantResolver.ts); no API-key path, since it has no database + whitelist: src/shapes/registry.ts; request validation: src/shapes/request.ts + pins (src/electric/ElectricClient.ts): table + `"org_id" = $1` (+ per-shape extra WHERE + or `$2` scope), params[1]=orgId, secret forwards ONLY offset/handle/live/cursor from the client streams Electric's response back (buffers the long-poll body) @@ -46,29 +48,31 @@ Electric (apps/electric on ECS Fargate in prod / docker `electric` locally) ← logical replication ← PlanetScale Postgres (direct 5432, publication electric_publication_default) writes: endpoint captures the Postgres txid on the mutating statement - (`pg_current_xact_id()::xid::text`, apps/api/src/lib/electric-txid.ts) and returns it; + (`pg_current_xact_id()::xid::text`, packages/backend/src/platform/electric-txid.ts) and returns it; the collection's write handler passes it to awaitTxId, which drops optimistic state once that transaction arrives on the shape stream. ``` -### Shapes (server-pinned whitelist, `apps/electric-sync/src/routes/shape.http.ts`) +### Shapes (server-pinned whitelist, `apps/electric-sync/src/shapes/registry.ts`) -| shape | table | pinned columns / extra WHERE (besides org scope) | -| -------------------- | ------------------ | ------------------------------------------------ | -| `dashboards` | dashboards | — | -| `alert_rules` | alert_rules | — | -| `alert_rule_states` | alert_rule_states | — | -| `alert_incidents` | alert_incidents | — | -| `alert_destinations` | alert_destinations | columns: drops the encrypted `secret_*` | -| `api_keys` | api_keys | columns: drops `key_hash` / `metadata_json` | +| shape | table | pinned columns / extra WHERE (besides org scope) | +| -------------------- | ------------------ | --------------------------------------------------------------- | +| `dashboards` | dashboards | none | +| `alert_rules` | alert_rules | none | +| `alert_rule_states` | alert_rule_states | none | +| `alert_incidents` | alert_incidents | none | +| `alert_destinations` | alert_destinations | columns: drops the encrypted `secret_*` | +| `api_keys` | api_keys | columns: drops `key_hash` / `metadata_json` | +| `investigation_v2` | investigations | scoped to one `id` (client supplies the value as `$2`); columns | -Shape `where`/columns are **immutable** — changing a pinned predicate forces a +Shape `where`/columns are **immutable**. Changing a pinned predicate forces a full re-sync for every client. If you must change one, version the shape name -(e.g. `dashboards.v2`) so old clients keep working during a deploy overlap. +(as `investigation_v2` did when it gained `progress_json`) so old clients keep +working during a deploy overlap. -The whitelist and `electric_publication_default` must stay in step **both ways**: -a shape over an unpublished table never receives changes (Electric runs with -`ELECTRIC_MANUAL_TABLE_PUBLISHING=true` and will not publish one itself), and a +The whitelist and `electric_publication_default` must stay in step **both ways**. +A shape over an unpublished table never receives changes (Electric runs with +`ELECTRIC_MANUAL_TABLE_PUBLISHING=true` and will not publish one itself). A published table with no shape is pure replication cost. `error_issues`, `actors`, `error_incidents` and `scrape_target_checks` were published for verticals that have since moved back to the typed `/v2` endpoints, and were pruned from both by @@ -76,19 +80,19 @@ have since moved back to the typed `/v2` endpoints, and were pruned from both by ## Local development -1. `bun db:up` starts the docker Postgres (now with `wal_level=logical`) and the - `electric` service (port 3473) — see `docker-compose.development.yml`. +1. `bun db:up` starts the docker Postgres (with `wal_level=logical`) and the + `electric` service (port 3473). See `docker-compose.development.yml`. If your Postgres volume predates the `wal_level` change, recreate it: `docker compose -f docker-compose.development.yml up -d --force-recreate postgres electric`. 2. `bun db:migrate:local` applies migrations, including `0009_electric_publication`. 3. `.env.local`: `ELECTRIC_URL=http://localhost:3473` (already in `.env.example`), read by the `apps/electric-sync` worker. Under `bun dev` the web app finds that worker at - `https://electric-sync.localhost` on its own; `VITE_ELECTRIC_SYNC_URL` only matters when + `https://electric-sync.localhost` on its own. `VITE_ELECTRIC_SYNC_URL` only matters when running the web app on a raw port without the portless proxy. -4. Run the app (`bun dev`) — the `electric-sync` worker comes up in the `alchemy dev` +4. Run the app (`bun dev`). The `electric-sync` worker comes up in the `alchemy dev` stack with everything else (`bun dev api electric-sync web` for just the pieces that - matter here). The dashboards/alerts/errors lists read exclusively from the sync path, so - steps 1–3 are required for them to load. + matter here). The synced lists (dashboards, alerts, API keys, investigation detail) + read from the sync path, so steps 1-3 are required for them to load. Smoke-test the proxy directly (through the standalone worker; needs a bearer): `curl -g 'https://electric-sync.localhost/api/sync/shape?shape=dashboards&offset=-1' -H "authorization: Bearer "`, @@ -96,19 +100,19 @@ or hit Electric with no proxy: `curl -g 'http://localhost:3473/v1/shape?table=da ### Troubleshooting -**`Electric sync is not configured` (HTTP 503)** — the worker's 503 body when it has +**`Electric sync is not configured` (HTTP 503).** The worker's 503 body when it has no upstream `ELECTRIC_URL`. Two causes: 1. `ELECTRIC_URL` isn't set in `.env.local`. Set `ELECTRIC_URL=http://localhost:3473`, - then **restart** `bun dev` — `--env-file` is read once when `alchemy dev` starts, so + then **restart** `bun dev`. `--env-file` is read once when `alchemy dev` starts, so a hot source reload won't pick it up. -2. The docker `electric` service isn't running on `:3473`. `bun db:up` starts it now; +2. The docker `electric` service isn't running on `:3473`. `bun db:up` starts it; confirm with `docker compose ps` (expect `maple-electric-1`). **Shapes 404 / `Database table public. is missing from the publication`** (or `does not have its replica identity set to FULL`). The early publication migrations (`0009`, `0011`, `0014`, `0037`) wrap their DDL in `DO $$ … EXCEPTION WHEN OTHERS THEN -RAISE NOTICE … END $$`, so on real Postgres a failure inside one is swallowed and drizzle +RAISE NOTICE … END $$`. On real Postgres a failure inside one is swallowed, and drizzle or alchemy still records it as applied. The case that actually happened: on the fresh EU database the publication existed, empty, before the first migration ran. `0009`'s `CREATE PUBLICATION` raised `duplicate_object`, and that handler rolls back the entire @@ -116,11 +120,11 @@ block, `REPLICA IDENTITY FULL` included. The later migrations then `ADD`ed their tables, so only `dashboards`, `alert_rules`, `alert_rule_states` and `alert_incidents` were missing. -`electric_publication_reconcile` closes this for every new database: it runs after all -of them, unguarded, and converges the publication on `SYNCED_TABLES` (creates it if -absent, sets FULL, adds what is missing, drops and resets anything extra). It is a no-op -on a database that is already correct. A database that ran it and then drifted by hand -needs the check below. +`electric_publication_reconcile` closes this for every new database. It runs after all +of them, unguarded, and converges the publication on `SYNCED_TABLES`: it creates the +publication if absent, sets FULL, adds what is missing, and drops and resets anything +extra. It is a no-op on a database that is already correct. A database that ran it and +then drifted by hand needs the check below. ```bash docker exec maple-postgres-1 psql -U maple -d maple -c " @@ -137,7 +141,7 @@ docker exec maple-postgres-1 psql -U maple -d maple -c " Every row should show `f`, `published = t` and `expected = t`. A missing synced table shows `published = f`; an extra member shows `expected = f`. To self-heal, apply the current membership (`SYNCED_TABLES` in -`packages/db/src/migrations.test.ts`), every statement idempotent: +`packages/db/src/migrations.test.ts`). Every statement is idempotent: ```bash docker exec -i maple-postgres-1 psql -U maple -d maple <<'SQL' @@ -160,130 +164,135 @@ SQL ``` On PlanetScale run the same SQL through `pscale shell main` (as done for -`maple-eu` on 2026-09-23). `SET TABLE` replaces the membership in one statement; a table +`maple-eu` on 2026-09-23). `SET TABLE` replaces the membership in one statement. A table it removes keeps FULL until you reset it with `REPLICA IDENTITY DEFAULT`. -**Nothing syncs but no error** — check `VITE_ELECTRIC_SYNC_URL` points at the +**Nothing syncs but no error.** Check that `VITE_ELECTRIC_SYNC_URL` points at the running `electric-sync` worker and that the docker `electric` service is up. It's a build-time constant, so a Vite restart is needed after changing it. ## Production (PlanetScale + self-hosted Electric on ECS) Electric Cloud is gone. `apps/electric` runs the upstream `electricsql/electric` -image on ECS Fargate at `electric.maple.dev`, with +image on ECS Fargate at `electric.maple.dev` (`electric.eu.maple.dev` for EU), with its own cluster, ALB, security groups and certificate **inside the ingest fleet's -VPC**. The shared VPC is forced, not an economy: two `AWS.EC2.Network`s in one -alchemy stack fight over the internet gateway — under `--adopt` the second one's +VPC**. The shared VPC is forced, not an economy. Two `AWS.EC2.Network`s in one +alchemy stack fight over the internet gateway: under `--adopt` the second one's create resolves to the first's IGW and tries to detach it, which AWS refuses on a VPC whose tasks hold public IPs (`DependencyViolation: … has some mapped public address(es)`). Both services want the same network anyway: public subnets, public -IPs, no NAT. -Nothing was migrated to get there: Postgres is the source of truth and Electric +IPs, no NAT. The root `alchemy.run.ts` therefore creates Electric only on stages +that also deploy ingest. +Nothing was migrated to get there. Postgres is the source of truth and Electric is a cache over its logical replication stream. **What guards it.** The service is public, because its only caller is a Worker at the Cloudflare edge with no private route into a VPC. `ELECTRIC_SECRET` is the -control — Electric requires it on every shape request, and the sync worker -appends it as `?secret=` exactly as it did for Cloud's source secret. One secret, -both ends of the hop. The task security group additionally admits `ELECTRIC_PORT` -only from the ALB's group, so a task's public IP is not a way around TLS. +control. Electric requires it on every shape request, and the sync worker +appends it as `?secret=`, as it did for Cloud's source secret. One secret covers +both ends of the hop. The task security group admits `ELECTRIC_PORT` only from +the ALB's group, so a task's public IP is not a way around TLS. **It is a singleton.** Two Electrics cannot share a replication slot, so the -service runs `desiredCount: 1` with `minimumHealthyPercent: 0` — stop the old +service runs `desiredCount: 1` with `minimumHealthyPercent: 0`: stop the old task, then start the new one. Every deploy therefore has a ~60s window with no -sync: `/dashboards` degrades to its HTTP snapshot (`SyncDegradedBanner`), while +sync. `/dashboards` degrades to its HTTP snapshot (`SyncDegradedBanner`), while the alerts lists have no fallback and sit in their retry state. This is why the image tag is pinned in `apps/electric/Dockerfile` rather than tracking `:latest` -like local docker does. +as local docker does. ### Standing it up 1. **PlanetScale cluster params:** `wal_level=logical`, `max_replication_slots>=10`, `max_wal_senders>=10`, `max_slot_wal_keep_size>=4096`, `sync_replication_slots=on`, `hot_standby_feedback=on`. Already set for Cloud; unchanged. -2. **The role is declared:** `Planetscale.PostgresRole("electric", { withReplication: true, +2. **The role is declared:** `Planetscale.PostgresRole("electric-db-role", { withReplication: true, inheritedRoles: ["postgres"] })` in `alchemy.run.ts`. The `REPLICATION` _attribute_ is never - inherited through role membership, Electric's database validation rejects a role without it + inherited through role membership. Electric's database validation rejects a role without it with a message that does not say so, and PlanetScale issues it only alongside `postgres`. - Its DIRECT 5432 URL (logical replication cannot run through PSBouncer or Hyperdrive), rewritten - to `sslmode=require` because Electric refuses `verify-full`, is the task's `DATABASE_URL`. -3. **Env:** `ELECTRIC_SECRET`. Both secrets reach the task through Secrets Manager, never the - task definition's plaintext `env`; the role id sits in `env` so a replaced role restarts the - singleton on the new secret before alchemy deletes the old one. + Its DIRECT 5432 URL is the task's `DATABASE_URL`. Logical replication cannot run through + PSBouncer or Hyperdrive, and the URL is rewritten to `sslmode=require` (`pgUrlRequireSsl`) + because Electric refuses `verify-full`. +3. **Env:** `ELECTRIC_SECRET`. Both secrets (`DATABASE_URL`, `ELECTRIC_SECRET`) reach the task + through Secrets Manager, never the task definition's plaintext `env`. The role id sits in + `env` (`MAPLE_PG_ROLE_ID`) so a replaced role restarts the singleton on the new secret before + alchemy deletes the old one. 4. **Migrate,** then `alchemy deploy`. No new migration is needed: the service reads the publication the migrations maintain, and `electric_publication_reconcile` makes a fresh database's membership exact even if the publication already exists. 5. **DNS.** The stack publishes the ACM validation CNAME into the `maple.dev` zone and waits for the certificate to reach `ISSUED` before attaching the 443 listener (`@maple/infra/acm`), so the first deploy needs no second pass. The - one manual record is a **proxied CNAME for `electric.maple.dev` at the ALB** — - the deploy output carries the hostname. + one manual record is a **proxied CNAME for `electric.maple.dev` at the ALB**. + The deploy output carries the hostname. 6. **Verify** before pointing anything at it: - `curl https://electric.maple.dev/v1/health`, then a shape through the proxy — + `curl https://electric.maple.dev/v1/health`, then a shape through the proxy: `curl -g 'https://sync.maple.dev/api/sync/shape?shape=dashboards&offset=-1' -H "authorization: Bearer "`. 7. **Cut over:** set `ELECTRIC_URL=https://electric.maple.dev` and clear - `ELECTRIC_SOURCE_ID`, then redeploy the sync worker. Reverting is the same env - change backwards, which is the entire point of running the two side by side. + `ELECTRIC_SOURCE_ID`, then redeploy the sync worker. This is done. While Cloud + still existed, reverting was the same env change backwards; with Cloud gone there + is nothing to revert to. ### The publication `ELECTRIC_MANUAL_TABLE_PUBLISHING=true`, and `ELECTRIC_REPLICATION_STREAM_ID` is -left at Electric's `default` — so it reads `electric_publication_default`, the -migration-owned publication, and opens `electric_slot_default` for itself. +left at Electric's `default`. Electric therefore reads `electric_publication_default`, +the migration-owned publication, and opens `electric_slot_default` for itself. -Electric Cloud never used that pair: it created its own generated +Electric Cloud never used that pair. It created its own generated `cloud_electric_pub_*` / `cloud_electric_slot_*`. That is why the self-hosted -service can run beside it on the same database with no collision, and why -flipping `ELECTRIC_URL` between them is a reversible env change rather than a -leap. +service could run beside Cloud on the same database with no collision during the +cutover. -## PR previews (no Electric source — dormant since 2026-08, now also Cloud-less) +## PR previews (no Electric source; dormant since 2026-08, now also Cloud-less) **PR previews no longer have an Electric source.** They stopped provisioning a PlanetScale branch (see `resolveDatabaseMode` in `packages/infra/src/cloudflare/stage.ts`), and with no Postgres to replicate from -there is nothing for Electric to point at. `apps/electric-sync/alchemy.run.ts` +there is nothing for Electric to point at. `apps/electric-sync/src/worker.ts` therefore withholds `ELECTRIC_URL`/`ELECTRIC_SOURCE_ID`/`ELECTRIC_SECRET` on the -`pr` stage — deliberately, so a preview can never inherit the shared `dev` +`pr` stage. This is deliberate: a preview can never inherit the shared `dev` credentials and proxy its shapes at another stage's data. The sync worker deploys -unconfigured, returns 503, and the web app falls back to its effect-atom fetches. +unconfigured and returns 503, so `/dashboards` falls back to its HTTP snapshot and +the alerts lists show `SyncUnavailable`. -Both paths are now dead, not just `up`: Electric Cloud is gone, so there are no -environments left to reap and `scripts/electric-pr-branch.ts` has nothing to call. -What follows is kept as the record of what previews used to do — restoring live -sync in a preview means pointing it at a self-hosted Electric, not at Cloud. +Every path of `scripts/electric-pr-branch.ts` is now dead, not just `up`. The +workflows still call `down` (on PR close) and `sweep` (from +`cleanup-preview-orphans.yml`), but Electric Cloud is gone, so there are no +environments left to reap. What follows is kept as the record of what previews used +to do. Restoring live sync in a preview means pointing it at a self-hosted Electric, +not at Cloud. The former lifecycle: an ephemeral Electric Cloud **environment** `pr-` + a Postgres **source** per PR, mirroring the PlanetScale/Tinybird branch lifecycle. `scripts/electric-pr-branch.ts` (`up`/`down `, driven from -`.github/workflows/deploy-pr-preview.yml`) uses `@electric-sql/cli` -(`ELECTRIC_API_TOKEN` auth) to, on open/synchronize: reuse (or create under -`ELECTRIC_PROJECT_ID`) the `pr-` environment, reset its services, create a -fresh `postgres` source pointed at the PR branch's `MAPLE_PG_ELECTRIC_URL` -(direct 5432 through a dedicated `--with-replication` role — Electric requires -the REPLICATION role _attribute_, which is never inherited; the main CI role -stays non-replication because PlanetScale replication roles aren't grantable, -which would break the in-place reset's role assumption), polled until active, -and export -`ELECTRIC_URL`/`ELECTRIC_SOURCE_ID`/`ELECTRIC_SECRET` to `$GITHUB_ENV` (bound to -the electric-sync worker by alchemy). On close it deletes the environment -(cascades the source). Steps are gated on `ELECTRIC_API_TOKEN`, so previews stay -green (and the worker 503s) until the token lands in Infisical. - -- The web build always reads through the sync path, so with no source provisioned - a preview's synced lists fall back to their effect-atom fetches. Provisioning - the source is what would make live sync work in previews again. -- **Publication:** the migrate step runs `0009` (creates - `electric_publication_default`) before the source is created. The script passes - `--manual-table-publishing` by default (prod parity; Electric reads that - migration-owned publication — its default name — instead of owning the tables). - Set `ELECTRIC_MANUAL_TABLE_PUBLISHING=false` to let Electric auto-manage - publishing instead; `ELECTRIC_SERVICE_EXTRA_ARGS` remains the flag escape hatch. - The script pins `@electric-sql/cli@0.0.10` (interface verified — `--json` is a +`.github/workflows/deploy-pr-preview.yml`) used `@electric-sql/cli` +(`ELECTRIC_API_TOKEN` auth). On open/synchronize it reused (or created under +`ELECTRIC_PROJECT_ID`) the `pr-` environment, reset its services, and created a +fresh `postgres` source pointed at the PR branch's `MAPLE_PG_ELECTRIC_URL`. That URL +was direct 5432 through a dedicated `--with-replication` role: Electric requires +the REPLICATION role _attribute_, which is never inherited, and the main CI role +stays non-replication because PlanetScale replication roles aren't grantable, which +would break the in-place reset's role assumption. The script polled the source until +active and exported `ELECTRIC_URL`/`ELECTRIC_SOURCE_ID`/`ELECTRIC_SECRET` to +`$GITHUB_ENV`, where alchemy bound them to the electric-sync worker. On close it +deleted the environment (cascading the source). Steps were gated on +`ELECTRIC_API_TOKEN`, so previews stayed green (and the worker 503'd) until the token +landed in Infisical. + +- The web build always reads through the sync path. Provisioning a source is what + would make live sync work in previews again. +- **Publication:** the migrate step ran `0009` (creates + `electric_publication_default`) before the source was created. The script passed + `--manual-table-publishing` by default (prod parity: Electric reads that + migration-owned publication, its default name, instead of owning the tables). + `ELECTRIC_MANUAL_TABLE_PUBLISHING=false` let Electric auto-manage publishing + instead; `ELECTRIC_SERVICE_EXTRA_ARGS` was the flag escape hatch. + The script pins `@electric-sql/cli@0.0.10` (interface verified: `--json` is a global flag, `environments create` returns `environmentId`, the postgres service - id is the shape-API `source_id`); re-verify before bumping the pin. -- **Caps:** each source counts against the Electric plan's max-databases limit and - holds a PlanetScale replication slot; teardown on close is mandatory. + id is the shape-API `source_id`). Re-verify before bumping the pin. +- **Caps:** each source counted against the Electric plan's max-databases limit and + held a PlanetScale replication slot, so teardown on close was mandatory. ## Adding a synced table later @@ -294,12 +303,12 @@ electric_publication_default ADD TABLE ""`, each behind a catalog check `electric_publication_reconcile`. **No `EXCEPTION` handler.** PGlite (0.5+) runs `CREATE/ALTER PUBLICATION` and `pg_publication_tables`, so the guard the early migrations carry protects nothing, and it is exactly what hid the EU failure. -2. Add the shape to the whitelist in `apps/electric-sync/src/routes/shape.http.ts`. +2. Add the shape to the whitelist in `apps/electric-sync/src/shapes/registry.ts`. 3. Add a collection under `apps/web/src/lib/collections/` via - `createEffectCollection` (model on `dashboards.ts` for a write vertical, or - `alerts.ts` for a read-only one — an identity `Schema.Struct` row schema that - mirrors the table columns, plus a `timestamptz` parser normalizing to ISO), - register it in `org-collections.ts` (constructor + `cleanup()`), and point the + `createEffectCollection`. Model it on `dashboards.ts` for a write vertical or + `alerts.ts` for a read-only one: an identity `Schema.Struct` row schema that + mirrors the table columns, plus a `timestamptz` parser normalizing to ISO. + Register it in `org-collections.ts` (constructor + `cleanup()`), and point the consumer read at the collection. 4. Update `SYNCED_TABLES` in `packages/db/src/migrations.test.ts`. @@ -313,15 +322,15 @@ Electric **refuses to serve a shape** over a table whose replica identity is not ``` This was checked empirically against `electricsql/electric:latest`, and it is -unconditional — it holds with and without a `where`, with and without a `columns` +unconditional. It holds with and without a `where`, with and without a `columns` projection, and with `(org_id, id)` present via `REPLICA IDENTITY USING INDEX`. So `0009`'s stated rationale is wrong in its details (`DEFAULT` keys deletes on the -primary key perfectly well, composite or not) but binding in its conclusion. What has -_not_ held up is its other claim — "these are low-write control-plane tables, so the -extra WAL volume is negligible." `FULL` writes the entire old row into the WAL on top -of the new one, and since Electric Cloud consumes the slot over a direct connection, -every one of those bytes is billed PlanetScale egress. +primary key perfectly well, composite or not) but binding in its conclusion. Its other +claim has _not_ held up: "these are low-write control-plane tables, so the extra WAL +volume is negligible." `FULL` writes the entire old row into the WAL on top of the new +one. Electric consumes the slot over a direct connection, so every one of those bytes +is billed PlanetScale egress. Because the per-write multiplier is not negotiable, **the only lever on a synced table is its write rate.** Before adding a hot writer to one, gate it: @@ -334,14 +343,14 @@ is its write rate.** Before adding a hot writer to one, gate it: authenticated request no longer writes on the hot path; - `alert_rule_states` has had `STATE_HEARTBEAT_MS` for the same reason. -Do not try to reclaim this by relaxing the replica identity — Electric will reject the -shape and the synced lists will fail to load outright, with no fetch fallback. +Do not try to reclaim this by relaxing the replica identity. Electric will reject the +shape and the synced lists will fail to load outright. ## Removing a synced table -Reverse order, and do all of it — a half-removal is what left four dead tables on +Reverse order, and do all of it. A half-removal is what left four dead tables on the slot. Drop the consumer + collection, drop the shape from the whitelist, then -a migration that drops the table from the publication **and** resets +write a migration that drops the table from the publication **and** resets `REPLICA IDENTITY DEFAULT` (FULL costs full-old-row WAL on every UPDATE/DELETE whether or not the table is published). Move the table from `SYNCED_TABLES` to `UNSYNCED_TABLES` in `migrations.test.ts`. See `0022_electric_publication_prune`. @@ -351,43 +360,42 @@ whether or not the table is published). Move the table from `SYNCED_TABLES` to **Done and verified** - Infra: docker `electric` + `wal_level=logical`; `0009_electric_publication` - (applies via both `drizzle-kit migrate` and the PGlite test path — see + (applies via both `drizzle-kit migrate` and the PGlite test path; see `packages/db/src/migrations.test.ts`), with later publication migrations for - wave-1 control-plane tables and `api_keys`. + wave-1 control-plane tables, `api_keys` and `investigations`. - Shape proxy with org-scoping + client-param pinning, extracted into the - standalone `apps/electric-sync` worker (`src/routes/shape.http.ts`; the - security-critical pinning is unit-tested in `src/routes/shape.test.ts`). -- txid capture: dashboards (all writes), alert rules (create/update/delete), and - error issues `heartbeat`/`assign`/`setSeverity`. + standalone `apps/electric-sync` worker. The security-critical pinning is + unit-tested in `src/shapes/registry.test.ts`, `src/shapes/request.test.ts` and + `src/routes/shape.http.test.ts`. +- txid capture: dashboards (all writes), alert rules (create/update/delete), alert + destinations, API keys, and error issues `heartbeat`/`assign`/`setSeverity`. - **`@maple/effect-db`** package (typecheck-clean) + **dashboards** collection refactored onto `createEffectCollection` + the `useDashboardStore` collection path, with writes migrated to `/v2/dashboards` and reconciled by returned txid; proven against a live Electric 1.6.2 instance locally. - **Alerts read consumers (Phase 6):** `collections/alerts.ts` (read-only - collections; client-side live-query join `alert_rules ⟕ alert_rule_states`); - `useAlertRulesList` / `useAlertIncidentsList` / `useAlertDestinationsList` hooks - read from the collections (writes stay on the typed endpoints — the shape stream - delivers results). The row→document mappers mirror the server's + collections; client-side live-query join `alert_rules ⟕ alert_rule_states`). + The `useAlertRulesList` / `useAlertIncidentsList` / `useAlertDestinationsList` + hooks read from the collections. Writes stay on the typed endpoints, and the shape + stream delivers results. The row→document mappers mirror the server's `rowToRuleDocument`/`rowToDestinationDocument` and are unit-tested (`collections/alerts.test.ts`). The parallel **errors** vertical (`collections/errors.ts`, `error_issues ⟕ actors ⟕ open_error_incidents`) was built and then reverted to the typed `/v2` reads; its tables were pruned from the publication by `0022`. - **API keys:** `collections/api-keys.ts` behind a column-restricted shape. +- **Investigations:** `collections/investigations.ts` behind the id-scoped + `investigation_v2` shape. - **Self-heal:** a `collection:schema-error` listener in `org-collections.ts` - recreates the org's collections (generation bump) so a post-deploy shape-schema + recreates the org's collections (generation bump), so a post-deploy shape-schema drift re-fetches instead of getting stuck. **Remaining (follow-ups)** - **Live smoke of alerts:** the mappers/joins/timestamps typecheck and unit-test green, but the end-to-end sync for this vertical still needs the docker-Electric - smoke (verify each list streams in scoped to the org and updates live after a - write) — same validation dashboards already passed. + smoke: verify each list streams in scoped to the org and updates live after a + write. Dashboards already passed the same validation. - **Row-volume check** before enabling any further list sync: confirm the per-org - row counts are bounded; if not, add an archival tick or keep terminal-state tabs + row counts are bounded. If not, add an archival tick or keep terminal-state tabs on paged effect-atom reads. -- **`alert_rules` write churn:** the scheduler's claim-lock CAS writes - `alert_rules.last_scheduled_at` every minute per enabled rule, churning that - shape. The fix is write-side (move the claim lock off the synced table) — a - column-subset shape is not the answer. diff --git a/docs/error-issue-lifecycle.md b/docs/error-issue-lifecycle.md index ea02716cc0..6c075ef6bc 100644 --- a/docs/error-issue-lifecycle.md +++ b/docs/error-issue-lifecycle.md @@ -10,7 +10,7 @@ This is the flow both humans and agents are meant to follow. If you are changing | Thing | Where it lives | What it is | | ------------- | --------------------------------------- | ------------------------------------------------------- | | Occurrence | ClickHouse / Tinybird | One exception, one row. Never mutated. | -| Fingerprint | `cityHash64(org, service, type, frame)` | The identity of an error _class_. | +| Fingerprint | `cityHash64(org, service, type, frames, message signature)` | The identity of an error _class_. | | Candidate | `error_fingerprint_candidates` | A fingerprint seen, but not yet worth a row of its own. | | **Issue** | `error_issues` | The durable, assignable record. One per fingerprint. | | Incident | `error_incidents` | A time-bounded flare-up _under_ an issue. | @@ -23,8 +23,8 @@ then `error.type` / `error.message`, then `Unknown Error`. This keeps existing f stable while separating previously unlabelled errors. The SQL in `error_events_mv` (`packages/domain/src/tinybird/materializations.ts`) owns this precedence. -The distinction that matters: **an incident is a flare-up, an issue is the bug**. An issue can flare -up ten times; it gets fixed once. +**An incident is a flare-up. An issue is the bug.** An issue can flare up ten times; it gets +fixed once. ## The flow @@ -73,7 +73,7 @@ up ten times; it gets fixed once. │ an occurrence from a build that postdates the fix ▼ ┌───────────┐ -│ regressed │ NOT `triage` — the issue remembers it was fixed once, so the +│ regressed │ NOT `triage`. The issue remembers it was fixed once, so the └───────────┘ next person doesn't fix the same bug a second time ``` @@ -82,33 +82,33 @@ and wakes back into `triage` when it expires. ## Who owns which edge -Three actors move issues, and the split is the whole design: +Three actors move issues: - **Humans and agents** move an issue through the states that record an _intention_: `triage`, `todo`, `in_progress`, `in_review`, `done`, `cancelled`, `wontfix`. -- **The errors tick** (every minute) owns `regressed`. It records an _observation_ — this error - fired from a build that was not running when it was resolved — and it would overwrite any - human claim to the contrary on its next pass. +- **The errors tick** (every minute) owns `regressed`. It records an _observation_: this error + fired from a build that was not running when it was resolved. It would overwrite any human + claim to the contrary on its next pass. - **The verification tick** (every minute) owns `verifying` and the exit from it. `regressed` and `verifying` are therefore in `MACHINE_OWNED_WORKFLOW_STATES`: legal edges in `WORKFLOW_TRANSITIONS` because the ticks travel them, but filtered out of every surface that lets -somebody _choose_ a state — the web state picker, the bulk bar, and the `transition_error_issue` +somebody _choose_ a state: the web state picker, the bulk bar, and the `transition_error_issue` MCP tool, which rejects them with an explanation. -The single source of truth for all of this is `WORKFLOW_TRANSITIONS` in +The source of truth is `WORKFLOW_TRANSITIONS` in `packages/domain/src/http/errors.ts`. The MCP tool description is rendered from it at registration time by `describeWorkflowTransitions()`, so the rules an agent is told can never drift from the rules the server enforces. ## Where the AI enters -There are exactly **three** places, and they do different jobs: +There are **three** places, each with a different job. ### 1. Auto-investigation on a new incident -Off by default; an admin opts in per org (`ai_triage_settings`). When an incident opens — first-seen -or regression — `maybeEnqueueTriage` starts an investigation, subject to a daily budget in runs and +Off by default; an admin opts in per org (`ai_triage_settings`). When an incident opens (first-seen +or regression), `maybeEnqueueTriage` starts an investigation, subject to a daily budget in runs and in model passes (`maxRunsPerDay`, `maxPassesPerDay`; one investigation is one pass). Three gates stand between an open incident and that pass, cheapest first, and every refusal lands @@ -126,8 +126,8 @@ on the `maybeStartInvestigation` span as `maple.investigation.start_result`: explains it. `evaluateIncidentGate` skips confident noise (`noise`) and confident matches (`covered_by_prior`), never anything the detector called `high` or `critical`, and only ever raises the severity the run is seeded with. A skipped noise incident labels its issue's - severity if nobody had, without escalating. No verdict — no binding, no token, a failed or - slow call — always reads as investigate; a broken classifier must not become a policy of + severity if nobody had, without escalating. No verdict (no binding, no token, a failed or + slow call) always reads as investigate. A broken classifier must not become a policy of dropping incidents. 3. **The daily budget**, judged by the severity the classifier settled on, so the reserve for `high`/`critical` is reachable by an incident the detector left unclassified. @@ -138,8 +138,8 @@ The run is one turn of the investigate agent on the investigation's `ChatSession it gathers the evidence, tests the rival explanations itself, and closes on `submit_diagnosis`. A pass that stops in prose or dies on a model error gets one close-out turn over its own tool transcript; a pass that still files nothing is marked `failed` with `no_diagnosis`. The report's -`ruledOut` is what records the explanations it tested and dropped, which is what makes a conclusion -readable a week later. (Until 2026-09 this was a planner → hypothesis lanes → validator workflow; +`ruledOut` records the explanations it tested and dropped, which makes a conclusion readable a +week later. (Until 2026-09 this was a planner → hypothesis lanes → validator workflow; the handoffs lost the evidence and most passes never reached a verdict.) The result lands back on the issue as an `ai_triage` timeline event plus an applied severity. @@ -152,17 +152,17 @@ An external coding agent claims an issue, reads its timeline, fixes the bug, and The lease is what keeps two agents off the same bug; it renews on every action and is dropped when the issue closes. -**The claim is taken by the work, not by a ceremony before it.** `propose_fix` and a transition to -`in_progress` both acquire the lease. This is not a convenience — it is the fix to the flow's worst -failure. For as long as claiming was a separate step an agent was merely _told_ to take, it was +**Doing the work takes the claim.** `propose_fix` and a transition to `in_progress` both acquire +the lease. This fixes the flow's worst failure. For as long as claiming was a separate step an agent was merely _told_ to take, it was never taken once: across 50 live issues in the internal org every `lease_holder` was null, and the most common MCP error in the org was `Illegal transition from 'triage' to 'in_review'`, which is `propose_fix` being rejected on an issue nobody had claimed. Agents responded by hand-walking `transition_error_issue` instead, which "worked" and left the lease empty. A guarantee that depends on an agent reading a rule is not a guarantee. -The agent-facing map of this flow lives in the `maple://instructions` MCP resource, and the -`issue-workflow.eval.ts` cases check a real model still picks these tools. Keep both in sync with +The agent-facing map of this flow lives in the `maple://instructions` MCP resource +(`apps/ai/src/mcp/resources/instructions.ts`), and the +`apps/ai/src/mcp/__evals__/issue-workflow.eval.ts` cases check a real model still picks these tools. Keep both in sync with this page. ### 3. Post-merge verification @@ -177,10 +177,10 @@ issue had been seen from at merge time. An occurrence from a build already in th client still in the wild; one from a build absent from it is the fix demonstrably not working, and that alone refutes the fix without asking an agent anything. -Only a _clean_ window goes to an agent, and it is asked the inverted question: not "what is wrong" -but "find anything that contradicts this fix holding". Hence the mapping in -`verdictFromInvestigationStatus`, which reads backwards until you hold that question — an agent that -_establishes_ a cause means `not_fixed`, and an agent that finds nothing means `verified`. +Only a _clean_ window goes to an agent, and it is asked the inverted question: "find anything +that contradicts this fix holding". That explains the mapping in +`verdictFromInvestigationStatus`, which reads backwards until you hold that question. An agent +that _establishes_ a cause means `not_fixed`, and an agent that finds nothing means `verified`. An inconclusive verdict re-arms exactly one longer window and then gives up, so an issue can never loop in verification without a human ever seeing it. @@ -209,9 +209,9 @@ loop in verification without a human ever seeing it. | Transitions, leases, timeline events | `packages/backend/src/services/errors/ErrorIssueWorkflowService.ts` | | The errors tick (incidents, regression) | `packages/backend/src/services/errors/error-tick-persistence.ts` | | Starting an investigation | `packages/backend/src/services/errors/ai-triage-enqueue.ts` | -| The gates in front of it | `investigation-gate.ts`, `IncidentClassifier.ts`, `apps/ai/src/triage/` | +| The gates in front of it | `packages/backend/src/services/errors/investigation-gate.ts`, `IncidentClassifier.ts`, `apps/ai/src/triage/` | | The investigate agent and its close-out | `apps/ai/src/chat/turn-runner.ts`, `apps/ai/src/chat/prompts.ts` | | Writing a diagnosis back | `packages/backend/src/services/errors/apply-diagnosis.ts` | | PR links and verification windows | `packages/backend/src/services/errors/IssueFixVerificationService.ts` | | The verification tick | `packages/backend/src/services/errors/FixVerificationTickService.ts` | -| What agents are told | `apps/api/src/mcp/resources/instructions.ts` | +| What agents are told | `apps/ai/src/mcp/resources/instructions.ts` | diff --git a/docs/eu-region-plan.md b/docs/eu-region-plan.md index 0439f6bef7..1e5e7f4cb8 100644 --- a/docs/eu-region-plan.md +++ b/docs/eu-region-plan.md @@ -17,7 +17,7 @@ there is no per-org routing anywhere, because each instance knows exactly one re deploy-time axis: `MapleRegion` in `packages/infra/src/aws/stage.ts` maps `eu` to eu-central-1 and its own CIDR, resource names take a region suffix with `us` unsuffixed, and the root `alchemy.run.ts` reads `MAPLE_REGION` and guards it against `AWS_REGION`. -- The Cloudflare half of the stack does not honour the region: `resolveWorkerName` and +- The Cloudflare half of the stack does not honor the region: `resolveWorkerName` and `resolveMapleDomains` in `packages/infra/src/cloudflare/stage.ts` know only stage, and `CLOUDFLARE_WORKER_PLACEMENT` is a constant `aws:us-east-1`. - Every Worker binds one `MAPLE_DB` Hyperdrive, one Tinybird host, one replay bucket. There is no @@ -52,8 +52,8 @@ callback, both on `api.maple.dev`, so an EU install lands on the US API (whose d connect session for it) and EU repositories' push and pull request events are delivered to the US. The sandbox clones the customer's repository, so it deploys per instance from day one: -`stageDeploysSandbox` already gates it to prd, and the EU deploy is prd with `MAPLE_REGION=eu`, so the only work is the region-suffixed name. Cloudflare's Sandbox container -has no jurisdiction setting, so it sits under the same best-effort placement as the Workers. +`stageDeploysSandbox` already gates it to prd, and the EU deploy is prd (stage `prd-eu`), so the +only work is the region-suffixed name. Cloudflare's Sandbox container has no jurisdiction setting, so it sits under the same best-effort placement as the Workers. ## Why `app.eu.maple.dev` and not a shared app @@ -77,9 +77,8 @@ across regions because there is no routing. connection, the two declared Hyperdrive configs (`originConnectionLimit: 8` each, raise with the cluster), the gateway through PSBouncer and a few admin slots, so 50 or more. The size first chosen gave 25 and Electric crash-looped on `too_many_connections`. That is all: the - deploy adopts - its `main` branch and applies the migrations, and declares on it the gateway's role, Electric's - replication role, one role per Worker consumer and a Hyperdrive config on each + deploy adopts its `main` branch, applies the migrations, and declares on it the gateway's role, + Electric's replication role, one role per Worker consumer and a Hyperdrive config on each (`declareMapleDb` in `alchemy.run.ts`). The same PlanetScale service token serves both instances; `prod-eu` carries it under the same names. @@ -122,7 +121,7 @@ across regions because there is no routing. Until the EU GitHub App exists, remove its keys from `prod-eu`: connecting then fails up front with "not configured" instead of sending the user through a flow that cannot finish. -## Phase 1. The stack honours the region on Cloudflare (built) +## Phase 1. The stack honors the region on Cloudflare (built) Built on the `worktree-eu-region` branch; `docs/infra.md` § Regions is the reference. @@ -234,7 +233,6 @@ Taken 2026-09-16: 2. Regional Services: not available; EU Workers run on best-effort placement, with the DO-hosted request path as the upgrade if a customer requires a hard execution guarantee. 3. AI features off on `eu` at launch. Turned back on 2026-09-25 over OpenRouter's EU endpoint (Phase 4). - 4. Sandbox deploys per instance from day one. ## Order and size diff --git a/docs/eventing-extension-guide.md b/docs/eventing-extension-guide.md index 1d3e15c368..94272c08f6 100644 --- a/docs/eventing-extension-guide.md +++ b/docs/eventing-extension-guide.md @@ -1,21 +1,18 @@ # Extending Maple's signal-to-event system -This guide walks through adding either of the two main eventing extensions: +This guide covers the two eventing extensions: - a **source adapter**, which turns an authenticated source payload into typed, normalized signals; or - a **semantic projector**, which turns matching signals into versioned factual events. -You can add one or both, depending on what the source already provides. +Add one or both, depending on what the source already provides. -First, one important naming point: an eventing extension is a compile-time -registered module. It is not a runtime-loaded plugin, and projection -configuration cannot introduce executable code. - -The host decides which adapters and projectors are installed. An operator can -then activate installed projectors through bounded, durable projection -revisions. +An eventing extension is a compile-time registered module, not a runtime-loaded +plugin. Projection configuration cannot introduce executable code. The host +decides which adapters and projectors are installed. An operator then activates +installed projectors through bounded, durable projection revisions. The contracts in `@maple/eventing-core` are host-neutral. Hosted Maple and Maple Local can install the same source and projector definitions while using @@ -57,17 +54,15 @@ The host is responsible for: - authorizing consumers, delivering events, retrying work, and performing side effects. -That last boundary matters: projectors never perform I/O. Sending a message, -calling a provider, mutating source state, or deciding what action to take -belongs to a consumer after the event has crossed the durable boundary. +Projectors never perform I/O. Sending a message, calling a provider, mutating +source state, or deciding what action to take belongs to a consumer, after the +event has crossed the durable boundary. ## Do you need a new source adapter? -A useful rule of thumb is to reuse an installed source kind whenever it already -preserves the fact you need. - -For example, when a semantic fact already arrives in an OTLP log, you will -usually need only: +Reuse an installed source kind whenever it already preserves the fact you need. +For example, when a semantic fact already arrives in an OTLP log, you usually +need only: 1. a new projector; and 2. projection configuration that selects the relevant logs. @@ -77,9 +72,9 @@ You do not need another OTLP decoder. Add a source adapter when the source has its own authenticated payload, identity contract, or field vocabulary. A provider webhook is the usual example. -A new adapter should not decode the same request a second time just for -eventing. The host should decode once, authenticate once, and pass the -already-decoded value to the adapter. +A new adapter must not decode the same request a second time for eventing. The +host decodes once, authenticates once, and passes the decoded value to the +adapter. Before writing the implementation, settle the following contracts: @@ -101,12 +96,10 @@ rather than pretend the source offers retry-safe identity or time. ## Complete example -The following example takes a build-system message that the host has already -authenticated and runtime-decoded, normalizes it, and projects successful builds +This provider-neutral example takes a build-system message that the host has +already authenticated and decoded, normalizes it, and projects successful builds into a versioned factual event. -The example is deliberately provider-neutral. - ### 1. Define the source and normalize its messages ```ts @@ -177,18 +170,15 @@ export const BUILD_SOURCE: SignalSourceAdapter = { Authentication is intentionally absent from `normalize`. The host must verify the message before calling the adapter. -Normalization must also be deterministic. Given the same source occurrence, it -should produce the same normalized signal. It must not call `Date.now()`, -generate a UUID, query a database, make a network request, or depend on mutable -host state. +Normalization must be deterministic: the same source occurrence produces the +same normalized signal. It must not call `Date.now()`, generate a UUID, query a +database, make a network request, or depend on mutable host state. ### 2. Define the projector's runtime codecs Projector configuration and projector output both cross trust boundaries, so -each needs a runtime decoder. - -The output decoder is especially important: it makes `dataschema` an enforced -contract rather than a hopeful annotation. +each needs a runtime decoder. The output decoder is what turns `dataschema` into +an enforced contract. ```ts import { type JsonValue, type SignalProjector } from "@maple/eventing-core" @@ -240,18 +230,17 @@ export const BUILD_COMPLETED_PROJECTOR: SignalProjector` structure, explicit facts and numbers, fair sourcing. | -| 7 | AI Overviews cut clicks to the #1 result by 58% (and accelerating) | The payoff of being _the cited source_ keeps rising — this is why GEO is worth doing at all, and why we optimize for the citation, not just the rank. | -| 9 | For the same query, AI Mode & AI Overviews agree 86% of the time but share only **13.7%** of citations | Cover the topic across multiple surfaces/pages (pillar, listicle, comparisons, feature pages) rather than betting on one URL — different engines cite different sources. | +| 7 | AI Overviews cut clicks to the #1 result by 58% (and accelerating) | The payoff of being _the cited source_ keeps rising. This is why GEO is worth doing at all, and why we optimize for the citation, not just the rank. | +| 9 | For the same query, AI Mode & AI Overviews agree 86% of the time but share only **13.7%** of citations | Cover the topic across multiple surfaces/pages (pillar, listicle, comparisons, feature pages) rather than betting on one URL. Different engines cite different sources. | ## Where things live - Listicle data model: `apps/landing/src/lib/best-observability-tools.ts` - Listicle page: `apps/landing/src/pages/best-open-source-observability-tools.astro` - Observability pillar: `apps/landing/src/pages/observability.astro` -- Homepage FAQ: `apps/landing/src/pages/index.astro` +- Homepage FAQ: `apps/landing/src/components/home/Home.astro` (rendered by `src/pages/index.astro`) - AI crawler allowlist: `apps/landing/public/robots.txt` - Sitemap freshness: `apps/landing/astro.config.mjs` -- Shared schema components: `SeoHead.astro`, `FAQ.astro`, `Breadcrumb.astro` +- Shared schema components: `apps/landing/src/components/{SeoHead,FAQ,Breadcrumb}.astro` ## Next steps (in priority order) -1. **YouTube (#6)** — produce a short demo/walkthrough, embed it, add `VideoObject` schema. Highest correlation of anything studied. -2. More listicles off the same data model — "Best Datadog alternatives", "Best OpenTelemetry backends", "Best open-source APM". +1. **YouTube (#6):** produce a short demo/walkthrough, embed it, add `VideoObject` schema. Highest correlation of anything studied. +2. More listicles off the same data model: "Best Datadog alternatives", "Best OpenTelemetry backends", "Best open-source APM". 3. A glossary / more informational pillars (distributed tracing, OTLP, spans, sampling) to widen informational-intent coverage (#8) and surface breadth (#9). diff --git a/docs/github-app-setup.md b/docs/github-app-setup.md index 43a514d6b3..37ce7cd592 100644 --- a/docs/github-app-setup.md +++ b/docs/github-app-setup.md @@ -1,19 +1,21 @@ # Setting up the GitHub App for the GitHub integration -Maple's GitHub integration is powered by a **GitHub App** that you create and own. When self-hosting Maple, you must register your own GitHub App, give Maple its credentials, and point its webhook and post-installation URLs back at your Maple deployment. Once that's done, your users get the self-serve "Connect GitHub" flow: they install your app on their account or organization, pick repositories, and Maple backfills and live-syncs their commit history. +Maple's GitHub integration runs on a **GitHub App** that you create and own. When self-hosting Maple, you register your own GitHub App, give Maple its credentials, and point its webhook and callback URLs at your Maple deployment. Your users then get the self-serve "Connect GitHub" flow: they install your app on their account or organization, pick repositories, and Maple backfills and live-syncs their commit history. -This guide is exhaustive and prescriptive — follow every step in order. +Follow every step in order. --- ## What you will end up with - A GitHub App registered under your GitHub account or organization. -- Five values copied into your Maple API environment: +- Six values copied into your Maple API environment, plus one optional one: - `GITHUB_APP_ID` - `GITHUB_APP_SLUG` - `GITHUB_APP_PRIVATE_KEY` - `GITHUB_APP_WEBHOOK_SECRET` + - `GITHUB_APP_CLIENT_ID` + - `GITHUB_APP_CLIENT_SECRET` - (optionally) `GITHUB_API_BASE_URL` for GitHub Enterprise Server - A working "Connect GitHub" button in **Integrations → GitHub** in your Maple dashboard. @@ -21,7 +23,7 @@ This guide is exhaustive and prescriptive — follow every step in order. ## Before you begin: know your public Maple URL -You need the **public base URL where your Maple API is reachable** — the same origin that serves the `/api/...` routes and the dashboard. Throughout this guide it is written as: +You need the **public base URL where your Maple API is reachable**: the origin that serves the `/api/...` routes. Throughout this guide it is written as: ``` https://YOUR_MAPLE_DOMAIN @@ -36,48 +38,48 @@ Two URLs derive from it, and you will paste both into GitHub later. Note them no | Webhook delivery | `https://YOUR_MAPLE_DOMAIN/api/integrations/github/webhook` | | User authorization / post-install redirect ("Callback URL") | `https://YOUR_MAPLE_DOMAIN/api/integrations/github/callback` | -> **One App per Maple instance.** Both URLs are registered on the App itself rather than sent per request, so an App serves exactly one deployment. Maple Cloud's EU instance runs its own App pointed at `https://api.eu.maple.dev`; reusing the US App sends EU installs and webhooks to the US API. +> **One App per Maple instance.** Both URLs are registered on the App itself rather than sent per request, so an App serves exactly one deployment. Maple Cloud's EU instance runs its own App pointed at `https://api.eu.maple.dev`. Reusing the US App would send EU installs and webhooks to the US API. -> These paths are fixed in Maple's code. Do not change them — Maple receives webhooks at `/api/integrations/github/webhook` and completes the install flow at `/api/integrations/github/callback`. +> These paths are fixed in Maple's code. Do not change them. Maple receives webhooks at `/api/integrations/github/webhook` and completes the install flow at `/api/integrations/github/callback`. --- -## Step 1 — Create a new GitHub App +## Step 1: Create a new GitHub App 1. Decide who owns the app: - **Personal account:** open - **Organization:** open `https://github.com/organizations/YOUR_ORG/settings/apps` (replace `YOUR_ORG`) 2. Click **New GitHub App**. (Direct link for a personal account: .) -3. You are now on the **Register new GitHub App** form. Leave it open and continue with the steps below — they map field-by-field to that form. +3. You are now on the **Register new GitHub App** form. Leave it open. The steps below map field by field to that form. --- -## Step 2 — Fill in the basic details +## Step 2: Fill in the basic details -| Field | What to enter | -| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **GitHub App name** | A unique, human-readable name, e.g. `Maple` or `Maple — example.com`. GitHub requires global uniqueness; if the name is taken, add a suffix. This name also determines your app **slug** (used later). | -| **Homepage URL** | `https://YOUR_MAPLE_DOMAIN` | +| Field | What to enter | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **GitHub App name** | A unique, human-readable name, e.g. `Maple` or `Maple (example.com)`. GitHub requires global uniqueness; if the name is taken, add a suffix. This name also determines your app **slug** (used later). | +| **Homepage URL** | `https://YOUR_MAPLE_DOMAIN` | --- -## Step 3 — Configure user authorization (Callback URL) +## Step 3: Configure user authorization (Callback URL) -This is what closes the loop after a user installs your app, so Maple can record the installation **and prove the installer actually owns it**. +This closes the loop after a user installs your app, so Maple can record the installation **and prove the installer actually owns it**. 1. Find the **Identifying and authorizing users** section. 2. In **Callback URL**, enter exactly: ``` https://YOUR_MAPLE_DOMAIN/api/integrations/github/callback ``` -3. Check **Request user authorization (OAuth) during installation**. **Do not skip this — it is a security control, not a convenience.** See the box below. +3. Check **Request user authorization (OAuth) during installation**. **Do not skip this. It is a security control.** See the box below. 4. Leave **Expire user authorization tokens** at its default (checked). -> **Why this is required (confused-deputy guard).** `installation_id` values are small, **enumerable** integers and aren't tied to the connect `state`. Without OAuth, anyone who starts a connect flow could submit _someone else's_ `installation_id` and bind another org's private repos to their own Maple org. With OAuth on, GitHub also returns a `code`; Maple exchanges it and calls `GET /user/installations` to confirm the user actually administers that installation before binding it. GitHub delivers `code`, `installation_id`, and `state` to the Callback URL above — there is no separate Setup URL. +> **Why this is required (confused-deputy guard).** `installation_id` values are small, **enumerable** integers and are not tied to the connect `state`. Without OAuth, anyone who starts a connect flow could submit _someone else's_ `installation_id` and bind another org's private repos to their own Maple org. With OAuth on, GitHub also returns a `code`. Maple exchanges it and calls `GET /user/installations` to confirm the user administers that installation before binding it. GitHub delivers `code`, `installation_id`, and `state` to the Callback URL above. There is no separate Setup URL. --- -## Step 4 — Configure the webhook +## Step 4: Configure the webhook 1. Find the **Webhook** section. 2. Check **Active**. @@ -89,51 +91,51 @@ This is what closes the loop after a user installs your app, so Maple can record ```bash openssl rand -hex 32 ``` - **Save this value** — you will set it as `GITHUB_APP_WEBHOOK_SECRET` in Step 9. Maple verifies every webhook with an HMAC-SHA256 signature (`X-Hub-Signature-256`) computed from this secret and **rejects deliveries that don't match**, so the value in GitHub and in Maple's environment must be identical. + **Save this value.** You will set it as `GITHUB_APP_WEBHOOK_SECRET` in Step 9. Maple verifies every webhook with an HMAC-SHA256 signature (`X-Hub-Signature-256`) computed from this secret and **rejects deliveries that don't match**, so the value in GitHub and in Maple's environment must be identical. --- -## Step 5 — Set permissions +## Step 5: Set permissions -Maple **reads** commit, branch, and source-file data, and **writes** three things on a pull request: a check run, one summary comment (edited in place on later pushes), and a comment-only review with inline notes, and only on repositories where a Maple admin has switched on the pull request observability review. It never pushes commits or changes repository settings. Read-only source access lets Maple's investigation agent correlate observed failures with the exact deployed revision when telemetry includes repository and commit attributes. +Maple **reads** commit, branch, and source-file data. Read-only source access lets Maple's investigation agent correlate observed failures with the exact deployed revision when telemetry includes repository and commit attributes. + +On repositories where a Maple admin has switched on the pull request review, Maple also **writes** to pull requests: a check run, one summary comment (edited in place on later pushes), a comment-only review with inline notes, and replies when someone mentions `@maple` in a pull request comment. With **Contents: Read and write**, `@maple fix` can also commit the edits it proposes as one fast-forward commit on the pull request's own branch (never a fork, never the default branch, never forced). Maple never changes repository settings. 1. Find **Permissions → Repository permissions**. 2. Set the following, leaving every other permission at **No access**: | Repository permission | Access level | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| **Contents** | **Read-only** | +| **Contents** | **Read-only**, or **Read and write** to allow `@maple fix` commits | | **Pull requests** | **Read and write** (reads the diff; posts the review as a comment, never an approval or block) | | **Checks** | **Read and write** (posts the `Maple / observability` check run on the head commit; without it the review is still posted as a comment) | | **Metadata** | **Read-only** (GitHub pre-selects this and it is mandatory) | -> **Upgrading an existing App?** Adding a permission to an App that is already installed makes GitHub ask every installation's admin to accept the new permissions. Until they do, Maple still records each review but cannot post it: the review's row keeps a publish error and nothing appears on the pull request. The fix is to accept the permission request on GitHub (Settings → Applications → the App → Review request). +> **Upgrading an existing App?** Adding a permission to an App that is already installed makes GitHub ask every installation's admin to accept the new permissions. Until they do, Maple still records each review but cannot post it: the review's row keeps a publish error and nothing appears on the pull request. To fix it, accept the permission request on GitHub (Settings → Applications → the App → Review request). 3. Leave **Organization permissions** and **Account permissions** entirely at **No access**. --- -## Step 6 — Subscribe to webhook events +## Step 6: Subscribe to webhook events -1. Find the **Subscribe to events** section (directly below permissions). The events listed here depend on the permissions you set in Step 5; if an event is missing, re-check that **Contents** is **Read-only**. +1. Find the **Subscribe to events** section (directly below permissions). The events listed here depend on the permissions you set in Step 5. If an event is missing, re-check the permissions. 2. Check exactly these events: -| Event to check | Why Maple needs it | -| ----------------------- | ----------------------------------------------------------------------------------------------------- | -| **Create** | Branch or tag created — detect newly created branches. | -| **Delete** | Branch or tag deleted — detect deleted branches. | -| **Push** | Live-sync new commits on tracked branches; reconcile force-pushes. | -| **Pull request** | Link pull requests to error issues, and start the observability review on repositories that opted in. | -| **Release** | Sync releases. | -| **Repository** | Track repositories being created, renamed, deleted, or changing visibility. | -| **Meta** | Notify Maple when the GitHub App itself is deleted. | -| **Installation target** | Notify Maple when an installation's target account is renamed or transferred. | +| Event to check | Why Maple needs it | +| ------------------------------- | ----------------------------------------------------------------------------------------------------- | +| **Create** | Detect newly created branches. | +| **Delete** | Detect deleted branches. | +| **Push** | Live-sync new commits on tracked branches; reconcile force-pushes. | +| **Pull request** | Link pull requests to error issues, and start the observability review on repositories that opted in. | +| **Issue comment** | Answer `@maple` mentions in a pull request's conversation. | +| **Pull request review comment** | Answer `@maple` mentions in a pull request's review threads. | -> You do **not** need to (and cannot) subscribe to `installation` or `installation_repositories` here — GitHub always delivers those lifecycle events automatically once the app is installed, and Maple handles them. The `ping` event GitHub sends on setup is accepted as a harmless no-op. +> You do **not** need to (and cannot) subscribe to `installation` or `installation_repositories` here. GitHub always delivers those lifecycle events once the app is installed, and Maple handles them. The `ping` event GitHub sends on setup, and any other event, is accepted as a no-op. --- -## Step 7 — Choose where the app can be installed +## Step 7: Choose where the app can be installed Under **Where can this GitHub App be installed?**: @@ -144,7 +146,7 @@ Then click **Create GitHub App** at the bottom of the form. --- -## Step 8 — Collect the App ID, slug, and private key +## Step 8: Collect the App ID, slug, client credentials, and private key You are now on your new app's settings page (`https://github.com/settings/apps/`). @@ -163,19 +165,19 @@ Look at the page URL: `https://github.com/settings/apps/`. The final These are required for the user-authorization (OAuth) confused-deputy guard you enabled in Step 3. 1. On the **General** tab, find **Client ID** (a string like `Iv1.abc123...` or `Iv23li...`). This is your `GITHUB_APP_CLIENT_ID`. -2. Just below it, under **Client secrets**, click **Generate a new client secret**. Copy the generated value immediately — **GitHub shows it only once**. This is your `GITHUB_APP_CLIENT_SECRET`. +2. Just below it, under **Client secrets**, click **Generate a new client secret**. Copy the value immediately. **GitHub shows it only once.** This is your `GITHUB_APP_CLIENT_SECRET`. ### 8d. Private key 1. Scroll to the bottom of the **General** tab to **Private keys**. -2. Click **Generate a private key**. GitHub downloads a `*.pem` file — **this is the only time you can get it**, so store it securely. +2. Click **Generate a private key**. GitHub downloads a `*.pem` file. **This is the only time you can get it**, so store it securely. 3. The file contents (including the `-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines) are your `GITHUB_APP_PRIVATE_KEY`. --- -## Step 9 — Set the environment variables on your Maple API +## Step 9: Set the environment variables on your Maple API -Set these on the **Maple API** service (`apps/api`). For local development, put them in your root `.env` file; for a deployed environment, use your secrets manager / platform environment configuration. +Set these on the **Maple API** service (`apps/api`). For local development, put them in the root `.env.local` file (what `bun dev` reads). For a deployed environment, use your secrets manager or platform environment configuration. ```bash # Required @@ -187,19 +189,19 @@ GITHUB_APP_WEBHOOK_SECRET= GITHUB_APP_CLIENT_ID=Iv23liAbC123... GITHUB_APP_CLIENT_SECRET= -# Required — the full PEM contents from Step 8d, newlines preserved +# Required: the full PEM contents from Step 8d, newlines preserved GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA... ...rest of the key... -----END RSA PRIVATE KEY-----" ``` -> These are **mandatory**, not optional. The connect flow **fails closed**: when a user connects an installation that isn't already linked to their org, Maple requires the OAuth `code` to prove they administer it. If `GITHUB_APP_CLIENT_ID` / `GITHUB_APP_CLIENT_SECRET` are missing, or **Request user authorization (OAuth) during installation** (Step 3) is unchecked, GitHub sends no `code`, the ownership check can't run, and the connect is **rejected** — no installation is linked. (A same-org _reconnect_ of an already-linked installation still works without a fresh `code`.) +> These are **mandatory**. The connect flow **fails closed**: when a user connects an installation that isn't already linked to their org, Maple requires the OAuth `code` to prove they administer it. If `GITHUB_APP_CLIENT_ID` / `GITHUB_APP_CLIENT_SECRET` are missing, or **Request user authorization (OAuth) during installation** (Step 3) is unchecked, GitHub sends no `code`, the ownership check can't run, and the connect is **rejected**. No installation is linked. (A same-org _reconnect_ of an already-linked installation still works without a fresh `code`.) Notes on the private key: - **Preserve the line breaks.** The value must be the literal multi-line PEM. In a `.env` file, wrap it in double quotes as shown so the newlines are kept. -- If your secrets platform cannot store multi-line values, paste the PEM with literal `\n` escapes on a single line and ensure your platform un-escapes them — but multi-line is preferred. +- If your secrets platform cannot store multi-line values, paste the PEM with literal `\n` escapes on a single line and make sure your platform un-escapes them. Multi-line is preferred. ### Optional variables @@ -207,19 +209,19 @@ Notes on the private key: | --------------------- | ------------------------------------------------------------------------------------------------------------------- | | `GITHUB_API_BASE_URL` | Only for **GitHub Enterprise Server**. Set to `https://YOUR_GHE_HOST/api/v3`. Defaults to `https://api.github.com`. | -> Using GitHub Enterprise Server? In addition to `GITHUB_API_BASE_URL`, create the app under your GHE instance (`https://YOUR_GHE_HOST/settings/apps/new`) instead of github.com, and use your GHE host in the homepage/Setup/webhook URLs where appropriate. +> Using GitHub Enterprise Server? In addition to `GITHUB_API_BASE_URL`, create the app under your GHE instance (`https://YOUR_GHE_HOST/settings/apps/new`) instead of github.com. -After setting the variables, **restart the Maple API** so they take effect. Maple validates `GITHUB_APP_ID` and `GITHUB_APP_PRIVATE_KEY` lazily — if either is missing, the integration returns a "GitHub App is not configured" error rather than crashing the server. +After setting the variables, **restart the Maple API** so they take effect. Maple validates `GITHUB_APP_ID` and `GITHUB_APP_PRIVATE_KEY` lazily. If either is missing, the integration returns a "GitHub App is not configured" error rather than crashing the server. --- -## Step 10 — Verify the setup +## Step 10: Verify the setup -1. **Confirm the dashboard sees the app.** Open **Integrations → GitHub** in your Maple dashboard. The **Connect GitHub** button should be active. (If you get "GitHub App is not configured (set `GITHUB_APP_SLUG`)", the slug env var is missing or the API wasn't restarted.) +1. **Confirm the dashboard sees the app.** Open **Integrations → GitHub** in your Maple dashboard. The **Connect GitHub** button should be active. (If you get "GitHub App is not configured (set GITHUB_APP_SLUG)", the slug env var is missing or the API wasn't restarted.) 2. **Run the connect flow.** Click **Connect GitHub**. A popup should open `https://github.com/apps//installations/new`. Install the app on a test account and pick at least one repository. -3. **Confirm the callback completes.** After installing — and authorizing when GitHub prompts — the popup should close and the card should flip to **Connected**, then move repositories from **Queued → Syncing → Synced**. If it doesn't, your **Callback URL** (Step 3) is the first thing to recheck. +3. **Confirm the callback completes.** After installing (and authorizing when GitHub prompts), the popup should close and the card should flip to **Connected**, then move repositories from **Queued → Syncing → Synced**. If it doesn't, recheck your **Callback URL** (Step 3) first. 4. **Confirm webhooks are signed correctly.** In your app's settings, open the **Advanced** tab → **Recent Deliveries**. The initial `ping` and the install events should show a **`200`** response. A non-`200` here means the webhook URL is wrong or the **webhook secret in GitHub doesn't match `GITHUB_APP_WEBHOOK_SECRET`**. -5. **Confirm live sync.** Push a commit to a tracked branch in the test repository, then check **Recent Deliveries** for a `push` event returning `200`, and confirm the commit appears in Maple. +5. **Confirm live sync.** Push a commit to a tracked branch in the test repository, check **Recent Deliveries** for a `push` event returning `200`, and confirm the commit appears in Maple. --- @@ -229,12 +231,12 @@ After setting the variables, **restart the Maple API** so they take effect. Mapl | Variable | Required | Default | Description | | --------------------------- | -------- | ------------------------ | ---------------------------------------------------------------------------------- | -| `GITHUB_APP_ID` | Yes | — | Numeric App ID from the app's General settings. | -| `GITHUB_APP_SLUG` | Yes | — | URL slug of the app; used to build the install link. | -| `GITHUB_APP_PRIVATE_KEY` | Yes | — | Full PEM private key; used to mint app JWTs (RS256) and installation tokens. | -| `GITHUB_APP_WEBHOOK_SECRET` | Yes | — | Shared secret for HMAC-SHA256 verification of incoming webhooks. | -| `GITHUB_APP_CLIENT_ID` | Yes | — | OAuth client ID; used by the confused-deputy guard to exchange the install `code`. | -| `GITHUB_APP_CLIENT_SECRET` | Yes | — | OAuth client secret; paired with the client ID for the `code` exchange. | +| `GITHUB_APP_ID` | Yes | | Numeric App ID from the app's General settings. | +| `GITHUB_APP_SLUG` | Yes | | URL slug of the app; used to build the install link. | +| `GITHUB_APP_PRIVATE_KEY` | Yes | | Full PEM private key; used to mint app JWTs (RS256) and installation tokens. | +| `GITHUB_APP_WEBHOOK_SECRET` | Yes | | Shared secret for HMAC-SHA256 verification of incoming webhooks. | +| `GITHUB_APP_CLIENT_ID` | Yes | | OAuth client ID; used by the confused-deputy guard to exchange the install `code`. | +| `GITHUB_APP_CLIENT_SECRET` | Yes | | OAuth client secret; paired with the client ID for the `code` exchange. | | `GITHUB_API_BASE_URL` | No | `https://api.github.com` | Override for GitHub Enterprise Server (`https://HOST/api/v3`). | ### Fixed URLs (set these in GitHub) @@ -247,26 +249,26 @@ After setting the variables, **restart the Maple API** so they take effect. Mapl ### Permissions -| Permission | Access | -| -------------------------- | --------------------- | -| Repository → Contents | Read-only | -| Repository → Pull requests | Read and write | -| Repository → Checks | Read and write | -| Repository → Metadata | Read-only (mandatory) | +| Permission | Access | +| -------------------------- | --------------------------------------------------- | +| Repository → Contents | Read-only (Read and write to allow `@maple fix`) | +| Repository → Pull requests | Read and write | +| Repository → Checks | Read and write | +| Repository → Metadata | Read-only (mandatory) | ### Subscribed webhook events -`Create`, `Delete`, `Push`, `Pull request`, `Release`, `Repository`, `Meta`, `Installation target`. (`installation` and `installation_repositories` are delivered automatically and need no subscription.) +`Create`, `Delete`, `Push`, `Pull request`, `Issue comment`, `Pull request review comment`. (`installation` and `installation_repositories` are delivered automatically and need no subscription.) --- ## Troubleshooting - **"GitHub App is not configured" in the dashboard.** A required env var (`GITHUB_APP_SLUG`, `GITHUB_APP_ID`, or `GITHUB_APP_PRIVATE_KEY`) is missing, or the API wasn't restarted after setting it. -- **Install succeeds on GitHub but never connects in Maple.** The **Callback URL** is missing/incorrect, or **Request user authorization (OAuth) during installation** is unchecked (or `GITHUB_APP_CLIENT_ID` / `GITHUB_APP_CLIENT_SECRET` are unset). The flow fails closed: with no OAuth `code`, Maple rejects a new installation binding with "Could not verify you own this GitHub installation." Fix Step 3 / Step 8c and reinstall. -- **Webhook deliveries return `401`/signature errors.** The webhook secret in GitHub does not match `GITHUB_APP_WEBHOOK_SECRET`. Regenerate, paste it into both places, and restart the API. +- **Install succeeds on GitHub but never connects in Maple.** The **Callback URL** is missing or incorrect, or **Request user authorization (OAuth) during installation** is unchecked, or `GITHUB_APP_CLIENT_ID` / `GITHUB_APP_CLIENT_SECRET` are unset. The flow fails closed: with no OAuth `code`, Maple rejects a new installation binding with "Could not verify you own this GitHub installation." Fix Step 3 / Step 8c and reinstall. +- **Webhook deliveries return `401` or signature errors.** The webhook secret in GitHub does not match `GITHUB_APP_WEBHOOK_SECRET`. Regenerate it, paste it into both places, and restart the API. - **Webhook deliveries return `404`.** The Webhook URL is wrong, or your Maple API isn't publicly reachable at `https://YOUR_MAPLE_DOMAIN`. - **`Connect GitHub` opens a GitHub 404.** `GITHUB_APP_SLUG` does not match the app's real slug (the last path segment of `https://github.com/settings/apps/`). - **Private key errors when minting tokens.** The PEM was truncated or had its newlines stripped. Re-paste the full key, including the BEGIN/END lines, preserving line breaks. -For the end-user-facing description of what the integration does once configured, see [the GitHub integration doc](../apps/landing/src/content/docs/integrations/github.md). +For the end-user description of what the integration does once configured, see [the GitHub integration doc](../apps/landing/src/content/docs/integrations/github.md). diff --git a/docs/http-api-migration.md b/docs/http-api-migration.md index 0143f16fab..8884793a0c 100644 --- a/docs/http-api-migration.md +++ b/docs/http-api-migration.md @@ -15,7 +15,7 @@ This document decides where the remaining `/api/...` surface belongs and defines Two of those carry notes worth keeping: -- **PlanetScale** was removed against gate condition 3, deliberately. Telemetry for 2026-07-15 → 2026-08-14 showed exactly one call — `GET /api/integrations/planetscale/status` on 2026-08-05 — so the surface was not at zero. `/v2/integrations/planetscale` had been live and the v1 operations OpenAPI-deprecated for long enough that accepting one broken caller beat carrying the group. The OAuth callback and the webhook receiver stay: PlanetScale stores those URLs on its side. +- **PlanetScale** was removed against gate condition 3, deliberately. Telemetry for 2026-07-15 → 2026-08-14 showed exactly one call (`GET /api/integrations/planetscale/status` on 2026-08-05), so the surface was not at zero. `/v2/integrations/planetscale` had been live and the v1 operations OpenAPI-deprecated for long enough that accepting one broken caller beat carrying the group. The OAuth callback and the webhook receiver stay: PlanetScale stores those URLs on its side. - **`onboarding`** left `OnboardingService` without a consumer until the onboarding reward checklist (`/v2/onboarding/checklist`, `OnboardingChecklistService`) became one: it writes `org_onboarding_state` again (creation time on first read, `reward_claimed_at` on claim). `first_data_received_at` still has no writer, so setup-audit's no-data gate remains stale for orgs created after the v1 route went away. `sessionReplays`: the remaining four web operations now use v2 (2026-09-07). Search and retrieve expose the visitor/acquisition dimensions, full detail also preserves user traits, host, language and last activity, and transcript events preserve custom properties. Search carries opaque cursors through infinite scrolling; transcripts and reverse trace correlation collect every page. Manifest and bounded playback events were already v2. The four v1 operations remain mounted and are OpenAPI-deprecated while the external-usage retirement gate is pending. Deployment/sunset dates and a verified zero-traffic window have not been established by this local change. @@ -33,7 +33,7 @@ These v1 groups mix public resource operations with private orchestration. Split | `errors` | Issue events, comments, state transitions, assignee, and severity under `/v2/error_issues/{id}/...`. | Agent registration, claim, heartbeat, release, escalation-policy evaluation, and other worker coordination. | | `organizations` + `orgClickHouseSettings` | Organization update/delete and customer-managed ClickHouse configuration as organization subresources, with explicit admin scopes. | Setup wizards or UI-only probes that merely coordinate several public operations. | | `integrations` | Promote a provider only when a public resource or supported external automation needs it. Chat connectors and PlanetScale already meet that bar. | Cloudflare, GitHub, and Hazel dashboard control surfaces remain private until public demand exists. Split providers into independent contracts so one provider does not block retirement of another. | -| `warehouse` (**removed**), `observability` | Keep the existing stable telemetry resources: traces, logs, metrics, services, and service map. Add a specific public resource endpoint only after its request and response shape is stable. | Raw SQL, generic query documents, arbitrary warehouse execution, attribute/facet discovery used only by dashboard builders, infrastructure drill-down helpers, and provider-specific chart queries. (`queryEngine` is **done** — the whole group moved to `/internal/query-engine`, so nothing was left to split.) | +| `warehouse` (**removed**), `observability` | Keep the existing stable telemetry resources: traces, logs, metrics, services, and service map. Add a specific public resource endpoint only after its request and response shape is stable. | Raw SQL, generic query documents, arbitrary warehouse execution, attribute/facet discovery used only by dashboard builders, infrastructure drill-down helpers, and provider-specific chart queries. (`queryEngine` is **done**: the whole group moved to `/internal/query-engine`, so nothing was left to split.) | ### v2 telemetry gaps blocking full CLI parity @@ -44,7 +44,7 @@ Rebuilding the CLI's remote mode on v2 surfaced capabilities local mode has and - **No span-level search.** Search returns root-based `V2TraceSummary`; there is no way to list spans matching a name, and its `span_name` filter matches exactly where the CLI matches a substring. - **Breakdown returns one aggregation per request.** A combined ranking (count + latency + error rate) needs N calls and cannot be ordered server-side by a composite. - ~~**No fingerprint → issue lookup.**~~ Closed: `/v2/error_issues` takes a `fingerprint_hash` filter, and the CLI's `maple error ` runs on it remotely. -- **No exception-type aggregate.** `/v2/error_issues` holds one triage object per fingerprint — it covers only fingerprints a sweep has turned into issues, and cannot say how many services an error spans, which `maple errors` prints. +- **No exception-type aggregate.** `/v2/error_issues` holds one triage object per fingerprint. It covers only fingerprints a sweep has turned into issues, and cannot say how many services an error spans, which `maple errors` prints. - **No window comparison.** Nothing corresponds to `service_overview_compare`. - **Offset pagination.** v2 lists seek by opaque cursor and cap at 100 rows, so `--offset` cannot be honoured. @@ -56,27 +56,27 @@ The following v1 groups are dashboard workflows or protocol surfaces. Do not por | v1 group | destination | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `billing` | **Done** — `/internal/billing`. `billingPublic` (`listPlans`) stayed on `MapleApi` at `/api/billing/plans`: it is served unauthenticated so a Clerk token-settle gap renders prices instead of a 401, which session-only authorization would defeat. Provider webhooks remain raw signed receivers. | -| `onboarding` | Removed outright rather than moved — nothing called it. See the note above. | -| `demo` | **Done** — `/internal/demo`. | -| `chat` | **Done** — `/internal/chat`. The `apply` mutation moved; the four raw chat-session streaming routes stayed where they were. They are a separate top-level `HttpRouter`, not a `MapleApi` group, and they resolve tenancy through `resolveHttpMcpTenant` (which accepts API keys) rather than the HttpApi middleware — so the split cost nothing to make. | -| `digest` | **Done** — `/internal/digest`. Revisit if a public notification-subscription resource is ever designed. | -| `aiTriage` | **Done** — `/internal/ai-triage`. | +| `billing` | **Done**: `/internal/billing`. `billingPublic` (`listPlans`) stayed on `MapleApi` at `/api/billing/plans`: it is served unauthenticated so a Clerk token-settle gap renders prices instead of a 401, which session-only authorization would defeat. Provider webhooks remain raw signed receivers. | +| `onboarding` | Removed outright instead of moved; nothing called it. See the note above. | +| `demo` | **Done**: `/internal/demo`. | +| `chat` | **Done**: `/internal/chat`. The `apply` mutation moved; the four raw chat-session streaming routes stayed where they were. They are a separate top-level `HttpRouter`, not a `MapleApi` group, and they resolve tenancy through `resolveHttpMcpTenant` (which accepts API keys) instead of the HttpApi middleware, so the split cost nothing to make. | +| `digest` | **Done**: `/internal/digest`. Revisit if a public notification-subscription resource is ever designed. | +| `aiTriage` | **Done**: `/internal/ai-triage`. | | `auth` and `authPublic` | Keep standards-driven CLI/MCP/OAuth/JWT exchange routes version-neutral; they are authentication protocols, not v2 resources. | ## The one unauthenticated v2 group -`sharePublic` (`POST /v2/share/resolve`, `POST /v2/share/widget-data`) is on `MapleApiV2` **without** `AuthorizationV2` — the only group there that is. It is the viewer half of dashboard share links: the token in the request body is the credential, and the whole proposition is that the link resolves for someone with no Maple account, so requiring a bearer would mean it only worked for people who did not need it. Share _management_ (`/v2/dashboards/{id}/share`, and the widget-scoped twins) is ordinary authenticated v2 surface and carries the `dashboards` scope family. +`sharePublic` (`packages/domain/src/http/v2/share.ts`: `POST /v2/share/resolve`, `/widget-data`, `/og-meta`, `/og-card`, `/alert-chart`, `/chat-chart`) is the only group on `MapleApiV2` **without** `AuthorizationV2`. It is the viewer half of dashboard share links: the token in the request body is the credential, and the whole proposition is that the link resolves for someone with no Maple account, so requiring a bearer would mean it only worked for people who did not need it. Share _management_ (`/v2/dashboards/{id}/share`, and the widget-scoped twins) is ordinary authenticated v2 surface and carries the `dashboards` scope family. Three consequences, all deliberate: - **No scope.** `requiredScopeForRoute` would derive the family `share` from the path, but it is only called by `ApiAuthorizationV2Layer`, which these routes never run. There is no `share:read` scope and nothing issues one. -- **`security: []` and no `401`** in the OpenAPI spec. `openapi.test.ts` exempts the two operations by operationId through `PUBLIC_OPERATION_IDS` — an allowlist, so a third public operation cannot appear without someone editing it. Every other operation guarantee still applies to them. +- **`security: []` and no `401`** in the OpenAPI spec. `openapi.test.ts` exempts these operations by operationId through `PUBLIC_OPERATION_IDS`. It is an allowlist, so a new public operation cannot appear without someone editing it. Every other operation guarantee still applies to them. - **The v2 rate limiter never runs.** These routes carry their own per-token and per-IP limiter in the handler; it is the only one on the path, not a supplement. -Prefer this shape over a new unauthenticated group. If a future surface needs anonymous access, weigh `/api/…` (as `billingPublic` did) first — the exemption above is narrow on purpose. +Prefer this shape over a new unauthenticated group. If a future surface needs anonymous access, weigh `/api/…` (as `billingPublic` did) first. The exemption above is narrow on purpose. -Moving a group to `/internal` swaps `Authorization` for `SessionAuthorization`, which refuses API-key-shaped bearers. That is the intended narrowing, and telemetry showed no API-key traffic to any of these paths — but it means the move is a breaking change for any API-key caller, not a transparent reroute. +Moving a group to `/internal` swaps `Authorization` for `SessionAuthorization`, which refuses API-key-shaped bearers. That is the intended narrowing, and telemetry showed no API-key traffic to any of these paths. It still makes the move a breaking change for any API-key caller. The frontend halves of that move are easy to get wrong in ways nothing type-checks: @@ -107,11 +107,11 @@ If external traffic prevents removal, keep the compatibility adapter thin over t ## Execution order 1. **Boundary consistency (this change):** apply API-wide v1 request-validation and defect middleware; apply the same v2 middleware once at `MapleApiV2`; distinguish request-decode failures from server-side response drift; define one exhaustive public policy map per domain error union and preserve every typed error's semantic tag through the v2 envelope. -2. **Deprecate complete duplicates:** done for `apiKeys`, `ingestKeys`, `ingestAttributeMappings`, `recommendationIssues`, `scrapeTargets`, and `investigations` — per-operation telemetry showed no production caller (the only August 2026 requests were manual `curl` probes), so the groups were deleted outright rather than deprecated. The PlanetScale v1 operations followed in the 2026-08-14 sweep, against the gate — see the note above. +2. **Deprecate complete duplicates:** done for `apiKeys`, `ingestKeys`, `ingestAttributeMappings`, `recommendationIssues`, `scrapeTargets`, and `investigations`. Per-operation telemetry showed no production caller (the only August 2026 requests were manual `curl` probes), so the groups were deleted outright instead of deprecated. The PlanetScale v1 operations followed in the 2026-08-14 sweep, against the gate (see the note above). 3. **Finish near-complete resources:** done for dashboards and anomalies, both now deleted. Session replay web callers now use v2; its deprecated v1 routes await external-usage evidence. -4. **Build the internal tier:** done for `queryEngine`, which now serves from `MapleInternalApi` at `/internal/query-engine` behind `SessionAuthorization` — telemetry showed one API-key call across all 62 endpoints in 30 days, so the group was moved wholesale rather than split. Billing, demo, digest, and AI triage followed on 2026-08-14. Chat apply followed on 2026-08-15. Still to move: error-agent coordination. +4. **Build the internal tier:** done for `queryEngine`, which now serves from `MapleInternalApi` at `/internal/query-engine` behind `SessionAuthorization`. Telemetry showed one API-key call across all 62 endpoints in 30 days, so the group was moved wholesale instead of split. Billing, demo, digest, and AI triage followed on 2026-08-14. Chat apply followed on 2026-08-15. Still to move: error-agent coordination. - The `warehouse` group (`POST /api/tinybird/query`) was **deleted** in the same change rather than moved. Its only caller was the CLI's remote mode, which had been broken since 6aed357503 renamed the payload field `pipe` → `pipeName` without updating `apps/cli/src/core/remote-executor.ts` — every remote command failed payload decode, and the executor's test stubbed `fetch` without asserting the body, so nothing caught it. Remote mode was rebuilt on the public v2 telemetry resources instead of repairing an endpoint that let clients name server-side pipe identifiers. + The `warehouse` group (`POST /api/tinybird/query`) was **deleted** in the same change rather than moved. Its only caller was the CLI's remote mode, which had been broken since 6aed357503 renamed the payload field `pipe` → `pipeName` without updating `apps/cli/src/core/remote-executor.ts`. Every remote command failed payload decode, and the executor's test stubbed `fetch` without asserting the body, so nothing caught it. Remote mode was rebuilt on the public v2 telemetry resources instead of repairing an endpoint that let clients name server-side pipe identifiers. 5. **Split mixed v1 groups:** separate `errors`, provider integrations, and query/warehouse operations so public promotions and private RPC moves can be retired independently. 6. **Delete by evidence:** remove each empty v1 group as soon as its retirement gate passes; do not wait for every v1 group to be ready. diff --git a/docs/infra.md b/docs/infra.md index 567bef92e4..0f26fc80b7 100644 --- a/docs/infra.md +++ b/docs/infra.md @@ -1,24 +1,23 @@ # Infrastructure notes -Background for the Alchemy stack (`alchemy.run.ts` + `apps/*/alchemy.run.ts` + -`packages/infra`). The stack files keep the rationale a reader needs **in order not to -break the code**; the history behind those decisions lives here, so the config stays -readable and the incidents stay findable. +Background for the Alchemy stack (`alchemy.run.ts`, the per-app Worker modules and +factories, and `packages/infra`). The stack files keep the rationale a reader needs **in +order not to break the code**. The history behind those decisions lives here, so the config +stays readable and the incidents stay findable. -If you are about to delete a comment in a stack file because "the history is in git" — -put it here instead. Git blame does not survive a refactor of the line it annotates. +If you are about to delete a comment in a stack file because "the history is in git", put +it here instead. Git blame does not survive a refactor of the line it annotates. ## The AI Worker (`maple-ai`) -`apps/ai` hosts every agent surface: the MCP transport and its tools, and the -chat `ChatSession` Durable Object, which also runs every investigation's -autonomous pass. `apps/api` -keeps the hostname and forwards `/mcp`, `/api/chat/*` and `/internal/chat/*` to -it over a service binding, so the OAuth issuer and the RFC 8707 resource -identifiers never move off api's origin. +`apps/ai` hosts every agent surface: the MCP transport and its tools, and the `ChatSession` +Durable Object, which also runs every investigation's autonomous pass. `apps/api` keeps the +hostname and forwards `/mcp`, `/api/chat/*` and `/internal/chat/*` to it over a service +binding, so the OAuth issuer and the RFC 8707 resource identifiers never move off api's +origin. -Measured before committing to the split (rolldown, unminified, same tree), -dropping the MCP registry, the chat routes and the two hosted classes from api: +Measured before committing to the split (rolldown, unminified, same tree), dropping the MCP +registry, the chat routes and the two hosted classes from api: | | with AI | without | | ----------------- | -------- | ------- | @@ -26,82 +25,83 @@ dropping the MCP registry, the chat routes and the two hosted classes from api: | bundle chunks | 85 | 50 | | module evaluation | ~336 ms | ~278 ms | -The per-request half is not in that table: a `/mcp` call no longer builds -`AllRoutes` and `ApiAuthLive`, and a `/v2` call no longer builds 47 tool schemas. -A 2026-09-08 attempt that moved only the transport measured 1.0%, which is what -moving the registry too is worth avoiding. +The per-request half is not in that table: a `/mcp` call no longer builds `AllRoutes` and +`ApiAuthLive`, and a `/v2` call no longer builds 47 tool schemas. A 2026-09-08 attempt that +moved only the transport measured 1.0%. Moving the registry too is what avoids that. Two things a future change here needs to know: -- **The `ChatSession` class carries `transferredFrom: "api"`.** Dropping a - locally hosted Durable Object class while keeping a cross-script reference is - the shape that destroys a namespace, and alchemy refuses it before uploading. - The property is inert once a stage has transferred, so it stays. -- **A Workflow has no equivalent.** Moving one to a new script mints a new - physical workflow and orphans in-flight runs. (The investigation fan-out - Workflow that used to live here was retired in 2026-09; `ClickHouseSchemaApplyWorkflow` - stays on `api`.) +- **The `ChatSession` class carries `transferredFrom: "api"`.** Dropping a locally hosted + Durable Object class while keeping a cross-script reference is the shape that destroys a + namespace, and alchemy refuses it before uploading. The property is inert once a stage + has transferred, so it stays. +- **A Workflow has no equivalent.** Moving one to a new script mints a new physical + workflow and orphans in-flight runs. (The investigation fan-out Workflow that used to + live here was retired in 2026-09. `ClickHouseSchemaApplyWorkflow` stays on `api`.) ## Layout -- `alchemy.run.ts` — the root stack. Provides `MapleStack` (stage, domains, public URLs, - the `bun dev` blocks) once, yields one module per app, and returns the deploy summary - (also emitted as GitHub step outputs). -- `apps//src/worker.ts` — a Worker as one module: the alchemy Worker class the root - yields, whose props are an Effect over `MapleStack`, and the bundle alchemy deploys - (`api`, `alerting`, `electric-sync`, `web`, `landing`, `local-ui`; see "Single-module - Workers" below). -- `apps//alchemy.run.ts` — a `create*` factory, only for the apps that are not - Workers (`ingest`, `electric` on ECS). Owns that app's resources and nothing else's. The one - api resource the ingest gateway shares (the replay bucket) is - `apps/api/src/resources/replay-blobs.ts`. -- `packages/infra` — stage/region/domain/naming logic, the shared deploy-time env groups, - and the few resources several Worker modules bind. - - `region.ts` — `MapleRegion` (`us` | `eu`), the one axis both clouds key on; see +- `alchemy.run.ts`: the root stack. Provides `MapleStack` (stage, region, domains, public + URLs, the `bun dev` blocks, the prd database branch) once, yields one module per app, and + returns the deploy summary (also emitted as GitHub step outputs). +- `apps//src/worker.ts`: a Worker as one module. It is the alchemy Worker class the + root yields, whose props are an Effect over `MapleStack`, and the bundle alchemy deploys + (`api`, `ai`, `alerting`, `chat-bot`, `electric-sync`, `web`, `landing`, `local-ui`; see + "Single-module Workers" below). +- `apps//alchemy.run.ts`: a `create*` factory for the apps that are not Workers + (`ingest`, `electric` on ECS), owning that app's resources and nothing else's. `sandbox` + also keeps its declaration here, because its `src/worker.ts` must stay a plain bundle + entry (see "The sandbox Worker" below). The one api resource the ingest gateway shares + (the replay bucket) is `apps/api/src/resources/replay-blobs.ts`. +- `packages/infra`: stage/region/domain/naming logic, the shared deploy-time env groups, and + the few resources several Worker modules bind. + - `region.ts`: `MapleRegion` (`us` | `eu`), the one axis both clouds key on. See "Regions" below. - - `cloudflare/stage.ts` — `MapleStage`, `parseMapleDeployment` (stage + region off the + - `cloudflare/stage.ts`: `MapleStage`, `parseMapleDeployment` (stage and region off the alchemy stage string), domains, worker names, placement, storage jurisdiction, Hyperdrive resolution. Pure functions, unit-tested, no cloud calls. - - `cloudflare/stack.ts` — `MapleStack`, what the root stack tells the Worker classes. - - `cloudflare/observability.ts` — the Workers Observability destinations, declared once - and yielded from every module that binds them (alchemy registers a resource by id; a - second yield returns the first's). - - `aws/stage.ts` — `MapleRegion`, AWS naming, task sizing, Cloud Map. - - `env.ts` — the deploy-time env primitives and the shared groups the workers spread. - - `cloudflare/maple-db.ts` — `MAPLE_DB` in the stage's flavor (`MapleDb`, yielded from a - Worker's init) and the runtime read of the binding (`readMapleDbBinding`). - - `config-helpers.ts` / `cloudflare/worker-runtime.ts` / `cloudflare/workers-cache.ts` — - the _runtime_ (in-Worker) side, each behind its own subpath export so a worker bundle + - `cloudflare/stack.ts`: `MapleStack`, what the root stack tells the Worker classes. + - `cloudflare/observability.ts`: the Workers Observability destinations, declared once + and yielded from every module that binds them (alchemy registers a resource by id, so + a second yield returns the first's). + - `aws/stage.ts`: AWS region, naming, task sizing, Cloud Map, and which stages get an + ingest fleet or a collector. + - `env.ts`: the deploy-time env primitives and the shared groups the Workers spread. + - `cloudflare/maple-db.ts`: `MAPLE_DB` in the stage's flavor (`MapleDb`, yielded from a + Worker's init, or `mapleDbEnv` from its props) and the runtime read of the binding + (`readMapleDbBinding`). + - `config-helpers.ts`, `cloudflare/worker-runtime.ts`, `cloudflare/workers-cache.ts`: + the _runtime_ (in-Worker) side, each behind its own subpath export so a Worker bundle never reaches the deploy graph through `./cloudflare`. `worker-runtime.ts` is the `WorkerEnvironment` tag (alchemy's key, typed `Record`) and `workerEnvLayer(env)`, the env plus its `ConfigProvider` for a graph built over one - env record — nothing reaches for `cloudflare:workers`. + env record. Nothing reaches for `cloudflare:workers`. **Read deploy-time config through `@maple/infra/env`, not `process.env`.** Alchemy resolves config through a ConfigProvider built as `fromDotEnv(--env-file ?? ".env")` **orElse** `fromEnv()` (`alchemy/Util/ConfigProvider.ts`), and never copies the file-sourced values into `process.env`. A `process.env` read therefore silently ignores `.env` and -`--env-file` — and does so _selectively_, since alchemy's own settings -(`CLOUDFLARE_ACCOUNT_ID`, `CI`, …) still pick them up, so half the deploy sees the file and -half does not. `Config` also reports every missing key in one pass instead of throwing on -the first, and keeps the failure in the typed error channel. `packages/alchemy-maple`'s -`MapleEnvironment` is the same pattern inside a provider; the runtime worker env schemas -use `@maple/infra/config-helpers`, which `env.ts` builds on. +`--env-file`. It does so _selectively_: alchemy's own settings (`CLOUDFLARE_ACCOUNT_ID`, +`CI`, …) still pick them up, so half the deploy sees the file and half does not. `Config` +also reports every missing key in one pass instead of throwing on the first, and keeps the +failure in the typed error channel. `packages/alchemy-maple`'s `MapleEnvironment` is the +same pattern inside a provider. The runtime Worker env schemas use +`@maple/infra/config-helpers`, which `env.ts` builds on. ## Regions: one stack, one instance per deploy -A geographic instance is the whole stack — every Worker, the ingest fleet, Electric, the -replay bucket, the chat Durable Object — deployed against that instance's own Tinybird -workspace, application database and secrets. There is no per-org routing anywhere: an org's +A geographic instance is the whole stack (every Worker, the ingest fleet, Electric, the +replay bucket, the chat Durable Object) deployed against that instance's own Tinybird +workspace, application database and secrets. There is no per-org routing anywhere. An org's region is the instance it was created on, and an EU hostname cannot reach a US resource because the EU Workers are bound to none. Plan and rationale: `docs/eu-region-plan.md`. -- **The alchemy stage string carries the region**: `prd` is the US instance, `prd-eu` the EU - one (`dev_makisuo-eu` a dev stage of it; PR previews are US-only). Alchemy keys its state - by stage, so the two instances never plan against each other's resources, and nothing has - to set a second variable in lockstep. `parseMapleDeployment` is the one parser; - `MapleStack` carries `region` to every Worker module. -- **`us` is unsuffixed** everywhere — Worker names, AWS names, hostnames — so adding `eu` +- **The alchemy stage string carries the region.** `prd` is the US instance and `prd-eu` + the EU one (`dev_makisuo-eu` is a dev stage of it; PR previews are US-only). Alchemy keys + its state by stage, so the two instances never plan against each other's resources, and + nothing has to set a second variable in lockstep. `parseMapleDeployment` is the one + parser. `MapleStack` carries `region` to every Worker module. +- **`us` is unsuffixed** everywhere (Worker names, AWS names, hostnames), so adding `eu` renamed nothing: `maple-api` / `maple-api-eu`, `app.maple.dev` / `app.eu.maple.dev`, `maple-ingest` / `maple-ingest-eu`. `regionSuffix` in `region.ts` is the single rule. - **Placement is a hint, jurisdiction is a pin.** `resolveWorkerPlacement` steers each @@ -111,22 +111,24 @@ because the EU Workers are bound to none. Plan and rationale: `docs/eu-region-pl jurisdiction is a property of the object id, so it is applied where ids are minted (`chatSessionStub`, reading the stack-derived `MAPLE_REGION`), not on the binding. Regional Services, the contractual execution guarantee, is an Enterprise add-on the - account does not carry; the residency claim says so. -- **Shared apps stay on `us`**: the marketing site and the local-mode SPA hold no customer + account does not carry. The residency claim says so. +- **Shared apps stay on `us`.** The marketing site and the local-mode SPA hold no customer data and there is one `maple.dev`, so `regionHostsSharedApps` keeps them off the EU deploy. - **Secrets** come from a per-instance Infisical environment (`prod`, `prod-eu`) holding the - same variable names with that instance's values; `deploy-prd-instance.yml` picks the + same variable names with that instance's values. `deploy-prd-instance.yml` picks the environment, the stage and the AWS region from one `region` input, and the stack refuses an `AWS_REGION` that disagrees with the stage. The EU deploy is opt-in through the `MAPLE_DEPLOY_EU` repository variable until its accounts exist. - **The EU database is declared, not pasted.** `resolveDatabaseMode` is `"declared"` for the - EU prd: the deploy adopts `maple-eu`'s `main` branch, declares a `Planetscale.PostgresRole` + EU prd. The deploy adopts `maple-eu`'s `main` branch, declares a `Planetscale.PostgresRole` per consumer on it and a `Cloudflare.Hyperdrive.Connection` on each role's direct origin (`declareMapleDb`), and the Workers bind theirs from their props (`mapleDbEnv`). The US - prd stays `"ref"`, on the dashboard configs it was measured on; moving it is the same + prd stays `"ref"`, on the dashboard configs it was measured on. Moving it is the same switch plus new config ids for its Workers. -- **AI features are off on the EU instance**: the model providers have no EU pin. +- **AI on the EU instance goes through OpenRouter's EU endpoint** (`eu.openrouter.ai`, + `apps/ai/src/platform/Llm.ts`, #1052). Workers AI has no region pin, so + `MAPLE_LLM_PROVIDER` stays unset there. ## Local dev: one `alchemy dev` stack @@ -134,82 +136,88 @@ because the EU Workers are bound to none. Plan and rationale: `docs/eu-region-pl ```bash bun dev # everything -bun dev api web # a subset: api, alerting, electric-sync, web, landing, ingest, local-ui, scraper +bun dev api web # a subset of: api, ai, alerting, chat-bot, electric-sync, web, landing, ingest, local-ui, scraper ``` -The Workers — **api, alerting, electric-sync** — are served by alchemy's local runtime from -the same Worker classes that deploy them. Everything that is not a Worker — web, landing -and local-ui (vite/astro dev servers), ingest (`cargo run`) and scraper — runs as a -`Command.Dev` child of the same stack: each app's own `dev` script, started by -`createDevProcess` in `alchemy.run.ts`, kept alive across stack restarts, stopped with the -stack. `Command.Dev` is a no-op on a deploy, so this is dev-only by construction; the asset -Workers' deploy shape (`Command.Build` + assets Worker) never runs in dev. +The Workers (**api, ai, alerting, chat-bot, electric-sync, web**) are served by alchemy's +local runtime from the same Worker classes that deploy them. web is a vite-source Worker, +so alchemy runs its vite dev server itself. Everything else (landing and local-ui as +vite/astro dev servers, ingest via `cargo run`, and the scraper) runs as a `Command.Dev` +child of the same stack (`DEV_PROCESS_APPS` in `@maple/infra/dev-urls`). Each child is that +app's own `dev` script, started by `createDevProcess` in `alchemy.run.ts`, kept alive across +stack restarts, and stopped with the stack. `Command.Dev` is a no-op on a deploy, so this is +dev-only by construction. The asset Workers' deploy shape (`Command.Build` plus an assets +Worker) never runs in dev. Portless provides what alchemy's local runtime does not: named HTTPS hosts that several -worktrees share without anyone caring about ports. The routes are alchemy resources — -`Portless.Route` from `lib/alchemy-portless`, one per app the run serves (`createDevRoute`). -A route reserves a loopback port, registers `portless alias `, and removes -the route again when it is torn down: with the dev session, on a config change, or on -`alchemy destroy`. Its `port` attribute feeds a child process's `PORT` like any other Output; -a Worker binds its port at plan time (`Portless.workerDev`) and its route follows the Worker. The -provider lives in alchemy's dev sidecar, so a route survives a hot reload of the stack file. +worktrees share without anyone caring about ports. The routes are alchemy resources: +`Portless.Route` from `lib/alchemy-portless`, one per app the run serves (`serveWorker` and +the dev-process loop in `alchemy.run.ts`). A route reserves a loopback port, registers +`portless alias `, and removes the route again when it is torn down: with +the dev session, on a config change, or on `alchemy destroy`. Its `port` attribute feeds a +child process's `PORT` like any other Output. A Worker binds its port at plan time +(`Portless.workerDev`) and its route follows the Worker. The provider lives in alchemy's dev +sidecar, so a route survives a hot reload of the stack file. **Ports are sticky, not pinned.** A route prefers a port derived from its identity (a hash -into 40000–49999; Workers use 50000–59999 so the two can never collide), walks forward if that -one is held (a second worktree running the same app), and only then takes an OS-chosen port. So `api` lands on the same port run after run -without anyone writing it down, and two checkouts never fight over one. Linked worktrees get -the branch-prefixed hostnames portless itself produces (last branch segment, none for `main`: -`fix-ui.api.localhost`), and -each non-Worker child is told its own name through `PORTLESS_URL`, which is how the -vite/astro configs find the api and ingest (`siblingUrl`). - -What is left in `scripts/dev.ts` is a shim: `bun dev api web` → `MAPLE_DEV_APPS=api,web`, then -`alchemy dev`. A subset run still declares every resource (a resource absent from the plan -would be deleted from the account); it only leaves the other Workers unserved -(`dev: { mode: "external" }`) and starts no child process for them. The inter-app URLs are -handed to the Workers as env by the stack itself, so `.env.local` cannot override them. - -`isDevServer` (`ALCHEMY_DEV=true`, set by `alchemy dev`) is what switches the stack into this -shape, and `MAPLE_DEV_APPS` narrows it to the requested apps. Neither is stage-derived: a dev +into 40000-49999; Workers use 50000-59999 so the two never collide). It walks forward if that +one is held (a second worktree running the same app), and only then takes an OS-chosen port. +So `api` lands on the same port run after run without anyone writing it down, and two +checkouts never fight over one. Linked worktrees get the branch-prefixed hostnames portless +itself produces (last branch segment, none for `main`: `fix-ui.api.localhost`). Each +non-Worker child is told its own name through `PORTLESS_URL`, which is how the vite/astro +configs find the api and ingest (`siblingUrl`). + +What is left in `scripts/dev.ts` is a shim: `bun dev api web` becomes +`MAPLE_DEV_APPS=api,web`, then `alchemy dev`. A subset run still declares every resource (a +resource absent from the plan would be deleted from the account). It only leaves the other +Workers unserved (`dev: { mode: "external" }`) and starts no child process for them. The +stack hands the inter-app URLs to the Workers as env itself, so `.env.local` cannot override +them. + +`isDevServer` (`ALCHEMY_DEV=true`, set by `alchemy dev`) switches the stack into this shape, +and `MAPLE_DEV_APPS` narrows it to the requested apps. Neither is stage-derived: a dev _stage_ can still be deployed to the cloud, and a deploy is never partial. What this buys over the old per-app `wrangler dev` under turbo + portless: -- **One definition.** Bindings, crons and exported classes come from `alchemy.run.ts`, for - dev and deploy alike. There are no `wrangler.jsonc` files any more; the crons/DO/KV/ - rate-limiter mirroring that used to drift between the two is gone with them. `wrangler` +- **One definition.** Bindings, crons and exported classes come from the stack, for dev and + deploy alike. There are no `wrangler.jsonc` files any more, and the crons/DO/KV/ + rate-limiter mirroring that used to drift between the two went with them. `wrangler` survives only as an `apps/api` devDependency for `bench:startup-cpu`, whose `worker` mode writes a throwaway config for `wrangler check startup`. - **One process tree.** No turbo fan-out, no per-app `portless` wrapper, no `dev:app` - indirection: `bun dev` is the stack, and Ctrl-C stops all of it. + indirection. `bun dev` is the stack, and Ctrl-C stops all of it. - **Crons fire on their real schedule.** `alchemy dev` runs each Worker's declared crons itself, and `/cdn-cgi/handler/scheduled` triggers one on demand (Miniflare's path, always - on — no `--test-scheduled` flag). -- **Almost everything is emulated locally.** Workers, KV, R2, Hyperdrive, queues + consumers, - Durable Objects, Workflows, rate limiters and `send_email` (written as `.eml` files under - `.alchemy/local/email/`) all come up `(local)`; storage lives under `.alchemy/local/`. - The AI Gateway is the one resource still created live in the account. + on, no `--test-scheduled` flag). +- **Almost everything is emulated locally.** Workers, KV, R2, Hyperdrive, queues and + consumers, Durable Objects, Workflows, rate limiters and `send_email` (written as `.eml` + files under `.alchemy/local/email/`) all come up `(local)`. Storage lives under + `.alchemy/local/`. The AI Gateway is declared on deployed stages only, since it has no + local emulation. -Gotchas worth knowing: +Gotchas: - **The dev Hyperdrive origin must set `sslmode: "disable"`.** Alchemy defaults a local - origin to `sslmode=prefer` (`Cloudflare/Hyperdrive/ConnectBinding.ts`), the driver then - attempts TLS against the docker Postgres, which has SSL off, and every DB call 503s after the - dial budget (`error.type = ConnectionError`; `CONNECT_TIMEOUT` under the old postgres.js driver). See `ManagedMapleDb`. + origin to `sslmode=prefer` (`Cloudflare/Hyperdrive/ConnectBinding.ts`). The driver then + attempts TLS against the docker Postgres, which has SSL off, and every DB call 503s after + the dial budget (`error.type = ConnectionError`; `CONNECT_TIMEOUT` under the old + postgres.js driver). See `ManagedMapleDb`. - **`MAPLE_OTEL_INGEST_KEY` is optional on dev stages only** (`selfObservabilityEnv`). The local stack resolves the same env contract as a deploy, and no developer has a real - ingest key; without the exemption the whole stack refuses to start over a key whose only + ingest key. Without the exemption the whole stack refuses to start over a key whose only job is exporting the Worker's own telemetry. - Dev stacks run with `ALCHEMY_LOCAL_STATE=1`, so they never touch the account state store. -- `--env-file .env.local` is read once at start: a changed variable needs a restart. +- `--env-file .env.local` is read once at start. A changed variable needs a restart. - The children's logs share one terminal. There are no per-app panes as under turbo's TUI. - A harness that cannot resolve `*.localhost` (the browser-verification preview) uses the - sticky raw ports `bun dev` prints; `Portless.Route`'s `port` prop pins one outright. -- Ctrl-C stops the whole tree: the shim runs alchemy in its own process group and forwards + sticky raw ports `bun dev` prints. `Portless.Route`'s `port` prop pins one outright. +- Ctrl-C stops the whole tree. The shim runs alchemy in its own process group and forwards the signal to the group, because alchemy's CLI is several node processes deep and a signal to any one of them stops nothing. The routes come off as alchemy tears the session down. - If portless is not installed or its proxy is down, a route logs a warning and the app is - reachable on `127.0.0.1:` only; the inter-app URLs still name the `*.localhost` + reachable on `127.0.0.1:` only. The inter-app URLs still name the `*.localhost` hosts, so start the proxy (`portless proxy start`) rather than work around it. ## Single-module Workers @@ -218,72 +226,76 @@ A Worker is one module, `apps//src/worker.ts`: the alchemy Worker class the yields (`export default class Alerting extends Cloudflare.Worker()("alerting", props, impl) {}`) and the bundle alchemy builds (`main: import.meta.url`). `props` is an Effect that reads `MapleStack` (`@maple/infra/cloudflare`, provided once by the root from -`Alchemy.Stage`) and yields whatever other resources the Worker binds — its `Command.Build`, +`Alchemy.Stage`) and yields whatever other resources the Worker binds: its `Command.Build`, the shared `ManagedMapleDb` or `WorkersObservabilityDestinations` (alchemy registers a resource by id, so a second module yielding the same one gets the first's). `impl` runs once per isolate on the first event and returns the handlers. No hand-written `export default -{ fetch }`, no per-app `alchemy.run.ts`, no factory arguments. Every Worker ships this way. -A Worker that binds another (web's `API` service binding to the api) takes it as a service -the root provides after yielding it (`ApiWorker` in `@maple/infra/cloudflare`) — a -`Worker.ref` reads stored state and cannot see a sibling the same deploy creates. +{ fetch }`, no factory arguments. A Worker that binds another (web's `API` service binding to +the api, api's and alerting's `AI_WORKER`) takes it as a service the root provides after +yielding it (`ApiWorker`, `AiWorker` in `@maple/infra/cloudflare`). A `Worker.ref` reads +stored state and cannot see a sibling the same deploy creates. What each kind of Worker keeps beside the module: - **Crons** (`alerting`, `api`): `Cloudflare.Workers.cron(expression, handler)` in `impl`, - under `CronEventSourceLive`, attaches the schedule at plan time and the listener at runtime. - The source reports every fire as successful, so the platform's retry never engages — and - nothing is lost: the ticks already log and swallow their own failures, the schedules - re-fire, and the shell logs a failure outside a tick. The ticks live in `src/scheduled.ts` - behind a dynamic import, so the api layer graph is off the startup path and out of the - deploy process (where `impl` also runs), and the test imports it without a runtime. A - cron fire is an event like any other: the bridge builds the telemetry into its scope and - flushes after it, so the tick graph carries no tracer or logger of its own (one there - would shadow the bridge's — `worker-telemetry.test.ts` pins that a tick's spans export). + under `CronEventSourceLive`, attaches the schedule at plan time and the listener at + runtime. The source reports every fire as successful, so the platform's retry never + engages. Nothing is lost: the ticks already log and swallow their own failures, the + schedules re-fire, and the shell logs a failure outside a tick. alerting's ticks live in + `src/scheduled.ts` behind a dynamic import, so the layer graph is off the startup path and + out of the deploy process (where `impl` also runs), and the test imports it without a + runtime. api's schedules are `src/worker/crons.ts`, with the tick bodies behind the same + kind of dynamic import (`src/worker/modules.ts`). A cron fire is an event like any other: + the bridge builds the telemetry into its scope and flushes after it, so the tick graph + carries no tracer or logger of its own. One there would shadow the bridge's + (`apps/alerting/src/worker-telemetry.test.ts` pins that a tick's spans export). - **Queues** (`api`): `Cloudflare.Queues.consumeQueueMessages(queue, settings, handler)` in `impl`, under `Queues.EventSourceLive`, yields the `Consumer` resource at plan time and the - listener at runtime. The consumers carry `renamedFrom({ fqn })` with the ids the retired api - factory declared them under (`vcs-sync-consumer`, …): alchemy migrates the state rows, so - the deploy plans a noop instead of re-creating each consumer — and the delete of the old - row would otherwise have removed the physical consumer the new row had adopted. Drop the - decoration once every stage has deployed past it. The init reads the queues back off the - host's props at plan time (`boundQueues`) rather than declaring them a second time: a - second declaration replaces the first's registration, props included. + listener at runtime (`src/worker/consumers.ts`, over the declarations in + `src/resources/queues.ts`). The consumers carry `renamedFrom({ fqn })` with the ids the + retired api factory declared them under (`vcs-sync-consumer`, …). Alchemy migrates the + state rows, so the deploy plans a noop instead of re-creating each consumer. Without it, + the delete of the old row would have removed the physical consumer the new row had + adopted. Drop the decoration once every stage has deployed past it. - **Background telemetry** (`api`): queue batches and cron ticks run under their own SDK - instance (`eventTelemetry` in `@maple/infra/worker-telemetry`, provided around the event) - so `maple-vcs-sync` and `maple-planetscale-webhooks` keep their - own service names — background work sharing `maple-api` skewed its p99 to 32s - (2026-09-04). The layer graphs those events build carry no tracer or logger of their own. -- **Durable Object and Workflows** (`api`): alchemy's Effect-native forms, yielded from the init. - `ChatSessionObject` (`src/chat/ChatSession.ts`) is `Cloudflare.DurableObject()("ChatSession", -impl)` over the plain `ChatSession` class — the outer Effect resolves state and env (it also runs - at plan time against a mock state, so it must not touch storage), the inner one builds the - session and returns its methods as Effects, which alchemy's bridge runs per RPC call and hands - back as-is. `ClickHouseSchemaApplyWorkflow` (`src/workflows/*.ts`) is - `Cloudflare.Workflow()(name, impl)` in the documented shape: the init resolves what the run - needs, then returns an `Effect.fn` body. The bodies (`*.run.ts`) are Effects on alchemy's step API — - `durableStep` (`src/workflows/durable-step.ts`) is `Cloudflare.Workflows.task` over an Effect - whose failure rejects the step, so Cloudflare retries it per config — reading `Database`, - `Cloudflare.WorkerEnvironment` and `Cloudflare.WorkflowStep` as services. The class wraps a run - in `withPgConnectionScope` + `layerPg` (one Postgres connection per run) and the run's own - `eventTelemetry` (`maple-schema-apply`), which flushes when alchemy - closes the run's scope. Everything is imported statically: the Worker evaluates in ~80 ms of - the 1 s startup-CPU budget on alchemy's bundle (`scripts/bench-startup-cpu.ts`, 2026-09-07). - The yield is the whole declaration: the binding (named after the class — `ChatSession`, - `ClickHouseSchemaApplyWorkflow`, which is what the services read - off the env), the namespace, the physical workflow (`--`, alchemy's - `makeWorkflowName`) and the generated entry's class export. No reference-form bindings, no - hand-written entry. + instance (`eventTelemetry` in `@maple/infra/worker-telemetry`, provided around the event), + so `maple-vcs-sync` and `maple-planetscale-webhooks` keep their own service names. + Background work sharing `maple-api` skewed its p99 to 32s (2026-09-04). The layer graphs + those events build carry no tracer or logger of their own. +- **Durable Objects** (`ai`) **and Workflows** (`api`): alchemy's Effect-native forms, + yielded from the init. `ChatSessionObject` (`apps/ai/src/chat/ChatSession.ts`) is + `Cloudflare.DurableObject()("ChatSession", impl)` over the plain `ChatSession` + class. The outer Effect resolves state and env. It also runs at plan time against a mock + state, so it must not touch storage. The inner one builds the session and returns its + methods as Effects, which alchemy's bridge runs per RPC call and hands back as-is. api + and the other consumers bind it cross-script by class name. + `ClickHouseSchemaApplyWorkflow` (`apps/api/src/workflows/*.ts`) is + `Cloudflare.Workflow()(name, impl)` in the documented shape: the init resolves what + the run needs, then returns an `Effect.fn` body. The body (`*.run.ts`) is an Effect on + alchemy's step API. `durableStep` (`packages/backend/src/platform/durable-step.ts`) is + `Cloudflare.Workflows.task` over an Effect whose failure rejects the step, so Cloudflare + retries it per config. The body reads `Database`, `Cloudflare.WorkerEnvironment` and + `Cloudflare.WorkflowStep` as services. The class wraps a run in `withPgConnectionScope` + + `layerPg` (one Postgres connection per run) and the run's own `eventTelemetry` + (`maple-schema-apply`), which flushes when alchemy closes the run's scope. The Workflow is + imported statically: the Worker evaluated in ~80 ms of the 1 s startup-CPU budget on + alchemy's bundle (`apps/api/scripts/bench-startup-cpu.ts`, 2026-09-07). The yield is the + whole declaration: the binding (named after the class, `ChatSession` or + `ClickHouseSchemaApplyWorkflow`, which is what the services read off the env), the + namespace, the physical workflow (`--`, alchemy's + `makeWorkflowName`) and the generated entry's class export, with no hand-written entry. The + only reference-form binding left is the cross-script `ChatSession` on the Workers that call it. - **The chat-bot Worker** (`chat-bot`): chat-platform ingress, and the one Worker in the fleet with a **resident** Durable Object. `ConnectorSocket` (`src/socket/ConnectorSocket.ts`) holds one outbound WebSocket per socket-ingress connector in `@maple/chat-platform`'s registry, addressed by the connector's id. Hibernation covers only sockets the platform - hands an object (`state.acceptWebSocket`); a socket the object dials itself keeps the + hands an object (`state.acceptWebSocket`). A socket the object dials itself keeps the object in memory for as long as it is open, so each socket connector costs one resident - object. That is what an @mention costs on a platform that delivers mentions no other way — - a platform that signs an HTTP webhook uses the same Worker's generic + object. That is what an @mention costs on a platform that delivers mentions no other way. + A platform that signs an HTTP webhook uses the same Worker's generic `POST /connectors/:connectorId/webhook` route and no Durable Object at all. - A `* * * * *` cron is the start and recovery trigger: the Worker has no traffic of its + A `* * * * *` cron is the start and recovery trigger. The Worker has no traffic of its own, so nothing would ever make a first request, and after a deploy or an eviction the next tick calls `ensureConnected` again. Between ticks the object's own alarm serves the connector's heartbeat, the reconnect backoff and a one-minute watchdog. A connector's @@ -292,81 +304,82 @@ impl)` over the plain `ChatSession` class — the outer Effect resolves state an recovery needs. A second Durable Object, `ConnectorRelay` (`src/relay/ConnectorRelay.ts`), carries the - turn a mention causes: one object per conversation, addressed by connector, workspace and - channel, which resolves the org, claims a turn on maple-ai's `ChatSession` and streams the - answer back into the conversation under its own `waitUntil` (an alarm every 30s keeps it - resident while it does, for the same reason the chat session arms one). It is a separate - object because the socket is a single one for the whole deployment: running turns there - would either block the next frame behind a model run or pile every concurrent turn in the - system into the object that holds the connection. It keeps two things in storage: which - conversations the bot opened itself, which is what lets a message that mentioned nobody - still be answered there (`src/relay/conversation.ts` holds the whole rule — the - conversation is the bot's own, its session has held a turn inside the last day, a human - wrote the message, and it has text), and a checkpoint per turn it is relaying - (`src/relay/settle.ts`). The checkpoint is the turn's identity, the cursor it was claimed - at and the refs of the platform messages posted for it — never what they say. An object - evicted or redeployed mid-turn loses only the fiber rendering it, and the answer is not - streamed any further: the keep-alive alarm wakes the fresh activation, which finds a - checkpoint no fiber of its own holds, leaves it until the turn's own `turn-end` is in the - session's log (or the session runs nothing), then renders the final answer into the same - messages in one pass - (the ordinary driver: surplus messages emptied, a missing tail posted), then clears it. A - checkpoint older than the session's own staleness ceiling plus a margin, one this build - cannot decode, or one whose session cannot be reached is dropped. It costs one small - storage write per platform message posted (not per edit), a delete per turn, and a prefix - `list` per 30s alarm; nothing new is resident. + turn a mention causes. There is one object per conversation, addressed by connector, + workspace and channel. It resolves the org, claims a turn on maple-ai's `ChatSession` and + streams the answer back into the conversation under its own `waitUntil`. An alarm every + 30s keeps it resident while it does, for the same reason the chat session arms one. It + is a separate object because there is a single socket for the whole deployment: running + turns there would either block the next frame behind a model run or pile every + concurrent turn in the system into the object that holds the connection. + + The relay keeps two things in storage. The first is which conversations the bot opened + itself, which is what lets a message that mentioned nobody still be answered there. + `src/relay/conversation.ts` holds the whole rule: the conversation is the bot's own, its + session has held a turn inside the last day, a human wrote the message, and it has text. + The second is a checkpoint per turn it is relaying (`src/relay/settle.ts`): the turn's + identity, the cursor it was claimed at and the refs of the platform messages posted for + it, never what they say. An object evicted or redeployed mid-turn loses only the fiber + rendering it, and the answer stops streaming. The keep-alive alarm wakes the fresh + activation, which finds a checkpoint no fiber of its own holds. It leaves it until the + turn's own `turn-end` is in the session's log (or the session runs nothing), then renders + the final answer into the same messages in one pass (the ordinary driver: surplus + messages emptied, a missing tail posted), then clears it. A checkpoint older than the + session's own staleness ceiling plus a margin, one this build cannot decode, or one + whose session cannot be reached is dropped. It costs one small storage write per + platform message posted (not per edit), a delete per turn, and a prefix `list` per 30s + alarm. Nothing new is resident. So the Worker binds, beyond its connector secrets: `ChatSession` cross-script on - maple-ai, `MAPLE_DB` (the api's Hyperdrive config — one row per mention, read inside a - connection scope that closes before the turn streams), `MAPLE_APP_BASE_URL` for the links - a reply carries, and an optional `MAPLE_SHARE_TOKEN_HMAC_KEY`, without which a chart in a - reply is relayed as text rather than as a picture. On a stage with no application - database (PR previews) the lookup fails, is logged, and the mention goes unanswered - rather than being told the workspace is unlinked. - - It takes **one public hostname on production instances** (`domains.chat` — - `chat.maple.dev`, `chat.eu.maple.dev`): a webhook connector's platform is configured with a - request URL inside the vendor's own application, and that URL has to keep working across - deploys, which is what buys the custom domain the socket half never needed. A dev stage - reaches the same route through portless and a PR preview gets none — a connector there would - have neither credentials nor a database to resolve a workspace in. The Worker is inert on a - stage with no connector credentials: every connector key is bound optional, and a connector - without its configuration is skipped with one log line, so no socket is opened, the webhook - route answers 503, and no turn is ever relayed. + maple-ai; `MAPLE_DB` (the api's Hyperdrive config, one row per mention, read inside a + connection scope that closes before the turn streams); `MAPLE_APP_BASE_URL` for the + links a reply carries; and an optional `MAPLE_SHARE_TOKEN_HMAC_KEY`, without which a + chart in a reply is relayed as text rather than as a picture. On a stage with no + application database (PR previews) the lookup fails, is logged, and the mention goes + unanswered rather than being told the workspace is unlinked. + + It takes **one public hostname on production instances** (`domains.chat`: + `chat.maple.dev`, `chat.eu.maple.dev`). A webhook connector's platform is configured with + a request URL inside the vendor's own application, and that URL has to keep working + across deploys. That is what the custom domain buys; the socket half never needed one. A + dev stage reaches the same route through portless and a PR preview gets none, since a + connector there would have neither credentials nor a database to resolve a workspace in. + The Worker is inert on a stage with no connector credentials. Every connector key is + bound optional, and a connector without its configuration is skipped with one log line, + so no socket is opened, the webhook route answers 503, and no turn is ever relayed. A connector whose install mints a credential per workspace (rather than using one deployment-wide secret) has it sealed into `chat_workspaces.credentials_{ciphertext,iv,tag}`, - AAD-bound to `(org_id, connector, external_workspace_id)`. That is why chat-bot also binds - `MAPLE_INGEST_KEY_ENCRYPTION_KEY`, optionally: without it such a workspace fails its lookup - and the mention goes unanswered, while every other connector runs normally. - -- **The sandbox Worker** (`sandbox`): the one Worker in the fleet whose own module is - its bundle entry. It hosts Cloudflare's Sandbox Durable Object (`@cloudflare/sandbox`), - which is a class the deployed script must export — and an Effect-native Worker cannot - export one, because alchemy generates its entry (`makeEffectVirtualEntry`) and exports - only the bridge classes it created. A plain module is used verbatim, so - `export { Sandbox }` in `apps/sandbox/src/worker.ts` is what binds. A separate app is - not the only way to run this image — an alchemy `Cloudflare.DurableObject` in the api - can front a `Cloudflare.Container` and talk to its port directly — but that means - owning the container's control protocol instead of using the vendor client, so this - buys the client at the price of an app. It has no route and no hostname: maple-ai, whose - agents run the sandbox tools, reaches it over a `SANDBOX` service binding, provided by the - root as `SandboxWorker`, and every - request carries `SANDBOX_INTERNAL_SERVICE_TOKEN` — deliberately not the shared - `INTERNAL_SERVICE_TOKEN`, which lets its holder act as any organization. - - Only `prd` gets one (`stageDeploysSandbox`). A PR preview has no application - database, so no repository resolves there; and on a dev stage `alchemy dev` would put a + AAD-bound to `(org_id, connector, external_workspace_id)`. That is why chat-bot also + binds `MAPLE_INGEST_KEY_ENCRYPTION_KEY`, optionally. Without it such a workspace fails + its lookup and the mention goes unanswered, while every other connector runs normally. + +- **The sandbox Worker** (`sandbox`): the one Worker in the fleet whose own module is its + bundle entry. It hosts Cloudflare's Sandbox Durable Object (`@cloudflare/sandbox`), which + is a class the deployed script must export. An Effect-native Worker cannot export one, + because alchemy generates its entry (`makeEffectVirtualEntry`) and exports only the bridge + classes it created. A plain module is used verbatim, so `export { Sandbox }` in + `apps/sandbox/src/worker.ts` is what binds, and the declaration lives in + `apps/sandbox/alchemy.run.ts`. A separate app is not the only way to run this image: an + alchemy `Cloudflare.DurableObject` in the api can front a `Cloudflare.Container` and talk + to its port directly. That means owning the container's control protocol instead of using + the vendor client, so this buys the client at the price of an app. It has no route and no + hostname. maple-ai, whose agents run the sandbox tools, reaches it over a `SANDBOX` + service binding, provided by the root as `SandboxWorker`. Every request carries + `SANDBOX_INTERNAL_SERVICE_TOKEN`, deliberately not the shared `INTERNAL_SERVICE_TOKEN`, + which lets its holder act as any organization. + + Only `prd` gets one (`stageDeploysSandbox`). A PR preview has no application database, + so no repository resolves there. On a dev stage, `alchemy dev` would put a multi-gigabyte `docker pull` between every developer and `bun dev`. What runs inside is one full `git clone` per commit under `/workspace/maple/`, kept to the newest three. The clone is a **background process** the Worker polls, because a - container request is capped well below what a cold clone of a real repository takes; a + container request is capped well below what a cold clone of a real repository takes. A call that arrives first gets `SandboxRunCheckoutPending` and retries. The credential is a GitHub token minted for that one repository with read-only contents, staged through the - container's file API into a root-only path and read by a git credential helper — never - put in a command, because every process's arguments are readable by the account the - agent's own commands run as. Commands run through a wrapper (`wrapCommand`): `env -i` + container's file API into a root-only path and read by a git credential helper. It is + never put in a command, because every process's arguments are readable by the account + the agent's own commands run as. Commands run through a wrapper (`wrapCommand`): `env -i` with a fixed environment, `runuser` to an unprivileged account that does not own the tree, `unshare -n` for a network namespace with no egress, and each stream cut to the request's bound where it is produced. The command's real exit status and whether the @@ -375,7 +388,7 @@ impl)` over the plain `ChatSession` class — the outer Effect resolves state an **`unshare -n` needs `CAP_SYS_ADMIN`, and whether Cloudflare's container runtime grants it is unverified.** Measured against the published image: under default container - capabilities it fails and the wrapper refuses to run the command; with the capability + capabilities it fails and the wrapper refuses to run the command. With the capability added, the namespace opens and a lookup inside it is denied while the same lookup outside succeeds. If the platform withholds it, every sandbox command returns `SandboxRunIsolationUnavailable` and the tools are dead until the request stops asking @@ -384,163 +397,171 @@ impl)` over the plain `ChatSession` class — the outer Effect resolves state an - **Assets** (`landing`, `local-ui`): the handler reads `Cloudflare.Workers.Request` and `env.ASSETS` and hands the web `Response` back through `HttpServerResponse.fromWeb`. - landing's negotiation is a plain function in `src/handler.ts` for the same test reason. -- **The application database** (`alerting`, `api`): `yield* MapleDb(consumer)` in the init - binds `MAPLE_DB` in the stage's flavor — `Hyperdrive.Connect(ManagedMapleDb)` on dev - stages, `host.bind` of the dashboard-managed config by id on prd (alchemy has no `env` - form for a Hyperdrive it did not create; its own `ConnectBinding` attaches the same raw - metadata), nothing on previews. The api's Workflows yield it too, from their outer phase. - The root yields `ManagedMapleDb` first on dev stages so its `MAPLE_PG_URL` read happens - outside any init, where alchemy's plan-time ConfigProvider would bind it as a secret. Every - Postgres layer reads the `MapleDbConnection` port (`packages/backend/src/platform/bindings.ts`), - never the env. - -Still a factory: `web` (takes `api`, for the service binding); `ingest` and `electric` are -ECS services. `web` follows once its props can `yield* MapleApi`; until then its -`src/worker.ts` is a plain `export default { fetch }` with no telemetry, because -`Telemetry.layer` only reaches handlers the bridge runs. - -Alchemy evaluates that module in three places — the deploy process, `alchemy dev`, and the -deployed isolate — and two rules keep it honest about which one it is in: + landing's negotiation is a plain function in `src/handler.ts`, so a test can drive it + without a runtime. +- **The application database** (`api`, `ai`, `alerting`, `chat-bot`): on dev stages and the + US prd, `yield* MapleDb(consumer)` in the init binds `MAPLE_DB`: + `Hyperdrive.Connect(ManagedMapleDb)` on dev stages, `host.bind` of the dashboard-managed + config by id on the US prd (alchemy has no `env` form for a Hyperdrive it did not create; + its own `ConnectBinding` attaches the same raw metadata). On the EU prd the props bind the + declared config through `mapleDbEnv`. Previews get nothing. The api's Workflow yields it + too, from its outer phase. The root yields `ManagedMapleDb` first on dev stages so its + `MAPLE_PG_URL` read happens outside any init, where alchemy's plan-time ConfigProvider + would bind it as a secret. Every Postgres layer reads the `MapleDbConnection` port + (`packages/backend/src/platform/bindings.ts`), never the env. + +**web** is a `Cloudflare.Website.Vite` Worker (`apps/web/src/worker.ts`). Alchemy owns its +vite build and its dev server, and the deployed entry is `src/worker-entry.ts`. Its props +bind the api Worker as `API`, handed over as `ApiWorker`. It carries no `WorkerTelemetry`. + +Alchemy evaluates a Worker module in three places (the deploy process, `alchemy dev`, and +the deployed isolate). Two rules keep it honest about which one it is in: - **Props are a plan-time Effect, guarded for the bundle.** The stage-derived props (`name`, `domain`, `env`, the portless `dev` block) read `MapleStack` (`@maple/infra/cloudflare`), a service the root stack provides once from `Alchemy.Stage`, so the module never imports portless or parses the stage itself. Alchemy also evaluates props inside the deployed bundle, where they are inert, so the props Effect returns early under - `globalThis.__ALCHEMY_RUNTIME__` — alchemy's bundler folds it to `true`, and the stack-side - branch plus the `@maple/infra` modules only it reaches are dead-code-eliminated. Check by - grepping the bundle under `.alchemy/bundles/electric-sync/` for a `maple.dev` hostname. + `globalThis.__ALCHEMY_RUNTIME__`. Alchemy's bundler folds that to `true`, and the + stack-side branch plus the `@maple/infra` modules only it reaches are dead-code-eliminated. + Check by grepping the bundle under `.alchemy/bundles/electric-sync/` for a `maple.dev` + hostname. - **The app layer is built on the first request, not in init.** `impl` (init) also runs at plan time, and alchemy's plan-time ConfigProvider auto-binds every `Config` it sees read - during init onto the Worker as a secret — which would override the explicit `env` contract + during init onto the Worker as a secret. That would override the explicit `env` contract (a PR preview deliberately gets no `ELECTRIC_URL`). So the route graph is dynamic-imported and built once per isolate on the first `fetch` (`Effect.cached`), against a scope that is - never closed: workerd has no isolate teardown, so nothing in the layer may need releasing. + never closed. workerd has no isolate teardown, so nothing in the layer may need releasing. - **The bridge serves the router and owns telemetry.** `fetch` is the `HttpRouter.toHttpEffect` - handler as-is: the bridge renders its typed failures before its tracer runs (`RouteNotFound` - → an Ok span with a 404; a defect → a 500, which the SDK records as an Error server span per - OTEL semconv), so no `orDie` sits on the request path — the one that did turned every 404 - into an Error span. `apps/electric-sync/src/worker-bridge.test.ts` drives the real bridge - path and pins all three outcomes; `apps/api/src/worker-bridge.test.ts` does the same for the - api's liveness, preflight and graph-failure fast paths. Telemetry is one line on init, + handler as-is. The bridge renders its typed failures before its tracer runs + (`RouteNotFound` becomes an Ok span with a 404; a defect becomes a 500, which the SDK + records as an Error server span per OTel semconv), so no `orDie` sits on the request path. + The one that did turned every 404 into an Error span. + `apps/electric-sync/src/worker-bridge.test.ts` drives the real bridge path and pins all + three outcomes. `apps/api/src/worker-bridge.test.ts` does the same for the api's liveness, + preflight and graph-failure fast paths. Telemetry is one line on init, `Effect.provide(WorkerTelemetry({ serviceName }))` from `@maple/infra/worker-telemetry`, on - the Workers that do work (api, electric-sync, alerting). The asset Workers (landing, local-ui) - deliberately have none: a server span per static page view is ingest volume spent - observing a file read, and it would put the internal ingest key in a marketing site's env - for it. Workers Observability covers their logs. It is the + the Workers that do work (api, ai, alerting, chat-bot, electric-sync). The asset Workers + (landing, local-ui) deliberately have none: a server span per static page view is ingest + volume spent observing a file read, and it would put the internal ingest key in a + marketing site's env. Workers Observability covers their logs. `WorkerTelemetry` is the published `Maple.Telemetry` from `@maple-dev/alchemy/telemetry` (Maple's counterpart of alchemy's `Axiom.Telemetry` sugar) with Maple's own defaults. It registers the SDK's `requestLayer` with alchemy's `Telemetry.layer`, which builds it into each request scope - and flushes after the response — no hand-rolled tracer, `waitUntil`, or flush shim. The - key/endpoint bindings `Maple.Telemetry` can do stay off for our Workers: + and flushes after the response. There is no hand-rolled tracer, `waitUntil`, or flush + shim. The key/endpoint bindings `Maple.Telemetry` can add stay off for our Workers: `selfObservabilityEnv(stage)` owns those, with the PR-preview rules. -What it costs: the root stack imports the worker module, so the Alchemy-entrypoints -typecheck (`tsconfig.alchemy.json`) covers electric-sync's runtime graph and needs -`@maple-dev/effect-sdk` built first and `@maple/electric-sync` installed in the quality -shard (`ci.yml`). Measured on the pilot (#745, local workerd A/B): +15ms startup CPU -(41→56ms, budget ~1s), ~+8ms cold first request, ~+0.2ms/request warm. +What it costs: the root stack imports the Worker modules, so the Alchemy-entrypoints +typecheck (`tsconfig.alchemy.json`) covers their runtime graphs and needs +`@maple-dev/effect-sdk` built first (`ci.yml`). Measured on the electric-sync pilot (#745, +local workerd A/B): +15ms startup CPU (41 to 56ms, budget ~1s), ~+8ms cold first request, +~+0.2ms/request warm. ### The api Worker's layout (2026-09-07) -`apps/api/src/worker.ts` is the composition root only — props plus an init that reads as a +`apps/api/src/worker.ts` is the composition root only: props plus an init that reads as a list of yields. What it composes lives beside it: -- `src/resources/*` — one file per resource the Worker binds, declared at module scope and - inert until yielded (`queues.ts`, `replay-blobs.ts`, `env.ts` for the - `Config` catalog). Stage-derived physical names come from `stageNamed` / `stageProps` - (`@maple/infra/cloudflare`), which read alchemy's own `Stage` — one of the platform - services a Worker's init may require, unlike `MapleStack` — behind the same - `__ALCHEMY_RUNTIME__` guard as a Worker's props, because alchemy evaluates a resource's props - Effect wherever it is yielded, the bundle included. The ingest factory yields the same - `ReplayBlobs` declaration to mint the gateway's writer token. -- `src/worker/*` — the runtime shell: `http.ts` (the lazily built route graph and `fetch`), - `rpc.ts`, `crons.ts`, `consumers.ts` (`consumeQueueMessages` over the declarations, so no - binding is read back off the host), `events.ts`, `modules.ts` (the dynamic imports), and - `bindings.ts`. +- `src/resources/*`: one file per resource the Worker binds, declared at module scope and + inert until yielded (`queues.ts`, `replay-blobs.ts`, and `env.ts` for the `Config` + catalog). Stage-derived physical names come from `stageNamed` / `stageProps` + (`@maple/infra/cloudflare`). They read alchemy's own `Stage`, one of the platform services + a Worker's init may require (unlike `MapleStack`), behind the same `__ALCHEMY_RUNTIME__` + guard as a Worker's props, because alchemy evaluates a resource's props Effect wherever it + is yielded, the bundle included. The ingest factory yields the same `ReplayBlobs` + declaration to mint the gateway's writer token. +- `src/worker/*`: the runtime shell. `http.ts` (the lazily built route graph and `fetch`), + `ai-forward.ts` (the forward to maple-ai), `crons.ts`, `consumers.ts` + (`consumeQueueMessages` over the declarations, so no binding is read back off the host), + `events.ts`, `modules.ts` (the dynamic imports), and `bindings.ts`. - **Bindings are alchemy capabilities, read as Maple ports.** The init yields - `Queues.WriteQueue(VcsSyncQueue)`, `R2.ReadBucket(ReplayBlobs)` and the four `Cloudflare.RateLimit(...)`s - (`worker/bindings.ts`); each yield attaches the native binding at plan time — under the - resource's logical id, so the queue and bucket bindings are `vcs-sync`, `replay-blobs`, … — - and resolves it from the env in the isolate. The clients become the ports in - `platform/bindings.ts` (`VcsSyncQueueProducer`, `ApiV2RateLimit`, `ReplayBlobBucket`, - `McpSessionStore`, …), which is what the services depend on: no service reads a binding off - `WorkerEnvironment` by name any more, tests provide fakes, and a host without the binding - (alerting, the CLI) provides nothing — the services that can degrade read the port through - `Effect.serviceOption`. The clients' methods are colored with alchemy's `RuntimeContext`, a - phantom that keeps them out of the init phase; the ports discharge it with - `RuntimeContext.phantom`, as alchemy's own runtime helpers do. -- What stays on `env:`: `AI` (the Gateway resource, read by name by the LLM shim), the stage - partition the rate limiters key under, and `EMAIL` on prd — alchemy's capabilities have no - "bound on some stages" form, and the plan/runtime split makes a conditional yield lie on - one side. `MAPLE_DB` is bound from the init (`MapleDb`, above). + `Queues.WriteQueue(VcsSyncQueue)`, `R2.ReadBucket(ReplayBlobs)` and the + `Cloudflare.RateLimit(...)`s (`worker/bindings.ts`). Each yield attaches the native + binding at plan time, under the resource's logical id (so the queue and bucket bindings + are `vcs-sync`, `replay-blobs`, …), and resolves it from the env in the isolate. The + clients become the ports in `platform/bindings.ts` (`VcsSyncQueueProducer`, + `ApiV2RateLimit`, `ReplayBlobBucket`, `McpSessionStore`, …), which is what the services + depend on. No service reads a binding off `WorkerEnvironment` by name any more, tests + provide fakes, and a host without the binding (alerting, the CLI) provides nothing. The + services that can degrade read the port through `Effect.serviceOption`. The clients' + methods are colored with alchemy's `RuntimeContext`, a phantom that keeps them out of the + init phase. The ports discharge it with `RuntimeContext.phantom`, as alchemy's own runtime + helpers do. +- What stays on `env:`: `EMAIL` on prd, the cross-script `ChatSession` binding, the + `AI_WORKER` service binding, and the stage partition the rate limiters key under. + Alchemy's capabilities have no "bound on some stages" form, and the plan/runtime split + makes a conditional yield lie on one side. `MAPLE_DB` is bound from the init (`MapleDb`, + above) or the props (`mapleDbEnv`). - The Worker env reaches a graph once, through the ports (`workerEnvLayer(env)` in - `apiPorts`): the runtime modules (`vcs-sync-runtime.ts`, …) declare `WorkerEnvironment` - and `ConfigProvider` as requirements and wire neither; a Durable Object or a Workflow run + `apiPorts`). The runtime modules (`vcs-sync-runtime.ts`, …) declare `WorkerEnvironment` + and `ConfigProvider` as requirements and wire neither. A Durable Object or a Workflow run hands its own env record to the same helper. ## The retired AWS opt-in flag (`MAPLE_DEPLOY_AWS_INGEST`) -The Rust OTLP gateway (`apps/ingest`) moved from Railway to ECS Fargate. While the -cut-over was in flight, `MAPLE_DEPLOY_AWS_INGEST=1` gated both `AWS.providers()` and the -ingest resources, so an unset variable produced a byte-identical pure-Cloudflare stack. +The Rust OTLP gateway (`apps/ingest`) moved from Railway to ECS Fargate, and on 2026-09-21 +to an ECS EC2 fleet on c7gd instances with local NVMe. While the Railway cut-over was in +flight, `MAPLE_DEPLOY_AWS_INGEST=1` gated both `AWS.providers()` and the ingest resources, +so an unset variable produced a byte-identical pure-Cloudflare stack. **The flag is gone (2026-08).** ECS is the only ingest path now, so the gate had nothing -left to protect: `AWS.providers()` is registered unconditionally (it cannot be -stage-derived — the `Alchemy.Stack` options are evaluated before `Alchemy.Stage` is -readable inside the stack effect), and `stageDeploysIngest` alone decides which stages get -a fleet. It covers prd **and PR previews**; dev stages run the gateway through -docker-compose. The spend gate moved to where the spend is: a preview only exists while -its PR carries the `preview` label. +left to protect. `AWS.providers()` is registered unconditionally. It cannot be +stage-derived, because the `Alchemy.Stack` options are evaluated before `Alchemy.Stage` is +readable inside the stack effect. `stageDeploysIngest` alone decides which stages get a +fleet. It covers prd **and PR previews**. Dev stages run the gateway through `cargo run` +under `bun dev`. The spend gate moved to where the spend is: a preview only exists while its +PR carries the `preview` label. Do not reintroduce a global on/off env flag for this. If a stage should not have a fleet, say so in `stageDeploysIngest`, where it is typed, unit-tested and visible in review. **The #378 hang.** The flag was _also_ introduced because turning the AWS half on wedged every production deploy with no log line and no network I/O. The cause was alchemy's -env-credential path (`CI=true`): it discovered the account with an STS `GetCallerIdentity` +env-credential path (`CI=true`). It discovered the account with an STS `GetCallerIdentity` issued while its own `AWSEnvironment` was still being constructed, and that call waited on -the half-built environment for its endpoint resolver — a self-deadlock. Supplying +the half-built environment for its endpoint resolver: a self-deadlock. Supplying `AWS_ACCOUNT_ID` skips the lookup. Reproduced locally with `CI=true` and the id unset, on -alchemy 2.0.0-beta.64 through beta.74. The deploy workflows now set it — **every workflow +alchemy 2.0.0-beta.64 through beta.74. The deploy workflows now set it. **Every workflow that deploys the stack must**, including `deploy-pr-preview.yml`. **Why the binary is compiled outside the image build.** Alchemy's docker build passes no `--cache-from`, and a fresh runner's layer cache is empty, so a Dockerfile that runs `cargo build` recompiles all 385 crates on every deploy however the layers are arranged. -Cold cost is **2m54s, measured** — an earlier version of this note guessed ~20 minutes, -which was wrong by ~7x and had already been quoted back as fact in a code review, so treat -the number as load-bearing. The workflow compiles inside `rust:1.94-bookworm` rather than -on the runner because the runtime base is `debian:bookworm-slim` (glibc 2.36) while -`ubuntu-24.04` ships 2.39 — a host-built binary dies with `version 'GLIBC_2.39' not found`. +Cold cost is **2m54s, measured**. An earlier version of this note guessed ~20 minutes, which +was wrong by ~7x and had already been quoted back as fact in a code review, so treat the +number as load-bearing. `build-ingest-binary.yml` compiles on a native arm64 runner +(`ubuntu-24.04-arm`) inside `rust:1.94-bookworm` rather than on the host, because the +runtime base is `debian:bookworm-slim` (glibc 2.36) while `ubuntu-24.04` ships 2.39. A +host-built binary dies with `version 'GLIBC_2.39' not found`. ## Schema migrations run in the deploy The instance's PlanetScale `main` branch (`maple`, `maple-eu`) is a `Planetscale.PostgresBranch` -yielded into `MapleStack` on prd (`db.schema`), with `migrations` at `packages/db/drizzle`. Alchemy -orders resources only by the Outputs their props reference, and a Hyperdrive bound by id references -nothing, so the api, ai and alerting Workers put the branch name in their env (`MAPLE_DB_BRANCH`, -via `mapleDbEnv`) to upload after it. The ingest gateway's Postgres credential is a -`Planetscale.PostgresRole` on the same branch; see `docs/persistence.md` for both. +yielded into `MapleStack` on prd (`db.schema`), with `migrations` at `packages/db/drizzle`. +Alchemy orders resources only by the Outputs their props reference, and a Hyperdrive bound +by id references nothing. So the api, ai, alerting and chat-bot Workers put the branch name +in their env (`MAPLE_DB_BRANCH`, via `mapleDbEnv`) to upload after it. The ingest gateway's +and Electric's Postgres credentials are `Planetscale.PostgresRole`s on the same branch. See +`docs/persistence.md` for both. ## Hyperdrive: why api and alerting have separate configs Measured over 6h on prd: `alerting` issued 60,688 Postgres queries/hour against the api's -1,415 — 97% versus 2%. Sharing one Hyperdrive config meant sharing one origin connection +1,415 (97% versus 2%). Sharing one Hyperdrive config meant sharing one origin connection pool, and the api spent its time queueing behind the alerting crons. A dial that found a -free slot took 12ms; one that did not stalled until Hyperdrive's 15s connection timeout, +free slot took 12ms. One that did not stalled until Hyperdrive's 15s connection timeout, which is what put `maple-api`'s p99 at 15.4s. -The two configs **partition** the origin's connections rather than creating more: the +The two configs **partition** the origin's connections rather than creating more. The per-config `origin_connection_limit`s sum against the branch's `max_connections`, and -Hyperdrive will not coordinate between them, so over-provisioning one starves the other at +Hyperdrive does not coordinate between them, so over-provisioning one starves the other at the database rather than at the pool. -**Resolved by deletion — staging pointed at production.** `resolveHyperdriveRefId` used to -return the prd config for `stg` (owner decision, 2026-07-14), so stg workers read and wrote +**Resolved by deletion: staging pointed at production.** `resolveHyperdriveRefId` used to +return the prd config for `stg` (owner decision, 2026-07-14), so stg Workers read and wrote the production database and the stg alerting crons overlapped prod's. The stage was removed -in full (2026-09): its deploy workflow had been disabled with no run history and neither +in full (2026-09). Its deploy workflow had been disabled with no run history and neither `api-staging.maple.dev` nor `ingest-staging.maple.dev` resolved, so the hazard was the only thing it still cost. `prd` is now the only stage `resolveHyperdriveRefId` answers for, and `parseMapleStage` rejects `stg` outright rather than letting it fall through to a dev stage. @@ -551,15 +572,15 @@ per consumer the way prd is, before it gets a `MAPLE_DB` binding at all. alchemy ≥ beta.70 sets rolldown `strictExecutionOrder: true`, which wraps ~every chunk in a lazy `__esmMin` initializer. The DB module graph (drizzle `pgTable` schemas + Effect Schema -ASTs) then evaluates on first use — inside the first Postgres call of each fresh isolate — +ASTs) then evaluates on first use, inside the first Postgres call of each fresh isolate, instead of at script startup. That stepped the cold dial from ~2s to ~9-11s on 2026-08-08 -(deploy 2679ba80) and produced the CONNECT_TIMEOUT incident; see the 2026-08-11 +(deploy 2679ba80) and produced the CONNECT_TIMEOUT incident. See the 2026-08-11 investigation. -The override in `apps/api/src/worker.ts` moves that cost back to script startup, off the -request path. If chunking ever regresses into upstream #749 (`ScriptStartupError: Cannot -access '' before initialization`), the deploy fails loudly at upload — remove the -override and warm the DB graph off the request path instead. +The override in `apps/api/src/worker.ts` (and `apps/ai/src/worker.ts`) moves that cost back +to script startup, off the request path. If chunking ever regresses into upstream #749 +(`ScriptStartupError: Cannot access '' before initialization`), the deploy fails +loudly at upload. Remove the override and warm the DB graph off the request path instead. ## Alchemy v1 → v2 notes @@ -568,50 +589,50 @@ when reading old code or docs: - **`HyperdriveRef` has no v2 equivalent.** Binding a dashboard-managed config by ID is done by attaching raw `{ type: "hyperdrive", name, id }` binding metadata from the Worker's - init (`host.bind` inside `MapleDb`, `cloudflare/maple-db.ts`) — the same mechanism the env - binder uses. No cloud - resource is created and the origin credentials stay in the dashboard. + init (`host.bind` inside `MapleDb`, `cloudflare/maple-db.ts`), the same mechanism the env + binder uses. No cloud resource is created and the origin credentials stay in the + dashboard. - **`Ai()` became an AI Gateway resource.** v2 emits the `{ type: "ai" }` binding by - attaching `Cloudflare.AI.Gateway`, which also fronts model calls with caching, - rate-limits and logging. The deploy token needs account-level "AI Gateway: Edit". + attaching `Cloudflare.AI.Gateway`, which also fronts model calls with caching, rate limits + and logging. The deploy token needs account-level "AI Gateway: Edit". - **`eventSources` became `Queues.Consumer`.** The consumer is a sibling resource pointing - at the worker by `scriptName`. + at the Worker by `scriptName`. - **Resource attributes are lazy Outputs.** `worker.url` and friends cannot be string-interpolated at plan time. This is why every deployed stage gets custom domains (`resolveMapleDomains`) and why inter-app URLs are plain strings chosen by the stack rather than read off resources. - **DO classes are SQLite-backed by default** in v2. - **The vendored runtime lib is gone.** `lib/effect-cloudflare` was a hand-copied subset of - `alchemy-effect`'s `Cloudflare/Workers/*`, from before that package shipped; ~1600 of its - 2520 lines had no consumer at all (the DO/Workflow/RPC/KV/fetcher/websocket cluster — - apps/api's own DO and Workflows extend `cloudflare:workers` directly). It was deleted; the - ~250 lines with consumers live in `packages/infra` behind runtime-only subpaths - (`/worker-runtime`, `/workers-cache`, `/r2`, `/config-helpers`). -- **Alchemy's runtime services were not importable from a hand-written Worker entry** - (its exports map has no entry finer than a directory, and the `Cloudflare/Workers` barrel - drags `fdir`, rolldown glue and `node:module` into a bundle — 426 KB against 14 KB for the - tag alone). That is why `packages/infra` carried its own `WorkerEnvironment` and R2 - client. With api and alerting on the class form (2026-09-07) the Workers use alchemy's - binding capabilities directly (`Queues.WriteQueue`, `KV.ReadWriteNamespace`, - `R2.ReadBucket`, `RateLimit`, `Hyperdrive.Connect`), the R2 client is gone, and the - `WorkerEnvironment` tag stays only for its stricter type — alchemy's is - `Record` — under alchemy's exact key, so both resolve to the same service. + `alchemy-effect`'s `Cloudflare/Workers/*`, from before that package shipped. ~1600 of its + 2520 lines had no consumer at all (the DO/Workflow/RPC/KV/fetcher/websocket cluster). It + was deleted (#760). The ~250 lines with consumers live in `packages/infra` behind + runtime-only subpaths (`/worker-runtime`, `/workers-cache`, `/config-helpers`). +- **Alchemy's runtime services were not importable from a hand-written Worker entry.** Its + exports map has no entry finer than a directory, and the `Cloudflare/Workers` barrel drags + `fdir`, rolldown glue and `node:module` into a bundle (426 KB against 14 KB for the tag + alone). That is why `packages/infra` carried its own `WorkerEnvironment` and R2 client. + With api and alerting on the class form (2026-09-07) the Workers use alchemy's binding + capabilities directly (`Queues.WriteQueue`, `KV.ReadWriteNamespace`, `R2.ReadBucket`, + `RateLimit`, `Hyperdrive.Connect`), and the R2 client is gone. The `WorkerEnvironment` tag + stays only for its stricter type (alchemy's is `Record`), under alchemy's + exact key, so both resolve to the same service. ## Cost decisions -These are cash-flow calls, not design ones, and should be revisited rather than treated as -architecture: +These are cash-flow calls, not design ones. Revisit them rather than treating them as +architecture. - **No NAT gateway** in the ingest VPC. NAT bills $0.045/GB _processed_ on top of egress, - and the gateway exists to push gzipped telemetry outbound — at current volume NAT alone + and the gateway exists to push gzipped telemetry outbound. At current volume NAT alone would cost more than the compute and the egress combined, and it scales linearly with - growth. Tasks therefore carry public IPs, which is why the security-group split between - the ALB and the tasks is load-bearing rather than tidy. The S3 gateway endpoint keeps ECR - image pulls off the public path and is free. + growth. Tasks (and, on the EC2 fleet, the hosts, which run tasks in host networking) + therefore carry public IPs, which is why the security-group split between the ALB and the + tasks is load-bearing rather than tidy. The S3 gateway endpoint keeps ECR image pulls off + the public path and is free. - **Same-region Tinybird.** AWS bills $0.09/GB to the public internet but $0.01/GB to a - public IP in the same region, and export traffic dwarfs every other line item — at 200k + public IP in the same region, and export traffic dwarfs every other line item. At 200k req/s that is ~$83k/mo vs ~$16k/mo. A workspace on `https://api.tinybird.co` is GCP - Frankfurt, where NO AWS region colocates and the move costs more than Railway did. + Frankfurt, where no AWS region colocates, and the move costs more than Railway did. Verify `TINYBIRD_HOST` before changing `resolveAwsRegion`. - **The OTel collector is prd-only** (`stageDeploysCollector`). The intent is every stage that deploys the gateway, at ~$13.5/mo per stage at the non-prd size. Adding the @@ -619,48 +640,48 @@ architecture: which is how it was verified on Fargate before it reached prod. - **PR previews get an ingest fleet, but no database.** PlanetScale PR branches billed continuously and consumed the account's Hyperdrive config cap, so `resolveDatabaseMode` - returns `"none"` for `pr`: DB-backed routes 500 and the rest of the preview works. The - reverse path is documented on that function. The AWS half _is_ deployed — a preview gets + returns `"none"` for `pr`. DB-backed routes 500 and the rest of the preview works. The + reverse path is documented on that function. The AWS half _is_ deployed: a preview gets its own VPC + ALB + ECS fleet, which is real money, so it only runs while the PR carries - the `preview` label and is destroyed the moment the label comes off or the PR closes. - A preview has no ingest domain, so its ALB answers plain HTTP on 80 with no ACM - certificate; the URL is posted on the PR comment. There is no longer a separate - on-demand ingest preview stack (`scripts/ingest-preview.run.ts` and - `deploy-pr-ingest.yml` are deleted) — two alchemy stacks claiming the same - `maple-ingest-pr-` physical names is how orphan fleets accumulate. -- **x86_64, not Graviton.** Flipping `cpuArchitecture` to `"ARM64"` is the whole switch - (~20% cheaper) but needs an ARM builder: cross-compiling Rust under QEMU is 10-30 min a - build, for single-digit dollars a month at this size. Revisit with an ARM runner. + the `preview` label and is destroyed the moment the label comes off or the PR closes. A + preview has no ingest domain, so its ALB answers plain HTTP on 80 with no ACM certificate. + The URL is posted on the PR comment. There is no longer a separate on-demand ingest + preview stack (`scripts/ingest-preview.run.ts` and `deploy-pr-ingest.yml` are deleted). + Two alchemy stacks claiming the same `maple-ingest-pr-` physical names is how orphan + fleets accumulate. +- **Graviton (ARM64).** The gateway and collector tasks run `cpuArchitecture: "ARM64"` + (`runtimePlatform` in `apps/ingest/alchemy.run.ts`), ~20% cheaper than x86_64. The + blocker used to be the builder: cross-compiling Rust under QEMU is 10-30 min a build. + `build-ingest-binary.yml` now compiles natively on an arm64 runner. ## Things that have broken a deploy before -Short list; each has a comment at the site. +Short list. Each has a comment at the site. - A **relative `dockerfile`** path. Alchemy has flipped how it resolves one between - releases — `ECR.Image` joins it onto `context`, the ECS image source (beta.73+) resolves - it against the cwd — and each flip broke the deploy. Absolute paths pass through both. -- **`listenerPort` vs `port`** on `ECS.Service`. `port` is the container port; the listener - defaults to 443 once `certificateArn` is set. Setting `listenerPort: INGEST_PORT` puts - the listener on 3474, which Cloudflare's proxy does not forward to, _and_ drops `port` - back to alchemy's default 3000 while the gateway binds 3474 — so no target ever passes - `/health`. + releases (`ECR.Image` joins it onto `context`; the ECS image source, beta.73+, resolves it + against the cwd), and each flip broke the deploy. Absolute paths pass through both. +- **`listenerPort` vs `port`** on `ECS.Service`. `port` is the container port. The listener + defaults to 443 once `certificateArn` is set. Setting `listenerPort: INGEST_PORT` puts the + listener on 3474, which Cloudflare's proxy does not forward to, _and_ drops `port` back to + alchemy's default 3000 while the gateway binds 3474, so no target ever passes `/health`. - **A security group that does not admit the listener port.** A stage without an ingest domain gets an HTTP listener on 80, not 443. The first preview deploy came up healthy and timed out on every request for exactly this reason. -- **An arm64 image on an X86_64 task** — fails at start with "image Manifest does not - contain descriptor matching platform 'linux/amd64'". Hence the explicit - `runtimePlatform`. +- **An image whose architecture does not match the task.** It fails at start with "image + Manifest does not contain descriptor matching platform 'linux/amd64'" (or the arm64 + equivalent). Hence the explicit `runtimePlatform`. - **A cargo `--target-dir` inside the image context.** Alchemy hashes the context with `hashDirectory`, which has no `.dockerignore` support and derives exclusions from - gitignore rules _without_ rebasing root-anchored ones onto the context dir — + gitignore rules _without_ rebasing root-anchored ones onto the context dir. `/apps/ingest/target` becomes the glob `apps/ingest/target/**` evaluated with `cwd=apps/ingest`, matching nothing. The whole target dir would be walked and hashed on every deploy. -- **A first deploy of a new stage with the AWS half on** used to fail: each service's ACM +- **A first deploy of a new stage with the AWS half on** used to fail. Each service's ACM certificate landed `PENDING_VALIDATION` and its 443 listener refused it, and the workflows recovered by publishing the validation CNAMEs with a script and deploying - again. `@maple/infra/acm` now does that inside the stack — it resolves the Cloudflare - zone from the certificate's own hostname, publishes the CNAME ACM asks for, and blocks - the listener on `ISSUED` — so a new certificate-bearing service needs no registration - anywhere and no second pass. The remaining manual record is the **proxied CNAME at the - ALB** for each public service, which the deploy output names. + again. `@maple/infra/acm` now does that inside the stack: it resolves the Cloudflare zone + from the certificate's own hostname, publishes the CNAME ACM asks for, and blocks the + listener on `ISSUED`. A new certificate-bearing service needs no registration anywhere + and no second pass. The remaining manual record is the **proxied CNAME at the ALB** for + each public service, which the deploy output names. diff --git a/docs/ingest-wal-durability.md b/docs/ingest-wal-durability.md index 11bb9a94f8..8e5ff1808a 100644 --- a/docs/ingest-wal-durability.md +++ b/docs/ingest-wal-durability.md @@ -2,15 +2,17 @@ The ingest gateway (`apps/ingest`, Rust) acknowledges a request once the rows are committed to a local write-ahead log, then exports them to Tinybird or ClickHouse in the background. That WAL is -the only copy of an accepted row until it reaches the warehouse, and it lives on Fargate ephemeral -storage, which is destroyed with the task. +the only copy of an accepted row until it reaches the warehouse. In prd it lives on the EC2 host's +NVMe instance store (`/mnt/wal`, bind-mounted into the task), which survives a reboot but is wiped +when the instance stops or is replaced. On a Fargate task it lives on ephemeral storage, which is +destroyed with the task. This is how it stays durable. ## The log -One **lane** per `(shard, destination)` — see `lane_index` in `apps/ingest/src/telemetry.rs`. Lanes -are independent so a stalled ClickHouse export cannot back up the Tinybird lane sharing its shard. +One **lane** per `(shard, destination)` (see `lane_index` in `apps/ingest/src/telemetry.rs`). Lanes +are independent, so a stalled ClickHouse export cannot back up the Tinybird lane sharing its shard. Each lane is a directory of sealed segments: @@ -24,7 +26,7 @@ $INGEST_QUEUE_DIR/shard-003-clickhouse/ Appends go to the active segment and `sync_data` before the request is acknowledged. At `INGEST_WAL_SEGMENT_MAX_BYTES` (8 MiB) the segment is **sealed**: the file is closed and the next -one opened. Sealing is an `open` plus a directory fsync — there is no copy on the append path. +one opened. Sealing is an `open` plus a directory fsync. There is no copy on the append path. The exporter advances `lane.cursor` and unlinks every segment behind it. **Appends and exports take different locks**, so an export can never stall a commit. That separation is the fix for the @@ -36,7 +38,7 @@ Ordering rules that matter: - The cursor is written **before** segments are unlinked. A crash in between costs a re-delete on the next boot; the other order strands a cursor pointing at bytes that are gone. -- Boot always opens a **fresh** segment, so "sealed" means "will never grow again" — which is what +- Boot always opens a **fresh** segment, so "sealed" means "will never grow again". That is what lets a sealed segment be shipped or deleted without coordinating with the appender. - A lost or unparseable cursor replays the lane from its oldest surviving segment. Everything here is **at-least-once**: replaying an exported frame is a duplicate row, skipping one is silent loss. @@ -55,12 +57,12 @@ s3:///wal/v1/ segments//shard-003-clickhouse/000000000041.seg ``` -An **owner** is one boot, identified by a fresh UUID — never a task ARN — so a sequence number can +An **owner** is one boot, identified by a fresh UUID (never a task ARN), so a sequence number can never be reused across boots and a claim can never race a live writer for the same key. **Shipping.** A sealed segment is announced to a shipper task (`SEGMENT_SHIPPER_WORKERS`, one lane always handled by the same worker so its events stay ordered). The shipper skips any segment the -exporter has already passed, which in a healthy pipeline is nearly all of them — this is why the +exporter has already passed, which in a healthy pipeline is nearly all of them. That is why the tier costs single-digit dollars a month at ~2B traces: the bucket holds the current backlog, not the traffic. When a segment exports, its object is deleted. @@ -68,12 +70,12 @@ traffic. When a segment exports, its object is deleted. `INGEST_WAL_S3_ORPHAN_AFTER_SECS` (10 min), and claims it with a conditional `PUT … If-None-Match: *`. S3 answers 412 when the key exists, so exactly one task wins without a lock service. The winner downloads that owner's segments, re-commits their frames to its own lanes, **re-ships them under its -own owner id, and only then** deletes the source objects — so the frames are never only on one +own owner id, and only then** deletes the source objects, so the frames are never only on one task's disk. A claim older than 30 minutes is taken over unconditionally, because the task that wrote it evidently died before finishing. **Shutdown.** After `INGEST_SHUTDOWN_DRAIN_SECS`, whatever did not export is sealed and shipped, and -the owner heartbeat is deleted — so the next task claims it immediately instead of waiting out the +the owner heartbeat is deleted, so the next task claims it immediately instead of waiting out the staleness window. Credentials come from the ECS task role (`AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`, refreshed 5 @@ -84,11 +86,11 @@ has an S3 gateway endpoint and no NAT, so this traffic is free. | Variable | Default | Notes | | --------------------------------- | ----------------------------------- | ------------------------------------------------------- | -| `INGEST_QUEUE_MAX_BYTES` | — | Total WAL budget; divided evenly across lanes | -| `INGEST_WAL_SHARDS` | `max(cpus × 2, 2)` | Shards, not lanes — lanes are `shards × destinations` | +| `INGEST_QUEUE_MAX_BYTES` | 20 GiB | Total WAL budget; divided evenly across lanes | +| `INGEST_WAL_SHARDS` | `max(cpus × 2, 2)` | Shards, not lanes: lanes are `shards × destinations` | | `INGEST_WAL_SEGMENT_MAX_BYTES` | 8 MiB | Seal threshold, and so the shipped object size | | `INGEST_WAL_S3_BUCKET` | unset | Unset keeps the WAL local-only (self-hosted, local dev) | -| `INGEST_WAL_S3_REGION` | `$AWS_REGION` | Required with a bucket | +| `INGEST_WAL_S3_REGION` | `$AWS_REGION` | Required with a bucket (either variable) | | `INGEST_WAL_S3_ENDPOINT` | `https://s3..amazonaws.com` | For an S3-compatible target | | `INGEST_WAL_S3_PREFIX` | `wal` | Key prefix inside the bucket | | `INGEST_WAL_S3_ORPHAN_AFTER_SECS` | 600 | How stale a heartbeat must be to be claimable | @@ -106,16 +108,17 @@ task-role policy are in `apps/ingest/alchemy.run.ts`. The byte caps bound how much the WAL holds, not how long a frame waits in it, and those are different limits. A lane's channel holds `INGEST_QUEUE_CHANNEL_CAPACITY` frames (100k) and the WAL holds `INGEST_QUEUE_MAX_BYTES` (20 GiB), so a stalled downstream target absorbs minutes of traffic -before either cap is reached — and for all of those minutes the accept path queues behind it on the +before either cap is reached. For all of those minutes the accept path queues behind it on the WAL append rather than shedding, because `try_reserve_owned` only refuses once the slots are gone. Two limits close that gap: - **`INGEST_QUEUE_MAX_AGE_SECS`** sheds a frame the export worker picks up more than this long after it was committed. This is **deliberate data loss** and is metered as `ingest_queue_age_shed_total` - (per org, destination and datasource) — alert on it. A shed frame still advances the lane cursor + (per org, destination and datasource). Alert on it. A shed frame still advances the lane cursor and releases its org bytes, or it would keep the backlog it was shed to cut. Frames recovered by WAL replay are stamped at replay time, so a restart's backlog gets a full window to export in - rather than being shed on sight for having been written before the restart. Set to 0 to queue without an age bound. + rather than being shed on sight for having been written before the restart. Set to 0 to queue + without an age bound. - **`INGEST_REQUEST_TIMEOUT_SECS`** bounds one HTTP request. Without it the accept path had no deadline anywhere, so saturation did not surface as errors: requests queued on the WAL append and sat there, p95 in the tens of seconds with per-route error rates flat near zero. Timed-out @@ -133,12 +136,12 @@ only redistribute the same IO. | Metric | Read it for | | ----------------------------------- | ------------------------------------------------------------------- | | `ingest_wal_shard_bytes` | Bytes a lane holds on disk, exported prefix included | -| `ingest_wal_shard_full_total` | Appends rejected because a lane hit its cap — customer-visible 429s | +| `ingest_wal_shard_full_total` | Appends rejected because a lane hit its cap: customer-visible 429s | | `ingest_wal_segments_sealed_total` | Segment rotation rate | | `ingest_wal_reclaimed_bytes_total` | Bytes freed by deleting exported segments | | `ingest_wal_shipped_bytes_total` | Bytes that reached the bucket | | `ingest_wal_ship_outcomes_total` | `outcome=exported_first` (healthy), `queue_full`, `failed` | -| `ingest_wal_frames_recovered_total` | Frames claimed from a dead task — non-zero means a task died dirty | +| `ingest_wal_frames_recovered_total` | Frames claimed from a dead task; non-zero means a task died dirty | | `ingest_wal_commit_bytes` | Per-append size; the `ingest.wal_commit` span carries the latency | `queue_full` means the object store cannot keep up with segment rotation, and those segments stay diff --git a/docs/legacy-cleanup-sweep-2026-09-05.md b/docs/legacy-cleanup-sweep-2026-09-05.md index d623cbd728..5606def710 100644 --- a/docs/legacy-cleanup-sweep-2026-09-05.md +++ b/docs/legacy-cleanup-sweep-2026-09-05.md @@ -1,4 +1,4 @@ -# Legacy cleanup sweep — 2026-09-05 +# Legacy cleanup sweep (2026-09-05) Baseline: `0e8250f2f0` (`refactor: remove retired adapters and error-service facade`). @@ -91,9 +91,11 @@ Keep this behavior consistent across local and remote commands. **Confirmed missed reuse; small-to-medium effort; behavior-risk.** -[turn-runner.ts](../apps/api/src/chat/turn-runner.ts):357–360 creates a fresh runtime with +Both files have since moved from `apps/api` to `apps/ai`; line numbers refer to the baseline. + +[turn-runner.ts](../apps/ai/src/chat/turn-runner.ts):357–360 creates a fresh runtime with `layerPg`, but line 480 runs the program without `withPgConnectionScope`. -[ChatSession.ts](../apps/api/src/chat/ChatSession.ts):354,414–415 calls it as a plain promise; +[ChatSession.ts](../apps/ai/src/chat/ChatSession.ts):354,414–415 calls it as a plain promise; there is no inherited Effect connection scope. Each tool database callback therefore uses `DatabasePgLive`'s fresh-client fallback. diff --git a/docs/local-event-consumers.md b/docs/local-event-consumers.md index 496d53c98d..9dc720d509 100644 --- a/docs/local-event-consumers.md +++ b/docs/local-event-consumers.md @@ -1,27 +1,30 @@ # Maple Local event consumer protocol -Status: version 1 durable downstream-consumer boundary for the Maple Local event outbox. +Status: version 1 of the durable downstream-consumer boundary for the Maple Local event outbox. +Routes live in `apps/cli/src/server/serve.ts`; state lives in +`apps/cli/src/server/eventing/control-store.ts`. This protocol lets a local consumer deliver ready Maple CloudEvents without destructive reads or a -second delivery database. It is intentionally transport-neutral: Maple does not select a downstream -transport, store downstream credentials, or choose delivery destinations. +second delivery database. It is transport-neutral. Maple does not pick a downstream transport, store +downstream credentials, or choose delivery destinations. ## Credentials -Maple creates two independent 32-byte hexadecimal credentials beside the configured data directory: +Maple creates two independent random 32-byte credentials (64 hex characters) beside the configured +data directory: - `.maintenance-token` administers projection and consumer configuration. - `.event-consumer-token` permits only claim and acknowledgement requests. -Both files must be real regular files. The consumer token is sent in -`x-maple-event-consumer-token`; it does not grant access to projection configuration, outbox -inspection, checkpoints, or retention controls. The existing maintenance token is sent in -`x-maple-maintenance-token` and cannot be substituted for the consumer token. +Both files must be regular files, not symlinks. The consumer token goes in +`x-maple-event-consumer-token`. It grants no access to projection configuration, outbox inspection, +checkpoints, or retention controls. The maintenance token goes in `x-maple-maintenance-token` and +cannot stand in for the consumer token. A wrong or missing consumer token returns `403`. ## Consumer administration -Consumer IDs match `^[a-z][a-z0-9._-]{0,63}$` and are unique. Disabled IDs remain reserved so an -operator cannot accidentally replace one consumer's durable position with an unrelated process. +Consumer IDs match `^[a-z][a-z0-9._-]{0,63}$` and are unique. Disabled IDs stay reserved, so an +operator cannot accidentally hand one consumer's durable position to an unrelated process. Register a consumer with the maintenance credential: @@ -33,13 +36,13 @@ X-Maple-Maintenance-Token: {"consumerId":"automation","startAt":"beginning"} ``` -`startAt` is exact: +`startAt` takes one of two values: - `beginning` starts immediately before the earliest ready event still retained for the tenant. - `latest` atomically skips every ready event visible at registration and receives later events. -Successful registration returns `201` and the consumer record. Reusing any existing or disabled ID -returns `409`. `GET /local/eventing/consumers` lists records under maintenance authorization. +Registration returns `201` and the consumer record. Reusing an existing or disabled ID returns +`409`. `GET /local/eventing/consumers` lists records under maintenance authorization. Disable a consumer explicitly: @@ -51,12 +54,12 @@ X-Maple-Maintenance-Token: {"consumerId":"automation"} ``` -Disabling clears any active lease and removes that cursor from the retention quorum. It does not -delete the audit record or permit the ID to be reused. +Disabling clears any active lease and removes the cursor from the retention quorum. The record +stays, and the ID cannot be reused. ## Claim and acknowledgement -Claim between 1 and 1,000 ready events for a lease of 5 through 300 seconds: +Claim 1 to 1,000 ready events with a lease of 5 to 300 seconds: ```http POST /local/eventing/claims @@ -89,13 +92,13 @@ A non-empty response has this shape: } ``` -The real `event` member is the complete validated CloudEvent. An empty claim returns null lease -fields and an empty event array. Only a SHA-256 hash of the lease token is stored. A second claim -while the lease is live returns `409`; at or after expiry it returns the same unacknowledged prefix, -possibly with a new token. +The `event` member (abbreviated above) is the complete validated CloudEvent. An empty claim returns +null lease fields, a null `throughSequence`, and an empty `events` array. Maple stores only a SHA-256 +hash of the lease token. A second claim while the lease is live returns `409`. At or after expiry, a claim +restarts after the last acknowledged sequence and issues a new lease token. -After every event in the claimed batch has been accepted by the downstream system, acknowledge the -exact `throughSequence` returned by the claim: +Once the downstream system has accepted every event in the batch, acknowledge the exact +`throughSequence` from the claim: ```http POST /local/eventing/acks @@ -105,26 +108,36 @@ X-Maple-Event-Consumer-Token: {"consumerId":"automation","leaseToken":"","throughSequence":42} ``` -Partial, extended, expired, missing, and wrong-token acknowledgements return `409`. Success returns: +Partial, extended, expired, missing, and wrong-token acknowledgements return `409`. Success +returns: ```json { "consumerId": "automation", "acknowledgedThrough": 42, "prunedEvents": 0 } ``` -Claims are at-least-once. A consumer crash after a downstream send and before acknowledgement causes -re-delivery after lease expiry. A consumer must therefore use the immutable Maple CloudEvent `id` as -its downstream idempotency key whenever the destination supports one. +Claims are at-least-once. If a consumer crashes after a downstream send and before +acknowledgement, the batch is re-delivered after lease expiry. Consumers must use the immutable +Maple CloudEvent `id` as the downstream idempotency key whenever the destination supports one. ## Retention, capacity, and checkpoints -Ready events are eligible for pruning only through the lowest acknowledged sequence among all active -consumers for the tenant. Maple retains the newest 1,000 otherwise-prunable ready events by default. -Disabled consumers do not block pruning; staged events are never pruned by consumer acknowledgement. -If no consumer is active, acknowledgement retention performs no deletion. +Ready events can be pruned only up to the lowest acknowledged sequence among the tenant's active +consumers. By default Maple keeps the newest 1,000 otherwise-prunable ready events. Disabled +consumers do not block pruning. Consumer acknowledgement never prunes staged events. With no active +consumer, acknowledgement retention deletes nothing. -The outbox defaults to 10,000 events and 256 MiB of canonical event JSON. Transactional counters enforce both caps without scanning every event for each ingest. If a new projection cannot fit, Maple drops that projection and continues warehouse ingestion. Existing staged and ready events remain intact. The OTLP response includes `x-maple-eventing-dropped` with the number of dropped projection attempts. Health includes a durable `deliveryGap` with a generation, cumulative dropped-event count, and last-drop time. Repeated overflow retries can count the same source occurrence more than once; this is a loss indicator, not a count of unique missing facts. +The outbox defaults to 10,000 events and 256 MiB of canonical event JSON. Transactional counters +enforce both caps without scanning every event on each ingest. If a new projected event does not fit, +Maple drops it and continues warehouse ingestion. Existing staged and ready events stay intact. The +OTLP response carries `x-maple-eventing-dropped` with the number of dropped projection attempts. +Health includes a durable `deliveryGap` with a generation, a cumulative dropped-event count, and the +last drop time. Repeated overflow retries can count the same source occurrence more than once. Treat +the count as a loss indicator, not a count of unique missing facts. -A consumer with an unaccepted gap receives HTTP 409 with the `EventConsumerDeliveryGap` error, current generation and dropped count. Existing leases can still be acknowledged. After investigating the gap, an operator may explicitly accept the current generation for a consumer: +A consumer with an unaccepted gap gets HTTP `409` with a JSON body: `error` +(`@maple/cli/eventing/EventConsumerDeliveryGap`), `message`, `consumerId`, `generation`, and +`droppedEvents`. Existing leases can still be acknowledged. After investigating the gap, an operator +can accept the current generation for a consumer: ```http POST /local/eventing/consumers/accept-gap @@ -134,9 +147,12 @@ X-Maple-Maintenance-Token: {"consumerId":"automation","generation":1} ``` -A stale generation is rejected. Acceptance resumes delivery of retained events; it does not recover missing events. A new `latest` consumer intentionally skips existing history, including previous gaps. A `beginning` consumer must accept any recorded gap before claiming. +A stale generation is rejected. Acceptance resumes delivery of retained events. It does not recover +missing events. A new `latest` consumer skips existing history, including earlier gaps. A +`beginning` consumer must accept any recorded gap before claiming. -To free stranded or unwanted events, inspect the outbox first, then explicitly abandon 1–1,000 distinct event IDs: +To free stranded or unwanted events, inspect the outbox first, then abandon 1 to 1,000 distinct +event IDs: ```http POST /local/eventing/outbox/abandon @@ -146,10 +162,18 @@ X-Maple-Maintenance-Token: {"eventIds":["sha256:..."]} ``` -Abandonment drains admitted requests, validates every ID belongs to this tenant, and atomically deletes the selected staged or ready records. Any missing ID rejects the entire batch. It records another delivery gap and clears tenant leases, so consumers cannot acknowledge a deleted batch. Consumers must accept the new generation before claiming again. This operation loses delivery history deliberately; ordinary inspection and acknowledgement never abandon staged records. Reconcile or replay source facts separately when needed. - -The initial eventing control schema is version 1. Its version and DDL digest are recorded in the local schema gate. Snapshot validation rejects staged source-backed rows with missing or malformed fingerprints. - -Consumer cursors and leases are part of the same SQLite backup as projection and outbox state. -Consumer mutations enter the server admission gate, so checkpoint exclusivity cannot capture a -half-applied claim or acknowledgement. +Abandonment drains admitted requests, checks that every ID belongs to this tenant, and atomically +deletes the selected staged or ready records. One missing ID rejects the whole batch. Abandonment +records another delivery gap and clears the tenant's leases, so no consumer can acknowledge a deleted +batch. Consumers must accept the new generation before claiming again. This operation discards +delivery history on purpose. Ordinary inspection and acknowledgement never abandon staged records. +Reconcile or replay source facts separately when needed. + +The eventing control schema is version 1 (`LOCAL_CONTROL_SCHEMA_VERSION` in +`apps/cli/src/server/local-schema-version.ts`). Its DDL digest is recorded in +`LOCAL_CONTROL_SCHEMA_HISTORY` (`apps/cli/src/server/local-schema-history.ts`). Snapshot validation rejects staged source-backed rows +with missing or malformed fingerprints. + +Consumer cursors and leases are in the same SQLite backup as projection and outbox state. Consumer +mutations pass through the server admission gate, so a checkpoint cannot capture a half-applied claim +or acknowledgement. diff --git a/docs/local-mode.md b/docs/local-mode.md index 2417ad4031..9a4f9640bd 100644 --- a/docs/local-mode.md +++ b/docs/local-mode.md @@ -1,8 +1,8 @@ # Local mode Local mode runs Maple as a single self-contained binary: OTLP ingest, an -embedded ClickHouse (chDB) store, a query API, and a UI — no cloud, no Tinybird, -no auth. It's for poking at telemetry on your own machine and for the +embedded ClickHouse (chDB) store, a query API, and a UI. No cloud, no Tinybird, +no auth. It is for poking at telemetry on your own machine and for the distributable "try Maple locally" bundle. Everything is single-tenant: every row is written under `org_id = "local"`, and @@ -20,7 +20,7 @@ Homebrew downloads the matching release bundle, verifies its checksum, installs `maple` and `libchdb.so` together in the Homebrew Cellar, and links `maple` onto your PATH. If Homebrew asks you to trust the third-party tap, run `brew trust Makisuo/tap` once and retry the install. The tap lives in -`Makisuo/homebrew-tap`, not this repo, so its platform coverage is set there — +`Makisuo/homebrew-tap`, not this repo, so its platform coverage is set there. Intel macOS currently installs via the manual installer below. Manual installer: @@ -29,9 +29,9 @@ Manual installer: curl -fsSL https://maple.dev/cli/install | sh ``` -(`maple.dev/cli/install` is [scripts/install.sh](../scripts/install.sh) served by -`apps/landing` — the build copies it to `public/cli/install`. The raw GitHub URL -`https://raw.githubusercontent.com/MapleTechLabs/maple/main/scripts/install.sh` works too.) +`maple.dev/cli/install` is [scripts/install.sh](../scripts/install.sh) served by +`apps/landing`. Its `sync:cli` script copies it to `public/cli/install`. The raw GitHub URL +`https://raw.githubusercontent.com/MapleTechLabs/maple/main/scripts/install.sh` works too. The manual installer detects your OS/arch, downloads the matching bundle from the latest GitHub release, verifies its checksum, installs the two files into @@ -39,11 +39,10 @@ the latest GitHub release, verifies its checksum, installs the two files into onto your PATH. Released targets: macOS (Apple Silicon & Intel) and Linux (x86_64 & arm64). -Intel macOS builds on GitHub's `macos-15-intel` runner, which is the scarcest -capacity in the pool — its tarball can land minutes after the others on a -release. Each build job publishes independently, so the rest of the release is -never held up. GitHub retires Intel macOS in Fall 2027; the target goes away -with it. +Intel macOS builds on GitHub's `macos-15-intel` runner, the scarcest capacity +in the pool, so its tarball can land minutes after the others. Each build job +publishes independently, so the rest of the release is never held up. GitHub +retires Intel macOS in Fall 2027, and the target goes away with it. Then: @@ -74,24 +73,23 @@ no credentials. Use it only on a trusted network or behind a TLS proxy with browser-compatible authentication (for example, a session cookie or HTTP authentication). The bundled UI does not inject a Bearer API key or propagate an entry-page query parameter to its API requests. Open the advertised URL from -another machine; the default UI hosted at -`local.maple.dev` always talks to the browser machine's loopback address and is -therefore not suitable for a remote local-mode server. +another machine. The default UI hosted at `local.maple.dev` always talks to the +browser machine's loopback address, so it cannot reach a remote local-mode server. By default `maple start` points you at the auto-updating dashboard hosted at -`local.maple.dev` (it talks back to this binary on loopback — see +`local.maple.dev`, which talks back to this binary on loopback (see [Where the UI comes from](#where-the-ui-comes-from)). `--offline` serves the copy bundled into the binary instead, which also avoids the browser's local-network permission prompt. The startup banner prints the right URL for the mode you chose. Query commands accept `--format table` for an aligned table instead of JSON, and -`--debug` to print the compiled SQL + per-query timing to stderr (stdout stays +`--debug` to print the compiled SQL and per-query timing to stderr (stdout stays clean JSON). Pin the backend with `maple use local|remote` (or `auto` to clear). Manual installer env overrides: `MAPLE_VERSION` (pin a release tag), `MAPLE_INSTALL_DIR` (bundle location, default `~/.maple/bin`), `MAPLE_BIN_DIR` -(PATH symlink location), `MAPLE_SKIP_CHECKSUM=1` (skip SHA-256 verification — -only for air-gapped mirrors without the `.sha256`; not recommended). +(PATH symlink location), `MAPLE_SKIP_CHECKSUM=1` (skip SHA-256 verification; +only for air-gapped mirrors without the `.sha256`, not recommended). ### Updating @@ -108,23 +106,23 @@ maple`. Manual-installer builds keep themselves current: - **Startup notice.** On any command, `maple` checks GitHub Releases for a newer - version — at most **once per 24h** (the result is cached in - `~/.maple/config.json` as `lastUpdateCheck` / `latestKnownVersion`, so every - other run stays instant and offline). When a newer release exists it prints a - one-line `update available` notice to stderr; it never changes behavior + version at most **once per 24h**. The result is cached in + `~/.maple/config.json` as `lastUpdateCheck` / `latestKnownVersion`, so other + runs stay instant and offline. When a newer release exists it prints a + one-line `update available` notice to stderr. It never changes behavior mid-run. The check is skipped for dev builds, non-interactive shells (CI/pipes), and the `--version`/`--help`/`update` paths. Opt out entirely with `MAPLE_NO_UPDATE_CHECK=1`. - **`maple update`** downloads the latest release bundle, verifies its SHA-256, - and installs it **in place** — an atomic rename over both files, safe even - though the running binary is being replaced (the install dir's `cp`-based - installer can't overwrite a running executable; the rename swaps the directory - entry while the live process keeps its old inode). It then clears the macOS - quarantine flag. Restart any running `maple start` afterward. - - `maple update --check` — report current vs. latest without installing. - - `maple update --tag ` — install a specific release (e.g. `v0.6.0`); also - the way to downgrade. (Named `--tag`, not `--version`, because the CLI - reserves `--version` for printing the binary version.) + and installs it **in place** with an atomic rename over both files. This is + safe while the binary is running: a `cp`-based install can't overwrite a + running executable, but the rename swaps the directory entry while the live + process keeps its old inode. It then clears the macOS quarantine flag. + Restart any running `maple start` afterward. + - `maple update --check`: report current vs. latest without installing. + - `maple update --tag `: install a specific release (e.g. `v0.6.0`). + This is also how to downgrade. The flag is `--tag` because the CLI + reserves `--version` for printing the binary version. This is the same artifact the installer fetches, so `maple update` and re-running `curl … | sh` are interchangeable. @@ -145,8 +143,8 @@ curl -fsSL https://maple.dev/cli/uninstall | sh The manual uninstaller removes the `maple` symlink and the `~/.maple/bin` bundle. Your data dir (`~/.maple/data`) is kept unless you confirm its removal -when prompted. Honors the same `MAPLE_INSTALL_DIR` / `MAPLE_BIN_DIR` overrides -as the installer. +when prompted (or set `MAPLE_REMOVE_DATA=1`). It honors the same +`MAPLE_INSTALL_DIR` / `MAPLE_BIN_DIR` overrides as the installer. If you migrate from the manual installer to Homebrew, run the manual uninstaller or remove the old PATH symlink so your shell resolves Homebrew's `maple`. @@ -155,19 +153,19 @@ or remove the old PATH symlink so your shell resolves Homebrew's `maple`. There is a single binary, `maple`, compiled from **`apps/cli`** (package `@maple/cli`, Effect + Bun) with `bun build --compile`. It is both the CLI and -the server, and it talks to the embedded ClickHouse engine **directly via -`bun:ffi`** — no subprocess, no second language at the front: +the server. It talks to the embedded ClickHouse engine **directly via +`bun:ffi`**, with no subprocess and no second language in front: | Concern | Where | How | | -------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| CLI commands | `apps/cli/src/commands` | `maple services`, `traces`, `errors`, … run against **either** the local server **or** a remote workspace — every command bottoms out at the shared `WarehouseExecutor`, and only the executor layer swaps per [mode](#local-vs-remote-mode). | -| `maple start` server | `apps/cli/src/server/serve.ts` | A `Bun.serve` hosting OTLP/HTTP ingest (`POST /v1/{traces,logs,metrics}`), the query API (`POST /local/query`), and the bundled SPA — all on one port. | +| CLI commands | `apps/cli/src/commands` | `maple services`, `traces`, `errors`, … run against **either** the local server **or** a remote workspace. `apps/cli/src/core/operations.ts` picks the path per [mode](#local-vs-remote-mode). | +| `maple start` server | `apps/cli/src/server/serve.ts` | A `Bun.serve` hosting OTLP/HTTP ingest (`POST /v1/{traces,logs,metrics}`), the query API (`POST /local/query`), and the bundled SPA, all on one port. | | Embedded ClickHouse | `apps/cli/src/server/chdb.ts` | `dlopen`s `libchdb` via `bun:ffi` (the `chdb_*` accessor C API) and holds a single connection for the process. | | OTLP → rows | `apps/cli/src/server/otlp/` | Decodes OTLP protobuf/JSON (protobufjs) and encodes each signal to per-table NDJSON, matching the generated `local-inserts.json` schema exactly. Ported from the production Rust encoders so row shapes can't diverge. | | UI (SPA) | `apps/local-ui` (Vite + React) | Hooks compile queries with `CH.compile(...)` and POST to `/local/query`. The same build is deployed to `local.maple.dev` (the default) **and** inlined into the binary as the `--offline` fallback (see [release bundle](#release-bundle)); it picks its query base URL from `window.location` at runtime (see [Where the UI comes from](#where-the-ui-comes-from)). | chDB allows exactly one connection per process and isn't safe to call -concurrently — so the long-lived `maple start` process owns the connection, and +concurrently. The long-lived `maple start` process owns the connection, and short-lived query commands (`maple traces`, …) reach it over HTTP via [`executeLocalQuery`](../packages/query-engine/src/local.ts). `bun:ffi` calls are synchronous and serialize naturally on the single JS thread, which preserves @@ -178,7 +176,7 @@ chDB's single-writer requirement. The on-disk store at `~/.maple/data` is guarded by two sentinels beside it (`apps/cli/src/server/store-version.ts`): -- **`maple-store-version.json`** — the chDB version and versioned Maple schema +- **`maple-store-version.json`**: the chDB version and versioned Maple schema identity that bootstrapped the store. A different chDB build can't be trusted to reload another's persisted materialized views (it may crash the C++ runtime natively, which JS can't catch), so `maple start` **refuses up front** when the @@ -186,14 +184,24 @@ The on-disk store at `~/.maple/data` is guarded by two sentinels beside it creation provenance, a full schema digest, and an `active`/`staging` state. Recover a store with an unsupported schema using `maple start --reset` only when losing its live telemetry is acceptable. -- **`maple-store-open`** — a clean-shutdown sentinel (not a concurrency lock; the - PID file already guards that). It's written right after chDB opens and removed - as the last step of a clean close. If `maple start` finds it still present over - a populated store, the previous server died without closing cleanly and the - store may be inconsistent — reopening could crash chDB natively. Rather than - risk the crash, `maple start` **auto-wipes the store and bootstraps fresh**, - printing a warning. Local telemetry data is **not recoverable** after an - unclean kill of chDB; re-ingest to repopulate. +- **`maple-store-open`**: a clean-shutdown sentinel (not a concurrency lock; the + PID file already guards that). It is written right after chDB opens and removed + as the last step of a clean close. If `maple start` finds it over a populated + store, the previous server died without closing cleanly and the store may be + inconsistent. Reopening it could crash chDB natively, so `maple start` never + reopens it. `--on-dirty-store` picks the recovery: + - `fail` (default): refuse to start and name the next command. That is + `maple restore --yes` when a checkpoint exists, otherwise + `maple start --reset`. + - `restore-checkpoint`: restore the last checkpoint and quarantine the dirty + store. + - `wipe`: discard the live telemetry, keep the checkpoints, and bootstrap + fresh. + + Live telemetry since the last checkpoint is **not recoverable** after an + unclean kill of chDB. `maple start` refreshes a checkpoint every 30 minutes by + default (`--checkpoint-interval`, `off` to disable). `maple checkpoint` + creates one on demand. ### Versioned local-store migrations @@ -269,7 +277,7 @@ Structural identities live in the append-only history at current identity against the history tip, preserves the base branch's prior entries in CI, and requires every historical identity to reach the current one through registered migration edges. Changing a schema digest or manifest -therefore requires a new versioned entry and executable edge together. +therefore requires a new versioned entry and an executable edge together. Everything that pairing demands except the DDL is derived from the new version number, so `bun run local-schema:bump ` writes it: the retained snapshot, @@ -311,7 +319,7 @@ wraps line-delimited rows into a JSON array, so it always needs `FORMAT JSONEachRow`. `CH.compile(...)` appends `FORMAT JSON`, so the handler (`forceJsonEachRow` in `apps/cli/src/server/serve.ts`) strips any trailing `FORMAT ` the client sent and re-appends `FORMAT JSONEachRow`. Clients -therefore POST `compiled.sql` verbatim — no client-side format rewriting. +therefore POST `compiled.sql` verbatim, with no client-side format rewriting. ## Where the UI comes from @@ -319,10 +327,10 @@ The dashboard SPA is a single build served two ways, and it decides which `/local/query` base URL to use from `window.location` (`localApiBase()` in [apps/local-ui/src/lib/constants.ts](../apps/local-ui/src/lib/constants.ts)): -- **Default — `local.maple.dev`.** `maple start` points you at the SPA deployed to - `local.maple.dev` (a Cloudflare worker, `apps/local-ui/alchemy.run.ts`). This - decouples UI updates from binary releases: ship a UI fix by deploying, no new - binary. Because that page is a _public_ origin, its queries to +- **Default: `local.maple.dev`.** `maple start` points you at the SPA deployed to + `local.maple.dev` (a Cloudflare Worker, `apps/local-ui/src/worker.ts`, declared + in the root `alchemy.run.ts`). This decouples UI updates from binary releases: + a UI fix ships with a deploy, not a new binary. Because that page is a _public_ origin, its queries to `http://127.0.0.1:/local/query` are a **public → loopback** request, which trips the browser's **Private Network Access** gate. The server answers the preflight with `Access-Control-Allow-Private-Network: true` only when the @@ -331,17 +339,17 @@ The dashboard SPA is a single build served two ways, and it decides which still show a one-time "wants to access devices on your local network" prompt; Safari/Firefox differ. The banner encodes the bound port as `?port=` and adds `maple-local-api=loopback`, so custom hosted UI origins use the same routing. -- **`--offline` (and dev) — same origin.** The binary serves the bundled SPA from +- **`--offline` (and dev): same origin.** The binary serves the bundled SPA from its selected bind address, so queries are same-origin even through a LAN hostname or reverse proxy: no CORS, no Private Network Access, no permission prompt, and it works with no internet. In dev the Vite server proxies - `/local/*` to the binary, which is the same same-origin path. This is the - recommended escape hatch whenever the default path hits a browser prompt. + `/local/*` and `/v1/*` to the binary, which keeps the same same-origin path. + Use `--offline` whenever the default path hits a browser prompt. Because the remote UI auto-updates independently of the binary, keep the `/local/query` contract and the local chDB schema ([apps/cli/src/server/schema/local-schema.sql](../apps/cli/src/server/schema/local-schema.sql)) -backward compatible — a newer UI may run against an older binary. +backward compatible. A newer UI may run against an older binary. `MAPLE_LOCAL_UI_URL` overrides the default UI origin (e.g. point a binary at a locally served build for testing). The startup link marks that custom origin as @@ -358,16 +366,16 @@ their matching loopback address instead of the unusable `0.0.0.0` or `::`. No Rust toolchain needed. Run the server and the SPA dev server in two terminals: ```bash -# Terminal 1 — the server (OTLP ingest + query API + chDB) on :4318. +# Terminal 1: the server (OTLP ingest + query API + chDB) on :4318. # Needs libchdb: set MAPLE_LIBCHDB, or keep libchdb.so in ~/.maple/bin. bun run apps/cli/src/bin.ts start -# Terminal 2 — the Vite SPA dev server on :4319, proxying /local → :4318 +# Terminal 2: the Vite SPA dev server on :4319, proxying /local and /v1 → :4318 bun --filter @maple/local-ui dev ``` -Open . Vite proxies `/local/*` to the server (override the -target with `MAPLE_LOCAL_URL`). +Open . Vite proxies `/local/*` and `/v1/*` to the server +(override the target with `MAPLE_LOCAL_URL`). Query from the CLI against the same server: @@ -414,21 +422,21 @@ maple auth logout # revoke browser-issued credentia Env overrides: `MAPLE_API_URL`, `MAPLE_API_TOKEN`, `MAPLE_LOCAL_URL`, `MAPLE_LOCAL_BIND_HOST`, and `MAPLE_LOCAL_ADVERTISE_HOST`. -**How queries route.** Local mode compiles the pipe → SQL client-side and POSTs -it to `/local/query`. Remote mode does **not** compile pipes at all: it calls +**How queries route.** Local mode compiles the pipe to SQL client-side and POSTs +it to `/local/query`. Remote mode does **not** compile pipes at all. It calls Maple's public v2 API (`/v2/traces/*`, `/v2/logs/*`, `/v2/services`, -`/v2/service_map`, `/v2/metrics`) with the API key `maple auth login` stored, -and maps each response into the same output type the local path produces. The +`/v2/service_map`, `/v2/metrics`, `/v2/error_issues`) with the API key +`maple auth login` stored, and maps each response into the same output type the local path produces. The branch lives in `apps/cli/src/core/operations.ts`; the v2 implementations are in `core/remote-ops.ts`. This replaced a generic `POST /api/tinybird/query` endpoint that let the client -name a pipe and have the server compile it. That endpoint has been retired — a -pipe name is an internal compiler detail, not a public contract, and shipping it -as one meant every CLI binary pinned the server's query catalog. +name a pipe and have the server compile it. That endpoint is retired. A pipe +name is an internal compiler detail, and treating it as a public contract meant +every CLI binary pinned the server's query catalog. **Some commands are local-only.** Where v2 has no equivalent, the command fails -with the reason rather than returning a narrower answer: +with the reason instead of returning a narrower answer: | Command | Why it needs local mode | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------- | @@ -447,7 +455,7 @@ sample traces. It reads Maple's triage issues rather than raw error events, so a fingerprint no sweep has turned into an issue fails instead of showing traces. Remote mode also inherits v2's pagination: lists cap at 100 rows per page and -seek by opaque cursor, so `--offset` is rejected rather than silently ignored. +seek by opaque cursor, so `--offset` is rejected instead of silently ignored. ### Seeding data @@ -455,16 +463,15 @@ Send OpenTelemetry to the server's OTLP/HTTP endpoints (`POST /v1/{traces,logs,metrics}`, protobuf or JSON, optionally gzip-encoded). Most OTLP exporters default to protobuf and work out of the box. -For OTLP/JSON, `traceId`/`spanId`/`parentSpanId` follow the OTLP/JSON convention -— **hex strings** (32 chars for a trace id, 16 for a span id), the spec's +For OTLP/JSON, `traceId`/`spanId`/`parentSpanId` follow the OTLP/JSON convention: +**hex strings** (32 chars for a trace id, 16 for a span id). This is the spec's deliberate deviation from proto3 JSON, and what every OTel language SDK emits. -Base64 of the raw bytes (24 and 12 chars — the proto3 JSON encoding, and what -the protobuf path decodes to internally) is also accepted; the two are told -apart by length, so hand-written payloads work either way. Ids that decode to -any other length are rejected with a `400` naming the field, rather than stored -mangled: a hex trace id read as base64 yields a _deterministic_ 24-byte value, -so the trace still self-joins and looks correct right up until you compare it -against the emitting service's logs. +Base64 of the raw bytes (24 and 12 chars, the proto3 JSON encoding) is also +accepted. The two are told apart by length, so hand-written payloads work either +way. Ids that decode to any other length are rejected with a `400` naming the +field instead of being stored mangled. A hex trace id read as base64 yields a +_deterministic_ 24-byte value, so the trace still self-joins and looks correct +until you compare it against the emitting service's logs. ## Release bundle @@ -476,13 +483,21 @@ maple # single Bun-compiled binary: CLI + ingest/query server + embedded libchdb.so # the chDB engine (~320 MB), downloaded from chdb-io/chdb-core releases ``` -The build (1) builds the SPA, (2) inlines `apps/local-ui/dist` into -`apps/cli/src/server/ui-embed.gen.ts` so `bun build --compile` bakes it into the -binary as the `--offline` fallback (the default UI is served from -`local.maple.dev`), (3) compiles `apps/cli`, and (4) downloads the matching `libchdb` beside -the binary. At runtime `maple` `dlopen`s the sibling `libchdb` (resolved relative -to its own path), so keep both files in the same directory — no `LD_LIBRARY_PATH` -or rpath tricks. +The build: + +1. builds the workspace packages consumed through gitignored `dist/` paths; +2. builds the SPA; +3. inlines `apps/local-ui/dist` into `apps/cli/src/server/ui-embed.gen.ts`, so + `bun build --compile` bakes it into the binary as the `--offline` fallback + (the default UI is served from `local.maple.dev`); +4. compiles `apps/cli`; +5. downloads the pinned `libchdb` (`LIBCHDB_VERSION`, checksum-verified) beside + the binary; +6. restores the committed `ui-embed.gen.ts` stub. + +At runtime `maple` `dlopen`s the sibling `libchdb`, resolved relative to its own +path. Keep both files in the same directory. No `LD_LIBRARY_PATH` or rpath +tricks are needed. ```bash scripts/build-local-binary.sh # full 2-file bundle into ./dist diff --git a/docs/local-telemetry-archives.md b/docs/local-telemetry-archives.md index b737a54245..4a07ac498a 100644 --- a/docs/local-telemetry-archives.md +++ b/docs/local-telemetry-archives.md @@ -2,15 +2,13 @@ Maple's embedded chDB store is a bounded **hot store**: it retains recent telemetry (logs and traces for 30 days, metrics for 90 days) for fast local -querying. Local telemetry archives extend Maple with **long-term, portable -Parquet storage** exported from immutable checkpoints, queryable independently -with DuckDB — without reloading history into the live store or running a second -always-on database. +querying. Local telemetry archives add **long-term, portable Parquet storage** +exported from immutable checkpoints. You query them with DuckDB, without +reloading history into the live store or running a second always-on database. -This document is the operator and architecture guide for local archives. It -covers the model, the happy path, every major off-happy-path outcome, the -independent query path, the full tuning configuration reference, calibration, -and the directory and manifest layouts. +This is the operator and architecture guide for local archives: the model, the +happy path, the off-happy-path outcomes, querying, the tuning reference, +calibration, and the directory and manifest layouts. ## What archives are (and are not) @@ -26,8 +24,9 @@ and the directory and manifest layouts. **Are not:** - A live export endpoint. v1 archives are created explicitly by an operator. -- Automatic hot-store pruning. Existing chDB TTLs govern the hot store; archives - do not delete from it. +- Automatic hot-store pruning. Existing chDB TTLs govern the hot store, and + creating an archive does not delete from it. Removing an archived day from the + live store is a separate, explicit `maple archive retire-live` step. - Archive rehydration into the Maple UI. Historical data is queried in DuckDB, not reloaded into the dashboard. - A second always-running database. Archives are files; DuckDB opens them on @@ -36,9 +35,9 @@ and the directory and manifest layouts. ## The six signals Archives export exactly the six **raw** telemetry tables. Aggregation and -materialized-view tables are deliberately excluded: they are rebuildable from -raw telemetry and would balloon archive volume without preserving any fact the -raw tables do not already carry. +materialized-view tables are excluded on purpose. They are rebuildable from raw +telemetry and would balloon archive volume without adding any fact the raw +tables lack. | Signal (table / directory name) | Event-time column used for the UTC-day range | | ------------------------------- | -------------------------------------------- | @@ -49,25 +48,24 @@ raw tables do not already carry. | `metrics_histogram` | `TimeUnix` | | `metrics_exponential_histogram` | `TimeUnix` | -Each signal drives fixed half-open UTC-day range semantics. Production queries -implement that range with UTC `toDate(...) = ` and per-hour -`toHour(...)` predicates, equivalent to `eventTime >= start AND eventTime < end` -for valid timestamps. +Each signal uses a fixed half-open UTC-day range. The export queries implement it +with UTC `toDate(...) = ` and per-hour `toHour(...)` predicates, +equivalent to `eventTime >= start AND eventTime < end` for valid timestamps. ## Architecture ```text Live Maple store (chDB) Archive volume (operator-configured, SEPARATE from data/) data/ / - backups/ logs/ - state.json 2026-06-01/ - snapshots// active.json ← atomic active pointer (formatVersion 1) - backup/ generations// - manifest.json manifest.json ← generation manifest (formatVersion 3) - pins//.json shards/HH-NNNN.parquet ← one or more shards per hour - operations/active/ catalog.jsonl ← canonical rebuildable JSONL index - quarantine/ traces/ ... - retiring/ building// (in-progress; owned temp output) + backups/ logs/ + state.json 2026-06-01/ + snapshots// active.json ← atomic active pointer (formatVersion 1) + backup/ generations// + manifest.json manifest.json ← generation manifest (formatVersion 3) + pins//.json shards/HH-NNNN.parquet ← one or more shards per hour + operations/active/ catalog.jsonl ← canonical rebuildable JSONL index + quarantine/ traces/ ... + retiring/ building// (in-progress; owned temp output) quarantine/ building-/ (retained pre-publication debris) calibration/ (calibration ownership + samples) @@ -90,14 +88,14 @@ archive into (or beneath) the live store. The only proven safe source for an archive is a native chDB checkpoint restored into sacrificial scratch. A raw copy of the live data directory is unsafe: it captures an inconsistent on-disk state, may include half-written merges, and -races concurrent ingest. A checkpoint, by contrast, is a validated, consistent -snapshot. Archive export restores one checkpoint into a private scratch chDB -(reusing the same scoped instance that checkpoint validation uses), exports from -it, then removes the scratch. The live store is never opened for export. +races concurrent ingest. A checkpoint is a validated, consistent snapshot. +Archive export restores one checkpoint into a private scratch chDB (the same +scoped instance that checkpoint validation uses), exports from it, then removes +the scratch. The live store is never opened for export. -A consequence: archive export holds the **maintenance lock** so it cannot overlap -checkpoint creation, restore, or reset. This is by design — the two operations -share one sacrificial chDB and must serialize. +As a consequence, archive export holds the **maintenance lock** so it cannot +overlap checkpoint creation, restore, or reset. The two operations share one +sacrificial chDB and must serialize. ### Why generations supersede instead of deduplicating by TraceId @@ -105,17 +103,17 @@ There is no universal deduplication key across the six raw tables. `TraceId` is shared by many spans, may be absent from logs, and does not exist on metrics. An archive therefore seals a fixed UTC-day range into an immutable **generation**. Late-arriving telemetry for an already-sealed day creates a **new generation** -that structurally supersedes the old one. The `active.json` pointer atomically -selects the new generation; the old generation is retained on disk but never -returned to listings or queries. This avoids scanning all generations to dedup -and makes each generation independently reproducible. +that supersedes the old one. The `active.json` pointer atomically selects the +new generation. The old generation stays on disk but is never returned to +listings or queries. This avoids scanning all generations to dedup, and each +generation stays independently reproducible. ### Separation of logical chunks, physical shards, and row groups Three distinct units, all configurable and calibratable: - A **logical chunk** is a provisioning target (the `targetChunkBytes` tuning - value); it is not a hard limit. + value), not a hard limit. - A **physical shard** is one Parquet file, bounded by `maxShardRows` and `maxShardBytes`. In v1, each shard covers one UTC hour within the sealed day; if a single hour exceeds `maxShardBytes` uncompressed, it is recursively @@ -130,16 +128,16 @@ Archive export holds Maple's **maintenance lock** so it cannot overlap checkpoin creation, restore, or reset. Inside the lock, it acquires a **persistent pin** on the source checkpoint so retention cannot delete the snapshot between resolution and export. A stale pin (e.g. from a crashed archive that never -released it) safely over-retains data rather than risking deletion. The pin is +released it) over-retains data instead of risking deletion. The pin is released after the generation is durable. Calibration pins use the purpose `archive-calibrate:` so they are unambiguous and operation-scoped. ## Commands -`maple archive` has eight operator-facing subcommands (`create`, `list`, -`rebuild`, `reconcile`, `gc`, `calibrate`, `retire-live`, `expire`) plus the internal -`calibrate-session` and `calibrate-run` commands used by calibration and its -fault probes. There are no short flags anywhere in this command tree. Root +`maple archive` has nine operator-facing subcommands (`create`, `list`, +`verify`, `rebuild`, `reconcile`, `gc`, `calibrate`, `retire-live`, `expire`) +plus the internal `calibrate-session` and `calibrate-run` commands used by +calibration and its fault probes. There are no short flags anywhere in this command tree. Root flags fall back to `~/.maple` defaults when omitted. | Flag | Default | @@ -189,7 +187,7 @@ maple archive list --output paths --signal traces # machine-readable paths maple archive list --output json # full JSON ``` -`--output` modes (`summary` is default, only `list` has this flag): +`--output` modes (`summary` is the default; only `list` has this flag): - `summary`: one line per active generation: signal, range, rows, shards, short generation id. @@ -199,8 +197,14 @@ maple archive list --output json # full JSON - `json`: the full `listActiveGenerations` object, pretty-printed. `list` verifies every shard's actual SHA-256 and byte size against the manifest -before returning it; a tampered shard fails closed (the affected range surfaces -in `errors`, other ranges still list). Only the active generation is exposed. +before returning it. A tampered shard fails closed: the affected range surfaces +in `errors`, and other ranges still list. Only the active generation is exposed. + +### `maple archive verify` + +Stream every active shard and check its SHA-256 with bounded memory. `--signal` +limits it to one signal. It prints the shard, generation and byte counts it +verified. ### `maple archive rebuild ` @@ -213,7 +217,8 @@ recovering from a truncated or missing catalog without rescanning Parquet bytes. Remove one sealed UTC day from all six live raw telemetry tables. The CLI first reconciles interrupted archive work, then asks the running server to close ingest/query admission, drain accepted requests, re-hash the frozen active -generation, and compare a canonical day-wide content digest—not only counts. +generation, and compare a canonical day-wide content digest as well as counts. +`--port` (default `4318`) names the running server. The server durably commits the retired day and its archive evidence to `.retired-days.json` before deleting any live row. That authority is @@ -225,7 +230,7 @@ SQL writes through `/local/query` are refused before execution, preventing insert-trigger materialized views from retaining a late contribution. A crash before the ledger commit leaves the live day intact; a crash after it is repaired by replay. The default `--sealing-lag-hours 24` can be increased but -never bypassed accidentally. The command refuses to run without `--apply`. +not bypassed by accident. The command refuses to run without `--apply`. The command does not hard-code an active-data window. Scheduling policy chooses which eligible day to retire (for example, the deployment may configure 60, @@ -234,8 +239,9 @@ it independently with `maple start --minimum-raw-telemetry-retention-days N`. That setting is durable beside the data directory, survives reset/restore, and can only be increased; omitting it on later launches preserves the configured value. Maple validates the live table TTLs before persisting the request and -never shortens a higher schema TTL. `N` must be between 90 and 3650 days, and operators should keep it strictly above -their active rotation window so TTL cannot delete a day before archival. +never shortens a higher schema TTL. `N` must be between 90 and 3650 days. Keep it +strictly above the active rotation window so TTL cannot delete a day before +archival. ### `maple archive expire --apply` @@ -255,7 +261,7 @@ Reconcile an interrupted `create` or `gc` operation to its intended state anything. - Apply: execute the decision function's verdict. -The decision is one of: `NoOp` (nothing active), `FailClosed` (unsafe state — +The decision is one of: `NoOp` (nothing active), `FailClosed` (unsafe state: zero mutation, exits non-zero), `CreateVerifyComplete`, `CreateAbortPrepublication`, `CreateFinishPublication`, `GcVerifyComplete`, or `GcResume`. A subsequent `create` also runs this reconciliation automatically as its first step. @@ -275,7 +281,7 @@ maple archive gc --keep 0 --dry-run # preview reclaiming all superseded - `--dry-run`: plan only, no mutation. If an operation is active in `operations/active/`, the dry run reports the blocker and reclaims nothing. -GC is conservative to a fault: it verifies every generation's manifest and shard +GC is deliberately conservative. It verifies every generation's manifest and shard checksums up front, excludes any signal/range whose catalog is not provably reconstructable or whose active pointer is missing, deletes by tombstone-rename (never in-place recursive delete), persists progress after every target, and @@ -382,8 +388,8 @@ PRAGMA temp_directory='/Volumes/External/duckdb-spill'; ## Tuning configuration -The tuning knobs are centralized, documented, and overridable. Defaults are the -measured research baselines — **not universal constants**. A deployment should +The tuning knobs live in one place and can be overridden. Defaults are the +measured research baselines, **not universal constants**. A deployment should calibrate against its checkpoint, archive volume, chDB version, and memory budget with `maple archive calibrate`. @@ -415,14 +421,14 @@ above), so a generation is reproducible and deployment drift is visible. `maple archive calibrate --write-config ` writes a **versioned calibration config document** (`formatVersion: 3`, mode `0o600`) with strict, exact-key -schema. It is a complete evidence record, not just the numbers, and the loader -re-derives every aggregate from the recorded evidence rather than trusting it. +schema. It records the full evidence as well as the numbers, and the loader +re-derives every aggregate from that evidence instead of trusting it. Top-level keys (all required; unknown keys rejected): | Key | Contents | | ----------------------- | ---------------------------------------------------------------------- | | `formatVersion` | `3` | -| `checkpoint` | `{ checkpointId, manifestFingerprint }` — the single source snapshot | +| `checkpoint` | `{ checkpointId, manifestFingerprint }`: the single source snapshot | | `candidateMatrix` | The exact four-candidate matrix evaluated | | `requiredSignals` | The exact six-signal set | | `budget` | The full `CalibrationBudget` the run used (see below) | @@ -440,9 +446,9 @@ Top-level keys (all required; unknown keys rejected): | `measuredAt` | Canonical UTC ISO-8601 timestamp | | `note` | Human-readable summary | -Each `results` entry carries a `sample` scope — `{ checkpointId, +Each `results` entry carries a `sample` scope (`{ checkpointId, checkpointManifestFingerprint, rangeDate, role, startRow, requestedRows, -rowCount }` — binding that measurement to one immutable checkpoint/range and an +rowCount }`) binding that measurement to one immutable checkpoint/range and an exact ordered-row window. Every training sample is `role: "training"`, `startRow: 0`; every held-out sample is `role: "held-out"`, `startRow: sampleRows`. The loader proves all scopes share one checkpoint/range, that the @@ -451,8 +457,8 @@ two windows are disjoint, and that actual `rowCount` equals `requestedRows` unrepresentative and cannot produce a loadable recommendation. `heldOut` and every complete `heldOutAttempts` entry persist a descriptive -aggregate `worstCase` plus a `signalComparisons` array — one entry per signal in -canonical order. Each entry carries its own `scaleRatio`, six metric comparison +aggregate `worstCase` plus a `signalComparisons` array with one entry per signal +in canonical order. Each entry carries its own `scaleRatio`, six metric comparison records (adjusted prediction, observation, tolerance, relative delta, pass/fail), and a per-signal `passed` flag; raw metrics are not duplicated (the loader re-derives them from the training/held-out results by candidate + signal). The @@ -480,7 +486,7 @@ is exactly: 5. Archive-volume replacement or filesystem change 6. Material telemetry-shape change (row width, cardinality, signal mix) -A document containing only `formatVersion` + `effective` is **rejected** — all +A document containing only `formatVersion` + `effective` is **rejected**. All evidence fields are required, so a config cannot be hand-edited into existence. The loader recomputes the worst cases, the held-out comparisons, the tuning derivation, and the sample scopes, and rejects any field that does not match @@ -492,11 +498,11 @@ wrong checkpoint). `loadTuningConfig` opens the file with defense-in-depth against tampering and TOCTOU: -1. `lstat` first — refuse if not a regular file (rejects symlinks/devices). +1. `lstat` first: refuse if not a regular file (rejects symlinks/devices). 2. Size cap: `16 MiB` (`MAX_CONFIG_BYTES`). -3. `open` with `O_NOFOLLOW` — the kernel refuses a symlink at the final path. -4. `fstat` the fd — refuse if not a regular file. -5. **fd-identity check** — the opened fd's `dev`/`ino` must equal the pre-`lstat` +3. `open` with `O_NOFOLLOW`: the kernel refuses a symlink at the final path. +4. `fstat` the fd: refuse if not a regular file. +5. **fd-identity check**: the opened fd's `dev`/`ino` must equal the pre-`lstat` `dev`/`ino` (detects a swap between lstat and open). 6. Bounded read to exactly the fd's size; SHA-256 is computed over those exact bytes. @@ -558,7 +564,7 @@ throughput, and temp disk. The selected candidate is re-measured on a **larger, disjoint** row window through the same shared writer. Training covered ordered rows `[0, sampleRows)`; held-out covers `[sampleRows, sampleRows + 2*sampleRows)`, equivalently -`[N, 3N)` — strictly larger than training and non-overlapping. Both requested +`[N, 3N)`, strictly larger than training and non-overlapping. Both requested windows and observed cardinalities are recorded in every result's `sample` scope and in `samplePolicy`; the loader requires actual `N`/`2N` rows. @@ -603,8 +609,8 @@ an incomplete or over-budget attempt records `signalComparisons: []`, small/unrepresentative (a signal's training row count below `sampleRows`) → the command exits non-zero with `selected: null` and **no config is written**. The CLI throws before writing a config when there is no recommendation. -- **An impossible resource budget** is not a special case: it composes from the - above — every candidate is rejected and no config is written, with no change +- **An impossible resource budget** is not a special case. It follows from the + above: every candidate is rejected and no config is written, with no change to existing configuration and no temporary data left behind. The calibrator never redefines the operator's goals to make a candidate pass, @@ -643,7 +649,7 @@ source == archived) are enforced. per-column-sum digest; v2 moved to UTC epoch-nanosecond strings and a multiset digest but carried a bare `tuningConfigName`; **v3** replaces that with the SHA-256-bound structured `tuningConfig` identity. v3 readers reject v2/v1 -fail-closed and preserve the files — older archives must be re-exported, not +fail-closed and preserve the files. Older archives must be re-exported, not migrated in place. ### Active pointer (`active.json`, formatVersion 1) @@ -674,20 +680,20 @@ the sole branch logic for create/GC recovery. Given an inspection of the on-disk state, it returns one of: -- `NoOp` — nothing active. -- `FailClosed` — an unsafe/impossible topology (e.g. both building and final +- `NoOp`: nothing active. +- `FailClosed`: an unsafe/impossible topology (e.g. both building and final state present, a published generation with no manifest, a final generation before the manifest-written phase, an aborted operation still active). Zero mutation; exits non-zero. -- `CreateVerifyComplete` — a create reached `complete`; verify terminal +- `CreateVerifyComplete`: a create reached `complete`; verify terminal invariants only. -- `CreateAbortPrepublication` — an interrupted create that had not published; +- `CreateAbortPrepublication`: an interrupted create that had not published; move its owned building dir into retained quarantine, remove exact owned scratch, and release its exact pin. -- `CreateFinishPublication` — an interrupted create that **had** published; +- `CreateFinishPublication`: an interrupted create that **had** published; re-select the pointer and rebuild the catalog. -- `GcResume` — resume collecting a frozen GC target set. -- `GcVerifyComplete` — a GC reached `complete`; prove terminal invariants and +- `GcResume`: resume collecting a frozen GC target set. +- `GcVerifyComplete`: a GC reached `complete`; prove terminal invariants and retire the journal. A phase label is never proof: the decision and the terminal checks re-read @@ -707,7 +713,7 @@ between pin creation and the phase advance (when the record still shows `pinPath: null`) still releases the exact pin. A checkpoint-fingerprint mismatch fails closed and preserves the record. Reconciliation removes the owned dirs only after classifying them as real directories, and clears the record only -after the pin is confirmed released and both dirs confirmed absent — otherwise +after the pin is confirmed released and both dirs confirmed absent. Otherwise it preserves the record for retry. ### GC recovery @@ -757,14 +763,14 @@ idempotently confirms an already-absent target. an interrupted GC (frozen target set resumed). - **Operator intervention:** a `FailClosed` reconciliation (impossible topology or suspected corruption), a persistently corrupt active pointer, or a shard - that repeatedly exceeds bounds requires manual inspection. `archive reconcile ---dry-run` reports the verdict without mutating. + that repeatedly exceeds bounds requires manual inspection. + `archive reconcile --dry-run` reports the verdict without mutating. ## Capacity and resource model For a 4 GiB hot-store target, live store plus current and previous checkpoints is roughly 3x the live footprint. Checkpoint creation, scratch restore, and archive -building can temporarily raise aggregate working storage toward 4–5x. That is +building can temporarily raise aggregate working storage toward 4 to 5x. That is an aggregate across volumes, not a free-space requirement for one disk. Checkpoint validation and archive export share **one** sacrificial chDB, so archive export does not add a second concurrent `f(4)` memory term. diff --git a/docs/mcp-oauth.md b/docs/mcp-oauth.md index e11c87502a..ce23537689 100644 --- a/docs/mcp-oauth.md +++ b/docs/mcp-oauth.md @@ -6,39 +6,41 @@ clients, and use authorization code + S256 PKCE. ## Discovery and endpoints -- `GET /.well-known/oauth-protected-resource/mcp` -- `GET /.well-known/oauth-authorization-server` -- `POST /register` — dynamic registration for public clients; redirect URIs must use HTTPS or a +- `GET /.well-known/oauth-protected-resource/mcp` (also served without the `/mcp` suffix) +- `GET /.well-known/oauth-authorization-server` (also served with a `/mcp` suffix) +- `POST /register`: dynamic registration for public clients. Redirect URIs must use HTTPS or a loopback HTTP address. -- `GET /oauth/authorize` — validates the client, registered redirect URI, `resource`, `mcp:tools` - scope, and PKCE challenge before redirecting to the Maple approval page. Redirect URIs match - exactly except that HTTP loopback clients may use a different port at authorization time, as - required by RFC 8252. -- `POST /oauth/token` — accepts form-encoded `authorization_code` and `refresh_token` grants. -- `POST /oauth/revoke` — revokes an access token or an entire refresh-token family. +- `GET /oauth/authorize`: validates the client, the registered redirect URI, `resource`, the + `mcp:tools` scope, and the PKCE challenge, then redirects to the Maple approval page. Redirect + URIs match exactly, except that HTTP loopback clients may use a different port at authorization + time, as RFC 8252 requires. +- `POST /oauth/token`: accepts form-encoded `authorization_code` and `refresh_token` grants. +- `POST /oauth/revoke`: revokes an access token or an entire refresh-token family. -Unauthenticated MCP requests return `401` with a `WWW-Authenticate` challenge containing the -protected-resource metadata URL and required `mcp:tools` scope. +Unauthenticated MCP requests return `401` with a `WWW-Authenticate` challenge that carries the +protected-resource metadata URL and the required `mcp:tools` scope. ## Credentials Authorization codes are single-use and expire after five minutes. Access tokens are opaque, hashed Maple MCP keys with a one-hour expiry. Their metadata binds the approving user's roles, OAuth client, and exact MCP resource. Refresh tokens are opaque, hashed, valid for 30 days, and -rotate on every use. Reusing a rotated refresh token revokes the whole grant family. +rotate on every use. A grant family expires 90 days after it was first issued, however often it refreshes, +so the client re-consents once a quarter. Reusing a rotated refresh token revokes the whole grant +family. Manual MCP keys remain supported for clients without OAuth. They continue to use the existing `Authorization: Bearer ...` configuration and are isolated to the MCP server by `kind: "mcp"`. ## Rate limiting -Authenticated `POST /mcp` requests share one budget per credential — the internal key ID for -OAuth tokens and manual MCP keys, the user for dashboard sessions — of **120 requests per 10 -seconds**, partitioned by deployment stage. Exceeding it returns `429` with +Authenticated `POST /mcp` requests share one budget per credential (the internal key ID for +OAuth tokens and manual MCP keys, the user for dashboard sessions) of **120 requests per 10 +seconds**, partitioned by deployment environment. Exceeding it returns `429` with `{ "error": "rate_limited" }` and `Retry-After: 10`. The limiter fails open with `maple.rate_limit.outcome=failed_open` telemetry, like the `/v2` limiter documented in -[api-v2.md](api-v2.md#rate-limiting); the OAuth handshake endpoints above have their own -separate 60/60s budget. +[api-v2.md](api-v2.md#rate-limiting). The OAuth handshake endpoints above have their own +60 requests per 60s budget. ## Browser approval @@ -47,4 +49,5 @@ existing sign-in and active-workspace redirects protect this page. Approval capt workspace, user, and roles; changing workspace before approval changes the workspace bound to the issued grant. -Deploy migration `0018_brown_brood.sql` before enabling the OAuth routes. +The OAuth tables come from migration `packages/db/drizzle/20260721133909_brown_brood`, which the +prd deploy applies. diff --git a/docs/openrouter-tracing.md b/docs/openrouter-tracing.md index 95a807411e..81b39d5539 100644 --- a/docs/openrouter-tracing.md +++ b/docs/openrouter-tracing.md @@ -3,7 +3,7 @@ Maple attributes every OpenRouter request to its app page on openrouter.ai and tags it with the surface it came from, the org it ran for, and the session it belongs to. Separately, OpenRouter Broadcast can send each completion as an OTLP/HTTP trace to any backend that accepts JSON OTLP on -`/v1/traces` — including Maple's own ingest gateway. +`/v1/traces`, including Maple's own ingest gateway. References verified on August 4, 2026: @@ -13,41 +13,41 @@ References verified on August 4, 2026: ## Where OpenRouter Is Called -| Path | Client | Surfaces | -| ------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------- | -| `apps/ai/src/platform/Llm.ts` | Effect AI (`@effect/ai-openrouter`) | chat turns and autonomous investigation passes | -| `apps/api/src/mcp/__evals__/model.ts` | `@ai-sdk/openai-compatible` | MCP evals in CI — **not** attributed or tagged | +| Path | Client | Surfaces | +| ------------------------------------ | ----------------------------------- | ---------------------------------------------------------- | +| `apps/ai/src/platform/Llm.ts` | Effect AI (`@effect/ai-openrouter`) | chat turns, investigation passes, and pull request reviews | +| `apps/ai/src/mcp/__evals__/model.ts` | `@ai-sdk/openai-compatible` | MCP evals in CI, **not** attributed or tagged | -`apps/api` can also run on Cloudflare Workers AI instead (`MAPLE_LLM_PROVIDER=workers-ai`). None of -the attribution below applies on that path — the headers and tag fields are OpenRouter's, and -`resolveTriageModel` deliberately keeps them off the Workers AI branch. +`apps/ai` can also run on Cloudflare Workers AI (`MAPLE_LLM_PROVIDER=workers-ai`). None of the +attribution below applies on that path. The tag fields are OpenRouter's, and `resolveTriageModel` +keeps them off the Workers AI branch. ## App Attribution -`HTTP-Referer` is what creates the app page on openrouter.ai; a title on its own does nothing and -usage without a referer never appears in the rankings. Every caller sends the same URL and title on -purpose — the referer _is_ the app's identity, so a second value would mint a second app entry and +`HTTP-Referer` creates the app page on openrouter.ai. A title on its own does nothing, and usage +without a referer never appears in the rankings. Every caller sends the same URL and title on +purpose: the referer is the app's identity, so a second value would mint a second app entry and split the rankings. -| Header | Value | Set at | -| -------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------- | -| `HTTP-Referer` | `https://maple.dev` | `apps/api/src/platform/Llm.ts` (`OPENROUTER_APP_URL`) | -| `X-Title` / `X-OpenRouter-Title` | `Maple` | same, `OPENROUTER_APP_TITLE` / `appName` | +| Header | Value | Set at | +| -------------- | ------------------- | --------------------------------------------------------------------- | +| `HTTP-Referer` | `https://maple.dev` | `apps/ai/src/platform/Llm.ts` (`OPENROUTER_APP_URL`, `openRouterHttp`) | +| `X-Title` | `Maple` | same file, `OPENROUTER_APP_TITLE` | Per-app analytics then live at https://openrouter.ai/apps. ## Per-Request Tags -`resolveTriageModel(env, tags)` takes an optional `LlmCallTags` (`surface`, `orgId`, `sessionId`) -and folds it into the OpenRouter request body as route defaults, so every `LLM.request` / -`generate` / `stream` made with the returned model carries it without each call site threading it -through. +`resolveTriageModel(env, tags)` and `resolveReviewModel(env, tags)` take an optional `LlmCallTags` +(`surface`, `orgId`, `sessionId`, plus `turnId` and `workflowName` for spans). The OpenRouter +fields are folded into the model config, so every call made with the returned model carries them +without each call site threading them through. -| Field | Maple value | Where it shows up | -| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `user` | Maple org id | the `/activity` page, activity exports, and the `/generations` API. OpenRouter folds it into a hashed identity and never forwards it raw upstream. | -| `session_id` | the chat session id — `:inv-` for an investigation's passes — truncated to OpenRouter's 256-character limit | groups the requests of one conversation or investigation, and makes OpenRouter route the whole session to a single provider so prompt caches actually hit | -| `trace.trace_name` | `chat` or `bot` | forwarded to configured Broadcast destinations only — it does **not** appear in the OpenRouter dashboard | +| Field | Maple value | Where it shows up | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| `user` | Maple org id | the `/activity` page, activity exports, and the `/generations` API. OpenRouter folds it into a hashed identity and never forwards it raw upstream. | +| `session_id` | the chat session id (`:inv-` for an investigation's passes), truncated to OpenRouter's 256-character limit | groups the requests of one conversation or investigation, and makes OpenRouter route the whole session to one provider so prompt caches hit | +| `trace.trace_name` | `chat` or `bot` | forwarded to configured Broadcast destinations only. It does **not** appear in the OpenRouter dashboard. | The same session id goes onto Maple's own model-call spans as `maple_ai.session.id`, so a call and its Broadcast mirror land in one agent session. @@ -83,8 +83,8 @@ https:///v1/traces 6. Use OpenRouter's Test Connection action, then send a Maple chat message. OpenRouter only emits Broadcast traces for traffic under the OpenRouter account or workspace where -Broadcast is enabled. Maple has no BYOK path — every org's traffic runs on Maple's own -`OPENROUTER_API_KEY` — so Broadcast is configured once, on Maple's OpenRouter account. +Broadcast is enabled. Maple has no BYOK path: every org's traffic runs on Maple's own +`OPENROUTER_API_KEY`, so Broadcast is configured once, on Maple's OpenRouter account. ## Querying In Maple @@ -94,8 +94,8 @@ model, usage, and cost data. The tag fields arrive under OpenRouter's `trace.met Useful filters: ```text -trace.metadata.trace_name = "ai-triage" -session.id = "triage_error_" +trace.metadata.trace_name = "chat" +session.id = ":inv-" ``` If prompt or completion content should not leave OpenRouter, enable Privacy Mode for the OpenRouter @@ -107,7 +107,7 @@ content while still sending timing, model, token usage, cost, and metadata. The attribution and tagging contract is covered by: ```bash -bun run --cwd apps/api vitest run src/platform/Llm.test.ts +bun run --cwd apps/ai vitest run src/platform/Llm.test.ts ``` Those tests swap `FetchHttpClient.Fetch` for a capture and assert, on the outgoing request, that diff --git a/docs/otel-coverage-roadmap.md b/docs/otel-coverage-roadmap.md index 219ba7a0cd..40ec009d82 100644 --- a/docs/otel-coverage-roadmap.md +++ b/docs/otel-coverage-roadmap.md @@ -1,12 +1,16 @@ # OTel Coverage Roadmap +Status (2026-09-25): `LogsFilters` now accepts `attributeFilters`, `resourceAttributeFilters`, and +`minSeverity` (`packages/domain/src/query-engine.ts`), but the logs UI does not expose them yet. +GenAI data has a dedicated surface in `/agent-sessions`. Other items below are still open. + Tracks OpenTelemetry-standard capabilities Maple does **not** yet surface for filtering and -visualization. Maple ingests the full OTel triad and stores a near-complete span/log/metric model in -ClickHouse/Tinybird — but a lot of stored data (span events, span links, span kind, trace state, log +visualization. Maple ingests all three OTel signals and stores a near-complete span/log/metric model +in ClickHouse/Tinybird. Much of the stored data (span events, span links, span kind, trace state, log attributes, metric exemplars, exponential histograms) is never exposed in the product. -**Framing:** most high-value gaps need _zero_ ingestion work — the columns are already populated, they -just need query-engine + UI plumbing. Tier 3 items require changes to the Rust ingest encoder +**Framing:** most high-value gaps need _zero_ ingestion work. The columns are already populated and +only need query-engine and UI plumbing. Tier 3 items require changes to the Rust ingest encoder (`apps/ingest/src/telemetry.rs`). Effort key: **S** = schema + query + UI wiring · **M** = new query shape or visual component · @@ -19,66 +23,66 @@ Effort key: **S** = schema + query + UI wiring · **M** = new query shape or vis Before building, we confirmed which columns are actually populated. A "quick win" is only quick if data exists: -- **Span kind** — all 5 kinds present (Client/Server/Internal/Consumer/Producer). ✅ -- **Span status** — Ok / Unset / Error all present; large `Client/Unset` volume the current - `has_error` boolean can't separate from Ok. ✅ -- **Span events** — present, concentrated on error spans (exception events). ✅ -- **Span links** — **0 rows have links.** ❌ No producers emit them → #6 deprioritized until data exists. -- **Scope name** — ~100% populated on both traces and logs. ✅ -- **Log attributes** — ~100% populated. ✅ -- **Log severity** — numbers 5/9/13/17 present; severity _text_ casing is inconsistent - (`info` vs `Info`, `debug`) → confirms a severity-**number** range filter beats text matching. ✅ -- **Metric exemplars** — 6.7M on histograms, ~46K on sums. ✅ -- **Exponential histograms** — **0 rows.** ❌ → #10 deprioritized. -- **Explicit histograms** — 36.8M rows. ✅ +- **Span kind:** all 5 kinds present (Client/Server/Internal/Consumer/Producer). ✅ +- **Span status:** Ok / Unset / Error all present. The large `Client/Unset` volume can't be separated + from Ok by the current `has_error` boolean. ✅ +- **Span events:** present, concentrated on error spans (exception events). ✅ +- **Span links:** **0 rows have links.** ❌ No producers emit them, so #6 is deprioritized until data exists. +- **Scope name:** ~100% populated on both traces and logs. ✅ +- **Log attributes:** ~100% populated. ✅ +- **Log severity:** numbers 5/9/13/17 present. Severity _text_ casing is inconsistent (`info` vs + `Info`, `debug`), so a severity-**number** range filter beats text matching. ✅ +- **Metric exemplars:** 6.7M on histograms, ~46K on sums. ✅ +- **Exponential histograms:** **0 rows.** ❌ #10 deprioritized. +- **Explicit histograms:** 36.8M rows. ✅ --- -## Tier 1 — Quick wins (data already stored, surface only) - -- [ ] ~~**1. Span kind filter** (Server/Client/Producer/Consumer/Internal)~~ — ✗ **REJECTED (tried & reverted).** Built end-to-end and verified working (Client → 36 traces), but judged not useful enough for the sidebar. Don't rebuild without a clearer use case. -- [ ] ~~**2. Span status filter** (Ok / Unset / Error, not just `has_error`)~~ — ✗ **REJECTED (tried & reverted).** Same as #1 — the existing `has_error` toggle covers the common case; granular Ok/Unset/Error wasn't worth the sidebar space. -- [ ] **3. Log attribute filters** (arbitrary `log_attributes`) — _S–M, logs_ - `LogsFilters` has no `attributeFilters` — only service/severity/trace_id/body-search. Reuse `AttributeFilter` + `traces-shared.ts` operators. -- [ ] **4. Log severity range** (`>= ERROR`) — _S, logs_ - `severity_number` is stored but only exact-match on text is exposed. Add a `minSeverityNumber` filter. -- [ ] **5. Span events on the waterfall/timeline** — _M, trace visuals_ - `events_*` arrays stored and completely unrendered. Add event markers to `trace-timeline.tsx` / `span-hierarchy.tsx` + events list in span detail. Highest-visibility visual win. -- [ ] **6. Span links rendered** (clickable links to related traces) — _M, trace visuals_ — ⏸ **DEPRIORITIZED: 0 links in prod data.** `links_*` arrays stored, never shown. Revisit once a producer emits links. -- [ ] **7. Scope / instrumentation filter** (`scope_name` / `scope_version`) — _S, trace + log filtering_ +## Tier 1: quick wins (data already stored, surface only) + +- [ ] ~~**1. Span kind filter** (Server/Client/Producer/Consumer/Internal)~~. ✗ **REJECTED (tried and reverted).** Built end-to-end and verified working (Client → 36 traces), but judged not useful enough for the sidebar. Don't rebuild without a clearer use case. +- [ ] ~~**2. Span status filter** (Ok / Unset / Error, not just `has_error`)~~. ✗ **REJECTED (tried and reverted).** The existing `has_error` toggle covers the common case; granular Ok/Unset/Error wasn't worth the sidebar space. +- [ ] **3. Log attribute filters** (arbitrary `log_attributes`). _S–M, logs_ + Backend done: `LogsFilters` has `attributeFilters` and `resourceAttributeFilters`. The logs sidebar has no control for them yet. Reuse the trace-side `AttributeFilter` UI. +- [ ] **4. Log severity range** (`>= ERROR`). _S, logs_ + Backend done: `LogsFilters.minSeverity` filters on `severity_number` and is accepted by `apps/web/src/api/warehouse/logs.ts`. The `/logs` route does not expose it yet. +- [ ] **5. Span events on the waterfall/timeline.** _M, trace visuals_ + `events_*` arrays are stored and never rendered. Add event markers to `trace-timeline.tsx` / `span-hierarchy.tsx` and an events list in span detail. Highest-visibility visual win. +- [ ] **6. Span links rendered** (clickable links to related traces). _M, trace visuals_. ⏸ **DEPRIORITIZED: 0 links in prod data.** `links_*` arrays are stored, never shown. Revisit once a producer emits links. +- [ ] **7. Scope / instrumentation filter** (`scope_name` / `scope_version`). _S, trace + log filtering_ Stored on all three signals, filterable on none. Isolates one SDK/library. -- [ ] **8. Generic attribute filter UI builder** — _M, trace filtering_ - `attributeFilters` + `resourceAttributeFilters` already work end-to-end in the query engine (and `hasActiveFilters` counts them) — but the sidebar renders no control to add them. Wire a key/op/value row builder backed by `explore-attributes` autocomplete. +- [ ] **8. Generic attribute filter UI builder.** _M, trace filtering_ + `attributeFilters` and `resourceAttributeFilters` work end-to-end in the query engine, and `hasActiveFilters` counts them. Span detail attribute rows can open a filtered trace list (`trace-attribute-filter-provider.tsx`), but the sidebar has no control to add a filter. Wire a key/op/value row builder backed by `explore-attributes` autocomplete. -## Tier 2 — Medium (data stored, new query/visual work) +## Tier 2: medium (data stored, new query/visual work) -- [ ] **9. Exemplar → trace links on charts** — _M, metric visuals_ - `exemplars_trace_id/span_id/value/timestamp` stored per metric point, never read. Overlay exemplar dots on latency/line charts that deep-link to the trace — the canonical OTel metrics↔traces bridge. -- [ ] **10. Exponential-histogram heatmap** — _M–L, metric visuals_ — ⏸ **DEPRIORITIZED: 0 exp-histogram rows in prod.** Revisit once data exists; explicit-bucket histograms (#11) have 36.8M rows and are the better first target. -- [ ] **11. Explicit-bucket histogram distribution view** — _M, metric visuals_ - Histogram datasource has bounds + bucket counts; registry has a generic histogram/heatmap but nothing wired to OTel histogram buckets. -- [ ] **12. trace_state / sampling filter** — _S, trace filtering_ - `TraceState` + `SampleRate` stored; useful for "show only head-sampled" debugging. Lower demand. -- [ ] **13. GenAI / LLM semantic conventions** (`gen_ai.*`) — _M, filtering + visuals_ - Token usage, model, system land in the generic attrs map — no dedicated facets or token/cost visuals. Emerging, high-interest namespace. +- [ ] **9. Exemplar → trace links on charts.** _M, metric visuals_ + `exemplars_trace_id/span_id/value/timestamp` are stored per metric point and never read. Overlay exemplar dots on latency/line charts that deep-link to the trace. This is the standard OTel bridge from metrics to traces. +- [ ] **10. Exponential-histogram heatmap.** _M–L, metric visuals_. ⏸ **DEPRIORITIZED: 0 exp-histogram rows in prod.** Revisit once data exists. Explicit-bucket histograms (#11) have 36.8M rows and are the better first target. +- [ ] **11. Explicit-bucket histogram distribution view.** _M, metric visuals_ + The histogram datasource has bounds and bucket counts. The chart registry has a generic histogram/heatmap, but nothing is wired to OTel histogram buckets. +- [ ] **12. trace_state / sampling filter.** _S, trace filtering_ + `TraceState` and `SampleRate` are stored. Useful for "show only head-sampled" debugging. Lower demand. +- [ ] **13. GenAI / LLM semantic conventions** (`gen_ai.*`). _M, filtering + visuals_ + Partly done: `/agent-sessions` renders GenAI sessions, tool calls, and transcripts. The trace list still has no dedicated `gen_ai.*` facets or token/cost visuals. -## Tier 3 — Needs ingest-encoder changes (`apps/ingest/src/telemetry.rs`) +## Tier 3: needs ingest-encoder changes (`apps/ingest/src/telemetry.rs`) -- [ ] **14. Summary metrics** — _M._ Currently silently dropped at encode time. Add datasource + encode path if customers send Prometheus-summary-style data. -- [ ] **15. Span flags / dropped-attribute counts** — _M._ `flags` and all `dropped_*_count` hardcoded to 0; needed for spec-complete fidelity + "data loss" indicators. -- [ ] **16. Structured (nested) log body** — _M._ OTel AnyValue body is flattened to string; nested bodies lose shape. -- [ ] **17. Profiles signal** — _L._ No OTLP profiles ingestion at all. Large, separable effort. -- [ ] **18. Baggage persistence** — _L._ Parsed but not stored; low product value. +- [ ] **14. Summary metrics.** _M._ Silently dropped at encode time. Add a datasource and encode path if customers send Prometheus-summary-style data. +- [ ] **15. Span flags / dropped-attribute counts.** _M._ `flags` and all `dropped_*_count` are hardcoded to 0. Needed for spec-complete fidelity and "data loss" indicators. +- [ ] **16. Structured (nested) log body.** _M._ The OTel AnyValue body is flattened to a string, so nested bodies lose shape. +- [ ] **17. Profiles signal.** _L._ No OTLP profiles ingestion at all. Large, separable effort. +- [ ] **18. Baggage persistence.** _L._ Parsed but not stored. Low product value. --- ## Recommended build order -1. Filter quick-wins batch (Tier 1: #1, #2, #4, #7) — shared schema → query operator → facet → sidebar pattern. -2. Log attribute filters (#3) — logs filtering parity with traces. -3. Span events visualization (#5) — most visible "we have OTel data you can't see" gap. -4. Generic attribute filter UI builder (#8) — backend done; pure UI, multiplies value of every stored attribute. -5. Span links (#6) then exemplar links (#9) — the two OTel correlation bridges. +1. Filter quick-wins batch (Tier 1: #4, #7; #1 and #2 were rejected). Shared schema → query operator → facet → sidebar pattern. +2. Log attribute filters (#3): logs filtering parity with traces. +3. Span events visualization (#5): the most visible "we have OTel data you can't see" gap. +4. Generic attribute filter UI builder (#8): backend done, pure UI, multiplies the value of every stored attribute. +5. Span links (#6), then exemplar links (#9): the two OTel correlation bridges. 6. Histogram / exp-histogram visuals (#10, #11) and GenAI conventions (#13). 7. Tier 3 ingest work as separate, demand-driven efforts. @@ -88,9 +92,9 @@ data exists: - **Filter operator → SQL:** `packages/query-engine/src/traces-shared.ts`. - **Trace/log search + facets:** `packages/query-engine/src/observability/{search-traces,search-logs,explore-attributes}.ts`. - **DSL queries:** `packages/query-engine/src/ch/queries/*.ts` (+ export from `packages/query-engine/src/ch/index.ts`). -- **Filter UIs:** `apps/web/src/components/traces/traces-filter-sidebar.tsx`, `apps/web/src/components/logs/logs-filter-sidebar.tsx`, shared `filter-section.tsx`. +- **Filter UIs:** `apps/web/src/components/traces/traces-filter-sidebar.tsx`, `apps/web/src/components/logs/logs-filter-sidebar.tsx`, shared `apps/web/src/components/filters/filter-section.tsx`. - **Trace visuals:** `packages/ui/src/components/traces/{trace-timeline,span-hierarchy,flamegraph,flow-view}.tsx`, `apps/web/src/components/traces/span-detail-panel.tsx`. - **Chart registry / metric visuals:** `packages/ui/src/components/charts/registry.ts`; verify via `/lab/widgets`. - **Datasource schemas:** `packages/domain/src/tinybird/datasources.ts` (traces, logs, metric\_\* tables). - **Ingest encoder (Tier 3 only):** `apps/ingest/src/telemetry.rs`. -- **MCP parity:** `apps/api/src/mcp/tools/{search-traces,search-logs,explore-attributes}.ts`. +- **MCP parity:** `apps/ai/src/mcp/tools/{search-traces,search-logs,explore-attributes}.ts`. diff --git a/docs/otel-spec/README.md b/docs/otel-spec/README.md index 18f8f0a2c4..567f90794b 100644 --- a/docs/otel-spec/README.md +++ b/docs/otel-spec/README.md @@ -1,18 +1,18 @@ # OpenTelemetry Specification Map -An internal, source-linked map of the OpenTelemetry specifications, built for three uses: +An internal, source-linked map of the OpenTelemetry specifications. It has three uses: -1. **Spec-compliance checks** — especially for the ingest gateway (`apps/ingest`), which is an - OTLP server and must honor the server-side MUSTs in [otlp.md](otlp.md). -2. **Best-practices material** — the normative rules here are the raw input for an internal +1. **Spec-compliance checks**, mainly for the ingest gateway (`apps/ingest`). It is an OTLP + server and must honor the server-side MUSTs in [otlp.md](otlp.md). +2. **Best-practices material.** The normative rules here are the raw input for an internal best-practice skill (instrumentation hygiene, status semantics, attribute rules). -3. **Fact-checking reference** — every section in every file carries inline `Source:` deep links - to the official spec pages so claims can be re-verified as the spec moves. +3. **Fact-checking reference.** Every section in every file carries inline `Source:` deep links + to the official spec pages, so claims can be re-verified as the spec moves. **Snapshot:** researched 2026-07-05 against specification **v1.58.0** (released 2026-06-22, per the [spec CHANGELOG](https://github.com/open-telemetry/opentelemetry-specification/blob/main/CHANGELOG.md)). Method: each file was written from freshly fetched official pages (opentelemetry.io/docs/specs, -the specification/semconv/proto GitHub repos, W3C TRs) — not from model memory. Stability labels +the specification/semconv/proto GitHub repos, W3C TRs), not from model memory. Stability labels are the spec's own, recorded per section. ## Files @@ -32,47 +32,54 @@ are the spec's own, recorded per section. ## Maple's three compliance surfaces **1. Ingest gateway as an OTLP server** ([otlp.md](otlp.md) is the contract): -partial-success responses are a MUST (200 + `rejected_*` counts for partially-bad batches — -never fail the whole batch for a few bad items; 400 only for fully undecodable requests); -the retryable HTTP status set is exactly {429, 502, 503, 504}; every 4xx/5xx body MUST be a -protobuf `Status`; gzip decoding is a server MUST; OTLP/JSON uses hex IDs, lowerCamelCase -fields, numeric enums, and unknown fields MUST be ignored. + +- Partial-success responses are a MUST. A partially bad batch gets 200 with `rejected_*` counts; + never fail the whole batch for a few bad items. Return 400 only for fully undecodable requests. +- The retryable HTTP status set is exactly {429, 502, 503, 504}. +- Every 4xx/5xx body MUST be a protobuf `Status`. +- gzip decoding is a server MUST. +- OTLP/JSON uses hex IDs, lowerCamelCase fields, and numeric enums. Unknown fields MUST be ignored. **2. Self-instrumentation as an SDK consumer** ([traces.md](traces.md), [resource-and-config.md](resource-and-config.md), [stability-and-compliance.md](stability-and-compliance.md)): -per-component instrumentation scopes over one global tracer; SDKs MUST NOT throw at runtime; -SERVER-span status = Error only on 5xx (already implemented in `apps/ingest` — -`otel_status_for_rejection`); `deployment.environment.name` is the canonical key (the legacy -`deployment.environment` is formally deprecated — our dual-emission is a migration bridge, not -the end state). + +- Use per-component instrumentation scopes over one global tracer. +- SDKs MUST NOT throw at runtime. +- SERVER-span status is Error only on 5xx. `apps/ingest` already implements this in + `otel_status_for_rejection` (`apps/ingest/src/main.rs`). +- `deployment.environment.name` is the canonical key. The legacy `deployment.environment` is + formally deprecated; our dual emission is a migration bridge, not the end state. **3. UI/query semantics as a data consumer** ([semantic-conventions.md](semantic-conventions.md), -[logs.md](logs.md), [metrics.md](metrics.md)): span status stored as `"Ok"/"Error"/"Unset"` -matches the spec enum spelling (the wire form `STATUS_CODE_*` is translated once at the OTLP -boundary); log error classification should key on `SeverityNumber >= 17`, not `SeverityText` -string-matching; the `db.statement`→`db.query.text` / `db.system`→`db.system.name` coalescing in -warehouse reads mirrors the official DB-semconv stabilization renames; an "event" is now a -LogRecord with `EventName` set, and span events / `RecordException()` are on a deprecation path -toward log-based events. +[logs.md](logs.md), [metrics.md](metrics.md)): + +- Span status stored as `"Ok"/"Error"/"Unset"` matches the spec enum spelling. The wire form + `STATUS_CODE_*` is translated once at the OTLP boundary. +- Log error classification should key on `SeverityNumber >= 17`, not string matching on + `SeverityText`. +- Warehouse reads coalesce `db.statement` to `db.query.text` and `db.system` to `db.system.name`. + This mirrors the official DB-semconv stabilization renames. +- An "event" is now a LogRecord with `EventName` set. Span events and `RecordException()` are on a + deprecation path toward log-based events. ## Known gaps / items to re-verify -Honest residue from the research pass — each is also flagged inline in its file: +Residue from the research pass. Each is also flagged inline in its file. -- **Jaeger `uber-trace-id` format** was sourced via search synthesis (official page 404'd); - spot-check against current Jaeger docs before quoting externally +- **Jaeger `uber-trace-id` format** was sourced via search synthesis (the official page returned + 404). Spot-check against current Jaeger docs before quoting externally ([context-propagation.md](context-propagation.md)). -- **W3C Baggage document-track status** (Editor's Draft vs Candidate Recommendation) can shift; - re-check before citing externally. -- **Global metrics cardinality-limit env var**: none found in the general spec — recorded as +- **W3C Baggage document-track status** (Editor's Draft vs Candidate Recommendation) can shift. + Re-check before citing externally. +- **Global metrics cardinality-limit env var**: none found in the general spec. Recorded as unverified absence, not confirmed absence ([metrics.md](metrics.md)). - **Env-var vs declarative-config precedence** when both are present is not precisely specified by the spec yet ([resource-and-config.md](resource-and-config.md)). - **Semconv general-naming and registry landing pages** were partially sourced from search snippets after URL churn ([semantic-conventions.md](semantic-conventions.md)). -- ~~Attribute value-type unification~~ — resolved 2026-07-05 against raw - `specification/common/README.md`: attribute values are full `AnyValue` (nested maps/arrays - allowed); older SDKs emit the stricter primitives+homogeneous-arrays subset. +- ~~Attribute value-type unification~~: resolved 2026-07-05 against raw + `specification/common/README.md`. Attribute values are full `AnyValue` (nested maps/arrays + allowed); older SDKs emit the stricter primitives + homogeneous-arrays subset. ## Canonical sources diff --git a/docs/otel-spec/context-propagation.md b/docs/otel-spec/context-propagation.md index a31c56406f..8358529e3e 100644 --- a/docs/otel-spec/context-propagation.md +++ b/docs/otel-spec/context-propagation.md @@ -3,48 +3,53 @@ This page covers the OpenTelemetry **Context** abstraction, the **Baggage** API and its W3C wire format, and the **Propagators API** (W3C TraceContext primary; B3/Jaeger at reference level). It is part of the internal OTel spec reference used for spec-compliance checks and the best-practices -skill. Every section links to its normative source — verify against the source, not this summary, -before treating anything here as authoritative for a compliance decision. - -> **Stability at a glance:** Context API — Stable. Propagators API — Stable (a few per-language -> details, e.g. `GetAll`, are pre-stable). Baggage API — Stable. W3C Trace Context — W3C -> Recommendation (normative, versioned `00`). W3C Baggage — W3C Editor's Draft / Candidate -> Recommendation track (not yet a final REC as of this writing — treat as stable-in-practice but -> re-check status). OTel `tracestate` handling (the `ot=` vendor key, consistent probability -> sampling `th`/`rv`) — **Development** (not stable) at the OTel-spec level. +skill. Every section links to its normative source. Verify against the source, not this summary, +before using anything here for a compliance decision. + +> **Stability at a glance:** +> +> - Context API: Stable. +> - Propagators API: Stable (a few per-language details, e.g. `GetAll`, are pre-stable). +> - Baggage API: Stable. +> - W3C Trace Context: W3C Recommendation (normative, version `00`). +> - W3C Baggage: on the W3C Editor's Draft / Candidate Recommendation track, not yet a final REC +> as of this writing. Treat it as stable in practice, but re-check its status. +> - OTel `tracestate` handling (the `ot=` vendor key, consistent probability sampling `th`/`rv`): +> **Development** (not stable) at the OTel-spec level. ## Relevance to Maple -- Maple's Rust ingest gateway (`apps/ingest`) is primarily an **OTLP receiver**, not an HTTP - trace-context propagator: incoming spans already carry `TraceId` / `SpanId` / `ParentSpanId` / - `TraceState` as OTLP fields (populated upstream by the _sending_ SDK's propagator before OTLP - export), and these are inserted verbatim into ClickHouse/Tinybird (see - `apps/ingest/src/clickhouse_insert_mappings.rs`, columns `TraceId`, `SpanId`, `ParentSpanId`, - `TraceState`). Maple does not need to parse a `traceparent` HTTP header to ingest span data. -- Where W3C TraceContext propagation _does_ apply directly to Maple: the ingest gateway's own - **self-instrumentation** (`apps/ingest` exports its own OTLP traces per - `docs/otel-spec/context-propagation.md`'s sibling doc on self-observability — see root - `CLAUDE.md` "Self-Observability" section) and any outbound HTTP calls the API/web app makes that - should carry a live `Context` (e.g. `warehouse.sqlQuery` spans, Hyperdrive calls) rely on - `@effect/opentelemetry`'s context propagation, which implements this spec under the hood. +- Maple's Rust ingest gateway (`apps/ingest`) is an **OTLP receiver**, not an HTTP trace-context + propagator. Incoming spans already carry `TraceId` / `SpanId` / `ParentSpanId` / `TraceState` as + OTLP fields (populated upstream by the _sending_ SDK's propagator before OTLP export). These are + inserted verbatim into ClickHouse/Tinybird (see `apps/ingest/src/clickhouse_insert_mappings.rs`, + columns `TraceId`, `SpanId`, `ParentSpanId`, `TraceState`). Maple does not need to parse a + `traceparent` HTTP header to ingest span data. +- W3C TraceContext propagation _does_ apply directly in two places. First, the ingest gateway's + own **self-instrumentation** (`apps/ingest/src/otel.rs` exports its own OTLP traces). Second, + outbound calls from Maple's Effect workers (api, ai, alerting) that should carry a live + `Context`, such as `WarehouseQueryService` queries or Postgres calls over Hyperdrive. Those rely + on the context + propagation in `@maple-dev/effect-sdk` (`packages/effect-sdk`), which implements this spec. `service.name`, `deployment.environment.name`, etc. are Resource attributes, not part of Context - propagation, but they ride the same OTel SDK plumbing. -- Because Maple is a **backend/consumer of arbitrary customer OTLP**, expect payloads whose - `TraceId` came from a variety of upstream propagators (W3C, but also legacy B3/Jaeger bridged by - some customer collectors). The ingest gateway should treat a **128-bit** `TraceId` and a - **64-bit** `SpanId`/`ParentSpanId` as the canonical OTel shapes; anything shorter (e.g. a 64-bit - B3 trace ID zero-padded to 128 bits) is a valid _interop_ case, not a data error — see + propagation, though they ride the same SDK plumbing. +- Maple is a **backend for arbitrary customer OTLP**, so expect `TraceId`s from many upstream + propagators: W3C, but also legacy B3/Jaeger bridged by some customer collectors. The ingest + gateway should treat a **128-bit** `TraceId` and a **64-bit** `SpanId`/`ParentSpanId` as the + canonical OTel shapes. A shorter ID (e.g. a 64-bit B3 trace ID zero-padded to 128 bits) is a + valid _interop_ case, not a data error. See [B3/Jaeger interop notes](#b3-and-jaeger-reference-level) below. - `Baggage` is orthogonal to span data: **baggage entries do not automatically become span - attributes**. If Maple ever wants baggage values (e.g. a customer's `user.id` propagated via - baggage) to show up as searchable span/log attributes, an explicit "baggage → span attribute" - bridge (a processor) is required — this is not implicit OTel behavior. Useful to know when - triaging "why isn't this baggage key showing up as a facet" questions. -- The W3C `tracestate` field is captured today (`TraceState` column, `LinksTraceState` for span - links) but Maple does not currently parse or act on the OTel `ot=` sampling sub-keys (`th`, `rv`) - — that spec area is explicitly **Development**/unstable upstream, so there's no compliance - expectation to implement it yet. Worth revisiting if/when OTel's consistent probability sampling - spec stabilizes and Maple wants cross-service consistent sampling. + attributes**. If Maple wants baggage values (e.g. a customer's `user.id` propagated via baggage) + to show up as searchable span/log attributes, it needs an explicit "baggage → span attribute" + bridge (a processor). This is useful when triaging "why isn't this baggage key showing up as a + facet" questions. +- The W3C `tracestate` field is captured (`TraceState` column, `LinksTraceState` for span links). + Maple reads one OTel `ot=` sub-key: the `traces.SampleRate` column default in + `packages/domain/src/tinybird/datasources.ts` derives the sample rate from `th` when no + `SampleRate` span attribute is set. `rv` is not used. The spec area is **Development**, so there + is no compliance expectation beyond this. Revisit if OTel's consistent probability sampling + stabilizes and Maple wants cross-service consistent sampling. --- @@ -53,40 +58,40 @@ before treating anything here as authoritative for a compliance decision. **Stability: Stable.** Source: https://opentelemetry.io/docs/specs/otel/context/ -`Context` is defined as "a propagation mechanism which carries execution-scoped values across API +`Context` is "a propagation mechanism which carries execution-scoped values across API boundaries and between logically associated execution units." It underlies both distributed trace -propagation (via `SpanContext`) and Baggage propagation, but is a general-purpose mechanism — any -cross-cutting concern can ride in `Context`. +propagation (via `SpanContext`) and Baggage propagation. It is general-purpose: any cross-cutting +concern can ride in `Context`. ### Immutability > A `Context` **MUST** be immutable, and its write operations **MUST** result in the creation of a > new `Context` containing the original values and the specified values updated. -A "write" (`SetValue`) never mutates the context you started with; it returns a new `Context` +A write (`SetValue`) never mutates the context you started with. It returns a new `Context` layered on top of the old one. This is why context propagation composes safely across concurrent -call stacks — nobody can retroactively change a `Context` another goroutine/task/fiber already +call stacks: nobody can retroactively change a `Context` that another goroutine/task/fiber already captured. ### Core operations -| Operation | Signature (conceptual) | Notes | -| ----------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `CreateKey` | `(name: string) -> Key` | `name` is for debugging only. "Multiple calls to `CreateKey` with the same name SHOULD NOT return the same value unless language constraints dictate otherwise" — keys are unforgeable/opaque, not string-keyed maps. | -| `GetValue` | `(context, key) -> value` | Read-only lookup against a given `Context`. | -| `SetValue` | `(context, key, value) -> Context` | Returns a **new** `Context`; does not mutate the input. | +| Operation | Signature (conceptual) | Notes | +| ----------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CreateKey` | `(name: string) -> Key` | `name` is for debugging only. "Multiple calls to `CreateKey` with the same name SHOULD NOT return the same value unless language constraints dictate otherwise." Keys are unforgeable/opaque, not string-keyed maps. | +| `GetValue` | `(context, key) -> value` | Read-only lookup against a given `Context`. | +| `SetValue` | `(context, key, value) -> Context` | Returns a **new** `Context`; does not mutate the input. | ### Implicit-context operations (languages with implicit/ambient context, e.g. via thread-locals or async-local storage) -| Operation | Purpose | -| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Get Current Context` | Returns the `Context` associated with the current execution unit. | -| `Attach` | Associates a given `Context` with the current execution unit; returns a **token** for later restoration. | -| `Detach` | Resets the current context using the token returned by `Attach`; implementations are expected to be able to signal incorrect call ordering (e.g. detaching out of order/detaching twice). | +| Operation | Purpose | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `Get Current Context` | Returns the `Context` associated with the current execution unit. | +| `Attach` | Associates a given `Context` with the current execution unit; returns a **token** for later restoration. | +| `Detach` | Resets the current context using the token returned by `Attach`. Implementations are expected to be able to signal incorrect call ordering (e.g. detaching out of order or detaching twice). | -Attach/detach is explicitly a stack-like discipline: detach with the token you got from the -matching attach, not just "restore to whatever was current a moment ago" — misuse (double-detach, -out-of-order detach) is something implementations may — and are encouraged to — detect and signal. +Attach/detach is a stack-like discipline. Detach with the token from the matching attach; don't +just restore whatever was current a moment ago. Implementations may detect and signal misuse +(double detach, out-of-order detach), and are encouraged to. --- @@ -97,47 +102,45 @@ pre-stable). Source: https://opentelemetry.io/docs/specs/otel/context/api-propagators/ The Propagators API is how a `Context` (specifically, the active `SpanContext` and `Baggage`) -crosses a wire. It is deliberately format-agnostic; `TextMapPropagator` is the concrete shape used -for HTTP-style string-keyed carriers. +crosses a wire. It is format-agnostic. `TextMapPropagator` is the concrete shape used for +HTTP-style string-keyed carriers. ### `TextMapPropagator` -| Method | Contract | -| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Fields(carrier)` | Returns the list of propagation field/header names this propagator uses for a given carrier type — lets a caller pre-clear those fields before re-injecting into a reused carrier ("If your carrier is reused, you should delete the fields here before calling Inject"). | -| `Inject(context, carrier, setter?)` | Writes values from `context` into `carrier` using `setter`. | -| `Extract(context, carrier, getter?)` | Reads values from `carrier`, returns a **new** `Context` (per Context's immutability rule) with the extracted values layered on top of the passed-in `context`. **On parse failure, the implementation MUST NOT throw and MUST NOT store a new value** — i.e., extraction failures degrade to a no-op, they never crash the caller or poison the context. | +| Method | Contract | +| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Fields(carrier)` | Returns the propagation field/header names this propagator uses for a given carrier type. A caller can clear those fields before re-injecting into a reused carrier ("If your carrier is reused, you should delete the fields here before calling Inject"). | +| `Inject(context, carrier, setter?)` | Writes values from `context` into `carrier` using `setter`. | +| `Extract(context, carrier, getter?)` | Reads values from `carrier` and returns a **new** `Context` (per Context immutability) with the extracted values layered on top of the passed-in `context`. **On parse failure, the implementation MUST NOT throw and MUST NOT store a new value.** Extraction failures degrade to a no-op; they never crash the caller or poison the context. | ### `Setter` / `Getter` carrier interfaces -- **Setter.Set(carrier, key, value)** — writes one field into the carrier. Implementations +- **Setter.Set(carrier, key, value):** writes one field into the carrier. Implementations "SHOULD preserve casing" for case-insensitive protocols like HTTP headers (don't rewrite - `Traceparent` to `traceparent`, etc., gratuitously). -- **Getter.Get(carrier, key)** — returns the first value for `key`, or null/absent if not present; - for HTTP-like carriers this "MUST be case insensitive." -- **Getter.Keys(carrier)** — returns all keys in the carrier; enables prefix/pattern-based - extraction (the doc calls out B3's `X-B3-*` family as the motivating example). -- **Getter.GetAll(carrier, key)** — returns _all_ values for a repeated key, in original order - (relevant for headers that can legitimately repeat). Noted as pre-stable in some language - implementations. - -Both Setter and Getter "MUST be stateless and allowed to be saved as constants" — they carry no -per-call state, so a single shared instance is safe to reuse across concurrent injects/extracts. + `Traceparent` to `traceparent` gratuitously). +- **Getter.Get(carrier, key):** returns the first value for `key`, or null/absent if not present. + For HTTP-like carriers this "MUST be case insensitive." +- **Getter.Keys(carrier):** returns all keys in the carrier. This enables prefix/pattern-based + extraction; the spec cites B3's `X-B3-*` family as the motivating example. +- **Getter.GetAll(carrier, key):** returns _all_ values for a repeated key, in original order + (for headers that can legitimately repeat). Pre-stable in some language implementations. + +Setter and Getter "MUST be stateless and allowed to be saved as constants." They carry no +per-call state, so one shared instance is safe to reuse across concurrent injects/extracts. ### Composite Propagator Multiple `TextMapPropagator`s can be combined into one composite propagator. The composite runs -its constituent propagators **in the order they were specified**, for both `Inject` and `Extract` -— order matters (e.g., running `tracecontext` before `baggage` vs. after can matter if propagators -interact with the same carrier keys, though in the common case they touch disjoint headers). +its constituent propagators **in the order they were specified**, for both `Inject` and `Extract`. +Order matters only if propagators touch the same carrier keys; `tracecontext` and `baggage` +normally touch disjoint headers. ### Global propagator registration -The API defines a way to get/set a process-wide default propagator. Per spec, implementations -"MUST use no-op propagators unless explicitly configured otherwise" — propagation is opt-in by -default at the raw-API level; SDKs, not the bare API, are what typically wire up sane defaults. -Language/framework integration guidance (e.g. ASP.NET) suggests defaulting to a composite of W3C -TraceContext + W3C Baggage. +The API defines a way to get and set a process-wide default propagator. Per spec, implementations +"MUST use no-op propagators unless explicitly configured otherwise." Propagation is opt-in at the +raw-API level; SDKs, not the bare API, typically wire up defaults. Guidance for pre-configured +platforms (e.g. ASP.NET) is to default to a composite of W3C TraceContext + W3C Baggage. ### `OTEL_PROPAGATORS` environment variable @@ -151,31 +154,31 @@ Source: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-v | -------------- | -------------------------------------- | -------------- | | `tracecontext` | W3C Trace Context | Stable | | `baggage` | W3C Baggage | Stable | -| `b3` | B3 Single header | — | -| `b3multi` | B3 Multi header | — | +| `b3` | B3 Single header | (none listed) | +| `b3multi` | B3 Multi header | (none listed) | | `jaeger` | Jaeger `uber-trace-id` | **Deprecated** | -| `xray` | AWS X-Ray (third-party format) | — | +| `xray` | AWS X-Ray (third-party format) | (none listed) | | `ottrace` | OT Trace | **Deprecated** | -| `none` | No automatically configured propagator | — | +| `none` | No automatically configured propagator | (none listed) | -Values **MUST be deduplicated** so a propagator is only registered once even if named twice (or +Values **MUST be deduplicated**, so a propagator is registered only once even if named twice (or implied twice by other config). ### Propagators distribution Source: https://opentelemetry.io/docs/specs/otel/context/api-propagators/#propagators-distribution -- **Core/mandatory packages:** W3C TraceContext, W3C Baggage, B3. +- **OTel-maintained packages:** W3C TraceContext, W3C Baggage, B3. - **Additional/optional packages:** Jaeger (Deprecated), OT Trace (Deprecated), OpenCensus BinaryFormat. - Vendor-specific propagator formats "MUST NOT be maintained or distributed as part of the - OpenTelemetry Core packages" — they live in contrib/vendor repos instead. + OpenTelemetry Core packages." They live in contrib/vendor repos. --- -## W3C Trace Context — `traceparent` +## W3C Trace Context: `traceparent` -**Status: W3C Recommendation** (normative wire format). Version field in this spec assumes `00`. +**Status: W3C Recommendation** (normative wire format). This section assumes version `00`. Source: https://www.w3.org/TR/trace-context/#traceparent-header ### Grammar (§3.2, ABNF) @@ -194,50 +197,47 @@ Example: `traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01` ### Field-by-field -| Field | Size | Encoding | Invalid value | On invalid, receiver behavior | -| ---------------------------- | -------- | ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| `version` | 1 byte | 2 lowercase hex digits | `ff` is explicitly forbidden | Reject/ignore the header | -| `trace-id` | 16 bytes | 32 lowercase hex chars | All-zero (`00000000000000000000000000000000`) | Invalid — vendors **must ignore** a `traceparent` with an invalid trace-id (i.e., treat as if no `traceparent` was received; start a new trace) | -| `parent-id` (a.k.a. span-id) | 8 bytes | 16 lowercase hex chars | All-zero (`0000000000000000`) | Invalid — same treatment: ignore the header | -| `trace-flags` | 1 byte | 2 lowercase hex digits | — (all 256 bit-patterns are structurally valid) | Only the least-significant bit is currently defined | +| Field | Size | Encoding | Invalid value | On invalid, receiver behavior | +| ---------------------------- | -------- | ---------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `version` | 1 byte | 2 lowercase hex digits | `ff` is explicitly forbidden | Reject/ignore the header | +| `trace-id` | 16 bytes | 32 lowercase hex chars | All-zero (`00000000000000000000000000000000`) | Invalid. Vendors **must ignore** a `traceparent` with an invalid trace-id (treat it as if no `traceparent` was received; start a new trace) | +| `parent-id` (a.k.a. span-id) | 8 bytes | 16 lowercase hex chars | All-zero (`0000000000000000`) | Invalid. Same treatment: ignore the header | +| `trace-flags` | 1 byte | 2 lowercase hex digits | None (all 256 bit patterns are structurally valid) | Level 1 defines only the least-significant bit (`sampled`). Level 2 (CR) also defines bit 1 (`random`) | -### Sampled flag (§3.2.2.5.1... i.e. the trace-flags LSB) +### Sampled flag (trace-flags LSB) > When set, the least significant bit (right-most), denotes that the caller may have recorded > trace data. -This is **not** a command ("you must sample") — it communicates the upstream's own recording -decision so downstream participants can make consistent joint decisions, but each vendor's sampler -still owns its own decision. All other trace-flags bits are reserved for future use and MUST be set -to `0` by a conforming producer that doesn't understand them (don't invent bit meanings). +This is **not** a command to sample. It communicates the upstream's own recording decision so +downstream participants can make consistent decisions; each vendor's sampler still owns its own +decision. Other trace-flags bits are reserved. A producer MUST set bits it doesn't understand to +`0` (don't invent bit meanings). -### Version handling (§3.3 "Versioning of traceparent") +### Version handling (§3.2.4 "Versioning of traceparent") - `version = ff` is invalid outright. -- If a receiver sees a **higher** version number than it knows about, it MUST still try to parse - the fields it understands from a version-`00`-shaped payload: parse `trace-id` as the 32 hex - chars after the first dash, `parent-id` as the next 16 hex chars, and the sampled bit from the - `trace-flags` byte that follows — then ignore any additional trailing fields the newer version - might append. - - If the header is shorter than the minimum length required to contain those three fields - (roughly, under ~55 characters for the version-00 shape), the receiver should not attempt to - parse it and should restart the trace (treat as if absent) rather than guess. -- When forwarding an unknown-version header onward, unknown/未定义 flag bits must be reset to zero - on the outgoing request (don't blindly forward bits you don't understand as if you understood - them). +- If a receiver sees a **higher** version than it knows, it MUST still try to parse the fields it + understands from a version-`00`-shaped payload: `trace-id` as the 32 hex chars after the first + dash, `parent-id` as the next 16 hex chars, and the sampled bit from the `trace-flags` byte that + follows. It then ignores any trailing fields the newer version appends. + - If the header is shorter than 55 characters (the version-00 length), the receiver should not + parse it and should restart the trace (treat it as absent) rather than guess. +- When forwarding an unknown-version header, reset unknown flag bits to zero on the outgoing + request. Don't forward bits you don't understand as if you understood them. ### Practical MUSTs for an ingest gateway -- Reject (treat as absent / start fresh) a `traceparent` whose `trace-id` or `parent-id` is - all-zero — these are the two explicitly normative "invalid value" cases. -- Never throw on a malformed `traceparent` — per the Propagators API contract, extraction failure - degrades to "no context extracted," not an exception. -- Preserve `trace-flags` bits you don't understand as `0` when re-emitting, don't round-trip - garbage bits. +- Reject (treat as absent, start fresh) a `traceparent` whose `trace-id` or `parent-id` is + all-zero. These are the two explicitly normative "invalid value" cases. +- Never throw on a malformed `traceparent`. Per the Propagators API contract, extraction failure + degrades to "no context extracted", not an exception. +- Reset `trace-flags` bits you don't understand to `0` when re-emitting. Don't round-trip unknown + bits. --- -## W3C Trace Context — `tracestate` +## W3C Trace Context: `tracestate` **Status: W3C Recommendation.** Source: https://www.w3.org/TR/trace-context/#tracestate-header @@ -260,59 +260,58 @@ chr = %x20 / nblk-chr ### Key rules -| Key form | Format | Purpose | -| ---------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | -| Simple key | lowercase alphanum + `_ - * /`, ≤256 chars | Single-tenant vendor identifier, e.g. `congo` | -| Multi-tenant key | `tenant-id@system-id` | Lets a multi-tenant vendor (`system-id`) namespace by `tenant-id` so lookups can jump straight to `@system-id` | +| Key form | Format | Purpose | +| ---------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | +| Simple key | lowercase alphanum + `_ - * /`, ≤256 chars | Single-tenant vendor identifier, e.g. `congo` | +| Multi-tenant key | `tenant-id@system-id` | Lets a multi-tenant vendor (`system-id`) namespace by `tenant-id`, so lookups can jump straight to `@system-id` | ### Value rules -Opaque, printable-ASCII (`0x20`–`0x7E`), **excluding** comma and `=`; max **256 characters**. +Opaque, printable ASCII (`0x20`-`0x7E`), **excluding** comma and `=`. Max **256 characters**. ### Limits -| Limit | Value | -| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| Max list-members | 32 | -| Max total header size a vendor should be able to propagate | 512 characters | -| Truncation strategy when trimming to fit | Drop entries over 128 characters first; then drop from the end of the list; only whole list-members may be dropped, never partially | +| Limit | Value | +| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| Max list-members | 32 | +| Max total header size a vendor should be able to propagate | 512 characters | +| Truncation strategy when trimming to fit | Drop entries over 128 characters first, then drop from the end of the list. Only whole list-members may be dropped, never partial ones | -### Mutation rules (§3.4/"Mutating the tracestate field") +### Mutation rules (§3.5 "Mutating the tracestate Field") - A participant adding a new entry inserts it **at the front (left)** of the list. - A participant updating its own existing entry **moves it to the front** (as if delete + re-add). -- Only one entry per key is allowed — it represents that vendor's _last_ position in the trace; a +- Only one entry per key is allowed. It represents that vendor's _last_ position in the trace; a vendor re-entering the trace overwrites its previous entry rather than appending a duplicate. - When a vendor is only reading (not modifying) an entry, order of all _other_, unmodified - list-members must be preserved. -- Deleting keys: a vendor should not delete keys it did not itself generate — the spec's guidance - is that removing another vendor's entry breaks that vendor's correlation ability, so participants - should be conservative about deleting others' state. + All other, unmodified list-members must keep their order. +- Deleting keys: a vendor should not delete keys it did not generate. Removing another vendor's + entry breaks that vendor's correlation, so participants should be conservative about deleting + others' state. -### OTel's own `tracestate` usage — the `ot=` vendor key +### OTel's own `tracestate` usage: the `ot=` vendor key **Status: Development (not stable).** Source: https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/ , https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/ OpenTelemetry SDKs consolidate all OTel-internal `tracestate` data into a **single** list-member -under the key `ot`, itself an internal `;`-separated set of sub-keys (e.g. `ot=th:c8;rv:1a2b3c...`), -capped at 256 characters total for that entry. Individual instrumentation libraries get their own -separate list-member keys rather than writing into the shared `ot` entry. - -| Sub-key | Meaning | Format | -| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -| `th` | Sampling **threshold** (rejection threshold `T`), conveys effective sampling probability. `0` = 100% sampling. Probability = `(2^56 - Threshold) / 2^56`. | 1–14 lowercase hex digits (right-padded conceptually to 56 bits) | -| `rv` | Explicit **randomness value** — an alternative source of the common random value `R` used for consistent sampling decisions, instead of relying on the trailing bits of the `trace-id` | Exactly 14 lowercase hex digits | -| `p` | Legacy/older probability encoding predating `th`/`rv` | — | - -This mechanism underpins **OTel consistent probability sampling**: every participant compares the -shared randomness `R` (from `rv`, or else derived from the low 7 bytes of the `trace-id`) against -its own rejection threshold `T` (`th`), so independently-configured samplers along a trace make -_consistent_ keep/drop decisions without needing to agree out-of-band. This whole area is marked -**Development** stability in the OTel spec — do not treat it as a compliance requirement, but do -recognize `ot=th:...;rv:...` if you see it in customer `tracestate` values. +under the key `ot`. Its value is an internal `;`-separated set of sub-keys (e.g. +`ot=th:c8;rv:1a2b3c...`), capped at 256 characters for that entry. Individual instrumentation +libraries get their own list-member keys rather than writing into the shared `ot` entry. + +| Sub-key | Meaning | Format | +| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| `th` | Sampling **threshold** (rejection threshold `T`); conveys effective sampling probability. `0` = 100% sampling. Probability = `(2^56 - Threshold) / 2^56`. | 1-14 lowercase hex digits (conceptually right-padded to 56 bits) | +| `rv` | Explicit **randomness value**: an alternative source of the common random value `R` used for consistent sampling decisions, instead of the trailing bits of the `trace-id` | Exactly 14 lowercase hex digits | +| `p` | Legacy probability encoding that predates `th`/`rv` | (not specified here) | + +This mechanism underpins **OTel consistent probability sampling**. Every participant compares the +shared randomness `R` (from `rv`, or else the low 7 bytes of the `trace-id`) against its own +rejection threshold `T` (`th`). Independently configured samplers along a trace then make +_consistent_ keep/drop decisions without agreeing out of band. The whole area is **Development** +in the OTel spec. Do not treat it as a compliance requirement, but recognize `ot=th:...;rv:...` in +customer `tracestate` values. --- @@ -323,34 +322,32 @@ recognize `ot=th:...;rv:...` if you see it in customer `tracestate` values. **Stability: Stable.** Source: https://opentelemetry.io/docs/specs/otel/baggage/api/ -| Operation | Signature | Notes | -| ---------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Get Value` | `(baggage, name) -> value \| absent` | Simple lookup. | -| `Get All Values` | `(baggage) -> [(name, value, metadata)]` | Order is **not significant**; may be exposed as an iterator or immutable collection. | -| `Set Value` | `(baggage, name, value, metadata?) -> Baggage` | Returns a new `Baggage`; `metadata` is "an opaque wrapper for a string with no semantic meaning" to the API itself (vendor-defined use, e.g. W3C's `property` syntax). | -| `Remove Value` | `(baggage, name) -> Baggage` | Returns a new `Baggage` without that entry. | +| Operation | Signature | Notes | +| ---------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Get Value` | `(baggage, name) -> value \| absent` | Simple lookup. | +| `Get All Values` | `(baggage) -> [(name, value, metadata)]` | Order is **not significant**; may be exposed as an iterator or immutable collection. | +| `Set Value` | `(baggage, name, value, metadata?) -> Baggage` | Returns a new `Baggage`. `metadata` is "an opaque wrapper for a string with no semantic meaning" to the API itself (vendor-defined use, e.g. W3C's `property` syntax). | +| `Remove Value` | `(baggage, name) -> Baggage` | Returns a new `Baggage` without that entry. | -Entry shape: **key** — any non-empty valid UTF-8 string, case-sensitive; **value** — any valid -UTF-8 string, case-sensitive; **metadata** — optional opaque string. Exactly one value is -associated with a given name at a time (setting again replaces). +Entry shape: **key** is any non-empty valid UTF-8 string, case-sensitive; **value** is any valid +UTF-8 string, case-sensitive; **metadata** is an optional opaque string. A name has exactly one +value at a time (setting again replaces it). **Security requirement:** "To avoid sending any name/value pairs to an untrusted process, the -Baggage API MUST provide a way to remove all baggage entries from a context" — i.e., a -`ClearBaggage`-equivalent full-wipe capability is mandatory, specifically framed as a -trust-boundary safeguard. - -**Baggage vs. span attributes — explicitly separate concerns.** The Baggage API spec only defines -the `Baggage` value type and its context-scoped CRUD; it says nothing about auto-copying baggage -into span attributes. In practice across OTel: **baggage entries do NOT automatically become span -attributes** (or log/metric attributes). Any such correlation requires an explicit bridge/processor -that reads `Baggage` from `Context` and calls `SetAttribute` on the current span itself. Treat -"baggage shows up as a span attribute" as something that must be deliberately wired, not a -default. - -### W3C Baggage header — wire format - -**Status:** W3C Editor's Draft (Baggage spec has moved slower through W3C process than Trace -Context — re-verify current REC status before citing as a final Recommendation). +Baggage API MUST provide a way to remove all baggage entries from a context." A +`ClearBaggage`-equivalent full wipe is mandatory, framed as a trust-boundary safeguard. + +**Baggage and span attributes are separate concerns.** The Baggage API spec only defines the +`Baggage` value type and its context-scoped CRUD. It says nothing about copying baggage into span +attributes. Across OTel, **baggage entries do NOT automatically become span attributes** (or +log/metric attributes). That correlation needs an explicit bridge/processor that reads `Baggage` +from `Context` and calls `SetAttribute` on the current span. Treat "baggage shows up as a span +attribute" as something that must be deliberately wired. + +### W3C Baggage header: wire format + +**Status:** W3C Editor's Draft. Baggage has moved through the W3C process more slowly than Trace +Context; re-verify its current status before citing it as a final Recommendation. Source: https://www.w3.org/TR/baggage/ #### Grammar @@ -363,16 +360,16 @@ value = *baggage-octet ; percent-encode anything outside th property = key OWS "=" OWS value / key OWS ``` -`baggage-octet` excludes control characters, whitespace, `"`, `,`, `;`, and `\`; any character +`baggage-octet` excludes control characters, whitespace, `"`, `,`, `;`, and `\`. Any character outside the allowed set must be percent-encoded per RFC 3986. #### Limits -| Limit | Requirement | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Max list-members | Implementations **must** propagate all list-members when the total is **64 or fewer** | -| Max header size | Implementations **must** propagate the full baggage-string when it is **8192 bytes or fewer** | -| Over either limit | Implementations **may** drop list-members to come back into compliance (no single normative drop order is mandated the way `tracestate` prescribes one) | +| Limit | Requirement | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| Max list-members | Implementations **must** propagate all list-members when the total is **64 or fewer** | +| Max header size | Implementations **must** propagate the full baggage-string when it is **8192 bytes or fewer** | +| Over either limit | Implementations **may** drop list-members to come back into compliance (no normative drop order is mandated, unlike `tracestate`) | No minimum-length constraint is specified for individual keys/values. @@ -382,18 +379,18 @@ No minimum-length constraint is specified for individual keys/values. > stored in baggage, or they should ensure that baggage isn't present in requests that cross > trust-boundaries. -The spec also expects implementations to defend against malicious/oversized baggage strings -(length validation, careful parsing) to avoid buffer-overflow/injection-style failure modes on -receipt — relevant directly to Maple's ingest gateway if it ever parses inbound `baggage` headers -rather than only OTLP payloads. +The spec also expects implementations to defend against malicious or oversized baggage strings +(length validation, careful parsing) to avoid buffer-overflow/injection-style failures on receipt. +This applies to Maple's ingest gateway if it ever parses inbound `baggage` headers rather than +only OTLP payloads. --- ## B3 and Jaeger (reference level) -These are **not** part of OTel's core/mandatory propagator set; B3 ships in the optional contrib -distribution, Jaeger propagation is explicitly **Deprecated** in OTel. Documented here because -Maple, as an ingest backend, may see traffic from collectors/proxies still emitting them. +Neither is part of OTel's default propagator set (`tracecontext,baggage`). B3 is an OTel-maintained +extension package; Jaeger propagation is **Deprecated** in OTel. They are documented here because +Maple, as an ingest backend, may see traffic from collectors/proxies that still emit them. Source (B3): https://github.com/openzipkin/b3-propagation Source (Jaeger, reference): https://www.jaegertracing.io/docs/1.6/client-libraries/#tracespan-identity , @@ -411,8 +408,8 @@ deprecation noted at https://opentelemetry.io/docs/specs/otel/context/api-propag ### B3 single-header format -Header name: `b3`. Format: `b3: {TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}` — last two -fields optional. `SamplingState`: `1` = accept, `0` = deny, `d` = debug, absent = defer. +Header name: `b3`. Format: `b3: {TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}`; the last two +fields are optional. `SamplingState`: `1` = accept, `0` = deny, `d` = debug, absent = defer. Examples: `b3: 80f198ee56343ba864fe8b2a57d3eff7-e457b5a2e4d86bd1-1-05e3ac9a4f6e3b90`, `b3: 0` (deny only), `b3: 80f198ee56343ba864fe8b2a57d3eff7-e457b5a2e4d86bd1-d` (debug). @@ -424,40 +421,39 @@ precedence. Header: `uber-trace-id: {trace-id}:{span-id}:{parent-span-id}:{flags}`. -- `trace-id`: variable length, 16 or 32 hex chars (64-bit or 128-bit); shorter values are - zero-padded on the left when converting to/from OTel's fixed 128-bit `TraceId`. +- `trace-id`: 16 or 32 hex chars (64-bit or 128-bit). Shorter values are zero-padded on the left + when converting to/from OTel's fixed 128-bit `TraceId`. - `span-id`: 16 hex chars. -- `parent-span-id`: historically present but deprecated/unused — conventionally `0`. -- `flags`: one byte as two hex digits (bit 0 = sampled, similar spirit to W3C's trace-flags LSB). +- `parent-span-id`: historically present but deprecated/unused; conventionally `0`. +- `flags`: one byte as two hex digits (bit 0 = sampled, like W3C's trace-flags LSB). -OTel's own docs mark the Jaeger propagator/format as **Deprecated** in favor of W3C Trace Context; -new integrations should not add Jaeger propagation, only consume it for legacy interop. +OTel's docs mark the Jaeger propagator/format as **Deprecated** in favor of W3C Trace Context. +New integrations should not add Jaeger propagation; consume it only for legacy interop. ### 64-bit vs 128-bit trace-id interop note OTel's canonical `TraceId` is always 128-bit (32 hex chars) internally. B3 and Jaeger both allow -legacy 64-bit trace IDs. The standard interop rule (as implemented by OTel's B3/Jaeger propagators) -is: a 64-bit incoming trace-id is **zero-extended on the left** to fill the 128-bit OTel `TraceId`; -on the way back out to a 64-bit-only system, the low 64 bits are used (the propagator may drop the -high, zero-padded bits). This is why Maple ingest should not treat a `TraceId` with a long run of -leading zero bytes as inherently anomalous — it's a legitimate signature of a bridged 64-bit trace -ID — while still rejecting the fully-all-zero 128-bit case for **W3C `traceparent` parsing -specifically** (that all-zero rule is a W3C traceparent-parsing rule, not a generic "all-zero -trace ids are always invalid" rule for OTLP payloads already carrying a `TraceId` field). +legacy 64-bit trace IDs. The standard interop rule (as implemented by OTel's B3/Jaeger +propagators): a 64-bit incoming trace-id is **zero-extended on the left** to fill the 128-bit +`TraceId`. On the way back out to a 64-bit-only system, the low 64 bits are used. So Maple ingest +should not treat a `TraceId` with a long run of leading zero bytes as anomalous; it is a legitimate +signature of a bridged 64-bit trace ID. The all-zero rejection rule applies to **W3C `traceparent` +parsing specifically**. It is not a generic rule for OTLP payloads that already carry a `TraceId` +field. --- ## Key references -- OTel Context spec — https://opentelemetry.io/docs/specs/otel/context/ -- OTel Propagators API spec — https://opentelemetry.io/docs/specs/otel/context/api-propagators/ -- OTel Propagators distribution — https://opentelemetry.io/docs/specs/otel/context/api-propagators/#propagators-distribution -- OTel Baggage API spec — https://opentelemetry.io/docs/specs/otel/baggage/api/ -- OTel SDK environment variables (`OTEL_PROPAGATORS`) — https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ -- OTel TraceState handling (`ot=` vendor key) — https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/ -- OTel TraceState probability sampling (`th`/`rv`) — https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/ -- W3C Trace Context (Recommendation) — https://www.w3.org/TR/trace-context/ -- W3C Baggage (Editor's Draft/CR) — https://www.w3.org/TR/baggage/ -- B3 propagation (OpenZipkin) — https://github.com/openzipkin/b3-propagation -- Jaeger client trace/span identity (`uber-trace-id`) — https://www.jaegertracing.io/docs/1.6/client-libraries/#tracespan-identity -- Canonical spec repo — https://github.com/open-telemetry/opentelemetry-specification +- OTel Context spec: https://opentelemetry.io/docs/specs/otel/context/ +- OTel Propagators API spec: https://opentelemetry.io/docs/specs/otel/context/api-propagators/ +- OTel Propagators distribution: https://opentelemetry.io/docs/specs/otel/context/api-propagators/#propagators-distribution +- OTel Baggage API spec: https://opentelemetry.io/docs/specs/otel/baggage/api/ +- OTel SDK environment variables (`OTEL_PROPAGATORS`): https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ +- OTel TraceState handling (`ot=` vendor key): https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/ +- OTel TraceState probability sampling (`th`/`rv`): https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/ +- W3C Trace Context (Recommendation): https://www.w3.org/TR/trace-context/ +- W3C Baggage (Editor's Draft/CR): https://www.w3.org/TR/baggage/ +- B3 propagation (OpenZipkin): https://github.com/openzipkin/b3-propagation +- Jaeger client trace/span identity (`uber-trace-id`): https://www.jaegertracing.io/docs/1.6/client-libraries/#tracespan-identity +- Canonical spec repo: https://github.com/open-telemetry/opentelemetry-specification diff --git a/docs/otel-spec/emerging-and-compat.md b/docs/otel-spec/emerging-and-compat.md index 0cfae1cf67..dfac34f9f6 100644 --- a/docs/otel-spec/emerging-and-compat.md +++ b/docs/otel-spec/emerging-and-compat.md @@ -4,43 +4,44 @@ This page covers the parts of the OpenTelemetry spec tree that are either **not (Profiles signal, Entities) or **exist to bridge OTel with something else** (telemetry schemas' rename/transform machinery, and the OpenCensus/OpenTracing/Prometheus compatibility specs). Maple is primarily a _consumer_ of OTLP data and a _producer_ of its own -self-instrumentation, so this doc exists to flag: (a) data we may see on the wire that isn't -"plain" OTLP yet, (b) a signal (Profiles) we don't ingest today but should plan schema/storage -for, and (c) the principled alternative to the ad-hoc legacy-field coalescing we already do by -hand. +self-instrumentation. This doc flags (a) data we may see on the wire that isn't "plain" OTLP +yet, (b) a signal (Profiles) we don't ingest today but should plan schema and storage for, and +(c) the principled alternative to the legacy-field coalescing we already do by hand. -> **Stability key used below:** OpenTelemetry specs use a formal lifecycle — Draft → Experimental -> → Stable → Deprecated → Removed — plus, specifically for Profiles, a pre-Experimental **Alpha** -> stage. See `Source:` links per claim; do not assume anything in this file is Stable unless -> explicitly marked so. +> **Stability key used below:** spec documents carry a status from the ladder Development → +> Alpha → Beta → Release Candidate → Stable, then Deprecated → Removed (see +> [stability-and-compliance.md](stability-and-compliance.md)). See the `Source:` links per claim. +> Do not assume anything in this file is Stable unless it is marked so. ## Relevance to Maple -- **Profiles: not ingested today.** Maple's pipeline (Rust ingest gateway → collector → - ClickHouse/Tinybird) has no Profiles datasource, no `/v1development/profiles` route, and no - schema for `Profile`/`Sample`/`Location`/`Function` tables. If/when we add it, expect a - dictionary-compressed, pprof-derived shape (below), not flat rows like spans/logs — this - changes the storage model materially (string/location/function tables are shared across - samples within a profile, closer to how we already dedupe attribute tables than to how we - store span rows). -- **Telemetry schemas are the principled version of what we do by hand.** CLAUDE.md documents - that `WarehouseQueryService.executeSql` spans "coalesce both" legacy `db.statement*`/`db.system` - and current `db.query.text`/`db.system.name` spellings, and that semconv renamed - `deployment.environment` → `deployment.environment.name` (dual-emitted by `apps/ingest`, coalesced on read by `DEPLOYMENT_ENV_SQL`). That - is exactly the `rename_attributes` transform a schema file encodes formally. We do not currently - read `schema_url` off ingested `ResourceSpans`/`ResourceLogs`/`ResourceMetrics` to drive this - automatically — it's manual per-migration logic. This doc is the reference for what a - schema-aware coalescing layer would look like if we ever generalize it. +- **Profiles: not ingested today.** Maple's pipeline (Rust ingest gateway → + ClickHouse/Tinybird, optionally via a collector) has no Profiles datasource, no `/v1development/profiles` route, and no + schema for `Profile`/`Sample`/`Location`/`Function` tables. If we add it, expect a + dictionary-compressed, pprof-derived shape (below), not flat rows like spans and logs. That + changes the storage model: string, location, and function tables are shared across samples + within a profile, which is closer to how we dedupe attribute tables than to how we store span + rows. +- **Telemetry schemas are the principled version of what we do by hand.** The warehouse + coalesces legacy and current spellings on read: `db.system` / `db.system.name` and + `db.statement` / `db.query.text` (`DB_SYSTEM_ATTR_SQL`, `DB_STATEMENT_SQL` in + `packages/domain/src/tinybird/db-query-shape-sql.ts`), and `deployment.environment` / + `deployment.environment.name` (`DEPLOYMENT_ENV_SQL` in + `packages/domain/src/tinybird/semconv-renames.ts`). Maple's own services dual-emit both + environment keys (e.g. `apps/ingest/src/otel.rs`). That is the `rename_attributes` transform a + schema file encodes formally. Ingest stores `schema_url` (the `resource_schema_url` and + `scope_schema_url` columns) but nothing reads it to drive this; it is manual per-attribute + logic. This doc is the reference for a schema-aware coalescing layer if we ever generalize it. - **Prometheus/OpenMetrics naming rules matter directly** if Maple ever accepts Prometheus remote-write, scrapes `/metrics` from anything, or exposes an OTLP→Prometheus bridge for - customers. The `_total`/unit-suffix/`target_info` rules below are the exact translation table - to implement or validate against. -- **OpenCensus/OpenTracing shim artifacts can show up in customer data** from services mid-migration - (the "OpenTelemetry sandwich" pattern) — e.g., an `opentracing.ref_type` span attribute, or - span links that originated as OpenCensus `Link`s. Both shims are now deprecated specs, so this - is legacy-interop knowledge, not something to build new support for. + customers. The `_total`/unit-suffix/`target_info` rules below are the translation table to + implement or validate against. +- **OpenCensus/OpenTracing shim artifacts can show up in customer data** from services + mid-migration (the "OpenTelemetry sandwich" pattern), e.g. an `opentracing.ref_type` span + attribute, or span links that originated as OpenCensus `Link`s. Both shims are deprecated + specs, so this is legacy-interop knowledge, not something to build new support for. - **Entities** (pointer only) is a Development-stage remodeling of Resource that every signal, - including Profiles, will eventually attach to — worth tracking but not actionable yet. + including Profiles, will eventually attach to. Worth tracking, not actionable yet. --- @@ -48,18 +49,18 @@ hand. ### Status -- Signal/data-model stability: **Alpha** (pre-Experimental; explicitly "should not be used for - critical production workloads"). `Source: https://opentelemetry.io/docs/specs/otel/profiles/`, +- Signal/data-model stability: **Alpha** (explicitly "should not be used for critical + production workloads"). `Source: https://opentelemetry.io/docs/specs/otel/profiles/`, `Source: https://opentelemetry.io/docs/specs/status/`, `Source: https://opentelemetry.io/blog/2026/profiles-alpha/` -- Protocol (OTLP transport) stability: **Development** — the spec's own status table lists +- Protocol (OTLP transport) stability: **Development**. The spec's own status table lists "Protocol: development" for profiles, distinct from the Stable protocol status for traces/metrics/logs. `Source: https://opentelemetry.io/docs/specs/status/` - Wire/endpoint: default path is **`/v1development/profiles`** (an `ExportProfilesServiceRequest` - protobuf), the `v1development` path segment itself signaling non-stable status — it will drop - to `v1` only once the proto reaches release candidate. `Source: https://opentelemetry.io/docs/specs/otlp/` + protobuf). The `v1development` path segment signals non-stable status; it drops to `v1` only + once the proto reaches release candidate. `Source: https://opentelemetry.io/docs/specs/otlp/` - The signal entered **public Alpha on 2026-03-26**; roadmap targets Beta then GA, with GA - originally hoped for ~Q3 2026 per community commentary (treat that date as unconfirmed — - it's from secondary sources, not the spec itself). + originally hoped for ~Q3 2026 per community commentary (treat that date as unconfirmed: it + comes from secondary sources, not the spec). `Source: https://opentelemetry.io/blog/2026/profiles-alpha/` Every message-level type below (`Profile`, `Sample`, `Location`, `Function`, `Mapping`, `Link`, @@ -76,21 +77,21 @@ conversion to/from pprof "with no loss of information." Key design point for a backend: **profiles are not flat sample rows.** A `ProfilesDictionary` holds shared tables that samples reference by index, deduplicating everything that repeats -across a profile (this is conceptually the same dedup trick as an attribute/string table, just -applied to stack frames too): +across a profile (the same dedup trick as an attribute/string table, applied to stack frames +too): -- `mapping_table` (`Mapping[]`) — binary/library memory regions (load address, file offset, +- `mapping_table` (`Mapping[]`): binary/library memory regions (load address, file offset, filename index) that a stack address falls into. -- `location_table` (`Location[]`) — a single stack frame: which `Mapping`, an instruction +- `location_table` (`Location[]`): a single stack frame: which `Mapping`, an instruction `address`, and one or more `Line`s (to represent inlined frames) each pointing at a `Function`. -- `function_table` (`Function[]`) — function metadata: human-readable name, system - (e.g. mangled) name, source filename, starting line — all as string-table indices. -- `link_table` (`Link[]`) — the cross-signal join: a `trace_id` + `span_id` pair a sample can +- `function_table` (`Function[]`): function metadata (human-readable name, system + (e.g. mangled) name, source filename, starting line), all as string-table indices. +- `link_table` (`Link[]`): the cross-signal join: a `trace_id` + `span_id` pair a sample can point at. -- `string_table` (`string[]`) — shared strings referenced everywhere else by index. -- `attribute_table` (`KeyValueAndUnit[]`) — key/value/unit triples (UCUM unit as a string index), +- `string_table` (`string[]`): shared strings referenced everywhere else by index. +- `attribute_table` (`KeyValueAndUnit[]`): key/value/unit triples (UCUM unit as a string index), referenced by index from `Profile`, `Sample`, `Location`, and `Mapping`. -- `stack_table` (`Stack[]`) — a deduplicated call stack (sequence of `Location` indices), shared +- `stack_table` (`Stack[]`): a deduplicated call stack (sequence of `Location` indices), shared across samples that hit the same stack. `Profile` (the batch-level container) then carries: `sample_type` (a `ValueType` = measurement @@ -100,7 +101,7 @@ kind + unit, e.g. `cpu`/`nanoseconds`), `samples[]`, `time_unix_nano`, `duration (`original_payload_format` + `original_payload` bytes) for lossless pprof round-trip. `Sample` is the actual observation: a `stack_index` into `stack_table`, `attribute_indices` for -sample-specific attributes, `values[]` (the measured quantities — e.g. CPU ns, allocated bytes), +sample-specific attributes, `values[]` (the measured quantities, e.g. CPU ns, allocated bytes), optional `timestamps_unix_nano[]`, and a `link_index` into `link_table` (0 = none, following the dictionary convention that index 0 is a null sentinel throughout). `Source: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto` @@ -108,9 +109,9 @@ dictionary convention that index 0 is a null sentinel throughout). ### Linking profiles to traces A `Sample` MAY carry a `Link` (via `link_index` → `link_table` → `{trace_id, span_id}`), -enabling direct "what code was running during this span" correlation — the profiles concept -page frames this explicitly as answering "which code is responsible" as a complement to an -existing trace/span. Correlation otherwise happens the same way every signal correlates: shared +enabling direct "what code was running during this span" correlation. The profiles concept +page frames this as answering "which code is responsible" as a complement to an existing trace +or span. Otherwise correlation works as it does for every signal: shared Resource/InstrumentationScope context. `Source: https://opentelemetry.io/docs/specs/otel/profiles/`, `Source: https://opentelemetry.io/docs/concepts/signals/profiles/` @@ -121,19 +122,19 @@ Given the shape above, ingesting Profiles is architecturally closer to ingesting relational bundle per batch (dictionary tables + samples referencing them by index) than to appending flat rows. A ClickHouse-style backend would plausibly need either (a) to materialize the dictionary tables into their own keyed tables and resolve indices at query time, or (b) to -fully denormalize/flatten each sample (function names, file, line, mapping) at ingest — trading -storage for query simplicity, the same trade-off Maple already makes for span attributes. -Cross-signal correlation (`trace_id`/`span_id` on `Link`) is the one piece that maps cleanly onto -existing trace-join infrastructure. **Maple does not implement any of this today** — this section -is forward-looking, not a description of current behavior. +fully denormalize each sample (function names, file, line, mapping) at ingest, trading storage +for query simplicity, the same trade-off Maple makes for span attributes. Cross-signal +correlation (`trace_id`/`span_id` on `Link`) is the one piece that maps cleanly onto existing +trace-join infrastructure. **Maple does not implement any of this today.** This section is +forward-looking. ### Collector-side status (pointer, not spec) -The OpenTelemetry Collector is **not part of the specification** — it's a reference -implementation — but is worth noting as context: the 2026 Alpha announcement highlights a -`pprof` receiver, Kubernetes metadata enrichment, OTTL transforms for profiles, and an -eBPF eBPF profiling agent donated by Elastic now distributed as part of the official Collector -distribution. None of this is spec-mandated behavior. +The OpenTelemetry Collector is **not part of the specification**; it is a reference +implementation. As context: the 2026 Alpha announcement highlights a `pprof` receiver, +Kubernetes metadata enrichment, OTTL transforms for profiles, and an eBPF profiling agent +donated by Elastic, now distributed as part of the official Collector distribution. None of +this is spec-mandated behavior. `Source: https://opentelemetry.io/blog/2026/profiles-alpha/` · Collector docs (non-spec): `https://github.com/open-telemetry/opentelemetry-collector` @@ -145,16 +146,16 @@ Collector docs (non-spec): `https://github.com/open-telemetry/opentelemetry-coll **Stable.** `Source: https://opentelemetry.io/docs/specs/otel/schemas/` -The Schema File Format sub-spec (versions 1.0.0 and 1.1.0) is separately versioned; the current -file format spec (1.1.0) is marked **Development** in its own document header even though the -overall "Telemetry Schemas" concept page is Stable — read the file-format details as less settled -than the top-level mechanism. +The Schema File Format sub-spec (versions 1.0.0 and 1.1.0) is versioned separately. The current +file format spec (1.1.0) is marked **Development** in its own document header, even though the +"Telemetry Schemas" concept page is Stable. Read the file-format details as less settled than +the top-level mechanism. `Source: https://opentelemetry.io/docs/specs/otel/schemas/file_format_v1.1.0/` ### Why schemas exist Semantic conventions evolve (attributes get renamed, split, etc.), but telemetry producers, -telemetry consumers, and the semantic conventions themselves all evolve independently — "the +telemetry consumers, and the semantic conventions themselves all evolve independently: "the coupling complicates the independent evolution of these 3 parties." A schema is the versioned, machine-readable description of _what changed between semconv versions_ so a consumer can transform old-shaped data into the shape it expects (or vice versa) without bespoke code per @@ -163,19 +164,19 @@ field rename. `Source: https://opentelemetry.io/docs/specs/otel/schemas/` ### `schema_url` mechanics end-to-end 1. **Format:** `http[s]://server[:port]/path/`. Everything before `` is the - **Schema Family** identifier — every version in a family shares that prefix. + **Schema Family** identifier. Every version in a family shares that prefix. 2. **Where it's attached in OTLP:** `schema_url` fields exist on `ResourceSpans`, - `ResourceMetrics`, `ResourceLogs` (applies to the Resource and everything nested under it) and - separately on the `InstrumentationLibrary*`/scope-level messages (applies to just that - scope's data). **If both are set, the scope-level value takes precedence** over the - resource-level one for the data under that scope. -3. **Immutability + caching:** "Schema files are immutable once they are published" — once you've + `ResourceMetrics`, `ResourceLogs` and on the scope-level messages + `ScopeSpans`/`ScopeMetrics`/`ScopeLogs` (formerly `InstrumentationLibrary*`). They cover + different data. The resource-level value applies only to the `resource` field. Each scope-level + value applies to that scope's spans, metrics, or logs. +3. **Immutability + caching:** "Schema files are immutable once they are published". Once you've resolved a `schema_url` to a file, it is safe to cache that resolution permanently. Fetchers must follow HTTP redirects. Consumers/backends MAY bundle known schema files at build time instead of fetching over the network at all. -4. **Versioning:** MAJOR.MINOR.PATCH, SemVer 2.0 ordering, and — importantly for anyone mapping - this to semconv — **"OpenTelemetry schema version numbers match OpenTelemetry Semantic - Conventions version numbers,"** i.e. schema `1.27.0` corresponds to semconv `1.27.0`. +4. **Versioning:** MAJOR.MINOR.PATCH with SemVer 2.0 ordering. For anyone mapping this to + semconv: **"OpenTelemetry schema version numbers match OpenTelemetry Semantic Conventions + version numbers,"** so schema `1.27.0` corresponds to semconv `1.27.0`. `Source: https://opentelemetry.io/docs/specs/otel/schemas/` @@ -206,8 +207,8 @@ versions: - `schema_url`'s trailing version must equal the highest version key present under `versions`. - The **`all`** section's transforms apply first, to every signal type; then per-signal sections apply on top. -- Transformations are **intentionally minimal by design** ("we intentionally limit the types of - transformations of schemas to the bare minimum") — this is not a general ETL DSL. +- Transformations are **intentionally minimal** ("we intentionally limit the types of + transformations of schemas to the bare minimum"). This is not a general ETL DSL. Transform types by section, per the 1.1.0 format: @@ -220,7 +221,7 @@ Transform types by section, per the 1.1.0 format: | `metrics` | `rename_metrics`, `rename_attributes`, `split` | | `logs` | `rename_attributes` | -`rename_attributes` — the one most relevant to Maple's existing manual coalescing — takes an +`rename_attributes`, the one most relevant to Maple's manual coalescing, takes an `attribute_map` of old→new names, optionally scoped to specific span names / event names / metric names via `apply_to_spans` / `apply_to_events` / `apply_to_metrics`: @@ -237,9 +238,9 @@ names via `apply_to_spans` / `apply_to_events` / `apply_to_metrics`: attribute's value (e.g. `system.paging.operations` split by `direction` into `system.paging.operations.in` / `.out`). -Version 1.1.0's change versus 1.0.0 was introducing this structured, declarative transform -framework so tooling can walk a chain of schema versions and apply/reverse the diffs -automatically, rather than each consumer hand-rolling migrations. +Version 1.1.0 added the `split` transform to the `metrics` section; the rest of the declarative +transform framework already existed in 1.0.0. Tooling walks a chain of schema versions and +applies or reverses the diffs automatically, instead of each consumer hand-rolling migrations. `Source: https://opentelemetry.io/docs/specs/otel/schemas/file_format_v1.1.0/` ### Applying transformations (the pattern Maple's manual coalescing approximates) @@ -248,22 +249,21 @@ Two implementation patterns the spec describes: 1. **Schema-aware backend:** compare the `schema_url` on incoming data against the schema version the backend wants; walk the schema file's declared changes between those versions; - apply them (e.g., the spec's own example — renaming `deployment.environment` to - `environment`). This is precisely the shape of the `deployment.environment` → - `deployment.environment.name` rename Maple's `apps/ingest` currently handles by **dual-emitting - both keys** (per CLAUDE.md) rather than by reading `schema_url` and transforming — i.e. we're - doing the outcome without the mechanism. + apply them (e.g. the spec's own example, renaming `deployment.environment` to + `environment`). This is the shape of the `deployment.environment` → + `deployment.environment.name` rename, which Maple handles by **dual-emitting both keys** from + its own services and coalescing on read, not by reading `schema_url` and transforming. We get + the outcome without the mechanism. 2. **Collector-Assisted Schema Transformation:** delegate the conversion to an OpenTelemetry Collector's Schema Translate Processor sitting in the pipeline, so producers/consumers never need schema logic themselves. `Source: https://opentelemetry.io/docs/specs/otel/schemas/` -**Note for Maple:** we do not currently read `schema_url` from ingested `ResourceSpans` / -`ResourceMetrics` / `ResourceLogs` anywhere in the ingest gateway or collector config (verify -against `apps/ingest` if this needs to be load-bearing) — the legacy-field coalescing described in -CLAUDE.md (`db.statement*`/`db.system` ↔ `db.query.text`/`db.system.name`, -`deployment.environment` ↔ `deployment.environment.name`) is hand-written per-field logic, not +**Note for Maple:** `apps/ingest` stores `schema_url` from ingested `ResourceSpans` / +`ResourceMetrics` / `ResourceLogs` as a column but never uses it to transform data. The +legacy-field coalescing (`db.statement`/`db.system` ↔ `db.query.text`/`db.system.name`, +`deployment.environment` ↔ `deployment.environment.name`) is hand-written per-attribute SQL, not schema-driven. If schema-aware coalescing is ever generalized, this is the mechanism to build on. --- @@ -288,12 +288,12 @@ schema-driven. If schema-aware coalescing is ever generalized, this is the mecha suffixing). - **Character sanitization:** characters Prometheus discourages in metric names **SHOULD** be replaced with `_` by default, and repeated `_` collapsed to one. -- **Resource attributes → `job`/`instance`/`target_info`:** `service.namespace` + +- **Resource attributes → `job`/`instance`/`target_info`:** `service.namespace` and `service.name` **MUST** combine into `/` (or just `` if no namespace) to form the `job` label; `service.instance.id`, if present, **MUST** become the `instance` label (else `instance` is emitted empty). Other resource attributes **SHOULD** become a synthetic - `target_info` metric (an OpenMetrics "info" metric, or — if the client library lacks info-metric - support — a gauge named `target_info` with constant value `1`) or **MUST** be dropped. + `target_info` metric (an OpenMetrics "info" metric or, if the client library lacks info-metric + support, a gauge named `target_info` with constant value `1`) or **MUST** be dropped. _(Development status.)_ - **Untyped/Gauge round-trip:** exporting an OTLP Gauge back toward Prometheus, if the datapoint carries a `prometheus.type` attribute equal to `unknown`, it **MUST** become a Prometheus @@ -301,16 +301,15 @@ schema-driven. If schema-aware coalescing is ever generalized, this is the mecha ### Prometheus → OTLP naming rules -- By default, label keys/values **MUST NOT** be altered (e.g., no automatic `_`→`.` translation) - — the opposite direction is lossier/more speculative than OTLP→Prom, so the spec is - conservative here. +- By default, label keys/values **MUST NOT** be altered (e.g. no automatic `_`→`.` translation). + This direction is lossier and more speculative than OTLP→Prom, so the spec is conservative. - `UNIT` metadata, if present on the Prometheus side, **MUST** be converted into the OTLP metric's unit. - A Prometheus **Unknown**-typed metric **MUST** be converted to an **OTLP Gauge** on ingest. _(Development status.)_ - Summary quantiles carry a `quantile` label with the stringified float (e.g. `0.99`); histogram - buckets use Prometheus's standard `le` (less-than-or-equal) label — the spec does not add new - bucket-naming rules beyond these Prometheus/OpenMetrics conventions. + buckets use Prometheus's standard `le` (less-than-or-equal) label. The spec adds no bucket-naming + rules beyond these Prometheus/OpenMetrics conventions. `Source: https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/` @@ -343,7 +342,7 @@ OpenCensus, and the shim bridges the two until migration completes. ### Status **Deprecated** as of **March 2026**; removal not before **March 2027**. Same posture as -OpenCensus above — existing shims MAY continue, new implementation isn't required. +OpenCensus above: existing shims MAY continue, new implementation isn't required. `Source: https://opentelemetry.io/docs/specs/otel/compatibility/opentracing/` ### What it guarantees / span kind & reference mapping @@ -352,8 +351,8 @@ The shim implements the OpenTracing API as a bridge over the OpenTelemetry API/S exposing OTel internals to OpenTracing-instrumented code. The part most likely to surface as an artifact in ingested data: OpenTracing's two **Reference** types (`Child Of`, `Follows From`) become OpenTelemetry **Links**, with the original reference type preserved as an -**`opentracing.ref_type`** span attribute — so seeing that attribute key in customer traces is a -tell that the data passed through this shim. The first `Child Of` reference (or the first +**`opentracing.ref_type`** span attribute. Seeing that attribute key in customer traces is a tell +that the data passed through this shim. The first `Child Of` reference (or the first reference of any kind if none is `Child Of`) is chosen as the span's actual OTel parent; Baggage from referenced spans is merged into the new span's initial Baggage. @@ -368,23 +367,23 @@ relationships, violating OpenTracing's explicit-propagation-only semantics. ## 6. Other emerging spec-tree items (pointers) -- **Entities** — a Development-stage remodeling of the Resource concept: each signal (including +- **Entities**: a Development-stage remodeling of the Resource concept. Each signal (including Profiles) will be able to attach one or more typed Entities (`k8s.cluster`, `host`, `container`, …), each with stable identifying attributes plus mutable descriptive attributes, layered on top of the existing Resource attribute pool for OTLP 1.x backward compatibility. - Not otherwise covered in this doc set — treat as a future Resource-model change to watch, not + Not otherwise covered in this doc set. Treat it as a future Resource-model change to watch, not something to implement against yet. `Source: https://opentelemetry.io/docs/specs/otel/entities/data-model/` -- **Trace Context in non-OTLP Log Formats** — a compatibility spec (sibling to OpenCensus/ +- **Trace Context in non-OTLP Log Formats**: a compatibility spec (sibling to OpenCensus/ OpenTracing/Prometheus above) for propagating trace context through log formats that aren't - OTLP; noted here as a pointer since it's linked from the same compatibility index but out of - this pass's detailed scope. + OTLP. Listed as a pointer only; it is linked from the same compatibility index but out of + scope here. `Source: https://opentelemetry.io/docs/specs/otel/compatibility/logging_trace_context/` - **The Collector itself is not part of the OpenTelemetry specification.** Its docs (`https://github.com/open-telemetry/opentelemetry-collector`) describe a reference implementation's data flow, processors, and receivers (including the `pprof` receiver and - Schema Translate Processor mentioned above) — useful for understanding available tooling, but - none of it is spec-mandated or spec-versioned. + Schema Translate Processor mentioned above). Useful for understanding available tooling; none + of it is spec-mandated or spec-versioned. --- @@ -404,4 +403,4 @@ relationships, violating OpenTracing's explicit-propagation-only semantics. - OpenTracing compatibility (shim): https://opentelemetry.io/docs/specs/otel/compatibility/opentracing/ - Trace context in non-OTLP log formats: https://opentelemetry.io/docs/specs/otel/compatibility/logging_trace_context/ - Entities data model: https://opentelemetry.io/docs/specs/otel/entities/data-model/ -- Document status definitions (Draft/Experimental/Stable/Deprecated/Removed, Alpha): https://opentelemetry.io/docs/specs/otel/document-status/ +- Document status definitions (Development/Alpha/Beta/RC/Stable/Deprecated/Unmaintained): https://opentelemetry.io/docs/specs/otel/document-status/ diff --git a/docs/otel-spec/logs.md b/docs/otel-spec/logs.md index e4f6164b9e..164d4d263b 100644 --- a/docs/otel-spec/logs.md +++ b/docs/otel-spec/logs.md @@ -1,68 +1,65 @@ # Logs & Events (Data Model, Bridge API & SDK) -This page documents the OpenTelemetry **Logs** specification: the log data model, the Logs API -(a.k.a. the "Bridge API"), the Logs SDK, and the current (2026) direction on **Events** as -log-based records rather than a separate signal. It is a reference for spec-compliance checks and -for a best-practices skill — every claim below is sourced from the official specification (or the -OTLP proto, for wire-level field names). +This page covers the OpenTelemetry **Logs** specification: the log data model, the Logs API (the +"Bridge API"), the Logs SDK, and the current (2026) direction on **Events** as log-based records +instead of a separate signal. It backs spec-compliance checks and a best-practices skill. Every claim +is sourced from the official specification, or from the OTLP proto for wire-level field names. ## Relevance to Maple -> Maple's Rust ingest gateway (`apps/ingest`) receives OTLP logs and forwards them to the -> collector → ClickHouse/Tinybird. Maple is primarily a **consumer/backend** of OTel log data, not -> an app-side log-emitting SDK author. The parts of this spec that matter most to us: +> Maple's Rust ingest gateway (`apps/ingest`) receives OTLP logs and writes them to +> Tinybird/ClickHouse, or forwards them to a collector, depending on `INGEST_WRITE_MODE`. Maple is +> mainly a **consumer/backend** of OTel log data. The parts of this spec that matter most to us: > -> - **Data model field semantics** (`docs/otel-spec/logs.md#logrecord-fields`) — how to interpret -> `Timestamp` vs `ObservedTimestamp`, `SeverityNumber` vs `SeverityText`, and `Body` as an -> `AnyValue` when mapping incoming OTLP `LogRecord`s into our ClickHouse schema. -> - **Severity normalization** — our dashboards should key error/warn facets off `SeverityNumber` -> ranges (see CLAUDE.md: "Span Status Codes: use title case" — analogous care is needed for -> severity display strings, which the spec also title-cases: `Trace`/`Debug`/`Info`/`Warn`/ -> `Error`/`Fatal`). -> - **Trace correlation fields** (`TraceId`/`SpanId`/`TraceFlags`) — these are how our logs UI -> should link a log line back to a trace/span; validity rules below (e.g. `SpanId` implies -> `TraceId`) matter for defensive parsing of malformed producers. -> - **EventName / Events direction** — OpenTelemetry is actively deprecating `Span.AddEvent` in -> favor of log-based events (see [Events](#events)). If Maple ever ingests or displays -> "events" as a first-class concept, the wire format is `LogRecord.event_name` (proto field 12), -> not a separate signal. Self-instrumentation should also check whether our own Rust ingest -> gateway or web app should switch to log-based events for anything currently using span events. -> - **SDK batch/limits env vars** — not directly consumed by Maple's backend (we don't run a Logs -> SDK ourselves in the hot ingest path), but relevant if/when Maple's own services (API, web, -> alerting) adopt an OpenTelemetry Logs SDK for self-instrumented application logging, and for -> validating third-party SDK behavior when debugging customer data gaps (e.g. "why did 2000 logs -> arrive in one batch 30s late" → default `OTEL_BLRP_*` values below). +> - **Data model field semantics** ([§2.1](#21-logrecord-fields)): how to interpret `Timestamp` vs +> `ObservedTimestamp`, `SeverityNumber` vs `SeverityText`, and `Body` as an `AnyValue` when mapping +> incoming OTLP `LogRecord`s into our ClickHouse schema. +> - **Severity normalization.** Error/warn facets should key off `SeverityNumber` ranges, not +> `SeverityText`. The spec's display names are upper case (`TRACE`, `DEBUG`, `INFO`, `WARN`, +> `ERROR`, `FATAL`), unlike span status codes, which Maple writes in Title case (`Ok`/`Error`/`Unset`, +> see `CLAUDE.md`). +> - **Trace correlation fields** (`TraceId`/`SpanId`/`TraceFlags`). These link a log line to its +> trace and span in the logs UI. The validity rules below (e.g. `SpanId` implies `TraceId`) matter +> for defensive parsing of malformed producers. +> - **EventName and the Events direction.** OpenTelemetry is deprecating `Span.AddEvent` in favor of +> log-based events (see [Events](#5-events)). If Maple ingests or displays "events" as a first-class +> concept, the wire format is `LogRecord.event_name` (proto field 12), not a separate signal. Our own +> services should check whether anything that uses span events today should switch to log-based +> events. +> - **SDK batch/limit env vars.** The ingest gateway already runs a Logs SDK for its own logs +> (`SdkLoggerProvider` with the `opentelemetry-appender-tracing` bridge, in +> `apps/ingest/src/main.rs`). The defaults also help when debugging customer data gaps (e.g. "why +> did 2000 logs arrive in one batch 30s late": see the `OTEL_BLRP_*` defaults below). --- ## 1. Overview -**Stability: no single badge for the overview page; it links out to Stable Data Model, Stable -API, Stable SDK (see [Stability summary](#stability-summary) below).** +**Stability: the overview page has no single badge.** It links to the Stable Data Model, API, and SDK +(see [Stability summary](#6-stability-summary) below). Source: https://opentelemetry.io/docs/specs/otel/logs/ -Unlike traces and metrics, OpenTelemetry does **not** introduce a brand-new logging API that -applications are expected to adopt wholesale. Instead the logs spec is designed to **embrace -existing logging libraries** and unify their output with traces/metrics via three correlation -dimensions: - -1. **Temporal correlation** — logs, traces, and metrics all carry timestamps, the most basic form - of correlation. -2. **Contextual (execution) correlation** — a `LogRecord` can carry `TraceId`/`SpanId` so a log - line is attributable to the exact request/span that produced it. -3. **Resource correlation** — a `LogRecord` carries the same `Resource` model used by traces and - metrics (e.g. identical `k8s.pod.*` attributes), so a log, trace, and metric from the same - process describe their origin identically. - -Three ways logs reach OpenTelemetry are called out on this page: - -- **Existing/legacy log files**, parsed and enriched by the OpenTelemetry Collector (with or - without an intermediate agent like FluentBit). -- **Log appenders / bridges** — library-specific glue that hooks an existing logging framework - (e.g. Log4j, Winston, `slog`) so it emits through the OpenTelemetry data model and automatically - injects trace context into each record. -- **Direct logging** — new application code emits `LogRecord`s straight through the Logs API/SDK - over OTLP, skipping files/parsers/rotation entirely. +Unlike traces and metrics, OpenTelemetry does **not** introduce a new logging API that applications +must adopt wholesale. The logs spec **embraces existing logging libraries** and unifies their output +with traces and metrics along three correlation dimensions: + +1. **Temporal correlation.** Logs, traces, and metrics all carry timestamps, the most basic form of + correlation. +2. **Contextual (execution) correlation.** A `LogRecord` can carry `TraceId`/`SpanId`, so a log line + is attributable to the request and span that produced it. +3. **Resource correlation.** A `LogRecord` carries the same `Resource` model as traces and metrics + (e.g. identical `k8s.pod.*` attributes), so a log, trace, and metric from one process describe + their origin identically. + +The page names three ways logs reach OpenTelemetry: + +- **Existing/legacy log files**, parsed and enriched by the OpenTelemetry Collector (with or without + an intermediate agent like FluentBit). +- **Log appenders / bridges**: library-specific glue that hooks an existing logging framework (e.g. + Log4j, Winston, `slog`) so it emits through the OpenTelemetry data model and injects trace context + into each record. +- **Direct logging**: new application code emits `LogRecord`s through the Logs API/SDK over OTLP, + skipping files, parsers, and rotation. Source: https://opentelemetry.io/docs/specs/otel/logs/ @@ -76,31 +73,31 @@ Source: https://opentelemetry.io/docs/specs/otel/logs/data-model/ (spec source: ### 2.1 LogRecord fields -All fields are conceptually optional at the data-model level (a valid `LogRecord` may have zero -fields populated — see [§2.4](#24-minimal-validity--missing-fields)); the table below states each -field's type, meaning, and the specific SHOULD/MUST guidance the spec attaches to it. +All fields are optional at the data-model level. A valid `LogRecord` may have zero fields populated +(see [§2.4](#24-minimal-validity--missing-fields)). The table gives each field's type, meaning, and the +SHOULD/MUST guidance the spec attaches to it. | Field | Type | Meaning | Requirement-level notes | | ---------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Timestamp` | `uint64` nanoseconds since Unix epoch | Time the event occurred, per the **origin clock** (i.e., the producer's clock, which may be unsynchronized) | Optional. May be omitted for early instrumentation phases per the spec's incremental-adoption note. | +| `Timestamp` | `uint64` nanoseconds since Unix epoch | Time the event occurred, per the **origin clock** (the producer's clock, which may be unsynchronized) | Optional. May be omitted for early instrumentation phases per the spec's incremental-adoption note. | | `ObservedTimestamp` | `uint64` nanoseconds since Unix epoch | Time the OpenTelemetry collection/observation system (SDK, collector, receiver) **first observed/recorded** the event, e.g. file-read time for tailed logs | SHOULD be set when the event's true origin timestamp is unavailable or untrusted. If unspecified by an SDK's `Emit`, the SDK **SHOULD** set it to current time (see [§4 SDK](#41-logrecord-lifecycle--observedtimestamp-default)). | | `TraceId` | byte sequence (16 bytes on the wire) | W3C Trace Context trace identifier for the request being processed when the log was emitted | Optional. | | `SpanId` | byte sequence (8 bytes on the wire) | Identifier of the specific span being processed when the log was emitted | Optional, but **if `SpanId` is present, `TraceId` SHOULD also be present.** | | `TraceFlags` | 1 byte | W3C trace flags (currently only defines the `SAMPLED` bit) | Optional. | -| `SeverityText` | string | The **original**, source-native string representation of severity (e.g. `"WARN"`, `"ERR"`, framework-specific spelling) — preserved as-is, not normalized | Optional. | -| `SeverityNumber` | integer, 0–24 | The **normalized** severity, mapped onto OpenTelemetry's fixed 1–24 numeric scale (0 = unspecified) | Optional. See full table in [§2.2](#22-severitynumber-reference-table). | +| `SeverityText` | string | The **original**, source-native severity string (e.g. `"WARN"`, `"ERR"`, framework-specific spelling), preserved as-is | Optional. | +| `SeverityNumber` | integer, 0-24 | The **normalized** severity, mapped onto OpenTelemetry's fixed 1-24 numeric scale (0 = unspecified) | Optional. See full table in [§2.2](#22-severitynumber-reference-table). | | `Body` | `AnyValue` | The log payload: a human-readable message string, **or** structured data (maps/arrays/scalars) | Optional. Body **MUST support `AnyValue`** so structured logs from applications survive without lossy stringification. | | `Resource` | `Resource` | Describes the entity producing the log (same `Resource` model as traces/metrics) | Optional at the data-model layer; in practice always populated by the SDK. | | `InstrumentationScope` | `InstrumentationScope` | The logger (name/version/schema_url/attributes) that emitted the record; stable across many log events from the same source | Optional at the data-model layer; populated by the SDK. | -| `Attributes` | collection of key-value pairs | Additional structured data about this **specific occurrence** (as opposed to `Resource`, which describes the origin) | Optional. Subject to the same attribute limits as spans (count/length limits, see [§4.3](#43-logrecord-limits)). | -| `EventName` | string | Identifies the **class/type** of the event this log record represents | Optional. A non-empty `EventName` makes this record an **Event** — see [§5](#5-events). | +| `Attributes` | collection of key-value pairs | Additional structured data about this **specific occurrence** (as opposed to `Resource`, which describes the origin) | Optional. Subject to the same attribute limits as spans (count/length limits, see [§4.5](#45-logrecord-limits)). | +| `EventName` | string | Identifies the **class/type** of the event this log record represents | Optional. A non-empty `EventName` makes this record an **Event** (see [§5](#5-events)). | Source: https://opentelemetry.io/docs/specs/otel/logs/data-model/#log-and-event-record-definition ### 2.2 SeverityNumber reference table -The severity scale is a fixed 1–24 integer range divided into six named bands, each with a -"finer-grained" sub-level (`+1`..`+4`) for systems whose native severities don't align exactly: +The severity scale is a fixed 1-24 integer range in six named bands. Each band has four values +(`+1`..`+3` above the base) for systems whose native severities do not align exactly: | SeverityNumber | Name | Meaning | | -------------- | --------------- | --------------------------------------------------------------------------------------- | @@ -130,67 +127,62 @@ The severity scale is a fixed 1–24 integer range divided into six named bands, | 23 | `FATAL3` | | | 24 | `FATAL4` | | -Normative guidance for producers mapping a **source system's own severities** onto this scale: +Guidance for producers mapping a **source system's own severities** onto this scale: -- If a source severity level maps to a **single** OpenTelemetry `SeverityNumber` range (e.g. - source "WARN" → the `WARN` band), use the **lowest/smallest** value in that band (i.e. plain - `WARN` = 13, not 14–16). -- If a source has **multiple** distinct severities that all fall into the same OpenTelemetry band - (e.g. a framework with both "notice" and "warning" both conceptually "warn-ish"), spread them - across that band's four values (13–16) in order of increasing importance so relative ordering is - preserved. -- **`SeverityNumber >= ERROR (17)` is a signal the record represents an erroneous situation** — this - is the normative hook backends should use to classify a log line as an error for dashboards/alerts, - not string-matching on `SeverityText`. +- If one source severity maps to an OpenTelemetry band (e.g. source "WARN" → the `WARN` band), use + the **smallest** value in that band (plain `WARN` = 13, not 14-16). +- If a source has **several** severities that fall into one band (e.g. a framework with both "notice" + and "warning"), spread them across that band's four values (13-16) in order of increasing + importance, so relative ordering is preserved. +- **`SeverityNumber >= ERROR (17)` signals an erroneous situation.** Backends should use this to + classify a log line as an error for dashboards and alerts, not string-matching on `SeverityText`. Source: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitynumber and https://opentelemetry.io/docs/specs/otel/logs/data-model/#displaying-severity ### 2.3 Body / AnyValue semantics -`Body` is typed as `AnyValue`, the same recursive value type used for attribute values but -extended to structures: a scalar (string/bool/int/double/bytes), or a structured value — an array -of `AnyValue`, or a map of string keys to `AnyValue`. This is explicitly to preserve **structured -logging** semantics (e.g. a JSON-object log line) without forcing lossy stringification into a -single message field. The spec states the data model **MUST** support `AnyValue` for `Body` for -exactly this reason — different occurrences of a log from the same statement/call-site can still -vary in shape (e.g. conditionally-included fields). +`Body` is typed as `AnyValue`, the same recursive value type used for attribute values. It is either a +scalar (string/bool/int/double/bytes) or a structure: an array of `AnyValue`, or a map of string keys +to `AnyValue`. This preserves **structured logging** (e.g. a JSON-object log line) without forcing +lossy stringification into one message field. The data model **MUST** support `AnyValue` for `Body` +because occurrences of a log from the same call site can vary in shape (e.g. conditionally included +fields). Source: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-body ### 2.4 Minimal validity / missing fields -The data model does not mandate a minimum set of populated fields for a record to be "valid" — an -empty `LogRecord` is technically legal, though useless. Practical guidance: +The data model mandates no minimum set of populated fields. An empty `LogRecord` is legal, though +useless. Practical guidance: -- **`Timestamp` vs `ObservedTimestamp`**: when a consumer/exporter needs to reduce a record to a - single timestamp (e.g. exporting to a system that only understands one time field), the rule is: - **use `Timestamp` if it is present, otherwise fall back to `ObservedTimestamp`.** A consumer - should never treat a record with only `ObservedTimestamp` set as invalid — that's the expected - shape for tailed/parsed legacy logs where the true origin time is unknown or untrustworthy. -- **`SpanId` without `TraceId`** should be treated as a malformed/unusual record by a strict - consumer, since the spec says `TraceId` SHOULD accompany `SpanId`; a defensive backend (like - Maple's ingest path) should not assume the reverse is disallowed (a record MAY carry `TraceId` - alone, e.g. no active span at emit time). +- **`Timestamp` vs `ObservedTimestamp`**: when a consumer or exporter must reduce a record to one + timestamp (e.g. exporting to a system with a single time field), **use `Timestamp` if present, + otherwise `ObservedTimestamp`.** A record with only `ObservedTimestamp` set is valid. It is the + expected shape for tailed or parsed legacy logs whose origin time is unknown or untrusted. +- **`SpanId` without `TraceId`** is a malformed record to a strict consumer, since the spec says + `TraceId` SHOULD accompany `SpanId`. The reverse is allowed: a record MAY carry `TraceId` alone + (e.g. no active span at emit time), and a defensive backend like Maple's ingest path should accept + it. Source: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-observedtimestamp and https://opentelemetry.io/docs/specs/otel/logs/data-model/#trace-context-fields -### 2.5 Data Model Appendix — mappings from other systems +### 2.5 Data Model Appendix: mappings from other systems **Stability: Stable (companion to the data model).** Source: https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/ The appendix gives field-by-field mapping tables from common log formats onto the OpenTelemetry -`LogRecord`, useful when writing or auditing a receiver/exporter. Examples covered: RFC 5424 -Syslog, Windows Event Log, SignalFx Events, Splunk HEC, Log4j, Zap, Apache HTTP Server access -logs, AWS CloudTrail, Google Cloud Logging, and the Elastic Common Schema. Pattern is consistent -across all of them: native timestamp → `Timestamp`, native severity/level → `SeverityNumber` -(+ `SeverityText` for the original string), free-form message → `Body`, host/service/cloud -identity fields → `Resource`, everything else structured → `Attributes`. A companion **Appendix B** -gives severity-level equivalency mappings (e.g. Syslog `Debug`/Java `FINEST` → `TRACE`(1); Syslog -`Emergency`/Log4j `FATAL` → `FATAL`(21)), which is the authoritative source if Maple ever needs to -normalize a specific framework's severities for ingestion. +`LogRecord`, useful when writing or auditing a receiver or exporter. Formats covered: RFC 5424 Syslog, +Windows Event Log, SignalFx Events, Splunk HEC, Log4j, Zap, Apache HTTP Server access logs, AWS +CloudTrail, Google Cloud Logging, and the Elastic Common Schema. The pattern is the same for all of +them: native timestamp → `Timestamp`, native severity/level → `SeverityNumber` (+ `SeverityText` for +the original string), free-form message → `Body`, host/service/cloud identity → `Resource`, other +structured fields → `Attributes`. **Appendix B** gives severity equivalence mappings (e.g. Java +`FINEST` → `TRACE`(1); Syslog `Debug` → `DEBUG`(5); Syslog `Emergency` and Log4j `FATAL` → +`FATAL`(21)). It is the authoritative source if Maple needs to normalize a specific framework's +severities at ingest. Source: https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/ @@ -198,91 +190,88 @@ Source: https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/ ## 3. Logs API ("Bridge API") -**Stability: Stable, except where otherwise specified** (the page explicitly calls out an -"Ergonomic API" as still in **Development**). +**Stability: Stable, except where otherwise specified** (the page calls out an "Ergonomic API" as +still in **Development**). Source: https://opentelemetry.io/docs/specs/otel/logs/api/ (spec source: [`specification/logs/api.md`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/api.md)) ### 3.1 Intended audience -This is explicitly called the **Logs Bridge API** in the spec, and it is **not meant for -application developers to call directly in everyday code**. The primary audience is: +The spec calls this the **Logs Bridge API**. It is **not meant for application developers to call +directly** in everyday code. The primary audience: -- **Log appender / bridge authors** — people writing the glue that lets an existing logging - library (Log4j, Winston, `slog`, Python `logging`, etc.) emit through OpenTelemetry. -- Secondarily, instrumentation and instrumented-library authors, who _may_ call it directly, though - a language may offer a more ergonomic wrapper for that purpose. +- **Log appender / bridge authors**, who write the glue that lets an existing logging library (Log4j, + Winston, `slog`, Python `logging`, etc.) emit through OpenTelemetry. +- Secondarily, instrumentation and instrumented-library authors. They _may_ call it directly, though a + language may offer a more ergonomic wrapper for that purpose. -This matters for Maple only insofar as we should not expect our own application code (if we ever -add OTel-based logging to the Rust ingest gateway, `apps/api`, etc.) to call this low-level API -directly — we'd reach for a logging-library integration/bridge, or the language's ergonomic -wrapper, not `Logger.Emit` by hand. +For Maple: our services should not call `Logger.Emit` by hand. They should use a logging-library +bridge or the language's ergonomic wrapper. The ingest gateway already does this through the Rust +`tracing` bridge (`OpenTelemetryTracingBridge` in `apps/ingest/src/main.rs`). Source: https://opentelemetry.io/docs/specs/otel/logs/api/#overview ### 3.2 LoggerProvider -Entry point; **MUST** provide a "Get a Logger" operation, parameterized by the **Instrumentation -Scope**: `name` (required), `version` (optional), `schema_url` (optional), `attributes` (optional). -Also must support access to a global default `LoggerProvider`. +Entry point. It **MUST** provide a "Get a Logger" operation, parameterized by the **Instrumentation +Scope**: `name` (required), `version` (optional), `schema_url` (optional), `attributes` (optional). It +must also support access to a global default `LoggerProvider`. Source: https://opentelemetry.io/docs/specs/otel/logs/api/#loggerprovider -### 3.3 Logger — Emit a LogRecord +### 3.3 Logger: Emit a LogRecord The `Logger` interface's core operation accepts (all optional except where noted): - `Timestamp`, `ObservedTimestamp` -- `Context` — **"When implicit Context is supported, then this parameter SHOULD be optional and if +- `Context`: **"When implicit Context is supported, then this parameter SHOULD be optional and if unspecified then MUST use current Context. When only explicit Context is supported, this - parameter SHOULD be required."** This is the mechanism by which trace correlation - (`TraceId`/`SpanId`) gets attached to a log record automatically: the SDK reads the active - span/trace out of whatever `Context` is in scope (implicit ambient context, or an explicitly - passed one) at emit time and copies its `TraceId`/`SpanId`/`TraceFlags` onto the record. + parameter SHOULD be required."** This is how trace correlation gets attached automatically. At emit + time the SDK reads the active span out of the `Context` in scope (implicit or explicitly passed) and + copies its `TraceId`/`SpanId`/`TraceFlags` onto the record. - `SeverityNumber`, `SeverityText` - `Body` - `Attributes` - `EventName` -- `Exception` — optional/MAY be accepted as a convenience for exception-recording +- `Exception`: optional; MAY be accepted as a convenience for exception recording ### 3.4 Enabled `Logger` **SHOULD** provide an `Enabled(context, severityNumber, eventName)` operation returning a -boolean, so callers can cheaply skip expensive log-construction work when the record would be -dropped anyway. The spec notes the result "is not always static, it can change over time" (e.g. -dynamic sampling/config), so callers should be documented to call it fresh each time rather than -caching the result. +boolean, so callers can skip expensive log construction when the record would be dropped. The spec +notes the result "is not always static, it can change over time" (e.g. dynamic sampling or config), so +callers should call it each time instead of caching the result. ### 3.5 Concurrency -Both `LoggerProvider` and `Logger`: **all methods MUST be documented as safe for concurrent use by +For both `LoggerProvider` and `Logger`, **all methods MUST be documented as safe for concurrent use by default.** -Source (3.2–3.5): https://opentelemetry.io/docs/specs/otel/logs/api/ +Source (3.2-3.5): https://opentelemetry.io/docs/specs/otel/logs/api/ --- ## 4. Logs SDK -**Stability: Stable, except where otherwise specified** — the page explicitly marks +**Stability: Stable, except where otherwise specified.** The page marks these as Development: **`LoggerConfigurator`, per-`LoggerConfig` behavior (`enabled`/`minimum_severity`/`trace_based`), -`Enabled`-based filtering rules, and the "Event to span event bridge processor" as Development.** +`Enabled`-based filtering rules, and the "Event to span event bridge processor".** Source: https://opentelemetry.io/docs/specs/otel/logs/sdk/ (spec source: [`specification/logs/sdk.md`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/sdk.md)) ### 4.1 LogRecord lifecycle / ObservedTimestamp default -The SDK-level `Emit` operation implements the data-model rule from [§2.4](#24-minimal-validity--missing-fields): -**"If Observed Timestamp is unspecified, the implementation SHOULD set it equal to the current -time."** This is the concrete SDK-side guarantee behind the data model's "SHOULD be set" language -for `ObservedTimestamp`. +The SDK-level `Emit` operation implements the data-model rule from +[§2.4](#24-minimal-validity--missing-fields): **"If Observed Timestamp is unspecified, the +implementation SHOULD set it equal to the current time."** This is the SDK-side guarantee behind the +data model's "SHOULD be set" language for `ObservedTimestamp`. ### 4.2 LoggerProvider / Logger (SDK) -- A `LoggerProvider` **MUST** provide a way to configure/attach a `Resource`. -- Owns the set of registered `LogRecordProcessor`s and (Development-status) a `LoggerConfigurator`. -- Supports multiple independent `LoggerProvider` instances in one process. -- **Shutdown**: called once; SDK should make the `Logger` a no-op after shutdown. +- A `LoggerProvider` **MUST** provide a way to configure and attach a `Resource`. +- It owns the registered `LogRecordProcessor`s and (Development) a `LoggerConfigurator`. +- Multiple independent `LoggerProvider` instances may exist in one process. +- **Shutdown**: called once; the SDK should make the `Logger` a no-op after shutdown. - **ForceFlush**: tells all registered processors to flush pending records. - `LoggerConfig` (Development) parameters, per logger: `enabled` (default `true`), `minimum_severity` (default `0`, filters records below this `SeverityNumber`), `trace_based` @@ -292,11 +281,11 @@ for `ObservedTimestamp`. Interface operations: -- **OnEmit** — called synchronously on the emitting thread with a `ReadWriteLogRecord` and the - resolved `Context`; **SHOULD NOT block or throw**. -- **Enabled** (optional) — filtering hook taking context/instrumentation-scope/severity/event-name, - returns bool. -- **Shutdown**, **ForceFlush** — as usual for OTel pipeline components. +- **OnEmit**: called synchronously on the emitting thread with a `ReadWriteLogRecord` and the + resolved `Context`. It **SHOULD NOT block or throw**. +- **Enabled** (optional): filtering hook taking context, instrumentation scope, severity, and event + name; returns bool. +- **Shutdown**, **ForceFlush**: as for other OTel pipeline components. #### Simple Processor @@ -305,8 +294,8 @@ Passes each finished `LogRecord` to its configured `exporter` **immediately**, s #### Batching Processor -Buffers records into batches before handing them to the exporter. Configurable parameters and -**their environment variables** (general SDK env var spec): +Buffers records into batches before handing them to the exporter. Parameters and **their environment +variables** (general SDK env var spec): | Parameter | Env var | Default | Notes | | --------------------- | --------------------------------- | ------------ | ---------------------------------------------------- | @@ -315,8 +304,8 @@ Buffers records into batches before handing them to the exporter. Configurable p | Export timeout | `OTEL_BLRP_EXPORT_TIMEOUT` | `30000` (ms) | Max time allowed for a single export call. | | Max export batch size | `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` | `512` | Must be `<=` max queue size. | -All numeric values must be positive. (`BLRP` = **B**atch **L**og**R**ecord **P**rocessor — same -naming pattern as `BSP` for spans.) +All numeric values must be positive. `BLRP` stands for **B**atch **L**og**R**ecord **P**rocessor, +the same naming pattern as `BSP` for spans. Source: https://opentelemetry.io/docs/specs/otel/logs/sdk/#batching-processor and https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md @@ -325,16 +314,16 @@ https://github.com/open-telemetry/opentelemetry-specification/blob/main/specific ### 4.4 LogRecordExporter - **Export(batch of `ReadableLogRecord`s)** → `Success` | `Failure`. **Must not be called - concurrently** with other `Export` calls on the same exporter instance (the processor guarantees - serialization). -- **ForceFlush** — completes pending exports within a timeout. -- **Shutdown** — releases resources; any subsequent `Export` call **MUST** return `Failure`. + concurrently** with other `Export` calls on the same exporter instance (the processor serializes + calls). +- **ForceFlush**: completes pending exports within a timeout. +- **Shutdown**: releases resources; any later `Export` call **MUST** return `Failure`. Source: https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter ### 4.5 LogRecord limits -The SDK enforces the same family of attribute limits used for spans, applied to `LogRecord.Attributes`: +The SDK applies the same family of attribute limits used for spans to `LogRecord.Attributes`: | Parameter | Env var | Default | Notes | | ---------------------------- | --------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------- | @@ -356,10 +345,10 @@ Source: https://github.com/open-telemetry/opentelemetry-specification/blob/main/ ### 4.7 Concurrency -- `LoggerProvider`: creation of `Logger`s, `ForceFlush`, `Shutdown` must be thread-safe. +- `LoggerProvider`: creating `Logger`s, `ForceFlush`, and `Shutdown` must be thread-safe. - `Logger`: all methods thread-safe. -- `LogRecordExporter`: `ForceFlush`/`Shutdown` thread-safe; `Export` itself is explicitly **not** - required to be concurrency-safe against itself (the processor serializes calls instead). +- `LogRecordExporter`: `ForceFlush`/`Shutdown` thread-safe. `Export` is explicitly **not** required + to be safe against concurrent calls to itself; the processor serializes them. Source: https://opentelemetry.io/docs/specs/otel/logs/sdk/#concurrency-requirements @@ -367,68 +356,60 @@ Source: https://opentelemetry.io/docs/specs/otel/logs/sdk/#concurrency-requireme ## 5. Events -**Stability: Development / actively changing (2026).** This is the area of the spec most in flux -— report here reflects the _current_ normative direction, not a settled long-term API. +**Stability: Development, changing actively (2026).** This is the part of the spec most in flux. This +section reflects the _current_ normative direction, not a settled long-term API. -### 5.1 Current model: Events are log-based, not a separate signal +### 5.1 Current model: Events are log records, not a separate signal -As of the current spec, there is **no separate Event API/SDK**. The historical -`specification/logs/event-api.md` document (a standalone Event API layered on top of the Logs API, -present in older spec versions such as +The current spec has **no separate Event API/SDK**. The historical `specification/logs/event-api.md` +(a standalone Event API layered on the Logs API, present in older versions such as [v1.35.0](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.35.0/specification/logs/event-api.md)) -**no longer exists on `main`** (confirmed 404 against the current `main` branch tree at time of -writing). Events have been folded directly into the log data model: +**no longer exists on `main`** (confirmed 404 against the `main` tree at time of writing). Events are +folded into the log data model: -- An **Event** is defined as **a `LogRecord` with a non-empty `EventName`** — not a distinct - message type. The proto wire representation is `LogRecord.event_name` (field 12, - `opentelemetry/proto/logs/v1/logs.proto`), sitting alongside `time_unix_nano`, - `severity_number`, `body`, `attributes`, `trace_id`, `span_id`, etc. There is no separate +- An **Event** is **a `LogRecord` with a non-empty `EventName`**, not a distinct message type. On the + wire it is `LogRecord.event_name` (field 12, `opentelemetry/proto/logs/v1/logs.proto`), next to + `time_unix_nano`, `severity_number`, `body`, `attributes`, `trace_id`, `span_id`, etc. There is no `EventRecord` proto message. -- All Events sharing the same `EventName` **MUST** conform to the same schema for both their - `Attributes` and their `Body` — i.e. `EventName` acts like a schema/type discriminator. -- Per the events semantic-conventions page: Events **MUST** have `Timestamp` set to when the event - actually occurred, and semantic conventions **MUST NOT** define a value for `ObservedTimestamp` - (that field stays purely about when the observability pipeline itself observed/received the - record — SDK/collector populated, never spec'd by convention authors). -- Event semantic conventions **MUST NOT** define a `Body` value except to hold a plain string - display message — structured/queryable data belongs in `Attributes`, preferably flat rather than - deeply nested. -- Event semantic conventions **MUST** document the event name so that querying by `event.name` - reliably returns records conforming to that convention's schema. +- All Events sharing an `EventName` **MUST** conform to the same schema for both `Attributes` and + `Body`. `EventName` acts as a schema/type discriminator. +- Per the events semantic-conventions page, Events **MUST** set `Timestamp` to when the event + occurred, and semantic conventions **MUST NOT** define a value for `ObservedTimestamp`. That field + stays about when the pipeline observed the record; the SDK or collector fills it. +- Event semantic conventions **MUST NOT** define a `Body` value except a plain string display message. + Structured, queryable data belongs in `Attributes`, preferably flat. +- Event semantic conventions **MUST** document the event name, so querying by `event.name` returns + records that conform to that convention's schema. Source: https://opentelemetry.io/docs/specs/semconv/general/events/ and https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-eventname ### 5.2 Deprecation of Span Events in favor of log-based events -A significant, currently-in-progress spec direction (tracked as an accepted OTEP, referenced as -**OTEP 4430** in the announcement): OpenTelemetry is **deprecating the Span Events API** -(`Span.AddEvent()` and `Span.RecordException()`) in favor of emitting **log-based events** through -the Logs API/SDK, correlated back to the active trace/span via `Context` (the same mechanism -described in [§3.3](#33-logger--emit-a-logrecord)) rather than being attached directly to the span -object. +An in-progress spec direction, tracked as an accepted OTEP (referenced as **OTEP 4430** in the +announcement): OpenTelemetry is **deprecating the Span Events API** (`Span.AddEvent()` and +`Span.RecordException()`). Instrumentation should emit **log-based events** through the Logs API/SDK +instead, correlated to the active trace and span via `Context` (the mechanism in +[§3.3](#33-logger-emit-a-logrecord)) instead of being attached to the span object. -Rationale given: maintaining two overlapping event mechanisms (span events vs. log-based events) -produced inconsistent guidance for library authors, duplicated concepts for operators to learn, and -slowed spec evolution since every improvement had to be designed/implemented twice. +Stated rationale: two overlapping event mechanisms gave library authors inconsistent guidance, gave +operators duplicate concepts to learn, and slowed spec work because every improvement had to be +designed and implemented twice. -Migration posture (no exact version/date commitments found in the source): +Migration posture (the source gives no exact version or date commitments): - New instrumentation should prefer log-based events **now**. -- Existing instrumentation's current major version stays behaviorally compatible (no breaking - removal yet). -- Next major versions of instrumentation libraries and semantic conventions are expected to - migrate from span events to log-based events. -- Language SDKs are expected to offer compatibility/transform shims during the transition. - -**Practical implication for Maple:** if our self-instrumentation (or any vendored instrumentation -we depend on) currently uses `span.addEvent(...)` for things like exception recording, expect -upstream libraries to eventually shift that data into log-based Events (`LogRecord` with -`EventName` set) instead of `SpanEvent`s on the span. Our ingest/consumption path should already -handle both shapes since both arrive as normal OTLP signals (span events inside -`Span.events`, log-based events inside the Logs OTLP stream) — but any dashboard logic that reads -"events" **only** from span data (e.g. `Span.events`) will miss log-based events, and vice versa, -until this migration is complete across the ecosystem. +- Existing instrumentation's current major version stays compatible (no breaking removal yet). +- Next major versions of instrumentation libraries and semantic conventions are expected to move from + span events to log-based events. +- Language SDKs are expected to offer compatibility shims during the transition. + +**Practical implication for Maple:** if our self-instrumentation (or instrumentation we depend on) +uses `span.addEvent(...)`, e.g. for exception recording, expect upstream libraries to move that data +into log-based Events (`LogRecord` with `EventName` set). Our ingest path handles both shapes, since +both arrive as normal OTLP: span events inside `Span.events`, log-based events in the Logs stream. +Dashboard logic that reads "events" **only** from `Span.events` will miss log-based events, and vice +versa, until the ecosystem finishes migrating. Source: https://opentelemetry.io/blog/2026/deprecating-span-events/ @@ -445,20 +426,20 @@ Source: https://opentelemetry.io/blog/2026/deprecating-span-events/ | OTLP Logs wire format (`logs.proto`) | Stable (part of OTLP) | https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto | | Events (semantic-convention layer, `EventName`) | **Development** | https://opentelemetry.io/docs/specs/semconv/general/events/ | | Span Events API (`Span.AddEvent`) | **Deprecated** (transition to log-based events in progress) | https://opentelemetry.io/blog/2026/deprecating-span-events/ | -| Standalone Event API/SDK (`logs/event-api.md`) | **Removed** — folded into Logs Data Model / API | (confirmed absent on `main`; last present at tag [v1.35.0](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.35.0/specification/logs/event-api.md)) | +| Standalone Event API/SDK (`logs/event-api.md`) | **Removed**; folded into Logs Data Model / API | (confirmed absent on `main`; last present at tag [v1.35.0](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.35.0/specification/logs/event-api.md)) | --- ## Key references -- Logs specification overview — https://opentelemetry.io/docs/specs/otel/logs/ -- Logs Data Model — https://opentelemetry.io/docs/specs/otel/logs/data-model/ -- Data Model Appendix (format mappings) — https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/ -- Logs API (Bridge API) — https://opentelemetry.io/docs/specs/otel/logs/api/ -- Logs SDK — https://opentelemetry.io/docs/specs/otel/logs/sdk/ -- SDK environment variables (Batch LogRecord Processor, LogRecord Limits, `OTEL_LOGS_EXPORTER`) — https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md -- Events semantic conventions (general) — https://opentelemetry.io/docs/specs/semconv/general/events/ -- Deprecating Span Events API (blog, 2026) — https://opentelemetry.io/blog/2026/deprecating-span-events/ -- OTLP `logs.proto` (wire format, `LogRecord`/`SeverityNumber`/`ResourceLogs`/`ScopeLogs`) — https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto -- Specification status summary — https://opentelemetry.io/docs/specs/status/ -- Historical Event API (pre-removal, v1.35.0 snapshot, for context only — not current) — https://github.com/open-telemetry/opentelemetry-specification/blob/v1.35.0/specification/logs/event-api.md +- Logs specification overview: https://opentelemetry.io/docs/specs/otel/logs/ +- Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/ +- Data Model Appendix (format mappings): https://opentelemetry.io/docs/specs/otel/logs/data-model-appendix/ +- Logs API (Bridge API): https://opentelemetry.io/docs/specs/otel/logs/api/ +- Logs SDK: https://opentelemetry.io/docs/specs/otel/logs/sdk/ +- SDK environment variables (Batch LogRecord Processor, LogRecord Limits, `OTEL_LOGS_EXPORTER`): https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md +- Events semantic conventions (general): https://opentelemetry.io/docs/specs/semconv/general/events/ +- Deprecating Span Events API (blog, 2026): https://opentelemetry.io/blog/2026/deprecating-span-events/ +- OTLP `logs.proto` (wire format, `LogRecord`/`SeverityNumber`/`ResourceLogs`/`ScopeLogs`): https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto +- Specification status summary: https://opentelemetry.io/docs/specs/status/ +- Historical Event API (v1.35.0 snapshot, context only, not current): https://github.com/open-telemetry/opentelemetry-specification/blob/v1.35.0/specification/logs/event-api.md diff --git a/docs/otel-spec/metrics.md b/docs/otel-spec/metrics.md index 0cb6fcd160..a473264c67 100644 --- a/docs/otel-spec/metrics.md +++ b/docs/otel-spec/metrics.md @@ -1,10 +1,10 @@ # Metrics (API, SDK & Data Model) -This page is a compliance-oriented reference for the OpenTelemetry **Metrics** signal: the client-facing -**API** (instrument creation and recording), the **SDK** (Views, Aggregations, MetricReader/Exporter -config), and the **Data Model** that defines what actually goes over OTLP (point kinds, temporality, -exemplars, resets). It exists to support spec-compliance checks and a best-practices skill — every claim -below is sourced from the fetched spec pages, not from memory. +A compliance reference for the OpenTelemetry **Metrics** signal. It covers the client-facing **API** +(instrument creation and recording), the **SDK** (Views, Aggregations, MetricReader/Exporter config), +and the **Data Model** that defines what goes over OTLP (point kinds, temporality, exemplars, resets). +It backs spec-compliance checks and a best-practices skill. Every claim is sourced from the fetched +spec pages, not from memory. > Source pages fetched for this document: > @@ -19,32 +19,35 @@ below is sourced from the fetched spec pages, not from memory. ## Relevance to Maple -Maple is primarily a **consumer/backend** for OTel metrics (Rust ingest gateway → collector → -ClickHouse/Tinybird), plus we self-instrument our own services (see the ingest gateway's own OTLP -metrics export described in the root `CLAUDE.md`). The parts of this spec that matter most to us: +Maple is mainly a **consumer/backend** for OTel metrics. The Rust ingest gateway (`apps/ingest`) +writes OTLP metric points to Tinybird/ClickHouse, or forwards them to a collector, depending on +`INGEST_WRITE_MODE`. The gateway also self-instruments: its own OTLP metrics are defined in +`apps/ingest/src/metrics.rs`. The parts of this spec that matter most to us: -- **Data model correctness** when ingesting/storing OTLP metric points: point kinds, temporality +- **Data model correctness** when ingesting and storing OTLP metric points: point kinds, temporality (delta vs cumulative), `start_time_unix_nano` semantics for reset/gap detection, and the - `NoRecordedValue` staleness flag — these directly affect how we aggregate and query stored metrics. -- **Single-writer principle & overlap handling** — as a backend, we're a "receiver" under the spec's - terminology; the normative guidance on deduplicating overlapping streams and detecting resets applies - to any rollup/materialized-view logic we build over ingested metrics. -- **Exponential histograms** — we may need to render/aggregate these (scale/bucket mapping), so the - mapping-function details are captured at a working level. -- **OTLP exporter env vars** (temporality preference, default histogram aggregation) — relevant when - documenting expected behavior of instrumented clients sending to our ingest gateway, and for our own + `NoRecordedValue` staleness flag. These decide how we aggregate and query stored metrics. +- **Single-writer principle and overlap handling.** As a backend we are a "receiver" in the spec's + terms. The normative guidance on deduplicating overlapping streams and detecting resets applies to + any rollup or materialized-view logic we build over ingested metrics. +- **Exponential histograms.** We may need to render or aggregate these (scale/bucket mapping), so the + mapping function is captured at a working level. +- **Summary points** are dropped at encode time today (`metrics_summary_dropped` in + `apps/ingest/src/telemetry.rs`); there is no Summary datasource. +- **OTLP exporter env vars** (temporality preference, default histogram aggregation). Relevant when + documenting how instrumented clients should send to our ingest gateway, and for our own self-instrumentation config. -- Metrics are **not yet a primary product surface** in Maple's dashboard (traces/logs are further along - per `docs/otel-coverage-roadmap.md`), so SDK-side view/aggregation configuration is lower priority than - data-model fidelity, but is included here for completeness since we may need to validate producer - behavior during spec-compliance checks. +- Metrics have a product surface (`/metrics`), but exemplars and histogram distributions are stored and + never shown (see `docs/otel-coverage-roadmap.md`). SDK-side View/aggregation config is lower + priority than data-model fidelity. It is included because spec-compliance checks may need to + validate producer behavior. --- ## 1. Metrics API -**Stability: Stable**, except where individually marked _(Development)_ below (the `Bind` operation and -the `Attributes` advisory parameter). +**Stability: Stable**, except where individually marked _(Development)_ below (for example the +`Attributes` advisory parameter). Source: https://opentelemetry.io/docs/specs/otel/metrics/api/ ### 1.1 Core architecture @@ -52,13 +55,13 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/api/ | Concept | Role | Source anchor | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `MeterProvider` | Entry point; owns config (exporters, readers, views). "The API SHOULD provide a way to set/register and access a global default `MeterProvider`." | `#meterprovider` | -| `Meter` | Created from a `MeterProvider`; "Responsible for creating `Instruments`." Meter MUST NOT hold configuration — that's the MeterProvider's job. | `#meter` | +| `Meter` | Created from a `MeterProvider`; "Responsible for creating `Instruments`." Meter MUST NOT hold configuration; that is the MeterProvider's job. | `#meter` | | `Instrument` | Identified by `name`, `kind`, `unit`, `description` (plus language-level value-type distinction, e.g. integer vs floating point) | `#instrument` | `Meter` creation parameters (`Get a Meter`): -- `name` (required) — instrumentation scope name, e.g. `io.opentelemetry.contrib.mongodb` -- `version` (optional) — e.g. `1.0.0` +- `name` (required): instrumentation scope name, e.g. `io.opentelemetry.contrib.mongodb` +- `version` (optional): e.g. `1.0.0` - `schema_url` (optional, since spec v1.4.0) - instrumentation scope `attributes` (optional, since spec v1.13.0) @@ -66,8 +69,8 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#meter ### 1.2 Instruments -All instruments are namespaced under a `Meter`; "MUST NOT be any API for creating [an instrument] other -than with Meter" (normative MUST) — i.e., no free-standing instrument construction outside the Meter. +All instruments are namespaced under a `Meter`. There "MUST NOT be any API for creating [an +instrument] other than with Meter" (normative MUST): no free-standing instrument construction. #### Synchronous instruments @@ -88,8 +91,8 @@ being observed." | Instrument | Semantics | Monotonic? | Value reported | | ------------------------------ | ---------------------------------- | ---------- | ------------------------------------------------------------------------------- | -| **Asynchronous Counter** | Monotonically increasing | Yes | Absolute (not delta) — SDK derives rate from successive differences | -| **Asynchronous UpDownCounter** | Additive, can increase or decrease | No | Absolute — SDK calculates deltas | +| **Asynchronous Counter** | Monotonically increasing | Yes | Absolute (not delta); SDK derives rate from successive differences | +| **Asynchronous UpDownCounter** | Additive, can increase or decrease | No | Absolute; SDK calculates deltas | | **Asynchronous Gauge** | Non-additive | N/A | Absolute, from an accessor/poll (contrast with sync Gauge's subscription model) | Callback normative requirements: @@ -106,30 +109,30 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#asynchronous-instr ### 1.3 Instruments × default aggregation (cross-reference to SDK §2.2) -See the table in [§2.2](#22-aggregations--default-per-instrument) — the API doesn't define aggregation, -but it's the natural pairing readers need. +See the table in [§2.2](#22-aggregations--default-per-instrument). The API does not define +aggregation, but readers need the pairing. ### 1.4 Naming rules -**Instrument name syntax** — Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#instrument-name-syntax +**Instrument name syntax.** Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#instrument-name-syntax ``` instrument-name = ALPHA 0*254 ("_" / "." / "-" / "/" / ALPHA / DIGIT) ``` - Not null/empty; case-insensitive ASCII -- First character: alphabetic (A–Z, a–z) +- First character: alphabetic (A-Z, a-z) - Subsequent characters: alphanumeric, `_`, `.`, `-`, `/` - **Max length: 255 characters** - "The API SHOULD NOT validate the `name`; that is left to implementations." -**Unit** — Source: `#instrument-unit` +**Unit.** Source: `#instrument-unit` - Case-sensitive ASCII string, opaque - **Max length: 63 characters** (chosen to allow fixed-size array storage) - "The API SHOULD NOT validate the `unit`" -**Description** — Source: `#instrument-description` +**Description.** Source: `#instrument-description` - Must support BMP (Unicode Plane 0); minimum guaranteed support **1023 characters** - Opaque string @@ -139,10 +142,10 @@ instrument-name = ALPHA 0*254 ("_" / "." / "-" / "/" / ALPHA / DIGIT) - **Identical instruments**: all identifying parameters equal (name, kind, unit, description, and the language-level value-type distinction). - **Distinct instruments**: differ in at least one identifying parameter. -- The Metrics **API spec itself does not define behavior for duplicate/conflicting registration** — that - is left to the SDK. (See Data Model §3.6 below for the _data-model_-level normative handling of - conflicting `Metric` identities once they reach OTLP.) -- **Concurrency**: `MeterProvider`, `Meter`, and `Instrument` — "All methods MUST be documented that +- The Metrics **API spec does not define behavior for duplicate or conflicting registration**. That is + left to the SDK. (Data Model §3.6 below covers the normative handling of conflicting `Metric` + identities once they reach OTLP.) +- **Concurrency**: for `MeterProvider`, `Meter`, and `Instrument`, "All methods MUST be documented that implementations need to be safe for concurrent use by default." Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#concurrency-requirements @@ -160,13 +163,13 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/api/#measurement ## 2. Metrics SDK -**Stability: Mixed** — most areas below are Stable; a few are explicitly Development (noted inline). +**Stability: Mixed.** Most areas below are Stable; a few are explicitly Development (noted inline). Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/ ### 2.1 Views -Views let the SDK owner customize how instruments are turned into metric streams without touching -instrumentation code. +Views let the SDK owner change how instruments become metric streams without touching instrumentation +code. **Instrument selection criteria** (all optional; user picks any subset): @@ -191,7 +194,7 @@ instrumentation code. Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#instrument-selection-criteria , `#stream-configuration` -### 2.2 Aggregations — default per instrument +### 2.2 Aggregations: default per instrument | Instrument | Default aggregation | | ----------------------------------------- | ------------------------------------------------------------------------------------- | @@ -208,25 +211,25 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#instrument-selecti `RecordMinMax` defaults to `true`. -**Base2 Exponential Histogram** _(Recommended default config when this aggregation is selected)_: +**Base2 Exponential Histogram** _(recommended default config when this aggregation is selected)_: `MaxSize = 160` buckets, `MaxScale = 20`, `RecordMinMax = true`. **Drop aggregation**: discards all measurements for the instrument (emits nothing). Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#aggregation , -raw spec `specification/metrics/sdk.md` (boundary list + MaxSize/MaxScale defaults confirmed verbatim). +raw spec `specification/metrics/sdk.md` (boundary list and MaxSize/MaxScale defaults confirmed verbatim). ### 2.3 MetricReader / MetricExporter **Periodic Exporting MetricReader** _(Stable)_: -- Collects at a configurable interval — default **60,000 ms** -- Export timeout — default **30,000 ms** +- Collects at a configurable interval, default **60,000 ms** +- Export timeout, default **30,000 ms** - _(Development)_ `maxExportBatchSize` splits large batches while preserving order across collections **Collect operation**: triggers async-instrument callbacks, invokes `Produce` on registered -`MetricProducer`s, and ensures the configured aggregation temporality is applied (performing -delta↔cumulative conversion for synchronous instruments as needed). +`MetricProducer`s, and applies the configured aggregation temporality (converting delta↔cumulative for +synchronous instruments as needed). Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader-operations , `#periodic-exporting-metricreader` @@ -241,8 +244,9 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader-opera | `AlwaysOn` | All measurements eligible | | `AlwaysOff` | No measurements eligible | -**ExemplarReservoir** _(Stable)_ — one per timeseries; `Offer(value, attributes, context, timestamp)`; -`Collect()` returns accumulated exemplars respecting the stream's aggregation temporality. +**ExemplarReservoir** _(Stable)_: one per timeseries. `Offer(value, attributes, context, timestamp)` +adds a candidate; `Collect()` returns accumulated exemplars respecting the stream's aggregation +temporality. Default reservoir selection: @@ -258,30 +262,30 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#exemplar **Status: Stable.** -- Enforced **after** attribute filtering (i.e., after Views' `attribute_keys` allow/exclude list is +- Enforced **after** attribute filtering (after the View's `attribute_keys` allow/exclude list is applied). -- Resolution order: (1) View's `aggregation_cardinality_limit` → (2) MetricReader default for that - instrument kind → (3) **global default of 2000**. -- **Overflow attribute**: measurements beyond the limit are aggregated into a single synthetic series - tagged `otel.metric.overflow = true` (boolean). +- Resolution order: (1) View's `aggregation_cardinality_limit`, then (2) MetricReader default for that + instrument kind, then (3) **global default of 2000**. +- **Overflow attribute**: measurements beyond the limit are aggregated into one synthetic series tagged + `otel.metric.overflow = true` (boolean). - **Cumulative streams**: "continue to export all attribute sets that were observed prior to the - beginning of overflow" (pre-overflow series are retained individually). + beginning of overflow". Pre-overflow series are kept individually. - **Delta streams**: "MAY choose an arbitrary subset of attribute sets to output to maintain the stated - cardinality limit" — no guaranteed stability of which series survive. + cardinality limit". Which series survive is not stable. - **Asynchronous instruments**: prefer first-observed attribute sets when trimming. -There is **no dedicated global env var** for the default cardinality limit documented on the general SDK -environment-variables page fetched for this doc (only the View-level `aggregation_cardinality_limit` -stream-config parameter and the built-in 2000 default are spec-defined). Verify per-SDK docs (language -implementations may expose their own env var) before treating any specific name as normative. +The general SDK environment-variables page defines **no global env var** for the default cardinality +limit. Only the View-level `aggregation_cardinality_limit` stream parameter and the built-in 2000 +default are spec-defined. Language SDKs may expose their own env var; check per-SDK docs before +treating any name as normative. -Source: raw spec `specification/metrics/sdk.md` (Cardinality limits section, doc status "Mixed" at file -top — the specific cardinality-limits subsection content quoted above is Stable per the rendered site). +Source: raw spec `specification/metrics/sdk.md` (Cardinality limits section; the file header says +"Mixed", and the rendered site marks this subsection Stable). ### 2.6 MeterProvider / MeterConfigurator (Development) -_(Development)_ `MeterConfigurator`: a function computing `MeterConfig` from an `InstrumentationScope`, -returning either a configuration or a signal to use defaults; SDKs may provide helpers for common +_(Development)_ `MeterConfigurator`: a function computing `MeterConfig` from an `InstrumentationScope`. +It returns either a configuration or a signal to use defaults. SDKs may provide helpers for common patterns (e.g., select meters by name, disable specific meters). _(Development)_ `MeterConfig`: `enabled: boolean` (default `true`); disabled meters behave as no-ops. @@ -292,21 +296,21 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#meterprovider ## 3. Metrics Data Model **Stability: Stable** for all point kinds, temporality, exemplars, data point flags, and the -single-writer principle. **Development** for the "Resets and Gaps" mechanics and "Overlap"/out-of-order -handling sections specifically (noted inline below). +single-writer principle. **Development** for the "Resets and Gaps" mechanics and the +"Overlap"/out-of-order handling sections (noted inline below). Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/ ### 3.1 Point kinds -| Point kind | Temporality? | Key fields | Status | -| ------------------------ | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | -| **Sum** | delta or cumulative | attributes, `(start, end]` window, `monotonic: bool`, exemplars, flags | Stable | -| **Gauge** | n/a (no aggregation temporality) | attributes, sampled value, `time_unix_nano`, optional `start_time_unix_nano`, exemplars, flags | Stable | -| **Histogram** | delta or cumulative | attributes, window, `count`, `sum`, optional `min`/`max`, explicit bucket boundaries + per-bucket counts, exemplars, flags | Stable | -| **ExponentialHistogram** | delta or cumulative | same as Histogram but exponential bucket structure (`scale`, `zero_count`, `zero_threshold`, positive/negative bucket index ranges) | Stable | -| **Summary (legacy)** | n/a | attributes, `time_unix_nano`, `count`, `sum`, strictly-increasing quantile set `[0.0, 1.0]` | Stable, but "not recommended for new applications" — points "cannot always be merged in a meaningful way" | +| Point kind | Temporality? | Key fields | Status | +| ------------------------ | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| **Sum** | delta or cumulative | attributes, `(start, end]` window, `monotonic: bool`, exemplars, flags | Stable | +| **Gauge** | n/a (no aggregation temporality) | attributes, sampled value, `time_unix_nano`, optional `start_time_unix_nano`, exemplars, flags | Stable | +| **Histogram** | delta or cumulative | attributes, window, `count`, `sum`, optional `min`/`max`, explicit bucket boundaries + per-bucket counts, exemplars, flags | Stable | +| **ExponentialHistogram** | delta or cumulative | same as Histogram but exponential bucket structure (`scale`, `zero_count`, `zero_threshold`, positive/negative bucket index ranges) | Stable | +| **Summary (legacy)** | n/a | attributes, `time_unix_nano`, `count`, `sum`, strictly-increasing quantile set `[0.0, 1.0]` | Stable, but "not recommended for new applications"; points "cannot always be merged in a meaningful way" | -**Sum monotonicity note**: "Delta monotonic: reader SHOULD expect non-negative values" / "Cumulative +**Sum monotonicity**: "Delta monotonic: reader SHOULD expect non-negative values" / "Cumulative monotonic: reader SHOULD expect values that are not less than the previous value." **Histogram bucket inclusivity** (normative): "Bucket upper-bounds are inclusive (except for the case @@ -314,25 +318,24 @@ where the upper-bound is +Inf) while bucket lower-bounds are exclusive. That is, number of values that are greater than their lower bound and less than or equal to their upper bound." **Gauge semantics**: "a point within a Gauge stream represents the last-sampled event for a given time -window" — no aggregation semantic; "last sample value" wins when temporally aligning or resampling. +window". There is no aggregation semantic; the last sample wins when temporally aligning or resampling. Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#sums , `#gauge`, `#histogram`, `#exponentialhistogram`, `#summary-legacy` -### 3.2 Temporality — delta vs cumulative +### 3.2 Temporality: delta vs cumulative | | Delta | Cumulative | | ------------------------ | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Start timestamp behavior | "successive data points **advance** the starting timestamp" — `(T0,T1], (T1,T2], (T2,T3]` | "successive data points **repeat** the starting timestamp" — `(T0,T1], (T0,T2], (T0,T3]` | +| Start timestamp behavior | "successive data points **advance** the starting timestamp": `(T0,T1], (T1,T2], (T2,T3]` | "successive data points **repeat** the starting timestamp": `(T0,T1], (T0,T2], (T0,T3]` | | Typical origin | Statsd-style metrics; "enables sampling and supports shifting the cost of cardinality outside of the process" | Prometheus-style; "naturally simpler ... in terms of cost of adding reliability. When collection fails intermittently, gaps ... are naturally averaged" | -| Sender-side memory cost | Lower (no need to remember all-time totals) | Higher — sender must retain "all previous measurements, an 'up-front' memory cost proportional to cardinality" | -| Validity rule | intervals must be contiguous — no gaps/overlaps in a well-formed stream | repeats the same start; validity tracked via reset detection (§3.3) | +| Sender-side memory cost | Lower (no need to remember all-time totals) | Higher: sender must retain "all previous measurements, an 'up-front' memory cost proportional to cardinality" | +| Validity rule | Intervals must be contiguous: no gaps or overlaps in a well-formed stream | Repeats the same start; validity tracked via reset detection (§3.3) | Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality -**OTLP exporter default & env var** (producer-side preference, not the wire temporality itself — see -§4 for the table): -`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` default = `cumulative`. +**OTLP exporter default and env var** (a producer-side preference, not the wire temporality itself; +see §5.2 for the table): `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE`, default `cumulative`. ### 3.3 `start_time_unix_nano`, resets, and gaps @@ -347,8 +350,8 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#resets-and- | -------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **True reset (known start)** | `StartTimeUnixNano < TimeUnixNano` | "a new unbroken sequence of observations begins with a 'true' reset at a known start time. The zero value is implicit, it is not necessary to record the starting point." | | **Reset, unknown start** | `StartTimeUnixNano == TimeUnixNano` | "a new unbroken sequence of observations begins with a reset at an unknown start time. The initial observed value is recorded ... These points have zero duration." | -| **Subsequent point, delta** | — | `StartTimeUnixNano` of each point matches the `TimeUnixNano` of the _preceding_ point | -| **Subsequent point, cumulative/other** | — | `StartTimeUnixNano` of each point matches the `StartTimeUnixNano` of the _initial_ observation in the sequence | +| **Subsequent point, delta** | n/a | `StartTimeUnixNano` of each point matches the `TimeUnixNano` of the _preceding_ point | +| **Subsequent point, cumulative/other** | n/a | `StartTimeUnixNano` of each point matches the `StartTimeUnixNano` of the _initial_ observation in the sequence | **Gap definition**: "A metric stream has a gap, where it is implicitly undefined, anywhere there is a range of time such that no point covers that range with its `StartTimeUnixNano` and `TimeUnixNano` @@ -360,13 +363,13 @@ fields." Fields: -| Field | Description | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `trace_id` (optional) | Trace associated with the recording | -| `span_id` (optional) | Span associated with the recording | -| `time_unix_nano` | Time of the observation | -| `value` | The recorded value | -| `filtered_attributes` | Attributes present on the measurement but filtered out of the point's own attribute set — "provide additional insight into the Context when the observation was made" | +| Field | Description | +| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `trace_id` (optional) | Trace associated with the recording | +| `span_id` (optional) | Span associated with the recording | +| `time_unix_nano` | Time of the observation | +| `value` | The recorded value | +| `filtered_attributes` | Attributes present on the measurement but filtered out of the point's own attribute set; they "provide additional insight into the Context when the observation was made" | Value participation in the parent point: @@ -374,7 +377,7 @@ Value participation in the parent point: - **Sum**: exemplar's value "is already included in the overall sum" - **Gauge**: exemplar's value "was seen at some point within the gauge interval for the same source" -### 3.5 Data point flags — staleness marker +### 3.5 Data point flags: staleness marker **Status: Stable.** Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#data-point-flags @@ -388,7 +391,7 @@ Equivalent to the [Prometheus staleness marker](https://prometheus.io/docs/prome When set: "all other data point properties except attributes, time stamps, or time windows, SHOULD be ignored." -### 3.6 Single-writer principle & conflicting metric identities +### 3.6 Single-writer principle and conflicting metric identities **Status: Stable.** Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#single-writer @@ -401,11 +404,11 @@ ignored." time. **Note: This implies aggregated metric streams must reach one destination**." - "Multiple writers for a metric stream is considered an error state, or misbehaving system. Receivers SHOULD presume a single writer was intended and eliminate overlap / deduplicate." -- Distinguished from semantic errors: single-writer violations are usually **misconfiguration** - (fixable by differentiating `Resource` or ensuring non-overlapping time ranges), whereas semantic - errors are sometimes fixable via Views. +- Single-writer violations are usually **misconfiguration**, fixable by differentiating `Resource` or + ensuring non-overlapping time ranges. Semantic errors are a separate class, sometimes fixable via + Views. -**Conflicting `Metric` identity** for the same `name` + `Resource` + `Scope` — producer recommendations +**Conflicting `Metric` identity** for the same `name` + `Resource` + `Scope`. Producer recommendations _(Stable)_, source: `#opentelemetry-protocol-data-model-producer-recommendations`: | Conflict | Producer recommendation | @@ -416,10 +419,10 @@ _(Stable)_, source: `#opentelemetry-protocol-data-model-producer-recommendations | Other identifying-property conflict | SHOULD inform the user of a semantic error and pass through the conflicting data | "Consumers MAY reject OpenTelemetry Metrics data containing semantic errors (i.e., more than one -`Metric` identity for a given `name`, `Resource`, and `Scope`)." — **relevant to Maple as an ingest -consumer**: we are within spec to reject/flag such payloads rather than silently merge them. +`Metric` identity for a given `name`, `Resource`, and `Scope`)." **Relevant to Maple as an ingest +consumer**: we are within spec to reject or flag such payloads instead of silently merging them. -### 3.7 Overlap & out-of-order handling +### 3.7 Overlap and out-of-order handling _(Status: Development for this subsection.)_ Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#overlap @@ -434,25 +437,25 @@ Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#overlap overlapping points may be expected ... collectors SHOULD modify points at the change-over using interpolation for Sum data points, to reduce gaps to zero width in these cases, without any overlap." -**Delta-to-cumulative conversion algorithm** (out-of-order / restart detection), source: +**Delta-to-cumulative conversion algorithm** (out-of-order and restart detection), source: `#sums-detecting-alignment-issues`: - "if the current point precedes the start time, then drop this point. Note: there are algorithms which can deal with late arriving points." - "if the next point does NOT align with the expected next-time window, then reset the counter following the same steps performed as if the current point was the first point seen." -- Two triggers: (1) significant **overlap** with the previous interval → assume single-writer-principle - violation, recommend elimination/dedup or resource differentiation; (2) significant **gap** from the - last-seen time → assume a reboot/restart and reset the cumulative counter. -- Degenerate case: if timestamps are missing entirely from data points (can happen when adapting - non-OTel metric formats into OTLP), "the algorithm resets on every point." +- Two triggers: (1) significant **overlap** with the previous interval: assume a single-writer + violation and recommend dedup or resource differentiation; (2) significant **gap** from the last-seen + time: assume a restart and reset the cumulative counter. +- Degenerate case: if data points carry no timestamps (possible when adapting non-OTel metric formats + into OTLP), "the algorithm resets on every point." --- -## 4. Exponential histograms — mapping rules (working-level detail) +## 4. Exponential histograms: mapping rules (working-level detail) -**Status: Stable** (data model fields); the mapping-function reference math below is drawn from the -canonical spec markdown (`specification/metrics/data-model.md`, ExponentialHistogram section). +**Status: Stable** (data model fields). The mapping-function math below comes from the canonical spec +markdown (`specification/metrics/data-model.md`, ExponentialHistogram section). - **Base formula**: `base = 2**(2**(-scale))`. At `scale = 0`, `base = 2`. - **Bucket definition**: "bucket identified by `index`, a signed integer, represents values in the @@ -460,23 +463,23 @@ canonical spec markdown (`specification/metrics/data-model.md`, ExponentialHisto - **Index constraint**: producers must ensure "the bucket index of any encoded bucket falls within the range of a signed 32-bit integer." - **Scale = 0 mapping ("extract exponent")**: "the index of a value equals its normalized base-2 - exponent" — derived directly from the IEEE-754 bit layout (no logarithm needed). + exponent", derived from the IEEE-754 bit layout (no logarithm needed). - **Scale ≤ 0 mapping ("extract and shift")**: equals the scale-0 mapping "shifted to the right by `-scale`". -- **General (any scale) mapping via logarithm**: `index == Ceiling(log(value)/log(base)) - 1`, computed - in practice via a scaling factor `2**scale / log(2)` (e.g. `math.Ldexp(math.Log2E, scale)` in Go-style - pseudocode) for numerical stability instead of naively dividing by `log(base)` each time. +- **General (any scale) mapping via logarithm**: `index == Ceiling(log(value)/log(base)) - 1`. In + practice it is computed with a scaling factor `2**scale / log(2)` (e.g. `math.Ldexp(math.Log2E, scale)` + in Go-style pseudocode) for numerical stability, instead of dividing by `log(base)` each time. - **Negative values**: "mapped by their absolute value into the negative range using the same scale as - the positive range" — i.e., symmetric positive/negative bucket sets sharing one `scale`. -- **Zero handling**: a dedicated `zero_count` bucket, with optional `zero_threshold`; "`zero_count` + the positive range". Positive and negative bucket sets share one `scale`. +- **Zero handling**: a dedicated `zero_count` bucket, with optional `zero_threshold`. "`zero_count` contains the count of values whose absolute value is less than or equal to `zero_threshold`." When `zero_threshold` is unset, the zero bucket also "stores values that cannot be expressed using the standard exponential formula as well as values that have been rounded to zero." - **Downscaling (resolution reduction)**: "Buckets of an exponential Histogram with a given scale map exactly into buckets of exponential Histograms with lesser scales, which allows consumers to lower the - resolution of a histogram (i.e., downscale) without introducing error" — i.e., merging is lossless when - reducing scale, which is how implementations reconcile a too-large bucket count against `MaxSize` - (default 160, see §2.2) by repeatedly halving resolution until the active index range fits. + resolution of a histogram (i.e., downscale) without introducing error". Merging is lossless when + reducing scale. Implementations use this to fit `MaxSize` (default 160, see §2.2): they halve + resolution until the active index range fits. Source: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram (rendered) and `specification/metrics/data-model.md` (raw, for verbatim formula quotes). @@ -515,9 +518,8 @@ Notes verbatim: instrument kinds." - `lowmemory`: "uses Delta aggregation temporality for Synchronous Counter and Histogram and uses Cumulative aggregation temporality for Synchronous UpDownCounter, Asynchronous Counter, and - Asynchronous UpDownCounter instrument kinds" — the distinguishing difference from `delta` is that - Asynchronous Counter stays Cumulative under `lowmemory` (avoids the SDK having to retain prior - async-counter state to compute deltas). + Asynchronous UpDownCounter instrument kinds". The difference from `delta` is that Asynchronous + Counter stays Cumulative, so the SDK does not retain prior async-counter state to compute deltas. ### 5.3 Environment variables (Metrics) @@ -530,11 +532,9 @@ Notes verbatim: | `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `cumulative` | `cumulative`, `delta`, `lowmemory` (case-insensitive) | OTLP metrics exporter | Stable | | `OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION` | `explicit_bucket_histogram` | `explicit_bucket_histogram`, `base2_exponential_bucket_histogram` (case-insensitive) | OTLP metrics exporter | Stable | -No dedicated global env var for the cardinality-limit default (2000) was found on the fetched general -SDK env-vars page; the limit is a spec-defined constant applied via View/Reader config precedence (see -§2.5), not (per the pages fetched) an environment variable. Treat this as **unverified-absence** rather -than confirmed non-existence — some individual language SDKs may add their own env var; check -language-specific SDK docs if this matters for a specific client. +The fetched general SDK env-vars page has no env var for the cardinality-limit default (2000). The +limit is a spec-defined constant applied through View/Reader precedence (§2.5). Treat this as +unverified absence, not confirmed non-existence: some language SDKs add their own env var. Source: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ , https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/ @@ -544,7 +544,7 @@ https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/ | Point kind | Delta valid? | Cumulative valid? | Notes | | -------------------- | ------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Sum | Yes | Yes | `monotonic` flag independent of temporality | -| Gauge | n/a | n/a | No aggregation temporality field at all — last-sample semantics only | +| Gauge | n/a | n/a | No aggregation temporality field; last-sample semantics only | | Histogram | Yes | Yes | `min`/`max` "more useful for Delta temporality, since ... Cumulative min and max will stabilize as more events are recorded" | | ExponentialHistogram | Yes | Yes | Same temporality semantics as Histogram | | Summary (legacy) | n/a | n/a | No temporality field; compatibility-only point kind | diff --git a/docs/otel-spec/otlp.md b/docs/otel-spec/otlp.md index e31a3fde33..a0b0eaf497 100644 --- a/docs/otel-spec/otlp.md +++ b/docs/otel-spec/otlp.md @@ -2,13 +2,13 @@ OTLP (OpenTelemetry Protocol) is the wire format OpenTelemetry SDKs and collectors use to send traces, metrics, and logs. It defines the transports (gRPC, HTTP/protobuf, HTTP/JSON), the protobuf message schemas (`Resource` → `Scope*` → `Span`/`Metric`/`LogRecord`), the request/response contract (including partial-success semantics), and retry/backpressure rules a compliant server must implement. -> **Relevance to Maple:** Maple's Rust ingest gateway (`apps/ingest`) is an **OTLP server** — it receives OTLP/HTTP (binary protobuf and JSON) from customer SDKs/collectors and must implement the server-side MUSTs below: correct status codes, partial-success vs full-failure semantics, `Retry-After`/throttling signals, and JSON encoding rules (hex trace/span IDs, camelCase, numeric enums, stringified int64). Getting these wrong causes silent data loss or client retry storms. Maple is also an OTLP **client** for self-observability (`apps/ingest` and the API self-instrument via `@effect/opentelemetry`/OTLP exporters) — the exporter env-var section governs that config. Our dummy-data script (`scripts/ingest-dummy.ts`, `bun run ingest:dummy`) hand-constructs OTLP/JSON and depends on the JSON encoding rules (camelCase + hex IDs); see also the root `CLAUDE.md` self-observability section for the loop-prevention rules layered on top of this transport. +> **Relevance to Maple:** Maple's Rust ingest gateway (`apps/ingest`) is an **OTLP server**. It receives OTLP/HTTP (binary protobuf and JSON) from customer SDKs and collectors, plus OTLP/gRPC when `INGEST_OTLP_GRPC_PORT` is set. It must implement the server-side MUSTs below: correct status codes, partial-success vs full-failure semantics, `Retry-After`/throttling signals, and the JSON encoding rules (hex trace/span IDs, lowerCamelCase, numeric enums, stringified int64). Getting these wrong causes silent data loss or client retry storms. Maple is also an OTLP **client** for self-observability: the ingest gateway exports through the Rust OpenTelemetry SDK, and the API through `@maple-dev/effect-sdk` (`packages/effect-sdk`). The exporter env-var section governs that config. The dev script `scripts/ingest-dummy-traces.ts` hand-builds OTLP/JSON and depends on the JSON encoding rules. The API traces itself through ingest; see the "Self-tracing" rule in the root `CLAUDE.md` for the loop prevention layered on top of this transport. --- ## Stability -Per the OTLP spec (v1.10.0): OTLP is **Stable** for the **trace, metric, and log** signals. The **profiles** signal is **Development** status (entered public Alpha in March 2026 per the OpenTelemetry blog); its default HTTP path is versioned accordingly (`/v1development/profiles`, not `/v1/profiles`), and its proto messages/fields are explicitly excluded from stability guarantees until promoted. +Per the OTLP spec (v1.10.0): OTLP is **Stable** for the **trace, metric, and log** signals. The **profiles** signal is **Development** (public Alpha since March 2026, per the OpenTelemetry blog). Its default HTTP path is versioned to match (`/v1development/profiles`, not `/v1/profiles`), and its proto messages and fields carry no stability guarantees until promoted. Source: https://opentelemetry.io/docs/specs/otlp/ · https://opentelemetry.io/blog/2026/profiles-alpha/ @@ -91,7 +91,7 @@ Source: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentele ### Full success - **gRPC:** server responds with the appropriate `ExportServiceResponse` and **"MUST leave the `partial_success` field unset in case of a successful response."** -- **HTTP:** **"On success, the server MUST respond with `HTTP 200 OK`."** The response body **"MUST be a Protobuf-encoded `ExportServiceResponse` message"** (even for a JSON request — response encoding for HTTP is always the ordinary protobuf-JSON/binary mapping, matching the request's encoding). +- **HTTP:** **"On success, the server MUST respond with `HTTP 200 OK`."** The response body **"MUST be a Protobuf-encoded `ExportServiceResponse` message"**. The server MUST respond with the same `Content-Type` it received, so a JSON request gets a JSON-encoded response. ### Partial success (server MUST rules) @@ -103,7 +103,7 @@ Applicability: **"If the request is only partially accepted (i.e. when the serve Warnings-only case: **"Servers MAY also use the `partial_success` field to convey warnings/suggestions to clients even when the server fully accepts the request. In such cases, the `rejected_` field MUST have a value of `0`."** -Client behavior: **"The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated"** — partial success is a terminal outcome, not a retry signal, regardless of which items were rejected. +Client behavior: **"The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated"**. Partial success is terminal, whichever items were rejected. Degenerate case: a `partial_success` with `rejected_* == 0` and empty `error_message` is defined as equivalent to the field being unset (full success). @@ -111,14 +111,14 @@ Source: https://opentelemetry.io/docs/specs/otlp/ · https://github.com/open-tel ### When to use partial success vs full failure (server MUST) -- If **any** part of the request is decodable and acceptable, respond `200 OK` + partial success describing what was rejected and why — do not fail the whole request just because some items were bad. +- If **any** part of the request is decodable and acceptable, respond `200 OK` + partial success describing what was rejected and why. Do not fail the whole request because some items were bad. - If the request **cannot be decoded at all**, or is invalid such that nothing in it can be processed, and the failure is permanent: **"the server MUST respond with `HTTP 400 Bad Request`"** (full failure, no partial success). The client **"MUST NOT retry"** on a 400. --- ## Failure handling -### gRPC status codes — retryable matrix +### gRPC status codes: retryable matrix | gRPC code | Retryable? | | ------------------- | ------------------------------------------------------- | @@ -145,7 +145,7 @@ Source: https://opentelemetry.io/docs/specs/otlp/ · https://github.com/open-tel - Backoff: **"When retrying, the client SHOULD implement an exponential backoff strategy."** - gRPC backpressure signal: **"To signal backpressure when using gRPC transport, the server SHOULD return an error with code `Unavailable` and MAY supply additional details via status using `RetryInfo`."** -### HTTP status codes — retryable matrix +### HTTP status codes: retryable matrix | HTTP status | Retryable? | Notes | | ----------------------- | ---------- | ------------------------------------------------------------------------- | @@ -157,7 +157,7 @@ Source: https://opentelemetry.io/docs/specs/otlp/ · https://github.com/open-tel | 504 Gateway Timeout | **Yes** | | | other 4xx/5xx | **No** | **"All other `4xx` or `5xx` response status codes MUST NOT be retried."** | -- Error body: **"The response body for all `HTTP 4xx` and `HTTP 5xx` responses MUST be a Protobuf-encoded `Status` message that describes the problem."** — this applies regardless of whether the request was binary or JSON encoded. +- Error body: **"The response body for all `HTTP 4xx` and `HTTP 5xx` responses MUST be a Protobuf-encoded `Status` message that describes the problem."** It is encoded with the request's `Content-Type` (binary protobuf or JSON). - Bad data: **"If the processing of the request fails because the request contains data that cannot be decoded or is otherwise invalid and such failure is permanent, then the server MUST respond with `HTTP 400 Bad Request`."** ### Throttling / backpressure (server SHOULD) @@ -168,7 +168,7 @@ Source: https://opentelemetry.io/docs/specs/otlp/ · https://github.com/open-tel ### Duplicate data (known limitation, informational) -**"In edge cases (e.g. on reconnections, network interruptions, etc) the client has no way of knowing if recently sent data was delivered if no acknowledgement was received yet. The client will typically choose to re-send such data to guarantee delivery, which may result in duplicate data on the server side."** — downstream consumers (Tinybird MVs, dashboards) should tolerate duplicate spans/log records/data points; OTLP does not guarantee exactly-once delivery. +**"In edge cases (e.g. on reconnections, network interruptions, etc) the client has no way of knowing if recently sent data was delivered if no acknowledgement was received yet. The client will typically choose to re-send such data to guarantee delivery, which may result in duplicate data on the server side."** OTLP does not guarantee exactly-once delivery, so downstream consumers (Tinybird MVs, dashboards) should tolerate duplicate spans, log records, and data points. ### Empty envelopes @@ -178,15 +178,15 @@ Source: https://opentelemetry.io/docs/specs/otlp/ --- -## JSON encoding (OTLP/HTTP JSON) — deviations from standard protobuf JSON +## JSON encoding (OTLP/HTTP JSON): deviations from standard protobuf JSON -OTLP/HTTP JSON starts from the proto3 standard JSON mapping but overrides it in ways that matter for anyone hand-constructing or parsing OTLP JSON (e.g. Maple's ingest-dummy script): +OTLP/HTTP JSON starts from the proto3 standard JSON mapping but overrides it in ways that matter to anyone hand-building or parsing OTLP JSON (e.g. `scripts/ingest-dummy-traces.ts`): | Aspect | Standard protobuf JSON | **OTLP/JSON rule** | | ------------------------------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `trace_id`/`span_id` (`bytes`) | base64 | **Case-insensitive hex-encoded string.** Example: `{"traceId": "5B8EFFF798038103D269B633813FC60C"}` | -| Field names | configurable | **lowerCamelCase only** — original (snake_case) field names are **not valid**. Example: `droppedAttributesCount`, not `dropped_attributes_count` | -| Enums | name string OR number, receiver's choice | **Numbers only — enum name strings MUST NOT be used.** Example: `{"kind": 2}`, not `{"kind": "SPAN_KIND_SERVER"}` | +| Field names | configurable | **lowerCamelCase only**; original (snake_case) field names are **not valid**. Example: `droppedAttributesCount`, not `dropped_attributes_count` | +| Enums | name string OR number, receiver's choice | **Numbers only; enum name strings MUST NOT be used.** Example: `{"kind": 2}`, not `{"kind": "SPAN_KIND_SERVER"}` | | `int64`/`uint64` | string | Same as standard: **encoded as decimal strings**; **"either numbers or strings are accepted when decoding"** | | Unknown fields | mapping-defined | **"OTLP/JSON receivers MUST ignore message fields with unknown names and MUST unmarshal the message as if the unknown field was not present"** (forward compatibility) | @@ -258,7 +258,7 @@ message KeyValue { } ``` -`ArrayValue`/`KeyValueList` let `AnyValue` recurse arbitrarily (arrays of any value, maps of any value), which is how OTel represents nested/structured attributes over a flat protobuf wire format. +`ArrayValue`/`KeyValueList` let `AnyValue` recurse (arrays of any value, maps of any value). This is how OTel carries nested attributes over protobuf. ### `Span` (trace.proto) @@ -301,7 +301,7 @@ message Status { } ``` -> Note for Maple: `Status.code` on the wire is the numeric enum (`0`/`1`/`2` — `Unset`/`Ok`/`Error`); the project's own data convention (per root `CLAUDE.md`) renders these as title-case strings (`"Ok"`, `"Error"`, `"Unset"`) once ingested — that's a Maple display/storage convention, not an OTLP wire rule. +> Note for Maple: `Status.code` on the wire is the numeric enum (`0`/`1`/`2` = `Unset`/`Ok`/`Error`). Maple stores and displays these as Title case strings (`"Ok"`, `"Error"`, `"Unset"`, per root `CLAUDE.md`). That is a Maple storage convention, not an OTLP wire rule. ### `LogRecord` (logs.proto) @@ -321,11 +321,11 @@ message LogRecord { } ``` -`SeverityNumber` is a 1–24 numeric scale (`UNSPECIFIED=0`, then `TRACE`(1-4)/`DEBUG`(5-8)/`INFO`(9-12)/`WARN`(13-16)/`ERROR`(17-20)/`FATAL`(21-24) tiers, each tier having 4 numbered sub-levels e.g. `TRACE2`). +`SeverityNumber` is a 1-24 numeric scale: `UNSPECIFIED=0`, then `TRACE`(1-4), `DEBUG`(5-8), `INFO`(9-12), `WARN`(13-16), `ERROR`(17-20), `FATAL`(21-24). Each tier has four numbered sub-levels (e.g. `TRACE2`). ### Dropped-count fields (uniform pattern across all signals) -Every repeated collection that the SDK may have truncated carries a paired `dropped_*_count: uint32` sibling — `dropped_attributes_count` (Resource, InstrumentationScope, Span, LogRecord, metric data points), `dropped_events_count`/`dropped_links_count` (Span only). `0` always means "nothing was dropped," never "field not applicable." +Every repeated collection that the SDK may have truncated carries a paired `dropped_*_count: uint32` sibling: `dropped_attributes_count` (Resource, InstrumentationScope, Span, LogRecord, metric data points), `dropped_events_count`/`dropped_links_count` (Span only). `0` always means "nothing was dropped," never "field not applicable." ### `flags` fields @@ -337,7 +337,7 @@ Source: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentele ## Exporter configuration (`OTEL_EXPORTER_OTLP_*`) -All options are defined generically and per-signal (`TRACES`/`METRICS`/`LOGS`); **"Each configuration option MUST be overridable by a signal specific option"** — signal-specific always wins over the generic one. +All options are defined generically and per-signal (`TRACES`/`METRICS`/`LOGS`); **"Each configuration option MUST be overridable by a signal specific option"**. The signal-specific value always wins over the generic one. | Env var (generic → per-signal suffix pattern) | Default | Values / format | | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | @@ -351,10 +351,10 @@ All options are defined generically and per-signal (`TRACES`/`METRICS`/`LOGS`); | `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` → `_TRACES_CLIENT_CERTIFICATE` / etc. | none | PEM file path (mTLS client cert/chain) | | `OTEL_EXPORTER_OTLP_CLIENT_KEY` → `_TRACES_CLIENT_KEY` / etc. | none | PEM file path (mTLS client private key) | -### Endpoint path-appending rule — generic vs per-signal +### Endpoint path-appending rule: generic vs per-signal - **Generic `OTEL_EXPORTER_OTLP_ENDPOINT` (HTTP):** the base URL, with the signal path appended: `v1/traces`, `v1/metrics`, `v1/logs`. -- **Per-signal endpoint (e.g. `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`):** **"the URL MUST be used as-is without any modification. The only exception is that if a URL contains no path part, the root path `/` MUST be used."** — i.e. per-signal endpoints are NOT auto-suffixed with `/v1/traces`; you must include the full path yourself, or the request goes to `/`. +- **Per-signal endpoint (e.g. `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`):** **"the URL MUST be used as-is without any modification. The only exception is that if a URL contains no path part, the root path `/` MUST be used."** Per-signal endpoints are NOT suffixed with `/v1/traces`. Include the full path yourself, or the request goes to `/`. - **Precedence:** **"The per-signal endpoint configuration options take precedence and can be used to override this behavior (the URL is used as-is for them, without any modifications)."** ### Protocol defaults / SDK requirements @@ -371,7 +371,7 @@ Source: https://opentelemetry.io/docs/specs/otel/protocol/exporter/ ## Key references -- OTLP specification (main spec — transports, request/response, retries, throttling): https://opentelemetry.io/docs/specs/otlp/ +- OTLP specification (main spec: transports, request/response, retries, throttling): https://opentelemetry.io/docs/specs/otlp/ - OTLP/JSON encoding deviations: https://opentelemetry.io/docs/specs/otlp/#json-protobuf-encoding - Exporter configuration (env vars): https://opentelemetry.io/docs/specs/otel/protocol/exporter/ - opentelemetry-proto repo (all `.proto` sources): https://github.com/open-telemetry/opentelemetry-proto diff --git a/docs/otel-spec/resource-and-config.md b/docs/otel-spec/resource-and-config.md index d17d3972f0..a4c445865e 100644 --- a/docs/otel-spec/resource-and-config.md +++ b/docs/otel-spec/resource-and-config.md @@ -1,28 +1,28 @@ # Resource, Entities & SDK Configuration -This page is a compliance reference for the OpenTelemetry **Resource** model (what a "resource" is, how it's built and merged), the **resource semantic conventions** that matter most to Maple (`service.*`, `deployment.environment.name`, `telemetry.sdk.*`, plus summary-level `host.*`/`k8s.*`/`cloud.*`), the **SDK environment-variable spec** (the consolidated table is the centerpiece), the emerging **declarative/file configuration** mechanism, and the current **Entities** data model effort. It exists for spec-compliance checks and to back a best-practices skill — every non-obvious claim below carries an inline `Source:` link to the fetched spec page it was verified against. +A compliance reference for the OpenTelemetry **Resource** model (what a resource is, how it is built and merged) and the **resource semantic conventions** that matter most to Maple (`service.*`, `deployment.environment.name`, `telemetry.sdk.*`, plus summary-level `host.*`/`k8s.*`/`cloud.*`). It also covers the **SDK environment-variable spec** (the consolidated table in §5), **declarative/file configuration**, and the **Entities** data model effort. It backs spec-compliance checks and a best-practices skill. Every non-obvious claim carries an inline `Source:` link to the spec page it was verified against. ## Relevance to Maple -- Maple is primarily a **consumer/backend** of OTel data (Rust `ingest` gateway → collector → ClickHouse/Tinybird → web dashboard), plus we **self-instrument** our own services (`apps/ingest`, the API, etc.) — so we need to both parse resources correctly from arbitrary SDKs _and_ emit spec-correct resources ourselves. -- **Dashboards key off `service.name` + `deployment.environment.name`.** The legacy `deployment.environment` attribute is deprecated (replaced by `deployment.environment.name`); our ingest gateway dual-emits both today, and every materialized view coalesces the two (`DEPLOYMENT_ENV_SQL` in `packages/domain/src/tinybird/semconv-renames.ts`, ClickHouse migration 0020) so telemetry carrying either key resolves to the same environment — see `CLAUDE.md`. This doc's deprecation citation below is the spec basis for eventually dropping the legacy emission, which now waits only on pre-0020 rows aging out and on BYO-ClickHouse schemas catching up. -- Our ingest gateway stamps a **vendor-namespaced** `maple_org_id` (and `maple.*` span attributes) — this is intentionally outside semconv (no `maple.*` resource semantic convention exists upstream), so it must never collide with a real OTel resource attribute name. -- We do not currently consume declarative/file configuration or the Entities data model, but both are tracked here because they affect how upstream SDKs will emit resources in the near future (Entities) and how our own self-instrumentation could be configured (file config). +- Maple is mainly a **consumer/backend** of OTel data (Rust ingest gateway `apps/ingest` → Tinybird/ClickHouse, or a collector in forward mode → web dashboard). We also **self-instrument** our own services (`apps/ingest`, the API, etc.). So we must parse resources correctly from arbitrary SDKs _and_ emit spec-correct resources ourselves. +- **Dashboards key off `service.name` + `deployment.environment.name`.** The legacy `deployment.environment` attribute is deprecated (replaced by `deployment.environment.name`). The ingest gateway's own self-telemetry resource dual-emits both keys today (`apps/ingest/src/otel.rs`). Every materialized view coalesces the two (`DEPLOYMENT_ENV_SQL` in `packages/domain/src/tinybird/semconv-renames.ts`, ClickHouse migration `packages/domain/src/clickhouse/migrations/0020_semconv_key_renames.ts`), so telemetry carrying either key resolves to the same environment. The deprecation citation below is the spec basis for dropping the legacy emission. That now waits only on pre-0020 rows aging out and on BYO-ClickHouse schemas catching up. +- The ingest gateway stamps a Maple-internal `maple_org_id` resource attribute (and `maple_ai.*` span attributes on AI agent spans, see `apps/ingest/src/ai_session.rs`). These are intentionally outside semconv and must never collide with a real OTel attribute name. The dotted `maple.*` namespace belongs to Maple's SDKs, not the gateway. +- We do not consume declarative/file configuration or the Entities data model yet. Both are tracked because they affect how upstream SDKs will emit resources (Entities) and how our self-instrumentation could be configured (file config). --- ## 1. Resource: definition, immutability, merge rules -**Stability: Stable** (the Resource SDK spec is stable except where individual parts are marked otherwise — resource _detector names_ are explicitly called out as Development). +**Stability: Stable**, except where individual parts are marked otherwise (resource _detector names_ are called out as Development). Source: https://opentelemetry.io/docs/specs/otel/resource/sdk/ - **Definition:** a Resource is "an immutable representation of the observed entity for which telemetry is being produced, expressed as Attributes." Once created, a Resource cannot be modified. - **Creation:** SDKs expose a `Create(attributes, schema_url)` factory. `schema_url` is optional and records the Schema URL associated with the emitted attributes. -- **Merge rule — critical for compliance checks:** when merging an _old_ resource with an _updating_ resource, "the value of the updating resource MUST be picked (even if the updated value is empty)." I.e., **the updating side always wins**, including overwriting a populated attribute with an empty one. Merge is not "fill in only missing keys." -- **Schema URL on merge:** combining two resources that carry **different, non-empty** schema URLs is a merging error — schema URLs must match (or one/both must be empty) for a clean merge. -- **`OTEL_RESOURCE_ATTRIBUTES`:** "The SDK MUST extract information from the `OTEL_RESOURCE_ATTRIBUTES` environment variable and merge this, as the secondary resource, with any resource information provided by the user." Format is `key1=value1,key2=value2`, percent-encoded for special characters — i.e. user-supplied/programmatic resource attributes win over the env var per the merge rule above (env var is the "old"/secondary side). -- **`OTEL_SERVICE_NAME`:** the `service` resource detector populates `service.name` from this env var; per the SDK environment variable spec (below), `OTEL_SERVICE_NAME` takes precedence over any `service.name` key set via `OTEL_RESOURCE_ATTRIBUTES`. -- **Default `service.name`:** if `service.name` is not otherwise specified, "SDKs MUST fallback to `unknown_service:` concatenated with the process executable name, e.g. `unknown_service:bash`." If the executable name is unavailable, the value degrades to plain `unknown_service`. (Note: many SDKs/docs colloquially write this as `unknown_service[:process]"`, which is what this pattern means in practice — the colon-joined executable name is included whenever known.) +- **Merge rule (critical for compliance checks):** when merging an _old_ resource with an _updating_ resource, "the value of the updating resource MUST be picked (even if the updated value is empty)." **The updating side always wins**, even overwriting a populated attribute with an empty one. Merge does not only fill missing keys. +- **Schema URL on merge:** combining two resources with **different, non-empty** schema URLs is a merging error. Schema URLs must match (or one or both must be empty) for a clean merge. +- **`OTEL_RESOURCE_ATTRIBUTES`:** "The SDK MUST extract information from the `OTEL_RESOURCE_ATTRIBUTES` environment variable and merge this, as the secondary resource, with any resource information provided by the user." Format is `key1=value1,key2=value2`, percent-encoded for special characters. User-supplied (programmatic) resource attributes win over the env var, which is the "old"/secondary side of the merge. +- **`OTEL_SERVICE_NAME`:** the `service` resource detector populates `service.name` from this env var. Per the SDK environment variable spec (§5), it takes precedence over a `service.name` key set via `OTEL_RESOURCE_ATTRIBUTES`. +- **Default `service.name`:** if `service.name` is not otherwise specified, "SDKs MUST fallback to `unknown_service:` concatenated with the process executable name, e.g. `unknown_service:bash`." If the executable name is unavailable, the value is plain `unknown_service`. Docs often write this as `unknown_service[:process]`: the colon-joined executable name is included whenever known. Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/service/ ## 2. Resource data model (Entities-aware) @@ -32,10 +32,10 @@ Source: https://opentelemetry.io/docs/specs/otel/resource/data-model/ The current logical Resource data model has two fields: -- `attributes` — the flat map of resource attribute key/values, which "MUST not change during the lifetime of the resource." -- `entities` — (newer, Development) a set of Entity objects the resource is composed of, per the Entities data model (§7). Resources also carry a `schema_url`, derived during entity merging: shared across all entities' schema URLs when they agree, left blank on conflict. +- `attributes`: the flat map of resource attribute key/values, which "MUST not change during the lifetime of the resource." +- `entities` (newer, Development): the set of Entity objects the resource is composed of, per the Entities data model (§7). Resources also carry a `schema_url`, derived during entity merging: the shared schema URL when all entities agree, blank on conflict. -This model is the forward-looking replacement for "resource = flat attribute bag"; see §7 for the Entities effort itself. +This model replaces "resource = flat attribute bag" going forward. See §7 for the Entities effort. ## 3. Key resource semantic conventions @@ -46,7 +46,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/service/ | Attribute | Requirement | Notes | | --------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `service.name` | Recommended | "MUST be the same for all instances of horizontally scaled services." Falls back to `unknown_service:` if unset (see §1). Configurable via `OTEL_SERVICE_NAME` (SDK env var spec) or provided as a default by the SDK (Resource SDK spec). | +| `service.name` | Required | "MUST be the same for all instances of horizontally scaled services." Falls back to `unknown_service:` if unset (see §1). Configurable via `OTEL_SERVICE_NAME` (SDK env var spec) or provided as a default by the SDK (Resource SDK spec). | | `service.namespace` | Optional | Logical grouping for a set of services; service names are expected to be unique **within** the same namespace. "Zero-length namespace string is assumed equal to unspecified namespace." | | `service.version` | Optional | Version string of the service; format is undefined (examples: `2.0.0`, `a01dbef8a`). | | `service.instance.id` | Optional | "MUST be unique for each instance of the same `service.namespace`, `service.name` pair." Recommendation: generate a random **UUID v1 or v4** (RFC 4122), or derive a **UUID v5** from a stable namespace UUID (`4d63009a-8d0f-11ee-aad7-4c796ed8e320`) when a stable seed exists. Treat data like pod names, used as input to instance IDs, as potentially confidential. | @@ -55,10 +55,10 @@ Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/service/ Source: https://opentelemetry.io/docs/specs/semconv/resource/deployment-environment/ and https://opentelemetry.io/docs/specs/semconv/registry/attributes/deployment/ -- **`deployment.environment.name`** — **Stable**, Recommended, type `string`. "Name of the deployment environment (aka deployment tier)." Well-known values (all Stable): `development`, `production`, `staging`, `test` — a well-known value MUST be used when it applies; otherwise a custom value MAY be used. -- Explicit note: this attribute "does not affect the uniqueness constraints defined through the `service.namespace`, `service.name` and `service.instance.id` attributes" — i.e. the same logical service running in two environments is still considered the same service identity for uniqueness purposes; the environment is an orthogonal dimension. -- **Legacy `deployment.environment`** (no `.name`) is registered as **Deprecated**, with the exact registry note: **"Replaced by `deployment.environment.name`."** This is the citable spec basis for the rename Maple handles by dual-emitting both keys on the write side and coalescing them on every read (see `CLAUDE.md` self-observability section) — compliant new code should emit only `deployment.environment.name`. -- The broader "Deployment" resource/entity group itself is at **Development** status overall (per the resource semconv landing page), even though the two attributes above are individually Stable — Development status here reflects unfinished entity-role modeling, not attribute-level instability. +- **`deployment.environment.name`**: **Stable**, Recommended, type `string`. "Name of the deployment environment (aka deployment tier)." Well-known values (all Stable): `development`, `production`, `staging`, `test`. A well-known value MUST be used when it applies; otherwise a custom value MAY be used. +- Explicit note: this attribute "does not affect the uniqueness constraints defined through the `service.namespace`, `service.name` and `service.instance.id` attributes". The same logical service running in two environments keeps one service identity for uniqueness purposes; environment is an orthogonal dimension. +- **Legacy `deployment.environment`** (no `.name`) is registered as **Deprecated**, with the exact registry note: **"Replaced by `deployment.environment.name`."** This is the spec basis for the rename Maple handles by dual-emitting both keys in its own self-telemetry and coalescing them on every read (see Relevance to Maple above). Compliant new code should emit only `deployment.environment.name`. +- The broader "Deployment" resource/entity group is **Development** overall (per the resource semconv landing page), even though the attributes above are individually Stable. The group status reflects unfinished entity-role modeling, not attribute-level instability. ### 3.3 `telemetry.sdk.*` / `telemetry.distro.*` @@ -73,15 +73,15 @@ Source: https://opentelemetry.io/docs/specs/semconv/resource/ | `telemetry.distro.name` | Recommended | Name of the auto-instrumentation agent/distro, if used. | | `telemetry.distro.version` | Recommended | Distro version string. | -### 3.4 `host.*`, `k8s.*`, `cloud.*` — summary level +### 3.4 `host.*`, `k8s.*`, `cloud.*`: summary level -Full definitions: https://opentelemetry.io/docs/specs/semconv/resource/ (index; links to each group's registry page). Per-group stability below reflects each attribute's individual maturity badge, independent of the overall resource page's blanket "Development" note for these groups. +Full definitions: https://opentelemetry.io/docs/specs/semconv/resource/ (index; links to each group's registry page). Stability below is each attribute's own badge, independent of the resource page's blanket "Development" note for these groups. -**`host.*`** — Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/host/ — all listed attributes currently **Development**: `host.name`, `host.id` (mandatory shape for cloud: provider's `instance_id`), `host.type` (cloud machine type), `host.arch`, `host.image.id`/`host.image.name`/`host.image.version`. +**`host.*`** (source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/host/). All listed attributes are **Development**: `host.name`, `host.id` (mandatory shape for cloud: provider's `instance_id`), `host.type` (cloud machine type), `host.arch`, `host.image.id`/`host.image.name`/`host.image.version`. -**`k8s.*`** — Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/k8s/ — the widely-used identity attributes are **Stable**: `k8s.cluster.name`, `k8s.cluster.uid`, `k8s.node.name`/`k8s.node.uid`, `k8s.namespace.name`, `k8s.pod.name`/`k8s.pod.uid`, `k8s.container.name`, `k8s.deployment.name`, `k8s.statefulset.name`, `k8s.daemonset.name`, `k8s.job.name`, `k8s.cronjob.name`. Newer additions are **Development**: `k8s.persistentvolume.name`, `k8s.persistentvolumeclaim.name`, `k8s.service.name`, `k8s.service.type`. +**`k8s.*`** (source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/k8s/). The widely used identity attributes are **Stable**: `k8s.cluster.name`, `k8s.cluster.uid`, `k8s.node.name`/`k8s.node.uid`, `k8s.namespace.name`, `k8s.pod.name`/`k8s.pod.uid`, `k8s.container.name`, `k8s.deployment.name`, `k8s.statefulset.name`, `k8s.daemonset.name`, `k8s.job.name`, `k8s.cronjob.name`. Newer additions are **Development**: `k8s.persistentvolume.name`, `k8s.persistentvolumeclaim.name`, `k8s.service.name`, `k8s.service.type`. -**`cloud.*`** — Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/cloud/ — all currently **Development**: `cloud.provider` (well-known values incl. `aws`/`azure`/`gcp`/`alibaba_cloud`, must use a well-known value when applicable), `cloud.platform` (e.g. `aws_lambda`, `gcp_cloud_run`), `cloud.region`, `cloud.account.id`, `cloud.availability_zone`, `cloud.resource_id` (provider-specific full identifier — ARN, etc.). +**`cloud.*`** (source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/cloud/). All are **Development**: `cloud.provider` (well-known values incl. `aws`/`azure`/`gcp`/`alibaba_cloud`, must use a well-known value when applicable), `cloud.platform` (e.g. `aws_lambda`, `gcp_cloud_run`), `cloud.region`, `cloud.account.id`, `cloud.availability_zone`, `cloud.resource_id` (provider-specific full identifier, e.g. an ARN). ## 4. Common attribute spec (applies to resource _and_ span/log/metric attributes) @@ -90,13 +90,13 @@ Source: https://opentelemetry.io/docs/specs/otel/common/ - **Permitted value types** for an attribute (`AnyValue`): primitive `string`, `boolean`, `double` (IEEE 754-1985), signed 64-bit `integer`; **homogeneous arrays** of those primitives; byte arrays; arrays of `AnyValue`; and `map`. - **Homogeneity rule (exact wording):** "A homogeneous array MUST NOT contain values of different types." -- **Empty values are meaningful:** empty strings, zero numbers, and empty arrays must be preserved through processors/exporters — they are not treated as "unset." -- **Null handling (exact wording):** "While `null` is a valid attribute value, its use within homogeneous arrays SHOULD generally be avoided unless language constraints make this impossible." If nulls do occur, "they MUST be preserved as-is (i.e., passed on to processors / exporters as `null`)" — nulls must not be silently dropped or coerced. +- **Empty values are meaningful:** empty strings, zero numbers, and empty arrays must be preserved through processors and exporters. They are not "unset". +- **Null handling (exact wording):** "While `null` is a valid attribute value, its use within homogeneous arrays SHOULD generally be avoided unless language constraints make this impossible." If nulls do occur, "they MUST be preserved as-is (i.e., passed on to processors / exporters as `null`)". Nulls must not be dropped or coerced. - **Attribute limits** (defaults, enforced by SDKs): - - `AttributeCountLimit` — default **128** (max attributes per record). - - `AttributeValueLengthLimit` — default **Infinity/no limit** (max length of string/byte-array values). + - `AttributeCountLimit`: default **128** (max attributes per record). + - `AttributeValueLengthLimit`: default **Infinity/no limit** (max length of string/byte-array values). - The count limit "applies only to top-level attributes, not to nested key-value pairs within maps." - - **Resource attributes and metric attributes are explicitly exempt** from these SDK-enforced limits (relevant: don't assume `OTEL_ATTRIBUTE_COUNT_LIMIT` truncates a resource — it doesn't, per this spec section, though exporters/backends may impose their own caps). + - **Resource attributes and metric attributes are explicitly exempt** from these SDK-enforced limits. `OTEL_ATTRIBUTE_COUNT_LIMIT` does not truncate a resource, though exporters and backends may impose their own caps. ## 5. SDK environment-variable spec (consolidated table) @@ -126,7 +126,7 @@ Source: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-v | Variable | Default | Notes | | ------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_TRACES_SAMPLER` | `parentbased_always_on` | e.g. `parentbased_traceidratio`, `always_on`, `always_off`, `traceidratio`. | -| `OTEL_TRACES_SAMPLER_ARG` | _(unset)_ | Argument shape depends on the chosen sampler (e.g. a ratio `0.1` for `traceidratio`/`parentbased_traceidratio` — used at high QPS by `apps/ingest`, per `CLAUDE.md`). | +| `OTEL_TRACES_SAMPLER_ARG` | _(unset)_ | Argument shape depends on the sampler (e.g. a ratio `0.1` for `traceidratio`/`parentbased_traceidratio`). `apps/ingest` sets its head sampling through these two vars. | ### Batch Span Processor (BSP) @@ -182,37 +182,37 @@ Base variable applies to all signals unless a signal-specific variant overrides | `OTEL_EXPORTER_OTLP_COMPRESSION` | _(unset)_; `none` or `gzip` | | `OTEL_EXPORTER_OTLP_TIMEOUT` | 10 s | | `OTEL_EXPORTER_OTLP_INSECURE` | `false` (gRPC only) | -| `OTEL_EXPORTER_OTLP_CERTIFICATE` | _(unset)_ — trusted server cert for TLS verification | -| `OTEL_EXPORTER_OTLP_CLIENT_KEY` | _(unset)_ — client private key (PEM), mTLS | -| `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` | _(unset)_ — client cert/chain (PEM), mTLS | +| `OTEL_EXPORTER_OTLP_CERTIFICATE` | _(unset)_; trusted server cert for TLS verification | +| `OTEL_EXPORTER_OTLP_CLIENT_KEY` | _(unset)_; client private key (PEM), mTLS | +| `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` | _(unset)_; client cert/chain (PEM), mTLS | Note: "SDKs SHOULD default endpoint variables to use `http` scheme unless they have good reasons to choose `https` scheme." ## 6. Declarative / file configuration -**Stability: the core schema is now Stable** (JSON schema, YAML file representation, in-memory representation, `ConfigProperties`, `PluginComponentProvider`, and the SDK `Parse`/`Create` operations were stabilized alongside an `opentelemetry-configuration` `1.0.0` release, announced 2026). Portions still evolving are explicitly namespaced `.../development` in the schema so stable and experimental parts can coexist. +**Stability: the core schema is Stable.** The JSON schema, YAML file representation, in-memory representation, `ConfigProperties`, `PluginComponentProvider`, and the SDK `Parse`/`Create` operations were stabilized with the `opentelemetry-configuration` `1.0.0` release (announced 2026). Parts still evolving are namespaced `.../development` in the schema, so stable and experimental parts coexist. Sources: https://opentelemetry.io/blog/2026/stable-declarative-config/ , https://opentelemetry.io/docs/specs/otel/configuration/sdk/ , https://opentelemetry.io/docs/specs/otel/configuration/ - **Mechanism:** the SDK exposes a two-step `Parse` (reads/validates a YAML file, performs env-var substitution, returns an in-memory config model) → `Create` (instantiates `TracerProvider`/`MeterProvider`/`LoggerProvider`/propagators from that model) flow. - **File format:** YAML, validated against the `opentelemetry-configuration` JSON Schema. -- **Env var to opt in / point at a file:** **`OTEL_CONFIG_FILE`** (the stabilized name — treat any reference to `OTEL_EXPERIMENTAL_CONFIG_FILE` as the older/language-specific pre-stabilization spelling; some SDKs may still only support the experimental name until they finish adopting `1.0.0`). -- **Positioning vs. env vars:** declarative config is described as "more expressive and full-featured than the environment variable based scheme." The spec's own stated direction is to eventually "deprecate environment variables which don't interoperate well" with file config, but the pages fetched do **not** spell out a precise precedence/merge rule for the case where both an env-var-based and a file-based configuration are present simultaneously — treat that as unverified/TBD rather than assume a specific precedence. +- **Env var to opt in / point at a file:** **`OTEL_CONFIG_FILE`** (the stabilized name). `OTEL_EXPERIMENTAL_CONFIG_FILE` is the pre-stabilization spelling; some SDKs may support only that name until they adopt `1.0.0`. +- **Positioning vs. env vars:** declarative config is described as "more expressive and full-featured than the environment variable based scheme." The spec's own stated direction is to eventually "deprecate environment variables which don't interoperate well" with file config. The fetched pages do **not** define a precedence or merge rule when env-var and file configuration are both present. Treat that as unverified; do not assume a precedence. - **Language implementation status (as of the stabilization blog post):** C++, Go, Java, JavaScript, and PHP have implementations; .NET and Python are in progress. The Go implementation is also reused by the Collector for its own internal telemetry configuration. -- **Maple relevance:** not currently used anywhere in this repo; noted here so future self-instrumentation config changes can cite the correct stable env var (`OTEL_CONFIG_FILE`) rather than the deprecated experimental name. +- **Maple relevance:** not used in this repo. Noted so future self-instrumentation config can cite the stable env var (`OTEL_CONFIG_FILE`) instead of the experimental name. ## 7. Entities -**Stability: Development** (the whole effort — overview and data model pages both carry an explicit `Status: Development` marker). +**Stability: Development** (the overview and data model pages both carry an explicit `Status: Development` marker). Sources: https://opentelemetry.io/docs/specs/otel/entities/ , https://opentelemetry.io/docs/specs/otel/entities/data-model/ -- **Purpose:** a common data model for "entities" — objects of interest associated with produced telemetry (e.g. a `service`, `host`, `k8s.pod`, `k8s.cluster`) — so multiple distinct-but-related components can be represented within the same signal's resource, each with its own identity and metadata, rather than flattening everything into one undifferentiated attribute bag. +- **Purpose:** a common data model for "entities", the objects of interest associated with produced telemetry (e.g. a `service`, `host`, `k8s.pod`, `k8s.cluster`). Several related components can then be represented in one signal's resource, each with its own identity and metadata, instead of one flat attribute bag. - **Data model fields:** - - `type` — required, immutable string category (e.g. `"service"`, `"host"`). - - `id` — a map of identifying attributes; "must contain at least one attribute" and should be the minimal attribute set sufficient to uniquely identify the entity. - - `description` — a map of descriptive, non-identifying attributes that may change over time and may be empty. -- **Relationship to Resource:** entities are attached in the resource section of a signal and reference the same shared attribute pool defined by Resource — this preserves backward compatibility with existing Resource-attribute consumers while avoiding duplicating identifying data across multiple entities on the same signal. -- **Instantiation models:** _pull-based_ (an entity discovers itself from the current runtime/environment — analogous to today's resource detectors) and _push-based_ (entity info supplied externally, typically via environment variables or declarative configuration). -- **Practical takeaway for Maple:** this is not yet something ClickHouse/Tinybird ingestion needs to model explicitly (Development stability, not yet emitted by mainstream SDKs by default), but the ingest gateway's resource-attribute parsing should not assume the Resource model stays a flat attribute-only bag forever — the `entities` field on the Resource data model (§2) is the forward path. + - `type`: required, immutable string category (e.g. `"service"`, `"host"`). + - `id`: a map of identifying attributes; it "must contain at least one attribute" and should be the minimal set that uniquely identifies the entity. + - `description`: a map of descriptive, non-identifying attributes that may change over time and may be empty. +- **Relationship to Resource:** entities are attached in a signal's resource section and reference the Resource's shared attribute pool. This keeps existing Resource-attribute consumers working and avoids duplicating identifying data across entities on one signal. +- **Instantiation models:** _pull-based_ (an entity discovers itself from the runtime environment, like today's resource detectors) and _push-based_ (entity info supplied externally, typically via environment variables or declarative configuration). +- **Practical takeaway for Maple:** ClickHouse/Tinybird ingestion does not need to model entities yet (Development status, not emitted by mainstream SDKs by default). The ingest gateway's resource parsing should still not assume Resource stays a flat attribute bag. The `entities` field on the Resource data model (§2) is the forward path. --- diff --git a/docs/otel-spec/semantic-conventions.md b/docs/otel-spec/semantic-conventions.md index d64f49c1ad..e187592d58 100644 --- a/docs/otel-spec/semantic-conventions.md +++ b/docs/otel-spec/semantic-conventions.md @@ -2,7 +2,12 @@ OpenTelemetry Semantic Conventions ("semconv") define the common attribute names, span/metric naming rules, and per-domain required/recommended fields that give telemetry consistent meaning across languages, libraries, and vendors. They live in a single spec (currently **v1.43.0**) covering traces, metrics, logs, resources, and profiles, published from `github.com/open-telemetry/semantic-conventions` and rendered at `https://opentelemetry.io/docs/specs/semconv/`. -> **Relevance to Maple:** Maple is primarily a **consumer/backend** for OTel data — our ClickHouse/Tinybird schemas, dashboard UI, and query engine all key on semconv attribute names, so drift between "what the spec says" and "what our ingest/query layers expect" causes silent breakage (missing facets, wrong span-status coloring, unmatched filters). Our trace UI keys on `http.request.method`/`http.route`/`http.response.status_code`; our Rust ingest gateway (`apps/ingest`) emits new-semconv HTTP attributes and self-instruments per `service.name="ingest"` (see root `CLAUDE.md`); warehouse read paths alias legacy `db.statement`/`db.system` to `db.query.text`/`db.system.name` for spans recorded before the DB semconv stability migration; and per project convention our own span status values use **title case** (`"Ok"`, `"Error"`, `"Unset"`), matching the OTel API enum names rather than the lowercase wire values some SDKs historically emitted. +> **Relevance to Maple:** Maple is mainly a **consumer/backend** for OTel data. Our ClickHouse/Tinybird schemas, dashboard UI, and query engine all key on semconv attribute names, so drift between the spec and what our ingest/query layers expect causes silent breakage (missing facets, wrong span-status coloring, unmatched filters). +> +> - Our trace UI keys on `http.request.method`/`http.route`/`http.response.status_code`. +> - Our Rust ingest gateway (`apps/ingest`) emits new-semconv HTTP attributes and self-instruments with `service.name="ingest"` (`apps/ingest/src/otel.rs`). +> - Warehouse queries fall back from `db.query.text`/`db.system.name` to the legacy `db.statement`/`db.system` for instrumentations that predate the DB semconv stability migration. +> - Per project convention, our span status values use **title case** (`"Ok"`, `"Error"`, `"Unset"`). This matches the OTel API enum names, not the uppercase `STATUS_CODE_*` OTLP wire enum. --- @@ -12,12 +17,15 @@ OpenTelemetry Semantic Conventions ("semconv") define the common attribute names Source: https://opentelemetry.io/docs/specs/semconv/general/naming/ -- "Names SHOULD be lowercase" and use **namespacing**, delimited by a dot character (e.g. `http.response.status_code` is the `status_code` attribute in the `http` namespace, itself nested under `http.response`). +- "Names SHOULD be lowercase" and use **namespacing**, delimited by a dot. For example, `http.response.status_code` is the `status_code` attribute in the `http.response` namespace, itself nested under `http`. - Multi-word components within a dot-delimited segment use **snake_case** (`status_code`, not `statusCode` or `status-code`). -- Character rules: names must be a valid Unicode sequence restricted to printable Basic Latin; tooling further limits to lowercase Latin letters, digits, underscore, and dot; names **must start with a letter, end with an alphanumeric character**, and must not contain two or more consecutive delimiters. +- Character rules: names must be a valid Unicode sequence restricted to printable Basic Latin. Tooling further limits them to lowercase Latin letters, digits, underscore, and dot. Names **must start with a letter and end with an alphanumeric character**, and must not contain two or more consecutive delimiters. - Pluralization: singular for single-valued attributes (`host.name`), plural for array-valued attributes (`process.command_args`). - Custom/vendor attributes outside the registry should be namespaced under a reverse-DNS-style prefix (e.g. `com.acme.shopname.*`) to avoid collisions with future registry additions. -- Metric names additionally: namespaces and metric names generally **SHOULD NOT be pluralized** (`system.process.count`, not `system.processes`); counters/up-down counters **SHOULD NOT** append `_total`; units follow UCUM (Unified Code for Units of Measure), preferring un-prefixed units (`By` not `MiBy`) and `s` for durations, with bracketed annotations matching grammatical number (`{request}` not `{requests}`). +- Metric names additionally: + - Namespaces and metric names generally **SHOULD NOT be pluralized** (`system.process.count`, not `system.processes`). + - Counters and up-down counters **SHOULD NOT** append `_total`. + - Units follow UCUM (Unified Code for Units of Measure). Prefer un-prefixed units (`By`, not `MiBy`) and `s` for durations. Bracketed annotations use singular form (`{request}`, not `{requests}`). ### Attribute requirement levels @@ -32,20 +40,20 @@ Source: https://opentelemetry.io/docs/specs/semconv/general/attribute-requiremen ### Span naming -Source: https://opentelemetry.io/docs/specs/otel/trace/api/ (core span-naming rule, part of the tracing API spec that semconv domains implement against) +Source: https://opentelemetry.io/docs/specs/otel/trace/api/ (the core span-naming rule lives in the tracing API spec, which semconv domains implement against) -- "The span name SHOULD be the most general string that identifies a (statistically) interesting class of Spans, rather than individual Span instances" — i.e. **low cardinality is mandatory**. `get_user` is a good name; `get_user/314159` (embedding a user ID) is not. -- Generality is prioritized over human-readability when the two conflict. -- Each domain gives its own concrete template (e.g. HTTP: `{method} {route}`; DB: `{db.query.summary}` or `{db.operation.name} {target}`; see domain sections below) — always built from low-cardinality attribute values, never from raw high-cardinality identifiers (full URLs, raw SQL literals, IDs). +- "The span name SHOULD be the most general string that identifies a (statistically) interesting class of Spans, rather than individual Span instances." Low cardinality is effectively mandatory. `get_user` is a good name; `get_user/314159` (embedding a user ID) is not. +- Generality wins over human readability when the two conflict. +- Each domain gives its own template (e.g. HTTP: `{method} {route}`; DB: `{db.query.summary}` or `{db.operation.name} {target}`; see domain sections below). Templates are always built from low-cardinality attribute values, never from raw high-cardinality identifiers (full URLs, raw SQL literals, IDs). ### Recording errors / span status Source: https://opentelemetry.io/docs/specs/semconv/general/recording-errors/ - "Span Status Code MUST be left unset if the instrumented operation has ended without any errors." Only set `Error` on genuine failures. -- When an operation fails: set span status to `Error` and set `error.type`; an optional status description can add detail but shouldn't just repeat the status code or error type. -- `error.type` must be applied **consistently** between a span and its corresponding metric for the same operation: same value if failed, absent if succeeded. -- Errors that were retried/handled internally (the operation still completes successfully) **SHOULD NOT** be recorded as errors on the span/metric describing that operation. +- When an operation fails, set span status to `Error` and set `error.type`. An optional status description can add detail, but shouldn't just repeat the status code or error type. +- Apply `error.type` **consistently** between a span and its corresponding metric for the same operation: same value if failed, absent if succeeded. +- Errors that were retried or handled internally (the operation still completes successfully) **SHOULD NOT** be recorded as errors on the span/metric describing that operation. ### Stability levels @@ -60,41 +68,50 @@ OpenTelemetry signals (including semconv domains) move through a lifecycle: | **Deprecated** | Being phased out; "Signals MUST NOT be marked as deprecated unless the replacement is stable." Same support guarantees as stable until removed. | | **Removed** | No longer supported; removal requires a major version bump. | -Per-domain stability (see each section below): **HTTP is Stable**; **Database is Stable** (as of the v1.30-era stability migration); **Messaging is Development**; **RPC is Release Candidate**; **Exceptions**-as-span-events is **Deprecated** in favor of exceptions-on-logs; **Code attributes are Stable**; **General attributes (`server.*`, `client.*`, `network.*`, `url.*`) are Stable** (a handful of derived `url.*` sub-attributes remain Development); **Gen-AI is Development** and has moved to a separate spec repo. +Per-domain stability (details in each section below): + +- **HTTP:** Stable. +- **Database:** Stable (as of the v1.30-era stability migration). +- **Messaging:** Development. +- **RPC:** Release Candidate. +- **Exceptions** as span events: Deprecated in favor of exceptions on logs. +- **Code attributes:** Stable. +- **General attributes** (`server.*`, `client.*`, `network.*`, `url.*`): Stable. A handful of derived `url.*` sub-attributes remain Development. +- **Gen-AI:** Development, and moved to a separate spec repo. ### The `OTEL_SEMCONV_STABILITY_OPT_IN` migration pattern Source: https://opentelemetry.io/docs/specs/semconv/non-normative/http-migration/ · https://opentelemetry.io/docs/specs/semconv/non-normative/db-migration/ -When a domain graduates from experimental to stable, its attribute names/values often change (e.g. `http.method` → `http.request.method`). Because this breaks dashboards/alerts built on the old names, instrumentations expose an env var so users can migrate on their own schedule instead of the instrumentation silently flipping conventions: +When a domain graduates from experimental to stable, its attribute names and values often change (e.g. `http.method` → `http.request.method`). That breaks dashboards and alerts built on the old names, so instrumentations expose an env var that lets users migrate on their own schedule instead of silently flipping conventions: - **HTTP** (`OTEL_SEMCONV_STABILITY_OPT_IN`): `http` = emit only the new stable HTTP/network conventions; `http/dup` = emit **both** old and new (phased rollout); unset/default = keep emitting whatever the instrumentation emitted before. -- **Database**: same three-way shape with values `database` / `database/dup` / default. -- **Gen-AI**: `gen_ai_latest_experimental` opts into the newest experimental gen-ai shape instead of the pre-1.36.0 one; default keeps emitting whatever version was previously emitted. -- Constraint: this variable is "only intended to be used when migrating from an experimental semantic convention to its initial stable version" — not a general-purpose versioning knob. Instrumentations must support the `/dup` dual-emit mode for **at least six months** after introducing it, within their current major version, before dropping the old shape in a subsequent major version. -- Exceptions have an analogous but distinct variable, `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN` (`logs` / `logs/dup` / default) — see the Exceptions section. +- **Database**: the same three-way shape, with values `database` / `database/dup` / default. +- **Gen-AI**: `gen_ai_latest_experimental` opts into the newest experimental gen-ai shape instead of the pre-1.36.0 one. The default keeps emitting whatever version was previously emitted. +- Constraint: this variable is "only intended to be used when migrating from an experimental semantic convention to its initial stable version." It is not a general-purpose versioning knob. Instrumentations must support the `/dup` dual-emit mode for **at least six months** after introducing it, within their current major version, before dropping the old shape in a later major version. +- Exceptions have an analogous but distinct variable, `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN` (`logs` / `logs/dup` / default). See the Exceptions section. ### Schema URLs & telemetry schemas Source: https://opentelemetry.io/docs/specs/otel/schemas/ -- A **Schema URL** identifies a specific version of a telemetry schema and is the address from which the schema file can be fetched: `http[s]://server[:port]/path/`. The portion before the version is the **Schema Family** identifier, stable across all versions in that family. -- OTLP carries `schema_url` at the `Resource*` message level and at the `InstrumentationScope`/instrumentation-library level; when both are present, the scope-level one takes precedence for that scope's data. -- Purpose: semantic conventions evolve (attributes get renamed, e.g. `http.method` → `http.request.method`, or `deployment.environment` → `deployment.environment.name`); a schema file formally declares the transformation rules needed to convert telemetry between schema versions, so a consumer that knows the schema version of incoming data (via `schema_url`) can mechanically rewrite it to the version its dashboards/alerts expect, instead of every consumer hand-rolling compatibility shims. +- A **Schema URL** identifies a specific version of a telemetry schema and is the address the schema file can be fetched from: `http[s]://server[:port]/path/`. The portion before the version is the **Schema Family** identifier, stable across all versions in that family. +- OTLP carries `schema_url` at two levels: on `Resource*` messages (e.g. `ResourceSpans`) and on `Scope*` messages (e.g. `ScopeSpans`). They cover different data. The resource-level `schema_url` applies to the `resource` field only; each scope-level `schema_url` applies to that scope's spans/metrics/logs. +- Purpose: semantic conventions evolve (attributes get renamed, e.g. `http.method` → `http.request.method`, or `deployment.environment` → `deployment.environment.name`). A schema file declares the transformation rules needed to convert telemetry between schema versions. A consumer that knows the schema version of incoming data (via `schema_url`) can rewrite it mechanically to the version its dashboards/alerts expect, instead of every consumer hand-rolling compatibility shims. - The schema file format itself is versioned separately (`file_format_v1.0.0`, `file_format_v1.1.0`) from the schema content it describes. ### The semconv registry Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/ · https://github.com/open-telemetry/semantic-conventions -- The **attribute registry** (`/docs/specs/semconv/registry/attributes/`) is an auto-generated, searchable/browsable catalog of every registered attribute, organized by namespace (`http.*`, `db.*`, `network.*`, `code.*`, …), each with type, stability, and description — this is the fastest way to check an attribute's current name/stability without reading the full domain spec. -- The registry and all domain specs are generated from YAML model definitions in `github.com/open-telemetry/semantic-conventions` (the source of truth); the rendered docs on opentelemetry.io are a build artifact of that repo. Gen-AI conventions have since split out into their own sibling repo, `github.com/open-telemetry/semantic-conventions-genai` (see Gen-AI section). +- The **attribute registry** (`/docs/specs/semconv/registry/attributes/`) is an auto-generated, browsable catalog of every registered attribute, organized by namespace (`http.*`, `db.*`, `network.*`, `code.*`, …), each with type, stability, and description. It is the fastest way to check an attribute's current name and stability without reading the full domain spec. +- The registry and all domain specs are generated from YAML model definitions in `github.com/open-telemetry/semantic-conventions` (the source of truth). The rendered docs on opentelemetry.io are a build artifact of that repo. Gen-AI conventions have since split out into their own sibling repo, `github.com/open-telemetry/semantic-conventions-genai` (see the Gen-AI section). --- ## HTTP -**Stability: Stable** (both client and server spans/metrics). +**Stability: Stable** (client and server spans/metrics). Source: https://opentelemetry.io/docs/specs/semconv/http/http-spans/ · https://opentelemetry.io/docs/specs/semconv/http/http-metrics/ ### Span name @@ -107,7 +124,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/http/http-spans/ · https:// | --------------------------- | -------------- | --------------- | ---------------------------------------------------------------------- | | `http.request.method` | Required | Both | GET, POST, HEAD, … | | `server.address` | Required | Client | Domain/IP/socket the request targets | -| `server.port` | Required | Client | — | +| `server.port` | Required | Client | (none) | | `url.full` | Required | Client | Absolute URL, RFC3986; credentials MUST be redacted | | `url.path` | Required | Server | URI path component | | `url.scheme` | Required | Server | `http`/`https` | @@ -118,18 +135,18 @@ Source: https://opentelemetry.io/docs/specs/semconv/http/http-spans/ · https:// | `server.port` | Cond. Required | Server | If available and `server.address` set | | `url.query` | Cond. Required | Server | If a query string was present | -### Span status mapping (the important compliance rule) +### Span status mapping (the key compliance rule) -- **1xx/2xx/3xx**: leave span status **unset**, unless another error occurred (network failure, redirect-limit exceeded) — then `Error`. -- **4xx**: **SERVER** spans leave status **unset** (client errors aren't the server's fault); **CLIENT** spans **SHOULD** be set to `Error`. -- **5xx**: `Error` for **both** client and server spans, plus `error.type` set. -- Intentionally cancelled client requests (caller-initiated) should **not** be treated as errors. +- **1xx/2xx/3xx**: leave span status **unset**, unless another error occurred (network failure, redirect limit exceeded); then `Error`. +- **4xx**: **SERVER** spans leave status **unset** (client errors aren't the server's fault). **CLIENT** spans **SHOULD** be set to `Error`. +- **5xx**: `Error` for **both** client and server spans, plus `error.type`. +- Client requests cancelled intentionally by the caller should **not** be treated as errors. -This asymmetry (4xx = Ok for SERVER, Error for CLIENT) is the single most load-bearing rule for anyone building error-rate dashboards — filtering `StatusCode = 'Error'` on server spans will correctly exclude client 4xx noise, but the same filter on client spans will correctly include it. +This asymmetry (4xx = Unset for SERVER, Error for CLIENT) is the most load-bearing rule for error-rate dashboards. Filtering `StatusCode = 'Error'` on server spans correctly excludes client 4xx noise; the same filter on client spans correctly includes it. ### Retries/resends -Each client resend (redirect, auth failure, 5xx, network error — cause doesn't matter) increments `http.request.resend_count`, set to the ordinal number of the resend attempt. +Each client resend (redirect, auth failure, 5xx, network error; the cause doesn't matter) increments `http.request.resend_count`, which is set to the ordinal number of the resend attempt. ### Metrics (Stable) @@ -142,13 +159,13 @@ Each client resend (redirect, auth failure, 5xx, network error — cause doesn't When reported alongside a span, the duration metric value **SHOULD** equal the span's duration. -> **Maple hook:** our trace UI keys directly on `http.request.method` / `http.route` / `http.response.status_code`; the SERVER-4xx-is-Ok rule above is exactly what our error-rate widgets assume when they filter on span status. +> **Maple hook:** our trace UI keys directly on `http.request.method` / `http.route` / `http.response.status_code`. The SERVER-4xx-is-Unset rule above is what our error-rate widgets assume when they filter on span status. --- ## Database -**Stability: Stable** (as of the DB semconv stability migration; pre-migration instrumentations used `OTEL_SEMCONV_STABILITY_OPT_IN=database`/`database/dup` to move over — see General rules above). +**Stability: Stable** (as of the DB semconv stability migration; pre-migration instrumentations used `OTEL_SEMCONV_STABILITY_OPT_IN=database`/`database/dup` to move over, see General rules above). Source: https://opentelemetry.io/docs/specs/semconv/db/database-spans/ · https://opentelemetry.io/docs/specs/semconv/db/database-metrics/ · https://opentelemetry.io/docs/specs/semconv/non-normative/db-migration/ The stable migration renamed `db.system` → `db.system.name` (and `db.statement` → `db.query.text`), and changed many enum values to a `.` pattern. @@ -162,7 +179,7 @@ Priority order: 3. `{target}` alone 4. `{db.system.name}` as the final fallback -Where `{target}` prefers, in order: `db.collection.name` → `db.stored_procedure.name` → `db.namespace` → `server.address:server.port`. +`{target}` prefers, in order: `db.collection.name` → `db.stored_procedure.name` → `db.namespace` → `server.address:server.port`. ### Required / Conditionally Required attributes @@ -178,13 +195,13 @@ Where `{target}` prefers, in order: `db.collection.name` → `db.stored_procedur ### `db.query.text` sanitization -- Raw (non-parameterized) query text should only be collected **if sanitized to exclude sensitive info** — literals replaced with a placeholder (e.g. `?`). -- **"Parameterized query text SHOULD NOT be sanitized."** Using parameters is itself a strong signal from the application that sensitive values are passed out-of-band as parameters, not embedded in the query text. +- Raw (non-parameterized) query text should only be collected **if sanitized to exclude sensitive info**, with literals replaced by a placeholder (e.g. `?`). +- **"Parameterized query text SHOULD NOT be sanitized."** Using parameters is a strong signal from the application that sensitive values are passed out of band as parameters, not embedded in the query text. - IN-clauses may be collapsed for cardinality control (`IN (?, ?, ?, ?)` → `IN (?)`). ### Span status -Follows the general Recording Errors doc; each system-specific DB convention should specify which `db.response.status_code` values count as errors. +Follows the general Recording Errors doc. Each system-specific DB convention should specify which `db.response.status_code` values count as errors. ### Metrics @@ -196,18 +213,18 @@ Follows the general Recording Errors doc; each system-specific DB convention sho Batch operations **SHOULD** be recorded as a single operation (not one metric event per row/statement). -> **Maple hook:** warehouse read paths in `WarehouseQueryService` coalesce both the legacy (`db.statement`, `db.system`) and stable (`db.query.text`, `db.system.name`) spellings, since spans recorded before our own migration still carry the legacy names — mirrors exactly the dual-emit window the spec expects instrumentations to support. +> **Maple hook:** warehouse queries in `packages/query-engine/src/ch/queries/` (e.g. `service-map.ts`, `errors.ts`) coalesce the stable (`db.query.text`, `db.system.name`) and legacy (`db.statement`, `db.system`) spellings, because spans from older instrumentations still carry the legacy names. This mirrors the dual-emit window the spec expects instrumentations to support. --- ## Messaging -**Stability: Development** (not yet stable — per the spec, existing instrumentations on v1.24.0-era conventions should not change what they emit by default until this stabilizes). +**Stability: Development.** Per the spec, existing instrumentations on v1.24.0-era conventions should not change what they emit by default until this stabilizes. Source: https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/ ### Span name -`{messaging.operation.name} {destination}`, where `{destination}` prefers `messaging.destination.template` (low-cardinality template) → `messaging.destination.name` (for known, non-temporary destinations) → `server.address:server.port` (only when no specific destination is targeted). +`{messaging.operation.name} {destination}`. `{destination}` prefers `messaging.destination.template` (low-cardinality template), then `messaging.destination.name` (for known, non-temporary destinations), then `server.address:server.port` (only when no specific destination is targeted). ### Required / Conditionally Required attributes @@ -231,7 +248,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/ | `process` | CONSUMER | | `settle` | CLIENT | -Status follows the general Recording Errors doc; don't set `error.type` on success. +Status follows the general Recording Errors doc. Don't set `error.type` on success. --- @@ -242,7 +259,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/rpc/rpc-spans/ ### Span name -`{rpc.method}` if known and not `_OTHER`; else falls back to `{rpc.system.name}`. +`{rpc.method}` if known and not `_OTHER`; otherwise `{rpc.system.name}`. ### Required / Conditionally Required attributes @@ -261,10 +278,10 @@ Source: https://opentelemetry.io/docs/specs/semconv/rpc/rpc-spans/ **Stability: the span-event form is Deprecated; the log-record form is the current Stable recommendation.** Source: https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-spans/ · https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-logs/ -- Recording exceptions as **span events** (event name `exception`) is deprecated: "Use Semantic conventions for exceptions in logs instead." `exception.escaped` is likewise deprecated — "It's no longer recommended to record exceptions that are handled and do not escape the scope of a span." -- The current guidance: exceptions **SHOULD be recorded as attributes on the LogRecord** passed to `Logger` emit operations, not as span events. +- Recording exceptions as **span events** (event name `exception`) is deprecated: "Use Semantic conventions for exceptions in logs instead." `exception.escaped` is also deprecated: "It's no longer recommended to record exceptions that are handled and do not escape the scope of a span." +- Current guidance: exceptions **SHOULD be recorded as attributes on the LogRecord** passed to `Logger` emit operations, not as span events. - Migration is gated by `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN`: `logs` = emit as logs only; `logs/dup` = emit both span events and logs during a phased rollout; default = keep emitting span events. Even after an instrumentation moves to logs-only emission, "users will still have the option to route those to span events at the SDK layer." -- Attributes (same names/semantics in both forms): +- Attributes (same names and semantics in both forms): | Attribute | Level | Notes | | ---------------------- | -------------- | --------------------------------------------------------------------------- | @@ -272,7 +289,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-spans/ | `exception.message` | Cond. Required | Required if `exception.type` unset; may contain sensitive data | | `exception.stacktrace` | Recommended | Natural stack-trace string for the language runtime | -- Instrumentations **should not** record artificial/synthetic exceptions manufactured by a framework purely to represent an error status code. +- Instrumentations **should not** record synthetic exceptions that a framework manufactures purely to represent an error status code. --- @@ -289,7 +306,7 @@ Source: https://opentelemetry.io/docs/specs/semconv/registry/attributes/code/ · | `code.column.number` | Column in `code.file.path`, same "should point within" constraint | | `code.stacktrace` | Stack trace string in the language's natural representation; identical semantics to `exception.stacktrace` | -Note: as of the v1.29.0 → v1.33.0 migration, the older separate `code.namespace` and `code.function` attributes were folded into the single `code.function.name` (which now carries the fully-qualified name). All `code.*` attributes are disallowed on Profile signals (redundant with data profiling already captures). +Note: in the v1.29.0 → v1.33.0 migration, the older separate `code.namespace` and `code.function` attributes were folded into `code.function.name`, which now carries the fully-qualified name. All `code.*` attributes are disallowed on Profile signals, since profiling data already captures them. --- @@ -334,13 +351,13 @@ Guidance: "Consider always setting the transport when setting a port number, sin | `url.domain`, `url.port`, `url.registered_domain`, `url.subdomain`, `url.top_level_domain`, `url.extension` | Development | Derived sub-components parsed out of `url.full` | | `url.original` | Development | Unmodified original URL as seen at the source; may retain credentials (unlike `url.full`) | -> **Maple hook:** these are the backbone of the HTTP domain above — our trace UI and ingest gateway's "new-semconv HTTP attributes" are built directly on `server.address`/`server.port`/`network.protocol.*`/`url.*` alongside `http.*`. +> **Maple hook:** these underpin the HTTP domain above. Our trace UI and the ingest gateway's new-semconv HTTP attributes build on `server.address`/`server.port`/`network.protocol.*`/`url.*` alongside `http.*`. --- ## Gen-AI -**Stability: Development.** Note: gen-ai semantic conventions **moved out of the main spec** into a dedicated repository, `github.com/open-telemetry/semantic-conventions-genai` (the `opentelemetry.io/docs/specs/semconv/gen-ai/` pages now redirect/point there). +**Stability: Development.** Gen-AI semantic conventions **moved out of the main spec** into a dedicated repository, `github.com/open-telemetry/semantic-conventions-genai`. The `opentelemetry.io/docs/specs/semconv/gen-ai/` pages now redirect or point there. Source: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md ### Span name @@ -356,14 +373,14 @@ Source: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/d | `gen_ai.operation.name` | Required | e.g. `chat`, `embeddings`, `retrieval` | | `gen_ai.provider.name` | Required | Provider identifier, e.g. `openai`, `anthropic`, `gcp.vertex_ai` (supersedes the deprecated `gen_ai.system`) | | `gen_ai.request.model` | Cond. Required | If available | -| `gen_ai.response.model` | Cond. Required | If available; the actual model that served the response (may differ from the requested one) | -| `gen_ai.usage.input_tokens` | Recommended | — | -| `gen_ai.usage.output_tokens` | Recommended | — | +| `gen_ai.response.model` | Cond. Required | If available; the model that actually served the response (may differ from the requested one) | +| `gen_ai.usage.input_tokens` | Recommended | (none) | +| `gen_ai.usage.output_tokens` | Recommended | (none) | | `error.type` | Cond. Required | Should match the provider's/client library's error code | -Span kinds covered by the spec include Inference, Embeddings, Retrieval, Memory, and Execute-Tool spans. Migration is gated by `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` (opt into the newest experimental shape) vs. default (keep emitting whatever pre-1.36.0 shape was already in use) — the spec notes this transition plan will gain a genuine stable value once gen-ai conventions themselves stabilize. +The spec covers Inference, Embeddings, Retrieval, Memory, and Execute-Tool spans. Migration is gated by `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` (opt into the newest experimental shape) vs. the default (keep emitting whatever pre-1.36.0 shape was already in use). The spec notes this transition plan will gain a stable value once the gen-ai conventions stabilize. -> **Maple hook:** relevant to our AI features — treat `gen_ai.*` as Development/pre-stable, expect attribute churn (`gen_ai.system` → `gen_ai.provider.name` already happened), and don't hard-code assumptions about the current attribute set without checking the genai repo for the version in use. +> **Maple hook:** relevant to our AI features. Treat `gen_ai.*` as pre-stable and expect attribute churn (`gen_ai.system` → `gen_ai.provider.name` already happened). Check the genai repo for the version in use before hard-coding assumptions about the attribute set. --- diff --git a/docs/otel-spec/stability-and-compliance.md b/docs/otel-spec/stability-and-compliance.md index 2eddb2da8d..3d6864487f 100644 --- a/docs/otel-spec/stability-and-compliance.md +++ b/docs/otel-spec/stability-and-compliance.md @@ -1,54 +1,53 @@ # Versioning, Stability & Compliance -This document is the meta-spec reference: it covers how the OpenTelemetry -specification itself defines maturity/stability, what "stable" legally forbids -implementations from doing, the error-handling and performance principles -every SDK/instrumentation must follow, and how to read the cross-language -compliance matrix. Everything else in `docs/otel-spec/` (semconv, OTLP, etc.) -inherits its stability guarantees from the concepts defined here. +This is the meta-spec reference. It covers how the OpenTelemetry specification +defines maturity and stability, what "stable" forbids implementations from +doing, the error-handling and performance principles every SDK and +instrumentation must follow, and how to read the cross-language compliance +matrix. Everything else in `docs/otel-spec/` (semconv, OTLP, etc.) inherits its +stability guarantees from the concepts defined here. > **Spec version referenced:** OpenTelemetry Specification **v1.58.0** > (released 2026-06-22, per the > [specification CHANGELOG](https://github.com/open-telemetry/opentelemetry-specification/blob/main/CHANGELOG.md)). -> Stability language quoted below is current as of that release; re-check the +> Stability language quoted below is current as of that release. Re-check the > CHANGELOG's "Unreleased" section when bumping this doc. > [!NOTE] Relevance to Maple > Maple is primarily a **consumer/backend** of OTel data (Rust ingest gateway → -> collector → ClickHouse/Tinybird → web dashboard), plus a **self-instrumented -> emitter** of its own telemetry via `@effect/opentelemetry`. That means three -> of the four stability domains below (API, SDK, telemetry/semconv) matter to -> us mostly as _producers_ (our own services) and the fourth (OTLP wire format) -> matters as a _server implementor_ (`apps/ingest` accepting arbitrary -> upstream OTLP). We do not ship an OTel API/SDK for others to depend on, so -> the API/SDK LTS clauses are not obligations we owe — they're read as -> "what guarantees can we assume from the SDKs instrumenting the services that -> send us data." +> collector → ClickHouse/Tinybird → web dashboard). It is also a +> **self-instrumented emitter** of its own telemetry via `@maple-dev/effect-sdk` +> (`packages/effect-sdk`, built on Effect's native OTLP exporter in +> `effect/unstable/observability`). Three of the four stability domains below +> (API, SDK, telemetry/semconv) matter to us mostly as _producers_ (our own +> services). The fourth (OTLP wire format) matters to us as a _server +> implementor_ (`apps/ingest` accepting arbitrary upstream OTLP). We do not ship +> an OTel API/SDK for others to depend on, so the API/SDK LTS clauses are not +> obligations we owe. Read them as "what guarantees can we assume from the SDKs +> instrumenting the services that send us data." --- -## 1. Four stability domains, disentangled - -The spec deliberately separates stability into independent domains that -version and evolve on their own schedules. Conflating them is the most common -compliance mistake. - -| Domain | What it governs | Governing doc | Versioning | -| ------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -| **API stability** | Method signatures in `opentelemetry-api` packages (Tracer, Meter, Logger, Context, Propagators) | [Versioning and stability](https://opentelemetry.io/docs/specs/otel/versioning-and-stability/) | All stable API packages across all signals version together, one number, SemVer 2.0.0 | -| **SDK stability** | Public SDK surface: plugin interfaces (`SpanProcessor`, `Exporter`, `Sampler`) and constructors/config/builders | Same doc, "SDK" sections | SDK packages for all signals version together, independently from API | -| **Telemetry/semconv stability** | The _shape_ of emitted telemetry (span/metric/log names, attribute keys) that a stable instrumentation produces | [Telemetry Stability](https://opentelemetry.io/docs/specs/otel/telemetry-stability/) | Semantic Conventions have their own single version number, independent of API/SDK | -| **Wire (OTLP) stability** | The protobuf/JSON wire protocol between SDKs, collectors, and backends | OTLP protocol spec (see `docs/otel-spec` OTLP page if present) | Its own protocol version field, independent again | - -Each of these can be at a different maturity level simultaneously. For -example, the Trace **API** has been Stable since v1.0.0, but a given -**instrumentation library's telemetry** (e.g., HTTP semconv attribute names) -can still be in `Development`/`Alpha` even while riding on top of a fully -stable API and SDK. Maple's ingest gateway sees this directly: many spans -we accept still carry `http.method`/`net.peer.name` (old semconv) alongside -`http.request.method`/`server.address` (new semconv) because instrumentation -libraries move through telemetry-stability transitions independently of their -host language's API/SDK version. +## 1. Four stability domains + +The spec separates stability into independent domains that version and evolve +on their own schedules. Conflating them is the most common compliance mistake. + +| Domain | What it governs | Governing doc | Versioning | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | +| **API stability** | Method signatures in `opentelemetry-api` packages (Tracer, Meter, Logger, Context, Propagators) | [Versioning and stability](https://opentelemetry.io/docs/specs/otel/versioning-and-stability/) | All stable API packages across all signals version together, one number, SemVer 2.0.0 | +| **SDK stability** | Public SDK surface: plugin interfaces (`SpanProcessor`, `Exporter`, `Sampler`) and constructors/config/builders | Same doc, "SDK" sections | SDK packages for all signals version together, independently from API | +| **Telemetry/semconv stability** | The _shape_ of emitted telemetry (span/metric/log names, attribute keys) that a stable instrumentation produces | [Telemetry Stability](https://opentelemetry.io/docs/specs/otel/telemetry-stability/) | Semantic Conventions have their own single version number, independent of API/SDK | +| **Wire (OTLP) stability** | The protobuf/JSON wire protocol between SDKs, collectors, and backends | [OTLP spec](https://opentelemetry.io/docs/specs/otlp/); see [otlp.md](otlp.md) | Versioned by `opentelemetry-proto` releases; the `v1` proto packages and `/v1/*` paths mark the major version | + +Each domain can sit at a different maturity level at the same time. The Trace +**API** has been Stable since v1.0.0, but a given **instrumentation library's +telemetry** (e.g. HTTP semconv attribute names) can still be `Development` while +riding on a fully stable API and SDK. Maple's ingest gateway sees this directly. +Many spans we accept still carry `http.method`/`net.peer.name` (old semconv) +alongside `http.request.method`/`server.address` (new semconv), because +instrumentation libraries move through telemetry-stability transitions +independently of their host language's API/SDK version. Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/, https://opentelemetry.io/docs/specs/otel/telemetry-stability/ @@ -56,9 +55,8 @@ Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/, http ## 2. Signal lifecycle levels (spec-wide) -The specification's versioning-and-stability doc defines the lifecycle a -**signal** (Traces, Metrics, Logs, Profiles, Baggage) or a spec **feature** -moves through: +The versioning-and-stability doc defines the lifecycle a **signal** (Traces, +Metrics, Logs, Profiles, Baggage) or a spec **feature** moves through: | Level | Guarantee | Notes | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -71,9 +69,9 @@ Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/ ### Document-level status markers (separate taxonomy) -Individual **specification documents** (not signals) carry their own, -more granular maturity marker at the top of the page — this is the taxonomy -most relevant to reading any given spec page correctly: +Individual **specification documents** (not signals) carry their own, more +granular maturity marker at the top of the page. This is the taxonomy that +matters when reading any given spec page: | Status | Guarantee | Exact language | | --------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | @@ -87,20 +85,19 @@ most relevant to reading any given spec page correctly: | _(no marker)_ | Treated as **Alpha**. | Absence of a status is not "stable by default." | Documents whose sections carry differing statuses are labeled **"Mixed"** at -the top — always check the per-section status inline, not just the doc -banner, before treating any single paragraph as a stability guarantee. +the top. Check the per-section status inline, not just the doc banner, before +treating any single paragraph as a stability guarantee. Source: https://opentelemetry.io/docs/specs/otel/document-status/ **Practical rule for our compliance checks:** when citing a spec page as -justification for a design decision, always record the status banner (and, -for Mixed docs, the section-level status) alongside the citation — a -"Development"-status paragraph is not a compliance requirement, it's a -preview. +justification for a design decision, record the status banner (and, for Mixed +docs, the section-level status) with the citation. A "Development"-status +paragraph is a preview, not a compliance requirement. --- -## 3. What "Stable" forbids — breaking-change policy +## 3. What "Stable" forbids: breaking-change policy ### API packages @@ -110,43 +107,42 @@ preview. > "All existing API calls MUST continue to compile and function against all > future minor versions of the same major version." -Languages that ship binaries should additionally provide **ABI compatibility** -for API packages across minor versions. +Languages that ship binaries should also provide **ABI compatibility** for API +packages across minor versions. ### SDK packages > "Public portions of SDK packages MUST remain backward compatible." -"Public" is scoped to two categories: +"Public" covers two categories: -1. **Plugin interfaces** — `SpanProcessor`, `Exporter`, `Sampler`, and - equivalents. New methods may be _added_ to these interfaces without a - major bump only if the host language allows it in a backward-compatible - way (e.g., default interface methods). -2. **Constructors** — configuration objects, environment variables, builder +1. **Plugin interfaces**: `SpanProcessor`, `Exporter`, `Sampler`, and + equivalents. New methods may be _added_ to these interfaces without a major + bump only if the host language allows it in a backward-compatible way (e.g. + default interface methods). +2. **Constructors**: configuration objects, environment variables, builder APIs. ### Semantic conventions -Semconv defines a _breaking change_ as one that breaks "common usage of -tooling written against the telemetry it produces" — a narrower, more -practical bar than pure API compatibility. Two tiers: +Semconv defines a _breaking change_ as one that breaks "common usage of tooling +written against the telemetry it produces". That is a narrower, more practical +bar than pure API compatibility. Three tiers: - **Allowed only via a published schema file** (so old→new transforms are - mechanical): renaming span/metric/log/resource attributes; renaming - metrics and span events. + mechanical): renaming span/metric/log/resource attributes; renaming metrics + and span events. - **Always allowed, no schema needed**: adding new attributes to an existing - convention; adding entirely new conventions for resource/span/metric types - that didn't exist before. + convention; adding new conventions for resource/span/metric types that didn't + exist before. - **Prohibited outright**: anything else that would require inventing a new schema transform format. ### Deprecation process -A signal/feature can only be marked Deprecated once a Stable replacement -exists. Deprecated code retains full Stable-level support guarantees (no -extra breakage) until formally Removed, which itself requires a major version -bump. +A signal or feature can only be marked Deprecated once a Stable replacement +exists. Deprecated code keeps full Stable-level support guarantees until it is +formally Removed, which requires a major version bump. Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/, https://opentelemetry.io/docs/specs/otel/telemetry-stability/ @@ -157,15 +153,14 @@ Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/, http - Follows **Semantic Versioning 2.0.0**. - **Independent version tracks:** all stable API packages (across every signal) share one version number; all SDK packages share a separate one; - Semantic Conventions have their own single version number; contrib - packages version independently again. A major bump in one track does not - imply a bump in another. -- **Major bump** required for: any breaking change to a stable interface, or + Semantic Conventions have their own; contrib packages version independently. + A major bump in one track does not imply a bump in another. +- **Major bump** required for any breaking change to a stable interface, or removal of a deprecated signal. - **Minor bump**: backward-compatible additions, changes to Development-level - signals, or a signal's maturity transition (e.g., Development → Stable). + signals, or a signal's maturity transition (e.g. Development → Stable). - **Patch bump**: "Make no changes which would require recompilation or - potentially break application code" — bug fixes, security fixes, docs only. + potentially break application code". Bug fixes, security fixes, docs only. ### Long-term support (LTS) @@ -176,8 +171,7 @@ Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/, http | Contrib | 1 year | During the API LTS window, the latest SDK minor version keeps receiving -bug/security fixes, and contrib packages from that era keep receiving fixes -too. +bug and security fixes, and so do contrib packages from that era. Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/ @@ -185,12 +179,11 @@ Source: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/ ## 5. Error-handling principles (fail-safe requirements) -These are the rules that should drive our best-practices skill's "never let -telemetry take down the host app" checks — both for our own -`@effect/opentelemetry` self-instrumentation and for anything we tell users -to expect from well-behaved upstream SDKs sending us data. +These rules should drive our best-practices skill's "never let telemetry take +down the host app" checks. They apply to our own self-instrumentation and to +what we tell users to expect from well-behaved upstream SDKs. -- **No unhandled exceptions, ever, at runtime:** +- **No unhandled exceptions at runtime:** > "OpenTelemetry implementations MUST NOT throw unhandled exceptions at > runtime." / "API methods MUST NOT throw unhandled exceptions when used > incorrectly by end users." @@ -198,19 +191,19 @@ to expect from well-behaved upstream SDKs sending us data. safe defaults for missing or invalid arguments" instead of failing. - **Init-time failure is the one allowed exception:** libraries "MAY fail fast and cause the application to fail on initialization" but "MUST NOT - cause the application to fail later at runtime." (I.e., blow up at startup - if misconfigured — never mid-request.) + cause the application to fail later at runtime." Blow up at startup if + misconfigured, never mid-request. - **SDK internal errors are isolated:** SDKs "MUST NOT throw unhandled - exceptions for errors in their own operations" — e.g. an exporter's - connection to the collector dropping must not propagate into user code. + exceptions for errors in their own operations". An exporter losing its + connection to the collector must not propagate into user code. - **Callback safety:** "API methods that accept external callbacks MUST handle all errors" raised by those callbacks. -- **No-op over null:** on suppressed error, implementations "MUST return a - 'no-op' or any other 'default' object" rather than `null`/an exception. +- **No-op over null:** on a suppressed error, implementations "MUST return a + 'no-op' or any other 'default' object" rather than `null` or an exception. - **Self-diagnostics instead of silent failure:** "Whenever the library suppresses an error that would otherwise have been exposed to the user, the library SHOULD log the error using language-specific conventions." - Libraries are further "encouraged to expose self-troubleshooting metrics, + Libraries are also "encouraged to expose self-troubleshooting metrics, spans, and other telemetry that can be easily enabled and filtered out by default." - **User override:** "SDK implementations MUST allow end users to change the @@ -220,11 +213,11 @@ Source: https://opentelemetry.io/docs/specs/otel/error-handling/ **Best-practice-skill translation:** any span/metric/log emission path we write (in `apps/api`, `apps/ingest`, or client SDKs we recommend) must never -be able to throw past its call site into request-handling code; it should -swallow-and-self-log instead. This is exactly the shape of our own -`withTracerDisabledWhen` / OTLP-export-is-async design already documented in -CLAUDE.md's Self-Observability section — that design _is_ spec compliance -with error-handling, not just an internal safety measure. +throw past its call site into request-handling code. It should swallow and +self-log instead. Our own async OTLP export and the `HttpMiddleware.TracerDisabledWhen` +filter (`apps/api/src/http/api-observability.ts`) already have this shape. That +design is spec compliance with error handling, not just an internal safety +measure. --- @@ -233,60 +226,59 @@ with error-handling, not just an internal safety measure. - **Non-blocking by default:** "Library should not block end user application by default." - **Bounded memory:** "Library should not consume unbounded memory - resource." The spec explicitly frames overhead as a trade-off the - implementation must actively manage, not ignore: instrumentation "should - not degrade the end user application as possible." -- **Under load, choose consciously between two failure modes:** - 1. _Preserve everything, risk memory pressure_ — "Preserve all information + resource." The spec frames overhead as a trade-off the implementation must + actively manage: instrumentation "should not degrade the end user + application as possible." +- **Under load, choose between two failure modes:** + 1. _Preserve everything, risk memory pressure_: "Preserve all information but possible to consume many resources", or - 2. _Bound memory, drop data_ — "Dropping some information under + 2. _Bound memory, drop data_: "Dropping some information under overwhelming load and show warning log to inform when information loss starts and when recovered." This mode should have configurable thresholds and ideally expose a metric approximating the effective sampling ratio caused by the drops. - **Logs need their own filter valve:** "Logging could consume much memory - by default if the end user application emits too many logs" — the spec - says implementations should "provide a way to filter logs to capture by + by default if the end user application emits too many logs". The spec says + implementations should "provide a way to filter logs to capture by OpenTelemetry" independent of the application's own logging volume. - **Shutdown/flush must be boundedly blocking:** "The OpenTelemetry client could block the end user application when it shut down." Both `shutdown()` and explicit `flush()` should "support user-configurable timeout." -No concrete numeric overhead/allocation targets are specified — this is -qualitative guidance only, left to each SDK's own benchmarking. +The spec sets no numeric overhead or allocation targets. This is qualitative +guidance, left to each SDK's own benchmarking. Source: https://opentelemetry.io/docs/specs/otel/performance/ -**Maple-specific read:** `apps/ingest`'s WAL + async OTLP forward and its -startup loopback guard are the kind of bounded-blocking / configurable-drop -behavior this section calls for. Any new self-instrumentation on hot paths -(e.g. per-request auth token validation, per-span-batch processing) should be -checked against "does this block the request" and "is memory bounded under -burst," not just "does it produce useful telemetry." +**Maple-specific read:** `apps/ingest`'s WAL, async OTLP forward, and startup +loopback guard are the bounded-blocking, configurable-drop behavior this +section calls for. Check any new self-instrumentation on hot paths (e.g. +per-request auth token validation, per-span-batch processing) against "does +this block the request" and "is memory bounded under burst," not just "does it +produce useful telemetry." --- ## 7. Library / instrumentation guidelines -Two source documents cover this, one on general client design principles and -one specifically on native instrumentation practice. +Two source documents cover this: one on general client design principles, one +on native instrumentation practice. ### General client design (spec: library-guidelines) - **Depend on API only, never SDK:** "Libraries, frameworks, and applications that want to be instrumented with OpenTelemetry take a dependency only on - the API packages." This is what lets a library be instrumented without - forcing a specific backend on its consumers. -- **Negligible overhead is a design constraint, not an afterthought:** "It is - also important that minimal implementation incurs as little performance - penalty as possible, so that third-party frameworks and libraries that are - instrumented with OpenTelemetry impose negligible overheads..." -- **Exporter package naming:** separately-published exporters should be - named with the `opentelemetry-exporter-{vendor_name}` pattern (prefixed - with "OpenTelemetry" and "Exporter"). -- This document explicitly does _not_ cover span-granularity or attribute - hygiene — it defers to other spec sections (semantic conventions, API spec) - for that level of detail. + the API packages." This lets a library be instrumented without forcing a + specific backend on its consumers. +- **Negligible overhead is a design constraint:** "It is also important that + minimal implementation incurs as little performance penalty as possible, so + that third-party frameworks and libraries that are instrumented with + OpenTelemetry impose negligible overheads..." +- **Exporter package naming:** separately published exporters should follow + the `opentelemetry-exporter-{vendor_name}` pattern (prefixed with + "OpenTelemetry" and "Exporter"). +- This document does _not_ cover span granularity or attribute hygiene. It + defers to other spec sections (semantic conventions, API spec) for that. Source: https://opentelemetry.io/docs/specs/otel/library-guidelines/ @@ -303,108 +295,106 @@ Source: https://opentelemetry.io/docs/specs/otel/library-guidelines/ version: they show up on the telemetry and help users process and filter telemetry." Example: `getTracer("demo-db-client", "0.1.0-beta1")`. - **Pin to the earliest stable API:** "Use the earliest stable OpenTelemetry - API (1.0.\*) and avoid updating it unless you have to use new features" — - minimizes forced version churn for consumers of the instrumented library. + API (1.0.\*) and avoid updating it unless you have to use new features". + This minimizes forced version churn for consumers of the instrumented + library. - **Register it:** "Add your instrumentation library to the OpenTelemetry registry so users can find it." -### Instrumentation Scope — identity and naming (concepts: instrumentation-scope) +### Instrumentation Scope: identity and naming (concepts: instrumentation-scope) An Instrumentation Scope is "a logical unit of software with which emitted telemetry is associated. It can represent a module, package, class, library, -or framework." It is identified by a `(name, version, schema_url, -attributes)` tuple — only `name` is required, and "the `name` should uniquely -identify the logical unit of software — for example, the fully qualified name -of a library, class, or module." +or framework." It is identified by a `(name, version, schema_url, attributes)` +tuple. Only `name` is required, and "the `name` should uniquely identify the +logical unit of software, for example, the fully qualified name of a library, +class, or module." Naming guidance by situation: | Situation | Convention | Example | | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -| Library/framework with native instrumentation | Library's own fully qualified name + version | — | +| Library/framework with native instrumentation | Library's own fully qualified name + version | n/a | | Third-party library, no native support (external instrumentation library) | Fully qualified name/version of the _instrumentation_ library itself, often reverse-DNS | `io.opentelemetry.contrib.mongodb`, `io.opentelemetry.instrumentation.flask` | | OpenTelemetry-hosted contrib instrumentation | `opentelemetry-instrumentation-` package-name prefix | `opentelemetry-instrumentation-flask` | | Application-level code (not a library) | Class or module name | `CheckoutService` | -Every span/metric/log record produced by a given tracer/meter/logger -instance is tagged with that instance's scope — this is what lets backends -group/filter telemetry by originating component and compare across library -versions. +Every span, metric, and log record produced by a given tracer/meter/logger is +tagged with that instance's scope. This lets backends group and filter +telemetry by originating component and compare across library versions. Source: https://opentelemetry.io/docs/specs/otel/library-guidelines/, https://opentelemetry.io/docs/concepts/instrumentation/libraries/, https://opentelemetry.io/docs/concepts/instrumentation-scope/ -**Maple-specific read:** our own service names already follow the spec's -identity model at the _Resource_ level (`service.name="ingest"`, -`service.version`, `service.instance.id` — see CLAUDE.md's -Self-Observability section). Instrumentation Scope is the finer-grained -sibling of that: if we ever split internal tracer usage across modules -inside `apps/api` (e.g. a distinct tracer for the alerting service vs. the -query engine), each should get its own scope name (e.g. -`maple.alerting`, `maple.query-engine`) and version, not one blanket -service-wide tracer — so dashboard users can filter by component the same +**Maple-specific read:** our services already follow the spec's identity model +at the _Resource_ level (`service.name="ingest"`, `service.version`, +`service.instance.id`). Instrumentation Scope is the finer-grained sibling. If +we split internal tracer usage across modules inside `apps/api` (e.g. a +distinct tracer for alerting vs. the query engine), each should get its own +scope name (e.g. `maple.alerting`, `maple.query-engine`) and version instead of +one service-wide tracer. Dashboard users can then filter by component the same way they filter by service. --- ## 8. Telemetry stability guarantees (semconv-adjacent, distinct from API/SDK) -This document governs **the shape of telemetry emitted by instrumentations** -specifically — a different axis from API/SDK code stability. +This document governs **the shape of telemetry emitted by instrumentations**, +a different axis from API/SDK code stability. -- **Unstable instrumentations:** no guarantees at all. "Unstable - instrumentations provide no guarantees about the shape of the telemetry - they produce and how that shape changes over time." +- **Unstable instrumentations:** no guarantees. "Unstable instrumentations + provide no guarantees about the shape of the telemetry they produce and how + that shape changes over time." - **Stable instrumentations** split into two sub-cases: - - **Fixed-schema producers** (stable, but no Schema URL attached to their - telemetry): "Such instrumentations are prohibited from changing any - produced telemetry" — full stop, even to adopt a newer semconv release, - unless they migrate to schema-file-driven status. - - **Schema-file-driven producers** (stable, Schema URL attached): as of the - fetched page, this path is **under moratorium** — currently held to the - same no-change restriction as fixed-schema producers. Once the - moratorium lifts, changes become allowed only if they (a) match a - released OTel semconv version, (b) ship a corresponding published schema - file, and (c) correctly update the Schema URL. -- **Universally allowed regardless of tier:** "Adding of new metrics, spans, - span events or log records and adding of new attributes." + - **Fixed-schema producers** (stable, no Schema URL attached): "Such + instrumentations are prohibited from changing any produced telemetry", + even to adopt a newer semconv release, unless they migrate to + schema-file-driven status. + - **Schema-file-driven producers** (stable, Schema URL attached): as of + the fetched page, this path is **under moratorium** and held to the same + no-change restriction as fixed-schema producers. Once the moratorium + lifts, changes are allowed only if they (a) match a released OTel semconv + version, (b) ship a corresponding published schema file, and (c) + correctly update the Schema URL. +- **Allowed regardless of tier:** "Adding of new metrics, spans, span events + or log records and adding of new attributes." Source: https://opentelemetry.io/docs/specs/otel/telemetry-stability/ -**Why this matters for our ingest gateway specifically:** because -telemetry-shape stability is decoupled from API/SDK stability, a -long-since-1.0 SDK can still be emitting spans whose attribute names are -mid-migration (old vs. new HTTP semconv, for instance). Maple's ingest path -and ClickHouse materialized views (`service_overview_spans_mv`, etc.) already -coalesce legacy and current attribute spellings for this exact reason — see -CLAUDE.md's note on dual-emitting `deployment.environment.name`/ -`deployment.environment`. That coalescing logic is not a hack; it's the -correct way to consume telemetry from instrumentations that are Stable at the -API level but still transitioning at the telemetry-shape level. +**Why this matters for our ingest gateway:** telemetry-shape stability is +decoupled from API/SDK stability, so a long-since-1.0 SDK can still emit spans +whose attribute names are mid-migration (old vs. new HTTP semconv, for +instance). Maple's ClickHouse materialized views (`service_overview_spans_mv`, +etc.) coalesce legacy and current attribute spellings for this reason, e.g. +`deployment.environment.name` / `deployment.environment` (see +`packages/domain/src/tinybird/semconv-renames.ts` and +[attributes.md](../attributes.md)). That coalescing is the correct way to +consume telemetry from instrumentations that are Stable at the API level but +still transitioning at the telemetry-shape level. ### Schema URL / Telemetry Schema mechanics -A **Schema URL** identifies a **Schema File** — a YAML document describing -one version of a Schema Family, plus the transforms needed to convert older +A **Schema URL** identifies a **Schema File**: a YAML document describing one +version of a Schema Family, plus the transforms needed to convert older compatible telemetry in that family up to this version. Mechanically: -- The URL's last path segment is the schema version; everything before it is - the **Schema Family identifier** — all schema versions in one family share - that prefix. +- The URL's last path segment is the schema version. Everything before it is + the **Schema Family identifier**, which all schema versions in one family + share. - Fetchers **must** follow HTTP redirects when resolving a Schema URL. - In OTLP, `schema_url` on `ResourceSpans`/`ResourceMetrics`/`ResourceLogs` - applies to the contained `Resource` and its spans/metrics/logs; a + applies to the contained `Resource` and its spans/metrics/logs. A scope-level `schema_url` (on `ScopeSpans`/`ScopeMetrics`/`ScopeLogs`, the - successor to the deprecated `InstrumentationLibrarySpans` etc.) applies - only to the contained telemetry items for that scope. -- The whole mechanism exists for one narrow purpose: "to allow OpenTelemetry - Semantic Conventions to evolve over time" without breaking consumers who - pin to an older schema version. + successor to the deprecated `InstrumentationLibrarySpans` etc.) applies only + to the telemetry items for that scope. +- The mechanism exists for one narrow purpose: "to allow OpenTelemetry + Semantic Conventions to evolve over time" without breaking consumers who pin + to an older schema version. Source: https://opentelemetry.io/docs/specs/otel/schemas/, https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/schemas/README.md --- -## 9. Glossary — terms load-bearing for this doc +## 9. Glossary: terms load-bearing for this doc | Term | Definition | Source | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | @@ -418,67 +408,63 @@ Source: https://opentelemetry.io/docs/specs/otel/schemas/, https://github.com/op | **Schema URL** | Identifier for a Schema File; last path segment is the version, the prefix is the Schema Family identifier; resolution must follow redirects. | [Telemetry Schemas](https://opentelemetry.io/docs/specs/otel/schemas/) | | **Telemetry Schema** | "The expected shape and composition of emitted telemetry data," versioned so Semantic Conventions can evolve without breaking pinned consumers. | [Telemetry Schemas](https://opentelemetry.io/docs/specs/otel/schemas/) | -Note: the canonical glossary page does not itself define Resource, Baggage, -Sampler, or Context Propagation with standalone entries — those are defined -in their respective spec sections (Resource SDK spec, Baggage API spec, -Trace SDK spec, Context API spec) rather than the glossary. Treat the -glossary as authoritative only for the terms actually listed on it. +The glossary page has no standalone entries for Resource, Baggage, Sampler, or +Context Propagation. Those are defined in their own spec sections (Resource SDK, +Baggage API, Trace SDK, Context API). Treat the glossary as authoritative only +for the terms it lists. Source: https://opentelemetry.io/docs/specs/otel/glossary/ --- -## 10. The spec compliance matrix — how to read it +## 10. The spec compliance matrix: how to read it The living source of truth for "does implementation X support feature Y" is: **https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md** -Structure (do not copy the matrix itself — it changes frequently; always -link and re-fetch): +Structure (do not copy the matrix itself; it changes often, so link and +re-fetch): -- **Rows** are individual spec requirements/features, grouped under section - headers for the major component areas: - 1. **Traces** — TracerProvider ops, context interaction, Tracer ops, +- **Rows** are individual spec requirements, grouped under section headers + for the major component areas: + 1. **Traces**: TracerProvider ops, context interaction, Tracer ops, SpanContext, span creation/lifecycle, attributes, links, events, exceptions, sampling, ID generation. - 2. **Baggage** — basic support, header naming. - 3. **Metrics** — MeterProvider, Meter ops, instrument types, Views, + 2. **Baggage**: basic support, header naming. + 3. **Metrics**: MeterProvider, Meter ops, instrument types, Views, aggregations, exemplars, cardinality limits. - 4. **Logs** — LoggerProvider, Logger ops, LogRecord handling, processors. - 5. **Resource** — creation, merging, detection. - 6. **Context Propagation** — Context management, composite propagators, + 4. **Logs**: LoggerProvider, Logger ops, LogRecord handling, processors. + 5. **Resource**: creation, merging, detection. + 6. **Context Propagation**: Context management, composite propagators, standard propagators (TraceContext, B3, Jaeger, OpenCensus). - 7. **Environment Variables** — `OTEL_*` configuration knobs. - 8. **Declarative Configuration** — YAML-based SDK setup. - 9. **Exporters** — stdout, in-memory, OTLP, Zipkin, Prometheus + 7. **Environment Variables**: `OTEL_*` configuration knobs. + 8. **Declarative Configuration**: YAML-based SDK setup. + 9. **Exporters**: stdout, in-memory, OTLP, Zipkin, Prometheus compatibility. - **Columns** are per-language implementations: Go, Java, JavaScript, Python, Ruby, Erlang, PHP, **Rust**, C++, .NET, Swift, Kotlin. - **Legend:** `+` supported, `-` unsupported, `N/A` not applicable, blank = - unknown/unreported. An **Optional** column marks features with `X` when a - feature is optional rather than required — unmarked rows are required for - a language to claim compliance. - -**Caveats relevant to Maple's stack:** - -- **Rust** has its own column (relevant to `apps/ingest`) — but note the - matrix tracks _SDK/client_ conformance (an implementation emitting - telemetry), not _server/collector_ conformance (accepting OTLP). Our - ingest gateway is architecturally closer to an OTLP _receiver_ than to any - row in this matrix — the matrix doesn't directly grade us, but it's the - right place to check what guarantees we can assume from Rust-instrumented - services sending us data. -- **JavaScript** is the column that governs `@effect/opentelemetry`'s - underlying `@opentelemetry/sdk-*` dependencies (apps/web, apps/api on - Workers). There is no separate "Bun" column — Bun is a JS runtime, so it - inherits the JavaScript column's conformance status; any Bun-specific gaps - (e.g. Workers-runtime clock/timer quirks — see the `workerd-trace-timestamp-bug` - memory entry) are Bun/workerd runtime bugs, not spec non-compliance, and - won't show up in this matrix. -- The matrix is **self-reported per language SIG** and updated - asynchronously from actual releases — treat a blank cell as "unverified," - not "unsupported." + unknown/unreported. An **Optional** column marks optional features with `X`. + Unmarked rows are required for a language to claim compliance. + +**Caveats for Maple's stack:** + +- **Rust** has its own column (relevant to `apps/ingest`). The matrix tracks + _SDK/client_ conformance (an implementation emitting telemetry), not + _server/collector_ conformance (accepting OTLP). Our ingest gateway is closer + to an OTLP _receiver_ than to any row in this matrix, so the matrix doesn't + grade us directly. It is the right place to check what we can assume from + Rust-instrumented services sending us data. +- **JavaScript** governs the `@opentelemetry/*` packages under our browser SDK + (`@maple-dev/browser`) and any JS service sending us data. Our server-side + self-instrumentation uses Effect's own OTLP exporter, not the JS OTel SDK. + There is no separate Bun or workerd column: they are JS runtimes and inherit + the JavaScript column. Runtime-specific gaps (e.g. Workers-runtime clock and + timer quirks) are runtime bugs, not spec non-compliance, and won't show up in + this matrix. +- The matrix is **self-reported per language SIG** and updated asynchronously + from releases. Treat a blank cell as "unverified," not "unsupported." Source: https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md @@ -490,72 +476,72 @@ Source: https://github.com/open-telemetry/opentelemetry-specification/blob/main/ SDK, so most of Sections 1–4 (API/SDK versioning obligations) don't bind us directly. What does bind us: -- **Error-handling principles (Section 5) apply symmetrically to servers.** - A malformed/oversized/undecodable payload from a misbehaving client must - never crash the gateway — this is already implemented via the documented - 4xx-vs-5xx `otel_status_for_rejection` split (only 5xx sets span status to - `Error`; auth/billing/throttle/payload 4xx rejections stay `Ok` but fully - observable via `http.response.status_code`/`error.type`). That design _is_ - the fail-safe-defaults principle applied to a receiver role. +- **Error-handling principles (Section 5) apply to servers too.** A + malformed, oversized, or undecodable payload from a misbehaving client must + never crash the gateway. `otel_status_for_rejection` in + `apps/ingest/src/main.rs` sets the server span status: 5xx, and 4xx + rejections that drop the caller's telemetry (`rejection_loses_data` in + `apps/ingest/src/otel.rs`: `bad_request`, `decode`, `enrich`), are `Error`. + Rejections of the caller itself (bad key, throttle, oversized payload) stay + `Ok` and remain observable via `http.response.status_code`/`error.type`. + That is the fail-safe-defaults principle applied to a receiver. - **Performance/blocking guidance (Section 6)** governs the WAL + async - forward design and the startup loopback guard — bounded memory and - non-blocking behavior under load are exactly what the spec calls out. + forward design and the startup loopback guard. Bounded memory and + non-blocking behavior under load are what the spec calls for. - **Telemetry-shape stability (Section 8)** is why we must tolerate mixed - old/new semantic conventions in inbound payloads indefinitely — clients - are never obligated to be on the latest semconv, and "Stable" telemetry + old/new semantic conventions in inbound payloads indefinitely. Clients are + never obligated to be on the latest semconv, and "Stable" telemetry producers are largely frozen at whatever shape they shipped. Schema-URL - coalescing (Section 8's mechanism) is the spec-sanctioned way to handle - this, and is exactly what our materialized views already do by convention - even though we don't yet key off literal `schema_url` values. + transforms are the spec-sanctioned way to handle this. Our materialized + views reach the same result by coalescing spellings, without keying off + literal `schema_url` values. **(b) Our self-instrumentation.** When `apps/ingest`, `apps/api`, or any new service adds tracing: -- Use **Instrumentation Scope naming** (Section 7) deliberately — one scope +- Use **Instrumentation Scope naming** (Section 7) deliberately: one scope per logical component (e.g. `maple.ingest`, `maple.alerting`), not one - global tracer, so the scope tuple is actually useful for filtering. -- New spans on hot/high-frequency paths must be justified against the - performance guidance (Section 6) — non-blocking, bounded memory, and ideally - covered by the existing `withTracerDisabledWhen` pattern for noisy paths. - CLAUDE.md's explicit warning not to remove that filter, and to be careful - adding spans to auth-token validation, is a direct application of Section 6. -- Any place we might swallow/re-throw telemetry-related errors should follow - Section 5 exactly: log-and-continue, never propagate into request handling, - never let OTLP export failures affect the response to the end user (already - true today since export is async and bypasses the API entirely per - CLAUDE.md). + global tracer, so the scope tuple is useful for filtering. +- Justify new spans on hot or high-frequency paths against the performance + guidance (Section 6): non-blocking, bounded memory, and where it fits, + excluded by the `HttpMiddleware.TracerDisabledWhen` filter. The "Self-tracing" + rule in CLAUDE.md (keep that filter and avoid spans on token validation) is a direct + application of Section 6. +- Any place that swallows or re-throws telemetry-related errors should follow + Section 5: log and continue, never propagate into request handling, never + let OTLP export failures affect the response (already true today, since + export is async). **(c) The UI's interpretation of data.** The dashboard trusts specific semconv-adjacent contracts as if they were part of the wire stability -guarantee (Section 1's fourth domain) — e.g., **span status codes** should be -title case (`"Ok"`, `"Error"`, `"Unset"`) per Maple's own data convention, and -error-rate dashboards filter strictly on `StatusCode='Error'`. Per Section 5's -"only 5xx is Error" SERVER-span rule, the UI's error dashboards are implicitly -depending on upstream instrumentations following that exact OTEL HTTP -semconv status-mapping rule. If an upstream SDK sets status to `Error` for a -4xx (a spec violation, or simply an older/nonconforming instrumentation), our -dashboards will over-count errors — this is a real spec-compliance risk -surface worth a validation/lint pass on ingest, not just a UI concern. More -generally: any dashboard heuristic that assumes "stable telemetry never +guarantee (Section 1's fourth domain). **Span status codes** must be Title Case +(`"Ok"`, `"Error"`, `"Unset"`) per Maple's data convention, and error-rate +dashboards filter strictly on `StatusCode='Error'`. The HTTP semconv rule that +only 5xx is `Error` on server spans means our error dashboards depend on +upstream instrumentations following that status mapping. If an upstream SDK +sets `Error` for a 4xx (a spec violation, or an older or nonconforming +instrumentation), our dashboards over-count errors. This is a real +spec-compliance risk that deserves a validation pass on ingest, not just a UI +fix. More generally, a dashboard heuristic that assumes "stable telemetry never changes shape" (Section 8) is safe **only** for genuinely Stable, fixed-schema -producers — Alpha/Beta/Development-status instrumentations (the document -status ladder in Section 2) can and will change attribute names under us -without notice, and the UI/query layer should degrade gracefully (missing -attribute → omit facet, not crash) rather than assume presence. +producers. Alpha, Beta, and Development instrumentations (the document status +ladder in Section 2) can change attribute names without notice. The UI and +query layer should degrade gracefully (missing attribute → omit facet, not +crash) instead of assuming presence. --- ## Key references -- Versioning and stability — https://opentelemetry.io/docs/specs/otel/versioning-and-stability/ -- Document status (Development/Alpha/Beta/RC/Stable/Deprecated/Unmaintained) — https://opentelemetry.io/docs/specs/otel/document-status/ -- Telemetry stability guarantees — https://opentelemetry.io/docs/specs/otel/telemetry-stability/ -- Error handling principles — https://opentelemetry.io/docs/specs/otel/error-handling/ -- Performance and blocking guidance — https://opentelemetry.io/docs/specs/otel/performance/ -- Library guidelines (client design principles) — https://opentelemetry.io/docs/specs/otel/library-guidelines/ -- Native instrumentation for libraries — https://opentelemetry.io/docs/concepts/instrumentation/libraries/ -- Instrumentation Scope concept — https://opentelemetry.io/docs/concepts/instrumentation-scope/ -- Telemetry Schemas — https://opentelemetry.io/docs/specs/otel/schemas/ and https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/schemas/README.md -- Glossary — https://opentelemetry.io/docs/specs/otel/glossary/ -- Spec compliance matrix (living source, re-fetch before relying on cell values) — https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md -- Specification CHANGELOG (version history) — https://github.com/open-telemetry/opentelemetry-specification/blob/main/CHANGELOG.md +- Versioning and stability: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/ +- Document status (Development/Alpha/Beta/RC/Stable/Deprecated/Unmaintained): https://opentelemetry.io/docs/specs/otel/document-status/ +- Telemetry stability guarantees: https://opentelemetry.io/docs/specs/otel/telemetry-stability/ +- Error handling principles: https://opentelemetry.io/docs/specs/otel/error-handling/ +- Performance and blocking guidance: https://opentelemetry.io/docs/specs/otel/performance/ +- Library guidelines (client design principles): https://opentelemetry.io/docs/specs/otel/library-guidelines/ +- Native instrumentation for libraries: https://opentelemetry.io/docs/concepts/instrumentation/libraries/ +- Instrumentation Scope concept: https://opentelemetry.io/docs/concepts/instrumentation-scope/ +- Telemetry Schemas: https://opentelemetry.io/docs/specs/otel/schemas/ and https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/schemas/README.md +- Glossary: https://opentelemetry.io/docs/specs/otel/glossary/ +- Spec compliance matrix (living source, re-fetch before relying on cell values): https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md +- Specification CHANGELOG (version history): https://github.com/open-telemetry/opentelemetry-specification/blob/main/CHANGELOG.md diff --git a/docs/otel-spec/traces.md b/docs/otel-spec/traces.md index d08b5792bf..1eeda87405 100644 --- a/docs/otel-spec/traces.md +++ b/docs/otel-spec/traces.md @@ -3,40 +3,43 @@ This page documents the OpenTelemetry **Tracing** specification: the Trace API (`TracerProvider`, `Tracer`, `Span`, `SpanContext`), the Trace SDK (samplers, span processors, span limits, ID generators, the `SpanExporter` interface), and the normative rules a spec-compliant tracing -implementation (or a backend that consumes its output) must honor. Every claim below is sourced -from the official specification — see inline `Source:` links — and each section states the -stability level the spec itself declares (Stable / Development / Deprecated), because normative -strength differs materially by level (only Stable sections are locked against breaking change). +implementation (or a backend that consumes its output) must honor. Every claim is sourced from the +official specification via inline `Source:` links. Each section states the stability level the spec +itself declares (Stable / Development / Deprecated). Normative strength differs by level: only +Stable sections are locked against breaking change. ## Relevance to Maple -> Maple is primarily a **consumer/backend** of OTel trace data (Rust `apps/ingest` OTLP gateway → -> collector → ClickHouse/Tinybird → web dashboard), plus a **self-instrumented emitter** of its own -> traces (`@effect/opentelemetry` in the API/alerting/chat-agent workers, and `apps/ingest`'s own -> Rust OTLP self-instrumentation — see CLAUDE.md "Self-Observability"). Both angles are why this -> spec matters: +> Maple is mainly a **consumer/backend** of OTel trace data (Rust `apps/ingest` OTLP gateway → +> collector → ClickHouse/Tinybird → web dashboard). It is also a **self-instrumented emitter** of its +> own traces: `@maple-dev/effect-sdk` (`packages/effect-sdk`) in the api, ai and alerting workers, +> and `apps/ingest`'s own Rust OTLP self-instrumentation (`apps/ingest/src/otel.rs`, +> `service.name = "ingest"`). Both angles make this spec matter. > > - **As a consumer/backend:** we must defensively parse whatever a compliant (or non-compliant) -> SDK emits. Key invariants to enforce or tolerate: `TraceId`/`SpanId` validity (all-zero = invalid, -> §[SpanContext](#spancontext)), the `Sampled`/`IsRecording` combination that must never reach us -> (`Sampled=true, IsRecording=false` is spec-forbidden — §[Sampling](#sampling-sdk)), `StatusCode` -> ordering (`Ok > Error > Unset`, and `Description` is only meaningful on `Error` — -> §[Status](#set-status)), and the 5 `SpanKind` values that drive our service-map/flamegraph -> parent-child inference (§[SpanKind](#spankind)). Maple stores span status as title-case -> `"Ok"`/`"Error"`/`"Unset"` strings (see CLAUDE.md "Data Conventions"), which matches the spec's -> enum spelling directly — but the OTLP wire encoding uses a different, uppercase enum -> (`STATUS_CODE_OK` / `STATUS_CODE_ERROR` / `STATUS_CODE_UNSET`); our ingest/collector mapping -> layer is the one place that translation must stay correct. +> SDK emits. Invariants to enforce or tolerate: +> - `TraceId`/`SpanId` validity: all-zero is invalid (§[SpanContext](#spancontext)). +> - `Sampled=true, IsRecording=false` is spec-forbidden and should never reach us +> (§[Sampling](#sampling-sdk)). +> - `StatusCode` ordering is `Ok > Error > Unset`, and `Description` only has meaning on +> `Error` (§[Status](#set-status)). +> - The 5 `SpanKind` values drive our service-map/flamegraph parent-child inference +> (§[SpanKind](#spankind)). +> +> Maple stores span status as title-case `"Ok"`/`"Error"`/`"Unset"` strings (CLAUDE.md, +> "Span status codes"). That matches the spec's enum spelling. The OTLP wire encoding uses a +> different uppercase enum (`STATUS_CODE_OK` / `STATUS_CODE_ERROR` / `STATUS_CODE_UNSET`); the +> ingest/collector mapping layer is the one place that translation must stay correct. > - **As a self-instrumented emitter:** our own services configure `BatchSpanProcessor` knobs, > samplers (`OTEL_TRACES_SAMPLER` / `_ARG`), and span limits via the env vars in > §[Span Limits](#span-limits) and §[Span Processor](#span-processor-sdk). The -> `withTracerDisabledWhen` health-check filter and the `workerd` monotonic-clock timestamp bug -> (see MEMORY.md) are both examples of us needing to reason precisely about `Span` start/end -> timestamp semantics (§[Span](#span)) and `OnStart`/`OnEnd` processor timing +> `HttpMiddleware.TracerDisabledWhen` health-check filter (`apps/api/src/http/api-observability.ts`) +> and the `workerd` monotonic-clock timestamp bug both require precise reasoning about `Span` +> start/end timestamp semantics (§[Span](#span)) and `OnStart`/`OnEnd` processor timing > (§[Span Processor](#span-processor-sdk)). > - **Links added after span start** (§[Add Link](#add-link)) and **attributes preferred at -> creation** (§[Span Creation](#span-creation)) matter for our own instrumentation code review: -> sampling can only ever see what was present at span-start time. +> creation** (§[Span Creation](#span-creation)) matter when reviewing our own instrumentation. +> Sampling only sees what was present at span start. --- @@ -49,15 +52,14 @@ Both the Trace API spec and the Trace SDK spec carry the document-level banner: > "**Status**: Stable, except where otherwise specified" -This means: unless a specific subsection is explicitly marked otherwise, treat it as **Stable** -(locked against breaking changes). A number of subsections — mostly newer extension points — are -explicitly marked **Development** (pre-release, can still change) inline; these are called out -throughout this doc. `TraceIdRatioBased`'s _configuration/creation_ API is Stable, but its -_algorithm_ compatibility is separately flagged (see §[Samplers](#samplers)). No section in either -document is marked **Experimental** or **Deprecated** at time of writing (2026), though -`TraceIdRatioBased` is textually described as "deprecated in favor of" `ProbabilitySampler` (a -**Development**-status component) — see §[Samplers](#samplers) for the exact deprecation timeline -language. +Unless a subsection is explicitly marked otherwise, treat it as **Stable** (locked against breaking +changes). Several subsections, mostly newer extension points, are marked **Development** +(pre-release, can still change) inline; this doc calls them out. `TraceIdRatioBased`'s +_configuration/creation_ API is Stable, but its _algorithm_ compatibility is flagged separately +(see §[Samplers](#samplers)). No section in either document is marked **Experimental** or +**Deprecated** at time of writing (2026). `TraceIdRatioBased` is, however, textually described as +"deprecated in favor of" `ProbabilitySampler` (a **Development** component); see +§[Samplers](#samplers) for the exact deprecation timeline. --- @@ -66,7 +68,7 @@ language. **Stability: Stable.** **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#spancontext -A `SpanContext` is the serializable portion of a `Span` — the part that must be propagated across +A `SpanContext` is the serializable portion of a `Span`: the part that must be propagated across process boundaries. `SpanContext`s are **immutable**. The representation conforms to the [W3C TraceContext spec](https://www.w3.org/TR/trace-context/). @@ -76,25 +78,26 @@ process boundaries. `SpanContext`s are **immutable**. The representation conform | `SpanId` | 8-byte array; **valid** iff it has at least one non-zero byte (all-zero = invalid) | | `TraceFlags` | Present on every span context (unlike `TraceState`). Currently defines two flags: `Sampled` ([W3C sampled flag](https://www.w3.org/TR/trace-context-2/#sampled-flag)) and `Random` ([W3C random-trace-id flag](https://www.w3.org/TR/trace-context-2/#random-trace-id-flag)) | | `TraceState` | List of tracing-system-specific key-value pairs; lets multiple tracing systems co-participate in one trace; fully defined by [W3C `tracestate`](https://www.w3.org/TR/trace-context-2/#tracestate-header) | -| `IsRemote` | Boolean — was this `SpanContext` received from another process, or locally generated? | +| `IsRemote` | Boolean: was this `SpanContext` received from another process, or generated locally? | Normative rules: - The API **MUST** implement methods to create a `SpanContext`, and these **SHOULD** be the only - way to create one; this "MUST be fully implemented in the API, and SHOULD NOT be overridable." -- **Retrieving TraceId/SpanId:** the API **MUST** allow retrieval in hex (32-hex-char lowercase - string for `TraceId`, 16-hex-char lowercase for `SpanId`) and binary (16-byte / 8-byte array) - forms. The API **SHOULD NOT** expose how they are stored internally. -- **`IsValid`**: "MUST be provided," returns `true` iff both `TraceId` and `SpanId` are non-zero. + way to create one. This functionality "MUST be fully implemented in the API, and SHOULD NOT be + overridable." +- **Retrieving TraceId/SpanId:** the API **MUST** allow retrieval in hex (32-char lowercase hex + string for `TraceId`, 16-char for `SpanId`) and binary (16-byte / 8-byte array) forms. The API + **SHOULD NOT** expose how they are stored internally. +- **`IsValid`**: "MUST be provided." Returns `true` iff both `TraceId` and `SpanId` are non-zero. - **`IsRemote`**: "MUST be provided." When a `SpanContext` is extracted via the Propagators API, - `IsRemote` **MUST** return `true`; for any child span's own context it **MUST** return `false`. -- **`TraceState` operations** — the API **MUST** provide: get value for key, add key-value pair, - update existing value, delete pair. All mutating operations **MUST** return a new immutable - `TraceState`; all must validate their inputs and **MUST NOT** return a `TraceState` containing - invalid data on invalid input (follow "general error handling guidelines" instead). Because + `IsRemote` **MUST** return `true`. For any child span's own context it **MUST** return `false`. +- **`TraceState` operations:** the API **MUST** provide get value for key, add key-value pair, + update existing value, and delete pair. All mutating operations **MUST** return a new immutable + `TraceState`. All must validate their inputs and **MUST NOT** return a `TraceState` containing + invalid data on invalid input (follow the "general error handling guidelines" instead). Because `SpanContext` is immutable, a new `TraceState` can only take effect at [propagation](https://opentelemetry.io/docs/specs/otel/context/api-propagators/) or - [export](https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-exporter) time — propagators + [export](https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-exporter) time. Propagators and exporters may create a modified copy right before serializing to the wire. --- @@ -105,26 +108,26 @@ Normative rules: inherited from the SDK's `TracerConfig`). **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#span -A `Span` represents a single operation within a trace; spans nest into a trace tree with one root +A `Span` represents a single operation within a trace. Spans nest into a trace tree with one root span per trace. A `Span` encapsulates: name, an immutable `SpanContext`, a parent (`Span` / `SpanContext` / null), `SpanKind`, start/end timestamps, `Attributes`, a list of `Link`s, a list of timestamped `Event`s, and a `Status`. **Span name:** "the most general string that identifies a (statistically) interesting _class of -Spans_" rather than a per-instance identifier — generality is prioritized over human-readability -(e.g. `get_account` is good; `get_account/42` is not, due to cardinality; `get_account/{accountId}` -is also acceptable using the HTTP-route form). +Spans_" rather than a per-instance identifier. Generality wins over human readability. Example: +`get_account` is good; `get_account/42` is not (cardinality); `get_account/{accountId}` is +acceptable (HTTP-route form). **Timestamps:** "A `Span`'s start and end timestamps reflect the elapsed real time of the operation." Start time **SHOULD** default to span-creation time. After creation it **SHOULD** be -possible to change the name, set attributes, add events, and set status — but none of these -**MUST** be changed after the end timestamp has been set. +possible to change the name, set attributes, add events, and set status. None of these **MUST** be +changed after the end timestamp has been set. -**Isolation from application logic:** Spans are not meant to propagate information within a -process; implementations **SHOULD NOT** expose a `Span`'s own attributes/events/etc. back to the -user (only its `SpanContext` is retrievable). Vendors may implement `Span` for vendor-specific -logic, but alternative implementations **MUST NOT** allow callers to create `Span`s directly — all -`Span`s **MUST** be created via a `Tracer`. +**Isolation from application logic:** spans are not meant to propagate information within a +process. Implementations **SHOULD NOT** expose a `Span`'s own attributes/events back to the user; +only its `SpanContext` is retrievable. Vendors may implement `Span` for vendor-specific logic, but +alternative implementations **MUST NOT** allow callers to create `Span`s directly. All `Span`s +**MUST** be created via a `Tracer`. ### Span Creation @@ -132,39 +135,39 @@ logic, but alternative implementations **MUST NOT** allow callers to create `Spa - There **MUST NOT** be any API for creating a `Span` other than via a `Tracer`. - In languages with implicit context propagation, span creation **MUST NOT** set the new span as - the active span in the current `Context` by default (MAY be offered as a separate operation). + the active span in the current `Context` by default (this MAY be offered as a separate operation). - Required/accepted parameters: - - **Name** — required. + - **Name**: required. - **Parent `Context`** or an explicit "root span" indication. The API **MUST NOT** accept a - `Span` or `SpanContext` directly as parent — only a full `Context`. (See + `Span` or `SpanContext` directly as parent, only a full `Context`. (See [Determining the Parent Span from a Context](#determining-the-parent-span-from-a-context).) - - **`SpanKind`** — defaults to `SpanKind.Internal` if unspecified. - - **`Attributes`** — empty collection assumed if not given. "The API documentation MUST state - that adding attributes at span creation is preferred to calling `SetAttribute` later, as + - **`SpanKind`**: defaults to `SpanKind.Internal` if unspecified. + - **`Attributes`**: an empty collection is assumed if not given. "The API documentation MUST + state that adding attributes at span creation is preferred to calling `SetAttribute` later, as samplers can only consider information already present during span creation." - - **`Link`s** — an ordered sequence, see §[Link](#link). - - **Start timestamp** — defaults to current time; "SHOULD only be set when span creation time - has already passed" (i.e. don't pass "now" explicitly). + - **`Link`s**: an ordered sequence, see §[Link](#link). + - **Start timestamp**: defaults to current time. It "SHOULD only be set when span creation time + has already passed" (don't pass "now" explicitly). - Root spans: "Implementations MUST provide an option to create a `Span` as a root span, and MUST generate a new `TraceId` for each root span created." For a non-root span, `TraceId` **MUST** match the parent's, and the child **MUST** inherit all of the parent's `TraceState` values by default. -- A `Span` is said to have a **remote parent** if it's a child of a span created in another - process; each propagator's deserialization must set `IsRemote=true` on that parent `SpanContext`. -- "Any span that is created MUST also be ended. This is the responsibility of the user." (Failing - to end a span MAY leak memory/resources in the implementation.) +- A `Span` has a **remote parent** if it is a child of a span created in another process. Each + propagator's deserialization must set `IsRemote=true` on that parent `SpanContext`. +- "Any span that is created MUST also be ended. This is the responsibility of the user." Failing + to end a span MAY leak memory/resources in the implementation. #### Determining the Parent Span from a Context -If the input `Context` contains a `Span`, that is the parent; if not, the new span is a root span. -A bare `SpanContext` cannot be set active in a `Context` directly — it must first be +If the input `Context` contains a `Span`, that is the parent. If not, the new span is a root span. +A bare `SpanContext` cannot be set active in a `Context` directly. It must first be [wrapped in a (non-recording) Span](#wrapping-a-spancontext-in-a-span). #### Specifying links "During Span creation, a user MUST have the ability to record links to other Spans" (same or different trace). "Links added at Span creation may be considered by Samplers to make a sampling -decision" — a capability links added later do not have (see §[Add Link](#add-link)). +decision." Links added later lack this (see §[Add Link](#add-link)). ### Span operations @@ -173,17 +176,17 @@ decision" — a capability links added later do not have (see §[Add Link](#add- "With the exception of the function to retrieve the Span's SpanContext and `IsRecording`, none of the below may be called after the Span is finished." -| Operation | Key normative rule | -| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Get Context** | MUST return the `SpanContext`; usable even after the span ends; MUST be the same value for the entire span lifetime. MAY be called `GetContext`. | -| **IsRecording** | See dedicated subsection below. | -| **Set Attributes** | MUST provide a single-attribute setter (`SetAttribute`); MAY provide a batch setter. Duplicate keys overwrite. Samplers only ever see attributes present at creation time — later changes can't affect their decision. | -| **Add Events** | MUST provide `AddEvent(name, attributes?, timestamp?)`; timestamp defaults to call time if omitted. Events SHOULD preserve recording order (may differ from timestamp order if custom timestamps are used out-of-order). An event's timestamp may legally fall before span start / after span end — no normalization is required. | -| **Add Link** | MUST support adding links post-creation (see §[Add Link](#add-link)) — but these "may not be considered by Samplers." | -| **Set Status** | See §[Set Status](#set-status). | -| **UpdateName** | Changes span name post-creation; sampling behavior thereafter is implementation-defined, since samplers can't retroactively reconsider. | -| **End** | See dedicated subsection below. | -| **Record Exception** | Language-specific specialization of `AddEvent` for exceptions (see below). | +| Operation | Key normative rule | +| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Get Context** | MUST return the `SpanContext`; usable even after the span ends; MUST be the same value for the entire span lifetime. MAY be called `GetContext`. | +| **IsRecording** | See dedicated subsection below. | +| **Set Attributes** | MUST provide a single-attribute setter (`SetAttribute`); MAY provide a batch setter. Duplicate keys overwrite. Samplers only see attributes present at creation time; later changes can't affect their decision. | +| **Add Events** | MUST provide `AddEvent(name, attributes?, timestamp?)`; timestamp defaults to call time if omitted. Events SHOULD preserve recording order (may differ from timestamp order if custom timestamps are used out of order). An event's timestamp may legally fall before span start or after span end; no normalization is required. | +| **Add Link** | MUST support adding links post-creation (see §[Add Link](#add-link)), but these "may not be considered by Samplers." | +| **Set Status** | See §[Set Status](#set-status). | +| **UpdateName** | Changes span name post-creation. Sampling behavior afterwards is implementation-defined, since samplers can't retroactively reconsider. | +| **End** | See dedicated subsection below. | +| **Record Exception** | Language-specific specialization of `AddEvent` for exceptions (see below). | #### IsRecording @@ -191,18 +194,17 @@ the below may be called after the Span is finished." like `SetAttributes`, `AddEvent`, `SetStatus` is captured in some form (e.g. in memory). When a `Span` is not recording ... all this data is discarded right away." Further calls become no-ops. -Critically: **"This flag may be `true` despite the entire trace not being sampled."** This lets a -system record/process all spans locally (e.g. for SLA/SLO latency charts) while only exporting a -sampled subset to the backend — see §[Sampling](#sampling-sdk) for the full `IsRecording` × -`Sampled` matrix. +**"This flag may be `true` despite the entire trace not being sampled."** A system can record and +process all spans locally (e.g. for SLA/SLO latency charts) while exporting only a sampled subset +to the backend. See §[Sampling](#sampling-sdk) for the full `IsRecording` × `Sampled` matrix. "After a `Span` is ended, it SHOULD become non-recording and `IsRecording` SHOULD always return -`false`" (streaming implementations without local state are the one documented exception). +`false`." Streaming implementations without local state are the one documented exception. `IsRecording` **SHOULD NOT** take parameters and **SHOULD** be used to skip expensive attribute/event -computation when a span isn't recording. A child span's own recording state is independent of its -parent's `IsRecording` value (it's driven by the `Sampled` flag on `SpanContext` instead). -"Users of the API should only access the `IsRecording` property when instrumenting code and never -access `SampledFlag` unless used in context propagators." +computation when a span isn't recording. A child span's recording state is independent of its +parent's `IsRecording` value; the `Sampled` flag on `SpanContext` drives it instead. "Users of the +API should only access the `IsRecording` property when instrumenting code and never access +`SampledFlag` unless used in context propagators." #### Set Status @@ -210,7 +212,7 @@ access `SampledFlag` unless used in context propagators." Overrides the default `Unset` status. `Status` = `{ StatusCode, Description? }`. `Description` **MUST** only be used with `StatusCode=Error` (an empty description is treated as -absent); it **MUST be ignored** for `Ok`/`Unset`. +absent). It **MUST be ignored** for `Ok`/`Unset`. | `StatusCode` | Meaning | | ------------ | ---------------------------------------------------------------------------------------------------------- | @@ -218,74 +220,71 @@ absent); it **MUST be ignored** for `Ok`/`Unset`. | `Ok` | "The operation has been validated by an Application developer or Operator to have completed successfully." | | `Error` | The operation contains an error. | -These "form a total order: `Ok > Error > Unset`" — setting `Ok` overrides any prior _or future_ +These "form a total order: `Ok > Error > Unset`". Setting `Ok` overrides any prior _or future_ attempt to set `Error`/`Unset`. Additional rules: - An attempt to set `Unset` explicitly **SHOULD** be ignored. -- When Instrumentation Libraries set `Error`, the `Description` **SHOULD** be documented and - predictable, and the decision to set `Error` at all **should only follow semantic-convention - rules** (or the library's own published conventions if none exist for that operation). +- When instrumentation libraries set `Error`, the `Description` **SHOULD** be documented and + predictable. The decision to set `Error` at all **should only follow semantic-convention rules** + (or the library's own published conventions if none exist for that operation). - "Generally, Instrumentation Libraries SHOULD NOT set the status code to `Ok`, unless explicitly configured to do so." They **SHOULD** leave status `Unset` absent an error. -- Only **Application developers and Operators** (i.e., end-user code, not instrumentation - libraries) are expected to set `Ok`. +- Only **application developers and operators** (end-user code, not instrumentation libraries) are + expected to set `Ok`. - Once `Ok` is set it **SHOULD** be considered final; further changes **SHOULD** be ignored. - "Analysis tools SHOULD respond to an `Ok` status by suppressing any errors they would otherwise generate" (e.g. suppressing a noisy 404-as-error). -- "Only the value of the last call will be recorded" — implementations are free to ignore earlier +- "Only the value of the last call will be recorded." Implementations are free to ignore earlier `SetStatus` calls when a later, permitted one arrives. > **Maple note:** the spec's enum spelling (`Unset`/`Ok`/`Error`) is exactly the title-case string -> Maple stores (CLAUDE.md: "Span Status Codes: Use title case"). The OTLP wire proto uses a -> _different_, uppercase, prefixed spelling — `STATUS_CODE_UNSET` / `STATUS_CODE_OK` / -> `STATUS_CODE_ERROR` (see the +> Maple stores (CLAUDE.md, "Span status codes"). The OTLP wire proto uses a different, uppercase, +> prefixed spelling: `STATUS_CODE_UNSET` / `STATUS_CODE_OK` / `STATUS_CODE_ERROR` (see the > [OTLP trace.proto `Status.StatusCode`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto)). -> The API spec explicitly notes the OTLP proto refers to `Description` as `message`. Any ingest -> mapping code translating OTLP → Maple's internal representation is the seam where this spelling -> difference must be handled once and consistently. +> The API spec notes that the OTLP proto calls `Description` `message`. Ingest mapping code that +> translates OTLP to Maple's internal representation is where this spelling difference must be +> handled, once and consistently. #### End "Signals that the operation described by this span has now (or at the time optionally specified) -ended." Implementations **SHOULD** ignore subsequent calls to `End` or any other span method — the +ended." Implementations **SHOULD** ignore subsequent calls to `End` or any other span method; the span becomes non-recording once ended (streaming exceptions noted above). Other end-triggering language sugar (e.g. Python `with`) **MUST** internally call `End` and be documented as such. -- **`End` MUST NOT affect child spans** — they may still be running and end later independently. -- **`End` MUST NOT inactivate the span in any `Context`** it's active in — an ended span remains +- **`End` MUST NOT affect child spans.** They may still be running and end later independently. +- **`End` MUST NOT inactivate the span in any `Context`** it is active in. An ended span remains usable as a parent via that context, and any Context-attachment mechanism must keep working. -- Optional parameter: explicit end timestamp; if omitted, treated as "now." +- Optional parameter: explicit end timestamp. If omitted, "now" is used. - Performance: "Expect this operation to be called in the hot path of production applications. It needs to be designed to complete fast, if not immediately." `End` itself **MUST NOT** perform - blocking I/O on the calling thread, and locking should be minimized/eliminated where possible - (downstream test/debug-only processors/exporters are explicitly out of scope for this - requirement). + blocking I/O on the calling thread, and locking should be minimized where possible. + Test/debug-only processors and exporters are explicitly out of scope for this requirement. #### Record Exception -Languages that use exceptions **SHOULD** provide `RecordException` as a specialized `AddEvent` — +Languages that use exceptions **SHOULD** provide `RecordException` as a specialized `AddEvent`. The same requirements as `AddEvent` apply except where overridden here. The minimum required argument -**SHOULD** be just the exception object; an optional parameter for additional attributes **MUST** -be accepted if the method exists (existing conventional attributes take precedence over duplicates -supplied this way). +**SHOULD** be just the exception object. If the method exists, it **MUST** accept an optional +parameter for additional attributes; existing conventional attributes take precedence over +duplicates supplied this way. ### Span lifetime "Start and end time as well as Event's timestamps MUST be recorded at [the] time of ... calling of -[the] corresponding API" — i.e., timestamps reflect the moment the API call happens (or an -explicit override), not some other clock. +[the] corresponding API." Timestamps reflect the moment the API call happens (or an explicit +override), not some other clock. ### Wrapping a SpanContext in a Span -The API **MUST** provide an operation wrapping a bare `SpanContext` as a `Span` (for in-process -propagation scenarios, e.g. propagator extraction). If a new type is needed for this, it **SHOULD -NOT** be publicly exposed if avoidable; if it must be, it **SHOULD** be named `NonRecordingSpan`. -Behavior: +The API **MUST** provide an operation that wraps a bare `SpanContext` as a `Span` (for in-process +propagation, e.g. propagator extraction). If a new type is needed for this, it **SHOULD NOT** be +publicly exposed if avoidable. If it must be, it **SHOULD** be named `NonRecordingSpan`. Behavior: - `GetContext` **MUST** return the wrapped `SpanContext`. - `IsRecording` **MUST** return `false`. -- All remaining `Span` methods **MUST** be no-ops — including `End` (so, as the one exception to - "every span must be ended," it's not required, or even useful, to end a wrapped context). +- All remaining `Span` methods **MUST** be no-ops, including `End`. This is the one exception to + "every span must be ended": ending a wrapped context is neither required nor useful. This "MUST be fully implemented in the API, and SHOULD NOT be overridable." @@ -296,11 +295,11 @@ This "MUST be fully implemented in the API, and SHOULD NOT be overridable." **Stability: Stable.** **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#spankind -`SpanKind` communicates two independent properties for analysis tooling: (1) outgoing-remote-call -vs. incoming-external-request, and (2) request/response vs. deferred execution. "In order for +`SpanKind` communicates two independent properties for analysis tooling: (1) outgoing remote call +vs. incoming external request, and (2) request/response vs. deferred execution. "In order for `SpanKind` to be meaningful, callers SHOULD arrange that a single Span does not serve more than one -purpose" — e.g. a server-handling span should not double as the span for an outgoing RPC it makes; -instrumentation should start a _new_ span before injecting `SpanContext` for an outgoing call. +purpose." For example, a server-handling span should not double as the span for an outgoing RPC it +makes; instrumentation should start a _new_ span before injecting `SpanContext` for an outgoing call. | `SpanKind` | Call direction | Communication style | Definition | | ---------- | -------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | @@ -310,14 +309,14 @@ instrumentation should start a _new_ span before injecting `SpanContext` for an | `CONSUMER` | incoming | deferred execution | Processing of an operation initiated by a producer that does not wait for the outcome. | | `INTERNAL` | (n/a) | (n/a) | **Default value.** An internal operation, as opposed to one with remote parents/children. | -Notes: a `CLIENT` span may have a `CLIENT` child, or a `PRODUCER` may have a local `CLIENT` child — -kind describes the _edge_, not a strict alternating pattern. Technology-specific semantic +Notes: a `CLIENT` span may have a `CLIENT` child, and a `PRODUCER` may have a local `CLIENT` child. +Kind describes the _edge_, not a strict alternating pattern. Technology-specific semantic conventions document the expected kind per operation type (e.g. DB client calls use `CLIENT`; if a DB client itself talks HTTP, the nested HTTP instrumentation creates its own nested `CLIENT` spans). > **Maple relevance:** these 5 kinds (plus `PRODUCER`/`CONSUMER` pairing rules) drive service-map -> edge direction and any parent/child inference Maple's flamegraph or trace-topology code performs -> — a `PRODUCER` span legitimately ending before its `CONSUMER` starts is expected, not a data bug. +> edge direction and any parent/child inference Maple's flamegraph or trace-topology code performs. +> A `PRODUCER` span ending before its `CONSUMER` starts is expected, not a data bug. --- @@ -331,55 +330,53 @@ https://opentelemetry.io/docs/specs/otel/trace/api/#add-link be from the same or a different trace." A `Link` = `{ SpanContext, Attributes? }`. - The API **MUST** provide a single-link recorder (e.g. `AddLink`) taking the target `SpanContext` - plus optional attributes; **MAY** provide a batch-add variant. + plus optional attributes. It **MAY** provide a batch-add variant. - "Implementations SHOULD record links containing `SpanContext` with empty `TraceId` or `SpanId` - (all zeros) as long as either the attribute set or `TraceState` is non-empty" — i.e., a link can - legitimately carry an invalid ID pair if it's only there to smuggle attributes/tracestate. + (all zeros) as long as either the attribute set or `TraceState` is non-empty." A link can + legitimately carry an invalid ID pair if it only exists to carry attributes/tracestate. - Order of links **SHOULD** be preserved as set. -- **Links added at span creation** are preferred and documented as such: "because head sampling +- **Links added at span creation** are preferred and documented as such, "because head sampling decisions can only consider information present during span creation." -- **Links added after creation** (`AddLink` post-start) are explicitly allowed by the API — "A - `Span` MUST have the ability to add `Link`s associated with it after its creation" — but "Links - added after `Span` creation may not be considered by Samplers." +- **Links added after creation** (`AddLink` post-start) are explicitly allowed: "A `Span` MUST + have the ability to add `Link`s associated with it after its creation." But "Links added after + `Span` creation may not be considered by Samplers." -**Concurrency:** Links are immutable and **SHOULD** be safe for concurrent use by default (Events -are immutable and **MUST** be safe for concurrent use — a slightly stronger requirement than for -Links). +**Concurrency:** Links are immutable and **SHOULD** be safe for concurrent use by default. Events +are immutable and **MUST** be safe for concurrent use, a slightly stronger requirement. --- ## 6. TracerProvider / Tracer **Stability: Stable** for the core API-level operations described here; **Development** for -`TracerConfigurator`/`TracerConfig` (SDK-level dynamic enable/disable) — see +`TracerConfigurator`/`TracerConfig` (SDK-level dynamic enable/disable), see §[TracerProvider / Tracer (SDK)](#tracerprovider--tracer-sdk). **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#tracerprovider , https://opentelemetry.io/docs/specs/otel/trace/api/#tracer `TracerProvider` is the API's entry point and the stateful object holding configuration. The API -**SHOULD** provide a way to set/register and access a global default instance, and implementations -**SHOULD** allow creating an arbitrary number of independent `TracerProvider` instances (e.g. for +**SHOULD** provide a way to set/register and access a global default instance. Implementations +**SHOULD** allow creating any number of independent `TracerProvider` instances (e.g. for per-dependency-injection-scope configuration). -**Get a Tracer** — `TracerProvider` **MUST** provide this. Accepted parameters: +**Get a Tracer**: `TracerProvider` **MUST** provide this. Accepted parameters: -| Param | Required? | Notes | -| ------------ | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `name` | required | "SHOULD uniquely identify the instrumentation scope" (library/package/module/class name). Invalid (null/empty) input **MUST** still return a working no-op-free `Tracer`, not null/an exception — its `name` **SHOULD** be set to empty string and an invalid-input message **SHOULD** be logged. Implementations _may_ ignore `name` entirely if "named" tracers aren't supported. | -| `version` | optional | Instrumentation scope version, e.g. `"1.0.0"`. | -| `schema_url` | optional, since 1.4.0 | Schema URL recorded in emitted telemetry. | -| `attributes` | optional, since 1.13.0 | Instrumentation scope attributes to associate with emitted telemetry. | +| Param | Required? | Notes | +| ------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `name` | required | "SHOULD uniquely identify the instrumentation scope" (library/package/module/class name). Invalid (null/empty) input **MUST** still return a working `Tracer`, not null or an exception. Its `name` **SHOULD** keep the original invalid value, and an invalid-input message **SHOULD** be logged. Implementations _may_ ignore `name` entirely if "named" tracers aren't supported. | +| `version` | optional | Instrumentation scope version, e.g. `"1.0.0"`. | +| `schema_url` | optional, since 1.4.0 | Schema URL recorded in emitted telemetry. | +| `attributes` | optional, since 1.13.0 | Instrumentation scope attributes to associate with emitted telemetry. | -Two `Tracer`s are _identical_ if all params match, _distinct_ otherwise. "Implementations MUST NOT -require users to repeatedly obtain a `Tracer` again with the same identity to pick up configuration -changes" — either the old `Tracer` picks up new config live, or it keeps working with stale config; -either is compliant as long as re-acquisition isn't _required_. +Two `Tracer`s are _identical_ if all params match, and _distinct_ otherwise. "Implementations MUST +NOT require users to repeatedly obtain a `Tracer` again with the same identity to pick up +configuration changes." The old `Tracer` may pick up new config live or keep working with stale +config; either is compliant as long as re-acquisition isn't _required_. -**Tracer operations:** `Tracer` **MUST** provide span creation; **SHOULD** provide an `Enabled()` -check (see §[Tracer (SDK)](#tracerprovider--tracer-sdk) for the SDK-level semantics driving its -return value) so callers can skip expensive work when tracing is off. `Enabled`'s return value can -change over time — the API **SHOULD** document that instrumentation must re-check it per span, not -cache it. +**Tracer operations:** `Tracer` **MUST** provide span creation. It **SHOULD** provide an `Enabled()` +check (see §[Tracer (SDK)](#tracerprovider--tracer-sdk) for the SDK semantics behind its return +value) so callers can skip expensive work when tracing is off. `Enabled`'s return value can change +over time. The API **SHOULD** document that instrumentation must re-check it per span, not cache it. **Concurrency:** all `TracerProvider`, `Tracer`, and `Span` methods **MUST** be documented as safe for concurrent use by default. @@ -393,13 +390,13 @@ for concurrent use by default. The API **MUST** provide, against the generic [`Context`](https://opentelemetry.io/docs/specs/otel/context/): (1) extracting the `Span` from a -`Context`, (2) combining a `Span` with a `Context` to produce a new `Context`. These exist because +`Context`, and (2) combining a `Span` with a `Context` to produce a new `Context`. These exist because API users **SHOULD NOT** have direct access to the Context Key the tracing API uses internally. -If the language supports **implicit** context propagation, the API **SHOULD** additionally provide: -getting the currently-active span (equivalent to get-implicit-context-then-extract-span), and -setting the active span (equivalent to combine-then-make-implicit). These may be exposed as static -module-level functions and **SHOULD** be fully implemented at the API layer where possible. +If the language supports **implicit** context propagation, the API **SHOULD** also provide: +getting the currently active span (get implicit context, then extract span), and setting the active +span (combine, then make implicit). These may be exposed as static module-level functions and +**SHOULD** be fully implemented at the API layer where possible. --- @@ -408,14 +405,14 @@ module-level functions and **SHOULD** be fully implemented at the API layer wher **Stability: Stable.** **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#behavior-of-the-api-in-the-absence-of-an-installed-sdk -"In general, in the absence of an installed SDK, the Trace API is a no-op API" — operations on a +"In general, in the absence of an installed SDK, the Trace API is a no-op API." Operations on a `Tracer` or `Span` have no side effects. The one carve-out is `SpanContext` propagation continuity: the API **MUST** return a non-recording `Span` wrapping whatever `SpanContext` was in the parent -`Context` (explicit or current-implicit). If that parent context's span is already non-recording, +`Context` (explicit or current implicit). If that parent context's span is already non-recording, it **SHOULD** be returned directly (no new object). If the parent context has no span at all, an -**empty** non-recording span **MUST** be returned — all-zero `SpanContext`, empty `TraceState`, +**empty** non-recording span **MUST** be returned: all-zero `SpanContext`, empty `TraceState`, unsampled flags. Net effect: a `SpanContext` supplied by a configured `Propagator` still flows -through to children and eventually `Inject`, but no _new_ `SpanContext` gets created without an SDK. +through to children and eventually `Inject`, but no _new_ `SpanContext` is created without an SDK. --- @@ -423,51 +420,51 @@ through to children and eventually `Inject`, but no _new_ `SpanContext` gets cre **Source:** https://opentelemetry.io/docs/specs/otel/trace/api/#concurrency-requirements -- **TracerProvider / Tracer / Span** — all methods **MUST** be documented safe for concurrent use +- **TracerProvider / Tracer / Span:** all methods **MUST** be documented safe for concurrent use by default. -- **Event** — immutable, **MUST** be safe for concurrent use by default. -- **Link** — immutable, **SHOULD** be safe for concurrent use by default. +- **Event:** immutable, **MUST** be safe for concurrent use by default. +- **Link:** immutable, **SHOULD** be safe for concurrent use by default. --- ## 10. SDK: TracerProvider / Tracer (SDK) **Stability: mixed.** Core `TracerProvider` responsibilities (owning config, `Shutdown`, -`ForceFlush`) are **Stable**; per-tracer dynamic enable/disable via `TracerConfigurator` / +`ForceFlush`) are **Stable**. Per-tracer dynamic enable/disable via `TracerConfigurator` / `TracerConfig` is explicitly **Development**. **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#tracer-provider The SDK's `TracerProvider` **MUST** implement the API's Get-a-Tracer operation and **MUST** build the `InstrumentationScope` from the caller-supplied name/version/schema_url/attributes, storing it -on the returned `Tracer`. Configuration ownership — `SpanProcessor`s, `IdGenerator`, `SpanLimits`, -`Sampler`, and (Development) `TracerConfigurator` — belongs to the `TracerProvider`; it MAY be -applied at construction time and the provider MAY expose update methods, but any update **MUST** -retroactively apply to already-returned `Tracer`s (implementations typically achieve this by having -`Tracer`s hold a reference back to the provider rather than caching config directly). - -**Shutdown** — MUST be called only once per `TracerProvider`; subsequent `GetTracer` calls after -Shutdown are disallowed (SDKs **SHOULD** return a working no-op `Tracer` if possible instead of -erroring). **MUST** be implemented by invoking `Shutdown` on all internal processors. **SHOULD** -report success/failure/timeout to the caller and **SHOULD** complete or abort within some timeout -(sync or async, implementation's choice). - -**ForceFlush** — immediately exports all not-yet-exported spans across all internal processors. -**MUST** invoke `ForceFlush` on every registered `SpanProcessor`. Same SHOULD-report-outcome / -SHOULD-timeout guidance as Shutdown. +on the returned `Tracer`. The `TracerProvider` owns configuration: `SpanProcessor`s, `IdGenerator`, +`SpanLimits`, `Sampler`, and (Development) `TracerConfigurator`. Configuration MAY be applied at +construction time and the provider MAY expose update methods, but any update **MUST** apply +retroactively to already-returned `Tracer`s. Implementations typically do this by having `Tracer`s +hold a reference back to the provider rather than caching config. + +**Shutdown:** MUST be called only once per `TracerProvider`. `GetTracer` calls after Shutdown are +disallowed; SDKs **SHOULD** return a working no-op `Tracer` if possible instead of erroring. +**MUST** be implemented by invoking `Shutdown` on all internal processors. **SHOULD** report +success/failure/timeout to the caller and **SHOULD** complete or abort within some timeout (sync or +async, implementation's choice). + +**ForceFlush:** immediately exports all not-yet-exported spans across all internal processors. +**MUST** invoke `ForceFlush` on every registered `SpanProcessor`. The same SHOULD-report-outcome / +SHOULD-timeout guidance as Shutdown applies. **`TracerConfigurator` (Development):** a function `tracer_scope -> TracerConfig` (or a -"use-default" sentinel) that the `TracerProvider` calls per-`Tracer`, both at first creation and -(if updating is supported) for every outstanding `Tracer` when the configurator itself is updated — -so it "is important that it returns quickly." Modeled as a function for flexibility; SDKs MAY offer -shorthand helpers (select tracers by name/pattern, disable specific tracers, disable-all-then-allow­list). +"use-default" sentinel). The `TracerProvider` calls it per `Tracer` at first creation and, if +updating is supported, for every outstanding `Tracer` when the configurator itself is updated. So +"it is important that it returns quickly." It is modeled as a function for flexibility; SDKs MAY +offer shorthand helpers (select tracers by name/pattern, disable specific tracers, +disable-all-then-allowlist). **`TracerConfig` (Development):** currently one field, `enabled` (default `true`). A disabled `Tracer` **MUST** behave like the API's no-op/no-SDK `Tracer`. `enabled` directly determines -`Enabled()`'s return value (`false` → `Enabled()` returns `false`; `true` → returns `true`). -Implementations don't need config changes to be _immediately_ visible to `Enabled()` callers, but -they **MUST** be eventually visible. +`Enabled()`'s return value. Config changes need not be _immediately_ visible to `Enabled()` +callers, but they **MUST** be eventually visible. -**`Enabled` (SDK semantics, Development flag inside a Stable-by-default method):** **MUST** return +**`Enabled` (SDK semantics; a Development flag inside a Stable-by-default method):** **MUST** return `false` when there are no registered `SpanProcessor`s, or (Development) when `TracerConfig.enabled == false`. Otherwise it **SHOULD** return `true`, and MAY return `false` for other optimization reasons. @@ -478,38 +475,38 @@ other optimization reasons. **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#additional-span-interfaces -The API only defines write access to a `Span`; the SDK needs to read data back out for processors +The API only defines write access to a `Span`. The SDK needs to read data back out for processors and exporters, so it defines two SDK-internal contracts: -- **Readable span** — MUST expose everything added via the API-level `Span` interface (that spec - section is authoritative for the full list), MUST expose `InstrumentationScope` (since 1.10.0) - and `Resource`, MUST (for back-compat) also expose the deprecated `InstrumentationLibrary` view - with matching name/version, MUST let callers reliably determine whether the span has ended, and - MUST expose attribute/event/link **dropped counts** (for exporters, per the +- **Readable span:** MUST expose everything added via the API-level `Span` interface (that spec + section is authoritative for the full list). MUST expose `InstrumentationScope` (since 1.10.0) + and `Resource`. MUST, for back-compat, also expose the deprecated `InstrumentationLibrary` view + with matching name/version. MUST let callers reliably determine whether the span has ended. MUST + expose attribute/event/link **dropped counts** (for exporters, per the [non-OTLP mapping spec](https://opentelemetry.io/docs/specs/otel/common/mapping-to-non-otlp/#dropped-attributes-count)). Implementations MAY choose not to expose the full parent `Context`, but MUST expose at least the full parent `SpanContext`. May or may not be mutable. -- **Read/write span** — everything a readable span has, plus the full write API, and callers - **MUST** be able to obtain the _same_ `Span` instance/type that span-creation returned to the user - (e.g. passed as a parameter, or via a getter). +- **Read/write span:** everything a readable span has, plus the full write API. Callers **MUST** be + able to obtain the _same_ `Span` instance/type that span creation returned to the user (e.g. + passed as a parameter, or via a getter). --- ## 12. Sampling (SDK) {#sampling-sdk} -**Stability: mixed** — the core `Sampler`/`ShouldSample` contract, `AlwaysOn`/`AlwaysOff`, and -`TraceIdRatioBased`'s _configuration/creation_ surface are **Stable**; `ProbabilitySampler`, -`CompositeSampler`/`ComposableSampler` and the whole explicit-randomness / `Random`-flag apparatus -are **Development**; `TraceIdRatioBased`'s _algorithm compatibility_ note is separately flagged +**Stability: mixed.** The core `Sampler`/`ShouldSample` contract, `AlwaysOn`/`AlwaysOff`, and +`TraceIdRatioBased`'s _configuration/creation_ surface are **Stable**. `ProbabilitySampler`, +`CompositeSampler`/`ComposableSampler`, and the whole explicit-randomness / `Random`-flag apparatus +are **Development**. `TraceIdRatioBased`'s _algorithm compatibility_ note is separately flagged **Development** even though the sampler itself is Stable. **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#sampling -Sampling controls collection noise/overhead. Two API-level signals govern it: +Sampling controls collection noise and overhead. Two API-level signals govern it: -- **`IsRecording`** on `Span` — if `false`, all data is discarded. `SpanProcessor`s **MUST** only - receive spans with `IsRecording == true`; `SpanExporter`s **SHOULD NOT** receive them unless +- **`IsRecording`** on `Span`: if `false`, all data is discarded. `SpanProcessor`s **MUST** only + receive spans with `IsRecording == true`. `SpanExporter`s **SHOULD NOT** receive them unless `Sampled` is _also_ set. -- **`Sampled`** flag in `TraceFlags` on `SpanContext` — propagates to children via `SpanContext`; +- **`Sampled`** flag in `TraceFlags` on `SpanContext`: propagates to children via `SpanContext` and indicates the span _has been_ sampled and will be exported. Exporters **MUST** receive spans with `Sampled == true` and **SHOULD NOT** receive ones without it. @@ -522,61 +519,65 @@ Sampling controls collection noise/overhead. Two API-level signals govern it: | false | true | **Not allowed** | **Not allowed** | | false | false | false | false | -`IsRecording=false, Sampled=true` is a forbidden combination: "the OpenTelemetry SDK MUST NOT allow -this combination" because it would create gaps in the distributed trace. `IsRecording=true, +`IsRecording=false, Sampled=true` is forbidden: "the OpenTelemetry SDK MUST NOT allow this +combination" because it would create gaps in the distributed trace. `IsRecording=true, Sampled=false` is legal and means "this span records data but its children likely won't." -> **Maple relevance:** a spec-compliant SDK will never emit the forbidden combination, but -> defensive ingest code should treat `Sampled=true` + missing/absent span data as a signal of a +> **Maple relevance:** a spec-compliant SDK never emits the forbidden combination. Defensive +> ingest code should still treat `Sampled=true` with missing span data as a sign of a > non-compliant producer or transport-layer data loss, not assume it can't happen. ### SDK span creation order -When creating a span, the SDK **MUST** act as if, in order: (1) use the parent's trace ID if valid, -else generate a new one — _before_ calling `ShouldSample`, which requires a valid trace ID as -input; (2) call the `Sampler`'s `ShouldSample`; (3) generate a new span ID regardless of the -sampling decision (so other components — logs, exception handling — can rely on a unique span ID -even for a non-recording span); (4) construct the span per the `ShouldSample` decision (a -non-recording result MAY reuse the same "wrap a SpanContext" mechanism as the no-SDK API path). +When creating a span, the SDK **MUST** act as if it does the following, in order: + +1. Use the parent's trace ID if valid, else generate a new one. This happens _before_ calling + `ShouldSample`, which requires a valid trace ID as input. +2. Call the `Sampler`'s `ShouldSample`. +3. Generate a new span ID regardless of the sampling decision, so other components (logs, + exception handling) can rely on a unique span ID even for a non-recording span. +4. Construct the span per the `ShouldSample` decision. A non-recording result MAY reuse the same + "wrap a SpanContext" mechanism as the no-SDK API path. ### Sampler interface -**`ShouldSample`** — required inputs: parent `Context` (its `SpanContext` may be invalid → root -span), the new span's `TraceId` (MUST match the parent's if parent has a valid trace ID), span -name, `SpanKind`, initial `Attributes`, and the collection of `Link`s. Returns a `SamplingResult`: +**`ShouldSample`** required inputs: parent `Context` (its `SpanContext` may be invalid, meaning a +root span), the new span's `TraceId` (MUST match the parent's if the parent has a valid trace ID), +span name, `SpanKind`, initial `Attributes`, and the collection of `Link`s. Returns a +`SamplingResult`: | `SamplingResult` field | Meaning | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Decision` | One of `DROP` / `RECORD_ONLY` / `RECORD_AND_SAMPLE` — see table below. | +| `Decision` | One of `DROP` / `RECORD_ONLY` / `RECORD_AND_SAMPLE`; see table below. | | Attributes | Additional span attributes to add; the returned object MUST be immutable. | -| `Tracestate` | The `TraceState` for the new `SpanContext`. If the sampler returns an _empty_ `Tracestate`, the existing one is cleared — so samplers that don't intend to change it SHOULD pass through the incoming value unmodified. | +| `Tracestate` | The `TraceState` for the new `SpanContext`. If the sampler returns an _empty_ `Tracestate`, the existing one is cleared, so samplers that don't intend to change it SHOULD pass through the incoming value unmodified. | -| `Decision` | `IsRecording` | `Sampled` flag | -| ------------------- | ---------------------------------------------------------- | ------------------- | -| `DROP` | `false` — span not recorded, all events/attributes dropped | not set | -| `RECORD_ONLY` | `true` | **MUST NOT** be set | -| `RECORD_AND_SAMPLE` | `true` | **MUST** be set | +| `Decision` | `IsRecording` | `Sampled` flag | +| ------------------- | ------------------------------------------------------------ | ------------------- | +| `DROP` | `false`: span not recorded, all events/attributes dropped | not set | +| `RECORD_ONLY` | `true` | **MUST NOT** be set | +| `RECORD_AND_SAMPLE` | `true` | **MUST** be set | -**`GetDescription`** — returns the sampler's name/config as a debug string (e.g. -`"TraceIdRatioBased{0.000100}"`); MAY change over time (e.g. under dynamic reconfiguration); +**`GetDescription`** returns the sampler's name/config as a debug string (e.g. +`"TraceIdRatioBased{0.000100}"`). It MAY change over time (e.g. under dynamic reconfiguration); callers **SHOULD NOT** cache it. ### Built-in samplers **Default sampler: `ParentBased(root=AlwaysOn)`.** -| Sampler | Status | Behavior | -| ---------------------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `AlwaysOn` | Stable | Always returns `RECORD_AND_SAMPLE`. Description MUST be `AlwaysOnSampler`. | -| `AlwaysOff` | Stable | Always returns `DROP`. Description MUST be `AlwaysOffSampler`. | -| `TraceIdRatioBased` | Stable (config/creation API); algorithm compatibility notes are Development | Deterministic hash of `TraceId` decides sampling; ignores parent `Sampled` flag (compose with `ParentBased` to respect it). A given ratio MUST sample a superset of what any lower-ratio instance would sample (monotonic — lets a backend sample at a higher rate than the frontend safely). **Deprecation note:** being phased out in favor of `ProbabilitySampler`; "OpenTelemetry SDK implementors SHALL NOT remove or modify the behavior of the original `TraceIdRatioBased` sampler until at least January 1, 2027," after which they're encouraged to silently swap in an equally-configured `ProbabilitySampler`. Exact hash algorithm was never specified — cross-SDK/cross-version results may differ, so it's recommended only as a _root_ sampler. | -| `ProbabilitySampler` | Development | Composable-style ratio sampler built on W3C Trace Context Level 2's 56 bits of randomness (see [Probability Sampling in TraceState](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/)). Ignores parent `Sampled` flag (compose with `ParentBased`). Configured with a ratio in `[2^-56, 1.0]`; on `ShouldSample`, compares randomness value `R` against rejection threshold `T` derived from the ratio — `R >= T` ⇒ `RECORD_AND_SAMPLE`, sets `ot=th:T` in tracestate; else `DROP`. | -| `ParentBased` | Stable | Decorator dispatching on parent shape — see table below. | -| `JaegerRemoteSampler` | Stable | Periodically pulls sampling config from a remote endpoint (Jaeger Collector or OTel Collector implementing the [Remote Sampling API](https://www.jaegertracing.io/docs/2.14/architecture/apis/#remote-sampling-configuration)); can assign different strategies per span name (e.g. `/product` at 10%, `/admin` at 100%, never `/metrics`). Configurable: `endpoint`, `polling interval`, `initial sampler` (used before first fetch). | -| `AlwaysRecord` | Stable | Decorator: converts a wrapped sampler's `DROP` into `RECORD_ONLY` (all other decisions pass through unchanged) so every span reaches processors (e.g. for span-to-metrics) without necessarily being exported. | -| `CompositeSampler` / `ComposableSampler` | Development | Implements `Sampler` by delegating to a `ComposableSampler.GetSamplingIntent` (threshold + `adjusted_count_reliable` + optional attribute/tracestate providers), then deriving/compares a randomness value `R` against the returned threshold to reach the final decision. Built-in composables: `ComposableAlwaysOn`, `ComposableAlwaysOff`, `ComposableProbability` (ratio `[2^-56, 1.0]`), `ComposableParentThreshold` (propagate parent's decision/threshold), `ComposableRuleBased` (predicate → sampler rule list, first match wins), `ComposableAnnotating` (delegate + extra attributes on sampled spans). | - -**`ParentBased` dispatch table** — required param `root(Sampler)`; optional params +| Sampler | Status | Behavior | +| ---------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `AlwaysOn` | Stable | Always returns `RECORD_AND_SAMPLE`. Description MUST be `AlwaysOnSampler`. | +| `AlwaysOff` | Stable | Always returns `DROP`. Description MUST be `AlwaysOffSampler`. | +| `TraceIdRatioBased` | Stable (config/creation API); algorithm compatibility notes are Development | A deterministic hash of `TraceId` decides sampling. Ignores the parent `Sampled` flag (compose with `ParentBased` to respect it). A given ratio MUST sample a superset of what any lower-ratio instance would sample (monotonic, so a backend can safely sample at a higher rate than the frontend). **Deprecation note:** being phased out in favor of `ProbabilitySampler`. "OpenTelemetry SDK implementors SHALL NOT remove or modify the behavior of the original `TraceIdRatioBased` sampler until at least January 1, 2027"; after that, they are encouraged to silently swap in an equally configured `ProbabilitySampler`. The exact hash algorithm was never specified, so results may differ across SDKs and versions; it is recommended only as a _root_ sampler. | +| `ProbabilitySampler` | Development | Ratio sampler built on W3C Trace Context Level 2's 56 bits of randomness (see [Probability Sampling in TraceState](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/)). Ignores the parent `Sampled` flag (compose with `ParentBased`). Configured with a ratio in `[2^-56, 1.0]`. On `ShouldSample`, compares randomness value `R` against rejection threshold `T` derived from the ratio: `R >= T` ⇒ `RECORD_AND_SAMPLE` and sets `ot=th:T` in tracestate; otherwise `DROP`. | +| `ParentBased` | Stable | Decorator dispatching on parent shape; see table below. | +| `JaegerRemoteSampler` | Stable | Periodically pulls sampling config from a remote endpoint (Jaeger Collector or OTel Collector implementing the [Remote Sampling API](https://www.jaegertracing.io/docs/2.14/architecture/apis/#remote-sampling-configuration)). Can assign different strategies per span name (e.g. `/product` at 10%, `/admin` at 100%, never `/metrics`). Configurable: `endpoint`, `polling interval`, `initial sampler` (used before the first fetch). | +| `AlwaysRecord` | Stable | Decorator: converts a wrapped sampler's `DROP` into `RECORD_ONLY` (all other decisions pass through unchanged), so every span reaches processors (e.g. for span-to-metrics) without necessarily being exported. | +| `CompositeSampler` / `ComposableSampler` | Development | Implements `Sampler` by delegating to `ComposableSampler.GetSamplingIntent` (threshold + `adjusted_count_reliable` + optional attribute/tracestate providers), then compares a randomness value `R` against the returned threshold to reach the final decision. Built-in composables: `ComposableAlwaysOn`, `ComposableAlwaysOff`, `ComposableProbability` (ratio `[2^-56, 1.0]`), `ComposableParentThreshold` (propagate the parent's decision/threshold), `ComposableRuleBased` (predicate → sampler rule list, first match wins), `ComposableAnnotating` (delegate + extra attributes on sampled spans). | + +**`ParentBased` dispatch table.** Required param `root(Sampler)`. Optional params `remoteParentSampled` (default `AlwaysOn`), `remoteParentNotSampled` (default `AlwaysOff`), `localParentSampled` (default `AlwaysOn`), `localParentNotSampled` (default `AlwaysOff`): @@ -588,19 +589,18 @@ callers **SHOULD NOT** cache it. | present | false | true | `localParentSampled()` | | present | false | false | `localParentNotSampled()` | -### Sampling requirements — TraceID randomness (Development) +### Sampling requirements: TraceID randomness (Development) The [W3C Trace Context Level 2](https://www.w3.org/TR/trace-context-2/) CR defines a `Random` trace flag meaning "the rightmost 7 bytes / 56 bits of the TraceID are random." SDKs **SHOULD** set this -`Random` flag on root spans when they generate TraceIDs meeting that randomness bar. SDKs and -Samplers **MUST NOT** overwrite an explicit randomness value (the `rv` sub-key of OTel's -`tracestate`, see +flag on root spans when the TraceIDs they generate meet that randomness bar. SDKs and samplers +**MUST NOT** overwrite an explicit randomness value (the `rv` sub-key of OTel's `tracestate`, see [TraceState Handling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/#explicit-randomness-value-rv)) -once a user has set one; root samplers MAY insert an `rv` value themselves when the generated -TraceID doesn't meet the randomness bar and no `rv` is already present. Absent an explicit -randomness value, samplers **SHOULD** presume TraceIDs are W3C-Level-2-random. Custom -`IdGenerator`s **SHOULD** self-identify when all their generated TraceIDs meet the randomness bar so -the SDK can correctly set the `Random` flag. +once a user has set one. Root samplers MAY insert an `rv` value themselves when the generated +TraceID doesn't meet the randomness bar and no `rv` is present. Absent an explicit randomness +value, samplers **SHOULD** presume TraceIDs are W3C-Level-2-random. Custom `IdGenerator`s +**SHOULD** self-identify when all their generated TraceIDs meet the randomness bar, so the SDK can +set the `Random` flag correctly. --- @@ -612,34 +612,33 @@ here). Span attributes **MUST** follow the common [attribute-limits](https://opentelemetry.io/docs/specs/otel/common/#attribute-limits) rules. The -SDK MAY additionally discard links/events beyond a configured per-collection limit. If a limit is +SDK MAY also discard links/events beyond a configured per-collection limit. If a limit is implemented, the SDK **MUST** expose a way to change it via `TracerProvider` configuration. -Discarding an attribute/event/link due to a limit **SHOULD** log a message — but **MUST** be logged -**at most once per span** (not once per discarded item) to avoid log spam. +Discarding an attribute/event/link due to a limit **SHOULD** log a message, and it **MUST** be +logged **at most once per span** (not once per discarded item) to avoid log spam. | Limit | Default | Env var | Notes | | ------------------------------------ | -------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AttributeCountLimit` (common) | 128 | `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` (falls back to `OTEL_ATTRIBUTE_COUNT_LIMIT`) | Max attributes per span. | -| `AttributeValueLengthLimit` (common) | no limit | `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` (falls back to `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`) | Max attribute value length; strings/byte-arrays MUST be truncated to the limit; for arrays of strings/AnyValue, the limit applies per-element; all other value shapes MUST NOT be truncated. | +| `AttributeValueLengthLimit` (common) | no limit | `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` (falls back to `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`) | Max attribute value length; strings/byte-arrays MUST be truncated to the limit; for arrays of strings/AnyValue, the limit applies per element; all other value shapes MUST NOT be truncated. | | `EventCountLimit` | 128 | `OTEL_SPAN_EVENT_COUNT_LIMIT` | Max events per span. | | `LinkCountLimit` | 128 | `OTEL_SPAN_LINK_COUNT_LIMIT` | Max links per span. | | `AttributePerEventCountLimit` | 128 | `OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT` | Max attributes per event. | | `AttributePerLinkCountLimit` | 128 | `OTEL_LINK_ATTRIBUTE_COUNT_LIMIT` | Max attributes per link. | -**Source (env vars):** https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-span-processor -and the neighboring "Span limits" / "Attribute limits" tables on that same page. +**Source (env vars):** https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#span-limits +and the neighboring "Attribute limits" table on that page. -**Attribute value types** (common spec, inherited by span attributes) — Source: -https://opentelemetry.io/docs/specs/otel/common/#attribute — attribute keys **MUST** be non-null, -non-empty strings; key casing is preserved and case-sensitive (differently-cased keys are distinct -keys). Attribute values **MUST** be one of the types defined by `AnyValue` (string, boolean, +**Attribute value types** (common spec, inherited by span attributes). Source: +https://opentelemetry.io/docs/specs/otel/common/#attribute. Attribute keys **MUST** be non-null, +non-empty strings. Key casing is preserved and case-sensitive (differently cased keys are distinct +keys). Attribute values **MUST** be one of the types defined by `AnyValue`: string, boolean, double-precision float, signed 64-bit integer, byte array, homogeneous/nested arrays, maps, and -empty). Verified against the raw spec source (`specification/common/README.md` on `main`, -2026-07): "The attribute value MUST be one of types defined in AnyValue." _Note:_ earlier spec -versions restricted plain `Attribute` values to only primitives and homogeneous primitive arrays -(no maps/nesting); the current spec unifies `Attribute` with the broader `AnyValue` definition -used by log bodies. When validating third-party producers instrumented with older SDKs, expect -the stricter, pre-unification subset in practice. +empty. Verified against the raw spec source (`specification/common/README.md` on `main`, 2026-07): +"The attribute value MUST be one of types defined in AnyValue." _Note:_ earlier spec versions +restricted plain `Attribute` values to primitives and homogeneous primitive arrays (no +maps/nesting). The current spec unifies `Attribute` with the broader `AnyValue` definition used by +log bodies. Third-party producers on older SDKs will emit the stricter, pre-unification subset. --- @@ -650,58 +649,63 @@ extension. **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#id-generators "The SDK MUST by default randomly generate both the `TraceId` and the `SpanId`." The SDK **MUST** -provide a mechanism to customize ID generation for both (typically via an `IdGenerator`-style -interface exposing `generateSpanIdBytes()` / `generateTraceIdBytes()`). Vendor-specific ID -generators (e.g. AWS X-Ray's ID format) **MUST NOT** be maintained/distributed as part of -OpenTelemetry's own core packages. +provide a mechanism to customize ID generation for both, typically an `IdGenerator`-style +interface exposing `generateSpanIdBytes()` / `generateTraceIdBytes()`. Vendor-specific ID +generators (e.g. AWS X-Ray's ID format) **MUST NOT** be maintained or distributed as part of +OpenTelemetry's core packages. **(Development)** Custom `IdGenerator`s **SHOULD** self-identify when _all_ generated TraceIDs meet -the W3C Trace Context Level 2 randomness bar, so the SDK can set the `Random` trace flag -accordingly — typically inferred statically via a marker interface rather than per-call. +the W3C Trace Context Level 2 randomness bar, so the SDK can set the `Random` trace flag. This is +typically inferred statically via a marker interface rather than per call. --- ## 15. Span Processor {#span-processor-sdk} -**Stability: Stable**, except `OnEnding` which is explicitly **Development**. +**Stability: Stable**, except `OnEnding`, which is explicitly **Development**. **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-processor -A `SpanProcessor` hooks span start/end. Processors are invoked **only** when `IsRecording` is true. -Built-in processors handle batching + conversion to an export-friendly representation before -handing off to exporters. Multiple processors register directly on the `TracerProvider` and run **in -registration order**; each forms the head of its own pipeline (processor + optional exporter), and -the SDK **MUST** allow each pipeline to end with an individual exporter. The SDK **MUST** allow -users to implement and register custom processors. +A `SpanProcessor` hooks span start and end. Processors are invoked **only** when `IsRecording` is +true. Built-in processors batch spans and convert them to an export-friendly representation before +handing them to exporters. Multiple processors register directly on the `TracerProvider` and run +**in registration order**. Each forms the head of its own pipeline (processor + optional exporter), +and the SDK **MUST** allow each pipeline to end with its own exporter. The SDK **MUST** allow users +to implement and register custom processors. ### Interface The `SpanProcessor` interface **MUST** declare `OnStart`, `OnEnd`, `Shutdown`, `ForceFlush`, and **SHOULD** declare `OnEnding`. -| Method | Timing / threading | Contract | -| ---------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `OnStart(span, parentContext)` | Synchronous, on the span-starting thread | MUST NOT block/throw. Multiple processors' `OnStart` run in registration order. `span` is a read/write span; keeping a reference and observing live updates SHOULD work (e.g. a processor that periodically inspects all active spans from a background thread). `parentContext` is the SDK-determined parent (explicit / current / empty, per what was requested). Returns void. | -| `OnEnding(span)` — **Development** | Synchronous, inside `Span.End()`, _before_ `OnEnd` | Called once the end timestamp is computed (its own duration is excluded from span duration) but while the span is **still mutable** (`SetAttribute`/`AddLink`/`AddEvent` still legal). MUST NOT block/throw. Multiple processors' `OnEnding` run in registration order; the SDK MUST guarantee no other thread can modify the span once the first `OnEnding` starts — only synchronous in-callback modification is allowed from that point. **All** registered `OnEnding` callbacks run before **any** `OnEnd` callback runs. | -| `OnEnd(span)` | Synchronous, inside `Span.End()`, after end timestamp is set | MUST NOT block/throw. `span` is a readable span; even if technically mutable, modifying it here is not allowed (already ended). | -| `Shutdown()` | — | SHOULD be called only once; subsequent `OnStart`/`OnEnd`/`ForceFlush` calls SHOULD be gracefully ignored. MUST include the effects of `ForceFlush`. SHOULD report success/failure/timeout and SHOULD complete/abort within some timeout. | -| `ForceFlush()` | — | Hint to complete any in-flight span work "as soon as possible, preferably before returning." If the processor has an exporter, it SHOULD `Export` everything not yet exported, then call the exporter's `ForceFlush` — built-in processors **MUST** do so. If a timeout is set, the processor MUST prioritize the timeout over completeness (may abort/skip calls). SHOULD report outcome; SHOULD only be called when "absolutely necessary" (e.g. FaaS suspend-after-invocation risk). | +| Method | Timing / threading | Contract | +| ----------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `OnStart(span, parentContext)` | Synchronous, on the span-starting thread | MUST NOT block/throw. Multiple processors' `OnStart` run in registration order. `span` is a read/write span; keeping a reference and observing live updates SHOULD work (e.g. a processor that periodically inspects all active spans from a background thread). `parentContext` is the SDK-determined parent (explicit / current / empty, per what was requested). Returns void. | +| `OnEnding(span)` (**Development**) | Synchronous, inside `Span.End()`, _before_ `OnEnd` | Called once the end timestamp is computed (its own duration is excluded from span duration) while the span is **still mutable** (`SetAttribute`/`AddLink`/`AddEvent` still legal). MUST NOT block/throw. Multiple processors' `OnEnding` run in registration order. The SDK MUST guarantee no other thread can modify the span once the first `OnEnding` starts; from that point only synchronous in-callback modification is allowed. **All** registered `OnEnding` callbacks run before **any** `OnEnd` callback runs. | +| `OnEnd(span)` | Synchronous, inside `Span.End()`, after end timestamp is set | MUST NOT block/throw. `span` is a readable span; even if technically mutable, modifying it here is not allowed (already ended). | +| `Shutdown()` | (n/a) | SHOULD be called only once; subsequent `OnStart`/`OnEnd`/`ForceFlush` calls SHOULD be gracefully ignored. MUST include the effects of `ForceFlush`. SHOULD report success/failure/timeout and SHOULD complete/abort within some timeout. | +| `ForceFlush()` | (n/a) | Hint to complete any in-flight span work "as soon as possible, preferably before returning." If the processor has an exporter, it SHOULD `Export` everything not yet exported, then call the exporter's `ForceFlush`; built-in processors **MUST** do so. If a timeout is set, the processor MUST prioritize the timeout over completeness (may abort/skip calls). SHOULD report outcome. SHOULD only be called when "absolutely necessary" (e.g. FaaS suspend-after-invocation risk). | ### Built-in processors -The standard SDK **MUST** implement both. (Other cross-cutting processing scenarios are steered -toward the out-of-process [Collector](https://opentelemetry.io/docs/specs/otel/overview/#collector) -instead of more built-in processors.) +The standard SDK **MUST** implement both. Other cross-cutting processing is steered toward the +out-of-process [Collector](https://opentelemetry.io/docs/specs/otel/overview/#collector) rather +than more built-in processors. + +**Simple processor:** passes each span to the configured exporter as soon as it finishes. MUST +synchronize `Export` calls so they are never invoked concurrently. Configurable: `exporter`. + +**Batching processor:** batches finished spans before handing them to the exporter; also MUST +synchronize `Export` calls. Once the previous export has returned, it exports a batch when any of +these holds: -**Simple processor** — passes each finished span to the configured exporter immediately as it -finishes. MUST synchronize `Export` calls so they're never invoked concurrently. Configurable: -`exporter`. +- `scheduledDelayMillis` has passed since construction, or since the first span in a new window. +- `scheduledDelayMillis` has passed since the previous export timer ended or the previous export + completed. +- The queue reaches `maxExportBatchSize`. +- `ForceFlush()` is called. -**Batching processor** — batches finished spans before handing them to the exporter; also MUST -synchronize `Export` calls. Exports a batch when (previous export has returned AND) any of: -`scheduledDelayMillis` since construction / since first span in a new window, `scheduledDelayMillis` -since previous export timer ended/completed, the queue reaches `maxExportBatchSize`, or -`ForceFlush()` is called. An empty queue at export time MAY export an empty batch or skip export -entirely (implementation's choice). +An empty queue at export time MAY export an empty batch or skip the export (implementation's +choice). | Parameter | Default | Env var | Meaning | | ---------------------- | ------- | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | @@ -714,7 +718,7 @@ entirely (implementation's choice). ### Concurrency -`Span processor` — all methods **MUST** be safe for concurrent calls. +Span processor: all methods **MUST** be safe for concurrent calls. --- @@ -723,19 +727,19 @@ entirely (implementation's choice). **Stability: Stable.** **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-exporter -The `SpanExporter` interface is what protocol-specific exporters implement to plug into the SDK; it -minimizes exporter implementation burden — an exporter is meant to be "primarily a simple telemetry -data encoder and transmitter." Each implementation **MUST** document its own concurrency +Protocol-specific exporters implement the `SpanExporter` interface to plug into the SDK. The +interface keeps exporter implementation burden low: an exporter is meant to be "primarily a simple +telemetry data encoder and transmitter." Each implementation **MUST** document its own concurrency requirements. **Interface:** MUST support `Export`, `Shutdown`, `ForceFlush` (typically one interface per signal, e.g. `SpanExporter`). -| Method | Contract | -| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `Export(batch)` | Exports a batch of readable spans (serialize + transmit, typically). "should not be called concurrently with other Export calls for the same exporter instance" — but the underlying transmission work MAY still happen concurrently under the hood (language-specific). **MUST NOT block indefinitely** — must time out with `Failure` after a reasonable upper limit. Retry logic is the **exporter's** responsibility, not the built-in processors' — "The default SDK's Span Processors SHOULD NOT implement retry logic" since it's protocol/backend-specific (e.g. [OTLP](https://opentelemetry.io/docs/specs/otlp/) defines its own send/retry logic). Returns an `ExportResult`: `Success` ("the batch has been successfully exported" — e.g. delivered over the wire) or `Failure` ("exporting failed. The batch must be dropped," e.g. unserializable data). | -| `Shutdown()` | Opportunity for exporter cleanup; called on SDK shutdown. Should be called only once; after it, subsequent `Export` calls are disallowed and **should return `Failure`**. Should not block indefinitely even if flushing to an unavailable destination. | -| `ForceFlush()` | Hint that any spans already received prior to the call should be exported "as soon as possible, preferably before returning." SHOULD report success/failure/timeout; SHOULD only be called when "absolutely necessary" (same FaaS-suspend rationale as the processor's `ForceFlush`); SHOULD complete/abort within some timeout. | +| Method | Contract | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Export(batch)` | Exports a batch of readable spans (typically serialize + transmit). It "should not be called concurrently with other Export calls for the same exporter instance", though the underlying transmission MAY still happen concurrently (language-specific). **MUST NOT block indefinitely**: it must time out with `Failure` after a reasonable upper limit. Retry logic belongs to the **exporter**, not the built-in processors: "The default SDK's Span Processors SHOULD NOT implement retry logic", since retry is protocol/backend-specific (e.g. [OTLP](https://opentelemetry.io/docs/specs/otlp/) defines its own send/retry logic). Returns an `ExportResult`: `Success` ("the batch has been successfully exported", e.g. delivered over the wire) or `Failure` ("exporting failed. The batch must be dropped", e.g. unserializable data). | +| `Shutdown()` | Opportunity for exporter cleanup; called on SDK shutdown. Should be called only once. Afterwards, `Export` calls are disallowed and **should return `Failure`**. Should not block indefinitely, even if flushing to an unavailable destination. | +| `ForceFlush()` | Hint that spans already received before the call should be exported "as soon as possible, preferably before returning." SHOULD report success/failure/timeout. SHOULD only be called when "absolutely necessary" (same FaaS-suspend rationale as the processor's `ForceFlush`). SHOULD complete/abort within some timeout. | **Concurrency:** `ForceFlush` and `Shutdown` **MUST** be safe to call concurrently on a `SpanExporter`. @@ -745,12 +749,12 @@ e.g. `SpanExporter`). **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#concurrency-requirements -| Component | Requirement | -| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| TracerProvider | Tracer creation, `ForceFlush`, `Shutdown` MUST be safe for concurrent calls. | -| Sampler | `ShouldSample` and `GetDescription` MUST be safe for concurrent calls. | -| Span processor | All methods MUST be safe for concurrent calls. | -| Span Exporter | `ForceFlush` and `Shutdown` MUST be safe for concurrent calls (per-instance `Export` itself must NOT be called concurrently — see above). | +| Component | Requirement | +| -------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| TracerProvider | Tracer creation, `ForceFlush`, `Shutdown` MUST be safe for concurrent calls. | +| Sampler | `ShouldSample` and `GetDescription` MUST be safe for concurrent calls. | +| Span processor | All methods MUST be safe for concurrent calls. | +| Span Exporter | `ForceFlush` and `Shutdown` MUST be safe for concurrent calls (per-instance `Export` itself must NOT be called concurrently; see above). | --- @@ -760,12 +764,12 @@ e.g. `SpanExporter`). **Source:** https://opentelemetry.io/docs/specs/otel/trace/sdk/#self-observability "The Tracing SDK SHOULD support -[SDK self-observability](https://opentelemetry.io/docs/specs/otel/self-observability/)" — i.e. the -SDK itself emitting metrics/logs about its own internal health (queue depth, export failures, -etc.), distinct from the application's own OTel-instrumented telemetry. This is the spec-level -counterpart to what Maple's own `apps/ingest` does manually today (its `PeriodicReader`-pushed -operational metrics — see CLAUDE.md "Self-Observability" — mirror the shape this section describes -for SDKs generally). +[SDK self-observability](https://opentelemetry.io/docs/specs/otel/self-observability/)": the SDK +emits metrics/logs about its own internal health (queue depth, export failures, etc.), separate +from the application's own telemetry. This is the spec-level counterpart to what `apps/ingest` +does by hand today: its operational metrics (`apps/ingest/src/metrics.rs`, +`apps/ingest/src/usage_metrics.rs`, pushed via a `PeriodicReader`) follow the shape this section +describes for SDKs. --- @@ -777,10 +781,10 @@ for SDKs generally). | Env var | Default | Accepted values / meaning | | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `OTEL_TRACES_SAMPLER` | `parentbased_always_on` | `always_on` (`AlwaysOnSampler`), `always_off` (`AlwaysOffSampler`), `traceidratio` (`TraceIdRatioBased`), `parentbased_always_on` (`ParentBased(root=AlwaysOnSampler)`), `parentbased_always_off` (`ParentBased(root=AlwaysOffSampler)`), `parentbased_traceidratio` (`ParentBased(root=TraceIdRatioBased)`), `parentbased_jaeger_remote` (`ParentBased(root=JaegerRemoteSampler)`), `jaeger_remote` (`JaegerRemoteSampler`), `xray` (AWS X-Ray centralized sampling — third-party). | -| `OTEL_TRACES_SAMPLER_ARG` | unset | Meaning depends on the selected sampler. For `traceidratio` / `parentbased_traceidratio`: "Sampling probability, a number in the [0..1] range, e.g. '0.25'. Default is 1.0 if unset." For `jaeger_remote` / `parentbased_jaeger_remote`: a comma-separated list of `endpoint`, `pollingIntervalMs`, `initialSamplingRate`. | +| `OTEL_TRACES_SAMPLER` | `parentbased_always_on` | `always_on` (`AlwaysOnSampler`), `always_off` (`AlwaysOffSampler`), `traceidratio` (`TraceIdRatioBased`), `parentbased_always_on` (`ParentBased(root=AlwaysOnSampler)`), `parentbased_always_off` (`ParentBased(root=AlwaysOffSampler)`), `parentbased_traceidratio` (`ParentBased(root=TraceIdRatioBased)`), `parentbased_jaeger_remote` (`ParentBased(root=JaegerRemoteSampler)`), `jaeger_remote` (`JaegerRemoteSampler`), `xray` (AWS X-Ray centralized sampling, third-party). | +| `OTEL_TRACES_SAMPLER_ARG` | unset | Meaning depends on the selected sampler. For `traceidratio` / `parentbased_traceidratio`: "Sampling probability, a number in the [0..1] range, e.g. '0.25'. Default is 1.0 if unset." For `jaeger_remote` / `parentbased_jaeger_remote`: a comma-separated list of `endpoint`, `pollingIntervalMs`, `initialSamplingRate`. | | `OTEL_TRACES_EXPORTER` | `otlp` | `otlp`, `zipkin`, `console`, `logging` (deprecated alias), `none`. | -| `OTEL_PROPAGATORS` | `tracecontext,baggage` | Ordered list of propagators to install globally (not trace-specific, but governs how `SpanContext`/`Baggage` cross process boundaries). | +| `OTEL_PROPAGATORS` | `tracecontext,baggage` | Ordered list of propagators to install globally. Not trace-specific, but it governs how `SpanContext`/`Baggage` cross process boundaries. | --- diff --git a/docs/persistence.md b/docs/persistence.md index 0199525355..d50bc3bf3f 100644 --- a/docs/persistence.md +++ b/docs/persistence.md @@ -6,7 +6,7 @@ Maple stores relational application state in PostgreSQL with a schema defined by ## Runtime modes - **Production:** the PlanetScale Postgres `main` branch. Cloudflare Workers - connect through the `MAPLE_DB` Hyperdrive binding; the application never opens the direct + connect through the `MAPLE_DB` Hyperdrive binding. The application never opens the direct administrative connection. - **Wrangler development:** Docker Postgres on port 5499 through Hyperdrive's `localConnectionString`. @@ -16,38 +16,39 @@ Maple stores relational application state in PostgreSQL with a schema defined by need `Database` fail normally; DB-free routes such as health checks continue to work. Application code keeps timestamps as epoch-millisecond numbers and converts at the Drizzle -boundary — use `msToDate` / `dateToMs` from `packages/backend/src/platform/time.ts` rather than bare +boundary. Use `msToDate` / `dateToMs` from `packages/backend/src/platform/time.ts` instead of bare `new Date(ms)` / `.getTime()`, including inside Promise-land helpers. ## Connections on Workers -One connection pool per invocation — request, cron tick, or Workflow run — created lazily on the first -query and closed at the boundary. This is Cloudflare's documented Hyperdrive shape, and -`makePgConnectionScope` (`packages/backend/src/platform/pg-connection-scope.ts`) is the only implementation -of it: `withPgConnectionScope` installs a scope around each worker's request handler and cron tick, and -`executeOnFreshPgClient` is the same scope one call long for entry points that have none. +One connection pool per invocation (request, cron tick, or Workflow run), created lazily on the +first query and closed at the boundary. This is Cloudflare's documented Hyperdrive shape. +`makePgConnectionScope` (`packages/backend/src/platform/pg-connection-scope.ts`) is the only +implementation of it. `withPgConnectionScope` installs a scope around each worker's request +handler and cron tick. `executeOnFreshPgClient` is the same scope, one call long, for entry points +that have none. Workers tie TCP sockets to the invocation that opened them, so a connection may be reused freely -within one but must never outlive it. Two settings carry hard-won history: +within one but must never outlive it. -The driver is node-postgres through `@effect/sql-pg`: one `pg.Pool` per invocation, built lazily -by `createMaplePgPool` (`packages/db/src/client.ts`) and handed to `PgClient.fromPool` rather than -`PgClient.make`, because `make` probes with `SELECT 1` at acquire. Two settings carry hard-won +The driver is `@effect/sql-pg`'s own pooled client: `makeMaplePgClient` (`packages/db/src/client.ts`) +calls `PgClient.make` with no probe, so building it opens no socket. Two settings carry hard-won history: -- **`max: 5`** — Cloudflare's documented value. `max` is a ceiling, not a reservation: the pool - opens a second socket only when a second statement is genuinely in flight. It was 1 for one day - on the theory that Postgres should hold at most one of the Worker's six outbound slots, which - serialized every statement in a cron tick behind one connection (`SELECT actors` p50 928ms → +- **`MAX_CONNECTIONS = 5`**, Cloudflare's documented value. It is a ceiling, not a reservation: + the pool opens a second socket only when a second statement is in flight. It was 1 for one day, + on the theory that Postgres should hold at most one of the Worker's six outbound slots. That + serialized every statement in a cron tick behind one connection (`SELECT actors` p50 928ms to 5687ms at flat volume). -- **A bounded dial** (10s, `connectionTimeoutMillis` on each `Client`, never on the `Pool`) — - unset, a stalled dial hangs for the whole invocation and lands with no `error.type` to classify. - On the pool the same option also times out waiting for a free client, so a fan-out wider than - `max` would fail against a healthy server. A dial that hits the bound carries no driver code and - lands as `error.type = ConnectionError` (`postgres-errors.ts` classifies code-less acquire - failures); a refused one carries the socket's own code (`ECONNREFUSED`). The bound is generous and single: a - 2s cap alone once took production 5xx from 0.06% to 5.01%, and the retry ladder that followed - existed only to compensate for it. +- **A bounded dial** (`CONNECT_TIMEOUT_SECONDS = 10`, passed as the driver's `connectTimeout`). + Unset, a stalled dial hangs for the whole invocation and lands with no `error.type` to classify. + The driver applies it to one connection's connect, TLS and auth, never to the wait for a free + connection, so a fan-out wider than the pool queues instead of failing. A dial that hits the + bound carries no driver code and lands as `error.type = ConnectionError` + (`postgres-errors.ts` classifies code-less acquire failures). A refused one carries the + socket's own code (`ECONNREFUSED`). The bound is generous and single: a 2s cap once took + production 5xx from 0.06% to 5.01%, and the retry ladder that followed existed only to + compensate for it. ## Local development @@ -73,7 +74,7 @@ Review the generated folder in `packages/db/drizzle/`: one `_/` holding `migration.sql` and the DDL `snapshot.json` (drizzle-kit v1 layout, no journal). The migrator orders folders by name and applies every folder the database has not recorded. A hand-authored migration (data backfill, publication change) still needs a folder with both -files: run `drizzle-kit generate --custom --name ` to scaffold it rather than creating the +files. Scaffold it with `drizzle-kit generate --custom --name ` instead of creating the folder by hand, so the snapshot chain stays intact. Useful local commands: @@ -88,9 +89,9 @@ bun run --cwd packages/db db:studio ## Deployment and tests -The prd deploy applies migrations: `alchemy.run.ts` declares the instance's PlanetScale `main` branch -(`maple` on `prd`, `maple-eu` on `prd-eu`) as `Planetscale.PostgresBranch` with `migrations` pointed -at `packages/db/drizzle`, and the api, ai and +The prd deploy applies migrations. `declareMapleDb` in `alchemy.run.ts` declares the instance's +PlanetScale `main` branch (database `maple` on `prd`, `maple-eu` on `prd-eu`) as +`Planetscale.PostgresBranch` with `migrations` pointed at `packages/db/drizzle`. The api, ai and alerting Workers carry its name in their env so they upload after it. Bookkeeping is alchemy's `__alchemy_migrations`; `drizzle.__drizzle_migrations` was copied in once and is frozen, so never run `drizzle-kit migrate` against prd. The deploy migrates as a temporary role that is dropped with @@ -111,11 +112,12 @@ its id sits in the task env so a replaced role rolls the fleet onto the new secr deletes the old role. `MAPLE_INGEST_PG_URL` in Infisical remains only for stages that deploy a fleet without a database branch (PR previews). -Electric's is declared too, on both instances: `Planetscale.PostgresRole("electric", { withReplication: -true })`, whose direct 5432 URL is the task's `DATABASE_URL` (`docs/electric-sync.md`). `withReplication` -rides Maple's alchemy patch until [alchemy-run/alchemy#1777](https://github.com/alchemy-run/alchemy/pull/1777) -ships; alchemy renders every role URL with `sslmode=verify-full`, which neither ECS client accepts, so -`pgUrlRequireSsl` in `@maple/infra/aws` rewrites it for both. +Electric's is declared too, on both instances: `Planetscale.PostgresRole("electric-db-role", { +withReplication: true })`, whose direct 5432 URL is the task's `DATABASE_URL` (`docs/electric-sync.md`). +`withReplication` rides Maple's alchemy patch until +[alchemy-run/alchemy#1777](https://github.com/alchemy-run/alchemy/pull/1777) ships. Alchemy renders +every role URL with `sslmode=verify-full`, which neither ECS client accepts, so `pgUrlRequireSsl` in +`@maple/infra/aws` rewrites it for both. The EU instance's Worker credentials are declared the same way: `declareMapleDb` in `alchemy.run.ts` mints one role per consumer on `maple-eu` and a Hyperdrive config on each role's direct origin, and @@ -123,7 +125,8 @@ the Workers bind them from their props. No dashboard config and no hand-minted r (`resolveDatabaseMode` is `"declared"`); the US prd keeps its dashboard-managed configs, bound by id. The deploy reads `PLANETSCALE_API_TOKEN_ID` / `PLANETSCALE_API_TOKEN` / -`PLANETSCALE_ORGANIZATION` from Infisical prod; `bun dev` leaves the PlanetScale provider out. +`PLANETSCALE_ORGANIZATION` from the instance's Infisical environment. `bun dev` leaves the +PlanetScale provider out. PGlite applies the same bundled migrations while its layer is built. The test harness caches a fresh migrated PGlite snapshot and restores it per test, so integration tests exercise the @@ -132,13 +135,15 @@ PostgreSQL schema without a shared server. ## Tinybird Materialized Views and TTL Coupling Raw `traces` and `logs` are retained for 30 days. Projection targets that preserve one row per -span or log use the same 30-day ceiling; aggregate targets intentionally retain rollups for 90 or -365 days. The TTL belongs to the target datasource in +span or log use the same 30-day ceiling. Aggregate targets retain rollups for 90 or 365 days on +purpose. The TTL belongs to the target datasource in `packages/domain/src/tinybird/datasources.ts`, not to the materialized-view definition. Two operational consequences: -1. **Backfill ceiling.** When deploying a new MV with `POPULATE`, you can only backfill data the source table still has — anything aged past the source TTL is lost. Plan deploys before any TTL reduction. +1. **Backfill ceiling.** A new MV deployed with `POPULATE` can only backfill data the source + table still has. Anything aged past the source TTL is lost. Plan deploys before any TTL + reduction. 2. **TTL changes require a target audit.** Keep row-level projections in lockstep with their raw source. Preserve the independently documented retention of hourly and error rollups unless the @@ -162,8 +167,10 @@ LIMIT 50 Decision rule: -- p99 < 1K distinct → keep `SpanName` in MV dimensions (current setup) -- p99 1K–10K → keep but only route to MV when query has a `SpanName` filter -- p99 > 10K → drop `SpanName` from MV dimensions; group-by-span-name queries fall back to raw `traces` +- p99 < 1K distinct: keep `SpanName` in MV dimensions (current setup). +- p99 1K to 10K: keep it, but route to the MV only when the query has a `SpanName` filter. +- p99 > 10K: drop `SpanName` from MV dimensions. Group-by-span-name queries fall back to raw + `traces`. -High cardinality is usually a tenant emitting per-request data in span names (anti-pattern, but seen). Address at the source if found. +High cardinality usually means a tenant is putting per-request data in span names. It is an +anti-pattern, but it happens. Fix it at the source. diff --git a/docs/pr-review-agent-plan.md b/docs/pr-review-agent-plan.md index 3492e4bbba..b8369920f7 100644 --- a/docs/pr-review-agent-plan.md +++ b/docs/pr-review-agent-plan.md @@ -43,11 +43,12 @@ superseded row is dropped. ### The agent (`apps/ai`) - **Mode.** `pr-review` is a `ChatMode` with tab prefix `pr-`. Its `AGENTS` entry carries - `PR_REVIEW_SYSTEM_PROMPT`, `PR_REVIEW_BUDGET` (60 tool calls, 8 minutes, 800k tokens, 48k - completion reserve) and `autonomousPermission: PR_REVIEW_RULESET`, which `profileForTurn` uses - for the unattended pass only. + `PR_REVIEW_SYSTEM_PROMPT`, `PR_REVIEW_BUDGET` (`apps/ai/src/chat/budgets.ts`: 500 tool calls, + 10 minutes, 64M tokens, 48k completion reserve, sized as runaway guards since #1053) and + `autonomousPermission: PR_REVIEW_RULESET`, which `profileForTurn` uses for the unattended pass + only. - **Tools.** `PR_REVIEW_RULESET` denies everything and allows the names in `PR_REVIEW_TOOLS`: - `pr_changed_files`, `pr_file_diff`, the four `sandbox_*` tools, `list_source_repositories`, + `pr_changed_files`, `pr_context`, `pr_file_diff`, the four `sandbox_*` tools, `list_source_repositories`, `search_source_code`, `read_source_file`, and the read-only telemetry tools `list_services`, `get_service_top_operations`, `explore_attributes`, `search_traces`, `service_map`, `list_metrics`, `audit_setup` and `get_instrumentation_recommendations`. `permissions.test.ts` @@ -55,8 +56,8 @@ superseded row is dropped. - **Diff tools.** `pr_changed_files` and `pr_file_diff` are internal tools (`apps/ai/src/mcp/tools/pull-request.ts`) over GitHub's pull request files endpoint, so they work locally, on previews and for fork PRs, where the prd-only sandbox does not. `pr_changed_files` - classifies each file (source, test, generated, docs, config, infra, tooling) and states the call - budget, `reviewCallBudget`: two calls per reviewable file plus four, between 6 and 40. + classifies each file (source, test, generated, docs, config, infra, tooling) and counts the + reviewable ones. It no longer states a call budget: `reviewCallBudget` was removed after #1053. `pr_file_diff` takes up to 20 `paths` per call and annotates new-side line numbers, so a finding cites a line GitHub accepts. - **Completion.** `submit_review` is offered only to the review's unattended pass; a person's @@ -65,9 +66,9 @@ superseded row is dropped. `normalizePrReviewSubmission` drops any finding without a path or a positive line, and any observability finding without a check id from the `maple-audit` grammar (`SPAN-03`, `REN-DUAL`). It derives the verdict: `issues` exactly when a warn or critical finding survives, - otherwise the model's `not_applicable` or `clean`. A pass that stops without submitting gets the shared close-out turn - (`withToolTranscript` in `apps/ai/src/chat/close-out.ts`), which sees the tool calls and results - the pass gathered. + otherwise the model's `not_applicable` or `clean`. A pass that stops without submitting gets the + shared close-out turn (`withToolTranscript` in `apps/ai/src/chat/close-out.ts`), which sees the + tool calls and results the pass gathered. - **Billing.** The turn is metered with source `review` and an idempotency key of the review id and the turn. @@ -100,22 +101,30 @@ needs work (50+) or poor. A single warning scores 90 but grades good, so the hea gap excellent. The score is computed from the findings rather than asked of the model, so the same gaps always score the same, and it is stored in `pr_reviews.score`. +The headline is now a confidence level from 1 to 5 (`confidencePrReview`, same file), from "do not +merge" to "safe to merge". It starts from the quality score, deducts for missing tests, a high-risk +area and unobservable new work, and is capped by findings. The quality score is shown beneath it. + ### Publishing to GitHub `PrReviewService.submitReview` stores the report, then calls `VcsProviderClient.publishPullRequestReview`, which posts, in order: -1. **A check run** named `Maple / review` on the head SHA, titled `/100 · `, - concluding `neutral` for gaps and `success` otherwise, never `failure`. An installation that - has not granted `checks: write` answers 403, and the review is posted without the check run. A - rate-limited 403, which carries a retry time, fails the publish instead. -2. **One summary comment**, always, found again by its hidden `` line and - authored by the App, and edited in place on later pushes. It carries the score, grade, verdict, - counts, summary, findings linked to their lines at the head SHA, what to change, and the - coverage table. The comment and the check summary share `renderReviewMarkdown`. -3. **A `COMMENT` review** with the findings above `info` inline, only when there are any. GitHub - refuses the whole review with a 422 when one line is outside the diff; the inline notes are then - dropped, since the summary comment already carries every finding. +1. **A check run** named `Maple / review` on the head SHA, titled `Confidence /5 · `. + It concludes `success` only when the review finished, left nothing to address and did not + rate confidence 3 or lower; otherwise `neutral`, never `failure`. An installation that has not granted + `checks: write` answers 403, and the review is posted without the check run. A rate-limited + 403, which carries a retry time, fails the publish instead. +2. **One summary comment per review**, always, authored by the App and found again by its hidden + `` line (`prReviewCommentMarker`). A "reviewing" + notice is replaced by that review's result; the next review of the pull request starts its own + comment (#1051). It carries the confidence, verdict, counts, summary, findings linked to their + lines at the head SHA, what to change, and the coverage table. The comment and the check + summary share `renderReviewMarkdown`. +3. **A `COMMENT` review** with the findings at or above the repository's inline threshold (`warn` + by default) inline, only when there are any. GitHub refuses the whole review with a 422 when + one line is outside the diff; the inline notes are then dropped, since the summary comment + already carries every finding. The publisher takes the PR identity from the `pr_reviews` row bound at trigger time, never from tool arguments, so text in a diff cannot redirect where the review is posted. A refused post lands @@ -139,7 +148,8 @@ The trigger (`PrReviewService.onPullRequestEvent`) and `submitReview` spans carr `maple.pr_review.id`, `.outcome`, `.skip_reason`, `.started_today`, `.superseded`, `.verdict`, `.score`, `.findings`, `.coverage`, `.partial`, `.published` and `.stale_submission`, and the sink handler records `.trigger`. The `submit_review` tool span adds `.filled_fields`, `.filled_count` -and `.dropped_findings`, and the turn records `.closed_out` when the runner had to close it out. The publish span carries `vcs.pull_request.check_run_id`, `.comment_id`, +and `.dropped_findings`, and the turn records `.closed_out` when the runner had to close it out. +The publish span carries `vcs.pull_request.check_run_id`, `.comment_id`, `.review_id`, `.check_run_skipped` and `.review_comments_rejected`. The turn itself is an ordinary chat session, so Agent Sessions shows its transcript. @@ -214,9 +224,9 @@ Cost baseline from the first iteration (2026-09-22, `z-ai/glm-5.3-flash:nitro`): | #977 (3 files, web) | 374k input tokens, 23 calls | 52k, 5 calls, correct `100/100` | | Gap fixture (1 file, 4 planted gaps) | | 42k, 5 calls, all four found, `60/100` | -What moved it: `pr_changed_files` states a call budget (`reviewCallBudget`), the prompt reads the -repository's conventions before the diffs and verifies only what a finding depends on, and -`tooling` files are not reviewed. +What moved it: `pr_changed_files` stated a call budget (`reviewCallBudget`, since removed), the +prompt reads the repository's conventions before the diffs and verifies only what a finding +depends on, and `tooling` files are not reviewed. `--prompt-file` replaces the system prompt for that run (an `agent` override on `ChatRunInput`), so a prompt change can be compared against the committed one before it is edited in. @@ -226,6 +236,10 @@ The full path (webhook → trigger → Durable Object → GitHub post) is covere ## Limits and what is next +Status (2026-09-25): the full review agent below shipped fan-out for large pull requests, finding +dedupe across pushes, a reviews list and per-repository config. The abandoned-row sweep is still +open. + - **Sandbox is prd only.** Local and preview runs use the API diff tools and `read_source_file`. Fork PRs get the API path only; deep context needs `git fetch origin refs/pull/N/head` in `cloneScript`. @@ -258,10 +272,10 @@ one product behind the same `prreview` rollout flag. Observability is one lens o - **Context.** One `pr_context` call: commits, what people and other bots already said (the App's own comments excluded), and the head's checks, failing first. - **Large pull requests.** Past 12 reviewable files the pass calls `review_files` per group of - related files, in parallel. Each group runs a child `pr-review-worker` agent through - `effect-agent/subagent` `Subagent`: the parent's own read-only toolkit (the grant is exactly - those tools, depth one), 16 calls and 4 minutes each, at most 8 children and 4 at a time, reserved - from the parent's budget. The child answers findings one per line; the parent verifies and files. + related files, in parallel (`apps/ai/src/chat/review-fanout.ts`). Each group of up to 12 files + runs a child `pr-review-worker` agent through `effect-agent/subagent` `Subagent`: the parent's + own read-only toolkit (the grant is exactly those tools, depth one), 100 calls and 4 minutes + each, at most 12 children and 4 at a time, reserved from the parent's budget. The child answers findings one per line; the parent verifies and files. - **Quality gate.** `bun run --cwd apps/ai review:eval mine` blames each `fix:` commit's changed lines back to the squash-merged PR that wrote them; a person keeps the real bugs in `apps/ai/scripts/pr-review-eval/corpus.json`. `review:eval run --model ` reviews every case and @@ -336,4 +350,4 @@ Permissions: `Pull requests: Read and write`, `Checks: Read and write`, `Content - Failed-check log tails in `pr_context` need `actions: read`. - Fork pull requests use the API diff path only; the sandbox cannot fetch `refs/pull/N/head` yet. -- The eval corpus has five cases; widen it with `review:eval mine` before comparing models. +- The eval corpus is small; widen it with `review:eval mine` before comparing models. diff --git a/docs/product-events-dashboard-parity.md b/docs/product-events-dashboard-parity.md index 8a88965746..aa8be36642 100644 --- a/docs/product-events-dashboard-parity.md +++ b/docs/product-events-dashboard-parity.md @@ -1,11 +1,15 @@ -# Product events as a first-class dashboard source — plan +# Product events as a first-class dashboard source: plan Status: **phases 1 through 5 built 2026-09-13** on `feat/product-events-query-source` -(PR #877). Verified end to end against a seeded local Tinybird: the Top Events, Events by Page and -Recent Events tiles, a four-step person-stitched funnel built in the step editor, and an alert rule -prefilled from a product-event widget all ran through the real stack. Phase 6 (deleting the last -funnel special cases: `FunnelSource: "query_set"`, `isProductEventsFunnel` as a gate) is still open; -so is the `SignalEmptyState` wiring, which cannot be exercised on an org that has events. +(PR #877, merged to `main`). Verified end to end against a seeded local Tinybird: the Top Events, +Events by Page and Recent Events tiles, a four-step person-stitched funnel built in the step +editor, and an alert rule prefilled from a product-event widget all ran through the real stack. +Phase 6 (deleting the last funnel special cases: `FunnelSource: "query_set"`, +`isProductEventsFunnel` as a gate) is still open. The `SignalEmptyState` wiring is built +(`ProductEventsAbsentHint` in `query-panel.tsx`) but unverified, since it cannot be exercised on +an org that has events. The phase-1 query-set builders landed in +`packages/query-engine/src/ch/queries/product-events-explore.ts`; the funnel queries stay in +`product-events.ts`. ## Where it stood before this PR @@ -62,7 +66,7 @@ path uses the row's own `if(UserId != '', UserId, VisitorId)` for `uniq(persons) Each phase is independently shippable and typechecks on its own. Order matters: every later phase hangs off the `QuerySpec` arms from phase 1. -### Phase 1 — the engine: `product_events` as a `QuerySpec` source +### Phase 1: the engine, `product_events` as a `QuerySpec` source **Domain (`packages/domain/src/query-engine.ts`)** @@ -116,7 +120,7 @@ hangs off the `QuerySpec` arms from phase 1. `web-analytics-parity.clickhouse.e2e.test.ts` case asserting the timeseries `count` for `kind = navigation` equals `webAnalyticsPageviewsTimeseriesQuery` on the seeded data. -### Phase 2 — the draft: `product_events` in the query builder model +### Phase 2: the draft, `product_events` in the query builder model - `packages/query-model/src/query-draft.ts`: add `"product_events"` to `QUERY_BUILDER_DATA_SOURCES`; `ProductEventsQueryDraftSchema` = base fields + @@ -137,7 +141,7 @@ hangs off the `QuerySpec` arms from phase 1. - `packages/backend/src/dashboard-templates/helpers.ts`: accept the new source in `makeQuerySpec`. - Tests in `model.test.ts`: clause parsing table, group-by resolution, aggregation guard. -### Phase 3 — the builder UI +### Phase 3: the builder UI - `query-panel-shell.tsx`: delete the `QueryPanelSource = QueryBuilderDataSource | "product_events"` special case. `product_events` is now a `QueryBuilderDataSource`; `QUERY_BUILDER_PANEL_SOURCES` @@ -168,7 +172,7 @@ hangs off the `QuerySpec` arms from phase 1. render `SignalEmptyState` for `product_events` under the panel, as the explorers do. - `auto-contexts.ts`: `/analytics` already maps; nothing to do. -### Phase 4 — alerts +### Phase 4: alerts - `packages/backend/src/services/alerts/AlertRuleModel.ts`: `sampleCountStrategy` gains `"product_event_count"`; the evaluator (`makeQueryEngineEvaluate` / @@ -180,7 +184,7 @@ hangs off the `QuerySpec` arms from phase 1. - Alert templates: one "Signups dropped" example under `alert-templates/` is enough to prove the path; don't build a catalogue. -### Phase 5 — agents and public surfaces +### Phase 5: agents and public surfaces - `apps/ai/src/mcp/tools/query-data.ts`: `source` accepts `product_events`; new params `event_name`, `event_kind`, `host`, `page_path`; `metric` doc lists the five aggregations; @@ -195,7 +199,7 @@ hangs off the `QuerySpec` arms from phase 1. - v2 public API: `packages/domain/src/http/v2/telemetry-signals.ts` already lists the signal. No new v2 endpoint; product events are queried through the same widget/query-set routes. -### Phase 6 — retire the special case +### Phase 6: retire the special case - `product_events_funnel` route and `makeProductEventsFunnelDataSource` STAY: a funnel is a different query shape (windowFunnel + identity join), not a count. What goes is every place that diff --git a/docs/product-events-funnels.md b/docs/product-events-funnels.md index 9c5eff817e..88ed1dea58 100644 --- a/docs/product-events-funnels.md +++ b/docs/product-events-funnels.md @@ -1,18 +1,22 @@ -# Product events + funnels — plan +# Product events + funnels: plan Status: **implemented in code (2026-08-17), pending rollout.** Goal: answer "referral → signed up → started a plan" for any org (and for Maple itself), from browser, mobile and backend events, as a real funnel in the product rather than a hand-written `run_sql`. +Status (2026-09-25): the code is on `main`. Two details differ from the plan below: +`product_events` keeps 365 days (not 180), and `web_events` was dropped inside migration 0021 +rather than by a separate `drop_web_events` migration. + ## Rollout checklist (operator steps the code cannot do) 1. **Tinybird**: `bun run --cwd apps/api tinybird:deploy` creates `product_events`, `identity_links` and their MVs. Then populate both from their sources with an explicit `tb` - step (the SDK has no populate option — same caveat as 0014). Until populated, page views + step (the SDK has no populate option; same caveat as 0014). Until populated, page views read as zero on managed orgs. Old `web_events`/`web_events_mv` can be removed from the workspace afterwards. 2. **BYO ClickHouse**: migration 0021 bumps `clickHouseSchemaVersion` to 21 (`requiredForIngest` - default), so BYO orgs' ingest routing is un-ready until they apply schema. Deliberate — the + default), so BYO orgs' ingest routing is un-ready until they apply schema. This is deliberate: the gateway now writes the new `session_events` columns and `product_events` directly. 3. **Secrets** (api worker): `CLERK_WEBHOOK_SECRET`, `AUTUMN_WEBHOOK_SECRET` (both routes answer 503 until set); optional `MAPLE_PRODUCT_EVENTS_INGEST_KEY` (defaults to `MAPLE_INGEST_KEY`). @@ -25,9 +29,9 @@ as a real funnel in the product rather than a hand-written `run_sql`. 6. **Billing**: `bun run --cwd apps/api atmn push` (no package script exists; `atmn` is an `apps/api` dependency) so the `product_events` feature and its `startup` plan item exist in Autumn before the gateway starts reserving against it. Until pushed, Autumn answers - `allowed: false` for the unknown feature — a real denial, not an error, so `is_allowed`'s + `allowed: false` for the unknown feature. That is a real denial, not an error, so `is_allowed`'s fail-open does NOT rescue it. Neither event path rejects on `product_events` for exactly that - reason: usage is recorded fail-open and nothing is dropped — just unbilled. + reason: usage is recorded fail-open and nothing is dropped, just unbilled. ### Billing @@ -39,11 +43,11 @@ reserve → WAL enqueue → confirm/release shape as session starts (`metered_en rows in the batch (a browser `track()` call is the same unit as a server-side event) and keeps its entitlement REJECTION on `browser_sessions`. **Neither path rejects on `product_events`**: an exhausted (or un-provisioned) product-events allowance bills usage-based overage instead of 402-ing -a whole session transcript or a backend's buffered batch. Automatic session events (clicks, navigations, ...) stay unmetered. **Beta pricing -(2026-08-17): the `startup` item is `unlimited: true` with no price** — usage is tracked in Autumn -(and shown on the billing page as "Unlimited · free during beta") so real volumes are known before -a price is set. To start charging, swap `unlimited` for an `included` allowance + `price` in -`apps/api/autumn.config.ts` (the commented example is $0.05 per 1,000 events past 1M/month) and +a whole session transcript or a backend's buffered batch. Automatic session events (clicks, +navigations, ...) stay unmetered. **Beta pricing (2026-08-17): the `startup` item is +`unlimited: true` with no price.** Usage is tracked in Autumn (and shown on the billing page as +"Unlimited · free during beta") so real volumes are known before a price is set. To start +charging, swap `unlimited` for an `included` allowance + `price` in `apps/api/autumn.config.ts` (the commented example is $0.05 per 1,000 events past 1M/month) and push. `DailySpendService` emits `DailyVolume.productEvents` from `product_events` (`Kind != 'navigation'`, matching what the gateway meters). @@ -53,7 +57,7 @@ Verified against production on 2026-08-17: - `web_events` (`packages/domain/src/tinybird/datasources.ts`, `packages/query-engine/src/ch/tables.ts`) already exists as "the funnel substrate": time-sorted `$pageview` + `track()` rows, `EventName` - as the step key, `Seq` tiebreak, `idx_event_name` skip index. **Nothing queries it as a funnel** — + as the step key, `Seq` tiebreak, `idx_event_name` skip index. **Nothing queries it as a funnel**: no `windowFunnel`/`sequenceMatch` anywhere in the repo; the `funnel` widget is a name/value bar. - Referral (`Referrer`, `ReferrerHost`, `Utm*`), `VisitorId`, `UserId`, `GroupId` live only on `session_replays`. `web_events` carries `SessionId` only, so any cross-session question is a join. @@ -107,19 +111,19 @@ Decisions baked in: - **Person key on the row.** The SDK already resolves identity lazily when session rows post and spans start (`clerk-auth-bridge.tsx`, `identify()`); session events get the same stamping in `packages/browser-session/src/events/events-sink.ts` (`visitor_id`, `user_id`, `group_id` on each - NDJSON line). The MV copies them through. No MV-side join against `session_replays` — that would + NDJSON line). The MV copies them through. No MV-side join against `session_replays`; that would depend on insert ordering. - Requires adding `VisitorId`/`UserId`/`GroupId` (`DEFAULT ''`) to `session_events` too (migration + Tinybird forward query, same shape as the 0012 `Attributes` widening). - Rows from older SDKs arrive with `''`; the funnel falls back to a `session_replays` lookup for - those, or simply excludes them — pick "exclude + show coverage" (matches how the analytics page + those, or simply excludes them. Pick "exclude + show coverage" (matches how the analytics page already treats sessions without the 0011 block). - **Sorting key**: `Timestamp` second so time ranges are a primary-index scan (the reason `web_events` exists), then `VisitorId` so a per-person `windowFunnel` groups over contiguous rows. Keep `SessionId`/`Seq` for stable step order. - **Retention 180d**, not 30. Table is tiny (13 distinct event names, thousands of rows/month in the dogfood org). A referral→paid funnel spans weeks; 30d loses the tail. Browser rows can never - be rebuilt past 30d (source TTL) — accepted; direct-ingested rows have no source and are the + be rebuilt past 30d (source TTL), which is accepted; direct-ingested rows have no source and are the primary copy. `retention-matrix.test.ts` pins this; update it deliberately. - **`Source` column** so a re-run of the browser backfill can `DELETE WHERE Source = 'browser'` instead of `TRUNCATE` (which would destroy direct-ingested rows). This replaces the @@ -127,7 +131,7 @@ Decisions baked in: - Name: `product_events`. Not `events` (ambiguous next to `session_events`/`error_events`), not `analytics_events` (the page is called Web Analytics; that name will age the same way). -### 2. Direct ingest — `POST /v1/events` +### 2. Direct ingest: `POST /v1/events` Ingest gateway (`apps/ingest/src/main.rs`), NDJSON like `/v1/sessionEvents`, authenticated by ingest key (org from the key, never the body). Body per line: @@ -162,13 +166,13 @@ ingest key (org from the key, never the body). Body per line: `apps/api`: -- `ProductEventsService.track({ userId, groupId, name, attributes })` — Effect service, posts to +- `ProductEventsService.track({ userId, groupId, name, attributes })`: an Effect service that posts to the ingest gateway with Maple's own dogfood ingest key (same org that `maple-web`/`maple-landing` report into). `Schema.TaggedError` for the failure; never fails the caller (fire-and-forget on a - forked fiber, `root: true` — see the ambient-span footgun). + forked fiber, `root: true`; see the ambient-span footgun). - **`signup_completed`**: Clerk webhook `user.created` (Svix-signed) → new route - `apps/api/src/routes/webhooks/clerk.http.ts`. `user_id` from the payload; `visitor_id` unknown - server-side — stitching to the marketing visit happens via identity (below), not on this row. + `apps/api/src/routes/webhooks/clerk.http.ts`. `user_id` from the payload; `visitor_id` is unknown + server-side. Stitching to the marketing visit happens via identity (below), not on this row. Fallback/complement: `apps/web` fires `signup_completed` client-side on the first `/quick-start` landing after Clerk `createdAt` is within N minutes; keep the webhook as truth. - **`plan_started`**: webhook from the billing side (Autumn if it offers one for @@ -183,7 +187,7 @@ ingest key (org from the key, never the body). Body per line: The person key for a funnel step is `if(UserId != '', UserId, VisitorId)`. Anonymous marketing visits (VisitorId only) and server events (UserId only) meet through **`identity_links`**: an MV over `session_replays` (and later mobile identify calls) emitting `(OrgId, VisitorId, UserId, -FirstSeen)` — ReplacingMergeTree keyed `(OrgId, VisitorId, UserId)`. The funnel query resolves +FirstSeen)`, a ReplacingMergeTree keyed `(OrgId, VisitorId, UserId)`. The funnel query resolves each row's person key via a `LEFT JOIN identity_links` (or `dictGet` on BYO CH) so a visitor's pre-signup rows and their post-signup UserId rows collapse into one person. Cheap: the link table is one row per (visitor,user) pair. @@ -204,15 +208,15 @@ windowSeconds, filters })` → per-step `count`, `conversion_from_prev`, `conver Step = `{ eventName } | { pagePath, host? } | { referrerHost | utmSource | ... }`. A session-dimension step (referral) becomes step 0's condition through the person's first `session_replays` row; event steps are `EventName = …` conditions on `product_events`. -- `productEventsFunnelBreakdownQuery` — same, `GROUP BY` one dimension (`UtmSource`, +- `productEventsFunnelBreakdownQuery`: same, `GROUP BY` one dimension (`UtmSource`, `ReferrerHost`, `Country`, `Attributes[k]`). -- `productEventNamesQuery` — the step picker's autocomplete (names + counts, 30d). +- `productEventNamesQuery`: the step picker's autocomplete (names + counts, 30d). - Every query keeps `$.OrgId.eq(param.string("orgId"))`; register in `benchmark/catalog.ts` and add parity coverage in the ClickHouse e2e like `web-analytics-parity.clickhouse.e2e.test.ts` - (seed dates must be now-relative — TTL). + (seed dates must be now-relative because of the TTL). - `apps/api/src/routes/internal/query-engine.http.ts` fallback message and the `useWebEvents` switch get renamed (`useProductEvents`), same "absent table → raw - `session_events`" degrade for page views; funnels **require** the table (no raw fallback — + `session_events`" degrade for page views; funnels **require** the table (no raw fallback: server/mobile rows only exist there). ### 6. Surfaces @@ -228,7 +232,7 @@ windowSeconds, filters })` → per-step `count`, `conversion_from_prev`, `conver ## Migration / rename sequence Order matters because `web_events` has no dedup and `session_events` still holds every browser -row for 30d — the rename is a rebuild, not a `RENAME TABLE`. +row for 30d. The rename is a rebuild, not a `RENAME TABLE`. 1. Migration `0021_product_events` (BYO CH) + Tinybird datasource `product_events` + MV `product_events_mv` (from `session_events`, with the new columns), backfill spec from @@ -237,7 +241,7 @@ row for 30d — the rename is a rebuild, not a `RENAME TABLE`. 2. Local CLI: `local-schema-v11.sql`, `local-store-migrations/v10-to-v11-product-events.ts` (+ test), bump the schema-version gate. 3. SDKs stamp identity on session events (`browser-session` events-sink; effect-sdk client - `track.ts`). Backwards compatible — defaults cover old builds. + `track.ts`). Backwards compatible: defaults cover old builds. 4. Ingest: `/v1/events`, mappings, signal, env; deploy. 5. Query engine: flip page-view readers `web_events → product_events`; add funnel queries; catalog + parity tests; `useWebEvents` rename. @@ -268,9 +272,9 @@ missing only the plan-start step. The funnel widget's definition is edited as its **query panel** (`FunnelQueryPanel`, `apps/web/src/components/dashboard-builder/config/funnel-query-panel.tsx`): the panel's source select offers **Product events** beside Traces / Logs / Metrics (a query-builder source is the -original group-by breakdown drawn as a funnel). The product-events body is the ordered steps — -event steps take a per-step attribute filter (`plan = "pro" AND source = "cli"` → -`attributeEquals`), page steps an optional host — a population where-clause over the session +original group-by breakdown drawn as a funnel). The product-events body is the ordered steps +(event steps take a per-step attribute filter (`plan = "pro" AND source = "cli"` → +`attributeEquals`), page steps an optional host), a population where-clause over the session dimensions (`country = "DE" AND utm.source = "twitter"` → `display.funnel.filters`), and Count by / Window / Breakdown as add-ons. The definition persists on `display.funnel` and is mirrored flat into the `product_events_funnel` route params (`ProductEventsFunnelWidgetParams` in `@maple/query-model`), @@ -278,7 +282,7 @@ which the browser server function and the share API's route plan both decode. Wi route answers `{ name, value, group }` rows (top 6 groups by step-1 count) and the funnel chart draws one bar per group per step with a legend. -## Product events from traces — annotate in code (2026-09) +## Product events from traces: annotate in code (2026-09) The fourth feed into `product_events`, after browser (`session_events` MV), server and mobile (`POST /v1/events`). A team marks a span they already emit and it becomes a funnel step that @@ -286,7 +290,7 @@ links back to the request that performed it. ```ts span.setAttributes({ - "maple.product_event.name": "checkout_completed", // required — presence is the predicate + "maple.product_event.name": "checkout_completed", // required; presence is the predicate "maple.product_event.user_id": user.id, // optional identity "maple.product_event.group_id": org.id, "maple.product_event.visitor_id": anonId, @@ -295,7 +299,7 @@ span.setAttributes({ ``` **Every other attribute on the span becomes an event property by default.** Nothing has to be -declared to get started — whatever the team already sets on the span (`plan`, `order.total`, the +declared to get started. Whatever the team already sets on the span (`plan`, `order.total`, the full HTTP/DB semconv surface) lands in `Attributes` and is available to funnel breakdowns. The `maple.product_event.*` control keys are stripped, since they are already promoted to their own columns. @@ -313,14 +317,14 @@ Three tiers out of one mechanism rather than three modes to pick between: | `include` | `prop.*` | `Attributes` | | -------------- | -------- | ------------------------------------------------------------------ | -| absent | — | every span attribute | +| absent | unset | every span attribute | | absent | set | every span attribute, with the props overriding on a key collision | -| `"plan,seats"` | — | only `plan` and `seats` | -| `""` | set | only the props — the overwrite case | +| `"plan,seats"` | unset | only `plan` and `seats` | +| `""` | set | only the props (the overwrite case) | `include` switches on **key presence**, not on a non-empty value, which is what makes the empty string mean "no span attributes" rather than "no filter". There is no separate replace flag to get -wrong, and `mapUpdate(base, props)` argument order is the override rule — swapped, an override +wrong, and `mapUpdate(base, props)` argument order is the override rule. Swapped, an override would be discarded exactly when the key it meant to correct was already present. Verified against ClickHouse 26.2, all three tiers: @@ -331,7 +335,7 @@ Verified against ClickHouse 26.2, all three tiers: | `include: "plan, seats"` | + `noise` | `{plan:'free', seats:'5'}` | | `include: ""` + `prop.plan=pro` | `http.method`, `plan=free` | `{plan:'pro'}` | -The contract lives in one place — `packages/domain/src/tinybird/product-event-attributes.ts` — +The contract lives in one place, `packages/domain/src/tinybird/product-event-attributes.ts`, and is read by exactly two consumers that must agree byte for byte: `productEventsTracesMv` (managed orgs, via `tinybird deploy`) and the frozen copy inside ClickHouse migration 0028 (BYO clusters). The migration's copy is deliberately NOT imported from the constant: a delta migration @@ -345,14 +349,14 @@ performed it. Marking a trace by hand in the UI marks _one sampled trace_, canno history, and puts a mutable user-authored row into an append-only fact table. An attribute marks every trace the path produces, applies retroactively across the whole `traces` retention window, and is reviewable in the customer's own diff. There is no second store and no write path from the -dashboard — the span is the record, the product event is its projection. +dashboard. The span is the record; the product event is its projection. ### The link `product_events` gained `TraceId`/`SpanId` (migration 0028, `DEFAULT ''`, appended, plus a `bloom_filter` on `TraceId`). Non-empty only on `Source = 'trace'` rows. Real columns rather than `Attributes` keys because both directions filter on them, and a `Map` lookup on this table reads -the whole map per row — the exact cost `product_events` was split out of `session_events` to avoid. +the whole map per row, the exact cost `product_events` was split out of `session_events` to avoid. | Direction | Query | Surface | | ---------------------------- | ------------------------------- | ------------------------------------------------ | @@ -367,18 +371,18 @@ never change at all. The MV predicate is one `Map` value read per incoming span, on the same block every other `traces` MV already fires on. An MV sees the insert block, not the table, so `idx_span_attr_keys` does not -help it — this is a real, deliberately small per-span ingest cost. +help it. This is a real, deliberately small per-span ingest cost. Copying the whole `SpanAttributes` map **by default** is the deliberate expensive choice. A server span's map is dominated by HTTP/DB semconv keys, and `product_events` keeps 365 days against raw -`traces`' 30 — so an annotated span's attributes outlive the span itself by a factor of twelve. What +`traces`' 30, so an annotated span's attributes outlive the span itself by a factor of twelve. What the default buys is that nothing has to be declared to get a useful event; `include` is the lever for a team that has measured the cost and wants it back, and it is a one-line change on the span rather than a schema migration. The practical consequence to watch: attribute pickers over product events list the span's whole semconv surface for any team that has not set `include`. If that becomes the dominant cost across -orgs rather than for one of them, the lever is a per-org key denylist at the MV — the per-span +orgs rather than for one of them, the lever is a per-org key denylist at the MV. The per-span `include` handles the single-team case already. The whole `Attributes` expression only evaluates for rows passing the `WHERE`, i.e. annotated spans, @@ -389,41 +393,41 @@ lookup. 1. **Managed**: `bun run --cwd apps/api tinybird:deploy` creates `product_events_traces_mv` and adds the two columns, then an explicit `tb` populate from `traces` (bounded by its 30-day TTL). - Blocked on the same manual step the rest of this document's checklist is — see - `project_product_events_tinybird_rollout_pending`. + Blocked on the same manual step the rest of this document's checklist is (see the + `project_product_events_tinybird_rollout_pending` note). **The `FORWARD_QUERY` on `product_events` is what makes this deploy safe, and it is not optional.** Adding two defaulted columns is _not_ a free change here: without the forward query Tinybird satisfies the new schema by rebuilding the table from the datasources that feed it, and both (`session_events`, `traces`) keep 30 days against `product_events`' 365. It says so and - proceeds anyway — + proceeds anyway: > it is going to be backfilled using the following datasources which would lead to a deleting > historical data - — which on this dual-fed table is worse than it sounds: the server and mobile rows arrive by + On this dual-fed table that is worse than it sounds: the server and mobile rows arrive by `POST /v1/events` and have **no source datasource at all**, so a rebuild drops them at every age, not just past 30 days. Verified against a real deploy, not inferred. - Two traps around it. `DEPLOYMENT_METHOD alter` on `product_events_mv` does **not** substitute — - tested, and the same data-loss warning returns, because it is the datasource schema change that + Two traps around it. `DEPLOYMENT_METHOD alter` on `product_events_mv` does **not** substitute. We + tested it, and the same data-loss warning returns, because it is the datasource schema change that triggers the source backfill, not the view's. And once the forward query is in place Tinybird suggests the inverse ("could be applied with ALTER TABLE and no data movement at promotion time - if you remove the FORWARD*QUERY"); following that suggestion reintroduces the loss. Per the - Tinybird rules the forward query can be deleted in a \_later* deploy, once this one has compacted. + if you remove the FORWARD_QUERY"); following that suggestion reintroduces the loss. Per the + Tinybird rules the forward query can be deleted in a _later_ deploy, once this one has compacted. **The populate is one-shot and overlap-prone.** Unlike BYO and local, the managed surface has no `DELETE WHERE Source = 'trace'` step, so running it twice double-inserts, and running it after the MV is already live double-counts every annotated span ingested between MV creation and the populate's own snapshot. BYO risks a gap; managed risks duplicates. Same caveat 0014 and 0021 - accepted — but on a table feeding customer-visible funnels, a double-counted conversion is worse + accepted, but on a table feeding customer-visible funnels, a double-counted conversion is worse than a missing one. Populate once, immediately after deploy, and if it fails partway prefer deleting the trace rows by hand over re-running it blind. 2. **BYO ClickHouse**: migration 0028, `requiredForIngest: false`. That is safe for one reason worth knowing before anyone touches `datasources.ts`: `TraceId`/`SpanId` are declared with **no `jsonPath`**, so the insert-mapping generator omits them and the Rust gateway's - `INSERT INTO product_events (…)` never names them — a cluster stamped below 28 still accepts + `INSERT INTO product_events (…)` never names them, so a cluster stamped below 28 still accepts every row it sends. Give those columns a `jsonPath` and the flag becomes a data-loss bug: the readiness gate still says 21, so unmigrated BYO orgs keep routing to their own cluster, where the INSERT fails on the unknown column, retries, trips the breaker and drops the batch. @@ -443,7 +447,7 @@ is 1970, which would turn the bound back into "everything". routes it would sit on both exist. - **No SDK helper.** Teams set the attributes by hand on whatever span API they already use. A `markProductEvent(span, name, { props, include })` in `@maple-dev/effect-sdk` would be the obvious - next step — it is a wrapper over `setAttributes` that builds the `prop.*` keys and joins + next step. It is a wrapper over `setAttributes` that builds the `prop.*` keys and joins `include`, not new machinery, and it is where the empty-string overwrite idiom would get a name (`attributes: "none"`) instead of being a documented convention. @@ -459,7 +463,7 @@ the previous step. When the variant is set the route params carry (p50/p90 milliseconds between consecutive steps) and `productEventsFunnelLeaversQuery` (the first event after the last step a leaver reached, top 6 per step, `''` = nothing followed). Both walk each person's -chain from the first step-1 event — a first-occurrence approximation of the +chain from the first step-1 event: a first-occurrence approximation of the chain `windowFunnel` counted, so a person the funnel counts at level N can occasionally have no step-N timestamp and is left out of the quantiles. The extras ride on the same `{ name, value }` rows as optional per-step fields @@ -472,8 +476,8 @@ Endpoints: `/product-events-funnel-timing`, `/product-events-funnel-leavers`. definition on `display.paths`): what people do in the hops after (or before) one anchor event or page. `productEventsPathsQuery` finds each person's anchor instant (the first going forward, the last going backward), reads only the -rows inside `windowSeconds` of it in `(ts, seq)` order — the anchor row always -kept, `include` / `exclude` narrowing the rest — collapses consecutive repeats, +rows inside `windowSeconds` of it in `(ts, seq)` order (the anchor row always +kept, `include` / `exclude` narrowing the rest), collapses consecutive repeats, and emits one `{ hop, fromNode, toNode, count }` row per hop up to `depth` (1–5). Only the top `branches` (1–10) nodes per column are named; the rest fold into @@ -481,6 +485,6 @@ top `branches` (1–10) nodes per column are named; the rest fold into `''`. Page views are named by `PagePath`, everything else by `EventName`; `include` narrows to events or pages, `exclude` drops names before sequencing. The hop rows are a CTE so the ranking and both sides of the fold join read -them once. `docs/query-benchmarking.md`'s catalog carries fixtures for both, +them once. The benchmark catalog (`packages/query-engine/src/benchmark/builders.ts`) carries fixtures for both, and `web-analytics-parity.clickhouse.e2e.test.ts` runs them against a real ClickHouse. diff --git a/docs/profiling.md b/docs/profiling.md index 5ea4ef2c65..54be684457 100644 --- a/docs/profiling.md +++ b/docs/profiling.md @@ -1,12 +1,15 @@ # Profiling as Maple's fourth signal -_Plan, 2026-08-18. Status: proposal — nothing here is built. Companion to +_Plan, 2026-08-18. Status: proposal; nothing here is built. Companion to [`otel-spec/emerging-and-compat.md`](otel-spec/emerging-and-compat.md) §1 (the spec digest) and item 17 of [`otel-coverage-roadmap.md`](otel-coverage-roadmap.md)._ +Status (2026-09-25): still unbuilt. `apps/ingest` now pins `opentelemetry-proto` 0.32, which +also ships the `profiles` feature; it is still not enabled. + ## TL;DR -- **Yes, it is a standard** — OpenTelemetry Profiles is the official fourth OTLP signal +- **Yes, it is a standard.** OpenTelemetry Profiles is the official fourth OTLP signal (traces, metrics, logs, profiles). It is pprof-derived, round-trips losslessly with pprof, and links samples to spans by `trace_id`/`span_id`. It entered **public Alpha on 2026-03-26**; the OTLP transport is **Development**, the HTTP path is @@ -17,13 +20,13 @@ and item 17 of [`otel-coverage-roadmap.md`](otel-coverage-roadmap.md)._ the OTel blog itself says "production-ready backends have not yet emerged". A trace-native platform that answers _"which function burned this span's time"_ in the same UI is differentiated today, not in 2027. -- **Recommendation:** build it — but as a scoped MVP that flattens the pprof-shaped payload +- **Recommendation:** build it, as a scoped MVP that flattens the pprof-shaped payload into ClickHouse rows at the ingest gateway (the same "denormalize at ingest" trade Maple already makes for span attributes), ships a pure-JS CPU profiler in `effect-sdk` (Node and Bun via `node:inspector`, verified today), and puts a flame graph on the service page and on the span detail panel. Roughly **3 engineer-weeks to MVP** on the seams named below, plus a 2–3 day spike first. Alpha status is handled by pinning the proto version and never storing - the raw wire format — only our own flattened rows. + the raw wire format, only our own flattened rows. - **What hotpath does not solve and this does:** hotpath profiles _Maple's ingest process locally_. This gives every Maple customer CPU/allocation attribution for their own services, correlated with their traces. Different product. @@ -33,7 +36,7 @@ and item 17 of [`otel-coverage-roadmap.md`](otel-coverage-roadmap.md)._ | Aspect | State (Aug 2026) | Source | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Signal / data model | **Alpha** (pre-Experimental; "should not be used for critical production workloads") | [spec](https://opentelemetry.io/docs/specs/otel/profiles/), [alpha post](https://opentelemetry.io/blog/2026/profiles-alpha/) | -| OTLP transport | **Development** — Stable only for traces/metrics/logs | [OTLP spec](https://opentelemetry.io/docs/specs/otlp/) | +| OTLP transport | **Development**; Stable only for traces/metrics/logs | [OTLP spec](https://opentelemetry.io/docs/specs/otlp/) | | HTTP path / RPC | `POST /v1development/profiles`, `ProfilesService/Export`, `ExportProfilesServiceRequest{resource_profiles}` / `partial_success.rejected_profiles` | [OTLP spec](https://opentelemetry.io/docs/specs/otlp/) | | Wire format | pprof-derived, `ProfilesDictionary` (string / function / location / stack / link / attribute tables shared per request), ~40% smaller than pprof | [alpha post](https://opentelemetry.io/blog/2026/profiles-alpha/) | | Rust proto crate | `opentelemetry-proto 0.31` ships `profiles.v1development` behind a `profiles` cargo feature (present in our registry copy, **not enabled** in `apps/ingest/Cargo.toml`) | local crate | @@ -53,15 +56,15 @@ Implications we design around: ## 2. Who can actually send us profiles -This is the make-or-break question for a signal — the spec being real does not mean customers +This is the make-or-break question for a signal. The spec being real does not mean customers have producers. Four tiers, from zero-code to bespoke: | Producer | Languages | Where it runs | Trace link | Effort for customer | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | **OTel eBPF profiler** (in the collector distro) | Go, JVM, Node, .NET, Ruby, Python, PHP, BEAM, native | Linux hosts / k8s DaemonSet, privileged | per-runtime, partial | Deploy the collector with the receiver + point at Maple. No app changes. | | **pprof-emitting profilers** → collector `pprof` receiver → OTLP | Go `runtime/pprof`, Rust `pprof`, Node `pprof` npm, Python `py-spy`, JVM `async-profiler` (which now also emits OTLP directly) | anywhere | no (time-window only) | App-side profiler + a collector hop | -| **`effect-sdk` built-in** (new, this plan) | Node ≥18, Bun | Node/Bun servers. **Not Cloudflare Workers** (no profiler API) — must no-op there | time-window in MVP; span-linked later | `profiling: { cpu: true }` | -| **`packages/browser`** | — | — | — | Out of scope: browsers expose no sampling profiler worth shipping (JS Self-Profiling API is Chromium-only, origin-trial-grade). | +| **`effect-sdk` built-in** (new, this plan) | Node ≥18, Bun | Node/Bun servers. **Not Cloudflare Workers** (no profiler API); must no-op there | time-window in MVP; span-linked later | `profiling: { cpu: true }` | +| **`packages/browser`** | n/a | n/a | n/a | Out of scope: browsers expose no sampling profiler worth shipping (JS Self-Profiling API is Chromium-only, origin-trial-grade). | **Verified this session:** `node:inspector` `Profiler.start/stop` returns a full V8 sampling profile (`nodes`, `samples`, `timeDeltas`, `callFrame{functionName,url,lineNumber}`) with **no @@ -70,12 +73,12 @@ same way. That is what makes "one config flag" profiling possible for our own SD biggest lever in this plan because most Maple SDK users are on Node/Bun today. The eBPF path is what makes Maple credible for k8s shops without any SDK change, and it is -_the_ reference producer the OTel SIG tests against — the spike (§8) should start there. +_the_ reference producer the OTel SIG tests against. The spike (§8) should start there. **Deliberately optional (Phase 2): Pyroscope push-API compatibility.** Every Pyroscope SDK (Go, Node, Python, Ruby, .NET, Java, Rust) already `POST`s pprof to `/ingest?name=…&from=…`. If ingest accepts that shape too, all of them work against Maple with an endpoint swap. It is one -extra route that decodes plain pprof — the same flattener as §4 — but it is not OTel, so it is +extra route that decodes plain pprof (the same flattener as §4), but it is not OTel, so it is not in the MVP. ## 3. Product surface @@ -89,12 +92,12 @@ Scope for MVP is exactly two questions, both already asked in traces terms elsew resource-attribute facets the service page already has. 2. **Span detail → "Profile" tab.** _"What was running during this span?"_ Flame graph filtered to samples that (a) carry a link to this span (`SpanId = ?`, best fidelity), else (b) fall - inside `[span.start, span.end]` on the same service instance (time-window fidelity — mark it + inside `[span.start, span.end]` on the same service instance (time-window fidelity; mark it as such in the UI). This is the feature no traces-only tool has and where the OTel spec's own pitch lands. Not in MVP (Phase 2+): diff/compare view (two windows), sandwich view, per-line source view, -alerts on profile metrics, dashboards widgets, MCP tools (`top_functions`, `flame_graph` — small +alerts on profile metrics, dashboards widgets, MCP tools (`top_functions`, `flame_graph`: small once the queries exist and very on-brand for the agent story; put it early in Phase 2). ## 4. Data model → storage @@ -106,7 +109,7 @@ is a list of `Location → Line → Function{name, filename, start_line}` indice **Decision: flatten to one denormalized row per sample at the ingest gateway.** No dictionary tables in the warehouse. Reasons: (1) it is one NDJSON line per row, which is exactly what the -existing `rows_to_frames` / insert-mapping / Tinybird / BYO-ClickHouse machinery moves — a +existing `rows_to_frames` / insert-mapping / Tinybird / BYO-ClickHouse machinery moves. A multi-table bundle would need new plumbing everywhere; (2) `Array(LowCardinality(String))` dictionary-encodes repeated frame names per part, so the storage penalty of denormalizing is modest; (3) flame-graph queries become a single `GROUP BY Frames` with no join. @@ -121,9 +124,9 @@ profiles -- new datasource, packages/domain/src/tinybir ProfileId String 16-byte hex; groups samples of one profile SampleType LowCardinality(String) "cpu" | "alloc_space" | "alloc_objects" | "wall" | … SampleUnit LowCardinality(String) "nanoseconds" | "bytes" | "count" - Value Int64 one row per (sample, value index) — pprof allows N values + Value Int64 one row per (sample, value index); pprof allows N values Period Int64 sampling interval in SampleUnit terms (0 if unknown) - StackHash UInt64 cityHash64 of the joined frames — toString()'d in every SELECT + StackHash UInt64 cityHash64 of the joined frames; toString()'d in every SELECT Frames Array(LowCardinality(String)) leaf-first function names, "pkg.fn" as the producer gave them FrameFiles Array(LowCardinality(String)) parallel; "" when unknown (native/eBPF) FrameLines Array(Int32) parallel; 0 when unknown @@ -147,15 +150,15 @@ profiles_stacks_hourly_mv → profiles_stacks_hourly (SummingMergeTree, 30d TTL) Queries this supports (all in `packages/query-engine/src/ch/queries/profiles.ts`): -- **flameGraph** — `SELECT Frames, sum(Value) FROM … WHERE OrgId, ServiceName, SampleType, +- **flameGraph**: `SELECT Frames, sum(Value) FROM … WHERE OrgId, ServiceName, SampleType, window [AND SpanId = ?] GROUP BY Frames` → the client folds the (frames[], value) list into a tree. Wire size is bounded by distinct stacks, not samples; cap at N stacks by value and report the tail as an "other" node. -- **topFunctions** — `arrayJoin(Frames)` for total; `Frames[1]` for self; both `sum(Value)`. -- **timeseries** — `sum(Value)` per bucket per SampleType (feeds a sparkline on the tab and, +- **topFunctions**: `arrayJoin(Frames)` for total; `Frames[1]` for self; both `sum(Value)`. +- **timeseries**: `sum(Value)` per bucket per SampleType (feeds a sparkline on the tab and, later, dashboards/alerts). -- **sampleTypes** — distinct `(SampleType, SampleUnit)` for the selector. -- **spanOverlap** — the time-window fallback for the span tab: same service instance +- **sampleTypes**: distinct `(SampleType, SampleUnit)` for the selector. +- **spanOverlap**: the time-window fallback for the span tab: same service instance (`ResourceAttributes['service.instance.id']`) and `Timestamp BETWEEN span.start AND span.end`. Retention: 7 d raw / 30 d rollup by default, keyed like `logs` in @@ -164,10 +167,10 @@ Retention: 7 d raw / 30 d rollup by default, keyed like `logs` in Sizing sanity: the eBPF agent samples at ~20 Hz/core and pre-aggregates per stack before sending, so a 32-core host is on the order of a few hundred rows/s; a Node process with the SDK at 1 kHz over a 10 s window is ~10 k samples/10 s _before_ we aggregate them by stack in the SDK -(which we should — see §6). Rows are small (Frames dictionary-encode); this is well inside what +(which we should; see §6). Rows are small (Frames dictionary-encode); this is well inside what `logs` does today. -## 5. Ingest gateway (`apps/ingest`) — the seams +## 5. Ingest gateway (`apps/ingest`): the seams Everything a new signal touches, from the code as it is on `main` today: @@ -175,7 +178,7 @@ Everything a new signal touches, from the code as it is on `main` today: | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Proto | `Cargo.toml` `opentelemetry-proto` features | add `"profiles"`; pin the exact version and note it in the plan doc | | Signal enums | `main.rs` `Signal` (`Traces\|Logs\|Metrics`), `DecodedPayload`, `item_count`, `encode`; `telemetry.rs` `TelemetrySignal`, `DatasourceNames` | add `Profiles`; `INGEST_TINYBIRD_DATASOURCE_PROFILES` | -| HTTP route | `main.rs` router (`/v1/traces` …) | `POST /v1development/profiles` **and** `/v1/profiles` (future RC path); both map to `Signal::Profiles`. `format!("/v1/{}", signal.path())` at the `ingest` span and `forward_to_collector` needs a `Signal::route_path()` — the profiles path breaks that assumption. | +| HTTP route | `main.rs` router (`/v1/traces` …) | `POST /v1development/profiles` **and** `/v1/profiles` (future RC path); both map to `Signal::Profiles`. `format!("/v1/{}", signal.path())` at the `ingest` span and `forward_to_collector` needs a `Signal::route_path()`; the profiles path breaks that assumption. | | gRPC | `run_grpc_server` | 4th service, `ProfilesServiceServer`, same `export` shape as logs | | Decode + enrich | `decode_and_enrich_payload` (6-arm match), `otlp_json::normalize` root field | `resourceProfiles` arm for protobuf + JSON; `enrich_profiles_request` writes `maple_org_id` etc. into resource attributes exactly like the others | | Flatten | new `telemetry.rs::encode_profiles` | the real work: resolve dictionary indices → frame name/file/line arrays, `cityHash64` stack, one row per `(sample, value_index)`, link → `TraceId`/`SpanId`, `routing_key = hash64(trace_id)` when linked else `hash64(profile_id)` so a span's samples land in one lane | @@ -183,13 +186,13 @@ Everything a new signal touches, from the code as it is on `main` today: | Insert mapping | `scripts/generate-clickhouse-insert-mappings.ts` `OTLP_DATASOURCES` | add `"profiles"`; regenerates `apps/ingest/src/clickhouse_insert_mappings.rs` and `apps/cli/src/server/schema/local-inserts.json` | | BYO ClickHouse | `packages/domain/src/clickhouse/migrations/00NN_profiles.ts` + `migrations/index.ts` | new delta migration; bumps `clickHouseSchemaVersion` → ingest `SCHEMA_VERSION` gate | | Billing | `usage_metrics.rs` (`"logs" \| "traces" \| "metrics"`), `metered_enqueue`, Autumn feature ids | new feature id `profiles`; re-size `USAGE_CARDINALITY_LIMIT` (sized for 3 signals × orgs) | -| Forward mode | `forward_to_collector` | forward profiles too — collectors ≥0.148 accept them | +| Forward mode | `forward_to_collector` | forward profiles too; collectors ≥0.148 accept them | | Local CLI | `apps/cli/src/server/serve.ts` `Signal` union, decode/encode/route, `local-schema.sql` | same table in chDB so `maple local` shows profiles too | | Self-obs | `metrics::native_rows` etc. keyed by `signal.path()` | falls out once the enum exists; add `maple.ingest.frame_count` to the encode span | Frames come from the producer as-is. eBPF/native frames arrive **unsymbolized** (addresses + mapping filename); we render those as `mapping!0x1234` in MVP and treat symbolization -(symbol upload / debuginfod) as explicitly out of scope — say so in the docs. +(symbol upload / debuginfod) as explicitly out of scope, and say so in the docs. ## 6. `effect-sdk` profiler (Node + Bun) @@ -201,7 +204,7 @@ layer({ profiling: { cpu: true, heap: false, windowMs: 10_000, samplingUs: 1_000 - Opens one `inspector.Session`; every `windowMs` it stops the current CPU profile, converts the V8 `cpuprofile` (`nodes[]` tree + `samples[]` + `timeDeltas[]`) into **our own aggregated form - first** — group samples by stack, sum `timeDeltas` — then builds one `ExportProfilesServiceRequest` + first** (group samples by stack, sum `timeDeltas`), then builds one `ExportProfilesServiceRequest` with a real `ProfilesDictionary` (dedupe strings/functions/stacks) and posts to `${endpoint}/v1development/profiles` on the existing flush path (`flush-core.ts` gets a `profilesUrl` next to traces/logs/metrics). Aggregating client-side is what keeps a 1 kHz @@ -215,7 +218,7 @@ layer({ profiling: { cpu: true, heap: false, windowMs: 10_000, samplingUs: 1_000 - Overhead: V8's sampling profiler is ~1–3% CPU at 1 kHz on a busy process; make `samplingUs` configurable and document it. `inspector` throws on Workers → the module detects the runtime and becomes a no-op layer with one warn log, never a startup failure. -- **Browser SDK loop-prevention:** `packages/browser` ignores `${endpoint}/v1/` — a +- **Browser SDK loop-prevention:** `packages/browser` ignores `${endpoint}/v1/`, so a `/v1development/` upload would escape that filter. Fix the regex to `/v1(development)?/` even though the browser SDK does not send profiles, so nothing else regresses. @@ -229,30 +232,34 @@ route; the SDK overview stops saying "traces, logs, and metrics" (nine files say `CH.CHNumber` for every numeric column, `StackHash` always `toString()`-wrapped (SQL-catalog gate), exported from `ch/index.ts`; registry entries in `registry/profiles.ts` with `profile: "aggregation"` and cache policy like the service-overview queries. -- **API:** `V2ProfilesApiGroup` in `packages/domain/src/http/v2/telemetry.ts` (`/v2/profiles/ -flamegraph`, `/top-functions`, `/timeseries`, `/sample-types`) + `HttpV2ProfilesLive` in +- **API:** `V2ProfilesApiGroup` in `packages/domain/src/http/v2/telemetry.ts` (`/v2/profiles/flamegraph`, + `/top-functions`, `/timeseries`, `/sample-types`) + `HttpV2ProfilesLive` in `apps/api/src/routes/v2/telemetry.http.ts`, following the logs group exactly. Flame graph is the one user-data-sized response → `compiledQueryBounded`. -- **UI:** - `routes/services/$serviceName.tsx`: `ServiceDetailTab` gains `"profiles"` (tab literal + - trigger + body) → new `components/services/service-profiles-tab.tsx`. - `components/traces/span-detail-panel.tsx`: a `profile` tab next to `details / logs / -infrastructure`, hidden when the service has no profiles in range (one cheap `sampleTypes` - call). - New `packages/ui/src/components/profiles/flame-graph.tsx`: an **aggregate icicle**, not the - trace waterfall — it folds `(frames[], value)` rows into a tree, lays out by share, and - renders on canvas (the TanStack-canvas evaluation showed canvas is the right call for dense - rectangles). Reuse the waterfall's tooltip and `color-by` palette so it reads as one family. - Click-to-zoom, hover tooltip (self / total / share), text search that highlights matching - frames. Keep it to that. - Data reads through `makeQueryAtomFamily` in `warehouse-query-atoms.ts` like everything else. +- **UI:** + - `routes/services/$serviceName.tsx`: `ServiceDetailTab` gains `"profiles"` (tab literal + + trigger + body) → new `components/services/service-profiles-tab.tsx`. + - `components/traces/span-detail-panel.tsx`: a `profile` tab next to `details / logs / + infrastructure`, hidden when the service has no profiles in range (one cheap `sampleTypes` + call). + - New `packages/ui/src/components/profiles/flame-graph.tsx`: an **aggregate icicle**, not the + trace waterfall. It folds `(frames[], value)` rows into a tree, lays out by share, and + renders on canvas (the TanStack-canvas evaluation showed canvas is the right call for dense + rectangles). Reuse the waterfall's tooltip and `color-by` palette so it reads as one family. + Click-to-zoom, hover tooltip (self / total / share), text search that highlights matching + frames. Keep it to that. + - Data reads through `makeQueryAtomFamily` in `warehouse-query-atoms.ts` like everything else. ## 8. Phasing -**Phase 0 — spike (2–3 days).** Run the collector eBPF profiler on a Linux box → a hand-rolled +**Phase 0: spike (2-3 days).** Run the collector eBPF profiler on a Linux box → a hand-rolled `/v1development/profiles` handler on a branch of ingest that dumps flattened rows into a scratch ClickHouse table → run the flame-graph SQL from §4 by hand and eyeball a `d3-flame-graph` render. Also point `async-profiler` (JVM) at it. Goals: confirm the proto crate builds, measure rows/s and bytes/row for real producers, and see what unsymbolized eBPF frames look like in practice. If the numbers or the shape surprise us, §4 changes _before_ any of the codegen work. -**Phase 1 — MVP (~3 engineer-weeks).** +**Phase 1: MVP (~3 engineer-weeks).** 1. Schema: datasource + MV + migration + insert-mapping regen + `ttl-override` (2 d). 2. Ingest: enums/routes/gRPC/decode/flatten/pipeline/billing + tests with recorded eBPF and @@ -274,7 +281,7 @@ keep accepting `/v1development/`. Consider symbolization for native frames. | Risk | Mitigation | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Proto churn while Alpha | pinned crate; ingest is the only place that reads the wire; storage is our shape | -| Producer scarcity | three tiers in §2 — eBPF (zero-code, k8s), pprof via collector, our SDK; the spike proves two of them | +| Producer scarcity | three tiers in §2: eBPF (zero-code, k8s), pprof via collector, our SDK; the spike proves two of them | | Volume / cost | producers pre-aggregate; SDK aggregates per window; 7 d raw + 30 d hourly rollup; `Frames` LC-encoded; profiles metered as its own Autumn feature | | Unsymbolized native frames look bad | render `mapping!addr`, document that symbolization is out of scope, keep JS/JVM/Go/.NET (which arrive symbolized) as the headline | | Node inspector overhead / Workers | configurable rate, off by default, no-op on Workers | @@ -286,7 +293,7 @@ keep accepting `/v1development/`. Consider symbolization for native frames. 1. **Pricing:** meter profiles as GB at the same rate as logs/traces, or a distinct SKU? (Ingest only needs the feature id; Autumn config is where this lands.) 2. **Retention defaults:** 7 d raw / 30 d rollup as proposed? -3. **Pyroscope push compatibility** in Phase 2 — worth it? It is the fastest route to Go/Python/ +3. **Pyroscope push compatibility** in Phase 2: worth it? It is the fastest route to Go/Python/ Ruby/.NET customers but it is a non-OTel surface to maintain. 4. **Go / no-go on the spike.** It is the only cheap step; the rest of the plan is conditional on what it shows. diff --git a/docs/query-benchmarking.md b/docs/query-benchmarking.md index 32813228c3..11c090643e 100644 --- a/docs/query-benchmarking.md +++ b/docs/query-benchmarking.md @@ -13,7 +13,7 @@ execution, HTTP, files and printing use the published `ch-bench` CLI. See the `run` accepts a TypeScript suite directly and recompiles it on each invocation. `doctor`, `export`, `schema`, `--json`, per-case verification, and multi-metric -`--budgets` are also available through `bun run bench:queries`. Comparisons now +`--budgets` are also available through `bun run bench:queries`. Comparisons always return a nonzero exit code for a failed gate; `--fail-on-regression` remains accepted. Artifacts from generic commands default to `.bench/` under the working directory; pass `--out apps/api/scripts/.bench/...` for Maple evidence. @@ -21,7 +21,7 @@ No benchmark code is added to the query engine's root barrel or production paths This replaces the previous `@maple/query-engine/sql-catalog` and `@maple/query-engine-integrations/catalog` entry points. The core fixtures and -coverage checks now live in `packages/query-engine/src/benchmark/`; integration +coverage checks live in `packages/query-engine/src/benchmark/`; integration fixtures live in `packages/query-engine-integrations/src/benchmark/`. The shared `apps/api/scripts/query-bench/catalog.ts` composes them for both the CLI and the ClickHouse analyzer sweep. Decoder, tenant-scope, routing, and SQL snapshot checks @@ -206,6 +206,8 @@ the change. It does not automatically recommend an index or materialized view. ```sh # Run in the standalone effect-clickhouse repository: bun run test + +# Run in this repository: bun run --cwd packages/query-engine test -- src/benchmark/catalog.test.ts bun run --cwd packages/query-engine-integrations test -- src/benchmark/catalog.test.ts bun run --cwd packages/query-engine typecheck @@ -214,7 +216,7 @@ bun run --cwd apps/api bench:typecheck # Real migrations, all catalog cases through the analyzer, and a populated CLI # run/compare/inspect test. Uses its own temporary database, removed afterward. CLICKHOUSE_E2E=1 CLICKHOUSE_E2E_URL=http://127.0.0.1:8123 \ - bun run --cwd apps/api test -- src/services/warehouse/query-benchmark.clickhouse.e2e.test.ts + bun run --cwd apps/api test -- scripts/query-bench/catalog.clickhouse.e2e.test.ts ``` The live test inserts 50,000 spans across two tenants, verifies 40,000 belong to diff --git a/docs/query-optimization-implementation-2026-09-05.md b/docs/query-optimization-implementation-2026-09-05.md index 9b4c27cef5..cd62d2bf35 100644 --- a/docs/query-optimization-implementation-2026-09-05.md +++ b/docs/query-optimization-implementation-2026-09-05.md @@ -1,4 +1,4 @@ -# Query optimization implementation — 2026-09-05 +# Query optimization implementation (2026-09-05) Implemented the measured trace-alert and series-cap optimizations from the [first review](query-optimization-review-2026-09-05.md) and [second scan](query-optimization-second-scan-2026-09-05.md), plus scheduler-scoped bucket reuse. Replaying the actual modified builders against populated historical windows reduced alert reads by **51–74%** and capped-timeseries reads by **about 50%**. diff --git a/docs/query-optimization-review-2026-09-05.md b/docs/query-optimization-review-2026-09-05.md index 70c9609ec1..ca76ad2ab1 100644 --- a/docs/query-optimization-review-2026-09-05.md +++ b/docs/query-optimization-review-2026-09-05.md @@ -1,4 +1,4 @@ -# Frequent-query optimization review — 2026-09-05 +# Frequent-query optimization review (2026-09-05) **Prioritize metric-specific trace alert queries.** The tested count/error-rate variants read **51–74% fewer bytes** and used **10–24% less mean server time**. The returned values consumed by alert evaluation matched. End-to-end p95 did not improve consistently, so this is evidence of reduced warehouse work, not a demonstrated latency improvement for every alert. diff --git a/docs/query-optimization-second-scan-2026-09-05.md b/docs/query-optimization-second-scan-2026-09-05.md index d2cfdab22a..4db3861e77 100644 --- a/docs/query-optimization-second-scan-2026-09-05.md +++ b/docs/query-optimization-second-scan-2026-09-05.md @@ -1,4 +1,4 @@ -# Query optimization: second scan — 2026-09-05 +# Query optimization: second scan (2026-09-05) **The strongest new SQL candidate is the timeseries series cap:** a window-ranking experiment read approximately **50% fewer bytes** and used **30–54% less mean server time**, with matching results in four populated cases. The broader scan also found substantial repeated trace-alert work and raw-alert authentication failures. Switching service facets to the existing minute tier increased reads and is not recommended from this evidence. diff --git a/docs/sampling-throughput.md b/docs/sampling-throughput.md index 082944ddee..66b3b7411d 100644 --- a/docs/sampling-throughput.md +++ b/docs/sampling-throughput.md @@ -1,33 +1,33 @@ # Sampling & Throughput Estimation -Maple automatically detects trace sampling and extrapolates throughput metrics so you see realistic request rates even when only a fraction of traces are collected. +Maple detects trace sampling and extrapolates throughput, so request rates stay realistic when only a fraction of traces is collected. ## What is trace sampling? -Distributed tracing at scale generates a large volume of spans. Sampling reduces this volume by only exporting a subset of traces: +Distributed tracing at scale produces a large volume of spans. Sampling cuts that volume by exporting only a subset of traces: -- **Head sampling** -- a decision made at the start of a trace (e.g. "keep 10% of traces"). The OTel SDK or Collector makes the choice before any spans are processed. -- **Tail sampling** -- a decision made after the full trace completes, typically based on latency, errors, or other attributes. +- **Head sampling**: the decision is made at the start of a trace (e.g. "keep 10% of traces"). The OTel SDK or Collector decides before any spans are processed. +- **Tail sampling**: the decision is made after the trace completes, typically based on latency, errors, or other attributes. -Both approaches reduce storage and processing costs, but mean the raw span count no longer reflects actual throughput. +Both reduce storage and processing costs. Both also mean the raw span count no longer reflects actual throughput. ## How Maple detects sampling -OpenTelemetry propagates a `TraceState` header on every span. When probability-based sampling is active, the `TraceState` contains a `th` (threshold) key: +OpenTelemetry records the W3C `tracestate` on every span. When probability sampling is active, the `ot` entry carries a `th` (rejection threshold) key: ``` tracestate: ot=th:e668 ``` -At ingest time, Maple computes a per-row `SampleRate` weight on the `traces` datasource. The expression resolves three sources in priority order: +At ingest time, Maple computes a per-row `SampleRate` weight on the `traces` datasource (the `SAMPLE_RATE_EXPR` column default in `packages/domain/src/tinybird/datasources.ts`). The expression resolves three sources in priority order: -1. `SpanAttributes['SampleRate']` -- explicit collector-set value, takes precedence. -2. `TraceState th:` -- W3C threshold sampling, parsed inline. -3. Default `1.0` -- unsampled. +1. `SpanAttributes['SampleRate']`: an explicit collector-set value. It wins when it is `>= 1`. +2. `TraceState th:`: W3C threshold sampling, parsed inline. The weight is capped at 10000. +3. Default `1.0`: unsampled. -Because the weight is materialized per row, downstream queries don't need to know anything about TraceState parsing -- they just sum `SampleRate`. +The weight is materialized per row, so downstream queries don't parse TraceState. They sum `SampleRate`. -No manual configuration is needed -- if your OTel SDK or Collector sets the `th` value, Maple picks it up automatically. +No configuration is needed. If your OTel SDK or Collector sets the `th` value, Maple picks it up. ## How throughput is calculated @@ -42,7 +42,7 @@ const acceptanceProbability = 1 - rejectionRate const sampleRate = 1 / acceptanceProbability ``` -Then the query engine simply sums the column: +The query engine then sums the column: ``` estimatedTotal = sum(SampleRate) -- per-row weighted sum @@ -56,24 +56,24 @@ estimatedTotal = 500 * 10 = 5000 throughput = 5000 / 60 = ~83 req/s ``` -Critically, this also handles **mixed sampling rates** correctly. If 99 of those 500 spans were sampled at 50% (weight 2) and 1 was sampled at 99.99% rejection (weight 8192), the per-row sum yields `99 * 2 + 1 * 8192 ≈ 8390` -- not `500 * 8192` like a single-weight-per-bucket approximation would. +This also handles **mixed sampling rates** correctly. Take a bucket of 100 spans: 99 sampled at 50% (weight 2) and 1 sampled at 1/8192 (weight 8192). The per-row sum is `99 * 2 + 1 * 8192 = 8390`. A single weight per bucket would give `100 * 8192`. ## UI indicators When sampling is detected for a service: -- **Tilde prefix (`~`)** -- throughput values are prefixed with `~` to indicate the number is an estimate, not an exact count. This appears in the services table, service map nodes, and service map edges. -- **Secondary "traced" line** -- the services table shows a smaller line below the estimated throughput with the actual traced rate (e.g. `~8.3 traced`), so you can see both values. -- **Tooltip** -- hovering the throughput cell shows the sampling rate and extrapolation factor (e.g. "Estimated from 10% sampled traces (x10 extrapolation)"). +- **Tilde prefix (`~`)**: throughput values are prefixed with `~` to mark them as estimates. This appears in the services table, service map nodes, and service map edges. +- **Secondary "traced" line**: the services table shows the actual traced rate under the estimate (e.g. `~8.3 traced`). +- **Tooltip**: hovering the throughput cell shows the sampling rate and extrapolation factor (e.g. "Estimated from 10% sampled traces (x10 extrapolation)"). ## Limitations -- **Edge throughput** -- service-to-service call counts on the service map use the same per-row `SampleRate` weighting. The edge label shows `~` when sampling is active. -- **Error rate** -- sampled error spans use the same per-row weights as throughput. Maple computes `sumIf(SampleRate, StatusCode = 'Error') / sum(SampleRate)`, so mixed sampling rates do not over-represent aggressively retained errors. If an upstream sampler does not report reliable inclusion weights, exact pre-sampling error rates still require SpanMetrics. +- **Edge throughput**: service-to-service call counts on the service map use the same per-row `SampleRate` weighting. The edge label shows `~` when sampling is active. +- **Error rate**: sampled error spans use the same per-row weights as throughput. Maple computes `sumIf(SampleRate, StatusCode = 'Error') / sum(SampleRate)`, so mixed sampling rates do not over-represent aggressively retained errors. If an upstream sampler does not report reliable inclusion weights, exact pre-sampling error rates still require SpanMetrics. ## For best results -If you need 100%-accurate RED metrics (Rate, Errors, Duration) alongside sampled traces, use the **OpenTelemetry Collector SpanMetrics Connector**. It derives metrics from every span before sampling is applied, giving you exact counts regardless of trace sampling configuration. +For exact RED metrics (Rate, Errors, Duration) alongside sampled traces, use the **OpenTelemetry Collector SpanMetrics Connector**. It derives metrics from every span before sampling, so counts stay exact whatever the trace sampling configuration. - [SpanMetrics Connector docs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) @@ -95,4 +95,4 @@ service: exporters: [otlp] ``` -This way, your metrics pipeline sees every request while your traces pipeline can sample aggressively. +The metrics pipeline sees every request while the traces pipeline samples aggressively. diff --git a/docs/self-hosted-clickhouse.md b/docs/self-hosted-clickhouse.md index b0378eeb1e..7b7e2c58d4 100644 --- a/docs/self-hosted-clickhouse.md +++ b/docs/self-hosted-clickhouse.md @@ -1,87 +1,70 @@ # Self-hosted Maple on plain ClickHouse -Self-managed Maple can run on a vanilla ClickHouse instance — no Tinybird Cloud, no Tinybird-Local. The query engine (`@maple/query-engine`) emits standard ClickHouse SQL, and the schema is generated from the same TypeScript source the Tinybird path uses. +Self-managed Maple can run on a vanilla ClickHouse instance, with no Tinybird Cloud and no Tinybird-Local. The query engine (`@maple/query-engine`) emits standard ClickHouse SQL, and the schema is generated from the same TypeScript source the Tinybird path uses. ## Scope This doc covers two pieces: -1. **Schema setup** — generating + applying the ClickHouse DDL on a vanilla server. -2. **Runtime configuration** — pointing the Maple API at ClickHouse instead of Tinybird. +1. **Schema setup:** generating and applying the ClickHouse DDL on a vanilla server. +2. **Runtime configuration:** pointing the Maple API at ClickHouse instead of Tinybird. -Ingest is still bring-your-own — see [Ingest options](#ingest-options) below. +Ingest is still bring-your-own. See [Ingest options](#ingest-options) below. ## Required ClickHouse version -Tested on ClickHouse 24.8+. Earlier versions may work but aren't validated. +Tested on ClickHouse 24.8+. Earlier versions may work but aren't validated. The local dev stack and CI's ClickHouse e2e jobs run 26.2. ## How runtime config works -Self-managed Maple is a **per-org BYO** feature. Each org configures their own backend in Settings → BYO; the credentials live in the `org_tinybird_settings` table (encrypted at rest with `MAPLE_INGEST_KEY_ENCRYPTION_KEY`). A `backend` column discriminates between the two flavors: +Self-managed Maple is a **per-org BYO** feature. Each org configures its own ClickHouse under Settings → "Bring your own ClickHouse". The credentials live in the `org_clickhouse_settings` Postgres table (`packages/db/src/schema/org-clickhouse-settings.ts`), with the password encrypted at rest with `MAPLE_INGEST_KEY_ENCRYPTION_KEY`. There is no sync workflow: the schema lives in the org's ClickHouse instance and is applied from the settings page or the CLI below. -- `backend = "tinybird"` — the existing path. Maple deploys its Tinybird project into the org's workspace via the sync workflow; queries route to that workspace. -- `backend = "clickhouse"` — new. The org points Maple at a vanilla ClickHouse server they operate themselves. There is no sync workflow — schema lives in their CH instance and is applied via the CLI below. +Orgs without a BYO row use the deployment's managed warehouse. API query routing needs no new env vars for BYO ClickHouse. Postgres-backed direct ingest does need `MAPLE_INGEST_KEY_ENCRYPTION_KEY` so the ingest gateway can decrypt stored ClickHouse passwords. -The Maple deployment itself still uses the env-level `TINYBIRD_HOST` / `TINYBIRD_TOKEN` for any org without a BYO row. API query routing does not require new env vars for ClickHouse-BYO; Postgres-backed direct ingest does require `MAPLE_INGEST_KEY_ENCRYPTION_KEY` so the ingest gateway can decrypt stored ClickHouse passwords. +The managed warehouse is Tinybird (`TINYBIRD_HOST` / `TINYBIRD_TOKEN`) unless env-level `CLICKHOUSE_URL` is set. `CLICKHOUSE_PROVIDER` defaults to `tinybird`, so a set `CLICKHOUSE_URL` is treated as Tinybird's ClickHouse-compatible gateway: raw SQL substitutes a per-org JWT and removes Tinybird-restricted query settings. For a vanilla/self-managed server, set `CLICKHOUSE_PROVIDER=clickhouse`; Maple then keeps `CLICKHOUSE_PASSWORD` for raw SQL. Tinybird raw SQL also requires explicit `TINYBIRD_SIGNING_KEY` and `TINYBIRD_WORKSPACE_ID` values. Maple never derives either from the API token. Set `TINYBIRD_RAW_SQL_JWT_RPS_LIMIT` to a positive integer to add an optional Tinybird-enforced request ceiling; Maple gives each org an independent bucket. -Env-level `CLICKHOUSE_URL` defaults to Tinybird's ClickHouse-compatible gateway for -compatibility with existing deployments; raw SQL substitutes a per-org JWT and -removes Tinybird-restricted query settings. For a vanilla/self-managed server, set -`CLICKHOUSE_PROVIDER=clickhouse`; Maple then preserves `CLICKHOUSE_PASSWORD` for raw -SQL. Tinybird raw SQL also requires explicit `TINYBIRD_SIGNING_KEY` and -`TINYBIRD_WORKSPACE_ID` values; Maple never derives either from the API token. -Set `TINYBIRD_RAW_SQL_JWT_RPS_LIMIT` to a positive integer to add an optional -Tinybird-enforced request ceiling; Maple gives each org an independent bucket. +Env-level vanilla ClickHouse raw SQL is enabled only when `MAPLE_AUTH_MODE=self_hosted` (the default), where the deployment is single-org. Hosted multi-org deployments fail closed unless they use Tinybird's scoped JWT path or per-org BYO credentials. -Env-level vanilla ClickHouse raw SQL is enabled only when `MAPLE_AUTH_MODE=self_hosted`, -where the deployment is single-org. Hosted multi-org deployments fail closed unless -they use Tinybird's scoped JWT path or per-org BYO credentials. - -For user-authored SQL, configure the runtime ClickHouse account as SELECT-only on the -Maple database. Apply schema migrations with a separate administrative account, and -enforce server-side limits such as maximum execution time, memory, rows, and bytes -read. Maple's client-side caps are a final response boundary, not a substitute for a -least-privilege role and server-side resource controls. See -[ClickHouse's guidance for agent-authored queries](https://clickhouse.com/blog/how-to-set-up-clickhouse-for-agentic-analytics) -for the corresponding database-side setup. +For user-authored SQL, configure the runtime ClickHouse account as SELECT-only on the Maple database. Apply schema migrations with a separate administrative account, and enforce server-side limits such as maximum execution time, memory, rows, and bytes read. Maple's client-side caps are a final response boundary. They do not replace a least-privilege role and server-side resource controls. See [ClickHouse's guidance for agent-authored queries](https://clickhouse.com/blog/how-to-set-up-clickhouse-for-agentic-analytics) for the corresponding database-side setup. ### Routing precedence For any given query the API resolves the upstream in this order: -1. **Per-org BYO row** — if `org_tinybird_settings` has an active row for the requesting org, the row's `backend` discriminator picks Tinybird or ClickHouse, and the row's credentials drive the connection. -2. **Managed Tinybird** — fall back to `TINYBIRD_HOST` + `TINYBIRD_TOKEN`. +1. **Per-org BYO row:** if `org_clickhouse_settings` has an active row for the requesting org, the row's credentials drive the connection. +2. **Managed warehouse:** otherwise, `CLICKHOUSE_URL` if set, else `TINYBIRD_HOST` + `TINYBIRD_TOKEN`. -### Configuring ClickHouse-BYO via the UI +### Configuring BYO ClickHouse via the UI -`Settings → BYO Backend → ClickHouse` exposes: +Settings → "Bring your own ClickHouse" exposes: -- **ClickHouse URL** — the HTTP interface (e.g. `https://your-clickhouse.example.com:8123`) -- **User** — defaults to `default`. Must have DDL privileges (CREATE TABLE / CREATE MATERIALIZED VIEW) so the API can install the schema on save. -- **Database** — defaults to `default` -- **Password** — optional; encrypted at rest. Leave blank to keep an existing password when re-saving. +- **ClickHouse URL:** the HTTP interface (e.g. `https://your-clickhouse.example.com:8123`). +- **User:** defaults to `default`. Applying the schema from the UI needs DDL privileges (CREATE TABLE / CREATE MATERIALIZED VIEW). +- **Database:** defaults to `default`. +- **Password:** optional; encrypted at rest. Leave it blank to keep the stored password when re-saving. Changing the URL, user or database requires re-entering it. -On save, the API connects to the instance, creates the schema (or skips if it already exists) inside an idempotent migration loop, and only then persists the BYO row. There is no resync flow — schema lives in your CH instance and there's nothing to push back to it. +On save, the API validates the connection with `SELECT 1` and persists the row. Save runs no DDL. The Schema card then shows a diff against the bundled schema, and **Apply schema** runs the migrations. ## Applying the schema -Two ways to apply the schema. Both run the **same migrations** from `@maple/domain/clickhouse` and use the **same** [`qualifyStatementForDatabase`](../packages/domain/src/clickhouse/qualify.ts) helper, so they're interchangeable. +There are two ways to apply the schema. Both run the **same migrations** from `@maple/domain/clickhouse` and use the **same** [`qualifyStatementForDatabase`](../packages/domain/src/clickhouse/qualify.ts) helper, so they're interchangeable. ### Via the Maple UI (default) -`Settings → BYO Backend → ClickHouse`. On save, the API connects to the supplied URL with the supplied credentials and: +Settings → "Bring your own ClickHouse" → Schema → **Apply schema**. The API starts a background workflow (`apps/api/src/workflows/ClickHouseSchemaApplyWorkflow.ts`), because backfill migrations can outlast one request. The page polls its progress. The workflow: 1. Creates `_maple_schema_migrations` (the bookkeeping table) if missing. 2. Applies any unapplied migrations in version order. 3. Records each applied migration's `(version, applied_at, description)`. +4. Stamps `schema_version` in `org_clickhouse_settings` (see readiness below). -If the connection fails or the user lacks DDL privileges, the save returns an error and the row is **not** persisted — so an unconfigured org never gets stuck pointed at a half-migrated CH instance. +If the connection fails or the user lacks DDL privileges, the apply fails and the settings row shows the error. -Re-saving is safe. Already-applied migrations are skipped, and every statement uses `IF NOT EXISTS` as a second line of defense. Future schema upgrades land the same way: pull a new Maple API release, then have the org re-save (or any other write to the BYO row) to pick up new migrations. +Re-applying is safe. Already-applied migrations are skipped, and every statement uses `IF NOT EXISTS` as a second line of defense. Future schema upgrades land the same way: pull a new Maple API release, then apply again to pick up new migrations. ### Via the standalone CLI -For airgapped clusters, CI checks, or when your CH credentials shouldn't pass through Maple's API: +For airgapped clusters, CI checks, or when your ClickHouse credentials shouldn't pass through Maple's API: ```bash bunx @maple/clickhouse-cli@latest apply \ @@ -96,9 +79,9 @@ bunx @maple/clickhouse-cli@latest status bunx @maple/clickhouse-cli@latest dry-run ``` -Connection flags fall back to `MAPLE_CH_URL`, `MAPLE_CH_USER`, `MAPLE_CH_PASSWORD`, `MAPLE_CH_DATABASE` env vars — handy in CI. See [`packages/clickhouse-cli/README.md`](../packages/clickhouse-cli/README.md). +Connection flags fall back to the `MAPLE_CH_URL`, `MAPLE_CH_USER`, `MAPLE_CH_PASSWORD` and `MAPLE_CH_DATABASE` env vars, which is handy in CI. See [`packages/clickhouse-cli/README.md`](../packages/clickhouse-cli/README.md). -To inspect applied migrations directly on your CH server: +To inspect applied migrations directly on your ClickHouse server: ```sql SELECT version, applied_at, description FROM _maple_schema_migrations ORDER BY version; @@ -106,73 +89,76 @@ SELECT version, applied_at, description FROM _maple_schema_migrations ORDER BY v ## What gets created -On a clean install, migration 0001 creates **37 tables** (datasources) and **39 materialized views**. -Migration 0001 re-exports the _generated_ snapshot, so these counts track -`datasources.ts` / `materializations.ts` — regenerate with `bun run clickhouse:schema` +On a clean install, migration 0001 creates **40 tables** (datasources) and **42 materialized views**. +Migration 0001 re-exports the _generated_ snapshot (`packages/domain/src/generated/clickhouse-schema.ts`), so these counts track +`datasources.ts` / `materializations.ts`. Regenerate with `bun run clickhouse:schema` and `bun run tinybird:manifest` after editing either, or CI's drift gate fails. -- **Direct-ingest tables** (12): `alert_checks`, `logs`, `metrics_exponential_histogram`, - `metrics_gauge`, `metrics_histogram`, `metrics_sum`, `service_address_resolutions_hourly`, - `service_map_edges_hourly_ingest`, `session_events`, `session_replay_events`, - `session_replays`, `traces` -- **MV-populated tables** (25): `attribute_keys_hourly`, `attribute_values_hourly`, - `error_events`, `error_events_by_time`, `error_fingerprints_minutely`, - `logs_aggregates_hourly`, `metric_catalog`, `service_external_edges_hourly`, - `service_map_children`, `service_map_db_edges_hourly`, +- **Tables with no MV feeding them** (13): `alert_checks`, `audit_log`, `logs`, + `metrics_exponential_histogram`, `metrics_gauge`, `metrics_histogram`, `metrics_sum`, + `service_address_resolutions_hourly`, `service_map_edges_hourly_ingest` (a `Null`-engine + bridge), `session_events`, `session_replay_events`, `session_replays`, `traces` +- **MV-populated tables** (27): `ai_trace_index`, `attribute_keys_hourly`, + `attribute_values_hourly`, `error_events`, `error_events_by_time`, + `error_fingerprints_minutely`, `identity_links`, `logs_aggregates_hourly`, `metric_catalog`, + `product_events` (also written directly by `POST /v1/events`), + `service_external_edges_hourly`, `service_map_children`, `service_map_db_edges_hourly`, `service_map_db_query_shapes_hourly`, `service_map_edges_hourly`, `service_map_spans`, `service_operations_hourly`, `service_operations_minutely`, `service_overview_hourly`, `service_overview_minutely`, `service_overview_spans`, `service_platforms_hourly`, `service_usage`, `span_metrics_calls_hourly`, `trace_detail_spans`, `trace_list_mv`, - `traces_aggregates_hourly`, `web_events` -- **Materialized views** (39): fan out from the direct-ingest tables to populate the - MV-populated tables. Several targets are fed by more than one MV — `service_usage` by - six, `attribute_values_hourly` / `attribute_keys_hourly` / `metric_catalog` by four each. + `traces_aggregates_hourly` +- **Materialized views** (42): fan out from the source tables to populate the + MV-populated tables. Several targets are fed by more than one MV: `service_usage` by + six, `attribute_values_hourly` / `attribute_keys_hourly` / `metric_catalog` by four each, + and `product_events` by two. See [`warehouse-rollups.md`](warehouse-rollups.md) for when a materialized view is the right answer and which tier a query should read. -Every table is partitioned by date and carries a TTL, tiered by how raw the data is: +Every stored table is partitioned by date and carries a TTL, tiered by how raw the data is: -| Retention | Tables | -| ------------ | ------------------------------------------------------------------------------------------------------------------------------ | -| **30 days** | `traces`, `trace_detail_spans`, `logs`, `service_map_spans`, `service_map_children`, `service_overview_spans`, `trace_list_mv` | -| **90 days** | `error_events`, `error_events_by_time`, `metrics_*`, `attribute_*_hourly`, `metric_catalog` | -| **365 days** | hourly rollups (`*_hourly`), `service_usage`, `alert_checks` | +| Retention | Tables | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **30 days** | `traces`, `trace_detail_spans`, `logs`, `service_map_spans`, `service_map_children`, `service_overview_spans`, `trace_list_mv`, `ai_trace_index`, `session_events`, `session_replay_events`, `session_replays` | +| **90 days** | `error_events`, `error_events_by_time`, `error_fingerprints_minutely`, `metrics_*`, `attribute_*_hourly`, `metric_catalog`, `*_minutely` rollups, `span_metrics_calls_hourly` | +| **365 days** | the other hourly rollups (`*_hourly`), `service_usage`, `alert_checks`, `product_events`, `identity_links` | +| **2190 days** | `audit_log` | Adjust by writing a follow-up migration if your retention requirements differ. -> **Changing a TTL requires an `ALTER`, not just a datasource edit.** Migration 0001 re-exports the generated snapshot, and every statement in it is `CREATE TABLE IF NOT EXISTS` — so on a cluster whose tables already exist, re-running it is a no-op and the old TTL survives. Editing `datasources.ts` alone therefore changes **new installs only**. Ship a paired `ALTER TABLE … MODIFY TTL` delta (see `0011_align_raw_telemetry_retention.ts`) or existing clusters silently keep the previous retention. This exact gap let a production cluster accumulate 3× its intended trace volume until the disk filled. +> **Changing a TTL requires an `ALTER`, not just a datasource edit.** Migration 0001 re-exports the generated snapshot, and every statement in it is `CREATE TABLE IF NOT EXISTS`. On a cluster whose tables already exist, re-running it is a no-op and the old TTL survives. Editing `datasources.ts` alone therefore changes **new installs only**. Ship a paired `ALTER TABLE … MODIFY TTL` delta (see `0009_one_year_service_history.ts`) or existing clusters silently keep the previous retention. This exact gap let a production cluster accumulate 3× its intended trace volume until the disk filled. > -> When lowering a TTL, set `ttl_only_drop_parts = 1` **first**. These tables partition on the same expression their TTL keys off, so expired parts are always wholly expired — the setting turns eviction into a whole-partition drop instead of a multi-TiB part rewrite. +> When lowering a TTL, set `ttl_only_drop_parts = 1` **first**. These tables partition on the same expression their TTL keys off, so expired parts are always wholly expired. The setting turns eviction into a whole-partition drop instead of a multi-TiB part rewrite. ## Ingest options -The maintained standalone path is **Option A: Maple's prebuilt OTel Collector image** (`mapleexporter` baked in). Hosted/self-hosted Maple deployments can also use the Rust ingest gateway's direct ClickHouse path once an org is marked ready. Three escape hatches stay supported for advanced setups. +The maintained standalone path is **Option A: Maple's prebuilt OTel Collector image** (`mapleexporter` baked in). Hosted and self-hosted Maple deployments can also use the Rust ingest gateway's direct ClickHouse path once an org is marked ready. Three escape hatches stay supported for advanced setups. ### Maple ingest gateway direct ClickHouse path -For orgs whose `org_clickhouse_settings` row has `sync_status = 'connected'` and `schema_version` equal to the bundled `clickHouseSchemaVersion` (the latest **ingest-required** ClickHouse migration version — emitted into the gateway as `SCHEMA_VERSION` by `scripts/generate-clickhouse-insert-mappings.ts`), the Rust ingest gateway routes accepted native-ingest frames directly to that org's ClickHouse HTTP endpoint. Non-ready orgs continue using the managed Tinybird path. Performance-only migrations set `requiredForIngest: false`, so an index rollout cannot un-ready an otherwise compatible org. Readiness also does **not** use the Tinybird-coupled `clickHouseProjectRevision`, so Tinybird-only changes cannot alter BYO-ClickHouse routing. +The Rust ingest gateway routes an org's accepted native-ingest frames directly to that org's ClickHouse HTTP endpoint when its `org_clickhouse_settings` row has `sync_status = 'connected'` and a `schema_version` equal to the bundled `clickHouseSchemaVersion`. That value is the latest **ingest-required** ClickHouse migration version, emitted into the gateway as `SCHEMA_VERSION` by `scripts/generate-clickhouse-insert-mappings.ts`. Non-ready orgs continue using the managed Tinybird path. Performance-only migrations set `requiredForIngest: false`, so an index rollout cannot un-ready an otherwise compatible org. Readiness also does **not** use the Tinybird-coupled `clickHouseProjectRevision`, so Tinybird-only changes cannot alter BYO-ClickHouse routing. -Postgres-backed ingest deployments must set `MAPLE_INGEST_KEY_ENCRYPTION_KEY` before rolling out this mode; the gateway exits at startup without it because ClickHouse passwords are encrypted at rest with the same AES-256-GCM key format as private ingest keys. +Postgres-backed ingest deployments must set `MAPLE_INGEST_KEY_ENCRYPTION_KEY` before rolling out this mode. The gateway exits at startup without it, because ClickHouse passwords are encrypted at rest with the same AES-256-GCM key format as private ingest keys. Operational caveats: -- **Readiness keys on the latest ingest-required migration, which only the API marks.** The `schema_version` stored in Postgres is written to `clickHouseSchemaVersion` **only** by the API's `applySchema` workflow (or by `schemaDiff` self-heal, below). A credential re-save _preserves_ the prior value, and the standalone `clickhouse-cli` writes `_maple_schema_migrations` **on your CH server but never touches Maple's application database**. So an org whose ClickHouse schema was applied entirely via the CLI stays `schema_version`-stale and the gateway keeps routing to Tinybird, even though the cluster is fully migrated. Symptom: the dashboard (which reads CH whenever a settings row exists) shows collector-written data, but data sent through the public ingestor is invisible because it landed in Tinybird. -- **Self-heal:** calling `schemaDiff` (e.g. opening `Settings → BYO Backend → ClickHouse`, or `POST /orgClickHouseSettings/schemaDiff`) re-stamps `schema_version` to `clickHouseSchemaVersion` whenever the live schema is fully in sync (every diff entry `up_to_date`). This is the supported way to mark a CLI-applied org ready without forcing an Apply that has nothing to migrate. The read path also annotates a `clickhouse.schemaDrift` span attribute (`OrgClickHouseSettingsService.resolveRuntimeConfig`) — alert on it to catch stale orgs. +- **Readiness keys on the latest ingest-required migration, which only the API marks.** The `schema_version` stored in Postgres is set to `clickHouseSchemaVersion` **only** by the API's apply workflow (or by the `schemaDiff` self-heal, below). A credential re-save _preserves_ the prior value. The standalone `clickhouse-cli` writes `_maple_schema_migrations` **on your ClickHouse server but never touches Maple's application database**. So an org whose schema was applied entirely via the CLI stays `schema_version`-stale, and the gateway keeps routing to Tinybird even though the cluster is fully migrated. Symptom: the dashboard (which reads ClickHouse whenever a settings row exists) shows collector-written data, but data sent through the public ingestor is invisible because it landed in Tinybird. +- **Self-heal:** calling `schemaDiff` (opening the settings page, or `GET /api/org-clickhouse-settings/schema-diff`) re-stamps `schema_version` to `clickHouseSchemaVersion` whenever the live schema is fully in sync (every diff entry `up_to_date`). This is the supported way to mark a CLI-applied org ready without forcing an Apply that has nothing to migrate. The read path also annotates a `clickhouse.schemaDrift` span attribute (`OrgClickHouseSettingsService.resolveRuntimeConfig`). Alert on it to catch stale orgs. - ClickHouse-routed frames never fall back to Tinybird. After the configured export retry budget is exhausted, the batch is dropped, the WAL cursor advances, and `ingest_clickhouse_export_dropped_total` records the datasource and final drop reason. Alert on any non-zero increase in that counter. -- Password-authenticated ClickHouse endpoints must use `https://`; the gateway drops passworded `http://` targets before attaching `X-ClickHouse-Key`. -- Direct ClickHouse routing writes WAL v3 frames. Do not roll back to a pre-direct-ClickHouse ingest binary while v3 frames may remain in the queue; first drain the WAL, or accept that clearing the queue directory is a data-loss recovery step. +- Password-authenticated ClickHouse endpoints must use `https://`. The gateway drops passworded `http://` targets before attaching `X-ClickHouse-Key`. +- Direct ClickHouse routing writes WAL v3 frames. Do not roll back to a pre-direct-ClickHouse ingest binary while v3 frames may remain in the queue. Drain the WAL first, or accept that clearing the queue directory is a data-loss recovery step. -### Option A — Maple OTel Collector (recommended) +### Option A: Maple OTel Collector (recommended) -A custom build of `otelcol-contrib` with the `mapleexporter` baked in. The exporter writes JSON-each-row directly into Maple's `traces` / `logs` / `metrics_*` tables, no shim required. +A custom build of `otelcol-contrib` with the `mapleexporter` baked in. The exporter writes JSON-each-row directly into Maple's `traces` / `logs` / `metrics_*` tables, with no shim. -- **Image:** `ghcr.io/mapletechlabs/maple/otel-collector-maple` (multi-arch — amd64 + arm64). Pin a tag (e.g. `0.2.0`); see [the package page](https://github.com/orgs/MapleTechLabs/packages/container/package/maple%2Fotel-collector-maple) for available versions. -- **Source:** [`packages/otel-collector-maple-exporter/`](../packages/otel-collector-maple-exporter/) — builder config in [`deploy/k8s-infra/builder-config.yaml`](../deploy/k8s-infra/builder-config.yaml), Dockerfile in [`deploy/k8s-infra/Dockerfile.otel-collector-maple`](../deploy/k8s-infra/Dockerfile.otel-collector-maple). +- **Image:** `ghcr.io/mapletechlabs/maple/otel-collector-maple` (multi-arch: amd64 + arm64). Pin a tag (e.g. `0.2.0`); see [the package page](https://github.com/orgs/MapleTechLabs/packages/container/package/maple%2Fotel-collector-maple) for available versions. +- **Source:** [`packages/otel-collector-maple-exporter/`](../packages/otel-collector-maple-exporter/). Builder config in [`deploy/k8s-infra/builder-config.yaml`](../deploy/k8s-infra/builder-config.yaml), Dockerfile in [`deploy/k8s-infra/Dockerfile.otel-collector-maple`](../deploy/k8s-infra/Dockerfile.otel-collector-maple). #### Step 1: apply the schema -Use the standalone CLI — no Maple API required, and your CH credentials never leave the machine running it: +Use the standalone CLI. It needs no Maple API, and your ClickHouse credentials never leave the machine running it: ```bash bunx @maple/clickhouse-cli@latest apply \ @@ -181,11 +167,11 @@ bunx @maple/clickhouse-cli@latest apply \ --database=default ``` -Or save credentials in the Maple UI under `Settings → BYO Backend → ClickHouse` and the API will run the same migrations on your behalf. +Or save credentials in the Maple UI under Settings → "Bring your own ClickHouse" and click **Apply schema**; the API runs the same migrations on your behalf. #### Step 2: deploy the collector -**Kubernetes** — install the [`maple-otel`](../deploy/maple-otel/) Helm chart: +**Kubernetes:** install the [`maple-otel`](../deploy/maple-otel/) Helm chart: ```bash helm install maple-otel oci://ghcr.io/mapletechlabs/charts/maple-otel \ @@ -197,9 +183,9 @@ helm install maple-otel oci://ghcr.io/mapletechlabs/charts/maple-otel \ Apps then point `OTEL_EXPORTER_OTLP_ENDPOINT` at `http://maple-otel.maple.svc.cluster.local:4318`. -**Anywhere else (Docker / VM / ECS / Nomad / …)** — download a pre-rendered config from Maple: +**Anywhere else (Docker / VM / ECS / Nomad / …):** download a pre-rendered config from Maple: -1. `Settings → BYO Backend → ClickHouse → Download collector config` (or `GET /api/org-clickhouse-settings/collector-config`). +1. `GET /api/org-clickhouse-settings/collector-config` (there is no download button in the settings UI). 2. Drop the YAML next to a copy of the image and run: ```bash @@ -210,13 +196,13 @@ Apps then point `OTEL_EXPORTER_OTLP_ENDPOINT` at `http://maple-otel.maple.svc.cl ghcr.io/mapletechlabs/maple/otel-collector-maple:0.2.0 ``` -The rendered YAML carries your `org_id`, ClickHouse URL/user/database, and the standard memory_limiter → k8sattributes → batch → maple pipeline. The password is referenced via `${env:MAPLE_CLICKHOUSE_PASSWORD}` so the file is safe to share. +The rendered YAML carries your `org_id`, ClickHouse URL/user/database, and the standard memory_limiter → k8sattributes → batch → maple pipeline. The password is referenced via `${env:MAPLE_CLICKHOUSE_PASSWORD}`, so the file is safe to share. #### Org id resolution -The `mapleexporter` stamps `OrgId` from its own `org_id` config on every record — no upstream `resource/maple_org` processor required for the typical single-tenant deploy. For multi-tenant fan-out (one collector serving several Maple orgs) set `org_id_from_resource_attribute: maple_org_id` on the exporter and stamp the right id per-record upstream; see [`packages/otel-collector-maple-exporter/README.md`](../packages/otel-collector-maple-exporter/README.md). +The `mapleexporter` stamps `OrgId` from its own `org_id` config on every record, so the typical single-tenant deploy needs no upstream `resource/maple_org` processor. For multi-tenant fan-out (one collector serving several Maple orgs), set `org_id_from_resource_attribute: maple_org_id` on the exporter and stamp the right id per record upstream. See [`packages/otel-collector-maple-exporter/README.md`](../packages/otel-collector-maple-exporter/README.md). -### Option B — Tinybird exporter + a shim service +### Option B: Tinybird exporter + a shim service The original "drop-in for Tinybird Cloud users" path. Run otelcol-contrib's `tinybird` exporter and point it at a small shim that: @@ -224,22 +210,22 @@ The original "drop-in for Tinybird Cloud users" path. Run otelcol-contrib's `tin - Applies the JSONPath mappings from `packages/domain/src/tinybird/datasources.ts` to project each row into the right column shape. - Issues `INSERT INTO FORMAT JSONEachRow` against ClickHouse. -The shim is not in this repo — operators write or fork their own. The JSONPath spec required to drive it is exposed via `emitJsonPathSpec()` in `@maple/domain/clickhouse`. +The shim is not in this repo; operators write or fork their own. The JSONPath spec required to drive it is exposed via `emitJsonPathSpec()` in `@maple/domain/clickhouse`. -### Option C — Tinybird-Local +### Option C: Tinybird-Local -If you're not allergic to running another container, [tinybird-local](https://github.com/tinybirdco/tinybird-local) is a single-binary Tinybird-API-compatible local server backed by ClickHouse. The Tinybird exporter works against it unchanged. Heavier than Option A but useful when you want Tinybird's UI side-by-side. +[tinybird-local](https://github.com/tinybirdco/tinybird-local) is a single-binary, Tinybird-API-compatible local server backed by ClickHouse. The Tinybird exporter works against it unchanged. It is heavier than Option A, but useful when you want Tinybird's UI side by side. -### Option D — Direct INSERTs from your application +### Option D: Direct INSERTs from your application -If you have a small, well-defined ingest path (e.g. you control the SDK that emits to Maple), nothing stops you from `INSERT INTO traces FORMAT JSONEachRow` directly. The JSONPath spec defines what shape the rows should be in. Each row should look like the Tinybird exporter's output — see the `$.…` paths in `datasources.ts`. +If you have a small, well-defined ingest path (e.g. you control the SDK that emits to Maple), you can `INSERT INTO traces FORMAT JSONEachRow` directly. The JSONPath spec defines the row shape. Each row should look like the Tinybird exporter's output; see the `$.…` paths in `datasources.ts`. ### Comparing the options | | Option A (Maple OTel Collector) | Option B (shim) | Option C (Tinybird-Local) | Option D (direct INSERTs) | | -------------------------------------- | --------------------------------------- | ----------------- | ------------------------- | ------------------------- | | Setup steps | 2 (schema + collector) | Many (write shim) | 2 | Application-specific | -| Pre-built image | ✅ | — | ✅ (Tinybird's) | — | +| Pre-built image | ✅ | none | ✅ (Tinybird's) | none | | Multi-tenant fan-out | ✅ via `org_id_from_resource_attribute` | manual | manual | application | | k8s pod metadata enrichment | ✅ baked into the image | manual | manual | manual | | Standard OTel collector pipeline shape | ✅ | partial | partial | n/a | @@ -248,8 +234,8 @@ If you have a small, well-defined ingest path (e.g. you control the SDK that emi Schema lives in `packages/domain/src/tinybird/datasources.ts` and `materializations.ts`. These TypeScript files are consumed by **two** emitters: -- The Tinybird manifest emitter (existing) — produces `.datasource` / `.pipe` files for Tinybird Cloud -- The ClickHouse DDL emitter (new) — produces `CREATE TABLE` / `CREATE MATERIALIZED VIEW` statements +- The Tinybird manifest emitter produces `.datasource` / `.pipe` files for Tinybird Cloud. +- The ClickHouse DDL emitter (`packages/domain/src/clickhouse/ddl-emitter.ts`) produces `CREATE TABLE` / `CREATE MATERIALIZED VIEW` statements. To regenerate the ClickHouse schema after a TS change: @@ -257,7 +243,7 @@ To regenerate the ClickHouse schema after a TS change: bun run clickhouse:schema ``` -CI checks this stays in sync via `bun run clickhouse:schema:check`. +CI checks that it stays in sync via `bun run clickhouse:schema:check`. ## Extending the schema @@ -265,7 +251,7 @@ To add a new column, table, or materialized view: 1. Edit `packages/domain/src/tinybird/datasources.ts` or `materializations.ts`. 2. Run `bun run clickhouse:schema` to regenerate the snapshot. -3. Create a new file `packages/domain/src/clickhouse/migrations/0002_.ts`: +3. Create a new file `packages/domain/src/clickhouse/migrations/00NN_.ts`, using the next free version number: ```typescript export const migration_0002_add_foo_column = { @@ -279,9 +265,11 @@ To add a new column, table, or materialized view: } as const ``` + Set `requiredForIngest: false` on a performance-only migration (an index, a projection) so it does not un-ready BYO orgs for ingest. + 4. Append it to the `migrations` array in `packages/domain/src/clickhouse/migrations/index.ts`. -The next `clickhouse:schema:apply` will pick it up and run only the new migration. +The next Apply (UI or `clickhouse-cli apply`) picks it up and runs only the new migration. ### Replacing Tinybird's `forwardQuery` @@ -296,18 +284,18 @@ ALTER TABLE traces MATERIALIZE COLUMN NewCol; ## Migrating from Tinybird-Local -Clean break is recommended — Maple's data has a 90-day TTL by default, so most operators can: +A clean break is recommended. Raw telemetry keeps 30 or 90 days, so most operators can: -1. Stop your ingest path. +1. Stop the ingest path. 2. Bring up the new vanilla-ClickHouse stack and apply the schema. -3. Resume ingest. Old data ages out within 90 days. +3. Resume ingest. Old raw data ages out within 90 days; the 365-day rollups start empty. -If you need historical data preserved, Tinybird-Local exposes its underlying ClickHouse on port 7181, so a one-shot `INSERT INTO new. SELECT * FROM tinybird_local.
` over `remote()` is feasible. This isn't currently shipped as tooling. +If you need historical data preserved, Tinybird-Local exposes its underlying ClickHouse on port 7181, so a one-shot `INSERT INTO new.
SELECT * FROM tinybird_local.
` over `remote()` is feasible. This isn't shipped as tooling. ## Troubleshooting -- **Save fails with "ClickHouse rejected credentials"**: the user/password combo doesn't authenticate. Maple maps CH 401/403 responses to this error. -- **Save fails with "ClickHouse rejected statement"**: the configured user authenticates but lacks DDL privileges, or a migration ran into a CH-version-specific syntax issue. Check `system.query_log` on your CH server for the failing statement. -- **Save fails with "Could not reach ClickHouse"**: the API can't make an HTTP request to the URL. Network/DNS/firewall — verify the API can reach the URL. +- **"ClickHouse rejected credentials"**: the user/password combo doesn't authenticate. Maple maps ClickHouse 401/403 responses to this error. +- **"ClickHouse rejected statement"** (during apply): the configured user authenticates but lacks DDL privileges, or a migration hit a version-specific syntax issue. Check `system.query_log` on your ClickHouse server for the failing statement. +- **"Could not reach ClickHouse"**: the API can't make an HTTP request to the URL. This is network, DNS or firewall; verify the API can reach the URL. - **Migration appears to hang on `MATERIALIZE COLUMN`**: this is a background mutation. Watch `system.mutations` to see progress. -- **`schema:check` fails in CI but the diff looks empty**: someone changed `datasources.ts` without running `bun run clickhouse:schema`. Run it locally and commit the regenerated file. +- **`clickhouse:schema:check` fails in CI but the diff looks empty**: someone changed `datasources.ts` without running `bun run clickhouse:schema`. Run it locally and commit the regenerated file. diff --git a/docs/service-map-architecture.md b/docs/service-map-architecture.md index 9ec0b78864..c36a33b784 100644 --- a/docs/service-map-architecture.md +++ b/docs/service-map-architecture.md @@ -2,8 +2,8 @@ The map is the densest read surface in the product: one page load fans out across four warehouse tables, two rollups a scheduled job fills, and two control-plane inventories, then -merges all of it into one graph. This is what it reads today, what the invariants are, and -where it is going — read it before adding another overlay. +merges all of it into one graph. This page covers what it reads today, the invariants, and +where it is going. Read it before adding another overlay. --- @@ -22,7 +22,7 @@ Clicking a database node adds three more (`serviceDbQuerySummary`, `…Timeserie Two things about that table are not yet true but should be: - **The Cloudflare and PlanetScale query sets run unconditionally.** An org with neither - integration still pays five metrics scans per load to render nothing — and the inventory + integration still pays five metrics scans per load to render nothing, and the inventory that would prove them empty arrives in a _different_ request. - **`serviceOverview` is the services-_list_ query.** 500 rows, each carrying a 20-element `commits` tuple array built by a per-commit `GROUP BY` and tDigest merge. The map reads @@ -43,21 +43,23 @@ interior, and raw rows for the two partial hours at the ends. | service → external | `service_external_edges_hourly` | `traces` | MV | | db query shapes | `service_map_db_query_shapes_hourly` | `traces` | MV | -**The two tiers must tile the window exactly once — no gap, no overlap.** That boundary is -not written per query; it comes from `ch/queries/rollup-splice.ts`, and +**The two tiers must tile the window exactly once, with no gap and no overlap.** That boundary +is not written per query. It comes from +`packages/query-engine/src/ch/queries/rollup-splice.ts`, and `unsplicedTwoTierQueries` in the SQL catalog fails any two-tier query that computes its own. See [warehouse-rollups.md](warehouse-rollups.md) for the rule and the bug that motivated it. -`service-map-parity.clickhouse.e2e.test.ts` is the numeric proof: it seeds spans exactly on +`packages/backend/src/services/warehouse/service-map-parity.clickhouse.e2e.test.ts` is the +numeric proof: it seeds spans exactly on each seam and one second either side, then compares each spliced query against a flat scan of -raw `traces`. Boundary-exact rows are the only ones that expose a tiling bug — everything else -passes whether the inequalities are complements or not. +raw `traces`. Boundary-exact rows are the only ones that expose a tiling bug. Everything else passes +whether the inequalities are complements or not. ### Service→service is the fragile one It is the only layer whose interior tier is filled by a **scheduled job** -(`ServiceMapRollupService`, hourly at `0 * * * *`) rather than a materialized view — the -downstream service name is only recoverable by joining a Client/Producer span to its child +(`ServiceMapRollupService`, hourly at `0 * * * *` in `apps/alerting`) instead of a +materialized view. The downstream service name is only recoverable by joining a Client/Producer span to its child Server/Consumer span, which an MV cannot express. Two consequences, both open: @@ -67,9 +69,9 @@ Two consequences, both open: - `LOOKBACK_HOURS = 6`. A worker outage longer than that is a **permanent** hole: the source spans live 30 days, but nothing ever goes back for them. -The fix for both is a per-org rollup watermark — splice at the watermark rather than at -`toStartOfHour`, falling back to the live join for any unsealed hour, so a late tick degrades -into slower-but-correct instead of a gap. +The fix for both is a per-org rollup watermark: splice at the watermark instead of at +`toStartOfHour`, and fall back to the live join for any unsealed hour. A late tick then +degrades into slower-but-correct instead of a gap. ### Presence queries are deliberately not spliced @@ -86,13 +88,13 @@ they name no raw table. Today a new overlay touches roughly nine files: the CH query, `ch/index.ts`, a registry entry, a domain response field, a branch in the `serviceMapBundle` handler, a web transform, an atom, a prop threaded through `ServiceMapView` → `ServiceMapCanvas` → `buildFlowElements`, and a -merge loop inside that function — whose input interface is now twelve optional fields. +merge loop inside that function. Its input interface is now twelve optional fields. Each layer independently re-decides four things, and each is a chance to drift: | Concern | Owned by | Should be | | ----------------- | -------------------------------------------- | ----------------------------------- | -| window boundary | `rollup-splice` ✅ | done — enforced by the catalog gate | +| window boundary | `rollup-splice` ✅ | done, enforced by the catalog gate | | fan-out + caching | a hand-written `Effect.all` per handler | one runner over declared layers | | wire shape | `Schema.Record(String, Unknown)` passthrough | typed rows | | client merge | a new prop + a new loop | one contribution point | @@ -116,9 +118,9 @@ The direction, not yet built: ## Statistics: one name, one meaning A database node and the panel that opens when you click it read the **same edges**. They must -therefore render the same statistic, and for a long time they did not — a ScyllaDB node showed -`3s` and `3k/s` beside a panel showing `7ms` and `30k/s` off the same data. Two separate -causes, both now fixed, both worth not reintroducing: +therefore render the same statistic. For a long time they did not: a ScyllaDB node showed +`3s` and `3k/s` beside a panel showing `7ms` and `30k/s` off the same data. There were two +separate causes. Both are fixed; do not reintroduce them. - **Counts are sample-weighted everywhere.** The node divided the raw `callCount` and hardcoded `hasSampling: false`, while the panel used `estimatedQueryCount`. At a sample rate @@ -127,15 +129,15 @@ causes, both now fixed, both worth not reintroducing: them from the overview, and render the `~` estimate prefix. - **`maxDurationMs` is a max, and is named that.** The edge rollups store `MaxDurationMs` and carry no quantile state, so there is no p95 at edge grain. It was called `p95DurationMs` - from the SQL alias all the way to a "P95 Latency" label — 3s (the slowest call in 12 hours) + from the SQL alias all the way to a "P95 Latency" label: 3s (the slowest call in 12 hours) against the panel's real merged-tDigest 7ms p95. The field is renamed end to end, and `ServiceNodeData` keeps `p95LatencyMs` (service nodes, a real p95) and `maxLatencyMs` (database nodes, a max) as separate optional fields so they cannot be confused again. The related rule: **a loading fallback may substitute a different source, never a different statistic.** The panel's P50 and P95 tiles used to fall back to the edges' mean and max until -the summary resolved, so the number changed meaning — and magnitude — a second after opening. -They now render an em dash while waiting. Counts still fall back, because the edge estimate +the summary resolved, so the number changed meaning (and magnitude) a second after opening. +They now render a dash placeholder while waiting. Counts still fall back, because the edge estimate and the summary estimate are the same statistic. Storing a tDigest state in the edge rollups would give nodes a real p95 and is the remaining @@ -144,7 +146,7 @@ half of this; renaming was the cheap half. ## 2D and 3D presentation `/service-map` stores its renderer choice in the `view=2d|3d` search parameter; 2D remains -the default. The lazy-loaded voxel factory renderer in `components/service-map/three` consumes +the default. The lazy-loaded voxel factory renderer in `apps/web/src/components/service-map/three` consumes the same resolved `buildFlowElements` graph after decluttering, including integration nodes and links. It adds no warehouse queries. Environment, time range, focus, traffic thresholds, namespace expansion, and service/database detail panels remain shared. diff --git a/docs/service-map-infrastructure.md b/docs/service-map-infrastructure.md index 49d4810805..e9b62a34ad 100644 --- a/docs/service-map-infrastructure.md +++ b/docs/service-map-infrastructure.md @@ -27,17 +27,17 @@ To opt in many namespaces declaratively at install time, list them in `autoInstr | Stack | Works automatically? | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| Maple's `@maple-dev/effect-sdk/server` (>= 0.2.0) | Yes — reads `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. Calling `Maple.layer()` with no args picks up the operator-injected config. | -| Java with `opentelemetry-javaagent.jar` | Yes — auto-config reads every `OTEL_*` env var. | +| Maple's `@maple-dev/effect-sdk/server` (>= 0.2.0) | Yes. Reads `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. Calling `Maple.layer()` with no args picks up the operator-injected config. | +| Java with `opentelemetry-javaagent.jar` | Yes. Auto-config reads every `OTEL_*` env var. | | Python with `opentelemetry-distro` / `opentelemetry-instrument` | Yes. | | Node.js with `@opentelemetry/auto-instrumentations-node` | Yes. | -| .NET with `AddOtlpExporter()` and no explicit endpoint | Yes — falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`. | +| .NET with `AddOtlpExporter()` and no explicit endpoint | Yes. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`. | | Go with `otlptracehttp.New(ctx)` (no `WithEndpoint`) | Yes. | -| Go with `otlptracehttp.New(ctx, WithEndpoint("..."))` | No — hardcoded endpoint wins. | +| Go with `otlptracehttp.New(ctx, WithEndpoint("..."))` | No. The hardcoded endpoint wins. | | Rust / other | Depends on whether the exporter is constructed from env vars. | -| Apps with no OTel SDK linked at all | No — nothing produces spans for env vars to configure. | +| Apps with no OTel SDK linked at all | No. Nothing produces spans for env vars to configure. | -If you're on a stack from the first half of the table, you can stop reading here and do the quickstart. If you're hardcoding endpoints in app code, switch to the env-var form (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_RESOURCE_ATTRIBUTES`) before the injection helps you. If you have no SDK, you need to add one before any of this matters — `inject-sdk` mode (the one we use) deliberately does not bundle a language agent. +If your stack is in the first half of the table, run the quickstart and stop here. If you're hardcoding endpoints in app code, switch to the env-var form (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_RESOURCE_ATTRIBUTES`) before the injection helps you. If you have no SDK, add one first. `inject-sdk` mode (the one we use) deliberately does not bundle a language agent. To verify the env vars actually reach your container, exec into a freshly-restarted pod: @@ -45,13 +45,13 @@ To verify the env vars actually reach your container, exec into a freshly-restar kubectl exec -n shop -- env | grep OTEL ``` -Both `OTEL_EXPORTER_OTLP_ENDPOINT=http://maple-k8s-infra-agent.maple.svc.cluster.local:4318` and `OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=...,k8s.pod.uid=...` should be present. If they are but your spans still don't show up, the SDK isn't reading them — fix on the app side. +Both `OTEL_EXPORTER_OTLP_ENDPOINT=http://maple-k8s-infra-agent.maple.svc.cluster.local:4318` and `OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=...,k8s.pod.uid=...` should be present. If they are present but your spans still don't show up, the SDK isn't reading them. Fix it on the app side. ## How it works 1. **Operator webhook injects env vars at pod creation.** The `Instrumentation/maple-default` CR (`deploy/k8s-infra/templates/instrumentation.yaml`) lists `OTEL_EXPORTER_OTLP_ENDPOINT` plus five downward-API env vars (`K8S_POD_IP`, `K8S_POD_UID`, `K8S_POD_NAME`, `K8S_NAMESPACE_NAME`, `K8S_NODE_NAME`) layered into `OTEL_RESOURCE_ATTRIBUTES` via `$(VAR)` interpolation. 2. **App's OTel SDK reads them on startup** and tags every span with `k8s.pod.ip` / `k8s.pod.uid` / `k8s.pod.name` / `k8s.namespace.name` / `k8s.node.name`. The chart's `global.clusterName` is also baked in as `k8s.cluster.name`. -3. **Agent's `k8sattributes` processor enriches the span** by looking up the pod (via `k8s.pod.ip`, `k8s.pod.uid`, or `(k8s.pod.name, k8s.namespace.name)` — see `processors.kubernetesAttributes.podAssociation` in `values.yaml`) in its in-memory cache built from a Kubernetes API watch. It adds `k8s.deployment.name` / `k8s.statefulset.name` / `k8s.daemonset.name`. +3. **Agent's `k8sattributes` processor enriches the span** by looking up the pod (via `k8s.pod.ip`, `k8s.pod.uid`, or `(k8s.pod.name, k8s.namespace.name)`; see `processors.kubernetesAttributes.podAssociation` in `deploy/k8s-infra/values.yaml`) in its in-memory cache built from a Kubernetes API watch. It adds `k8s.deployment.name` / `k8s.statefulset.name` / `k8s.daemonset.name`. 4. **Web app joins on the result.** The `serviceWorkloads` query joins `service.name` (on spans) to workload identity (also on spans), then enriches with pod count and CPU / memory utilization from `metrics_gauge`. The end-to-end join key is `service.name` → `k8s.deployment.name`. If step 3 fails to enrich, the join fails silently and the Infrastructure tab stays empty. @@ -64,7 +64,7 @@ The end-to-end join key is `service.name` → `k8s.deployment.name`. If step 3 f | EKS Fargate | Agent DaemonSet can't run on Fargate-typed nodes. Either keep one EC2 node so the agent has somewhere to land, or override `autoInstrumentation.instrumentation.otlpEndpointOverride` to point at the cluster collector. Set `presets.fargateMetrics.enabled=true` so per-pod CPU/memory still feed the join. | | GKE Standard | None. | | GKE Autopilot | Mutating webhooks are rejected on Google-managed namespaces. Only annotate user namespaces. | -| AKS Azure CNI Overlay | None — pod IPs survive overlay. | +| AKS Azure CNI Overlay | None. Pod IPs survive overlay. | | Self-managed (k3s, kind, k0s) | None. Auto-generated webhook certs work without cert-manager. | | Service-mesh (Linkerd, Istio) | Sidecars rewrite source IPs, breaking the `k8s.pod.ip` association key. The `k8s.pod.uid` and `(k8s.pod.name, k8s.namespace.name)` keys travel inside the OTLP payload itself and rescue the join. No mesh-side config needed. | @@ -85,7 +85,7 @@ GROUP BY 1, 2, 3 ORDER BY spans DESC ``` -`deployment` should be populated (not blank, not `NULL`) for at least one row. If it's blank for every row, the `k8sattributes` processor isn't enriching — see Troubleshooting. +`deployment` should be populated (not blank, not `NULL`) for at least one row. If it's blank for every row, the `k8sattributes` processor isn't enriching. See Troubleshooting. In the UI: open the service map, find one of those services, confirm the pod-count badge on its node, click in, and confirm the Infrastructure tab shows a row with the correct kind, name, namespace, cluster, and pod count. @@ -107,7 +107,7 @@ kubectl logs -n maple -l app.kubernetes.io/name=opentelemetry-operator --tail=20 kubectl get namespace -o jsonpath='{.metadata.annotations}' | jq . ``` -The annotation must be exactly `instrumentation.opentelemetry.io/inject-sdk: "maple/maple-default"`. Anything else (just `maple-default`, just `true`, etc.) is silently ignored. If the operator pod is crash-looping, it's usually a webhook cert issue — switch to cert-manager via `opentelemetryOperator.admissionWebhooks.certManager.enabled=true` if your cluster requires it. +The annotation must be exactly `instrumentation.opentelemetry.io/inject-sdk: "maple/maple-default"`. Anything else (just `maple-default`, just `true`, etc.) is silently ignored. If the operator pod is crash-looping, it's usually a webhook cert issue. Switch to cert-manager via `opentelemetryOperator.admissionWebhooks.certManager.enabled=true` if your cluster requires it. **Service mesh in the way.** Sidecars rewrite source IPs but the `k8s.pod.uid` / `(name, namespace)` association keys ride inside the OTLP payload and rescue the join. Verify the env vars are present: @@ -116,7 +116,7 @@ Sidecars rewrite source IPs but the `k8s.pod.uid` / `(name, namespace)` associat kubectl describe pod -n | grep -E 'K8S_POD_(IP|UID|NAME)|K8S_NAMESPACE_NAME' ``` -All four should be set. If only some are, the operator's webhook didn't run on this pod — see the previous failure mode. +All four should be set. If only some are, the operator's webhook didn't run on this pod. See the previous failure mode. ## Opting out diff --git a/docs/signal-to-event-projection.md b/docs/signal-to-event-projection.md index 7c86f8ac2c..8453d09870 100644 --- a/docs/signal-to-event-projection.md +++ b/docs/signal-to-event-projection.md @@ -5,15 +5,24 @@ Related work: [issue #222](https://github.com/MapleTechLabs/maple/issues/222), Audience: Maple maintainers and implementers of hosted or Maple Local runtimes +Status (2026-09-25): slices 1, 2, 3 and 5 shipped. `packages/eventing-core` +landed in #363, the Local runtime lives in `apps/cli/src/server/eventing`, +PlanetScale uses +`packages/backend/src/services/integrations/planetscale/webhook-events.ts`, and +alert lifecycle events come from `packages/alerting-core`. The slice 4 example +exists only as test fixtures; Maple Local installs no projector by default. +Slice 6 (replay) is not implemented. "Settled implementation choices" describes +the shipped behavior. + Implementers adding a source adapter or semantic projector should also read -[`eventing-extension-guide.md`](./eventing-extension-guide.md), which provides a +[`eventing-extension-guide.md`](./eventing-extension-guide.md). It has a complete compile-time extension example, host wiring patterns, and review and test checklists. ## Decision summary -Maple will treat immediate, per-occurrence event generation as an ingest concern, -not as a scheduled warehouse-query concern. +Maple treats immediate, per-occurrence event generation as an ingest concern. +Scheduled warehouse queries do not generate these events. - Each accepted OTLP record or provider webhook is decoded and normalized into a typed signal once. @@ -32,15 +41,15 @@ not as a scheduled warehouse-query concern. projections. The configurable matching model is a small, structured, typed predicate tree. It -is not arbitrary SQL and it is not a new textual expression language. The live -runtime evaluates the tree in memory. A warehouse adapter may lower the supported -subset to parameterized ClickHouse expressions for explicit historical replay, -but SQL behavior does not define the predicate semantics. +is neither SQL nor a textual expression language. The live runtime evaluates the +tree in memory. A warehouse adapter may lower the supported subset to +parameterized ClickHouse expressions for explicit historical replay. SQL +behavior does not define the predicate semantics. ## Problem -Maple currently contains several mechanisms that are related but not expressed -through one event boundary: +Before this design, Maple had several related mechanisms with no shared event +boundary: - hosted alert rules periodically query telemetry, update incident lifecycle state, and request deliveries; @@ -49,15 +58,14 @@ through one event boundary: - future automation needs individual facts, such as a source record being observed, to become events that agents or other consumers can act on. -Using the alert scheduler for the last case would give it the wrong semantics. -A windowed query answers a question about a set of stored records and normally -produces one aggregate observation. It cannot faithfully represent every -individual occurrence without cursors, overlap windows, late-arrival handling, -and deduplication. +The alert scheduler has the wrong semantics for the last case. A windowed query +answers a question about a set of stored records and normally produces one +aggregate observation. It cannot represent every individual occurrence without +cursors, overlap windows, late-arrival handling, and deduplication. -The current Local `logs` table has no ingestion sequence or native event ID. Its -sort key is designed for observability queries, and arbitrary OTLP attributes are -stored as strings. Repeatedly querying that table once per rule would therefore: +The Local `logs` table has no ingestion sequence or native event ID. Its sort +key is designed for observability queries, and arbitrary OTLP attributes are +stored as strings. Querying that table once per rule would: - compete with ingest, UI queries, checkpoints, retention, and archive work; - miss late records or repeatedly rediscover records unless a second deduplication @@ -65,8 +73,8 @@ stored as strings. Repeatedly querying that table once per rule would therefore: - require casts that cannot always recover the source value's original type; - turn an embedded analytical database into an inefficient message queue. -The event layer is still useful. It belongs in front of chDB for live signals, -with chDB retained behind it for analytics and aggregate alert evaluation. +The event layer belongs in front of chDB for live signals. chDB stays behind it +for analytics and aggregate alert evaluation. ## Goals @@ -350,24 +358,26 @@ selector to: - no regular expressions, functions, arithmetic, joins, or user code. These bounds keep evaluation predictable and leave room for indexing active -projections by source kind and simple discriminating fields. `SignalScalar` -describes normalized source data and does not inherit the literal-only 1,024-code-point -limit; the OTLP adapter accepts source strings up to its separate 16 KiB bound. -The literal limit is normative in Unicode code points: both the shared decoder -and the generated JSON Schema enforce a maximum of 1,024. Since a Unicode code -point requires at most four UTF-8 bytes, this also bounds literals to 4 KiB. -ASCII literals are therefore limited to 1,024 characters, not 4,096. The shared -multibyte conformance vectors verify the same boundary. - -The generated predicate schema describes two additional mandatory checks: -consumers must enforce the depth and total-node budgets before recursive -validation. These whole-tree limits are not expressed by its JSON Schema -validation keywords, so a successful JSON Schema validation alone is insufficient -for selector acceptance. The shared decoder performs an iterative preflight. -Count each predicate occurrence, including repeated subtrees; JavaScript callers -may reuse a subtree object, and it counts the same as the duplicated JSON value. -Actual cycles fail the bounded preflight. Field declarations and operator/type -compatibility also require the semantic validation described above. +projections by source kind and simple discriminating fields. The 1,024-code-point +limit applies to selector literals only. `SignalScalar` describes normalized +source data, and the OTLP adapter accepts source strings up to its separate +16 KiB bound. + +The literal limit is normative in Unicode code points. The shared decoder and +the generated JSON Schema both enforce a maximum of 1,024. A code point needs at +most four UTF-8 bytes, so literals are also bounded to 4 KiB. ASCII literals are +limited to 1,024 characters, not 4,096. The shared multibyte conformance vectors +check the same boundary. + +The generated predicate schema describes two more mandatory checks: consumers +must enforce the depth and total-node budgets before recursive validation. JSON +Schema validation keywords cannot express these whole-tree limits, so passing +JSON Schema validation alone does not make a selector acceptable. The shared +decoder runs an iterative preflight. It counts each predicate occurrence, +including repeated subtrees. A JavaScript caller may reuse a subtree object; it +counts the same as the duplicated JSON value. Actual cycles fail the bounded +preflight. Field declarations and operator/type compatibility also need the +semantic validation described above. ### Signal projection @@ -393,14 +403,14 @@ retroactive: the new revision sees signals accepted after the runtime atomically installs its compiled registry snapshot. Historical processing requires an explicit replay operation. -Replaying the exact latest revision is a no-op only while its enabled/disabled -state still matches the active pointer. Replaying an older revision is a stale -revision conflict; an intentional rollback is a new monotonic revision that -copies the earlier configuration. +Resubmitting the exact latest revision is a no-op only while its +enabled/disabled state still matches the active pointer. Resubmitting an older +revision is a stale revision conflict. An intentional rollback is a new +monotonic revision that copies the earlier configuration. The configuration record is data. Source adapters and projector implementations -are registered code. This is how matching remains configurable without making -authentication, provider semantics, or executable code user-supplied. +are registered code. Matching stays configurable while authentication, provider +semantics, and executable code stay out of user hands. For example, an installed source adapter and projector can use this neutral contract: @@ -479,12 +489,17 @@ Produced events use CloudEvents 1.0 structured representation: "tenantid": "...", "projectionid": "...", "projectionrevision": 3, + "projectorid": "example.record", + "projectorversion": 1, "data": {} } ``` -Names above are illustrative until the repository reserves its canonical event -type and schema namespace. +Type and schema names above are the neutral fixture names (see "Settled +implementation choices"). `MapleCloudEventSchema` in +`packages/eventing-core/src/model.ts` is the canonical envelope. It requires +`projectorid` and `projectorversion` and allows optional `sourceoccurrenceid` +and `identityquality` extensions. The event ID is deterministic when stable occurrence identity exists: @@ -497,9 +512,10 @@ occurrence ID, projection ID, and the decimal projection revision. Encode each field as UTF-8, prefix it with its byte length as an unsigned four-byte big-endian integer, concatenate, and hash. The `sha256:` prefix is outside the hash. -The hash input uses a canonical length-delimited encoding, not string -concatenation. Projector version and output schema version are already fixed by -the immutable projection revision and must be recorded with the event. +The length prefixes make the encoding unambiguous, unlike plain string +concatenation. The immutable projection revision already fixes the projector +version and output schema version; the envelope records both. The reference +implementation is `makeEventId` in `packages/eventing-core/src/event.ts`. Sensitive source details belong in `data`, under the projector's explicit schema and redaction policy. They must not be copied into CloudEvents context attributes, @@ -526,10 +542,8 @@ zero, one, or several different factual events. ### Maple Local OTLP ingest -Maple Local already decodes an OTLP request and then passes the decoded payload -to the warehouse encoder. The event seam belongs between those operations. - -The implementation should refactor decoding/normalization so that: +Maple Local decodes an OTLP request and then passes the decoded payload to the +warehouse encoder. The event seam sits between those operations: 1. the OTLP request is parsed once; 2. typed record values remain available to the matcher; @@ -542,87 +556,95 @@ The implementation should refactor decoding/normalization so that: When no projection matches, the path adds only bounded predicate work before the existing chDB insert. -When the outbox reaches its event or byte cap, new projections are dropped while -warehouse ingestion continues. Maple preserves existing staged and ready events, -reports the dropped projection count, and records a durable delivery gap that an -operator must accept before consumers resume claiming. Infrastructure failures -while persisting eventing state remain retryable ingest failures. A retry of a -staged occurrence recovers its original event IDs and canonical bytes. Durable -OTLP log projection requires `timeUnixNano` or `observedTimeUnixNano`; server -receipt time is never incorporated into durable identity or event content. -OTLP permits both timestamp fields to be absent or zero; those records remain -accepted by the warehouse path but are skipped by durable event projection. -The same isolation applies to eventing-specific normalization bounds: an -oversized attribute map or nested value makes only that source occurrence -ineligible for projection and records a bounded normalization failure. The -existing warehouse encoder still decides independently whether the OTLP record -is valid for storage, so enabling a projection does not narrow ingest. Before -full event normalization, Maple performs a tolerant, bounded extraction of the -stable source URI and source-issued occurrence ID. An ineligible occurrence -that matches an existing staged obligation fails ingest rather than silently -acknowledging a changed retry and stranding the earlier event. - -Staging and chDB insertion are not one transaction. A process crash after the -chDB insert but before the OTLP acknowledgement can still cause a duplicate raw -telemetry row on retry; that is already possible with at-least-once OTLP -delivery. The staged/ready outbox protocol prevents an event from becoming -dispatchable before the ingest attempt reaches its warehouse commit point. -Staged rows retain the source occurrence identity and original projection -revision plus a bounded hash of the normalized source content. On redelivery, -Maple recovers those exact event IDs and does not reevaluate that occurrence +When the outbox reaches its event or byte cap, new projections are dropped and +warehouse ingestion continues. Maple keeps existing staged and ready events, +reports the dropped projection count, and records a durable delivery gap. An +operator must accept the gap before consumers resume claiming. Infrastructure +failures while persisting eventing state remain retryable ingest failures. A +retry of a staged occurrence recovers its original event IDs and canonical +bytes. + +Durable OTLP log projection requires `timeUnixNano` or `observedTimeUnixNano`. +Server receipt time never enters durable identity or event content. OTLP +permits both timestamp fields to be absent or zero. The warehouse path still +accepts those records; durable event projection skips them. + +Eventing-specific normalization bounds are isolated the same way. An oversized +attribute map or nested value makes only that source occurrence ineligible for +projection and records a bounded normalization failure. The warehouse encoder +still decides independently whether the OTLP record is valid for storage, so +enabling a projection does not narrow ingest. Before full event normalization, +Maple does a tolerant, bounded extraction of the stable source URI and +source-issued occurrence ID. If an ineligible occurrence matches an existing +staged obligation, ingest fails. Acknowledging it would accept a changed retry +and strand the earlier event. + +Staging and chDB insertion are not one transaction. A crash after the chDB +insert and before the OTLP acknowledgement can cause a duplicate raw telemetry +row on retry. At-least-once OTLP delivery already allows that. The staged/ready +outbox protocol keeps an event from becoming dispatchable before the ingest +attempt reaches its warehouse commit point. + +Staged rows keep the source occurrence identity, the original projection +revision, and a bounded hash of the normalized source content. On redelivery, +Maple recovers those exact event IDs. It does not reevaluate the occurrence against a newer or disabled projection snapshot. Recovery requires the source -hash to match; reuse of the same source identity with changed content fails as a -collision and leaves the staged event non-ready. Within one ingest batch, two -records that reuse the same tenant, source kind, source URI, and source-issued -occurrence ID must also have the same normalized source hash. Maple checks that -source tuple for every normalized occurrence before selectors divide it into -zero, one, or several projected event IDs. A projection-ineligible record that -reuses a normalized tuple makes the batch ambiguous and is rejected before any -event is staged. The fingerprint contract orders field keys by explicit -JavaScript code-unit order, not locale collation, so checkpoint recovery is -independent of host locale. - -The initial control schema is version 1 and includes source fingerprints. +hash to match. Reusing a source identity with changed content fails as a +collision and leaves the staged event non-ready. + +Within one ingest batch, two records with the same tenant, source kind, source +URI, and source-issued occurrence ID must also have the same normalized source +hash. Maple checks that tuple for every normalized occurrence before selectors +split it into zero, one, or several projected event IDs. A +projection-ineligible record that reuses a normalized tuple makes the batch +ambiguous; Maple rejects it before staging any event. The fingerprint orders +field keys by JavaScript code-unit order, not locale collation, so checkpoint +recovery does not depend on host locale. + +The control schema (`LOCAL_CONTROL_SCHEMA_VERSION = 1` in +`apps/cli/src/server/local-schema-version.ts`) includes source fingerprints. Opening a store or validating a snapshot rejects staged source-backed rows with missing or malformed fingerprints. Restore verifies the control snapshot against its checkpoint manifest, validates it, and copies it through a private scratch store before installing the restored data directory. Invalid snapshots fail before restore readiness or the live-directory swap. The checkpoint artifact -remains unchanged. +stays unchanged. -If atomic exactly-once storage across both systems later becomes a requirement, -the correct addition is a durable ingress journal before both writes. chDB -polling does not solve that problem. +If atomic exactly-once storage across both systems becomes a requirement, add a +durable ingress journal before both writes. chDB polling does not solve that +problem. ### Provider webhooks Provider authentication runs before normalization. The hosted PlanetScale route -acknowledges test, ignore, and log dispositions inline. Events requiring issue or -timeline persistence are projected and queued before acknowledgement. Their -canonical CloudEvent and routing metadata form the durable queue body. When the -provider omits its timestamp, projection uses the request's `receivedAt` value. -The complete serialized job is measured against a 120 KiB cap before send; -oversized queue bodies receive a deterministic `413` rather than a retryable -queue failure. During rolling upgrades, consumers accept the new event-only body -and the payload-only body already produced by upstream. Legacy payload-only jobs -are projected using their stored `receivedAt` when their timestamp is absent. - -Current queue jobs are decoded as one relational contract: the event tenant, -source, embedded connection, type, schema, and timestamp must agree with the -bounded routing fields. Unsupported or contradictory jobs are terminally -acknowledged as poison messages. Health-event issue mutations use a durable -`(org_id, event_id)` receipt inserted in the same PostgreSQL transaction as the -issue mutation; timeline insertion remains independently idempotent. A retry -after a failure between those phases therefore completes the issue once without -duplicating its occurrence count or history. Transactions also take a scoped -lock for `(org_id, issue fingerprint)` before claiming the receipt. Concurrent, -distinct events for the same issue are therefore all counted, while only one -transition reopens a resolved issue. +(`apps/api/src/routes/v1/planetscale-webhook.http.ts`) acknowledges test, +ignore, and log dispositions inline. Events that need issue or timeline +persistence are projected and queued before acknowledgement. The canonical +CloudEvent and routing metadata form the durable queue body. When the provider +omits its timestamp, projection uses the request's `receivedAt` value. The +complete serialized job is checked against a 120 KiB cap +(`MAX_PLANETSCALE_WEBHOOK_QUEUE_BYTES`) before send. An oversized body gets a +deterministic `413`, not a retryable queue failure. During rolling upgrades, +consumers accept both the new event-only body and the legacy payload-only body +(`LegacyPlanetScaleWebhookJob`). Legacy jobs without a provider timestamp are +projected with their stored `receivedAt`. + +Current queue jobs decode as one relational contract: the event tenant, source, +embedded connection, type, schema, and timestamp must agree with the bounded +routing fields. Unsupported or contradictory jobs are acknowledged as poison +messages and not retried. Health-event issue mutations insert a durable +`(org_id, event_id)` receipt in the same PostgreSQL transaction as the issue +mutation. Timeline insertion is independently idempotent. A retry after a +failure between those phases completes the issue once, without duplicating its +occurrence count or history. Each transaction takes +`pg_advisory_xact_lock` on `(org_id, issue fingerprint)` before claiming the +receipt. Concurrent distinct events for the same issue are all counted, and only +one transition reopens a resolved issue. The provider source adapter supplies the strongest available delivery or event identity. It then uses the same selector, projector, event ID, and outbox -contracts as OTLP. Provider-specific response behavior does not live in the core; -it can be migrated behind consumers of the emitted event types. +contracts as OTLP. Provider-specific response behavior stays out of the core +and can move behind consumers of the emitted event types. ### Query-driven alerts @@ -636,7 +658,7 @@ Scheduled alert rules retain their existing execution model: 5. the host persists it through the common event outbox. This path queries chDB or the hosted warehouse because its input is an aggregate -over time. It does not reuse the ingest-time signal selector, and the ingest-time +over time. It does not reuse the ingest-time signal selector. The ingest-time path does not impersonate an alert incident. ### Historical replay @@ -659,15 +681,14 @@ compiler emits parameterized expressions through existing query-building facilities; it never interpolates field names or literals supplied directly by a user. -Current Local OTLP attribute maps store strings, so arbitrary typed attributes -will generally be `coerced`, not `exact`. Replay event IDs are guaranteed to +Local OTLP attribute maps store strings, so arbitrary typed attributes are +generally `coerced`, not `exact`. Replay event IDs are guaranteed to deduplicate against live events only when the warehouse retained the same stable source occurrence ID. ## Processing and delivery guarantees -The architecture uses precise, layered guarantees rather than the blanket phrase -"exactly once". +Each boundary has its own guarantee. None of them is "exactly once". | Boundary | Guarantee | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------- | @@ -675,7 +696,7 @@ The architecture uses precise, layered guarantees rather than the blanket phrase | One accepted batch | One evaluation against one immutable projection-registry snapshot. | | Projection with source-stable identity | Effectively-once event creation through deterministic ID plus unique outbox insertion. | | Projection with derived identity | Best-effort deduplication; identical real occurrences may collapse and re-encoded retries may diverge. | -| Projection with no identity | At-least-once event creation only; durable automation should reject this configuration by default. | +| Projection with no identity | No event. `makeCloudEvent` rejects `identityQuality: "none"`, so the host records a projection failure. | | Outbox to consumer | At least once with an event ID/idempotency key; consumer-side external effects are outside this specification. | | chDB telemetry row | Existing OTLP semantics; duplicate storage remains possible after ambiguous failures. | @@ -755,14 +776,15 @@ The physical Local store is an implementation decision, but it must provide: ## Package and host ownership -The intended ownership is: +Ownership: -- `packages/eventing-core` (new): language-neutral schemas, selector validation, - the reference TypeScript evaluator, projector registry contracts, canonical - event identity, and conformance fixtures. No database, network, scheduler, or - global clock dependencies. -- `packages/alerting-core` (new): aggregate alert evaluation and incident - lifecycle. It remains distinct and later emits through an eventing-core port. +- `packages/eventing-core` (`@maple/eventing-core`): language-neutral schemas, + selector validation, the reference TypeScript evaluator, projector registry + contracts, canonical event identity, and conformance fixtures. No database, + network, scheduler, or global clock dependencies. +- `packages/alerting-core` (`@maple/alerting-core`): aggregate alert evaluation + and incident lifecycle. It stays distinct and builds lifecycle CloudEvents + with `makeCloudEvent` from eventing-core. - `packages/domain`: public/API schemas when projection CRUD becomes public. - `apps/cli`: Maple Local OTLP source adapter, compiled-registry lifecycle, durable Local ports, ingest staging, and optional replay adapter. @@ -771,8 +793,9 @@ The intended ownership is: - `apps/ingest`: a future Rust OTLP adapter only when hosted per-signal projection is required. -The canonical JSON schemas and fixture corpus, rather than TypeScript source -types, define cross-language behavior. A Rust implementation must pass the same +The canonical JSON schemas (`packages/eventing-core/schemas`) and fixture corpus +(`packages/eventing-core/fixtures`) define cross-language behavior. TypeScript +source types do not. A Rust implementation must pass the same valid/invalid selector cases, typed comparison cases, canonical event-ID vectors, and projection fixtures before it can claim compatibility. @@ -839,8 +862,8 @@ when those optional host operations run. ## Compatibility and migration -This design extends rather than replaces the host-neutral alert-core extraction -already on the issue-222 branch. +This design extends the host-neutral alert-core extraction from the issue-222 +branch. It does not replace it. 1. Existing hosted aggregate alerts continue using their scheduler, query, lifecycle, and delivery behavior while the event contract is introduced. @@ -865,7 +888,7 @@ sort key. ## Implementation slices for the next goal -### Slice 1 — Contract and evaluator +### Slice 1: Contract and evaluator - Add `packages/eventing-core`. - Define runtime schemas for typed values, fields, predicates, projection specs, @@ -874,21 +897,21 @@ sort key. - Add canonical JSON and event-ID test vectors. - Add complexity-limit and hostile-input tests. -### Slice 2 — Local durable control state +### Slice 2: Local durable control state - Select and document the Local transactional store. - Implement projection revision and outbox ports, migrations, recovery, and backup/restore hooks. - Expose headless health inspection before UI work. -### Slice 3 — Local ingest seam +### Slice 3: Local ingest seam - Refactor OTLP normalization to preserve typed values without decoding twice. - Load and atomically swap compiled projection snapshots. - Stage matching events, insert telemetry, mark events ready, and acknowledge. - Prove that the live path executes no chDB `SELECT` and adds no scheduler. -### Slice 4 — Example extension: source record to durable Maple event +### Slice 4: Example extension: source record to durable Maple event - Define a source adapter's OTLP field contract and stable occurrence identity. - Register its field catalog and a pure semantic projector outside the core. @@ -899,14 +922,14 @@ sort key. This slice stops at the outbox. Transport and agent-action behavior are downstream concerns using the produced typed event. -### Slice 5 — Existing producer convergence +### Slice 5: Existing producer convergence - Adapt PlanetScale webhook inputs to the source/projector contracts. - Project alert lifecycle intents into CloudEvents. - Preserve existing provider and alert behavior with parity fixtures before switching consumers. -### Slice 6 — Optional replay +### Slice 6: Optional replay - Add per-field replay capability declarations. - Implement bounded dry-run and explicit emission modes. @@ -937,18 +960,18 @@ The first usable implementation is complete when all of the following are true: ## Settled implementation choices -The TypeScript reference implementation settles the remaining host choices as -follows: +The TypeScript reference implementation settles the remaining host choices: - Maple Local stores projection revisions, failures, and the staged/ready outbox - in SQLite at `/control/eventing.sqlite`, using WAL and `synchronous = -FULL`. While ingest is quiesced, backup first completes and verifies a blocking - `wal_checkpoint(TRUNCATE)` so the serialized database contains every committed - control-store transaction rather than only the main SQLite file. A version-2 - Maple checkpoint contains `control.sqlite` beside the chDB - backup and binds its byte count, SHA-256 digest, schema version, and row counts - in the checkpoint manifest. Version-1 checkpoints remain readable and restore - an empty control store. + in SQLite at `/control/eventing.sqlite`, with WAL and + `synchronous = FULL` (`apps/cli/src/server/eventing/control-store.ts`). While + ingest is quiesced, backup first runs and verifies a blocking + `wal_checkpoint(TRUNCATE)`, so the serialized database holds every committed + control-store transaction and not only the main SQLite file. A version-2 + Maple checkpoint stores `control.sqlite` beside the chDB backup (under + `/backups/snapshots//`). Its manifest binds the file's + byte count, SHA-256 digest, schema version, and row counts. Version-1 + checkpoints remain readable and restore an empty control store. - The reference OTLP extension example uses a LogRecord event name such as `example.record.observed`. An installed adapter defines its own accepted stable occurrence identifiers, field catalog, validation rules, and semantic @@ -959,12 +982,12 @@ FULL`. While ingest is quiesced, backup first completes and verifies a blocking event type and schema names; neutral fixtures use `dev.maple.example.record.observed.v1` with `urn:maple:event-schema:example-record:v1`. -- Attribute strings are limited to 16 KiB, source/event identities to 256 - characters (long stable inputs are represented by a SHA-256 URN), each - attribute namespace to 256 entries, - nested values to depth 8 and 1,024 nodes, normalized source data to 256 KiB, - and a canonical outbox CloudEvent to 256 KiB. Secret-like attribute names are - excluded from the projection field and data views. +- Limits (`apps/cli/src/server/eventing/otlp.ts`, `MAX_CLOUD_EVENT_BYTES` in + eventing-core): attribute strings 16 KiB; source and event identities 256 + characters (longer stable inputs become a SHA-256 URN); 256 entries per + attribute namespace; nested values depth 8 and 1,024 nodes; normalized source + data 256 KiB; canonical outbox CloudEvent 256 KiB. Secret-like attribute names + are excluded from the projection field and data views. - The Local TypeScript path is the reference live implementation. Hosted Rust ingest remains a later adapter and must pass the shared schemas and fixture corpus before claiming parity. @@ -973,64 +996,97 @@ FULL`. While ingest is quiesced, backup first completes and verifies a blocking projector before queueing. Test, ignore, and log dispositions are acknowledged inline. The dedicated Cloudflare Queue carries `dev.maple.planetscale.webhook.received.v1` without duplicating the provider - payload. Consumers support the new event-only body and the existing upstream - payload-only body during rolling upgrades. Missing timestamps use `receivedAt`; - legacy jobs use the value stored in their queue body, so queue retries retain + payload. Consumers support the new event-only body and the legacy + payload-only body during rolling upgrades. Missing timestamps use `receivedAt`. + Legacy jobs use the value stored in their queue body, so queue retries keep the same projected identity. - Hosted query-alert delivery rows remain that producer's durable outbox. Their - payload now includes an additive deterministic - `dev.maple.alert.lifecycle.{trigger,resolve,renotify,test}.v1` CloudEvent while - retaining every legacy top-level delivery field. Retry creation preserves the - originally stored JSON, including the CloudEvent ID and future additive fields, - instead of round-tripping it through a lossy legacy schema. -- Historical replay execution remains deliberately unimplemented in this - change. Field catalogs already declare `exact`, `coerced`, or `unavailable`, - but Local's current arbitrary attribute maps have lost source scalar type and - its warehouse rows do not furnish a native occurrence ID. A later bounded, - operator-invoked replay adapter must require explicit coercion acknowledgement - and pass live-evaluator conformance tests; the live path never falls back to a - chDB poller in the meantime. + payload includes an additive deterministic + `dev.maple.alert.lifecycle.{trigger,resolve,renotify,test}.v1` CloudEvent and + keeps every legacy top-level delivery field. Retry creation preserves the + originally stored JSON, including the CloudEvent ID and future additive + fields. It does not round-trip the payload through a lossy legacy schema. +- Historical replay is not implemented. Field catalogs already declare `exact`, + `coerced`, or `unavailable`. Local's arbitrary attribute maps have lost the + source scalar type, and its warehouse rows carry no native occurrence ID. A + later bounded, operator-invoked replay adapter must require explicit coercion + acknowledgement and pass live-evaluator conformance tests. Until then the live + path never falls back to a chDB poller. - Projector failures with a source occurrence ID are idempotent per projection - revision. Local retains a bounded newest 10,000 failure rows per tenant and - exposes the count through the authenticated headless health endpoint. A - projector failure does not retry a valid telemetry occurrence forever; - infrastructure failure to persist required state remains retryable. + revision. Local keeps the newest 10,000 failure rows per tenant and exposes + the count through the authenticated headless health endpoint. A projector + failure does not retry a valid telemetry occurrence forever. Failure to + persist required state remains retryable. Maple Local activates immutable revisions with authenticated `POST /local/eventing/projections`. The same maintenance credential protects `GET /local/eventing/projections`, `/local/eventing/health`, -`/local/eventing/outbox`, and consumer administration. Ready records receive a separate, append-only -readiness `sequence` on their first staged-to-ready transition; -`?after=&limit=` therefore cannot skip an older staged event that is -recovered after newer events were already read. `?state=staged` uses the original -staging sequence for bounded inspection of records stranded before the chDB -commit point. The Local store defaults to at most 10,000 events and 256 MiB of -canonical event JSON. When either cap would be exceeded, new projections are dropped while warehouse ingestion continues. A durable delivery gap blocks subsequent consumer claims until an operator accepts its generation. Existing records remain intact; the maintenance-only abandon API can explicitly remove stranded records. Inspection remains non-destructive. Named downstream -consumers use the separate [Maple Local event consumer protocol](./local-event-consumers.md) for -leased, at-least-once claims and exact whole-batch acknowledgement. Ready-event pruning advances only -through the slowest active consumer and retains a bounded acknowledged tail; staged events are never -pruned by delivery acknowledgement. - -Re-delivery is the safe recovery operation: it locates staged rows by stable -source occurrence, preserves their original projection snapshot, and promotes -those exact event IDs only after the warehouse write succeeds. Maple never blindly -promotes an old staged record because, after a crash, the control store alone -cannot prove whether the corresponding chDB write committed. Activation requires -authentication, a bounded request body, structural budget validation, and full -registry compilation before acquiring global quiescence. Only the projection -revision commit and immutable runtime-registry swap occur while ingest is -quiesced, so invalid credentials, incomplete bodies, and expensive validation do -not close admission and every ingest request still observes exactly one registry -version. Concurrent maintenance requests receive an intentional conflict response. +`/local/eventing/outbox`, and consumer administration (routes in +`apps/cli/src/server/serve.ts`). + +A ready record gets a separate, append-only readiness `sequence` on its first +staged-to-ready transition. `?after=&limit=` therefore cannot skip +an older staged event that is recovered after newer events were read. +`?state=staged` uses the original staging sequence for bounded inspection of +records stranded before the chDB commit point. + +The Local store defaults to at most 10,000 events and 256 MiB of canonical event +JSON. When either cap would be exceeded, new projections are dropped and +warehouse ingestion continues. A durable delivery gap blocks consumer claims +until an operator accepts its generation. Existing records stay intact. The +maintenance-only abandon API can explicitly remove stranded records. Inspection +is non-destructive. + +Named downstream consumers use the separate +[Maple Local event consumer protocol](./local-event-consumers.md) for leased, +at-least-once claims and exact whole-batch acknowledgement. Ready-event pruning +advances only through the slowest active consumer and keeps a bounded +acknowledged tail. Delivery acknowledgement never prunes staged events. + +Re-delivery is the safe recovery operation. It locates staged rows by stable +source occurrence, keeps their original projection snapshot, and promotes those +exact event IDs only after the warehouse write succeeds. Maple never blindly +promotes an old staged record: after a crash, the control store alone cannot +prove whether the matching chDB write committed. + +Activation runs authentication, a bounded request body read, structural budget +validation, and full registry compilation before acquiring global quiescence. +Only the projection revision commit and the immutable runtime-registry swap run +while ingest is quiesced. Invalid credentials, incomplete bodies, and expensive +validation therefore never close admission, and every ingest request sees +exactly one registry version. Concurrent maintenance requests get a `409` +conflict. ## Compatibility and delivery notes -The initial Local adapter projects OTLP logs only; traces and metrics continue through ordinary warehouse ingestion. Alert webhook and Hazel payloads gain an additive `event` CloudEvent envelope (including `tenantid`, typically about 1 KB). Alert event identities are deterministic for a scheduled tick; retries reuse the retained payload and identity. - -Checkpoint format v2 includes the control database. Older CLIs cannot list or restore v2 checkpoints and their reset command rejects data directories containing `control/`. Restore a v1 checkpoint with the new CLI to recover warehouse data with an empty control store; projection definitions and consumer positions are not present in v1. - -A checkpoint drains admitted operations, captures immutable SQLite bytes, and runs the synchronous chDB backup before reopening admission. The bytes are written and verified after admission resumes. chDB’s native backup blocks the JavaScript event loop, so this does not promise query availability during the native backup; it avoids holding admission closed during the subsequent asynchronous control archive write. Taking unrelated snapshots with a time gap could restore acknowledged delivery state ahead of the warehouse, so that gap is not accepted. - -Deploy `0059_planetscale_issue_receipts` after main's onboarding and incident-hold migrations and before deploying the consumer. Its receipt insert shares the issue transaction and requires the table to exist. - -PlanetScale issue receipts are retained for 90 days after processing and swept hourly in batches of at most 5,000. Replays after that window may apply issue mutations again. Receipts intentionally survive issue deletion within the window, so redelivery is skipped instead of recreating a deleted issue. +The Local adapter projects OTLP logs only (`otel.log`). Traces and metrics go +through ordinary warehouse ingestion. Alert webhook and Hazel payloads carry an +additive `event` CloudEvent envelope (including `tenantid`, typically about +1 KB). Alert event identities are deterministic for a scheduled tick. Retries +reuse the retained payload and identity. + +Checkpoint format v2 includes the control database. Older CLIs cannot list or +restore v2 checkpoints, and their reset command rejects data directories +containing `control/`. Restoring a v1 checkpoint with the new CLI recovers +warehouse data with an empty control store. v1 checkpoints contain no +projection definitions or consumer positions. + +A checkpoint drains admitted operations, captures immutable SQLite bytes, and +runs the synchronous chDB backup before reopening admission. The SQLite bytes +are written and verified after admission resumes. chDB's native backup blocks +the JavaScript event loop, so queries may stall during it. The design only +avoids holding admission closed during the later asynchronous control +archive write. Separate snapshots taken with a time gap could restore +acknowledged delivery state ahead of the warehouse, so the design rejects that +approach. + +The receipt table comes from migration +`packages/db/drizzle/20260917223420_planetscale_issue_receipts`. The receipt +insert shares the issue transaction and requires the table to exist. + +PlanetScale issue receipts are kept for 90 days after processing. The API +worker's hourly retention cron sweeps them in batches of at most 5,000 +(`packages/backend/src/services/integrations/planetscale-event-retention.ts`). +Replays after that window may apply issue mutations again. Receipts survive +issue deletion within the window, so redelivery is skipped instead of +recreating a deleted issue. diff --git a/docs/sst-fork-workflow.md b/docs/sst-fork-workflow.md index 2ee366e9ce..58a8d7bf56 100644 --- a/docs/sst-fork-workflow.md +++ b/docs/sst-fork-workflow.md @@ -1,8 +1,12 @@ # SST Fork Workflow -Maple runs against a local fork of `anomalyco/sst` so we can add features to the SST platform (the TypeScript component library under `platform/src/components/`) before they land upstream, and open PRs back to upstream from the same branches. +> Status (2026-09-25): obsolete. Maple no longer depends on SST: the root `package.json` has no +> `sst` dependency and there is no `sst.config.ts`. Infrastructure is declared with alchemy +> (`alchemy.run.ts`, see `docs/infra.md`). Kept as a record of the fork workflow. -The SST CLI is a Go binary shipped as a prebuilt in the `sst` npm package. At runtime it extracts the platform source into `.sst/platform/` and runs it via Pulumi. We don't rebuild the Go binary — we just swap out the extracted platform source with a symlink to our fork. That means any edit in the fork is live in maple on the next `bun sst …` invocation, with no rebuild step. +Maple ran against a local fork of `anomalyco/sst`. This let us add features to the SST platform (the TypeScript component library under `platform/src/components/`) before they landed upstream, and open PRs back to upstream from the same branches. + +The SST CLI is a Go binary shipped prebuilt in the `sst` npm package. At runtime it extracts the platform source into `.sst/platform/` and runs it via Pulumi. We don't rebuild the Go binary. We replace the extracted platform source with a symlink to our fork, so any edit in the fork is live in maple on the next `bun sst …` invocation. ## Layout @@ -14,9 +18,9 @@ The SST CLI is a Go binary shipped as a prebuilt in the `sst` npm package. At ru ## Branches -- `dev` — tracks `upstream/dev`. Do not commit local patches here; this is the baseline you branch off for upstream PRs. -- `local-4.7.3` — local testing branch, based on the `v4.7.3` tag with our local-only patches cherry-picked on top. This is the branch you have checked out when running maple so the SST CLI sees the exact baseline it expects, plus our patches. -- `feat/*` — one branch per upstream PR, branched from `dev`. +- `dev`: tracks `upstream/dev`. Do not commit local patches here. This is the baseline you branch off for upstream PRs. +- `local-4.7.3`: local testing branch, based on the `v4.7.3` tag with our local-only patches cherry-picked on top. Keep it checked out when running maple, so the SST CLI sees the exact baseline it expects plus our patches. +- `feat/*`: one branch per upstream PR, branched from `dev`. When `v4.7.3` == `upstream/dev` (as it was at setup time), `dev` and `local-4.7.3` have the same content modulo the local patches. They diverge once upstream cuts a new release. @@ -29,7 +33,7 @@ Assume you're already on `local-4.7.3` in the fork. cd ~/Documents/GitHub/sst $EDITOR platform/src/components/cloudflare/worker.ts -# 2. Test in maple — no rebuild, the symlink reflects the edit immediately +# 2. Test in maple. No rebuild: the symlink reflects the edit immediately cd ~/Documents/GitHub/maple bun sst diff --stage ``` @@ -44,11 +48,11 @@ Do this whenever you want the latest `anomalyco/sst:dev`, typically before start cd ~/Documents/GitHub/sst git fetch upstream git checkout dev -git merge --ff-only upstream/dev # fast-forward only — dev should have no local commits +git merge --ff-only upstream/dev # fast-forward only: dev should have no local commits git push origin dev # keep Makisuo/sst:dev in sync with upstream ``` -If `git merge --ff-only` fails, someone (probably past-you) committed directly to `dev`. Move those commits to a feature branch with `git branch -f feat/whatever dev && git reset --hard upstream/dev`, then retry. +If `git merge --ff-only` fails, someone committed directly to `dev`. Move those commits to a feature branch with `git branch -f feat/whatever dev && git reset --hard upstream/dev`, then retry. ## Rebasing `local-4.7.3` onto a newer release @@ -62,7 +66,7 @@ git rebase v4.7.4 # replays local patches onto the new tag git branch -m local-4.7.3 local-4.7.4 ``` -Then bump maple: `bun update sst` in `~/Documents/GitHub/maple`, and **re-create the symlink** (SST wipes `.sst/platform/` on version mismatch — see Recovery below). +Then bump maple: `bun update sst` in `~/Documents/GitHub/maple`, and **re-create the symlink**. SST wipes `.sst/platform/` on version mismatch (see Recovery below). ## Adding a new feature + opening an upstream PR @@ -107,7 +111,7 @@ git checkout local-4.7.3 git cherry-pick feat/ ``` -After upstream merges and you rebase `local-4.7.3` onto the next release tag, drop the cherry-picked commit — it's already included via upstream. +After upstream merges and you rebase `local-4.7.3` onto the next release tag, drop the cherry-picked commit. Upstream already includes it. ### Style notes for upstream PRs @@ -115,7 +119,7 @@ From reading merged PRs on `anomalyco/sst`: - Titles are imperative and prose-style, no `feat:` / `fix:` prefix. "Add X", "Support X", "Fix X". Backticks around code identifiers are fine. - Commit messages match the PR title; squash-merges on the upstream side rewrite them anyway. -- PR bodies are conversational — what the change does, why it's needed, any API-design decisions, and a short test plan. Don't pad with boilerplate. +- PR bodies are conversational: what the change does, why it's needed, any API-design decisions, and a short test plan. Don't pad with boilerplate. - Reference PR #6744 as a recent example of the expected length and voice. ## Recovery: symlink was wiped @@ -146,6 +150,6 @@ The Go CLI expects the platform source to match the version it was built from. I - A new component file that references a Go RPC method the CLI doesn't know about - A new provider dep added to `platform/package.json` that isn't in the installed `.sst/platform/node_modules` -- Changes to the `dist/` layout (generated artifacts) — we don't symlink `dist/`, but component code that references a new `dist/` helper will fail +- Changes to the `dist/` layout (generated artifacts). We don't symlink `dist/`, so component code that references a new `dist/` helper will fail -The `local-` branch exists to avoid this: it pins local patches to the exact baseline the installed CLI expects. Only rebase it forward when you bump `sst` in maple. +The `local-` branch exists to avoid this. It pins local patches to the exact baseline the installed CLI expects. Only rebase it forward when you bump `sst` in maple. diff --git a/docs/tinybird-pr-branches.md b/docs/tinybird-pr-branches.md index cd7f14139d..647ee1630a 100644 --- a/docs/tinybird-pr-branches.md +++ b/docs/tinybird-pr-branches.md @@ -1,59 +1,70 @@ # Per-PR Tinybird branches -Every PR preview deploy gets its own **empty ephemeral Tinybird branch** so schema changes can be +Every PR preview deploy gets its own **empty ephemeral Tinybird branch**, so schema changes can be deployed and exercised without touching production. The branch is created on PR open, refreshed on each push, and removed when the PR closes. Branches carry **no production data**. They used to be created with `--last-partition`, which -attached the latest production partition of every datasource — live customer telemetry, in an -environment readable by anyone with the preview URL and retained until a best-effort teardown +attached the latest production partition of every datasource. That put live customer telemetry in +an environment readable by anyone with the preview URL, retained until a best-effort teardown removed it. A preview that needs rows seeds its own (see [Getting data in](#getting-data-in)). ## How it works -The PR-preview pipeline (`.github/workflows/deploy-pr-preview.yml`) wraps the existing Alchemy -deploy with two extra steps backed by `scripts/tinybird-pr-branch.ts`: +The PR-preview pipeline (`.github/workflows/deploy-pr-preview.yml`) wraps the Alchemy deploy with +two extra steps backed by `scripts/tinybird-pr-branch.ts`: 1. **`up `** (on `opened` / `synchronize` / `reopened`) - - `tb branch create pr_` — creates the branch empty. Idempotent: re-running on a new commit + - `tb branch create pr_` creates the branch empty. Idempotent: re-running on a new commit reuses the branch. - - `tb --branch=pr_ deploy` — deploys _this PR's_ datasources/materialized views into the - branch. - - Resolves the branch's admin token and writes `TINYBIRD_HOST` / `TINYBIRD_TOKEN` to - `$GITHUB_ENV`, **overriding** the Infisical `dev` values for the steps that follow. -2. **`alchemy:deploy:pr`** then binds the whole preview stack (api / web / alerting / chat-agent - and, if pointed at it, the Rust ingest gateway) to the branch — no app code changes, because - every `alchemy.run.ts` already reads `TINYBIRD_HOST` / `TINYBIRD_TOKEN` from `process.env` - (see `apps/api/src/lib/Env.ts`, used in `apps/api/src/lib/WarehouseQueryService.ts`). -3. **`down `** (on `closed`, after `alchemy:destroy:pr`) — `tb branch rm pr_ --yes`. + - Reads the branch's own admin token from the environments API (`GET /v1/environments`, + authorized by the parent workspace token). + - `tb deploy --allow-destructive-operations`, run as the branch token, deploys _this PR's_ + datasources and materialized views into the branch. + - Writes `TINYBIRD_HOST` / `TINYBIRD_TOKEN` to `$GITHUB_ENV`, **overriding** the Infisical + `dev` values for the steps that follow. The host stays the parent's; the branch token does + the routing. +2. **`alchemy:deploy:pr`** then binds the whole preview stack (the Workers and the Rust ingest + gateway) to the branch with no app code changes, because the stack reads `TINYBIRD_HOST` / + `TINYBIRD_TOKEN` from the deploy env (`packages/infra/src/env.ts`) and the runtime reads them + through `packages/backend/src/platform/Env.ts`. +3. **`down `** (on `closed`, after `alchemy:destroy:pr`) runs `tb branch rm pr_ --yes`. + +The close-event teardown is best effort. The scheduled `cleanup-preview-orphans.yml` workflow runs +`bun scripts/tinybird-pr-branch.ts sweep`, which removes every `pr_` branch whose PR is closed. + +Every `tb` call in the script is `tb --cloud --host … --token …`, never `--branch=`. The CLI +resolves `--branch=` through the user-workspaces endpoint, which lists branches only for a human +login token, so under a workspace token every branch is "not found". ## Getting data in A fresh branch has the PR's schema and no rows, so charts render empty until you seed it: -- **Demo seed** — the internal `demo.seed` route (`apps/api/src/routes/internal/demo.http.ts`, - backed by `packages/backend/src/services/org/DemoService.ts`) writes synthetic - services, routes and queries into the branch. Enough for most UI and query work. -- **Point ingest at the branch** — send real OTLP traffic from a local collector or +- **Demo seed.** The internal `demo.seed` route (`apps/api/src/routes/internal/demo.http.ts`, + backed by `packages/backend/src/services/org/DemoService.ts`) writes synthetic services, routes + and queries into the branch. Enough for most UI and query work. +- **Point ingest at the branch.** Send real OTLP traffic from a local collector, or run `bun scripts/ingest-dummy-traces.ts`, which generates synthetic spans. Neither path uses customer data. If a change can only be validated against production cardinality, -benchmark it against production instead (`apps/api/scripts/BENCH.md`) rather than copying rows into -a preview. +benchmark it against production instead (`apps/api/scripts/BENCH.md`, `docs/query-benchmarking.md`) +rather than copying rows into a preview. ## Caveats -- **Branches share compute with production.** That's why teardown on PR close is mandatory and we - scope branches to the open-PR set. Avoid leaving stray branches around. +- **Branches share compute with production.** That is why teardown on PR close is mandatory and + branches are scoped to the open-PR set. Do not leave stray branches around. - Requires a Tinybird plan that supports branches. -- The Infisical `dev` environment's `TINYBIRD_HOST` / `TINYBIRD_TOKEN` must be the **parent workspace - admin** host+token — the script uses them to create the branch before swapping in the branch's - own credentials. -- If the CLI's `token ls` output can't be parsed for the admin token, pin it explicitly with the - `TB_BRANCH_ADMIN_TOKEN_NAME` env var. +- The Infisical `dev` environment's `TINYBIRD_HOST` / `TINYBIRD_TOKEN` must be the **parent + workspace admin** host and token. The script uses them to create the branch and read its token + before swapping in the branch's own credentials. ## Doing it manually +`--branch=` works only when the CLI is logged in as a person (`tb login`), not with a workspace +token. + ```bash # create empty, then deploy this checkout's schema into it tb --cloud branch create pr_123 diff --git a/docs/warehouse-rollups.md b/docs/warehouse-rollups.md index 517dd35a5f..ee87996546 100644 --- a/docs/warehouse-rollups.md +++ b/docs/warehouse-rollups.md @@ -5,8 +5,8 @@ time, and for a long time nobody could answer "should this be an MV?" without re from scratch. This is that answer. Definitions live in `packages/domain/src/tinybird/materializations.ts` (MVs) and -`datasources.ts` (targets). Everything else — the ClickHouse DDL snapshot, `local-schema.sql` -for the embedded chDB engine, the Rust insert mappings — is generated from those two files. +`datasources.ts` (targets). Everything else is generated from those two files: the ClickHouse +DDL snapshot, `local-schema.sql` for the embedded chDB engine, and the Rust insert mappings. --- @@ -14,7 +14,7 @@ for the embedded chDB engine, the Rust insert mappings — is generated from tho | Tier | Grain | TTL | Answers | | ------------ | -------------------------- | ---- | -------------------------------------------------------------------------------------------------------- | -| **raw** | one row per span/log/point | 30d | "show me this exact trace / these log lines" — anything needing attributes, span names, or a specific id | +| **raw** | one row per span/log/point | 30d | "show me this exact trace / these log lines", or anything needing attributes, span names, or an id| | **minutely** | pre-aggregated per minute | 90d | sub-hour timeseries and alert evaluation, where a per-span scan is the only alternative | | **hourly** | pre-aggregated per hour | 365d | dashboards and trends over days-to-a-year | @@ -25,8 +25,8 @@ for the embedded chDB engine, the Rust insert mappings — is generated from tho Rollup routes union a **raw edge** with a **rollup interior**: the rollup answers whole buckets, and the raw table covers the partial buckets at each end of the window. Getting the edge grain wrong double-counts every span in the first and last partial bucket, and the SQL -looks perfectly reasonable while it does — see the comment on `edgeGrain` in -`queries/traces.ts`. +still looks reasonable. See the comment on `edgeGrain` in +`packages/query-engine/src/ch/queries/traces.ts`. --- @@ -37,13 +37,14 @@ of a table we already have. 1. **Re-sort for point lookup.** `trace_detail_spans` is a 1:1 copy of `traces` ordered by `(OrgId, TraceId, SpanId)` instead of `(OrgId, ServiceName, Timestamp)`. That turns - "fetch this trace" from a bloom-filter scan into a primary-key lookup. It costs 82 GB — - the single most expensive object we have — and it earns it. + "fetch this trace" from a bloom-filter scan into a primary-key lookup. It costs 82 GB, + the single most expensive object we have, and it earns it. 2. **Pre-aggregation for scans.** `*_aggregates_hourly`, `service_overview_*`, `service_operations_*`. Trades write amplification for orders-of-magnitude less read. 3. **Filtered projection.** `error_events` keeps only `StatusCode = 'Error'` and unwraps the - exception event — or, when both the event and status message are absent, the `exception.*` / `error.*` span attributes — - so error queries never touch the Map columns of the full traces table. + exception event. When both the event and the status message are absent, it falls back to the + `exception.*` / `error.*` span attributes. Error queries never touch the Map columns of the + full traces table. Storage is not free and the ratio is worse than it looks: `traces` is 110 GB, and its MV descendants total roughly 116 GB. **We store traces more than twice over.** Every new MV on @@ -53,14 +54,15 @@ descendants total roughly 116 GB. **We store traces more than twice over.** Ever ## Rules -These are not style preferences. Each one is here because its absence cost something -measurable, found during the 2026-08 sweep. +Each rule is here because its absence cost something measurable, found during the 2026-08 +sweep. ### Every MV names its consuming query No consumer, no MV. `error_spans` was materialized from every error span for the lifetime of -the product and **never read once** — no query in the DSL ever called `from(ErrorSpans)`. -Worse, it was still described to LLM agents in `warehouse-catalog.ts` as "use this instead of +the product and **never read once**. No query in the DSL ever called `from(ErrorSpans)`. +It was also described to LLM agents in +`packages/backend/src/services/warehouse/warehouse-catalog.ts` as "use this instead of `traces WHERE StatusCode = 'Error'`", so `run_sql` agents were actively steered onto it. ### Any MV that fans out over unbounded values must carry a cardinality bound @@ -69,18 +71,18 @@ Worse, it was still described to LLM agents in `warehouse-catalog.ts` as "use th `AttributeValue != ''` as its only filter, `attribute_values_hourly` reached **1.59 billion rows / 12.3 GB** to serve an autocomplete dropdown read a couple hundred times a week. -Bound it by measuring what is actually in the table, not by guessing. The intuitive rules — -a length cap and a denylist of id-ish key names — would have missed the top two keys, which -were `idle_ns` and `busy_ns`: short _numeric measurements_ that together were ~70% of the -rows. See `attributeValueCardinalityBound` for the rules that resulted and why. +Bound it by measuring what is actually in the table. The intuitive rules (a length cap and a +denylist of id-ish key names) would have missed the top two keys, `idle_ns` and `busy_ns`: +short _numeric measurements_ that together were ~70% of the rows. See +`attributeValueCardinalityBound` in `materializations.ts` for the resulting rules and why. ### Keep the write filter and the read guard in sync, and test that they agree `span_metrics_calls_hourly` sat at **0 rows for its entire existence**. The MV filtered `MetricName IN ('span.metrics.calls', 'calls')`; the collector emits the counter namespaced by its pipeline, as `traces.span.metrics.calls`. Both the write filter and the read fast-path -missed it, so they agreed with each other and disagreed with reality — and every read fell -back to a raw scan measured at ~7s p95. +missed it, so they agreed with each other and disagreed with reality. Every read fell back +to a raw scan measured at ~7s p95. A rollup whose target is empty is indistinguishable from a rollup nobody queries. Neither the schema lint nor the SQL catalog can catch this; only looking at row counts can. @@ -89,17 +91,17 @@ schema lint nor the SQL catalog can catch this; only looking at row counts can. The raw edge and the rollup interior have to tile the window exactly once. Written by hand that is two inequalities which must stay each other's exact complement, and when they drift -nothing errors — the counts simply come out wrong, on a chart that looks fine. +nothing errors. The counts come out wrong on a chart that looks fine. `packages/query-engine/src/ch/queries/rollup-splice.ts` owns that boundary: `interiorConditions(bucketColumn, grain)` for the rollup branch, `edgeCondition(tsColumn, -grain)` for the raw branch, defined as its complement. Traces, logs, services and -service-operations use them. +grain)` for the raw branch, defined as its complement. Traces, logs, services, +service-operations and the service map use them. -The service-map family did not, and drifted in both directions at once. `serviceDependencies` +The service-map family used to hand-roll it, and drifted. `serviceDependencies` hand-rolled the boundary correctly. `serviceDbEdges`, `serviceExternalEdges` and the db-query-shape drill-down hand-rolled the same boundary with the interior floored to -`toStartOfHour(startTime)` while their raw branch covered only the trailing hour — so every +`toStartOfHour(startTime)` while their raw branch covered only the trailing hour. Every window whose start was not hour-aligned counted the whole leading hour, including spans outside the window. The web app snaps a 12h range to a 5-minute grid, so the start was essentially never aligned. The DB nodes on the service map read high against the service @@ -110,27 +112,27 @@ Two things made it survive: nothing forced the shared helper, and those queries fixture in the SQL catalog at all, so neither the DESCRIBE sweep nor any structural gate ever looked at them. -Both are closed now. `unsplicedTwoTierQueries` in `benchmark/catalog.ts` fails any query naming both +Both are closed now. `unsplicedTwoTierQueries` in `packages/query-engine/src/benchmark/catalog.ts` fails any query naming both a rollup table and a raw table whose boundary is not a computed `firstFullBucket`, and -`service-map-parity.clickhouse.e2e.test.ts` compares the spliced result against a flat scan +`packages/backend/src/services/warehouse/service-map-parity.clickhouse.e2e.test.ts` compares the spliced result against a flat scan with spans seeded exactly on each seam. **There is no allowlist on that gate.** A single-tier query never trips it, and a deliberately-approximate one (`serviceHealthSnapshot` floors to -the hour on purpose) reads only its rollup and never trips it either. A new query that needs -an exemption is the gate finding something, not the gate needing a hole. +the hour on purpose) reads only its rollup and never trips it either. If a new query seems to +need an exemption, the gate has found a bug. ### A routing guard must be tested on the tier it selects, not on a table name `canUseAnnualServiceOverview` required `allMetrics === true`. Alert evaluation (`computeAlertBuckets`) never sets it, so every alert fell through to a flat scan of the -325M-row per-span `service_overview_spans` — **165k scans per 3 days** — while the 2.2M-row +325M-row per-span `service_overview_spans` (**165k scans per 3 days**) while the 2.2M-row minutely rollup built for exactly that shape sat unreachable. `benchmark/catalog.test.ts` already asserts every routing predicate is exercised both ways, and it passed throughout. The gap was that the test guarding this route asserted -`sql.toContain("FROM service_overview_spans")` — and the tiered union reads that table too, +`sql.toContain("FROM service_overview_spans")`, and the tiered union reads that table too, as its raw edge. The assertion could not tell the two routes apart. **Assert the tier that distinguishes the branch** (`FROM service_overview_minutely`), and include a fixture shaped -like a real alert rule, not just a dashboard query. +like a real alert rule as well as a dashboard query. --- @@ -140,18 +142,18 @@ like a real alert rule, not just a dashboard query. 2. If it fans out over values, decide the cardinality bound first. 3. Add a routing fixture that pins the **tier**, both ways. 4. `bun run clickhouse:schema` and `bun run tinybird:manifest` to regenerate. -5. Write the numbered migration in `packages/domain/src/clickhouse/migrations/` — +5. Write the numbered migration in `packages/domain/src/clickhouse/migrations/`. **`DROP VIEW` before `DROP TABLE`, always**. The inverse leaves an MV pointing at a missing target and wedges ingest with `Code: 60 UNKNOWN_TABLE`. 6. Set `requiredForIngest: false` unless the Rust gateway writes the table directly. Bumping `clickHouseSchemaVersion` un-readies ingest routing for every BYO-ClickHouse org. 7. `bun run local-schema:bump ` to bump `LOCAL_SCHEMA_VERSION`, retain the snapshot, append the history identity, and scaffold the local-store migration edge from the previous - one. It writes every mechanical edit — including the pinned literals in the bun test and the - native probe — and leaves the edge's `apply` for you. Write the DDL there, including explicit + one. It writes every mechanical edit, including the pinned literals in the bun test and the + native probe, and leaves the edge's `apply` for you. Write the DDL there, including explicit drops, because `assertPhysicalSchema` fails on leftover objects too. -Because step 5–6 are expensive, **batch removals into one migration** rather than shipping +Steps 5 and 6 are expensive, so **batch removals into one migration** instead of shipping them one at a time. Migration `0019_mv_sweep` is the worked example. --- @@ -160,18 +162,18 @@ them one at a time. Migration `0019_mv_sweep` is the worked example. Recorded so the next sweep does not re-derive them. -- **`alertRawQuery` — 173,000 s/week**, the single largest consumer of warehouse time. It is +- **`alertRawQuery`: 173,000 s/week**, the single largest consumer of warehouse time. It is user-authored SQL against raw tables, so no MV can see it and no routing guard applies. Needs its own design: a rollup users can target, or per-rule result caching. -- **`trace_detail_spans` TTL.** 82 GB, and TTL is the only remaining lever — column narrowing +- **`trace_detail_spans` TTL.** 82 GB, and TTL is the only remaining lever. Column narrowing recovered 1.19 GB (1.5%) and the rest is `SpanAttributes` (31 GB) and the incompressible `SpanId` (17 GB). A product decision about the trace-drilldown window. -- **`service_map_spans` + `service_map_children`** — 18.6 GB of intermediate producing a +- **`service_map_spans` + `service_map_children`**: 18.6 GB of intermediate producing a 223 MB hourly rollup. Suspicious amplification, but the recent-window branch reads them directly. - **`metrics_exponential_histogram` is empty.** Not dead: ingest writes it - (`apps/ingest/src/telemetry.rs`), and it is empty only because no SDK currently exports + (`apps/ingest/src/clickhouse_insert_mappings.rs`, table names in `telemetry.rs`). It is empty only because no SDK currently exports exponential histograms. Leave it and its two MVs alone. -- **Naming drift** — `trace_list_mv_mv`, and `serviceMapDbQuerySignaturesHourlyMv` producing +- **Naming drift**: `trace_list_mv_mv`, and `serviceMapDbQuerySignaturesHourlyMv` producing `service_map_db_query_shapes_hourly_mv`. Cosmetic; renaming costs a full migration plus a - local schema version, which is not worth spending on aesthetics. + local schema version, which is not worth it. diff --git a/skills/maple-audit/SKILL.md b/skills/maple-audit/SKILL.md index 0d9e78e210..74cc4fe336 100644 --- a/skills/maple-audit/SKILL.md +++ b/skills/maple-audit/SKILL.md @@ -5,11 +5,11 @@ description: "Audit an already-instrumented project against Maple's OpenTelemetr # Maple instrumentation audit -Review an existing OpenTelemetry setup against what Maple actually consumes, produce a findings report, then fix the gaps. This is the counterpart to `maple-onboard`: that skill installs telemetry from scratch; this one assumes instrumentation exists and asks whether it's *right*. +Review an existing OpenTelemetry setup against what Maple actually consumes, produce a findings report, then fix the gaps. This is the counterpart to `maple-onboard`. That skill installs telemetry from scratch; this one assumes instrumentation exists and checks whether it is *right*. -Before auditing, read `checks.md` in this skill's directory — it is the full check registry (check ids, severities, what each gap breaks in Maple). Every finding you report must cite a check id from there; never invent attribute keys or conventions that aren't in `checks.md` or the upstream OTel semconv. +Before auditing, read `checks.md` in this skill's directory. It is the full check registry: check ids, severities, and what each gap breaks in Maple. Every finding you report must cite a check id from it. Never invent attribute keys or conventions that aren't in `checks.md` or the upstream OTel semconv. -For *how* to fix what you find, use the companion skills — don't improvise recipes: +For *how* to fix what you find, use the companion skills. Don't improvise recipes: - `maple-onboarding-style` for general OTel taste, VCS attributes, log bridges, metrics, LLM conventions. - `maple-nextjs-style`, `maple-nodejs-style`, `maple-python-style`, `maple-effect-style`, `maple-go-style`, `maple-rust-style`, `maple-java-style`, `maple-csharp-style`, `maple-kotlin-style` for stack-specific bootstrap shapes. @@ -18,58 +18,59 @@ For *how* to fix what you find, use the companion skills — don't improvise rec | Severity | Meaning | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `critical` | Breaks a Maple feature outright or is a data risk: missing `service.name`, hand-stamped status strings (error analytics read zero rows), logs without trace correlation, missing `peer.service`/`db.system` on client spans, PII in attributes. | -| `warn` | Feature works degraded or work is invisible: missing environment/VCS resource attrs, deprecated semconv keys, double-emission, high-cardinality labels or span names, outbound calls left `Internal`, untraced operations (missing auto-instrumentation for a used framework/driver, business operations and background jobs with no spans). | +| `critical` | Breaks a Maple feature outright or is a data risk: missing `service.name`, hand-stamped status strings (error analytics read zero rows), logs without trace correlation, outbound calls that can't form a service-map edge (no `Client` span, broken `traceparent`, DB spans without `db.system.name`), PII in attributes. | +| `warn` | The feature works degraded or work is invisible: missing environment/VCS resource attrs, deprecated semconv keys, double-emission, high-cardinality labels or span names, outbound calls left `Internal`, untraced operations (missing auto-instrumentation for a used framework/driver, business operations and background jobs with no spans). | | `info` | Signal-quality improvements: trace-context propagation across async boundaries, span noise, missing metrics, naming style, missing `gen_ai.*` on LLM calls. | -## Step 1 — Enumerate services and classify +## Step 1: Enumerate services and classify -Map every app/service in the repo exactly as `maple-onboard` Step 1 does (workspace manifests, `apps/*`, `services/*`, mobile, edge/serverless functions — skip pure type/config packages). For each, find its OTel bootstrap and classify: +Map every app/service in the repo exactly as `maple-onboard` Step 1 does (workspace manifests, `apps/*`, `services/*`, mobile, edge/serverless functions; skip pure type/config packages). For each, find its OTel bootstrap and classify it: -- **instrumented** — has an SDK bootstrap; gets the full audit below. -- **partial** — some signals wired (e.g. traces but no log bridge); audit what exists, flag what's absent. -- **not instrumented** — no OTel at all. That is a single `critical` finding pointing the user at `maple-onboard`; don't run per-check audits against nothing. +- **instrumented**: has an SDK bootstrap. Gets the full audit below. +- **partial**: some signals wired (e.g. traces but no log bridge). Audit what exists and flag what's absent. +- **not instrumented**: no OTel at all. Report a single `critical` finding pointing the user at `maple-onboard`. Don't run per-check audits against nothing. Show the user the service list with classifications before auditing, so they can correct it. -## Step 2 — Static code audit +## Step 2: Static code audit Per instrumented service, work through the categories in `checks.md`: -1. **Bootstrap & resource (RES-\*)** — read the SDK init: is `service.name` explicit, are `service.version`, `deployment.environment.name`, `vcs.repository.url.full`, `vcs.ref.head.revision` on the resource? Any invented keys? -2. **Status & kind (STAT-\*)** — grep for status handling: SDK status enum vs hand-stamped strings; failure paths recording exceptions; outbound calls created as `Client`/`Producer` spans. -3. **Trace coverage (SPAN-\*)** — what *should* be traced but isn't. Compare the dependency manifest against the instrumentation registered in the bootstrap (a DB driver or queue client with no instrumentation package means that whole category emits nothing). Then read the code for critical business operations, background jobs, and queue consumers with no span in their call path — when this service misbehaves, what would an operator need to see that currently emits nothing? -4. **Service-map attribution (MAP-\*)** — for every outbound dependency in the code (HTTP clients to internal services, DB drivers, queues), check the corresponding spans set `peer.service` (and `db.system` for DBs) consistently. -5. **Attribute keys (REN-\*, NAME-\*)** — grep attribute call sites (`setAttribute`, `set_attribute`, `setAttributes`, attribute map literals) for the deprecated/camelCase keys in the REN table and for double-emission of old+new keys. -6. **Logs (LOG-\*)** — is an OTLP log bridge wired under the logger the app actually uses? Do in-span logs carry trace context? Structured fields? -7. **Metrics (MET-\*)** — instruments at module scope, low-cardinality labels, business coverage. -8. **PII (PII-01)** — scan attribute values for emails, tokens, auth headers, full bodies. -9. **LLM (LLM-\*)** — only if the project calls LLM providers. +1. **Bootstrap & resource (RES-\*)**: read the SDK init. Is `service.name` explicit? Are `service.version`, `deployment.environment.name`, `vcs.repository.url.full`, `vcs.ref.head.revision` on the resource? Any invented keys? +2. **Status & kind (STAT-\*)**: grep status handling. SDK status enum or hand-stamped strings? Do failure paths record exceptions? Are outbound calls `Client`/`Producer` spans? +3. **Trace coverage (SPAN-\*)**: find what *should* be traced but isn't. Compare the dependency manifest against the instrumentation registered in the bootstrap: a DB driver or queue client with no instrumentation package means that whole category emits nothing. Then read the code for critical business operations, background jobs, and queue consumers with no span in their call path. Ask what an operator would need to see when this service misbehaves that currently emits nothing. +4. **Service-map attribution (MAP-\*)**: for every outbound dependency in the code, check the call produces what the map joins on. Calls to internal services need a `Client`/`Producer` span that propagates `traceparent` to an instrumented callee. DB calls need `db.system.name` (and `db.namespace`). Queue/RPC calls need `messaging.*` / `rpc.*` attributes. +5. **Attribute keys (REN-\*, NAME-\*)**: grep attribute call sites (`setAttribute`, `set_attribute`, `setAttributes`, attribute map literals) for the deprecated/camelCase keys in the REN table and for double-emission of old and new keys. +6. **Logs (LOG-\*)**: is an OTLP log bridge wired under the logger the app actually uses? Do in-span logs carry trace context? Are fields structured? +7. **Metrics (MET-\*)**: instruments at module scope, low-cardinality labels, business coverage. +8. **PII (PII-01)**: scan attribute values for emails, tokens, auth headers, full bodies. +9. **LLM (LLM-\*)**: only if the project calls LLM providers. -Collect findings as you go: service, check id, severity, evidence (`file:line`), and which Maple feature it affects (from `checks.md`). +Record each finding as you go: service, check id, severity, evidence (`file:line`), and the Maple feature it affects (from `checks.md`). -## Step 3 — Live cross-check (only when the Maple MCP is connected) +## Step 3: Live cross-check (only when the Maple MCP is connected) -If `mcp__maple__*` tools are available, verify the static findings against what's actually arriving. If they're not available, say so in one line of the report and move on — the static audit stands alone. Don't ask the user to install the MCP mid-audit. +If `mcp__maple__*` tools are available, verify the static findings against what's actually arriving. If they aren't, say so in one line of the report and move on; the static audit stands alone. Don't ask the user to install the MCP mid-audit. -- **`get_instrumentation_recommendations`** — the authoritative list of deprecated/double-emitted/non-conforming attribute keys, reconciled against the org's live span data, plus resource-attribute coverage gaps. Where it disagrees with your static REN/NAME findings, the live data wins (the code you read may not be deployed, or other emitters exist). -- **`list_services`** — confirm every service you enumerated actually reports. A service that's instrumented in code but absent here is a `critical` export problem (bootstrap not loaded, key wrong, exporter blocked). -- **`explore_attributes`** (resource scope) — confirm environment/VCS/version resource attrs arrive in practice. -- **`service_map`** — a service whose code makes outbound calls but shows no outgoing edges confirms MAP-01/02/03 findings from the data side. -- **`get_service_top_operations`** (or `search_traces` per service) — confirm trace-coverage findings: a critical operation you flagged under SPAN-02/03 that never appears as a span name in live data is confirmed untraced; a DB-heavy service whose traces contain no DB client spans confirms SPAN-01. +- **`audit_setup`**: Maple's own live audit of the org. Its findings use the same check ids as `checks.md` (RES, LOG, STAT, MAP, NAME, PII, MET) plus trace-completeness (`TRC-*`) and config (`CFG-*`) checks. Attach its evidence to the matching static findings. +- **`get_instrumentation_recommendations`**: the authoritative list of deprecated, double-emitted, and non-conforming attribute keys, reconciled against the org's live span data, plus resource-attribute coverage gaps. Where it disagrees with your static REN/NAME findings, the live data wins (the code you read may not be deployed, or other emitters exist). +- **`list_services`**: confirm every service you enumerated actually reports. A service instrumented in code but absent here is a `critical` export problem (bootstrap not loaded, wrong key, exporter blocked). +- **`explore_attributes`** (resource scope): confirm environment/VCS/version resource attrs arrive in practice. +- **`service_map`**: a service whose code makes outbound calls but shows no outgoing edges confirms MAP-01/02/04 findings from the data side. +- **`get_service_top_operations`** (or `search_traces` per service): confirm trace-coverage findings. A critical operation you flagged under SPAN-02/03 that never appears as a span name in live data is confirmed untraced. A DB-heavy service whose traces contain no DB client spans confirms SPAN-01. Annotate existing findings with live evidence rather than duplicating them. -## Step 4 — Findings report +## Step 4: Findings report Present the report **before making any edit**. Format: ``` -## Instrumentation audit — +## Instrumentation audit: ### (instrumented | partial | not instrumented) | Severity | Check | Finding | Evidence | Affects | -| critical | MAP-01 | Client spans to billing lack peer.service | src/billing.ts:88 | service map edge missing | +| critical | MAP-01 | billing client call made as Internal span, no traceparent | src/billing.ts:88 | service map edge missing | | warn | REN-02 | emits http.status_code | live: 12.4k spans/24h | rename to http.response.status_code | ... @@ -77,31 +78,31 @@ Present the report **before making any edit**. Format: N critical · N warn · N info across M services. Recommended fix order: … ``` -Every row carries a check id and concrete evidence (file:line for static, counts/tool output for live). If the MCP wasn't connected, note "live cross-check skipped — Maple MCP not connected" here. +Every row carries a check id and concrete evidence (`file:line` for static, counts or tool output for live). If the MCP wasn't connected, note "live cross-check skipped: Maple MCP not connected" here. -## Step 5 — Apply fixes +## Step 5: Apply fixes After the report, fix in severity order (`critical` → `warn` → `info`), following the relevant style skill for each edit. Rules: -- Pure key renames (REN rows of kind *rename*) have two valid routes: rename at the SDK (preferred — fixes it at the source), or accept the matching Recommendation Issue in Maple Settings → Ingestion, which creates an ingest attribute mapping. Tell the user both; default to the SDK rename when you're already editing that file. -- Double-emission and naming issues can **only** be fixed at the SDK — an ingest mapping can't merge keys. -- Idempotent edits; preserve the project's package manager, logger, and formatting. -- Never remove an existing observability vendor (Sentry, Datadog, …) — coexistence is the default. -- Never put PII in attributes; when fixing PII-01, replace the value with an ID or drop the attribute, don't hash-and-keep. +- Pure key renames (REN rows of kind *rename*) have two valid routes: rename at the SDK (preferred, fixes it at the source), or accept the matching recommendation in Maple under Settings → Ingestion, which creates an ingest attribute mapping. Tell the user both. Default to the SDK rename when you're already editing that file. +- Double-emission and naming issues can **only** be fixed at the SDK. An ingest mapping can't merge keys. +- Make idempotent edits. Preserve the project's package manager, logger, and formatting. +- Never remove an existing observability vendor (Sentry, Datadog, …). Coexistence is the default. +- Never put PII in attributes. When fixing PII-01, replace the value with an ID or drop the attribute. Don't hash and keep it. - Never commit, push, or open PRs. -If the user only wants the review, stop after Step 4 — the report is a complete deliverable. +If the user only wants the review, stop after Step 4. The report is a complete deliverable. -## Step 6 — Re-verify +## Step 6: Re-verify -For every service you touched, run the same smoke as `maple-onboard` Step 4: the service's own dev/build command starts cleanly, and OTLP POSTs from the running process return 2xx. A fix that breaks startup is a regression — fix or revert it, don't paper over it. +For every service you touched, run the same smoke as `maple-onboard` Step 4: the service's own dev/build command starts cleanly, and OTLP POSTs from the running process return 2xx. A fix that breaks startup is a regression. Fix or revert it; don't paper over it. -Then close out: summarize what changed per service, and tell the user that after deploying, re-running `get_instrumentation_recommendations` (or checking Settings → Ingestion in Maple) will show issues auto-resolving as the offending keys stop arriving — reconciliation is automatic. +Then close out. Summarize what changed per service. Tell the user that after deploying, re-running `get_instrumentation_recommendations` (or checking Settings → Ingestion in Maple) shows issues auto-resolving as the offending keys stop arriving. Reconciliation is automatic. ## Hard rules - No edits before the findings report has been shown (Step 4). -- Every finding cites a check id from `checks.md`; no invented attribute keys or conventions. -- The static audit must work standalone — the Maple MCP is an enhancer, never a requirement. -- Never modify files outside the project root; never commit or push. +- Every finding cites a check id from `checks.md`. No invented attribute keys or conventions. +- The static audit must work standalone. The Maple MCP is an enhancer, never a requirement. +- Never modify files outside the project root. Never commit or push. - Fix recipes come from the `maple-*-style` skills, not from memory. diff --git a/skills/maple-audit/checks.md b/skills/maple-audit/checks.md index bc0a74972c..8b8a7df933 100644 --- a/skills/maple-audit/checks.md +++ b/skills/maple-audit/checks.md @@ -1,22 +1,22 @@ # Maple instrumentation audit checklist -The full check registry for the `maple-audit` skill. Each check has a stable id, what to look for in the code, a severity, the Maple feature affected, and where to find the fix recipe. +The full check registry for the `maple-audit` skill. Each check has a stable id, what to look for in the code, a severity, the Maple feature affected, and where to find the fix recipe. Maple's live `audit_setup` MCP tool reports findings under the same ids. Severities: -- **critical** — breaks a Maple feature outright or is a data risk. Fix first. -- **warn** — the feature works degraded or the data is harder to query than it should be. -- **info** — nice-to-have; improves signal quality but nothing is broken. +- **critical**: breaks a Maple feature outright or is a data risk. Fix first. +- **warn**: the feature works degraded, or the data is harder to query than it should be. +- **info**: improves signal quality; nothing is broken. Fix pointers reference the companion skills: `maple-onboarding-style` (general), `maple-nodejs-style`, `maple-nextjs-style`, `maple-python-style`, `maple-go-style`, `maple-rust-style`, `maple-java-style`, `maple-csharp-style`, `maple-kotlin-style`, `maple-effect-style`. -## RES — Resource attributes +## RES: Resource attributes Set once per process on the SDK resource, in the OTel bootstrap. Look for the resource construction (`resourceFromAttributes(...)` / `Resource.create(...)` in TS, `Resource(attributes={...})` in Python, `resource.NewWithAttributes(...)` in Go, `Resource::builder()` in Rust, `registerOTel({ attributes })` for `@vercel/otel`). | Id | Check | Severity | Feature affected | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------- | -| RES-01 | `service.name` is set explicitly per service (not the SDK default). | critical | Without it spans land in a synthetic `unknown_service` bucket — services list, map, alerts all blind. | +| RES-01 | `service.name` is set explicitly per service (not the SDK default). | critical | Without it spans land in a synthetic `unknown_service` bucket; the services list, map, and alerts are all blind. | | RES-02 | `service.version` is set (release/semver or commit short-SHA). | warn | Per-version slices on service overview. | | RES-03 | `deployment.environment.name` is set (the current semconv key). Maple also accepts the legacy `deployment.environment`; emitting either works, dual-emit is what Maple's SDKs do. | warn | Environment filtering/slicing everywhere (`env =` alias, per-env throughput/latency/error rate). | | RES-04 | `vcs.repository.url.full` is set to the canonical https repo URL (not SSH, not a local path). | warn | Links telemetry back to source; repo context in error debugging. | @@ -25,56 +25,58 @@ Set once per process on the SDK resource, in the OTel bootstrap. Look for the re | RES-07 | `service.instance.id` set when the service runs multiple replicas. | info | Distinguishing replicas in trace queries. | | RES-08 | Cloud/platform attrs (`cloud.provider`, `cloud.platform`, `cloud.region`, `process.runtime.name`) present where the SDK's auto-detectors don't already supply them. Keep `process.runtime.name` values consistent across services (`nodejs` everywhere, not `nodejs` + `node`). | info | Platform badge + runtime icon on the service map. | -Fix: `maple-onboarding-style` → "VCS resource attributes" + the per-language style skill's bootstrap section. +Fix: `maple-onboarding-style` → "VCS resource attributes", plus the per-language style skill's bootstrap section. -## STAT — Span status & kind +## STAT: Span status & kind These are span **fields**, not attributes, and both spellings are load-bearing on the Maple side: stored values must be Title Case (`Ok` / `Error` / `Unset`, `Server` / `Client` / `Producer` / `Consumer` / `Internal`). | Id | Check | Severity | Feature affected | | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | -| STAT-01 | Status is set via the SDK's status API (`span.setStatus({ code: SpanStatusCode.ERROR })`, `span.set_status(StatusCode.ERROR)`, `span.SetStatus(codes.Error, …)`) — never a hand-stamped string attribute or a hand-built OTLP value like `"ERROR"`/`"error"`. | critical | Error analytics filter `WHERE StatusCode = 'Error'` — wrong casing silently yields **zero errors** everywhere. | +| STAT-01 | Status is set via the SDK's status API (`span.setStatus({ code: SpanStatusCode.ERROR })`, `span.set_status(StatusCode.ERROR)`, `span.SetStatus(codes.Error, …)`). Never a hand-stamped string attribute or a hand-built OTLP value like `"ERROR"`/`"error"`. | critical | Error analytics filter `WHERE StatusCode = 'Error'`; wrong casing silently yields **zero errors** everywhere. | | STAT-02 | Failure paths actually set Error status + `span.recordException(err)` (or language equivalent) before rethrowing. A span that swallows errors as `Unset` hides them. | critical | Error rate, error issues, error-debug context. | -| STAT-03 | Outbound network calls (HTTP clients, DB drivers, queue producers) are `Client`/`Producer` kind spans — not `Internal`. Auto-instrumentation usually gets this right; hand-rolled spans often don't. | warn | The service map only draws edges for `Client`/`Producer` spans; an `Internal` call is invisible in the map. | +| STAT-03 | Outbound network calls (HTTP clients, DB drivers, queue producers) are `Client`/`Producer` kind spans, not `Internal`. Auto-instrumentation usually gets this right; hand-rolled spans often don't. | warn | The service map only draws edges from `Client`/`Producer` spans; an `Internal` call is invisible in the map. | | STAT-04 | Inbound request handlers produce `Server` (or `Consumer`) spans. | warn | Route rendering, throughput attribution, Apdex. | Fix: `maple-onboarding-style` + per-language style skill ("record exceptions" pattern; in TS/JS `withSpan` from `@maple/otel-helpers` handles status + exception recording). -## SPAN — Trace coverage +## SPAN: Trace coverage -Not just "are the spans right" but "is everything that should be traced actually traced". Read the code the way an on-call operator would: when this service misbehaves, which operations would they need to see? Auto-instrumentation covers HTTP in/out, DB queries, and framework lifecycle — that's the floor; anything with real latency or a failure mode above that floor needs its own span (the same bar `maple-onboard` Step 3 applies when installing). +These checks ask whether everything that should be traced is traced. Read the code the way an on-call operator would: when this service misbehaves, which operations would they need to see? Auto-instrumentation covers HTTP in/out, DB queries, and framework lifecycle. That is the floor. Anything above it with real latency or a failure mode needs its own span (the same bar `maple-onboard` Step 3 applies when installing). | Id | Check | Severity | Feature affected | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------ | -| SPAN-01 | Auto-instrumentation is registered for every framework and client library the service actually uses — HTTP server, HTTP client, DB driver, queue client. A used dependency with no instrumentation package/hook means that whole traffic category emits nothing. | warn | Entire categories of work (e.g. all DB queries, all outbound HTTP) invisible in traces, latency breakdowns, and the service map. | +| SPAN-01 | Auto-instrumentation is registered for every framework and client library the service actually uses: HTTP server, HTTP client, DB driver, queue client. A used dependency with no instrumentation package/hook means that whole traffic category emits nothing. | warn | Entire categories of work (e.g. all DB queries, all outbound HTTP) invisible in traces, latency breakdowns, and the service map. | | SPAN-02 | Critical business operations are wrapped in spans: payment/order/signup flows, batch jobs, LLM calls, anything a human would ask "how long did it take and did it fail?" about. Span names are `domain.verb` with entity-ID attributes (order.id, tenant.id) and branch outcomes. | warn | The trace shows one opaque parent span; no per-operation latency, error attribution, or top-operations view for the work that matters. | -| SPAN-03 | Background jobs, cron tasks, and queue consumers start their own root spans (or continue propagated context) — they don't run untraced just because no inbound HTTP request started a trace. | warn | Async work is completely absent from Maple; failures in jobs surface nowhere. | -| SPAN-04 | Trace context propagates across async boundaries the code crosses: queue publish→consume carries context (or links), fire-and-forget tasks inherit the active span, internal service calls send `traceparent`. | info | Traces fragment at the boundary — the consumer's work appears as a disconnected trace instead of one end-to-end view. | +| SPAN-03 | Background jobs, cron tasks, and queue consumers start their own root spans (or continue propagated context). They must not run untraced just because no inbound HTTP request started a trace. | warn | Async work is completely absent from Maple; failures in jobs surface nowhere. | +| SPAN-04 | Trace context propagates across async boundaries the code crosses: queue publish→consume carries context (or links), fire-and-forget tasks inherit the active span, internal service calls send `traceparent`. | info | Traces fragment at the boundary: the consumer's work appears as a disconnected trace instead of one end-to-end view. | | SPAN-05 | No span noise at the other extreme: trivial getters, pure transforms, and per-item spans inside tight loops don't each get a span. | info | Trace waterfalls become unreadable; span volume (and cost) inflates without signal. | Static heuristics for SPAN-01/02: compare the dependency manifest (HTTP frameworks, DB drivers, queue clients, LLM SDKs) against the instrumentation registered in the bootstrap; grep entry points for job/consumer handlers with no `startActiveSpan`/`withSpan`/decorator in their call path. Fix: `maple-onboard` Step 3 + `maple-onboarding-style` (span naming, attributes, `withSpan`); per-language style skill for the auto-instrumentation package list. -## MAP — Service-map attribution +## MAP: Service-map attribution + +Maple draws a service-to-service edge by joining a `Client`/`Producer` span to its child `Server`/`Consumer` span in the downstream service (same trace, `ParentSpanId` = the client span's id). `peer.service` does not draw edges. Database nodes key on `db.system.name` (legacy `db.system` accepted) plus `db.namespace`. Other outbound `Client`/`Producer` calls become external nodes keyed on `messaging.*`, `rpc.*`, or `server.address`. | Id | Check | Severity | Feature affected | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | -| MAP-01 | Every `Client`/`Producer` span calling another **internal** service sets `peer.service`, and its value exactly matches the target's `service.name`. | critical | No `peer.service` (or `peer.service` on a `Server` span) ⇒ no edge drawn; the call path is invisible. | -| MAP-02 | DB client spans set `db.system` (e.g. `postgresql`, `redis`, `clickhouse`) **and** `peer.service` on the same span — both are required for a database node. | critical | No DB node on the map; DB calls render as generic external HTTP. | -| MAP-03 | `peer.service` spelling is consistent across all emitting services (`tinybird`, not `tinybird` + `Tinybird` + `tb`). Pick one canonical name per peer. | warn | Inconsistent names fragment one logical dependency into multiple map nodes. | -| MAP-04 | Queue/RPC dependencies set `messaging.system` + `messaging.destination.name` (queues) or `rpc.system` + `rpc.service` (RPC). | warn | External-dependency edges and the External Dependencies tab. | +| MAP-01 | Every call to another **internal** service is a `Client`/`Producer` span that injects `traceparent`, and the callee is instrumented so its `Server`/`Consumer` span is a child of that span. | critical | No child span in the callee ⇒ no edge; the call shows at most as an external host node. | +| MAP-02 | DB client spans set `db.system.name` (e.g. `postgresql`, `redis`, `clickhouse`; legacy `db.system` is accepted) and `db.namespace` (the database name). | critical | No `db.system.name` ⇒ no DB node; the call renders as a generic external host. No `db.namespace` ⇒ every database behind that driver collapses into one anonymous node. | +| MAP-03 | Dependency-naming values (`db.system.name`, `messaging.system`, `rpc.system`, `peer.service` where emitted) are spelled consistently across services (`clickhouse`, not `clickhouse` + `ClickHouse`). | warn | Each spelling becomes its own node, fragmenting one logical dependency. | +| MAP-04 | Queue/RPC dependencies set `messaging.system` + `messaging.destination.name` (queues) or `rpc.system` + `rpc.service` (RPC). Outbound HTTP to third parties sets `server.address` (auto-instrumentation does). | warn | External-dependency nodes and edges on the map. | -Fix: `maple-onboarding-style`; the public reference is the "Service map" section of Maple's attributes docs (https://maple.dev/docs/concepts/otel-conventions). +Fix: `maple-onboarding-style` plus the per-language style skill (use the framework's instrumented HTTP client so `Client` spans and `traceparent` injection come for free). -## REN — Deprecated / non-conforming attribute keys +## REN: Deprecated / non-conforming attribute keys -Maple keeps fallback chains for the common legacy HTTP keys (legacy and current names feed the same fast columns), so these are renames for forward-compatibility and clean querying — `warn`, not breakage. Search the codebase for the deprecated key in `setAttribute`/`set_attribute`/attribute-map literals. +Maple keeps fallback chains for the common legacy HTTP keys (legacy and current names feed the same fast columns), so these renames are for forward compatibility and clean querying. They are `warn`, not breakage. Search the codebase for the deprecated key in `setAttribute`/`set_attribute`/attribute-map literals. | Id | Deprecated key | Canonical key | | ------ | -------------------------------- | ------------------------------- | @@ -118,45 +120,45 @@ Maple keeps fallback chains for the common legacy HTTP keys (legacy and current All REN checks are **warn**. Two extra rules: -- **REN-DUAL (warn):** if a service emits *both* the deprecated and the canonical key on the same spans (double-emission), standardize on the canonical key at the SDK. An ingest mapping cannot merge them — the canonical key already exists, so the mapping would silently no-op. -- **Fix routes:** pure renames can be fixed either at the SDK (preferred) or by accepting the corresponding Recommendation Issue in Maple Settings → Ingestion (creates an ingest attribute mapping). Double-emission and naming issues can only be fixed at the SDK. +- **REN-DUAL (warn):** if a service emits *both* the deprecated and the canonical key on the same spans (double-emission), standardize on the canonical key at the SDK. An ingest mapping cannot merge them: the canonical key already exists, so the mapping would silently no-op. +- **Fix routes:** pure renames can be fixed either at the SDK (preferred) or by accepting the corresponding recommendation in Maple under Settings → Ingestion (creates an ingest attribute mapping). Double-emission and naming issues can only be fixed at the SDK. -## LOG — Logs +## LOG: Logs | Id | Check | Severity | Feature affected | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | --------------------------------------------------------------------------------------------------- | -| LOG-01 | An OTLP log bridge/handler is wired under the app's existing logger (OTel `LoggerProvider` + the language's bridge: `logs` SDK + winston/pino transport, Python `LoggingHandler`, Rust `tracing` layer). Stdout-only logging never reaches Maple. | critical | Logs don't exist in Maple at all — no log search, no trace-correlated logs. | -| LOG-02 | Logs emitted inside an active span carry `trace_id`/`span_id` (most bridges inject automatically — verify the bridge sits on the path the app actually logs through). | critical | Log↔trace correlation: jumping from a trace to its logs and back. | -| LOG-03 | Logs are structured — fields like `order_id`, `user_id` as attributes, not interpolated into the message string. | warn | Attribute chips, filtering, log pattern mining. | +| LOG-01 | An OTLP log bridge/handler is wired under the app's existing logger (OTel `LoggerProvider` + the language's bridge: `logs` SDK + winston/pino transport, Python `LoggingHandler`, Rust `tracing` layer). Stdout-only logging never reaches Maple. | critical | Logs don't exist in Maple at all: no log search, no trace-correlated logs. | +| LOG-02 | Logs emitted inside an active span carry `trace_id`/`span_id`. Most bridges inject them automatically; verify the bridge sits on the path the app actually logs through. | critical | Log↔trace correlation: jumping from a trace to its logs and back. | +| LOG-03 | Logs are structured: fields like `order_id`, `user_id` are attributes, not interpolated into the message string. | warn | Attribute chips, filtering, log pattern mining. | | LOG-04 | Severity uses the standard levels (`TRACE`/`DEBUG`/`INFO`/`WARN`/`ERROR`/`FATAL`). Error-level logs are reserved for unrecoverable conditions needing intervention. | warn | Severity row coloring; `ERROR`/`FATAL` drive the error banner. | Fix: `maple-onboarding-style` "Signals" + per-language style skill's log-bridge section. -## MET — Metrics +## MET: Metrics | Id | Check | Severity | Feature affected | | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------- | -| MET-01 | Metric labels are low-cardinality — no user IDs, order IDs, request IDs, raw URLs as dimensions (tenant-like bounded IDs are fine if the repo already treats them that way). | warn | High-cardinality labels explode series counts and slow every metrics query. | +| MET-01 | Metric labels are low-cardinality: no user IDs, order IDs, request IDs, or raw URLs as dimensions (tenant-like bounded IDs are fine if the repo already treats them that way). | warn | High-cardinality labels explode series counts and slow every metrics query. | | MET-02 | Business state transitions have counters (created/completed/failed/retried) and key operations have latency histograms. | info | Dashboards and alerts on business health, not just HTTP plumbing. | | MET-03 | Meters/instruments are created once at module scope, not per call. | info | Avoids churn and duplicate instrument registration. | Fix: `maple-onboarding-style` "Signals" / "Metrics" sections. -## NAME — Naming conventions +## NAME: Naming conventions | Id | Check | Severity | Feature affected | | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------- | -| NAME-01 | Attribute keys are lowercase `dotted.snake_case` and namespaced — no camelCase, no bare un-namespaced keys (covered keys: see REN-31…37). | info | Un-namespaced keys score lowest in the chip strip; inconsistent keys fragment queries. | -| NAME-02 | Span names are low-cardinality `domain.verb` (`order.process`, `payment.charge`) — never IDs or full URLs in the span name. | warn | High-cardinality span names break operation grouping and top-operations views. | -| NAME-03 | No `maple_*`-prefixed attributes — that namespace is reserved for Maple platform internals and hidden from the UI. | warn | Attributes silently hidden from chips. | +| NAME-01 | Attribute keys are lowercase `dotted.snake_case` and namespaced: no camelCase, no bare un-namespaced keys (covered keys: see REN-31…37). | info | Un-namespaced keys score lowest in the chip strip; inconsistent keys fragment queries. | +| NAME-02 | Span names are low-cardinality `domain.verb` (`order.process`, `payment.charge`). Never put IDs or full URLs in the span name. | warn | High-cardinality span names break operation grouping and top-operations views. | +| NAME-03 | No `maple_*`-prefixed attributes. That namespace is reserved for Maple platform internals and hidden from the UI. | warn | Attributes silently hidden from chips. | -## PII — Sensitive data +## PII: Sensitive data | Id | Check | Severity | Feature affected | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------- | -| PII-01 | No emails, passwords, tokens, API keys, auth headers, or full request/response bodies in span/log attributes. Grep attribute call sites for these value sources. | critical | Data risk — telemetry is broadly visible and retained. | +| PII-01 | No emails, passwords, tokens, API keys, auth headers, or full request/response bodies in span/log attributes. Grep attribute call sites for these value sources. | critical | Data risk: telemetry is broadly visible and retained. | -## LLM — gen_ai conventions +## LLM: gen_ai conventions Only applies when the project calls an LLM provider (OpenAI, Anthropic, Google, …). diff --git a/skills/maple-csharp-style/SKILL.md b/skills/maple-csharp-style/SKILL.md index 13129c4a8d..c487cf261c 100644 --- a/skills/maple-csharp-style/SKILL.md +++ b/skills/maple-csharp-style/SKILL.md @@ -1,11 +1,11 @@ --- name: maple-csharp-style -description: ".NET / C# OpenTelemetry style for Maple: OpenTelemetry.Extensions.Hosting + OTLP HTTP exporter, ActivitySource for spans, ILogger bridging via OpenTelemetryLoggerProvider, inline endpoint + ingest key." +description: ".NET / C# OpenTelemetry style for Maple: OpenTelemetry.Extensions.Hosting + OTLP HTTP exporter, ActivitySource for spans, ILogger bridging via WithLogging, inline endpoint + ingest key." --- # Maple .NET / C# style -Use the official `OpenTelemetry` packages and wire them through the .NET hosting model — `IServiceCollection` for traces / metrics, `ILoggingBuilder` for logs. +Use the official `OpenTelemetry` packages and wire them through the .NET hosting model. One `AddOpenTelemetry()` call on `IServiceCollection` configures traces, metrics, and `ILogger` export with a shared resource (`WithLogging` needs `OpenTelemetry.Extensions.Hosting` 1.9 or later). ## Install @@ -18,7 +18,7 @@ dotnet add package OpenTelemetry.Instrumentation.Http ## Bootstrap (ASP.NET Core) -Inline the endpoint and ingest key — they're a project-scoped, write-only token (Sentry-DSN-shaped). No env-var indirection. +Inline the endpoint and ingest key. The key is a project-scoped, write-only token (shaped like a Sentry DSN). No env-var indirection. With `HttpProtobuf`, an `Endpoint` set in code is used as-is, so include the `/v1/` path. ```csharp using OpenTelemetry; @@ -40,40 +40,38 @@ void ConfigureOtlp(OtlpExporterOptions options, string path) options.Headers = $"authorization=Bearer {MapleKey}"; } -var resource = ResourceBuilder.CreateDefault() - .AddService("orders-api") - .AddAttributes(new Dictionary - { - ["deployment.environment.name"] = builder.Environment.EnvironmentName, - ["vcs.repository.url.full"] = "https://github.com/acme/orders-api", - ["vcs.ref.head.revision"] = Environment.GetEnvironmentVariable("GITHUB_SHA") ?? "", - }); - builder.Services.AddOpenTelemetry() - .ConfigureResource(r => r.AddService("orders-api")) + .ConfigureResource(r => r + .AddService("orders-api") + .AddAttributes(new Dictionary + { + ["deployment.environment.name"] = builder.Environment.EnvironmentName, + ["vcs.repository.url.full"] = "https://github.com/acme/orders-api", + ["vcs.ref.head.revision"] = Environment.GetEnvironmentVariable("GITHUB_SHA") ?? "", + })) .WithTracing(tracing => tracing + .AddSource("orders.api") // every custom ActivitySource name, or its spans are dropped .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddOtlpExporter(o => ConfigureOtlp(o, "traces"))) .WithMetrics(metrics => metrics .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() - .AddOtlpExporter(o => ConfigureOtlp(o, "metrics"))); - -builder.Logging.AddOpenTelemetry(logging => -{ - logging.SetResourceBuilder(resource); - logging.IncludeFormattedMessage = true; - logging.IncludeScopes = true; - logging.AddOtlpExporter(o => ConfigureOtlp(o, "logs")); -}); + .AddOtlpExporter(o => ConfigureOtlp(o, "metrics"))) + .WithLogging( + logging => logging.AddOtlpExporter(o => ConfigureOtlp(o, "logs")), + options => + { + options.IncludeFormattedMessage = true; + options.IncludeScopes = true; + }); var app = builder.Build(); ``` ## Bounded business spans -Use a static `ActivitySource` per module. ASP.NET / HttpClient auto-instrumentation handles the obvious spans; reach for `ActivitySource.StartActivity` for business operations. +Use a static `ActivitySource` per module and register its name with `.AddSource(...)`. Unregistered sources produce no spans: `StartActivity` returns `null`. ASP.NET Core and HttpClient instrumentation cover request spans. Use `ActivitySource.StartActivity` for business operations. ```csharp public static class Tracing @@ -102,12 +100,12 @@ public sealed class OrderService } ``` -`Activity.RecordException(Exception)` lives in `OpenTelemetry.Trace` — `using OpenTelemetry.Trace;` for the extension. +`ActivitySource` and `ActivityStatusCode` live in `System.Diagnostics`. `RecordException(Exception)` is an extension method in `OpenTelemetry.Trace`, so add `using OpenTelemetry.Trace;`. ## Logs -`ILogger` calls already carry the active `Activity`'s `TraceId` / `SpanId` once `AddOpenTelemetry` is wired into `builder.Logging`. Don't replace the user's existing logger; add OTLP underneath. +Once `WithLogging` is wired, `ILogger` records carry the active `Activity`'s `TraceId` / `SpanId`. Do not replace the user's existing logger or providers; OTLP export runs alongside them. ## Coexistence -If the project already exports to Application Insights, Datadog, or Honeycomb, keep those. `AddOpenTelemetry().WithTracing(...)` accepts multiple exporters — chain `.AddOtlpExporter(...)` for Maple alongside the existing one. +If the project already exports to Application Insights, Datadog, or Honeycomb, keep those. `WithTracing(...)` accepts multiple exporters: chain `.AddOtlpExporter(...)` for Maple next to the existing one. diff --git a/skills/maple-dashboard-widgets/SKILL.md b/skills/maple-dashboard-widgets/SKILL.md index b59c8ed2c6..3657737279 100644 --- a/skills/maple-dashboard-widgets/SKILL.md +++ b/skills/maple-dashboard-widgets/SKILL.md @@ -7,8 +7,8 @@ description: "Build, repair, or review Maple dashboard widgets via the MCP. Trig # Maple dashboard widgets via MCP Everything below is generated from the live widget schema by -`bun run --cwd apps/ai mcp:docs`. **Do not edit this file by hand** — edit -`apps/api/src/mcp/lib/dashboard-schema-doc.ts` and regenerate. The same module backs the +`bun run --cwd apps/ai mcp:docs`. **Do not edit this file by hand.** Edit +`apps/ai/src/mcp/lib/dashboard-schema-doc.ts` and regenerate. The same module backs the `describe_dashboard_schema` MCP tool, so an agent at runtime and a reader here see one truth. ## When to use this skill @@ -18,7 +18,7 @@ When constructing widget JSON for `mcp__maple__create_dashboard`, `mcp__maple__replace_dashboard_widgets`. For a brand-new dashboard, prefer the simplified `widgets` array on `create_dashboard` -(`{ title, source, metric, group_by?, service_name?, unit? }`) — it fills in the traps below. +(`{ title, source, metric, group_by?, service_name?, unit? }`). It fills in the traps below. Reach for raw JSON when the simplified spec can't express what you need: multi-query charts, formulas, hidden series, non-default transforms. @@ -43,46 +43,51 @@ derived for you. The legacy `visualization` parameter is still accepted, but it line/bar/area into `chart` and then needs a `display.chartId` to tell them apart. The two columns below are what `panel_type` resolves to, and are what you write directly when authoring an assembled widget rather than calling `add_dashboard_widget`. A panel whose -`chartId` is `—` takes none: the `visualization` alone identifies it. +`chartId` is `-` takes none: the `visualization` alone identifies it. | panel_type | Label | `visualization` | `display.chartId` | Raw-SQL type | Default w×h | Requirements | |---|---|---|---|---|---|---| -| `line` | Line | `chart` | `query-builder-line` | `line` | 4×6 | — | -| `bar` | Bar | `chart` | `query-builder-bar` | `bar` | 4×6 | — | +| `line` | Line | `chart` | `query-builder-line` | `line` | 4×6 | - | +| `bar` | Bar | `chart` | `query-builder-bar` | `bar` | 4×6 | - | | `hbar` | Horizontal Bar | `hbar` | `query-builder-hbar` | `hbar` | 4×6 | needs a group-by | -| `area` | Area | `chart` | `query-builder-area` | `area` | 4×6 | — | +| `area` | Area | `chart` | `query-builder-area` | `area` | 4×6 | - | | `pie` | Pie | `pie` | `query-builder-pie` | `pie` | 4×6 | needs a group-by | -| `stat` | Stat | `stat` | — | `stat` | 3×4 | needs `transform.reduceToValue` | -| `gauge` | Gauge | `gauge` | — | `stat` | 4×6 | needs `transform.reduceToValue` | -| `table` | Table | `table` | — | `table` | 6×5 | — | -| `list` | List | `list` | — | — | 6×5 | **no raw-SQL support** | -| `histogram` | Histogram | `histogram` | `query-builder-histogram` | `histogram` | 4×6 | — | +| `stat` | Stat | `stat` | - | `stat` | 3×4 | needs `transform.reduceToValue` | +| `gauge` | Gauge | `gauge` | - | `stat` | 4×6 | needs `transform.reduceToValue` | +| `table` | Table | `table` | - | `table` | 6×5 | - | +| `list` | List | `list` | - | - | 6×5 | **no raw-SQL support** | +| `histogram` | Histogram | `histogram` | `query-builder-histogram` | `histogram` | 4×6 | - | | `heatmap` | Heatmap | `heatmap` | `query-builder-heatmap` | `heatmap` | 4×6 | needs a group-by | | `funnel` | Funnel | `funnel` | `query-builder-funnel` | `funnel` | 4×6 | needs a group-by | -| `markdown` | Note | `markdown` | — | — | 4×5 | **no raw-SQL support** | +| `paths` | Paths | `paths` | `query-builder-paths` | - | 6×5 | **no raw-SQL support** | +| `markdown` | Note | `markdown` | - | - | 4×5 | **no raw-SQL support** | ### Choosing one -- **line / area / bar** — a value over time. `area` and `bar` accept `display.stacked`; `line` does not. -- **hbar** — a ranked “top N by volume”. Each row is labelled with its share of the **total**. -- **funnel** — sequential stages with a drop-off. Labels each bar as a share of the *largest*, +- **line / area / bar**: a value over time. `area` and `bar` accept `display.stacked`; `line` does not. +- **hbar**: a ranked “top N by volume”. Each row is labelled with its share of the **total**. +- **funnel**: sequential stages with a drop-off. Labels each bar as a share of the *largest*, so an unranked breakdown of four equal things reads “100%” four times. Use `hbar` for that. -- **pie** — composition, few slices. Collapses a long tail into “Other”. -- **stat / gauge** — one number. A gauge adds an arc; set `display.gauge.min`/`max` to match the unit. -- **table** — rows and columns; set `display.columns` for headers and per-column units. -- **list** — recent traces/logs. Configured by `display.listDataSource`, never by SQL. -- **heatmap / histogram** — a distribution. A histogram over traces can bucket raw values client-side. -- **markdown** — a static note. Takes no query at all. + `display.funnel.variant: "dropoff"` draws the same funnel as one column per step with the + loss between steps, the median time between them, and where the leavers went. +- **paths**: what people do in the steps after (or before) one event or page, as a flow. + Defined by `display.paths` alone; no query set. +- **pie**: composition, few slices. Collapses a long tail into “Other”. +- **stat / gauge**: one number. A gauge adds an arc; set `display.gauge.min`/`max` to match the unit. +- **table**: rows and columns; set `display.columns` for headers and per-column units. +- **list**: recent traces/logs. Configured by `display.listDataSource`, never by SQL. +- **heatmap / histogram**: a distribution. A histogram over traces can bucket raw values client-side. +- **markdown**: a static note. Takes no query at all. ## Data sources A widget's `dataSource` is a discriminated union over `kind`: `query`, `raw_sql`, `route`, `static`. Every arm requires its `kind`. -> **If you have seen `{ "endpoint": …, "params": … }` anywhere — that is the retired v2 shape +> **If you have seen `{ "endpoint": …, "params": … }` anywhere, that is the retired v2 shape > and it will not decode.** A `query` source spreads `queries`/`formulas` at the TOP LEVEL, > not under `params`, and requires `resultShape`. -### `kind: "query"` — the query builder +### `kind: "query"`: the query builder `resultShape` is required and is one of `timeseries` (a value over time), `breakdown` (one row per group) or `list` (raw rows). Optional: `formulas`, `comparison`, `limit`, @@ -121,7 +126,7 @@ A widget's `dataSource` is a discriminated union over `kind`: `query`, `raw_sql` } ``` -### `kind: "raw_sql"` — your own ClickHouse SQL +### `kind: "raw_sql"`: your own ClickHouse SQL ```json { @@ -131,7 +136,7 @@ A widget's `dataSource` is a discriminated union over `kind`: `query`, `raw_sql` } ``` -### `kind: "static"` — a markdown note, no request +### `kind: "static"`: a markdown note, no request ```json { @@ -139,7 +144,7 @@ A widget's `dataSource` is a discriminated union over `kind`: `query`, `raw_sql` } ``` -### `kind: "route"` — a curated built-in panel +### `kind: "route"`: a curated built-in panel `{ "kind": "route", "endpoint": "service_overview", "params": { … } }`. These back the prebuilt panels; you rarely author one by hand. @@ -149,7 +154,7 @@ prebuilt panels; you rarely author one by hand. A `stat` or `gauge` reads `data[0].value`. Without `transform.reduceToValue` it renders `[object Object]`. `add_dashboard_widget` injects `{ field: "value", aggregate: "first" }` when you omit it; set it explicitly to choose a different reducer. Valid aggregates: -`sum`, `first`, `count`, `avg`, `max`, `min` — **there is no `last`**. +`sum`, `first`, `count`, `avg`, `max`, `min`: **there is no `last`**. Which one depends on what the query returns, because the query is bucketed over time and the reducer collapses those buckets into one number: @@ -157,7 +162,7 @@ the reducer collapses those buckets into one number: - **A rate or a count** (`count`, a metrics `rate`) → `sum`, for a window total. - **A latency percentile or an average** (`p95_duration`, `avg_duration`, a gauge metric) → `avg` for the typical value over the window, or `max` for the worst bucket. **Not** - `sum` — adding percentiles together is meaningless, and it is the common wrong choice. + `sum`: adding percentiles together is meaningless, and it is the common wrong choice. - **A current reading**, where only the newest bucket matters → `first`. ```json @@ -199,7 +204,7 @@ the reducer collapses those buckets into one number: ### The breakdown shape -`resultShape: "breakdown"` returns one row per group instead of a series over time — the +`resultShape: "breakdown"` returns one row per group instead of a series over time, the shape `pie`, `hbar`, `funnel` and `heatmap` need. It requires a group-by, and `limit` caps the rows (honoured for 1–100). @@ -237,11 +242,130 @@ the rows (honoured for 1–100). } ``` +### Product-event funnels (`panel_type: "funnel"` + `display.funnel.steps`) + +A funnel widget has two modes. Without `display.funnel.steps` it draws a group-by breakdown +as descending stages (the shape above). With them it is a **conversion funnel over product +events** (page views, `track()` events and server-side events, stitched per person), and +the query set is not used at all. Set the definition on `display_json.funnel` and +`add_dashboard_widget` derives the data source (`kind: "route"`, +`endpoint: "product_events_funnel"`) for you; do not pass `data_source_json`. + +- `steps`: 1–10, in order. `{ kind: "event", eventName, attributeEquals? }`, + `{ kind: "page", pagePath, host? }`, or (**step 1 only**) + `{ kind: "session", dimension, value }` with `dimension` one of `referrerHost`, + `utmSource`, `utmMedium`, `utmCampaign`, `country`, `host`. +- `keyBy`: `person` (default; user id, else the visitor's linked user, else the visitor), + `visitor`, `user`, or `session`. +- `windowSeconds`: the whole chain must complete within this many seconds of step 1 + (default 86400). Must be positive. +- `breakdownBy`: split the funnel by a session dimension (`referrerHost`, `utmSource`, + `utmMedium`, `utmCampaign`, `country`, `host`) or `attribute:`; the widget draws one + bar per group per step for the top 6 groups by step-1 count. +- `filters`: the population filter: only persons with a session matching these + dimensions take part. `{ host?, pagePath?, referrerHost?, country?, deviceType?, + browserName?, osName?, language?, utmSource?, utmMedium?, utmCampaign?, + visitorType?: "new" | "returning" }`. They are spread FLAT into the derived route params. + +The step count, the step-1-only session rule and a positive `windowSeconds` are enforced on +write: a definition that breaks one of them is rejected rather than saved, because the +query engine would reject it again on every render. + +Use `list_product_events` to see which event names exist, and `query_funnel` to try a +definition before pinning it to a board. + +```json +{ + "id": "w-signup-funnel", + "visualization": "funnel", + "dataSource": { + "kind": "route", + "endpoint": "product_events_funnel", + "params": { + "steps": [ + { + "kind": "page", + "pagePath": "/pricing" + }, + { + "kind": "event", + "eventName": "signup_completed" + }, + { + "kind": "event", + "eventName": "plan_started", + "attributeEquals": { + "plan": "pro" + } + } + ], + "keyBy": "person", + "windowSeconds": 604800 + } + }, + "display": { + "title": "Signup funnel", + "chartId": "query-builder-funnel", + "funnel": { + "showStepPercent": true, + "steps": [ + { + "kind": "page", + "pagePath": "/pricing" + }, + { + "kind": "event", + "eventName": "signup_completed" + }, + { + "kind": "event", + "eventName": "plan_started", + "attributeEquals": { + "plan": "pro" + } + } + ], + "keyBy": "person", + "windowSeconds": 604800 + } + }, + "layout": { + "x": 0, + "y": 0, + "w": 6, + "h": 4 + } +} +``` + +`display.funnel.variant: "dropoff"` switches the same widget to the drop-off view, which +also fetches the time between steps and the most common next event of the persons who +stopped before each step. Needs at least two steps to say anything; with `breakdownBy` it +draws one cap per group and no timing or leavers. + +### Paths (`panel_type: "paths"` + `display.paths`) + +A flow of what people did in the hops after (or, with `direction: "before"`, before) +one anchor. Product events only; there is no query set. Set the definition on +`display_json.paths` and `add_dashboard_widget` derives the data source (`kind: "route"`, +`endpoint: "product_events_paths"`); do not pass `data_source_json`. + +- `anchor`: required. `{ kind: "event", eventName, attributeEquals? }` or + `{ kind: "page", pagePath, host? }`. A session step cannot anchor a sequence. +- `direction`: `"after"` (default) or `"before"`. +- `depth`: hops to draw, 1–5 (default 3). `branches`: nodes named per step, 1–10 + (default 4); the rest fold into an *Other* node. A sequence that stops short ends in + an *Ended* node. +- `keyBy`, `windowSeconds` (how far from the anchor a hop may be, default 86400) and + `filters`: as for funnels. +- `include`: `"all"` (default), `"events"` or `"pages"`. `exclude`: names (event names + or page paths) dropped before sequencing, e.g. `["heartbeat", "/"]`. + ### A complete widget The sections above describe `add_dashboard_widget`'s parameters, which it assembles into a widget for you. `update_dashboard_widget`, `replace_dashboard_widgets` and `dashboard_json` -take the assembled object instead — this is its shape. `timeRange` and `sectionId`/`tabId` +take the assembled object instead. This is its shape. `timeRange` and `sectionId`/`tabId` are the only other top-level keys, both optional. ```json @@ -332,19 +456,19 @@ A gauge's arc is independent of its unit and defaults to 0–100: on a `percent` ## Queries -A query draft is discriminated on `dataSource` (`traces` / `logs` / `metrics`). The +A query draft is discriminated on `dataSource` (`traces` / `logs` / `metrics` / `product_events`). The metric-only fields belong solely to `metrics` queries; do not add them to trace or log queries: -- `metricName` — required; discover real names with `list_metrics`. -- `metricType` — required, one of `sum`, `gauge`, `histogram`, `exponential_histogram`. Anything else fails to decode. -- `signalSource` — optional, one of `default`, `meter`. Omit it unless you know you need `meter`. -- `isMonotonic` — optional; `false` marks a Sum as an UpDownCounter, which changes the +- `metricName`: required; discover real names with `list_metrics`. +- `metricType`: required, one of `sum`, `gauge`, `histogram`, `exponential_histogram`. Anything else fails to decode. +- `signalSource`: optional, one of `default`, `meter`. Omit it unless you know you need `meter`. +- `isMonotonic`: optional; `false` marks a Sum as an UpDownCounter, which changes the aggregations that make sense (`rate`/`increase` assume a monotonic counter). ### `addOns` is required, and all five keys must be present -`addOns: { groupBy, having, orderBy, limit, legend }` — every key, every time. A missing +`addOns: { groupBy, having, orderBy, limit, legend }`: every key, every time. A missing one fails to decode. Each flag gates whether the matching field is read at all, which is why `groupBy` without `addOns.groupBy: true` silently does nothing. @@ -355,10 +479,11 @@ why `groupBy` without `addOns.groupBy: true` silently does nothing. | `traces` | `count`, `avg_duration`, `p50_duration`, `p95_duration`, `p99_duration`, `error_rate` | | `logs` | `count` | | `metrics` | `avg`, `sum`, `min`, `max`, `count`, `rate`, `increase` | +| `product_events` | `count`, `sessions`, `persons`, `users`, `visitors` | On traces only, setting `valueField: "attr."` switches the query to numeric-attribute mode, where the aggregation is one of `avg`, `sum`, `min`, `max`, `p50`, `p95`, `p99`. -This is the **only** place a bare `p50`/`p95`/`p99` is valid — latency percentiles are +This is the **only** place a bare `p50`/`p95`/`p99` is valid; latency percentiles are spelled `p95_duration`. Metrics never accept percentiles. ### Group-by tokens @@ -371,15 +496,16 @@ silent failure: the array is present, the chart shows an ungrouped total, and no | `traces` | `service`, `service.name`, `service_name`, `span`, `span.name`, `span_name`, `status`, `status.code`, `status_code`, `http.method`, `none`, `all` | `attr.` | | `logs` | `service`, `service.name`, `service_name`, `severity`, `severity_text`, `none`, `all` | **none** | | `metrics` | `service`, `service.name`, `none`, `all` | `attr.`, `resource.` | +| `product_events` | `event`, `event.name`, `event_name`, `kind`, `event.kind`, `source`, `host`, `page`, `page.path`, `page_path`, `service`, `service.name`, `service_name`, `group`, `group.id`, `none`, `all` | `attr.` | Anything outside the literal list must use a supported prefix; unrecognised tokens are dropped, which makes the write tools reject the widget rather than save a mis-scoped chart. ### `whereClause` is a custom grammar, not SQL -Operators — the only ones: `=`, `!=`, `>`, `<`, `>=`, `<=`, `contains`, `!contains`, +Operators (the only ones): `=`, `!=`, `>`, `<`, `>=`, `<=`, `contains`, `!contains`, `exists`, `!exists`. Clauses join with ` AND `; there is no `OR` and no parentheses. -Values use double quotes. **There is no `IS NULL` / `IS NOT NULL`** — write ` exists` +Values use double quotes. **There is no `IS NULL` / `IS NOT NULL`**: write ` exists` or ` !exists`. `exists` means present *and* non-empty, because attributes live in ClickHouse `Map` columns where a missing key reads back as `''`. @@ -392,7 +518,7 @@ a span attribute, so `db.system = "clickhouse"` works directly. Cap: 5 `attr.*` `formulas: [{ id, name, expression, legend }]` references queries by `name` (`A / B`), and is valid on the **timeseries** shape only. Marking a query `hidden: true` is UI-only in raw -JSON — also add `transform.hideSeries.baseNames: ["A"]` or the auxiliary series renders at +JSON, so also add `transform.hideSeries.baseNames: ["A"]` or the auxiliary series renders at full scale and flattens the axis. ## Display config @@ -400,7 +526,7 @@ full scale and flattens the axis. | Key | Applies to | Notes | |---|---|---| | `title`, `description` | all | | -| `unit` | all | See the `units` section — read it before choosing a percent token. | +| `unit` | all | See the `units` section: read it before choosing a percent token. | | `thresholds` | stat, gauge, charts | `[{ value, color, label? }]`; highest matching value wins. | | `prefix`, `suffix` | stat, gauge | Wrap the formatted value. | | `chartPresentation.legend` | charts | `visible` \| `hidden` \| `right`. | @@ -413,11 +539,12 @@ full scale and flattens the axis. | `columns` | table, list | `[{ field, header, unit?, width?, align?, hidden? }]`. | | `listDataSource`, `listWhereClause`, `listLimit`, `listRootOnly` | list | | | `pie` | pie | `{ donut, innerRadius, showLabels, showPercent }`. | -| `gauge` | gauge | `{ min, max }` — defaults to 0–100, which is wrong for a `percent` unit. | +| `gauge` | gauge | `{ min, max }`: defaults to 0–100, which is wrong for a `percent` unit. | | `histogram` | histogram | `{ bucketCount, bucketWidth, logScaleY }`. | | `heatmap` | heatmap | `{ colorScale, scaleType }`. | -| `funnel` | funnel | `{ showStepPercent }`. | -| `markdown` | markdown | `{ content }` — the note body. | +| `funnel` | funnel | `{ showStepPercent, variant?, steps?, keyBy?, windowSeconds?, breakdownBy?, filters? }`: with `steps` it is a product-event funnel (see Data sources); `variant: "dropoff"` is the drop-off view. | +| `paths` | paths | `{ anchor, direction?, depth?, branches?, keyBy?, windowSeconds?, include?, exclude?, filters? }`: the whole definition (see Data sources). | +| `markdown` | markdown | `{ content }`: the note body. | | `sparkline` | stat | `{ enabled, dataSource? }`; embeds a full nested data source. | ### Stored but not rendered @@ -439,7 +566,7 @@ existing override. ## Raw SQL widgets Pass `sql` to `add_dashboard_widget` and the tool builds the data source for you. -**Call `describe_warehouse_tables` first** — a hallucinated table or column silently +**Call `describe_warehouse_tables` first**: a hallucinated table or column silently produces an empty chart. ### Macros @@ -451,32 +578,32 @@ produces an empty chart. ### Conventions that catch everyone -- Columns are PascalCase (`ServiceName`, `Timestamp`) — never snake_case. +- Columns are PascalCase (`ServiceName`, `Timestamp`), never snake_case. - `StatusCode` / `SeverityText` / `SpanKind` values are **Title Case** (`'Error'`, not `'ERROR'`). Wrong casing runs fine and matches zero rows. - Span `Duration` is **nanoseconds**. Divide by `1e6` for ms. -- `SpanAttributes['key']` — square brackets. A missing key returns `''`, not NULL. +- `SpanAttributes['key']`: square brackets. A missing key returns `''`, not NULL. - One statement only; writes are rejected; every query is wrapped in `LIMIT 1001`. ### What to SELECT, per panel type The renderer is opinionated. Wrong aliases give an empty chart or `[object Object]`. -- **line / area / bar** — a DateTime bucket as the FIRST column (alias `bucket`) plus one or +- **line / area / bar**: a DateTime bucket as the FIRST column (alias `bucket`) plus one or more **numeric** columns; each becomes a series named after the column. **String columns - are dropped**, so multi-series must be pivoted in SQL with `countIf(...)` — tall form + are dropped**, so multi-series must be pivoted in SQL with `countIf(...)`; tall form (`bucket, ServiceName, count()`) collapses to one aggregate line. -- **stat / gauge** — one scalar aliased `value`. -- **pie / funnel / hbar** — a string column aliased `name` plus a numeric column. Cap at ~8–10 rows. -- **heatmap** — three columns aliased `x`, `y`, `value`; string-cast numeric `x`/`y`. -- **histogram** — one numeric column aliased `value`, one row per observation; add `LIMIT 5000`. -- **table** — any rows; columns render in order, so use `AS` for readable headers. -- **list** — not supported. A list is configured by `display.listDataSource`. +- **stat / gauge**: one scalar aliased `value`. +- **pie / funnel / hbar**: a string column aliased `name` plus a numeric column. Cap at ~8–10 rows. +- **heatmap**: three columns aliased `x`, `y`, `value`; string-cast numeric `x`/`y`. +- **histogram**: one numeric column aliased `value`, one row per observation; add `LIMIT 5000`. +- **table**: any rows; columns render in order, so use `AS` for readable headers. +- **list**: not supported. A list is configured by `display.listDataSource`. ```sql SELECT toStartOfInterval(Timestamp, INTERVAL $__interval_s SECOND) AS bucket, - countIf(SeverityText = 'Error') AS Error, - countIf(SeverityText = 'Warn') AS Warn + countIf(SeverityNumber BETWEEN 17 AND 20) AS Error, + countIf(SeverityNumber BETWEEN 13 AND 16) AS Warn FROM logs WHERE $__orgFilter AND $__timeFilter(Timestamp) GROUP BY bucket @@ -485,4 +612,4 @@ ORDER BY bucket `granularity_seconds` only matters if the SQL references `$__interval_s`. Either use `toStartOfInterval(…, INTERVAL $__interval_s SECOND)` with it, or a fixed `toStartOf*` -without it — mixing them means the setting silently does nothing. +without it; mixing them means the setting silently does nothing. diff --git a/skills/maple-effect-style/SKILL.md b/skills/maple-effect-style/SKILL.md index 37e229befe..97a04feae0 100644 --- a/skills/maple-effect-style/SKILL.md +++ b/skills/maple-effect-style/SKILL.md @@ -5,7 +5,7 @@ description: "Effect-TS OpenTelemetry style for Maple via @maple-dev/effect-sdk: # Maple Effect style -For Effect apps, use `@maple-dev/effect-sdk` — Maple's first-class Effect SDK. It wraps Effect's built-in `Otlp.layerJson` exporter and handles batching, shutdown, and resource attributes for you. +For Effect apps, use `@maple-dev/effect-sdk`. It wraps Effect's built-in `Otlp.layerJson` exporter and handles batching, shutdown, and resource attributes. ## Install @@ -13,11 +13,11 @@ For Effect apps, use `@maple-dev/effect-sdk` — Maple's first-class Effect SDK. npm install @maple-dev/effect-sdk effect ``` -For Effect 3, use `@maple-dev/effect-sdk@effect-v3` instead. Same API, different peer ranges. +The current release requires Effect 4 (`effect >= 4.0.0-rc.113`). For Effect 3, install `@maple-dev/effect-sdk@effect-v3 effect @effect/platform @effect/opentelemetry`. The API and import paths are the same. ## Bootstrap -Pick the entry point per runtime — they have different lifecycle requirements: +Pick the entry point per runtime. Each has different lifecycle requirements: - **Server (Node.js, Bun, Deno):** background-export fiber, env-var auto-detection, graceful shutdown. - **Browser:** explicit config (no env vars), browser metadata baked in. @@ -27,15 +27,13 @@ Pick the entry point per runtime — they have different lifecycle requirements: ```ts import { Maple } from "@maple-dev/effect-sdk" -import { Effect, Layer } from "effect" +import { Effect } from "effect" const TracerLive = Maple.layer({ serviceName: "orders-api", endpoint: "https://ingest.maple.dev", ingestKey: "MAPLE_TEST", // set by maple-onboard skill on pairing - attributes: { - "vcs.repository.url.full": "https://github.com/acme/orders-api", - }, + repositoryUrl: "https://github.com/acme/orders-api", }) const program = Effect.gen(function* () { @@ -47,13 +45,13 @@ Effect.runPromise(program.pipe(Effect.provide(TracerLive))) The default import resolves to the server build under Node.js. Import `@maple-dev/effect-sdk/server` explicitly when needed. -If `endpoint` is omitted, the server layer auto-detects it from `MAPLE_ENDPOINT` (falling back to `OTEL_EXPORTER_OTLP_ENDPOINT`, then the public Maple ingest). `Maple.layer` always exports — a missing ingest key does not disable it, which is what keeps keyless local-mode and self-hosted-collector setups working. Inline the key when you want telemetry to flow regardless of env (matches the rest of the maple-onboard inline-key pattern). `MapleFlush.make` and the Cloudflare `make()` differ: those no-op without a key. +If `endpoint` is omitted, the server layer reads `MAPLE_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then falls back to the public ingest for the region (`https://ingest.maple.dev`, or `https://ingest.eu.maple.dev` with `region: "eu"` / `MAPLE_REGION=eu`). The key falls back to `MAPLE_INGEST_KEY`. `Maple.layer` always exports. A missing ingest key does not disable it, so keyless local-mode and self-hosted-collector setups keep working. Inline the key when telemetry must flow regardless of env (the maple-onboard inline-key pattern). `MapleFlush.make` and the Cloudflare `make()` differ: they no-op without a key. -The server layer also auto-fills `vcs.ref.head.revision` from `COMMIT_SHA` / `RAILWAY_GIT_COMMIT_SHA` / `VERCEL_GIT_COMMIT_SHA` / `CF_PAGES_COMMIT_SHA` / `RENDER_GIT_COMMIT` (first match wins). For `vcs.repository.url.full`, use the `repositoryUrl` option or `MAPLE_REPOSITORY_URL` — don't hand-write the attribute. +The server layer also auto-fills `vcs.ref.head.revision` from `COMMIT_SHA` / `RAILWAY_GIT_COMMIT_SHA` / `VERCEL_GIT_COMMIT_SHA` / `CF_PAGES_COMMIT_SHA` / `RENDER_GIT_COMMIT` (first match wins). For `vcs.repository.url.full`, use the `repositoryUrl` option or `MAPLE_REPOSITORY_URL`. Do not hand-write the attribute. The layer also dual-emits `deployment.environment` and `deployment.environment.name` from the `environment` option or `MAPLE_ENVIRONMENT`. ### Cloudflare Workers -The Cloudflare entry point exports `make()`, not a `Maple` namespace. Build the telemetry object **once at module scope** — it buffers in-isolate and resolves `env` lazily on the first flush: +The Cloudflare entry point exports `make()`, not a `Maple` namespace. Build the telemetry object **once at module scope**. It buffers in-isolate and resolves `env` lazily on the first flush: ```ts import * as MapleCloudflareSDK from "@maple-dev/effect-sdk/cloudflare" @@ -79,7 +77,7 @@ export default { } ``` -The Cloudflare entry point requires `ctx.waitUntil(telemetry.flush(env))` so telemetry survives the isolate exit — note `flush` takes `env`. Forgetting the `waitUntil` is the most common reason traces don't show up from Workers. +Call `ctx.waitUntil(telemetry.flush(env))` on every request so telemetry survives the isolate exit. `flush` takes `env`. A missing `waitUntil` is the most common reason Worker traces never arrive. When routes go through `HttpRouter.toWebHandler`, provide `telemetry.layer` to the layer you pass it (`Layer.provideMerge(telemetry.layer)`), not to a separate per-request runtime. ### Browser @@ -93,31 +91,30 @@ const TracerLive = Maple.layer({ }) ``` -No env-var fallback in the browser entry point — config is always explicit. +The browser entry point has no env-var fallback. Pass all config explicitly (add `region: "eu"` for EU organizations). It records session replays by default; opt out with `replay: { enabled: false }`. ## Custom spans -Use `Effect.withSpan` to trace operations and `Effect.annotateCurrentSpan` for attributes — don't reach for the raw `@opentelemetry/api` tracer when an Effect-native primitive is available. +Use `Effect.withSpan` to trace operations and `Effect.annotateCurrentSpan` for attributes. Do not reach for the raw `@opentelemetry/api` tracer when an Effect-native primitive exists. ```ts const processOrder = (orderId: string) => Effect.gen(function* () { yield* Effect.annotateCurrentSpan("order.id", orderId) - yield* Effect.annotateCurrentSpan("peer.service", "payment-api") const result = yield* chargePayment(orderId) return result }).pipe(Effect.withSpan("order.process")) ``` -Setting `peer.service` on outgoing calls makes them visible on Maple's service map. +Maple's service map draws a service-to-service edge by joining a Client span to the downstream service's child Server span. `peer.service` does not draw edges. Make outgoing calls through Effect's `HttpClient`: it creates the Client span and injects `traceparent` by default. Calls to uninstrumented dependencies appear as external nodes keyed on `server.address`. -`Effect.fail` and uncaught defects are recorded as exceptions and set the span status to ERROR automatically — you don't need to wrap with `try` / `catch` / `finally`. +`Effect.fail` and uncaught defects end the span with status `Error` and an `exception` event. Do not wrap with `try` / `catch` / `finally`. -`@maple/otel-helpers` `withSpan` is for non-Effect TypeScript code; in Effect code prefer the Effect-native span primitives. +`@maple/otel-helpers` `withSpan` is for non-Effect TypeScript code. In Effect code, use the Effect-native span primitives. ## Logs -`Effect.log` automatically includes trace context when called inside a span — no additional setup needed: +`Effect.log` inside a span carries the trace context. No extra setup is needed: ```ts const program = Effect.gen(function* () { @@ -131,13 +128,4 @@ Logs emitted inside spans are correlated with the active trace in the Maple dash ## Coexistence -If the project already uses `@effect/opentelemetry` or `Otlp.layerJson` with a custom exporter (e.g. for Honeycomb, Datadog), keep it. `Maple.layer()` can compose alongside via `Layer.merge`: - -```ts -const TracerLive = Layer.merge( - Maple.layer({ serviceName: "api", endpoint: "https://ingest.maple.dev", ingestKey: "MAPLE_TEST" }), - HoneycombLayer, -) -``` - -Both vendors receive the same spans. +If the project already exports through `@effect/opentelemetry` or `Otlp.layerJson` to another vendor (Honeycomb, Datadog), keep that exporter. Do not `Layer.merge` `Maple.layer()` with another tracer layer. Effect runs one `Tracer` service, so the last layer wins and the other vendor stops receiving spans. To ship to both, add Maple as a second destination inside the existing setup: a second OTLP span processor on the `@effect/opentelemetry` SDK, or a fan-out in the team's OpenTelemetry Collector. Ask the user before replacing the incumbent exporter. diff --git a/skills/maple-go-style/SKILL.md b/skills/maple-go-style/SKILL.md index 71fbc1f331..d5b7f0df4c 100644 --- a/skills/maple-go-style/SKILL.md +++ b/skills/maple-go-style/SKILL.md @@ -14,7 +14,7 @@ go get \ go.opentelemetry.io/otel \ go.opentelemetry.io/otel/sdk \ go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \ - go.opentelemetry.io/otel/exporters/otlp/otlplogs/otlploghttp \ + go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp \ go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp \ go.opentelemetry.io/otel/log \ go.opentelemetry.io/otel/sdk/log @@ -22,7 +22,7 @@ go get \ ## Bootstrap -Inline the endpoint and ingest key — they're a project-scoped, write-only token (Sentry-DSN-shaped). No env-var indirection. +Inline the endpoint and ingest key. The key is a project-scoped, write-only token (shaped like a Sentry DSN). No env-var indirection. `WithEndpoint` takes a host without scheme, uses HTTPS, and appends the default `/v1/` path. ```go package telemetry @@ -33,7 +33,7 @@ import ( "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" - "go.opentelemetry.io/otel/exporters/otlp/otlplogs/otlploghttp" + "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/log/global" @@ -132,7 +132,7 @@ func main() { ## Bounded business spans -Acquire the tracer at package scope, start spans where auto-instrumentation is blind, set the status on error, end the span via `defer`. +Acquire the tracer at package scope. Start spans where auto-instrumentation is blind, set the status on error, and end the span via `defer` (imports: `go.opentelemetry.io/otel/attribute`, `go.opentelemetry.io/otel/codes`). ```go var tracer = otel.Tracer("orders.api") @@ -154,8 +154,8 @@ func SubmitOrder(ctx context.Context, orderID string) (err error) { ## Auto-instrumentation -Go does not have framework-level auto-discovery. Add the official contrib packages for the libraries the app actually uses (`otelhttp`, `otelgrpc`, `otelsql`, `otelpgx`, `otelmux`, `otelfiber`, etc.). Don't add packages the app doesn't import. +Go has no framework-level auto-discovery. Add instrumentation packages only for libraries the app imports: `otelhttp`, `otelgrpc`, `otelmux` from `go.opentelemetry.io/contrib/instrumentation/...`, and community packages such as `github.com/XSAM/otelsql`, `github.com/exaring/otelpgx`, and Fiber's `otelfiber` (in `github.com/gofiber/contrib`). Wrap outgoing HTTP clients with `otelhttp.NewTransport` so trace context propagates; Maple's service map draws edges from a Client span to the callee's child Server span. ## Coexistence -If the project already exports to Honeycomb / Datadog / Tempo, install Maple's exporter alongside via `trace.WithBatcher(traceExp)` for each backend. Don't strip the existing exporter unless the user asks. +If the project already exports to Honeycomb, Datadog, or Tempo, add Maple's exporter alongside with one `trace.WithBatcher(exp)` option per backend. Do not strip the existing exporter unless the user asks. diff --git a/skills/maple-java-style/SKILL.md b/skills/maple-java-style/SKILL.md index 505c5184b5..267f90addd 100644 --- a/skills/maple-java-style/SKILL.md +++ b/skills/maple-java-style/SKILL.md @@ -5,7 +5,7 @@ description: "Java OpenTelemetry style for Maple: zero-code Java agent or manual # Maple Java style -The fastest path is the OpenTelemetry Java Agent — it auto-instruments the JVM with zero code changes. +The fastest path is the OpenTelemetry Java agent. It auto-instruments the JVM with no code changes. ## Zero-code: Java agent @@ -13,7 +13,7 @@ The fastest path is the OpenTelemetry Java Agent — it auto-instruments the JVM curl -sLO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar ``` -Inline the endpoint and ingest key as JVM args (these are the agent's only configuration surface — they map straight onto the inline-key model): +Inline the endpoint and ingest key as JVM system properties. The agent also reads `OTEL_*` env vars, but inline `-D` flags fit the inline-key model. The agent appends `/v1/traces`, `/v1/logs`, and `/v1/metrics` to the base endpoint: ```bash java \ @@ -26,13 +26,13 @@ java \ -jar build/libs/app.jar ``` -Replace `MAPLE_TEST` with the project's real Maple ingest key once it's available. Keep the args inline (in `Procfile` / `Dockerfile` / `systemd` unit / `application.yml`) — don't move them behind unset env vars. +Replace `MAPLE_TEST` with the project's real Maple ingest key once it exists. Keep the flags inline where the JVM is launched (`Procfile`, `Dockerfile`, `systemd` unit, or `JAVA_TOOL_OPTIONS`). Do not move them behind unset env vars. The agent does not read Spring's `application.yml`. The agent auto-instruments Spring (Boot, MVC, WebFlux), Servlet containers, Apache HttpClient, OkHttp, JDBC, R2DBC, Hibernate, Kafka, gRPC, AWS SDK, and many more. ## Manual SDK (when the agent isn't an option) -For cases where the agent can't run (GraalVM native image, embedded JVM, sealed module path), use the SDK directly: +Where the agent can't run (GraalVM native image, embedded JVM, sealed module path), use the SDK directly. Import `io.opentelemetry:opentelemetry-bom` and `io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha` in `` so the artifacts below resolve without explicit versions: ```xml @@ -61,8 +61,8 @@ public final class Telemetry { public static OpenTelemetrySdk init() { var headers = Map.of("authorization", "Bearer " + MAPLE_KEY); var resource = Resource.getDefault().merge(Resource.create(Attributes.builder() - .put(ServiceAttributes.SERVICE_NAME, "orders-api") - .put(DeploymentIncubatingAttributes.DEPLOYMENT_ENVIRONMENT_NAME, + .put("service.name", "orders-api") + .put("deployment.environment.name", System.getenv().getOrDefault("DEPLOYMENT_ENV", "development")) .put("vcs.repository.url.full", "https://github.com/acme/orders-api") .put("vcs.ref.head.revision", System.getenv().getOrDefault("GITHUB_SHA", "")) @@ -74,23 +74,25 @@ public final class Telemetry { .build(); // … same shape for OtlpHttpLogRecordExporter and OtlpHttpMetricExporter - return OpenTelemetrySdk.builder() + var sdk = OpenTelemetrySdk.builder() .setTracerProvider(SdkTracerProvider.builder() .addSpanProcessor(BatchSpanProcessor.builder(spanExporter).build()) .setResource(resource) .build()) .buildAndRegisterGlobal(); + OpenTelemetryAppender.install(sdk); // Logback bridge, see Logs + return sdk; } } ``` ## Logs -Bridge the existing Logback / SLF4J / Log4j2 setup through OTLP — don't replace it. With the Java agent, log appenders are auto-bridged. With the manual SDK, add the `opentelemetry-logback-appender-1.0` (or the Log4j 2 equivalent) and configure it in `logback.xml` so existing logger calls carry `trace_id` / `span_id` and reach Maple. +Bridge the existing Logback / SLF4J / Log4j2 setup through OTLP. Do not replace it. With the Java agent, Logback and Log4j2 are bridged automatically. With the manual SDK, add `opentelemetry-logback-appender-1.0` (or `opentelemetry-log4j-appender-2.17`), declare `io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender` in `logback.xml`, and call `OpenTelemetryAppender.install(sdk)` after building the SDK. Existing logger calls then carry `trace_id` / `span_id` and reach Maple. ## Bounded business spans -Acquire the tracer at class scope; wrap operations the agent's auto-instrumentation can't see. +Acquire the tracer at class scope. Wrap operations the agent's auto-instrumentation can't see. ```java private static final Tracer TRACER = GlobalOpenTelemetry.getTracer("orders.api"); @@ -114,4 +116,4 @@ public Order submit(String orderId, String tenantId) { ## Coexistence -If the project already runs Datadog / New Relic / Honeycomb agents, leave them in place. The OpenTelemetry Java Agent coexists with most APMs (test the combination once before shipping). Don't strip an incumbent agent unless the user asks. +If the project already runs a Datadog, New Relic, or Honeycomb agent, leave it in place. Two bytecode agents on one JVM can conflict, so test the combination once before shipping. Do not strip an incumbent agent unless the user asks. diff --git a/skills/maple-kotlin-style/SKILL.md b/skills/maple-kotlin-style/SKILL.md index 227b826a75..0e42932c72 100644 --- a/skills/maple-kotlin-style/SKILL.md +++ b/skills/maple-kotlin-style/SKILL.md @@ -22,9 +22,9 @@ java \ -jar build/libs/app.jar ``` -Replace `MAPLE_TEST` with the project's real Maple ingest key once available. Keep these flags inline (in `Procfile` / `Dockerfile` / `application.yml`) — don't move them behind unset env vars. +Replace `MAPLE_TEST` with the project's real Maple ingest key once it exists. Keep these flags inline where the JVM is launched (`Procfile`, `Dockerfile`, or `JAVA_TOOL_OPTIONS`). Do not move them behind unset env vars. The agent does not read `application.yml`. -The agent auto-instruments Ktor, Spring Boot (MVC, WebFlux), Coroutines, Exposed, R2DBC, Kafka, gRPC, OkHttp, AWS SDK, and many more. +The agent auto-instruments Ktor, Spring Boot (MVC, WebFlux), kotlinx.coroutines context propagation, JDBC (so Exposed), R2DBC, Kafka, gRPC, OkHttp, AWS SDK, and more. ## Manual SDK (Ktor, no agent) @@ -36,8 +36,8 @@ fun initTelemetry(): OpenTelemetrySdk { val headers = mapOf("authorization" to "Bearer $MAPLE_KEY") val resource = Resource.getDefault().merge(Resource.create( Attributes.builder() - .put(ServiceAttributes.SERVICE_NAME, "orders-api") - .put(DeploymentIncubatingAttributes.DEPLOYMENT_ENVIRONMENT_NAME, + .put("service.name", "orders-api") + .put("deployment.environment.name", System.getenv("DEPLOYMENT_ENV") ?: "development") .put("vcs.repository.url.full", "https://github.com/acme/orders-api") .put("vcs.ref.head.revision", System.getenv("GITHUB_SHA") ?: "") @@ -57,7 +57,7 @@ fun initTelemetry(): OpenTelemetrySdk { } ``` -Add the equivalent log + metric exporters in the same builder. +Add the equivalent log and metric exporters in the same builder. Resolve versions through `io.opentelemetry:opentelemetry-bom`. ## Bounded business spans @@ -70,7 +70,7 @@ suspend fun submitOrder(orderId: String, tenantId: String) { .setAttribute("order.id", orderId) .startSpan() try { - span.makeCurrent().use { + withContext(span.asContextElement()) { chargeOrder(orderId) } } catch (e: Exception) { @@ -83,12 +83,12 @@ suspend fun submitOrder(orderId: String, tenantId: String) { } ``` -For coroutines, use the `kotlinx-coroutines-extension` (`opentelemetry-kotlin-extension`) so context propagates across `withContext` boundaries. +`asContextElement()` comes from `io.opentelemetry:opentelemetry-extension-kotlin` (`import io.opentelemetry.extension.kotlin.asContextElement`). Do not call `span.makeCurrent()` in a `suspend` function: the scope is thread-local and leaks or is lost when the coroutine resumes on another thread. ## Logs -Bridge whatever the project uses (Logback, SLF4J, Log4j2). With the agent, log appenders are auto-bridged. With the manual SDK, add `opentelemetry-logback-appender-1.0` (or Log4j 2 equivalent) and configure it in `logback.xml` so existing logger calls carry `trace_id` / `span_id` and reach Maple. Don't replace the user's existing logger. +Bridge whatever the project uses (Logback, SLF4J, Log4j2). With the agent, Logback and Log4j2 are bridged automatically. With the manual SDK, add `opentelemetry-logback-appender-1.0` (or `opentelemetry-log4j-appender-2.17`), declare its appender in `logback.xml`, and call `OpenTelemetryAppender.install(sdk)` after building the SDK. Existing logger calls then carry `trace_id` / `span_id` and reach Maple. Do not replace the user's existing logger. ## Coexistence -If the project runs Datadog / New Relic / Honeycomb, leave them in place. Test the combination once before shipping. +If the project runs a Datadog, New Relic, or Honeycomb agent, leave it in place. Two bytecode agents on one JVM can conflict, so test the combination once before shipping. diff --git a/skills/maple-nextjs-style/SKILL.md b/skills/maple-nextjs-style/SKILL.md index f45284e232..4254f82d93 100644 --- a/skills/maple-nextjs-style/SKILL.md +++ b/skills/maple-nextjs-style/SKILL.md @@ -5,7 +5,7 @@ description: "Next.js / Vercel OpenTelemetry style for Maple: instrumentation.ts # Maple Next.js style -For Next.js apps, prefer the framework entrypoint. +For Next.js apps, use the framework entrypoint, `instrumentation.ts` with `@vercel/otel`. ```ts // instrumentation.ts @@ -30,9 +30,9 @@ export function register() { } ``` -Do not replace this with a custom `NodeSDK` bootstrap unless the repo is not a normal Next/Vercel app or already has a custom provider that must be extended. +Do not replace this with a custom `NodeSDK` bootstrap unless the repo is not a standard Next/Vercel app or already has a custom provider to extend. -For JavaScript/TypeScript LLM providers, prefer provider instrumentation over manual child spans. For Anthropic, add OpenInference in the same bootstrap and keep call sites native. This example uses `@vercel/otel@2.x`; if the installed types are v1, use `logRecordProcessor` singular instead. +For JavaScript/TypeScript LLM providers, use provider instrumentation instead of manual child spans. For Anthropic, add OpenInference in the same bootstrap and keep call sites native. This example uses `@vercel/otel@2.x`. If the installed types are v1, use `logRecordProcessor` (singular). ```ts import Anthropic from "@anthropic-ai/sdk" @@ -99,11 +99,11 @@ export async function POST(request: Request) { } ``` -For TypeScript route handlers, use `@maple/otel-helpers` `withSpan` for bounded business spans and add `@maple/otel-helpers` to `package.json` when it is not already present. This is required when the package can be installed. It keeps span lifecycle / error handling out of the handler body and avoids a large indentation diff. Do not expand the whole route into `tracer.startActiveSpan(...)` plus `try` / `catch` / `finally` unless the helper cannot be added or the span has a true cross-callback lifecycle. +For TypeScript route handlers, use `@maple/otel-helpers` `withSpan` for bounded business spans. Add `@maple/otel-helpers` to `package.json` if it is missing; this is required whenever the package can be installed. It keeps span lifecycle and error handling out of the handler body and avoids a large indentation diff. Do not expand the whole route into `tracer.startActiveSpan(...)` plus `try` / `catch` / `finally` unless the helper cannot be added or the span has a true cross-callback lifecycle. If a route has an LLM call and OpenInference / provider instrumentation supports that SDK, do not wrap the provider call. Leave `client.messages.create(...)` / equivalent in place and put business context on the active product span or structured log. Do not duplicate provider/model/token attributes in route-level spans, logs, or metrics when OpenInference already reports them. Do not calculate LLM cost in route handlers; Maple derives estimated cost in the UI/query layer from OpenInference provider/model/token attributes. For Anthropic in Next.js/ESM, keep the instrumentation instance and `manuallyInstrument(Anthropic)` call at module scope so it runs once and before route code. -Match the `@vercel/otel` logs option to the installed package/types: `@vercel/otel@1.x` uses `logRecordProcessor` singular, while `@vercel/otel@2.x` uses `logRecordProcessors` plural. For normal Next.js / Vercel apps, do not guard `registerOTel(...)` behind `NEXT_RUNTIME`; Next calls `instrumentation.ts` in the appropriate runtime and `@vercel/otel` handles its own runtime differences. +Match the `@vercel/otel` logs option to the installed version: `@vercel/otel@1.x` takes `logRecordProcessor` (singular), `@vercel/otel@2.x` takes `logRecordProcessors` (plural). For normal Next.js / Vercel apps, do not guard `registerOTel(...)` behind `NEXT_RUNTIME`; Next calls `instrumentation.ts` in the appropriate runtime and `@vercel/otel` handles its own runtime differences. `console.info` is not OTLP log export. If there is no existing logger bridge, use `@opentelemetry/api-logs` for production log records. Remove pre-existing `console.*` calls that duplicate the same structured OTel log event: @@ -128,6 +128,6 @@ logger.emit({ ## Configuration and smoke -Inline the endpoint and ingest key in `instrumentation.ts` — pass them explicitly to `registerOTel`. Don't rely on `OTEL_EXPORTER_OTLP_*` env vars: the Maple ingest key is project-scoped + write-only and inline config sidesteps Vercel's env-propagation quirks during preview builds. +Inline the endpoint and ingest key in `instrumentation.ts` and pass them explicitly to `registerOTel`. Do not rely on `OTEL_EXPORTER_OTLP_*` env vars. The Maple ingest key is project-scoped and write-only, and inline config avoids Vercel's env-propagation quirks in preview builds. -Smoke checks should use tools already in the repo, e.g. `npm run typecheck` or `npm run build`, plus a real app request where practical. Do not invent fragile inline Node scripts that import TypeScript source files directly, and do not assume `ts-node` exists unless it is already installed. +Smoke-check with tools already in the repo (`npm run typecheck`, `npm run build`), plus a real app request where practical. Do not invent fragile inline Node scripts that import TypeScript source files directly, and do not assume `ts-node` exists unless it is already installed. diff --git a/skills/maple-nodejs-style/SKILL.md b/skills/maple-nodejs-style/SKILL.md index 88f09b075c..3a0525cc22 100644 --- a/skills/maple-nodejs-style/SKILL.md +++ b/skills/maple-nodejs-style/SKILL.md @@ -5,10 +5,11 @@ description: "Plain Node.js (Express, Fastify, Hono, Bun) OpenTelemetry style fo # Maple Node.js style -Use `@opentelemetry/sdk-node` with `--import` (or the equivalent Bun `--preload`) so the SDK starts before any framework code runs. +Use `@opentelemetry/sdk-node` loaded with `--import` (Bun: `--preload`) so the SDK starts before any framework code runs. ```ts // telemetry.ts +import { register } from "node:module" import { NodeSDK } from "@opentelemetry/sdk-node" import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node" import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http" @@ -18,6 +19,9 @@ import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs" import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics" import { resourceFromAttributes } from "@opentelemetry/resources" +// ESM apps only: lets auto-instrumentation patch `import`ed modules. Omit for CommonJS. +register("@opentelemetry/instrumentation/hook.mjs", import.meta.url) + const MAPLE_ENDPOINT = "https://ingest.maple.dev" const MAPLE_KEY = "MAPLE_TEST" // set by maple-onboard skill on pairing @@ -60,22 +64,22 @@ Run the app with the bootstrap loaded first: node --import ./telemetry.js app.js ``` -For TypeScript projects, use the loader the repo already uses (`tsx`, `ts-node/esm`, native Bun, etc.) — do not introduce a new loader. +For TypeScript projects, use the loader the repo already uses (`tsx`, `ts-node/esm`, native Bun). Do not introduce a new loader. For ESM apps, keep the `register(...)` hook call and add `@opentelemetry/instrumentation` to `package.json`; without it, ESM-imported libraries are not instrumented. ## Bootstrap rules - HTTP OTLP exporters only, never gRPC. gRPC pulls in native bindings that complicate containers. -- `getNodeAutoInstrumentations()` covers HTTP, Express, Fastify, Hono, pg, MySQL, Redis, and many more out of the box. Disable specific instrumentations only when they actively break the app: +- `getNodeAutoInstrumentations()` covers HTTP, `fetch` (undici), Express, Fastify, pg, MySQL, Redis, and more. Frameworks without their own instrumentation (Hono on `@hono/node-server`) are still traced at the HTTP server layer. Disable an instrumentation only when it breaks the app: ```ts getNodeAutoInstrumentations({ "@opentelemetry/instrumentation-fs": { enabled: false }, }) ``` -- For Bun, use the same SDK with `bun --preload ./telemetry.ts run app.ts`. Bun's HTTP exporter compatibility is good; if you hit an issue, fall back to `node` for the bootstrap process. +- For Bun, use the same SDK with `bun --preload ./telemetry.ts app.ts`. Bun ignores Node's module hooks, so some auto-instrumentations do not fire. Add manual spans where auto-instrumentation is blind. ## Route handlers and business operations -Use the native API; reach for `withSpan` from `@maple/otel-helpers` for bounded operations. +Use the native API. Use `withSpan` from `@maple/otel-helpers` for bounded operations. Its signature is `withSpan(name, fn, { tracer })`; it ends the span and records exceptions and `Error` status on throw. ```ts import { trace, metrics } from "@opentelemetry/api" @@ -104,8 +108,8 @@ app.post("/orders", async (req, res) => { ## Logs -Bridge the existing logger (Pino, Winston, console) through OTLP rather than replacing it. For Pino, install `@opentelemetry/instrumentation-pino` and include it in `instrumentations`. For Winston, install `@opentelemetry/instrumentation-winston`. The user's logger keeps its current sinks; you're adding OTLP underneath so logs carry `trace_id` / `span_id` and reach Maple. Do not rip out the existing logger. +Bridge the existing logger through OTLP. Do not replace it. `getNodeAutoInstrumentations()` already includes the Pino, Winston, and Bunyan instrumentations: they inject `trace_id` / `span_id` into records and forward them to the `logRecordProcessors` configured above. Winston forwarding also needs `@opentelemetry/winston-transport` installed. `console.*` is not bridged. The user's logger keeps its current sinks. ## Coexistence -If the repo has Sentry, Datadog, New Relic, Honeycomb, Logtail, or a Pino transport, leave them in place. They sit alongside Maple, not instead of it. +If the repo has Sentry, Datadog, New Relic, Honeycomb, Logtail, or a Pino transport, leave them in place alongside Maple. diff --git a/skills/maple-onboard/SKILL.md b/skills/maple-onboard/SKILL.md index b7c13f46ea..220393a7f1 100644 --- a/skills/maple-onboard/SKILL.md +++ b/skills/maple-onboard/SKILL.md @@ -5,7 +5,7 @@ description: "Onboard a project to Maple by installing OpenTelemetry traces, log # Maple onboarding -Wire OpenTelemetry traces, logs, and metrics into the user's project so telemetry streams to Maple. Cover **every app and service in the repo** — not just the one the user is currently sitting in. +Wire OpenTelemetry traces, logs, and metrics into the user's project so telemetry streams to Maple. Cover **every app and service in the repo**, not only the one the user is currently working in. Prefer native OpenTelemetry APIs and the framework's documented bootstrap over custom helper layers. If a specific stack stumps you, search the OTel docs for that language; don't guess. @@ -15,76 +15,76 @@ Before editing, read the applicable companion skills: - `maple-nextjs-style` for Next.js / Vercel apps. - `maple-nodejs-style` for plain Node servers (Express, Fastify, Hono, Bun). - `maple-python-style` for Python services (FastAPI, Django, Flask). -- `maple-effect-style` for Effect-based services — Maple's first-class SDK. +- `maple-effect-style` for Effect-based services (Maple's first-class SDK). - `maple-go-style`, `maple-rust-style`, `maple-java-style`, `maple-csharp-style`, `maple-kotlin-style` for those stacks. If none match (Ruby, Elixir, PHP, plain Deno, …), use `maple-onboarding-style` as the fallback and consult the upstream OTel docs for that language. -## Step 0 — Endpoint and key handling +## Step 0: Endpoint and key handling -The **OTLP endpoint** is always `https://ingest.maple.dev` and **goes inline** in the bootstrap code — it's not a secret, no env-var indirection needed. +The **OTLP endpoint** is always `https://ingest.maple.dev` and **goes inline** in the bootstrap code. It is not a secret; no env-var indirection is needed. -The **ingest API key** is **project-scoped + write-only** — it can only ingest events for one project, can't read anything, can't change settings. Treat it like a Sentry DSN, a PostHog public key, or a Datadog RUM client token: **inline it directly in the OTel bootstrap source** alongside the endpoint. No `.env` files, no deploy-target wiring, no `process.env.OTEL_EXPORTER_OTLP_HEADERS`. The user deploys their code and events flow. +The **ingest key** is **project-scoped and write-only**. It can only ingest events for one project; it can't read anything or change settings. Treat it like a Sentry DSN, a PostHog public key, or a Datadog RUM client token: **inline it directly in the OTel bootstrap source** alongside the endpoint. Use the public key (`maple_pk_…`) from Settings → Ingestion; it is the one meant to ship in source. No `.env` files, no deploy-target wiring, no `process.env.OTEL_EXPORTER_OTLP_HEADERS`. The user deploys their code and events flow. Two paths, no questions asked: ### Key in the prompt -If the invoking prompt already contains a Maple ingest key, inline it in every bootstrap file you write. Done — move on to Step 1. +If the invoking prompt already contains a Maple ingest key, inline it in every bootstrap file you write, then move on to Step 1. ### No key -Inline the literal sentinel `MAPLE_TEST` in the bootstrap source as a placeholder. Maple's ingest gateway accepts it from anyone, returns 200 without forwarding anywhere, so the user's app can boot and exercise the OTel bootstrap path while they go create a real key. +Inline the literal sentinel `MAPLE_TEST` in the bootstrap source as a placeholder. Maple's ingest gateway accepts it from anyone and returns 200 without storing anything, so the user's app can boot and exercise the OTel bootstrap path while they get a real key. -Tell the user *briefly* at the top of your work: "I'm using `MAPLE_TEST` as a placeholder so the bootstrap can run. Create a real key at https://app.maple.dev/settings (Settings → API Keys) and search-replace the literal `MAPLE_TEST` in the files I write to swap it in." +Tell the user *briefly* at the top of your work: "I'm using `MAPLE_TEST` as a placeholder so the bootstrap can run. Copy your ingest key from https://app.maple.dev/settings?tab=ingestion (Settings → Ingestion) and search-replace the literal `MAPLE_TEST` in the files I write to swap it in." Then keep going. Don't block install on signup. -## Step 1 — Map every app/service in the repo +## Step 1: Map every app/service in the repo Before instrumenting anything, enumerate what's here. Check workspace manifests (`pnpm-workspace.yaml`, root `package.json` `workspaces`, `bun` workspaces, `go.work`, Cargo workspace, Python `pyproject.toml` workspace setups, `apps/*` and `services/*` conventions). Identify each service: web frontend, API, workers, background jobs, CLIs, sample/demo apps, mobile apps, Supabase and/or server functions. Mobile and serverless/edge functions are in scope; do not skip them merely because they are client-side or short-lived. Skip pure type/config packages with no runtime entry point. Do not skip any runnable services or leave them "out of scope": instrument absolutely everything in this run; there may be no follow-up. Show the user the list before you start, so they can correct it. -## Step 2 — For each service, install native OTel and bootstrap +## Step 2: For each service, install native OTel and bootstrap **Use the language's native OpenTelemetry SDK.** Don't reach for vendor wrappers or hand-rolled helpers when an official package exists. Examples of what "native" means here: `@opentelemetry/sdk-node` for Node servers, `@vercel/otel` for normal Next.js/Vercel apps (sdk-node breaks Next's webpack and misses the framework bootstrap), `@opentelemetry/sdk-trace-web` + browser/mobile-compatible exporters for Vite/SPA/Expo, `opentelemetry-instrumentation-*` + `opentelemetry-sdk` for Python, `go.opentelemetry.io/otel` for Go. No broad wrapper APIs. Avoid reusable helpers like `sendMapleSpan`, `recordCounter`, `recordLog`, `startTelemetrySpan`, or `withTelemetry`. Acquire native tracers/meters/loggers at module scope and use the SDK's own APIs directly. In TypeScript/JavaScript, use the published `@maple/otel-helpers` `withSpan` helper for bounded business spans and add `@maple/otel-helpers` to `package.json`; this is required when the package can be installed because it avoids expanding a whole function into `startActiveSpan` plus `try` / `catch` / `finally`. Do not use helpers around provider SDK calls that OpenInference/provider instrumentation can observe directly. -Wire all three signals — traces, logs, metrics. **Logs go through OTLP, not just stdout** — set up the OTel log bridge for the language so app logs (with their existing log levels and structured fields) carry the active `trace_id` / `span_id` automatically. The user's existing logger keeps working; you're just adding an OTLP handler/processor underneath. +Wire all three signals: traces, logs, metrics. **Logs go through OTLP, not just stdout.** Set up the OTel log bridge for the language so app logs (with their existing log levels and structured fields) carry the active `trace_id` / `span_id` automatically. The user's existing logger keeps working; you only add an OTLP handler/processor underneath. Bootstrap rules: - The bootstrap file must run before any framework imports. Use the language/framework's documented hook (`--import` flag, `instrumentation.ts`, top-of-`main.py` import, etc.). -- Inline the endpoint (`https://ingest.maple.dev`) and the project's ingest key directly in the bootstrap source. Don't read from `process.env.OTEL_EXPORTER_OTLP_*` or write any `.env` files — the key is write-only, and inline configuration removes a whole class of "OTel didn't start because env vars weren't set" deploy failures. (See the framework-specific style skills for the exact shape per stack.) +- Inline the endpoint (`https://ingest.maple.dev`) and the project's ingest key directly in the bootstrap source. Don't read from `process.env.OTEL_EXPORTER_OTLP_*` or write any `.env` files. The key is write-only, and inline configuration removes a whole class of "OTel didn't start because env vars weren't set" deploy failures. (See the framework-specific style skills for the exact shape per stack.) - Use HTTP OTLP exporters, not gRPC. gRPC pulls in native bindings that break bundlers and complicate containers. - Use the project's existing package manager (detect via lockfile). - Prefer idempotent edits. If a config file already exists, edit don't overwrite. -- Set resource attributes on the OTel resource for every service: `service.name`, `service.version`, `deployment.environment.name`, and **`vcs.repository.url.full`** — the canonical https URL of the repo (e.g. `https://github.com/acme/api`). The repo URL is the important one and is fine to hardcode alongside `service.name` in the SDK init; if the build platform exposes the slug (Vercel `VERCEL_GIT_REPO_OWNER`/`VERCEL_GIT_REPO_SLUG`, Railway `RAILWAY_GIT_REPO_OWNER`/`RAILWAY_GIT_REPO_NAME`), prefer reading from env. Also set **`vcs.ref.head.revision`** (commit SHA) on a best-effort basis from whatever env the runtime already injects (`VERCEL_GIT_COMMIT_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `GITHUB_SHA`, `SOURCE_COMMIT`, `GIT_COMMIT`, `HEROKU_SLUG_COMMIT`, …). Do not shell out to `git` from the running process. Skipping the SHA is fine, skipping the URL is not. Use the OTel semantic-convention keys exactly — do not invent `git.repo` / `app.repo_url` / `deployment.commit_sha`. +- Set resource attributes on the OTel resource for every service: `service.name`, `service.version`, `deployment.environment.name`, and **`vcs.repository.url.full`**, the canonical https URL of the repo (e.g. `https://github.com/acme/api`). The repo URL is the important one and is fine to hardcode alongside `service.name` in the SDK init; if the build platform exposes the slug (Vercel `VERCEL_GIT_REPO_OWNER`/`VERCEL_GIT_REPO_SLUG`, Railway `RAILWAY_GIT_REPO_OWNER`/`RAILWAY_GIT_REPO_NAME`), prefer reading from env. Also set **`vcs.ref.head.revision`** (commit SHA) on a best-effort basis from whatever env the runtime already injects (`VERCEL_GIT_COMMIT_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `GITHUB_SHA`, `SOURCE_COMMIT`, `GIT_COMMIT`, `HEROKU_SLUG_COMMIT`, …). Do not shell out to `git` from the running process. Skipping the SHA is fine, skipping the URL is not. Use the OTel semantic-convention keys exactly. Do not invent `git.repo` / `app.repo_url` / `deployment.commit_sha`. Framework rules: - **Next.js/Vercel:** use `instrumentation.ts` with `@vercel/otel` `registerOTel(...)` as the bootstrap. Do not substitute a raw `@opentelemetry/sdk-node` / `NodeSDK` bootstrap unless the repo already uses that architecture and you are extending it. Use `@opentelemetry/api` tracers/meters inside route handlers only where auto-instrumentation is blind. -- **Expo/React Native:** preserve existing Expo Go / unsupported-runtime guards. In supported builds, call `initObservability()` before Sentry and before app registration/user code. Inline the endpoint + ingest key in the observability module — no `EXPO_PUBLIC_*` env vars. The bootstrap reads them straight from constants. -- **Supabase Edge Functions / Cloudflare Workers:** native Deno / Workers OpenTelemetry can be quirky. Keep the exporter shim tiny, provider-neutral, and OTel-shaped: `tracer.startActiveSpan`, `span.setAttributes`, `SpanStatusCode`, `meter.createCounter`, `histogram.record`. For Effect-on-Workers, prefer `@maple/infra/worker-runtime` (see `maple-effect-style`). +- **Expo/React Native:** preserve existing Expo Go / unsupported-runtime guards. In supported builds, call `initObservability()` before Sentry and before app registration/user code. Inline the endpoint + ingest key in the observability module, with no `EXPO_PUBLIC_*` env vars. The bootstrap reads them straight from constants. +- **Supabase Edge Functions / Cloudflare Workers:** native Deno / Workers OpenTelemetry can be quirky. Keep the exporter shim tiny, provider-neutral, and OTel-shaped: `tracer.startActiveSpan`, `span.setAttributes`, `SpanStatusCode`, `meter.createCounter`, `histogram.record`. For Effect on Workers, use `@maple-dev/effect-sdk/cloudflare` (see `maple-effect-style`). - **Python/FastAPI:** use native instrumentation such as `FastAPIInstrumentor.instrument_app(app)` rather than replacing request handling with manual middleware. -**Coexist with existing observability vendors. Don't remove Sentry, Datadog, New Relic, Honeycomb, Logtail, Pino transports, etc.** OTel sits alongside them. The user explicitly wants both signals flowing during migration; ripping out the incumbent is not your call. +**Coexist with existing observability vendors. Don't remove Sentry, Datadog, New Relic, Honeycomb, Logtail, Pino transports, etc.** OTel sits alongside them. The user wants both flowing during migration; removing the incumbent is not your call. -## Step 3 — Add custom spans, metrics, and logs around business operations +## Step 3: Add custom spans, metrics, and logs around business operations -Auto-instrumentation gets you HTTP in/out, DB queries, framework lifecycle. That's the floor, not the ceiling. Read the project to find the operations a human operator would actually want to see when something looks wrong. +Auto-instrumentation covers HTTP in/out, DB queries, and framework lifecycle. That is the floor. Read the project to find the operations an operator would want to see when something looks wrong. ### Traces -Wrap **every critical business operation** with an active span. Auto-instrumented spans are fine where they exist — but if an operation isn't already getting a span, add one. +Wrap **every critical business operation** with an active span. Auto-instrumented spans are fine where they exist. If an operation isn't already getting a span, add one. - Naming: `domain.verb` (`order.process`, `payment.charge`, `email.send`, `agent.run`, `interview.create`, `job.`). - Attributes: entity IDs (order.id, user.id, workspace.id, tenant.id), counts, key boolean branch outcomes, model name / provider for LLM calls. - Record exceptions: `span.recordException(err)` + `span.setStatus({ code: ERROR })` on failure paths. **In TS/JS use `withSpan` from `@maple/otel-helpers`** so the body stays flat instead of nested under `try` / `catch` / `finally`. - For Python functions with clear boundaries, prefer `@tracer.start_as_current_span("operation.name")`. Use a context manager when a decorator does not fit. Do not use detached `start_span()` + manual `end()` for bounded work. -- Skip trivial getters, pure transforms, internal helpers — anything with no real latency or failure mode. +- Skip trivial getters, pure transforms, and internal helpers: anything with no real latency or failure mode. - **Never put PII in attributes** (emails, passwords, tokens, full request bodies). ### Logs @@ -97,22 +97,22 @@ Use logs for narrative ("starting batch reconcile", "retrying after 3xx") and ex Cover **business + performance + cost**. Three categories to look for: -- **Business logic counters.** Every meaningful state transition: created, started, completed, failed, retried. Per-tenant, per-channel, per-status — low-cardinality dimensions only (never user/order IDs). -- **Performance histograms.** Latency of operations the user cares about, queue depth, batch sizes, payload sizes. Reuse existing timing instrumentation if the project already has any (`time.perf_counter` blocks, custom `LatencyTracker`s, "[TIMING]" log lines) — emit a histogram from those measurements rather than measuring twice. -- **Costs — especially LLM costs.** If the project calls OpenAI / Anthropic / Google / any LLM provider, prefer provider instrumentation such as OpenInference where available so native SDK calls stay readable. Do not add pricing constants or LLM cost math in product handlers; Maple computes estimated cost centrally in the UI/query layer from captured provider/model/token attributes. Avoid duplicating token counters already captured by provider instrumentation. +- **Business logic counters.** Every meaningful state transition: created, started, completed, failed, retried. Break down per tenant, channel, or status, using low-cardinality dimensions only (never user/order IDs). +- **Performance histograms.** Latency of operations the user cares about, queue depth, batch sizes, payload sizes. Reuse existing timing instrumentation if the project has any (`time.perf_counter` blocks, custom `LatencyTracker`s, "[TIMING]" log lines). Emit a histogram from those measurements rather than measuring twice. +- **Costs, especially LLM costs.** If the project calls OpenAI / Anthropic / Google / any LLM provider, prefer provider instrumentation such as OpenInference where available so native SDK calls stay readable. Do not add pricing constants or LLM cost math in product handlers; Maple computes estimated cost centrally in the UI/query layer from captured provider/model/token attributes. Avoid duplicating token counters already captured by provider instrumentation. Get the meter once at module level, create instruments at module level, increment in the hot path. Don't create a fresh meter per call. -## Step 4 — Verify the app still works +## Step 4: Verify the app still works Per service: -1. **Run the project's own dev or build command** (whatever its `package.json` / `pyproject` / `Makefile` already wires up). Confirm it starts cleanly with no errors that trace back to your OTel install. Also run a telemetry bootstrap smoke that imports or starts the app, so provider setup, exporter construction, log bridging, and framework instrumentation all initialize. For a Python server this can be an import/startup command such as `uv run python -c 'from app.main import app; print(app.title)'`; for Node/Next use the repo's build/start path. For a server, hit at least one route with curl so traffic flows through the instrumentation; choose a route that exercises an instrumented operation when practical, not only a static health route. For a CLI, invoke a real command. **Don't ship if the app's own startup is now broken** — that's a regression. -2. **Confirm telemetry leaves the process.** With the inline `MAPLE_TEST` (or real key) in the bootstrap, OTLP POSTs from the dev server should return 2xx — that proves the bootstrap is reaching the network. As an extra sanity check, you can `curl -X POST https://ingest.maple.dev/v1/traces -H "authorization: Bearer MAPLE_TEST" -H "content-type: application/json" -d '{}'` and confirm 200. The real signal is the running app's own POSTs succeeding by the time the dev server shuts down — if they don't, the bootstrap is wrong (most often: SDK loaded too late, or shutdown not flushing). +1. **Run the project's own dev or build command** (whatever its `package.json` / `pyproject` / `Makefile` already wires up). Confirm it starts cleanly with no errors that trace back to your OTel install. Also run a telemetry bootstrap smoke that imports or starts the app, so provider setup, exporter construction, log bridging, and framework instrumentation all initialize. For a Python server this can be an import/startup command such as `uv run python -c 'from app.main import app; print(app.title)'`; for Node/Next use the repo's build/start path. For a server, hit at least one route with curl so traffic flows through the instrumentation; choose a route that exercises an instrumented operation when practical, not only a static health route. For a CLI, invoke a real command. **Don't ship if the app's own startup is now broken.** That is a regression. +2. **Confirm telemetry leaves the process.** With the inline `MAPLE_TEST` (or real key) in the bootstrap, OTLP POSTs from the dev server should return 2xx. That proves the bootstrap reaches the network. As an extra sanity check, you can `curl -X POST https://ingest.maple.dev/v1/traces -H "authorization: Bearer MAPLE_TEST" -H "content-type: application/json" -d '{}'` and confirm 200. The real signal is the running app's own POSTs succeeding by the time the dev server shuts down. If they don't, the bootstrap is wrong (most often the SDK loads too late, or shutdown doesn't flush). A bootstrap that loads but never POSTs is not a partial success. Fix it before moving on. -## Step 5 — Hand-off (final message to the user) +## Step 5: Hand-off (final message to the user) ### What changed @@ -122,19 +122,19 @@ A bootstrap that loads but never POSTs is not a partial success. Fix it before m If `MAPLE_TEST` is still inline, tell the user: -> "The bootstrap currently uses `MAPLE_TEST` as a placeholder so the install could complete end-to-end. Create a real key at https://app.maple.dev (Settings → API Keys), then search-replace `MAPLE_TEST` in the files I wrote with your new key and redeploy." +> "The bootstrap currently uses `MAPLE_TEST` as a placeholder so the install could complete end-to-end. Copy your ingest key from https://app.maple.dev (Settings → Ingestion), then search-replace `MAPLE_TEST` in the files I wrote with your new key and redeploy." ### Deploy -Tell the user to deploy as they normally would — push to their hosting platform, run their existing CI, or run locally. There are no env vars to wire and nothing platform-specific to configure: the endpoint and key are inline in the bootstrap, so events start flowing the moment the instrumented code runs. +Tell the user to deploy as they normally would: push to their hosting platform, run their existing CI, or run locally. There are no env vars to wire and nothing platform-specific to configure: the endpoint and key are inline in the bootstrap, so events start flowing the moment the instrumented code runs. -If the user asks "where do I put the key in production?" — the answer is "you already did, it's in the source you just deployed." +If the user asks "where do I put the key in production?", the answer is "you already did; it's in the source you just deployed." -## Step 6 — Drive Maple MCP install +## Step 6: Drive Maple MCP install -Suggest installing the Maple MCP server so the agent (Claude Code, Codex, Cursor, etc.) can query telemetry directly next time the user is debugging — search logs, pull traces, check error rates from inside the chat without context-switching to the dashboard. +Suggest installing the Maple MCP server so the agent (Claude Code, Codex, Cursor, etc.) can query telemetry directly the next time the user is debugging: search logs, pull traces, and check error rates from the chat without switching to the dashboard. -For **Claude Code** (most common — that's where this skill is running), offer to run it for them: +For **Claude Code** (most common, and where this skill is running), offer to run it for them: ``` claude mcp add --transport http maple https://api.maple.dev/mcp @@ -147,7 +147,7 @@ For other agents the user might also use, mention but do *not* run: - **Codex:** `codex mcp add maple --url https://api.maple.dev/mcp` - **Cursor / others:** copy the `mcpServers` snippet from https://app.maple.dev/mcp. -When done (or skipped), close out with a single line directing the user to deploy their app — they're ready to ship. +When done (or skipped), close with a single line telling the user to deploy their app. ## Hard rules @@ -157,4 +157,4 @@ When done (or skipped), close out with a single line directing the user to deplo - Never remove an existing observability vendor unless the user asks for it. - Use the project's existing package manager and existing logger. - Prefer native OTel packages for the language; don't reinvent telemetry plumbing the SDK already provides. -- If the dev/build command errors out *because of* your instrumentation, that's a failure — fix it or report it, don't paper over it. +- If the dev/build command errors out *because of* your instrumentation, that is a failure. Fix it or report it; don't paper over it. diff --git a/skills/maple-onboarding-style/SKILL.md b/skills/maple-onboarding-style/SKILL.md index 499ed7d675..2a02786472 100644 --- a/skills/maple-onboarding-style/SKILL.md +++ b/skills/maple-onboarding-style/SKILL.md @@ -50,14 +50,14 @@ withTelemetry(...) ## Endpoint and key -Inline the endpoint and the project's ingest key directly in the bootstrap source — don't read from `OTEL_EXPORTER_OTLP_*` env vars and don't write `.env` files. The Maple ingest key is project-scoped + write-only (Sentry DSN shaped), so source-level configuration is the right default; env-var indirection only adds deploy-time failure modes. +Inline the endpoint and the project's ingest key directly in the bootstrap source. Don't read from `OTEL_EXPORTER_OTLP_*` env vars and don't write `.env` files. The Maple ingest key is project-scoped and write-only (shaped like a Sentry DSN), so source-level configuration is the right default. Env-var indirection only adds deploy-time failure modes. ```text MAPLE_ENDPOINT = "https://ingest.maple.dev" -MAPLE_KEY = "maple_…" # or "MAPLE_TEST" while pairing +MAPLE_KEY = "maple_pk_…" # public ingest key, or "MAPLE_TEST" while pairing ``` -While pairing is in flight, use the literal `MAPLE_TEST` sentinel — Maple's ingest accepts it without forwarding events anywhere, so the bootstrap exercises the full code path before the real key arrives. +While pairing is in flight, use the literal `MAPLE_TEST` sentinel. Maple's ingest accepts it and drops the events, so the bootstrap exercises the full code path before the real key arrives. Pass the inline values to the SDK explicitly via the exporter constructor's `endpoint` / `headers` options. Do not configure the SDK off implicit env-var reads. @@ -67,11 +67,11 @@ Include standard resource attributes when values are available: `service.name`, ## VCS resource attributes -Set `vcs.repository.url.full` on the OTel resource for every instrumented service. The value is the canonical https URL of the repo (e.g. `https://github.com/acme/api`) — the same URL the user would paste in a browser, not an SSH URL, not a local working-tree path. This is the important one: it lets Maple link telemetry back to the source of truth. It is fine to hardcode this string alongside `service.name` in the SDK init; if a build env already exposes the slug (e.g. `VERCEL_GIT_REPO_OWNER` + `VERCEL_GIT_REPO_SLUG`, `RAILWAY_GIT_REPO_OWNER` + `RAILWAY_GIT_REPO_NAME`), prefer reading from env so a fork or rename doesn't drift. +Set `vcs.repository.url.full` on the OTel resource for every instrumented service. The value is the canonical https URL of the repo (e.g. `https://github.com/acme/api`): the URL the user would paste in a browser, not an SSH URL or a local working-tree path. This is the important one; it lets Maple link telemetry back to the source. It is fine to hardcode this string alongside `service.name` in the SDK init; if a build env already exposes the slug (e.g. `VERCEL_GIT_REPO_OWNER` + `VERCEL_GIT_REPO_SLUG`, `RAILWAY_GIT_REPO_OWNER` + `RAILWAY_GIT_REPO_NAME`), prefer reading from env so a fork or rename doesn't drift. -Also set `vcs.ref.head.revision` (the commit SHA) on a best-effort basis. Read it from whatever env var the runtime/build platform already injects: `VERCEL_GIT_COMMIT_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `GITHUB_SHA`, `SOURCE_COMMIT`, `GIT_COMMIT`, `HEROKU_SLUG_COMMIT`, etc. Do not shell out to `git` from the running process — many production images do not have git or a working tree. If no env source is available, omit the attribute; skipping the SHA is fine, skipping the URL is not. +Also set `vcs.ref.head.revision` (the commit SHA) on a best-effort basis. Read it from whatever env var the runtime/build platform already injects: `VERCEL_GIT_COMMIT_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `GITHUB_SHA`, `SOURCE_COMMIT`, `GIT_COMMIT`, `HEROKU_SLUG_COMMIT`, etc. Do not shell out to `git` from the running process. Many production images have no git binary or working tree. If no env source is available, omit the attribute; skipping the SHA is fine, skipping the URL is not. -Use `vcs.repository.url.full` and `vcs.ref.head.revision` exactly as named — these are the OTel semantic-convention keys. Do not invent parallel attributes like `git.repo`, `app.repo_url`, or `deployment.commit_sha`. +Use `vcs.repository.url.full` and `vcs.ref.head.revision` exactly as named. These are the OTel semantic-convention keys. Do not invent parallel attributes like `git.repo`, `app.repo_url`, or `deployment.commit_sha`. ## Signals diff --git a/skills/maple-otel-spec-review/SKILL.md b/skills/maple-otel-spec-review/SKILL.md index 350bfb7fe7..b13cc4e01c 100644 --- a/skills/maple-otel-spec-review/SKILL.md +++ b/skills/maple-otel-spec-review/SKILL.md @@ -1,13 +1,19 @@ --- name: maple-otel-spec-review -description: "Review a diff, PR, or specific file in this repo for OpenTelemetry *specification* compliance, grounded in the source-linked spec corpus at docs/otel-spec/ (snapshot v1.58.0). Triggers on requests like 'is this spec compliant', 'review this PR against the OTel spec', 'spec-review this diff', 'check my partial-success handling', 'are these retryable status codes right', 'does apps/ingest honor the OTLP spec', and on reviews of changes touching the OTLP server surface in apps/ingest (partial success, retryable set {429, 502, 503, 504}, protobuf Status bodies, gzip, OTLP/JSON encoding), self-instrumentation (apps/api tracer setup, apps/ingest/src/otel.rs, packages/effect-sdk), or consumers of span status / SeverityNumber / db.query.text (WarehouseQueryService, query-engine). Spec MUSTs and SHOULDs only — for Maple house conventions use maple-telemetry-conventions; for whole-project instrumentation audits use maple-audit; for general diff correctness use /code-review." +description: "Review a diff, PR, or specific file in this repo for OpenTelemetry *specification* compliance, grounded in the source-linked spec corpus at docs/otel-spec/ (snapshot v1.58.0). Triggers on requests like 'is this spec compliant', 'review this PR against the OTel spec', 'spec-review this diff', 'check my partial-success handling', 'are these retryable status codes right', 'does apps/ingest honor the OTLP spec', and on reviews of changes touching the OTLP server surface in apps/ingest (partial success, retryable set {429, 502, 503, 504}, protobuf Status bodies, gzip, OTLP/JSON encoding), self-instrumentation (apps/api tracer setup, apps/ingest/src/otel.rs, packages/effect-sdk), or consumers of span status / SeverityNumber / db.query.text (WarehouseQueryService, query-engine). Spec MUSTs and SHOULDs only. For Maple house conventions use maple-telemetry-conventions; for whole-project instrumentation audits use maple-audit; for general diff correctness use /code-review." --- # OTel spec-compliance review -Review a diff, PR, or named file of *Maple's own code* against the *OpenTelemetry specification* as snapshotted in `docs/otel-spec/` (spec **v1.58.0**, researched 2026-07-05). This is not `maple-audit` (whole-project instrumentation-quality audit against Maple conventions, mostly for external projects), not `maple-telemetry-conventions` (the house-conventions reference — cross-referenced here, never enforced from here), and not generic `/code-review` (diff correctness). The deliverable is spec findings: MUST/SHOULD violations with normative citations. +Review a diff, PR, or named file of *Maple's own code* against the *OpenTelemetry specification* as snapshotted in `docs/otel-spec/` (spec **v1.58.0**, researched 2026-07-05). The deliverable is spec findings: MUST/SHOULD violations with normative citations. -`docs/otel-spec/README.md` is the routing hub; the nine spec files are the **only authority this skill may cite**. Every section in them carries an inline `Source:` deep link to the official spec — findings copy that link. +Neighbouring skills cover other ground: + +- `maple-audit`: whole-project instrumentation-quality audit against Maple conventions, mostly for external projects. +- `maple-telemetry-conventions`: the house-conventions reference. Cross-reference it here; never enforce it from here. +- `/code-review`: generic diff correctness. + +`docs/otel-spec/README.md` is the routing hub. The nine spec files beside it are the **only authority this skill may cite**. Every section in them carries an inline `Source:` deep link to the official spec; findings copy that link. ## Severity @@ -16,32 +22,32 @@ Severity comes from the spec's own normative word, not intuition: | Severity | Meaning | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `critical` | Violates a MUST / MUST NOT / SHALL / REQUIRED in a Stable spec section: failing a whole batch instead of returning partial success, wrong retryable status set, non-protobuf error body, an SDK code path that can throw into the host app, dropping gzip decode. | -| `warn` | Violates a SHOULD / SHOULD NOT, or newly relies on a deprecated path: fresh span-event usage for event-shaped data, deprecated semconv keys without coalescing, missing throttle hints on 429/503. | +| `warn` | Violates a SHOULD / SHOULD NOT, or newly relies on a deprecated path: new span-event usage for event-shaped data, deprecated semconv keys without coalescing, missing throttle hints on 429/503. | | `info` | MAY/recommendation divergence; anything grounded in a Development/Experimental-stability section; **any finding touching a README known-gap item (mandatory cap, tagged "unverified")**. | ## Routing table -Map what the diff touches to the minimal spec sections. Route by heading text, never line numbers. Surfaces are the three from `docs/otel-spec/README.md`: **1** ingest gateway as OTLP server · **2** self-instrumentation as SDK consumer · **3** UI/query as data consumer. +Map what the diff touches to the minimal spec sections. Route by heading text, never by line numbers. The surfaces are the three from `docs/otel-spec/README.md`: **1** ingest gateway as OTLP server · **2** self-instrumentation as SDK consumer · **3** UI/query as data consumer. | Diff touches (path or content signal) | Surface | Load (file → `##`/`###` headings) | | --- | --- | --- | -| `apps/ingest` OTLP handlers: request decode, response building, status codes | 1 | otlp.md → `Request/response shapes` (esp. `Partial success (server MUST rules)`, `When to use partial success vs full failure (server MUST)`), `Failure handling` (esp. `HTTP status codes — retryable matrix`, `Throttling / backpressure (server SHOULD)`) | -| OTLP/JSON parsing or serialization (IDs, enums, unknown fields) | 1 | otlp.md → `JSON encoding (OTLP/HTTP JSON) — deviations from standard protobuf JSON`; proto field questions → `Proto schema essentials` | +| `apps/ingest` OTLP handlers: request decode, response building, status codes | 1 | otlp.md → `Request/response shapes` (esp. `Partial success (server MUST rules)`, `When to use partial success vs full failure (server MUST)`), `Failure handling` (esp. `HTTP status codes: retryable matrix`, `Throttling / backpressure (server SHOULD)`) | +| OTLP/JSON parsing or serialization (IDs, enums, unknown fields) | 1 | otlp.md → `JSON encoding (OTLP/HTTP JSON): deviations from standard protobuf JSON`; proto field questions → `Proto schema essentials` | | gzip / content-type / transport handling in ingest | 1 | otlp.md → `Transports: gRPC vs HTTP` (esp. `Compression`, `Content types`) | -| Inbound `traceparent`/`tracestate`/baggage handling; propagator config | 1/2 | context-propagation.md → `W3C Trace Context — traceparent` (esp. `Practical MUSTs for an ingest gateway`), `W3C Trace Context — tracestate`, `Baggage`; propagator wiring → `Propagators API` | +| Inbound `traceparent`/`tracestate`/baggage handling; propagator config | 1/2 | context-propagation.md → `W3C Trace Context: traceparent` (esp. `Practical MUSTs for an ingest gateway`), `W3C Trace Context: tracestate`, `Baggage`; propagator wiring → `Propagators API` | | Span creation/status/kind sites: `apps/ingest/src/otel.rs`, `tracing::info_span!` / `otel.kind`, `apps/api` tracer setup, `packages/effect-sdk/` | 2 | traces.md → `3. Span`, `4. SpanKind`, `6. TracerProvider / Tracer`; error-handling posture → stability-and-compliance.md → `5. Error-handling principles (fail-safe requirements)`; scope naming → stability-and-compliance.md → `7. Library / instrumentation guidelines` | | Span processors, exporters, sampling config | 2 | traces.md → `12. Sampling (SDK)`, `15. Span Processor`, `16. Span Exporter`, `13. Span Limits` | -| Resource attributes, `OTEL_*` env vars, exporter endpoints | 2 | resource-and-config.md → `1. Resource: definition, immutability, merge rules`, `3. Key resource semantic conventions`, `5. SDK environment-variable spec (consolidated table)`; endpoint rules → otlp.md `Exporter configuration ('OTEL_EXPORTER_OTLP_*')` | +| Resource attributes, `OTEL_*` env vars, exporter endpoints | 2 | resource-and-config.md → `1. Resource: definition, immutability, merge rules`, `3. Key resource semantic conventions`, `5. SDK environment-variable spec (consolidated table)`; endpoint rules → otlp.md `Exporter configuration (OTEL_EXPORTER_OTLP_*)` | | New/renamed attribute keys (`setAttribute`, `annotateCurrentSpan`, `record(...)`, `#[instrument(fields(...))]`) | 2/3 | semantic-conventions.md → `General rules` + the one matching domain section (`HTTP`, `Database`, `Messaging`, `RPC`, `Exceptions`, `Gen-AI`) | -| Consumers of status / severity / db keys: `WarehouseQueryService`, `packages/query-engine`, `packages/domain/src/tinybird/`, UI classification code | 3 | semantic-conventions.md → matching domain section (HTTP status rule: `Span status mapping (the important compliance rule)`); logs.md → `2. Log Data Model` (esp. `2.2 SeverityNumber reference table`), `5. Events` | -| Metrics instruments, views, temporality, cardinality | 2 | metrics.md → `1. Metrics API` (esp. `1.4 Naming rules`), `2. Metrics SDK` (esp. `2.2 Aggregations — default per instrument`, `2.5 Cardinality limits`); data-model questions only → `3. Metrics Data Model` | +| Consumers of status / severity / db keys: `WarehouseQueryService` (`packages/backend/src/services/warehouse/`), `packages/query-engine`, `packages/domain/src/tinybird/`, UI classification code | 3 | semantic-conventions.md → matching domain section (HTTP status rule: `Span status mapping (the key compliance rule)`); logs.md → `2. Log Data Model` (esp. `2.2 SeverityNumber reference table`), `5. Events` | +| Metrics instruments, views, temporality, cardinality | 2 | metrics.md → `1. Metrics API` (esp. `1.4 Naming rules`), `2. Metrics SDK` (esp. `2.2 Aggregations: default per instrument`, `2.5 Cardinality limits`); data-model questions only → `3. Metrics Data Model` | | Log bridge wiring / log record emission | 2 | logs.md → `3. Logs API ("Bridge API")`, `4. Logs SDK` | | Schema URLs, Prometheus interop, profiles | 2/3 | emerging-and-compat.md → `2. Telemetry schemas (deep dive)` / `3. Prometheus & OpenMetrics compatibility` / `1. Profiles signal` | -| Anything OTel-shaped that doesn't map above | — | README `Files` table → pick ONE file; read its `Relevance to Maple` intro (where present), then decide which sections to load | +| Anything OTel-shaped that doesn't map above | none | README `Files` table → pick ONE file; read its `Relevance to Maple` intro (where present), then decide which sections to load | -**Loading rule:** `Grep -n '^##' docs/otel-spec/.md` to locate the routed heading, then `Read` with offset/limit from that line to the next same-level heading. If the file has a `Relevance to Maple` intro (traces, metrics, logs, resource-and-config, context-propagation, emerging-and-compat), read that first (~30 lines) as orientation. +**Loading rule:** run `Grep -n '^##' docs/otel-spec/.md` to locate the routed heading, then `Read` with offset/limit from that line to the next same-level heading. If the file has a `Relevance to Maple` intro (traces, metrics, logs, resource-and-config, context-propagation, emerging-and-compat), read that first (~30 lines) for orientation. -**Budget:** never read a spec file end-to-end (traces.md is 801 lines). Default is ≤2 spec files per review, section-scoped. If a third is genuinely needed, say why in the report's "Spec sections loaded" line. +**Budget:** never read a spec file end-to-end (traces.md is ~800 lines). Load at most 2 spec files per review by default, section-scoped. If a third is genuinely needed, say why in the report's "Spec sections loaded" line. ## Load-bearing MUSTs (quick table) @@ -51,75 +57,75 @@ The most common review facts, cached so trivial reviews load zero spec files. Th | --- | --- | --- | --- | | 1 | 1 | Partially-bad batch → HTTP 200 + `partial_success` with `rejected_*` count and `error_message`. Never fail the whole batch for a few bad items. | otlp.md § Partial success (server MUST rules) | | 2 | 1 | 400 is only for fully undecodable/invalid requests, and is non-retryable. | otlp.md § When to use partial success vs full failure (server MUST) | -| 3 | 1 | The retryable HTTP status set is exactly {429, 502, 503, 504}. | otlp.md § HTTP status codes — retryable matrix | -| 4 | 1 | Every 4xx/5xx response body MUST be a protobuf `Status`. | otlp.md § Failure handling | +| 3 | 1 | The retryable HTTP status set is exactly {429, 502, 503, 504}. | otlp.md § HTTP status codes: retryable matrix | +| 4 | 1 | Every 4xx/5xx response body MUST be a protobuf `Status`. | otlp.md § HTTP status codes: retryable matrix | | 5 | 1 | Server MUST accept gzip-compressed request bodies. | otlp.md § Transports: gRPC vs HTTP → Compression | -| 6 | 1 | OTLP/JSON: hex trace/span IDs, lowerCamelCase field names, enums accepted as numbers, unknown fields MUST be ignored. | otlp.md § JSON encoding — deviations | +| 6 | 1 | OTLP/JSON: hex trace/span IDs, lowerCamelCase field names, enums accepted as numbers, unknown fields MUST be ignored. | otlp.md § JSON encoding (OTLP/HTTP JSON): deviations from standard protobuf JSON | | 7 | 1 | When throttling, server SHOULD respond 429/503 and MAY include `Retry-After`; clients SHOULD honor it. | otlp.md § Throttling / backpressure (server SHOULD) | -| 8 | 2 | API/SDK calls MUST NOT throw into the host application — fail-safe, self-diagnose to own log. | stability-and-compliance.md § 5. Error-handling principles (fail-safe requirements) | -| 9 | 2 | HTTP SERVER span status = `Error` only for 5xx; 4xx leaves server-span status `Unset`. (CLIENT spans: `Error` on any 4xx/5xx.) | semantic-conventions.md § HTTP → Span status mapping (the important compliance rule) | -| 10 | 2 | Per-component instrumentation scope names, not one global tracer. | stability-and-compliance.md § 7 → Instrumentation Scope — identity and naming | +| 8 | 2 | API/SDK calls MUST NOT throw into the host application. Fail safe and self-diagnose to the SDK's own log. | stability-and-compliance.md § 5. Error-handling principles (fail-safe requirements) | +| 9 | 2 | HTTP SERVER span status = `Error` only for 5xx; 4xx leaves server-span status `Unset`. (CLIENT spans: `Error` on any 4xx/5xx.) | semantic-conventions.md § HTTP → Span status mapping (the key compliance rule) | +| 10 | 2 | Per-component instrumentation scope names, not one global tracer. | stability-and-compliance.md § 7 → Instrumentation Scope: identity and naming | | 11 | 2 | `deployment.environment.name` is canonical; legacy `deployment.environment` is deprecated. | resource-and-config.md § 3.2 | -| 12 | 2 | Resource is immutable once the provider is built; merge = updating resource wins per key. | resource-and-config.md § 1. Resource: definition, immutability, merge rules | +| 12 | 2 | Resource is immutable once the provider is built; on merge, the updating resource wins per key. | resource-and-config.md § 1. Resource: definition, immutability, merge rules | | 13 | 2 | Metric instrument names: first char alphabetic, ≤255 chars, case-insensitive ASCII, charset `A–Z a–z 0–9 _ . - /`. | metrics.md § 1.4 Naming rules | | 14 | 3 | Log error classification keys on `SeverityNumber >= 17` (ERROR), never `SeverityText` string-matching. | logs.md § 2.2 SeverityNumber reference table | | 15 | 3 | An "event" is a LogRecord with `EventName` set; span events / `RecordException()` are deprecating toward log-based events. | logs.md § 5. Events | -| 16 | 1/2 | All-zero `trace-id`/`parent-id` in `traceparent` is invalid → treat as absent, start fresh; extraction never throws — malformed degrades to "no context extracted". | context-propagation.md § traceparent → Practical MUSTs for an ingest gateway | +| 16 | 1/2 | All-zero `trace-id`/`parent-id` in `traceparent` is invalid: treat it as absent and start fresh. Extraction never throws; malformed input degrades to "no context extracted". | context-propagation.md § traceparent → Practical MUSTs for an ingest gateway | -## Step 1 — Scope the diff +## Step 1: Scope the diff -Resolve what's under review: the working diff (`git diff` / `git diff --staged`), a PR (its diff via the GitHub tooling available), a commit (`git show `), or files the user named. List the touched files grouped by path. If nothing touches a compliance surface or OTel-shaped content (no spans, exporters, OTLP handling, telemetry attributes, or telemetry consumers), say so in two lines and stop — don't spec-review generic code. +Resolve what's under review: the working diff (`git diff` / `git diff --staged`), a PR (its diff via the available GitHub tooling), a commit (`git show `), or files the user named. List the touched files grouped by path. If nothing touches a compliance surface or OTel-shaped content (no spans, exporters, OTLP handling, telemetry attributes, or telemetry consumers), say so in two lines and stop. Don't spec-review generic code. -## Step 2 — Classify +## Step 2: Classify Map each file group to surface(s) and routing-table row(s). Show the classification before reviewing so the user can correct it: ``` apps/ingest/src/main.rs (response building) → surface 1 → otlp.md §Request/response shapes, §Failure handling -apps/api/src/services/WarehouseQueryService.ts → surface 3 → semantic-conventions.md §Database +packages/backend/src/services/warehouse/WarehouseQueryService.ts → surface 3 → semantic-conventions.md §Database ``` -## Step 3 — Load minimally +## Step 3: Load minimally -Check the quick table first. If every implicated rule is covered there and no finding looks `critical`, skip spec loading entirely. Otherwise grep-then-read only the routed sections per the loading rule. Track what you loaded — it goes in the report header. +Check the quick table first. If it covers every implicated rule and no finding looks `critical`, skip spec loading entirely. Otherwise grep-then-read only the routed sections per the loading rule. Track what you loaded; it goes in the report header. -## Step 4 — Review +## Step 4: Review -Walk the diff hunks against the loaded normative text. For each candidate finding: quote (to yourself) the normative sentence, take severity from its MUST/SHOULD/MAY word and the section's stability label, and record evidence as `file:line` in the diff. Keep two piles as you go: spec findings, and house-convention observations (Maple-specific rules from `maple-telemetry-conventions` that the spec doesn't mandate). +Walk the diff hunks against the loaded normative text. For each candidate finding, quote the normative sentence to yourself, take severity from its MUST/SHOULD/MAY word and the section's stability label, and record evidence as `file:line` in the diff. Keep two piles: spec findings, and house-convention observations (Maple-specific rules from `maple-telemetry-conventions` that the spec doesn't mandate). -## Step 5 — Findings report +## Step 5: Findings report Present the report **before making any edit**. Format: ``` -## OTel spec review — +## OTel spec review: Spec snapshot: v1.58.0, researched 2026-07-05 (docs/otel-spec/). Re-verify Source links before quoting externally. Surfaces touched: <1 OTLP server | 2 SDK consumer | 3 data consumer | none> -Spec sections loaded: otlp.md §Partial success, §HTTP status codes — retryable matrix +Spec sections loaded: otlp.md §Partial success, §HTTP status codes: retryable matrix | Severity | Spec ref | Finding | Evidence | Source | | critical | otlp.md §Partial success (server MUST rules) | returns 400 for a batch with 3 bad spans instead of 200 + rejected_spans | apps/ingest/src/main.rs:512 | | ... ### House-convention notes (not spec findings) -- — see maple-telemetry-conventions rules/.md +- : see maple-telemetry-conventions rules/.md ### Summary N critical · N warn · N info. Known-gap caveats: . ``` -If the user only wants the review, stop here — the report is a complete deliverable. +If the user only wants the review, stop here. The report is a complete deliverable. -## Step 6 — Apply fixes (only if asked) +## Step 6: Apply fixes (only if asked) -Fix in severity order. *What* to change comes from the cited spec section; *how* to write it defers to the style skills (`maple-rust-style` for `apps/ingest`, `maple-effect-style` for `apps/api`) and `maple-telemetry-conventions` for attribute spellings. After edits, re-verify: the touched app's build/dev command starts cleanly (`bun typecheck` for TS, `cargo check` for `apps/ingest`). A fix that breaks the build is a regression — fix or revert it. +Fix in severity order. *What* to change comes from the cited spec section. *How* to write it defers to the style skills (`maple-rust-style` for `apps/ingest`, `maple-effect-style` for `apps/api`) and to `maple-telemetry-conventions` for attribute spellings. After edits, re-verify that the touched app still builds (`bun typecheck` for TS, `cargo check` for `apps/ingest`). A fix that breaks the build is a regression: fix or revert it. ## Hard rules -- Every finding cites a `docs/otel-spec/` file + section heading and copies that section's inline `Source:` link. If the corpus is silent on a question, report "not covered by the corpus — not reviewed"; never reason from model memory about what the spec says. -- The README known-gap items (Jaeger `uber-trace-id`, W3C Baggage doc status, global metrics cardinality-limit env var, env-var vs declarative-config precedence) are capped at `info` and tagged "unverified"; re-check the canonical URL before asserting them as fact. -- Documented intentional Maple deviations are **not** findings: title-case `"Ok"/"Error"/"Unset"` status storage (one-time translation at the OTLP boundary), the `deployment.environment` dual-emit migration bridge, and `db.statement`→`db.query.text` read-side coalescing (all in README §Maple's three compliance surfaces). Flag one only if the diff *breaks* the documented pattern. -- No edits before the findings report. Never commit, push, or open PRs. Never expand into a whole-repo audit — that's `maple-audit`. +- Every finding cites a `docs/otel-spec/` file + section heading and copies that section's inline `Source:` link. If the corpus is silent on a question, report "not covered by the corpus, not reviewed". Never reason from model memory about what the spec says. +- The README known-gap items (Jaeger `uber-trace-id`, W3C Baggage document status, global metrics cardinality-limit env var, env-var vs declarative-config precedence, semconv naming/registry landing pages) are capped at `info` and tagged "unverified". Re-check the canonical URL before asserting them as fact. +- Documented intentional Maple deviations are **not** findings: Title Case `"Ok"/"Error"/"Unset"` status storage (one-time translation at the OTLP boundary), the `deployment.environment` dual-emit migration bridge, and read-side coalescing of `db.statement`→`db.query.text` and `db.system`→`db.system.name` (all in README § Maple's three compliance surfaces). Flag one only if the diff *breaks* the documented pattern. +- No edits before the findings report. Never commit, push, or open PRs. Never expand into a whole-repo audit; that's `maple-audit`. - Maple-convention issues go under "House-convention notes" pointing at the specific `maple-telemetry-conventions` `rules/*.md`, never mislabeled as spec violations. A line violating both gets a spec finding plus a cross-reference. -- Respect the loading budget: section-scoped reads only, ≤2 spec files by default, never a full file. +- Respect the loading budget: section-scoped reads only, at most 2 spec files by default, never a full file. diff --git a/skills/maple-python-style/SKILL.md b/skills/maple-python-style/SKILL.md index 6f17811819..05289d686e 100644 --- a/skills/maple-python-style/SKILL.md +++ b/skills/maple-python-style/SKILL.md @@ -5,6 +5,10 @@ description: "Python OpenTelemetry style for Maple: module-scope tracers/meters, # Maple Python style +```bash +pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation-logging +``` + Acquire OTel objects at module scope. ```python @@ -19,7 +23,7 @@ orders_submitted = meter.create_counter("orders.submitted", unit="1") ## Bounded work -Prefer decorators for functions with clear boundaries. +Use decorators for functions with clear boundaries. The decorator works on both sync and `async def` functions. ```python @tracer.start_as_current_span("order.submit") @@ -64,15 +68,16 @@ If logs are claimed as OTLP-forwarded, configure all of: - `set_logger_provider(logger_provider)` from `opentelemetry._logs` - Log correlation for existing records, e.g. `LoggingInstrumentor().instrument(set_logging_format=True)` -Preserve existing `logging.basicConfig`, console / file handlers, and log levels. The user's logger keeps working — you're adding an OTLP handler underneath so log lines carry `trace_id` / `span_id` and reach Maple. +Preserve existing `logging.basicConfig`, console / file handlers, and log levels. The user's logger keeps working. The OTLP handler sits underneath so log lines carry `trace_id` / `span_id` and reach Maple. ## Init behavior -Inline the endpoint and ingest key directly in the init module — don't read them from env. The ingest key is project-scoped + write-only (Sentry DSN shaped), so source-level configuration is the right default; env indirection just adds a class of "OTel didn't start because env wasn't set" deploy failures. +Inline the endpoint and ingest key in the init module. Do not read them from env. The ingest key is project-scoped and write-only (shaped like a Sentry DSN), so source-level configuration is the default. Env indirection adds "OTel didn't start because env wasn't set" deploy failures. ```python # telemetry.py import logging +import os from opentelemetry import _logs, metrics, trace from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter @@ -150,7 +155,7 @@ Counters: - `llm.tokens.output` - requests/events/jobs/errors -Use semantic units when the SDK supports them: token counters use `unit="tokens"`. Do not add app-side `llm.cost_usd` pricing metrics for normal LLM calls; Maple estimates cost centrally from provider/model/token data. +Use UCUM units: token counters use `unit="{token}"`. Do not add app-side `llm.cost_usd` pricing metrics for normal LLM calls; Maple estimates cost centrally from provider/model/token data. Histograms: @@ -162,7 +167,7 @@ Avoid raw high-cardinality values in metric attributes. Prefer tenant/org/projec ## FastAPI -Use the native instrumentation rather than replacing request handling with manual middleware. +Use the native instrumentation (`pip install opentelemetry-instrumentation-fastapi`). Do not replace request handling with manual middleware. ```python from fastapi import FastAPI diff --git a/skills/maple-rust-style/SKILL.md b/skills/maple-rust-style/SKILL.md index 2e14f9b494..3e11c4d065 100644 --- a/skills/maple-rust-style/SKILL.md +++ b/skills/maple-rust-style/SKILL.md @@ -5,48 +5,51 @@ description: "Rust OpenTelemetry style for Maple: opentelemetry + opentelemetry_ # Maple Rust style -Use the official `opentelemetry` + `opentelemetry_sdk` crates with `opentelemetry-otlp` (HTTP exporter, not gRPC). Bridge the `tracing` crate via `tracing-opentelemetry` so existing `info!` / `error!` calls flow through OTLP. +Use the official `opentelemetry` + `opentelemetry_sdk` crates with `opentelemetry-otlp` (HTTP exporter, not gRPC). Bridge the `tracing` crate: `tracing-opentelemetry` turns `tracing` spans into OTel spans, and `opentelemetry-appender-tracing` turns `info!` / `error!` events into OTLP log records. ## Cargo.toml ```toml [dependencies] -opentelemetry = "0.27" -opentelemetry_sdk = { version = "0.27", features = ["rt-tokio"] } -opentelemetry-otlp = { version = "0.27", features = ["http-proto", "reqwest-client", "logs", "metrics"] } -opentelemetry-semantic-conventions = "0.27" +opentelemetry = "0.32" +opentelemetry_sdk = { version = "0.32", features = ["trace", "logs", "metrics"] } +opentelemetry-otlp = { version = "0.32", features = ["http-proto", "reqwest-blocking-client", "reqwest-rustls", "trace", "logs", "metrics"] } +opentelemetry-appender-tracing = "0.32" tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } -tracing-opentelemetry = "0.28" +tracing-opentelemetry = "0.33" ``` +Keep the `opentelemetry*` crates on one minor version, and pair `tracing-opentelemetry` with the release built for it (0.33 for 0.32). The builder-default batch processors export from their own thread, so use the blocking reqwest client. The async `reqwest-client` panics there with "there is no reactor running". `reqwest-rustls` is required for HTTPS: `opentelemetry-otlp` builds reqwest without default TLS. + ## Bootstrap -Inline the endpoint and ingest key — they're a project-scoped, write-only token (Sentry-DSN-shaped). +Inline the endpoint and ingest key. The key is a project-scoped, write-only token (shaped like a Sentry DSN). ```rust -use opentelemetry::{global, KeyValue}; -use opentelemetry_otlp::{LogExporter, MetricExporter, Protocol, SpanExporter, WithExportConfig}; -use opentelemetry_sdk::{ - logs::LoggerProvider, metrics::SdkMeterProvider, trace::TracerProvider, Resource, +use opentelemetry::{global, trace::TracerProvider as _, KeyValue}; +use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge; +use opentelemetry_otlp::{ + ExporterBuildError, LogExporter, MetricExporter, Protocol, SpanExporter, WithExportConfig, + WithHttpConfig, }; -use opentelemetry_semantic_conventions::resource::{ - DEPLOYMENT_ENVIRONMENT_NAME, SERVICE_NAME, +use opentelemetry_sdk::{ + logs::SdkLoggerProvider, metrics::SdkMeterProvider, trace::SdkTracerProvider, Resource, }; use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt}; const MAPLE_ENDPOINT: &str = "https://ingest.maple.dev"; const MAPLE_KEY: &str = "MAPLE_TEST"; // set by maple-onboard skill on pairing -pub fn init() -> Result<(TracerProvider, LoggerProvider, SdkMeterProvider), opentelemetry_otlp::ExporterBuildError> { +pub fn init() -> Result<(SdkTracerProvider, SdkLoggerProvider, SdkMeterProvider), ExporterBuildError> { let auth = format!("Bearer {MAPLE_KEY}"); let mut headers = std::collections::HashMap::new(); headers.insert("authorization".to_string(), auth); let resource = Resource::builder() + .with_service_name("orders-api") .with_attributes([ - KeyValue::new(SERVICE_NAME, "orders-api"), - KeyValue::new(DEPLOYMENT_ENVIRONMENT_NAME, std::env::var("DEPLOYMENT_ENV").unwrap_or_else(|_| "development".into())), + KeyValue::new("deployment.environment.name", std::env::var("DEPLOYMENT_ENV").unwrap_or_else(|_| "development".into())), KeyValue::new("vcs.repository.url.full", "https://github.com/acme/orders-api"), KeyValue::new("vcs.ref.head.revision", std::env::var("GITHUB_SHA").unwrap_or_default()), ]) @@ -56,9 +59,9 @@ pub fn init() -> Result<(TracerProvider, LoggerProvider, SdkMeterProvider), open .with_http() .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/traces")) .with_headers(headers.clone()) - .with_protocol(Protocol::HttpJson) + .with_protocol(Protocol::HttpBinary) .build()?; - let tracer_provider = TracerProvider::builder() + let tracer_provider = SdkTracerProvider::builder() .with_batch_exporter(trace_exporter) .with_resource(resource.clone()) .build(); @@ -68,9 +71,9 @@ pub fn init() -> Result<(TracerProvider, LoggerProvider, SdkMeterProvider), open .with_http() .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/logs")) .with_headers(headers.clone()) - .with_protocol(Protocol::HttpJson) + .with_protocol(Protocol::HttpBinary) .build()?; - let logger_provider = LoggerProvider::builder() + let logger_provider = SdkLoggerProvider::builder() .with_batch_exporter(log_exporter) .with_resource(resource.clone()) .build(); @@ -79,7 +82,7 @@ pub fn init() -> Result<(TracerProvider, LoggerProvider, SdkMeterProvider), open .with_http() .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/metrics")) .with_headers(headers) - .with_protocol(Protocol::HttpJson) + .with_protocol(Protocol::HttpBinary) .build()?; let meter_provider = SdkMeterProvider::builder() .with_periodic_exporter(metric_exporter) @@ -87,8 +90,8 @@ pub fn init() -> Result<(TracerProvider, LoggerProvider, SdkMeterProvider), open .build(); global::set_meter_provider(meter_provider.clone()); - let otel_layer = tracing_opentelemetry::layer().with_tracer(global::tracer("orders.api")); - let otel_log_layer = opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge::new(&logger_provider); + let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer_provider.tracer("orders.api")); + let otel_log_layer = OpenTelemetryTracingBridge::new(&logger_provider); tracing_subscriber::registry() .with(tracing_subscriber::EnvFilter::from_default_env()) @@ -119,18 +122,18 @@ async fn main() { ## Bounded business spans via `tracing` -The point of bridging `tracing` is so existing instrumentation works unchanged. Use `#[tracing::instrument]` on bounded async operations: +Bridging `tracing` keeps existing instrumentation unchanged. Use `#[tracing::instrument]` on bounded async operations: ```rust -#[tracing::instrument(name = "order.submit", skip_all, fields(order.id = %order_id))] +#[tracing::instrument(name = "order.submit", skip_all, err, fields(order.id = %order_id))] async fn submit_order(order_id: &str) -> Result<(), Error> { charge_order(order_id).await?; Ok(()) } ``` -`tracing::error!` and `?err` field interpolation will record the exception and set the span status to ERROR via the bridge. +Add `err` to `#[instrument]`. An `Err` return then emits an event with an `error` field, which `tracing-opentelemetry` records as an `exception` event and span status `Error` (both on by default). ## Coexistence -If the project already uses `tracing` with a Honeycomb / Datadog / Jaeger layer, leave it in place — add Maple's `tracing-opentelemetry` layer alongside. Don't strip the existing exporter unless the user asks. +If the project already uses `tracing` with a Honeycomb, Datadog, or Jaeger layer, leave it in place and add Maple's layers alongside. Do not strip the existing exporter unless the user asks.