Skip to content

feat(heartbeat): add an embedded TCP/UDP push transport for constrained devices - #301

Merged
fclairamb merged 58 commits into
mainfrom
batch/2026-09-01
Sep 2, 2026
Merged

fclairamb merged 58 commits into
mainfrom
batch/2026-09-01

Conversation

@fclairamb

Copy link
Copy Markdown
Owner

Batch of four specs from specs/todos/, each implemented on its own and gated on an independent completeness audit before being archived to specs/done/2026/09/.

Embedded TCP/UDP heartbeat push transport

Heartbeat checks could only be fed over HTTPS, which shuts out the fleet the primitive fits best — microcontrollers, cellular modems driven by AT commands, battery sensors, legacy PLCs — because TLS costs tens of KB of RAM, a CA store and a correct clock. This adds two small listeners (one port number, 4001, on both TCP and UDP, off by default) speaking a one-line, netcat-debuggable protocol in two forms:

  • SP1 — SP1 <org>/<check> <token> — plaintext token, convenience default.
  • SP2 — SP2 <org>/<check> <ts> <ctr> <mac> — HMAC-SHA256 keyed with the check's token, no secret on the wire, ~80 bytes.

Replay protection is a strictly-increasing per-check counter enforced by the database itself (a single conditional upsert), so an old datagram fails and even the latest datagram cannot be replayed. The new per-check require_hmac option rejects SP1 outright; the dashboard pairs the toggle with a token-rotation nudge, because a check that ever accepted SP1 has leaked the key.

Beats carry an optional annotation (a status word plus logfmt key=value pairs). Numeric values land in the result's metrics jsonb, so battery volts and RSSI become first-class time series charted on the check page. An annotation never invalidates a beat — a firmware typo must not make a healthy device look dead.

Security posture is stated explicitly in the code and the docs: SP1 is replayable to mask an outage (that is what require_hmac closes), SP2 authenticates the message rather than the transport, and UDP source addresses are spoofable and are never treated as identity. UDP answers nothing at all on any failure — a nonexistent org is byte-identical to a bad MAC — and never replies with more bytes than it received.

New docs page: Embedded / push monitoring. Enabling the ports is a deployment decision (k8s Service / LB) and is documented, not automated.

gRPC checks: real phase breakdown and request metadata

connection_time_ms was fiction — grpc.NewClient is lazy, so the metric measured struct construction while the real DNS + TCP + TLS cost hid inside rpc_time_ms. The connection is now established eagerly and instrumented, reporting genuine dns_time_ms / connect_time_ms / tls_time_ms / rpc_time_ms, with failures classified by the phase that failed instead of collapsing into one opaque RPC error. Checking a service the server never registered now says so in words rather than surfacing a raw NotFound.

gRPC checks also gain metadata and encrypted secretMetadata, so a health endpoint behind an authenticating proxy can be checked at all — and the dashboard form is no longer a stub: TLS skip-verify, timeout and both metadata editors are now reachable.

Uptime report emails: trends, latency and worst-first ordering

The scheduled report showed one period in isolation. It now carries period-over-period deltas, a response-time block, average incident duration, and a check table sorted worst-first with a per-day availability strip — so a broken check can no longer sit invisibly at the bottom of a 50-row alphabetical table.

Considerable care went into the degenerate cases, because this is exactly where a digest starts lying: zero is never green, a missing baseline omits the delta entirely rather than rendering "±0.00%", a monitor down all period gets a plain factual sentence instead of a grid of zeros and a celebrated response-time "improvement" measured on error responses, and days before a check existed are gray rather than red. No MTBF, and no plan-upsell content in report emails by explicit product decision.

Dashboard polish

The "Getting started" checklist rows had no background of their own, so the card's tint bled through and completed rows looked identical to pending ones. Rows now carry an explicit surface with a faint emerald wash when done.


Changelog entries

feat(heartbeat): add an embedded TCP/UDP push transport so constrained devices can beat without TLS
feat(heartbeat): add the SP1/SP2 beat wire protocol, annotation parsing and the per-check require_hmac option
feat(dash0): show the embedded beat examples and the require_hmac toggle on heartbeat checks
feat(grpc): report a real DNS/connect/TLS/RPC phase breakdown and classify failures by phase
feat(grpc): support request metadata and encrypted secretMetadata, and complete the dashboard form
feat(uptimereport): add period-over-period trends, a response-time summary and a worst-first check table with a per-day strip
feat(heartbeat): expire an SP2 replay counter a week after the last beat
feat(dash0): give the Getting started checklist rows their own backgrounds and a done state
fix(checks): a heartbeat config PATCH no longer destroys the check's ping token
fix(uptimereport): render a zero previous-incident count instead of ""
fix(dash0): render the TCP beat one-liner's newline escape

Notes for review

  • Two bugs were found and fixed in passing, neither in this batch's own new code: a heartbeat config PATCH silently destroyed the check's ping token (the public patch path is REPLACE, and Validate's regenerated token was discarded because it ran on a deep copy), and Data.PreviousIncidentCount carried omitempty, so the commonest healthy report — no incidents this month or last — printed <no value> where it meant 0.
  • No new migration. The SP2 counter lives in the existing generic state_entries store rather than a dedicated table, matching the consolidation already made for OAuth storage in batch/2026-06-23: realtime v2 (WebSocket), scheduler lanes & fairness, discovery, maintenance windows, availability API (41 specs) #89. Strict monotonicity stays in the SQL WHERE clause on both dialects, never in Go — a read-modify-write there would reintroduce the replay race under exactly the concurrency a retried UDP datagram produces. Net DDL for this cycle is nothing, so there is no 018 file.
  • Counters expire on a sliding 7-day window, refreshed by every accepted beat, so a beating device never expires its own counter and a check that went quiet stops occupying a row forever. That sweep is a soft delete and the guard ignores deleted_at, so a swept counter keeps refusing replays — pinned by a test per dialect. The one thing a future change should weigh is recorded in the code and the spec: if a hard purge of soft-deleted state entries is ever added, a clockless (ts=0) device silent for over a week could have an old beat replayed at it.
  • Dev-database note: an earlier commit in this branch briefly added an 018_v0_22_0 migration that was withdrawn. A local dev database that ran it carries a bun_migrations row for a file that no longer exists. The runner tolerates that and boots normally, but reset any dev DB that ran it before a future 018 lands, rather than using migrate repair.

gRPC checks now report where the time actually went, and where a failure
actually happened. The check used to report a `connection_time_ms` that
measured nothing: the gRPC client is lazy and dials nothing when it is
created, so the real DNS, TCP and TLS cost was hidden inside the health
RPC's own timing. The connection is now established and measured phase by
phase before the RPC is sent, and a check reports `dns_time_ms`,
`connect_time_ms`, `tls_time_ms` (only where each phase actually applies),
`rpc_time_ms` and `total_time_ms`. A failure names the phase it died in, so
a hostname that does not resolve, a refused port, a broken TLS handshake and
an unhealthy service are no longer the same `rpc error: …` message — and a
refused connection now carries the reachability marker that lets an incident
run a path trace. Checking a service name the server never registered, the
most common gRPC health misconfiguration, says exactly that instead of
returning a raw `NotFound`. A service answering `NOT_SERVING` is still
reported down *with* its measured latency, so it can be seen slowing down
before it drains.

gRPC checks can also send request metadata, which is what lets you monitor a
health endpoint sitting behind an authenticating proxy: `metadata` stays in
the public, searchable configuration for routing or tenant headers, and
`secretMetadata` is encrypted at rest and never returned by the API — for
`authorization` bearers and API keys.

The dashboard form finally exposes the whole check. "Skip TLS certificate
verification" (with a warning about what it costs you), both metadata
editors, and the per-check timeout are all editable from the browser instead
of being API-only, and the documentation covers the TLS modes, named service
versus overall server health, the metadata rules and the metrics the check
reports. The old `keyword` / `invertKeyword` options — which matched a
substring of the serving-status enum the check already evaluates — keep
working but are deprecated and no longer offered in the form.
…nds, latency summary, worst-first check list, degenerate-case guards
… trends, latency summary, worst-first check list, degenerate-case guards
… trends, latency summary, worst-first check list, degenerate-case guards
… trends, latency summary, worst-first check list, degenerate-case guards
…trends, latency summary, worst-first check list, degenerate-case guards
Give each getting-started step row its own background (bg-card, matching
the done-state emerald wash on the all-set banner) instead of letting the
card's primary tint bleed through undifferentiated. Also drop the alerts
CTA's forced outline variant when there is no test-alert button to defer
to, and add a done-state statusPage CTA label.
Fixture now mixes a pending alerts row (no test-alert button, default CTA
variant) and a done statusPage row (ctaDone label) to showcase both
optional niceties, and the section description documents the new
row-background pattern.
fclairamb and others added 20 commits September 2, 2026 00:09
Recovered the gate run that a dead session left unrecorded: backend build,
golangci-lint, the full Go suite, dash0 build + eslint + vitest, the docs
build and the heartbeat-push Playwright file all pass against the tree with
origin/main merged in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t its own table

Repo precedent decides this: issue #89 deliberately collapsed three dedicated
OAuth tables into `state_entries`/`user_tokens`, which already backs
notification state, password resets, invites and OAuth grants. A whole table
for one integer per check cut against it.

The counter now lives at `heartbeat_counter/<checkUID>`, org-scoped, valued
`{"lastCounter": N}`. Two things improve for free: `organization_uid` cascades
on org delete (the standalone table never did), and no DDL is needed at all —
`state_entries` has existed since 001, so the `heartbeat-push-counters` section
comes out of the still-unreleased consolidated 018 rather than being reverted
by a new migration. 018 held nothing else, so its net final DDL is now empty
and all four files go with it.

What did NOT change is the part that matters. `TryAdvanceHeartbeatCounter` is
still ONE conditional upsert with the strictly-greater test in the WHERE
clause; it does not go through `GetStateEntry` -> compare -> `SetStateEntry`,
because that generic setter is unconditional and would let two concurrent
beats both read the old value and both be accepted. A retried UDP datagram is
the normal case here, so that race is the replay the counter exists to stop.
The 40-goroutine Postgres test and the sqlite same-value race are re-pointed at
the new store rather than dropped, for exactly that reason.

Three storage properties are now asserted rather than assumed, on both engines:

- `expires_at` stays NULL and is forced back to NULL on every advance, so
  `DeleteExpiredStateEntries` can never sweep a live device's counter. The
  tests run the sweeper instead of only reading the column.
- the unique constraint carries no `deleted_at` predicate, so a soft-deleted
  row still owns the slot and still gates the comparison. That is the fail-safe
  direction: deleting the entry must not rewind the check. A winning advance
  un-deletes the row rather than leaving live state the reader would have to
  lie about, and `GetHeartbeatCounter` deliberately ignores `deleted_at` so it
  reports what the guard actually enforces.
- a forged MAC still writes no row at all — now asserted against
  `state_entries` directly, not just via the counter accessor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gate re-run after moving the SP2 replay counter into state_entries:
make build-backend, lint-back (0 issues) and the full Go suite (exit 0,
zero FAIL) all green, including the 40-goroutine Postgres concurrency
test and the new state_entries storage-contract tests on both engines.
dash0 and the docs site were not touched by this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…at ingest — add a minimal TCP/UDP push transport
The counter was stored with no expiry at all, so a check that stopped
beating kept its state_entries row forever. It now carries a sliding
models.HeartbeatCounterTTL window, refreshed on every winning advance:
a device that is actually beating never expires its own counter, and
only a check quiet for a week is swept.

Sweeping stays safe because DeleteExpiredStateEntries is a soft delete
and the advance ignores deleted_at, so a swept counter keeps refusing
replays -- pinned by a test in each dialect, each with a positive
control proving a real device can still advance.
@fclairamb
fclairamb enabled auto-merge (squash) September 2, 2026 01:26
fclairamb and others added 3 commits September 2, 2026 03:41
The onboarding checklist's new statusPage `ctaDone` label ("View status
pages", spec 2026-09-01) renders an `<a>` to /orgs/$org/status-pages on
the org dashboard whenever the step is done — and the seeded `test` org
ships with a status page, so it is done from the first page load.

`getByRole("link", { name: "Status Pages" })` matches accessible names by
case-insensitive substring, so "View status pages" now matches it too and
every unscoped call site died with a strict-mode violation (8 tests across
these two files). The old pending label, "Create a status page", did not
match — the plural is what made the collision appear.

Scope the four call sites to the sidebar, which is what they meant all
along and what status-pages.spec.ts and docs-links.spec.ts already do.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Version

The clear half of "pins a family, persists it, and clears back to auto"
submits from /checks/<uid>/edit, which already matches the waitForURL
regex it then waits on — so the wait resolved instantly and the getCheck
that follows raced the PATCH, reading back the pre-clear "ipv6". It
failed roughly two runs in three in isolation.

The create half of the same test already waits for networkidle, which is
why only the clear half flaked. Predates this batch; found by running the
full suite as a batch gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fclairamb
fclairamb merged commit ecde564 into main Sep 2, 2026
12 checks passed
@fclairamb
fclairamb deleted the batch/2026-09-01 branch September 2, 2026 02:47
@fclairamb fclairamb mentioned this pull request Sep 2, 2026
fclairamb added a commit that referenced this pull request Sep 2, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.22.0](v0.21.2...v0.22.0)
(2026-09-02)


### Features

* **heartbeat:** add an embedded TCP/UDP push transport for constrained
devices ([#301](#301))
([ecde564](ecde564))
* **heartbeat:** add an embedded TCP/UDP push transport so constrained
([ecde564](ecde564))


### Bug Fixes

* **deps:** update github.com/dop251/goja digest to 43234fa
([#300](#300))
([9c9b3e2](9c9b3e2))
* **deps:** update module github.com/aws/aws-sdk-go-v2/service/s3 to
v1.110.0 ([#298](#298))
([2482111](2482111))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant