Skip to content

feat(mq): warn at startup when embedded JetStream store dir is non-persistent #99

Description

@jfwoods

In cmd/wavehouse standalone mode, with no volume mounted, embedded JetStream
silently writes to ./data/nats/jetstream inside the container's writable overlay
layer — visible only as Store Directory: "data/nats/jetstream" in startup logs,
with no warning that the path isn't persistent. Every container recreation
(Portainer redeploy, image rebuild, healthcheck restart) wipes JetStream state:
in-flight events between an ingest POST and the Bento→ClickHouse flush are lost,
SSE/WS gap-fill stops bridging restarts, and NATS retention (WH_MQ_MAX_BYTES_GB,
default 50) accumulates inside /var/lib/docker instead of the operator's chosen
SSD tier. The reference compose mounts a volume so the docs imply persistence is
required — but a fresh user wiring their own compose can easily miss it.

Proposed Solution

When the resolved mq.embedded_dir is a relative path and nothing is mounted at
it (or it resolves onto the container's overlay rather than a volume), log a startup
WARN. One line that catches the mistake before any data is lost.

Alternatives Considered

  • Make a persistent dir mandatory (hard-fail if not mounted) — too aggressive;
    ephemeral JetStream is legitimate for throwaway/dev stacks.
  • Document-only — easy to miss; a runtime warning catches the case the docs already
    imply.

Additional Context

From nas-observability WHissues.md, 2026-05-06. Mirrors the volume pattern in
deployments/compose/standalone.yaml (established by #42, Docker Compose:
Standalone vs. Clustered). Adjacent to #84 (JetStream SyncAlways durability) —
same internal/mq/embedded.go footgun territory, different axis: #84 is about the
sync-mode/throughput tradeoff, this is about the store dir silently being
ephemeral.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ingestIngest pipeline (Bento, batching, DLQ)area/observabilityMetrics, logs, traces, health, profilingenhancementNew feature or request

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions