Graceful Shutdown & Resource Cleanup

Problem

Forcefully killing the application can leave NATS messages unacknowledged, ClickHouse batches half-written, or metrics unflushed, potentially leading to data inconsistencies.

Proposed Solution

Implement a signal listener (SIGTERM/SIGINT) that triggers a coordinated shutdown sequence. This will stop the API listeners first, then flush all pending data buffers to ClickHouse, and finally close database connections.

Alternatives Considered

None. Maybe some database crash recovery but we should not depend on that nor use it as a graceful shutdown method.

Additional Context

None

Closing — implementation has shipped.

The coordinated shutdown sequence requested in the issue body is in cmd/wavehouse/main.go:378-393:

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh

shutCtx, shutCancel := context.WithTimeout(context.Background(), time.Duration(cfg.Server.ShutdownTimeout)*time.Second)

_ = ingestStream.Stop(shutCtx)            // flush ingest buffers
if err := srv.Shutdown(shutCtx); err != nil { ... }    // stop API listener
if err := promSrv.Shutdown(shutCtx); err != nil { ... } // stop Prometheus exporter

The earlier otelShutdown(ctx) (~line 123) flushes telemetry providers (traces/metrics/logs) on the same shutdown path.

Matches the proposed-solution requirements:

✅ SIGTERM/SIGINT signal listener
✅ Bounded shutdown context (configurable timeout via cfg.Server.ShutdownTimeout)
✅ Stops API listeners first (srv.Shutdown)
✅ Flushes pending data buffers (ingestStream.Stop)
✅ Flushes observability (otelShutdown)

Reopen if a specific resource is leaking — e.g. if you find the ClickHouse driver isn't being closed via this path, that's a follow-up.

chore: claude code native improvements (#147)

Navigation Menu

Repository navigation

Graceful Shutdown & Resource Cleanup #46

Problem

Proposed Solution

Alternatives Considered

Additional Context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Graceful Shutdown & Resource Cleanup #46

Description

Problem

Proposed Solution

Alternatives Considered

Additional Context

Activity

EricAndrechek commented on May 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions