Skip to content

bug(mq): embedded NATS installs its own SIGINT handler — races main shutdown, panics, and zeroes e2e coverage #287

Closed

Description

@EricAndrechek

What happens

natsserver.Server.Start() installs a process-wide signal handler unless opts.NoSigs is set (nats-server v2.14.0 server.go:948 → signal.go). Its SIGINT path is:

case syscall.SIGINT:
    s.Shutdown()
    s.WaitForShutdown()
    os.Exit(0)

WaveHouse's own graceful shutdown (cmd/wavehouse/main.go:421) runs concurrently on the same SIGINT, and main.go:227 already defers embeddedMQ.Close() → server.Shutdown(). So every SIGINT is a three-way race:

  1. NATS handler wins → os.Exit(0) mid-drain — skips all of run()'s defers (dedup.Close, cache.Close, chConn.Close), truncating graceful shutdown. Exit 0, so nobody notices.
  2. Main's path wins cleanly → correct shutdown.
  3. Both interleave → double Server.Shutdown() → panic: close of nil channel in shutdownEventing (events.go:1907) → exit status 2.

Observed

Mode 3 hit a local make ci run today (tree = main + docs-only changes). The e2e orchestrator SIGINTs the instrumented binary to flush coverage; the panic exit skips the Go coverage atexit hooks, so:

2026/06/05 20:18:27 → SIGINT-ing WaveHouse (flushes coverage)...
2026/06/05 20:18:27   wavehouse exit: exit status 2
...
  e2e             0.0%  >=60  ✗      ← only covmeta written, no covcounters
==> Coverage gate FAILED: e2e
panic: close of nil channel

goroutine 89 [running]:
github.com/nats-io/nats-server/v2/server.(*Server).shutdownEventing(0x45fb95fad008)
	nats-server/v2@v2.14.0/server/events.go:1907 +0x134
github.com/nats-io/nats-server/v2/server.(*Server).Shutdown(0x45fb95fad008)
	nats-server/v2@v2.14.0/server/server.go:2569 +0x3c
github.com/nats-io/nats-server/v2/server.(*Server).handleSignals.func1()
	nats-server/v2@v2.14.0/server/signal.go:52 +0x318

Mode 1 has likely been silently truncating shutdowns all along (the Trapped "interrupt" signal NATS log line is visible in every e2e shutdown).

Fix

NoSigs: true in the embedded server options (internal/mq/embedded.go) — an embedded server must never own process signals; WaveHouse's shutdown path is the single owner and already calls Close().

Activity

  1. added
    bugSomething isn't working
    area/ingestIngest pipeline (Bento, batching, DLQ)
    on Jun 6, 2026
  2. added
    area/apiHTTP handlers, routing, middleware
    area/infraCI, build, deploy, Docker, release
    breaking-changeBreaking change to public API, CLI, or config
    on Jun 6, 2026
  3. moved this from Backlog to In progress in WaveHouse Task Boardon Jun 6, 2026
  4. moved this from In progress to Ready in WaveHouse Task Boardon Jun 8, 2026
  5. moved this from Ready to Backlog in WaveHouse Task Boardon Jun 9, 2026
  6. moved this from Backlog to Ready in WaveHouse Task Boardon Jul 1, 2026
  7. moved this from Ready to In progress in WaveHouse Task Boardon Jul 1, 2026
  8. self-assigned this
    on Jul 1, 2026
  9. taitelee commented on Jul 1, 2026

    @taitelee
    Member

    This issue was fixed in PR #290, where the embedded NATS servers were configured to now start with NoSigs: true in both internal/mq/embedded.go and internal/testutil/natsjs.go, so nats-server no longer installs its own process-wide SIGINT handler and WaveHouse's graceful shutdown path is the single lifecycle owner. So now there's no more os.Exit(0) skipping run()'s defers, and no more double-Server.Shutdown() panicking with close of nil channel (this also restores the e2e coverage flush that was zeroing out make ci). Both shutdown paths additionally WaitForShutdown() after Shutdown() so the process can't exit while JetStream is still tearing down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/apiHTTP handlers, routing, middlewarearea/infraCI, build, deploy, Docker, releasearea/ingestIngest pipeline (Bento, batching, DLQ)breaking-changeBreaking change to public API, CLI, or configbugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions