feat(api): admin DLQ replay endpoint (re-validate + re-publish dead-lettered events)

Area: api · ingest — operability gap · found via nas-observability dogfooding (jfwoods)

Expected: after fixing the cause of a dead-letter (e.g. a schema migration), re-process the DLQ'd events without manual NATS surgery.

Actual: there's a read-only /v1/dlq/stats (internal/api/router.go:153) but no replay path. Once events land in dlq.<table> the only options are accept the data loss or write a one-off NATS consumer to re-publish them.

Real-world impact (nas-observability): a producer added six columns without migrating the table; WaveHouse's strict validator dead-lettered every event for ~23 min (it looked like a silently-dead stream — SSE open, /v1/query healthy, dashboard "LIVE"; see the schema-rejection-metric gap in #94). After the ALTER TABLE + /v1/schema/refresh fix, 12 stranded DLQ messages had no replay path and were effectively lost.

Scope: an admin POST /v1/dlq/replay?table=… that re-validates and re-publishes dead-lettered events, with the obvious guards (dedupe-idempotent, batch/limit, dry-run count). Pairs with the existing /v1/dlq/stats.

Related: DLQ stats endpoint (router.go:153); transient-retry buffer #50 (pre-DLQ retries, not post-DLQ replay); DLQ settings → policy #188.

From WHissues.md (nas-observability dogfooding, jfwoods, 2026-05-08); validated against 0f8826c on 2026-06-04.

🔗 Related PRs

#122 - fix(ingest): break infinite-Nak loop on permanent delete errors [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

docs(ingest): fix the drain runbook, the split exception, and lock cl…

fix(ingest): retry ClickHouse outages instead of dead-lettering (#619)

A related behaviour, observed on v0.2.1, that affects any replay design here:

With dedupe on, an event's id is committed when the event is accepted. If that row later dead-letters (say the table lacked a column, or a value didn't fit its type), the id stays committed. When the producer fixes the schema and resends the same event, the resend is answered as a duplicate and the row never lands. With only /v1/ops/dlq/stats available, there is also no way to re-drive the parked row.

This is the current behaviour. It should either be treated as an issue and changed or documented as a deliberate decision. Possible changes:

release (or mark) the id when its row is parked, so a corrected resend is accepted; or
have replay (this issue) clear the id for the rows it re-publishes, and have a DLQ delete clear it too.

If it's deliberate, the dedupe docs should say that a dead-lettered event can't be resent under the same id.

Navigation Menu

Repository navigation

feat(api): admin DLQ replay endpoint (re-validate + re-publish dead-lettered events) #237

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat(api): admin DLQ replay endpoint (re-validate + re-publish dead-lettered events) #237

Description

Activity

coderabbitai commented on Jun 4, 2026

EricAndrechek commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions