refactor: move DLQ settings to policy (per-table/role) and general DLQ + NATS/JetStream cleanup

Summary

Currently, the DLQ enabled/disabled flag is a global setting handled at the process level. Following discussion in PR #182, we want to refactor DLQ configuration to be more granular and policy-driven.

Goals

Move DLQ settings to policy — allow DLQ behavior to be configured per table and per role, enabling fine-grained control over which events get routed to the DLQ and under what conditions.
Leave DLQ on by default — DLQ routing should be active out of the box; policy can be used to opt out for specific tables/roles.
General DLQ refactoring and cleanup — as part of this work, clean up existing DLQ handling logic and address any remaining rough edges.
General NATS/JetStream refactoring and cleanup — resolve outstanding NATS/JetStream technical debt introduced or surfaced during the ingest pipeline refactor in PR refactor: full api --> ingest --> clickhouse --> dlq refactor #182.

Background

This was identified in PR #182 (full API → ingest → ClickHouse → DLQ refactor) during review of StartIngestWorker. The current implementation does not consult a dlq.enabled flag, which is acceptable for now (DLQ is always on), but longer-term we need per-table and per-role DLQ policy support.

See: #182 (comment)

PR refactor: full api --> ingest --> clickhouse --> dlq refactor #182 — refactor: full api → ingest → clickhouse → dlq refactor
Closes feat(ingest): Prevent infinite retry loops on permanent ClickHouse failures (Poison Pills) #91 (begun in PR refactor: full api --> ingest --> clickhouse --> dlq refactor #182 — poison pill / DLQ isolation)

🔗 Related PRs

#122 - fix(ingest): break infinite-Nak loop on permanent delete errors [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Reprioritization (launch-gated pass): re-anchored P1→P2 — moving DLQ settings to policy is a breaking refactor; valuable pre-1.0 but not alpha-blocking. Rationale on #149.

The DLQ switch this asked for (per table and on by default) exists in the tenant settings directory. dlq.enabled is seeded true, a dlq.tables.<table>.enabled override sits under it, and settings.Store.DLQFor (internal/settings/store.go) resolves the two; the ingest worker reads the result per tenant and table (#508). It lives in settings rather than the policy file. Per-role routing was not built: queued events carry no caller role, and no use case for it has come up. Specific DLQ follow-ups are tracked in their own issues, for example #237 and #532.

Navigation Menu

Repository navigation

refactor: move DLQ settings to policy (per-table/role) and general DLQ + NATS/JetStream cleanup #188

Summary

Goals

Background

Related

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

refactor: move DLQ settings to policy (per-table/role) and general DLQ + NATS/JetStream cleanup #188

Description

Summary

Goals

Background

Related

Activity

coderabbitai commented on May 27, 2026

EricAndrechek commented on Jun 4, 2026

EricAndrechek commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions