Skip to content

Retain exhausted RabbitMQ deliveries and confirm retry handoffs - #105

Draft
niemyjski wants to merge 1 commit into
feature/rabbitmq-broker-verificationfrom
feature/rabbitmq-delivery-recovery
Draft

niemyjski wants to merge 1 commit into
feature/rabbitmq-broker-verificationfrom
feature/rabbitmq-delivery-recovery

Conversation

@niemyjski

@niemyjski niemyjski commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

With Automatic acknowledgements, classic retries return only to their source queue, and retry or terminal transfers are confirmed before acknowledging the original. Recovery and shutdown preserve delivery ownership for both classic and quorum queues.

Breaking change: exhausted deliveries without a dead-letter destination are retained instead of silently discarded. This can stop consumption and grow backlog: provision quarantine and capacity limits, or explicitly enable DiscardOnDeliveryLimit for discardable work. Handlers must tolerate duplicates. Previously accepted invalid direct options now fail at construction, including retry limits below -1, invalid recovery/heartbeat intervals, and a terminal exchange equal to the source. Required-delivery combinations are also validated there. Changing the explicit queue type after construction fails before declaration. Shutdown waits are bounded; handlers that ignore cancellation may continue afterward. FireAndForget remains the default and still acknowledges before processing.

Broker upgrade impact: RabbitMQ 4.3 makes quorum priorities strict, removes guaranteed service for lower-priority traffic, and defaults an absent priority to 4. RabbitMQ.Client 7.2.2 omits zero on the wire. Audit mixed-priority workloads when upgrading; these broker changes are separate from this PR's acknowledgement fixes.

Validation: The rebased aggregate builds and passes formatting. Nine separate priority/routing cases passed on 3.13.7, 4.2.5, and 4.3.6 without skips. All updated-head Linux checks passed. The push run passed 390 tests with five expected strict-quorum version skips. The full broker/TLS suite targets 4.2.5; dedicated 4.3.6 coverage is limited to priority/routing.

Verification and implementation details

The version matrix compares fresh brokers. A rolling upgrade with existing queue data was not exercised.

Classic retries use confirmed mandatory routing to the original queue; quorum retries use broker requeue. Both confirm application terminal transfers before source acknowledgement. Broker-managed replication and at-least-once dead-lettering remain quorum-only. Replacement publisher channels recheck delayed-exchange support and redeclare their topic. Native delayed retries and per-queue consumer timeout retain their RabbitMQ 4.3+ restrictions.

The original prefetch settings, delay, and assertion are preserved, with an added nonempty-delivery assertion. The rolling-restart test retains its original loss-rate assertion and additionally reconciles every confirmed message ID. Abrupt raw-consumer recovery has a separate named test. New tests put infrastructure guards before Arrange and use Arrange–Act–Assert. Existing XML documentation and version guidance are preserved. This latest update only rebases this PR onto the added compatibility tests; it adds no delivery-recovery implementation changes.

Required routing, required dispatch, and broker-delayed-delivery options are opt-in. Strict dispatch requires Automatic acknowledgements and a separately provisioned terminal destination. A confirmed transfer can be replayed if source acknowledgement is interrupted. FireAndForget can still lose an in-flight delivery during cancellation.

Shared constants retain delivery-count and original-message-ID wire names. Handoffs strip publisher-owned UserId, CC/BCC, and delay controls while preserving application metadata. Handler-owned body copies remain necessary because handlers may outlive callback buffers. The small ProcessProbe executable verifies delayed work survives publisher exit; it is not shipped as library code. Relevant queues and exchanges require configure/read/write permissions, with no new broker-wide privilege.

This remains the largest PR because acknowledgement, recovery, and shutdown share channel ownership. Optional required-delivery APIs are the clearest future separation point. Further moving existing methods into files would add churn without reducing behavior under review.

Merge after #104 and before #100.

@niemyjski
niemyjski added this pull request to stack #107 September 25, 2026 20:21
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 6923527 to 962b7f7 Compare September 25, 2026 20:39
@niemyjski niemyjski changed the title Prevent RabbitMQ delivery loss during retries and recovery Retain exhausted RabbitMQ deliveries and prevent retry loss Sep 25, 2026
@niemyjski niemyjski changed the title Retain exhausted RabbitMQ deliveries and prevent retry loss Prevent Automatic retry loss and retain exhausted deliveries Sep 25, 2026
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from 962b7f7 to f34aedc Compare September 25, 2026 21:29
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from f34aedc to b7c1a1f Compare September 25, 2026 21:43
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from b7c1a1f to 2d7f928 Compare September 25, 2026 21:49
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 2995cf3 to d6aca4a Compare September 26, 2026 02:20
@niemyjski niemyjski changed the title Prevent Automatic retry loss and retain exhausted deliveries Retain exhausted RabbitMQ deliveries and confirm retry handoffs Sep 26, 2026
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch 2 times, most recently from 6bcdb4d to c170bb3 Compare September 29, 2026 02:04
@niemyjski
niemyjski force-pushed the feature/rabbitmq-delivery-recovery branch from c170bb3 to 1528a24 Compare October 2, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant