Repository navigation
bug(mq): storage-limit errors (10002/10023) map to 500 instead of 503 + Retry-After #687
Description
Area: mq — external NATS error mapping
internal/mq/external.go's publishError (called from publish, external.go:559) maps a failed publish to one of two sentinels the API layer treats as retryable: ErrQueueFull (matched by the server's error text for "maximum bytes exceeded" / "maximum messages per subject exceeded") and ErrUnavailable (no answer, connection closed/draining, a stream that no longer matches, jetstream.ErrNoStreamResponse with a confirmed-missing stream). Both map to a 503 with Retry-After in internal/api/ingest.go's publishFailed (case definite: for ErrQueueFull, case errors.Is(err, mq.ErrUnavailable):). Anything that is neither of those falls through to the final, unconditional case: a plain 500 "publish failed".
JetStream's account-resource-limit error (code 10002, "resource limits exceeded for account") and its server-resource error (code 10023, "insufficient resources") are not checked for anywhere in publishError — they are not text-matched into ErrQueueFull, and they are not one of the sentinel error values wrapped into ErrUnavailable. Both are the server telling the client "this is full/over capacity right now," which is exactly the kind of transient, capacity-related condition ErrQueueFull/ErrUnavailable exist to signal — but as written, a publish that fails with either code reaches the client as an undifferentiated 500, with no Retry-After, indistinguishable from a genuine server bug.
- Evidence: inferred — the error codes and their meaning are read from the NATS/JetStream error catalogue;
publishError's lack of a case for them, and the500fallthrough inpublishFailed, are read directly from currentinternal/mq/external.goandinternal/api/ingest.go. - Scenario: an account's resource limits (e.g., total storage across streams) are exceeded, or a server itself is short on resources. A publish to a partition stream fails with
apiErr.ErrorCode == 10002or10023.publishErrordoes not recognize either, so the error returned is the raw*jetstream.APIError, wrapped in neither sentinel;publishFailediningest.gothen answers the client with a flat500, instead of the503+Retry-Aftera client would otherwise get for a full queue or an unreachable broker.
Reproduce/confirm: force an account or server resource-limit condition against a test NATS server (e.g., a very small account byte limit) and publish through ExternalNATS.Publish; inspect the resulting *jetstream.APIError.ErrorCode and confirm it is 10002 or 10023, then trace it through publishError to confirm it falls to the final return err and ultimately to publishFailed's default case.
Found in review of #624.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsBacklog
Activity
feat(mq): external nats broker with sharded, pinned ingest workers (#624