Skip to content

bug(mq): storage-limit errors (10002/10023) map to 500 instead of 503 + Retry-After #687

Open

Description

@EricAndrechek

Area: mq — external NATS error mapping

internal/mq/external.go's publishError (called from publish, external.go:559) maps a failed publish to one of two sentinels the API layer treats as retryable: ErrQueueFull (matched by the server's error text for "maximum bytes exceeded" / "maximum messages per subject exceeded") and ErrUnavailable (no answer, connection closed/draining, a stream that no longer matches, jetstream.ErrNoStreamResponse with a confirmed-missing stream). Both map to a 503 with Retry-After in internal/api/ingest.go's publishFailed (case definite: for ErrQueueFull, case errors.Is(err, mq.ErrUnavailable):). Anything that is neither of those falls through to the final, unconditional case: a plain 500 "publish failed".

JetStream's account-resource-limit error (code 10002, "resource limits exceeded for account") and its server-resource error (code 10023, "insufficient resources") are not checked for anywhere in publishError — they are not text-matched into ErrQueueFull, and they are not one of the sentinel error values wrapped into ErrUnavailable. Both are the server telling the client "this is full/over capacity right now," which is exactly the kind of transient, capacity-related condition ErrQueueFull/ErrUnavailable exist to signal — but as written, a publish that fails with either code reaches the client as an undifferentiated 500, with no Retry-After, indistinguishable from a genuine server bug.

  • Evidence: inferred — the error codes and their meaning are read from the NATS/JetStream error catalogue; publishError's lack of a case for them, and the 500 fallthrough in publishFailed, are read directly from current internal/mq/external.go and internal/api/ingest.go.
  • Scenario: an account's resource limits (e.g., total storage across streams) are exceeded, or a server itself is short on resources. A publish to a partition stream fails with apiErr.ErrorCode == 10002 or 10023. publishError does not recognize either, so the error returned is the raw *jetstream.APIError, wrapped in neither sentinel; publishFailed in ingest.go then answers the client with a flat 500, instead of the 503 + Retry-After a client would otherwise get for a full queue or an unreachable broker.

Reproduce/confirm: force an account or server resource-limit condition against a test NATS server (e.g., a very small account byte limit) and publish through ExternalNATS.Publish; inspect the resulting *jetstream.APIError.ErrorCode and confirm it is 10002 or 10023, then trace it through publishError to confirm it falls to the final return err and ultimately to publishFailed's default case.

Found in review of #624.

Related: #624, #613.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ingestIngest pipeline (Bento, batching, DLQ)bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions