Custom OpenAI-compatible provider 401 "Invalid or missing API key" floods Sentry #3671

Source

Sentry: TAURI-RUST-DHM (self-hosted, tauri-rust)
Events: 5,636 · Users affected: 1 · First seen: 2026-06-14 · Last seen: 2026-06-14 · Release: openhuman@0.57.40 · Platform: Windows

Symptom

kiro API error (401 Unauthorized): {"error":{"message":"Invalid or missing API key","type":"authentication_error"}}

domain=llm_provider · operation=native_chat · failure=non_2xx · model=claude-sonnet-4.5 · provider=kiro

Breadcrumbs show the same outbound chat-completions URL repeating, with memory_tree::extract::llm transport failure, retrying in 250ms (attempt 2/3) and [memory::jobs] terminal failure … reason=Some("transient") — i.e. memory-tree extraction and memory background jobs keep firing through the broken credential, producing thousands of identical events from one user in a few hours.

"kiro" does not exist anywhere in the codebase. It's a user-configured custom OpenAI-compatible provider (endpoint kiro.unix86.us.kg/v1/chat/completions) where the bearer key is missing or invalid.

Root cause

The provider HTTP-error gate matrix in src/openhuman/inference/provider/compatible.rs:1456-1542 (and its sibling in ops.rs::api_error, line ~956) treats 401/403 as Sentry-actionable for any non-OpenHuman-backend provider:

1. is_backend_auth_failure(name, status) — only matches provider == openhuman_backend (publishes SessionExpired, skips Sentry). "kiro" doesn't match → falls through.
2. is_budget_exhausted_http_400 / is_custom_openai_upstream_bad_request_http_400 / is_provider_access_policy_denied_http_403 / is_provider_config_rejection_http — none match a plain 401 with an authentication_error body.
3. should_report_provider_http_failure(401) returns true (only 429 is in TRANSIENT_PROVIDER_HTTP_STATUSES) → report_error to Sentry with failure=non_2xx.

The justifying comment at ops.rs:925-927 says:

401/403 from other providers (OpenAI, Anthropic, …) still go to Sentry — those mean a misconfigured API key, which is actionable.

In practice this is the same anti-pattern that produced TAURI-RUST-1T (backend session-expiry, 5,414 events from one user) and TAURI-RUST-52S / #3354 (Cohere 401 "no api key supplied", 8,775 events from one user): a deterministic user-state error gets hammered by retry loops (memory-tree extract, memory jobs, cron) and floods Sentry. A misconfigured BYO API key is actionable for the user, not for OpenHuman engineering — Sentry has no remediation path.

Solution

Mirror the established pattern (is_backend_auth_failure, is_provider_config_rejection_http, is_provider_access_policy_denied_http_403): classify 401/403 from a non-backend provider with an OpenAI-style auth-error body as user-config state and demote it from Sentry to an info! log. The error still propagates via anyhow::bail!(message) so the chat / UI surface is unchanged — only the Sentry report is dropped.

Concrete plan

1. Add is_byo_provider_auth_failure_http(provider, status, body) in src/openhuman/inference/provider/ops.rs:

   • status ∈ {401, 403}, AND
   • provider != openhuman_backend::PROVIDER_LABEL (backend keeps its publish_backend_session_expired branch), AND
   • body matches one of: "type":"authentication_error", "invalid api key", "invalid or missing api key", "missing api key", "no api key supplied" (the last to subsume fix(embeddings): Cohere embed floods Sentry with 401 'no api key supplied' #3354's defense-in-depth wording).

2. Add log_byo_provider_auth_failure(operation, provider, model, status) info-log helper, matching the shape of log_provider_config_rejection / log_provider_access_policy_denied_http_403.

3. Wire the new branch into both Sentry report sites:

   • compatible.rs::chat_completions (line ~1485), between is_backend_auth_failure and is_budget_exhausted_http_400.
   • ops.rs::api_error (line ~956), in the same order.

4. Defense-in-depth: add "invalid api key" / "invalid or missing api key" / "no api key supplied" to the ApiKeyMissing classifier arm in src/core/observability.rs:266-270 so any higher-layer re-report (agent / web_channel domain) is also caught.

5. Out of scope, follow-up: a credential-cooldown mechanism (after the first 401/403 from a provider, mark the credential invalid for the process / until the user re-edits it, and short-circuit the memory-tree + memory-jobs retry loops). That fixes the root cause of the volume (loops hammer a known-bad credential), but is cross-cutting and belongs in its own issue.

Scope

Provider-scoped: backend 401/403 keeps its existing SessionExpired branch (unchanged). Third-party providers that 401/403 with a body that doesn't look like an auth-error envelope (e.g. weird gateways returning 401 on quota / geo-block) still reach Sentry — the gate is body-shape-anchored, not blanket-status.

Reproduces on

upstream/main (0.57.40 and later) — compatible.rs gate matrix unchanged.

Related

• fix(embeddings): Cohere embed floods Sentry with 401 'no api key supplied' #3354 — sibling, Cohere embed 401 "no api key supplied" flood (closed, same class).
• fix(observability): classify OpenHuman/Embedding/streaming backend 'Invalid token' 401 as SessionExpired (TAURI-RUST-4P0 + 4K5 + 1EE) #2786 — sibling, backend session-expiry 401 flood that introduced is_backend_auth_failure + publish_backend_session_expired.
• fix(providers): reasoning-v1 model name sent to DeepSeek API which rejects it (regression) #2079 / fix(providers): no per-model temperature guard — 'only 1 is allowed' errors #2076 / fix(providers): requests for non-existent model names (deepseek-v4-pro, claude-opus-4-7) — no validation or stale catalog #2202 — config-rejection demotion pattern this fix mirrors.

Acceptance criteria

• Repro gone — A 401/403 with an authentication_error body from a non-backend provider no longer produces a Sentry event (verified via core::observability test or recorded fixture).
• Backend session-expiry unchanged — openhuman_backend 401/403 still publishes SessionExpired and still skips Sentry (existing test stays green).
• Non-auth 401/403 still reported — A 401 with a non-auth-envelope body (e.g. quota / geo-block phrasing) still reaches Sentry (new test asserts this).
• Regression safety — Unit tests in src/openhuman/inference/provider/ops_tests.rs cover the new predicate and both wire-in sites.
• Diff coverage ≥ 80% — fix PR meets the changed-lines coverage gate.