Skip to content

fix(failover): fail over on relayed upstream failures wearing 4xx statuses - #607

Merged
SantiagoDePolonia merged 1 commit into
mainfrom
feat/failover-400
Jul 28, 2026
Merged

SantiagoDePolonia merged 1 commit into
mainfrom
feat/failover-400

Conversation

@SantiagoDePolonia

@SantiagoDePolonia SantiagoDePolonia commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #605.

OpenCode Zen (and other aggregator-style providers) can relay a transient failure of their upstream as a 4xx client error — e.g. 400 "Error from provider (Console Go): Upstream request failed". Failover previously ran only on 5xx, 429, and model-availability errors, so a request whose primary model had configured failover targets still returned the raw 400.

ShouldAttemptFailover now also fires when the error message blames an upstream: it must contain "upstream" plus failure phrasing (failed / error / unavailable / timed out / timeout), regardless of the status code the provider chose. Messages that merely mention an upstream without failure phrasing (e.g. "parameter tools is not supported by the upstream provider") still do not fail over, so genuine validation errors don't sweep targets.

No new configuration: this follows the same message-heuristic pattern as the existing model-availability trigger, keeping the default behavior right without a knob.

Changes

  • internal/gateway/failover.go: new upstream-failure trigger; the three inline fragment-scan loops are extracted into named fragment lists (modelUnavailableFragments, upstreamFailureFragments, retiredModel404Fragments) checked via one containsAny helper.
  • internal/gateway/failover_test.go: table cases for the exact reported OpenCode message, upstream timeout/unavailable, and a no-failover guard for upstream capability errors.
  • docs/features/failover.mdx: "When It Runs" documents the new trigger.

User-visible impact

  • Requests failing with a relayed upstream error now try their failover targets instead of returning the provider's 4xx.
  • Existing sweep semantics apply: if every failover target also fails, the client receives the last target's error rather than the primary 4xx.
  • Provider note: motivated by OpenCode Zen, but the trigger is provider-agnostic and applies to any translated route with failover rules.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Expanded automatic failover support for transient upstream failures reported with HTTP 4xx responses.
    • Added handling for model unavailability and retired-model availability messages.
  • Documentation
    • Updated failover documentation to describe upstream-relayed 4xx failure conditions that can trigger fallback.

…tuses

Aggregator-style providers can relay a transient failure of their own
upstream as a client error: OpenCode Zen returns 400 "Upstream request
failed" when its upstream breaks, so requests with configured failover
targets still died with the raw 400.

ShouldAttemptFailover now treats a message that blames an upstream
("upstream" plus failure phrasing) as failover-eligible regardless of
the status code, mirroring the existing model-availability heuristic.
The three inline fragment-scan loops are extracted into named fragment
lists checked via one containsAny helper.

Closes #605

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 28, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 4c3d510e-034d-4d0b-8dce-462abf8fde77

📥 Commits

Reviewing files that changed from the base of the PR and between 953d600 and 34b7798.

📒 Files selected for processing (3)
  • docs/features/failover.mdx
  • internal/gateway/failover.go
  • internal/gateway/failover_test.go

📝 Walkthrough

Walkthrough

Failover detection now centralizes message matching and recognizes selected upstream transient failure messages relayed with 4xx statuses. Tests cover matching and non-matching upstream errors, and documentation describes the new trigger.

Changes

Failover detection

Layer / File(s) Summary
Message matching primitives
internal/gateway/failover.go
Shared message fragments and a containsAny helper centralize case-insensitive failover matching.
Failover decision and coverage
internal/gateway/failover.go, internal/gateway/failover_test.go, docs/features/failover.mdx
ShouldAttemptFailover handles model-unavailable, upstream transient, and retired-model 404 messages; tests and documentation cover upstream failures relayed as 4xx responses.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • ENTERPILOT/GoModel#451: Both changes modify failover predicate logic in internal/gateway/failover.go and its tests.

Poem

I’m a bunny guarding the route,
Catching upstream troubles en route.
Four-oh-oh? Hop, don’t delay—
Try the next model right away!
- 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: failover now triggers on relayed upstream 4xx failures.
Description check ✅ Passed The description covers the why, what changed, tests, and user impact, even though it uses different section headings than the template.
Linked Issues check ✅ Passed The changes satisfy #605 by extending failover to relayed upstream failures that were previously returned as 400s.
Out of Scope Changes check ✅ Passed The docs, helper refactor, and added tests all support the failover fix and do not appear unrelated.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/failover-400

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mintlify

mintlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
gomodel 🟢 Ready View Preview Jul 28, 2026, 9:50 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge with no actionable defects identified.

The new predicate reaches the intended relayed upstream failures, retains the established failover behavior, and does not alter the existing model-availability or status-based checks.

T-Rex T-Rex Logs

What T-Rex did

  • Ran the contract-validation sequence and observed the initial state: eligible=false, zero target calls, and no failover.
  • After triggering failover by requesting OpenAI GPT-5, the results changed to eligible=true with one target call to openai/gpt-5 and did_failover=true, with target-ok.
  • Re-ran capability-validation and confirmed it produced eligible=false with zero target calls.
  • All captured commands exited with exit code 0.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "fix(failover): fail over on relayed upst..." | Re-trigger Greptile

@SantiagoDePolonia
SantiagoDePolonia merged commit 01d6f30 into main Jul 28, 2026
21 checks passed
@mintlify

mintlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
gomodel 🟡 Building – Jul 28, 2026, 9:50 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

This branch was successfully deployed

1 active deployment
staging - docs — 34b77988 Deployed Jul 28, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: configurable failover reasons

3 participants