[CO owner] eliminate orchestrator/free provider-family SPOF before gateway release #1915

Activity

1. seonghobae

   changed the title [-]Decision needed: Strix's orchestrator/free SPOF is a 6+ day open, unreviewed risk[/-] [+][CO owner] eliminate orchestrator/free provider-family SPOF before gateway release[/+] on Sep 5, 2026
2. claude commented on Sep 6, 2026

   

   claudeBot

   on Sep 6, 2026 – with Claude

   More actions

   Reproduction 7 days later, larger catalog, same single-family collapse

   This issue's evidence is the 2026-08-30 run where the free pool's discovered routes collapsed to one provider family with 4/4 candidates rejected. Here is the same collapse on 2026-09-06, with the catalog six times larger — so catalog growth has not restored diversity.

   Source: required noema-review on .github#1884@396b4dee, job 101521704729, 17:33:26Z → 18:14:14Z; artifact noema-sidecar-evidence (id 9994541963), contract strix-plain-chat-preflight-v2.

   The gateway itself was healthy — this is a pool-composition failure, not an unreachable-gateway one:

   "gateway": {"attempts": 1, "endpoint": "chat/completions", "finish_reason": "stop", "status": "ready"}
   "candidate_count": 24, "probed_count": 16, "ready_count": 6,
   "deferred_count": 2, "rejected_count": 8, "skipped_count": 4, "target_ready": 8

   All 6 ready routes are nvidia_nim / nvidia_nim_sub — three models, each counted twice because the two NVIDIA accounts are separate credentials on the same provider family:

   model	providers
   deepseek-ai/deepseek-v4-flash-0731	nvidia_nim, nvidia_nim_sub
   deepseek-ai/deepseek-v4-pro-0813	nvidia_nim, nvidia_nim_sub
   meta/llama-3.2-11b-vision-instruct	nvidia_nim, nvidia_nim_sub

   The two independent free sources that could have supplied diversity each failed, in different ways:

   • Bytez never entered the pool at all: provider_discovery_failed provider=bytez code=http_status_500 (first line of the sidecar log — discovery, not inference).
   • OpenRouter produced its only two candidates and both deferred on HTTPError / http_status: 429: cohere/north-mini-code:free, dots-studio/dots-3-note-preview:free.

   The 8 rejects are NVIDIA 404s on retired hosted models (google/gemma-3-12b-it, google/gemma-3-4b-it, both accounts) plus TimeoutErrors (google/gemma-4-31b-it, both accounts).

   target_ready is 8 and only 6 were reached, so the pool was already under its own diversity target before inference began.

   What happened next is the failure this issue predicts. With no other family to fall back to, every attempt against those six routes hit the socket idle timeout — eight measured provider_attempt → provider_attempt_failed … error_type=TimeoutError intervals of 90.1, 90.1, 90.1, 90.2, 90.1, 90.1, 90.0, 90.1 s — through circuit_failure → circuit_opened(reset_seconds=30) → circuit_cleared → retry, then attempts=3 exhausted on the last route and:

   request_failed status=502 code=provider_connection_error
   

   The required gate went fully dark, exactly the "no fallback" outcome in the trade-off section, and it did so without any paid model being needed to diagnose it — the free pool had a healthy gateway and simply had nowhere else to route.

   Two points for the acceptance tracker, offered as measurement rather than as a decision:

   1. The free pool currently has a hard dependency on NVIDIA NIM. Diversity is not merely thin; it is zero — the two NVIDIA credentials are one family. This sits directly against the organization direction that direct NIM communication is a removal target: today, removing NIM from the free pool would leave it empty.
   2. Provider-family diversity should be measurable from the preflight artifact. Every run already emits routes[].provider, so a distinct-family count over status == "ready" is derivable with no new probing, and target_ready already exists as a companion threshold. That would let "free-pool availability" be an observed number in this tracker instead of a per-incident reconstruction.

   I am not proposing a code change here, and specifically not reintroducing paid or direct-provider fallback, per this issue's ownership correction.

   ---

   Generated by Claude Code
3. claude commented on Sep 6, 2026

   

   claudeBot

   on Sep 6, 2026 – with Claude

   More actions

   Refining my own suggestion above: the metric exists, but on the wrong axis and the wrong scope

   In my previous comment I suggested that provider-family diversity "should be measurable from the preflight artifact," implying nothing computes it. That was imprecise, and acting on it as written would mean rebuilding something that partly exists. Three corrections, from the source rather than the artifact:

   1. A diversity metric already exists — in a different report. scripts/ci/contextual_orchestrator_review_launcher.py::_with_discovery_counts emits free_account_diversity and free_pool_account_diversity, gated on FREE_POOL_CREDENTIAL_NAMES (BYTEZ_API_KEY, NVIDIA_NIM_API_KEY, NVIDIA_NIM_API_KEY_SUB, OPENROUTER_API_KEY).

   2. It is scoped to the discovery catalog, not to what inference can actually use. Its input is free_rows = [row for row in rows if row.get("cost_evidence") == "free"], and its own docstring pins the contract as "among all discovered free routes." The strix-plain-chat-preflight-v2 report — the one uploaded as noema-sidecar-evidence when the gate goes dark — is built separately and carries candidate_count / probed_count / ready_count / target_ready / routes[], with no diversity field of any kind. So on the 2026-09-06 run I posted above, catalog-scoped diversity would have looked healthy (OpenRouter contributed two candidates) while the ready set had collapsed to one family. The metric cannot see the failure at the moment the failure matters, and the operator diagnosing the dark gate cannot see the metric.

   3. The axis is accounts, not families — deliberately, and correctly for its own purpose.

   def provider_account(provider_name: str) -> str:
       """Return the independently credentialed provider account identity."""
       return provider_name

   The identity function is right for rate-limit isolation: NVIDIA_NIM_API_KEY and NVIDIA_NIM_API_KEY_SUB were confirmed on 2026-09-03 to have independent rate limits rather than a shared pool, and test_strix_serializes_provider_evidence_per_repository_and_pr records PR-scoped Strix concurrency being restored on exactly that finding. But it makes account diversity a poor proxy for the SPOF this issue tracks. On the run above, an account-diversity count over the ready set reports 2 (nvidia_nim, nvidia_nim_sub) while outage diversity is 1:

   model	nvidia_nim	nvidia_nim_sub
   google/gemma-4-31b-it	TimeoutError 90.1 s	TimeoutError 90.1 s
   deepseek-ai/deepseek-v4-flash-0731	TimeoutError 90.1 s	TimeoutError 90.2 s, then 90.1 s
   deepseek-ai/deepseek-v4-pro-0813	TimeoutError 90.1 s	— (circuit already open)

   Two independently credentialed accounts, same upstream, same 90 s failure, minutes apart. Independent rate limits do not buy independent availability, so the two axes need to stay distinct rather than one standing in for the other.

   So the concrete ask is narrower than what I wrote before: not "add a diversity metric," but give the existing one a family-level companion and evaluate it over status == "ready", then emit it in strix-plain-chat-preflight-v2 alongside ready_count and target_ready. Every input is already in the report — routes[].provider and routes[].status — so this needs no additional probing and no change to what is discovered.

   I am not implementing that here: the axis semantics (what counts as one family, and what a family-diversity floor should be) is an acceptance decision on this issue rather than a mechanical edit, and per this issue's ownership correction I am not the one to declare it. Flagging it as a precise spec for whoever takes it.

   ---

   Generated by Claude Code
4. seonghobae

   added a commit that references this issue on Sep 7, 2026

   docs(gap-baseline): record the dispatch-actor outage and its fix, the…

   ...

   767e740
5. seonghobae

   added

   bugSomething isn't workingSomething isn't working

   priority: mediumNormal-priority or P2 workNormal-priority or P2 work

   on Sep 7, 2026
6. claude commented on Sep 7, 2026

   

   claudeBot

   on Sep 7, 2026 – with Claude

   More actions

   A near-simultaneous success/failure pair — and why the pair cannot be compared

   Two noema-review runs on this repository overlapped almost entirely this morning and ended differently:

   PR	head	window	result
   #1884	767e740c (docs-only)	07:37:15Z → 08:02:07Z (25 min)	success, APPROVE published on the exact head
   #1967	8ca96943 (544-line test + 94-line workflow)	07:35:44Z → 08:08:48Z (33 min)	failure, request_failed status=502 code=provider_connection_error

   The failing run reproduces this issue's shape for the fourth time (artifact noema-sidecar-evidence 10009710244):

   "gateway": {"attempts": 1, "finish_reason": "stop", "status": "ready"}
   "candidate_count": 24, "ready_count": 6, "deferred_count": 2,
   "rejected_count": 8, "skipped_count": 4, "target_ready": 8

   All six ready routes are again nvidia_nim / nvidia_nim_sub — the same three models across the two accounts. 39 provider attempts, 22 recorded failures, 16 TimeoutError against 6 HTTPError, ending 502.

   The useful part is what the pair cannot tell us. The obvious reading is that the single remaining family is marginal rather than dead, and serves a small request while timing out on a larger one. I can't support that from the evidence: noema-review.yml uploads noema-sidecar-evidence only if: failure(), so the succeeding run produced no preflight at all. Its ready_count, its provider mix, and how many attempts it needed are unobservable. Whether the two runs even saw the same pool composition is not something the artifacts can answer.

   So a controlled comparison is impossible by construction, and that is a second instrumentation gap alongside the missing family-diversity field I described in #1915 (comment). Both point the same way: the acceptance signal this tracker needs — "is the free pool healthy right now?" — is only ever emitted when the pool has already failed. A preflight report retained on success as well as failure would make success-vs-failure pairs like this one comparable, and would cost one artifact upload on a path that already builds the report either way.

   Recording this as measurement. I am not proposing a change to what gets probed, and not reintroducing paid or direct-provider fallback.

   ---

   Generated by Claude Code
7. seonghobae commented on Sep 13, 2026

   

   seonghobae

   on Sep 13, 2026

   ContributorAuthor

   More actions

   Current consumer recurrence for the existing provider-availability work: fast-mlsirm#1863 at bc94a2991b8cae10d5bdb35a408610a59e1ea360, Noema run34749709996/job103703885704.

   The GitHub App credential setup succeeded, but review never began because preflight found zero ready routes. Downloaded full artifact10315098630 reports24 candidates,16 probed/rejected,8 skipped:11 HTTP429,4 HTTP404,1 TimeoutError. Sidecar stderr also records Bytez discovery HTTP500. That diagnostic alone is not the fatal cause: the CO owner confirmed pinned414f's existing provider-error test passes and preserves another provider when available. The full no-ready result is the evidence here.

   Next action belongs to this existing CO availability investigation: use the recorded route/status outcomes to validate free-route discovery, rejection classification, skipped-candidate policy and recovery; then verify a real consumer review completes with an allowed free route. Do not substitute a paid route, fabricate a verdict or treat a provider mock as live success. Internal next review:2026-09-14 09:00UTC, justified by the current review delivery failure; no provider restoration date is promised. Consumer source/security review and hosted approval remain incomplete.