Skip to content

fix(runtime): probe the fetched account inventory before hard-coded fallback models - #5803

Open
Aaron-Ben wants to merge 3 commits into
apache:mainfrom
Aaron-Ben:fix/connection-test-probe-model
Open

Aaron-Ben wants to merge 3 commits into
apache:mainfrom
Aaron-Ben:fix/connection-test-probe-model

Conversation

@Aaron-Ben

Copy link
Copy Markdown

Summary

With nothing enabled, the connection test probed the provider's first hard-coded fallback model even when the account's own model list had already been fetched. For OpenRouter that is anthropic/claude-sonnet-5, so a credential check billed a premium model the user never chose, and a key with a model allow-list could be reported as failing although it works.

The probe now picks, in order: an enabled model (unchanged) → the fetched account inventory, :free variants first → the provider fallback. A shipped snapshot (modelSource: 'fallback') or a connection without an inventory keeps the old fallback order, per #1584.

The rule now lives in @maka/core as connectionTestModelId, so the Runtime probe and the settings page share one owner. The Test connection button uses it to name the model it will probe in its tooltip, before a request is spent.

Fixes #5493

Verification

  • Runtime provider-conformance: 3 new probe tests; the two inventory tests fail on main with actual: 'anthropic/claude-sonnet-5' and pass here. File 42/42; full @maka/runtime suite 3676 pass, 0 fail.
  • Core llm-connections: new connectionTestModelId test, 15/15.
  • Runtime Host connection-effect-coordinator + connection-effects-protocol: 34/34.
  • typecheck for core, runtime and desktop; Biome on changed files; check:renderer-architecture; check-locale-hygiene --base upstream/main; knip for desktop and ui — all pass.
  • Not run: full npm test, E2E. The tooltip has no automated test; checked by hand (screenshot below).

Review focus

  • :free preference: OpenRouter's free variants are rate-limited, so a probe could hit a 429 on a busy free model. I followed the issue's expected behavior here; happy to drop the preference if you'd rather keep the first inventory entry.
  • Out of scope: a new connection still pre-enables the provider fallback list (feat(runtime-host): complete M5 production cutover #2420), so with default selections the probe still uses the first enabled model. This PR only changes the nothing-enabled path.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code — root-cause analysis, implementation and tests; reviewed and verified by me. Both commits carry Generated-by: Claude Code.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No
before after

Move the probe-model rule into @maka/core so the Runtime probe and the settings preview share one owner.

Refs apache#5493

Generated-by: Claude Code
@github-actions github-actions Bot added the effort/M Under 500 readable lines label Sep 28, 2026

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 60f0814. The change moves connection-test model selection into core for both Runtime and the settings preview. With no enabled model, it now probes a fetched account inventory before the shipped fallback, preferring IDs ending in :free; it adds provider and core tests.

I found one P2 in the new inventory path (inline): it can select an explicitly non-chat model and report a valid connection as failed instead of reaching a chat-capable candidate. This should be resolved before merging. The 57 focused core/runtime tests pass locally, and the fetched-main merge-tree and diff check are clean. Only the label check is currently visible for this head; no current-head hosted test is green. The full local build stops on UI component-contract type errors outside the changed files. I did not run a live provider account or packaged Desktop.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

Comment thread packages/core/src/llm-connections.ts Outdated
: enabled;
const accountInventory = listed
? [
...discoveredIds.filter(isNoCostModelId),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Filter account-inventory probe candidates for chat capability. This path collects every fetched ID and moves every :free ID ahead of the fallback without checking the model's capabilities.chat/output modalities. The same codebase explicitly marks non-text/image-only models as unsupported for chat (isModelExplicitlyUnsupportedForChat), but testConnectionStrict sends a chat probe to this selected ID and never tries the next candidate. With no enabled models and a fetched inventory [image:free (chat:false), chat (chat:true)], a direct probe selected image:free; a valid credential can therefore be shown as a failed connection. Please skip explicitly non-chat inventory entries before selecting the probe model and cover that case in a regression test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The account-inventory candidates now drop entries isModelExplicitlyUnsupportedForChat rejects before the :free ordering. Enabled ids are left as they are, since those are the user's own choice to test.
  • Regression tests for your exact shape ([image:free (chat:false), chat (chat:true)]) in provider-conformance and core llm-connections; both fail on 60f0814 with acme/image:free and pass now. The core test also covers an audio-only :free entry, and a list with no chat-capable entry falling back to the provider fallback.
  • To share the check without an import cycle, isModelExplicitlyUnsupportedForChat moved beside ModelInfo in llm-connections.ts; model-catalog re-exports it, so its existing callers are unchanged.

Local: core 920/920, runtime 3677/3677, runtime-host connection suites 34/34, typecheck for core/runtime/runtime-host/desktop, and check:renderer-architecture --base upstream/main --strict-base pass.

The probe sends one chat request, so an image- or audio-only entry sorted first would report a valid credential as failed. isModelExplicitlyUnsupportedForChat moves beside ModelInfo so the probe can share it without an import cycle; model-catalog re-exports it for existing callers.

Refs apache#5493

Generated-by: Claude Code

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit be46b8f. This follow-up filters fetched account-inventory candidates that are explicitly unsupported for chat before choosing the one-shot connection probe. It moves the existing chat-capability predicate beside ModelInfo, re-exports it for catalog callers, and adds core/runtime regressions for image-only and audio-only entries.

The P2 I reported on the previous head is addressed: with an image-only :free entry ahead of a chat-capable model, the new selector probes the chat model. I found no additional substantiated P0-P3 issue in the new diff and its affected paths. The core and runtime builds and 81 focused tests pass locally; the fetched-main merge-tree and diff check are clean. No current-head hosted checks are visible yet, so this is not a green merge gate. I did not run a live provider account, packaged Desktop, or the full local suite; the earlier full local build stopped on unchanged UI component-contract type errors.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/M Under 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Connection test probes a hard-coded premium model when the connection has no enabled model (OpenRouter → anthropic/claude-sonnet-5)

2 participants