Repository navigation
feat(combos): JEV level mode with quota-aware selection (stacked on #6302) - #6348
Closed
SeongwoongCho wants to merge 11 commits into
Closed
SeongwoongCho wants to merge 11 commits into
SeongwoongCho wants to merge 11 commits into
Conversation
…sion models A JEV Combo can name its decision service with `decisionProvider`. Omitted or "jev" keeps the canonical TypeSafe request byte-identical (same URL, env-key fallback, isCanonicalUrl proof, structured criteria), except that a canonical `jev` row's defaultModel may replace jev-latest. Any other id must be a configured `adapter: "jev-decision"` row, e.g. Ollama tev1 at http://127.0.0.1:11434/v1/systemone: its baseUrl is the full endpoint, its model is defaultModel/models[0]/jev-latest, and route options are sent as description strings, which Ollama's System One endpoint requires. Security invariant preserved: TypeSafe credentials stay pinned to the canonical URL. The `jev` id always resolves there (a retargeted `jev` row is still ignored, not treated as a custom destination), and a self-hosted row receives only its own apiKey, or no Authorization header at all. TYPESAFE_API_KEY, JEV_API_KEY and the jev row key are never sent to a custom URL (regression-tested). The row itself is the outbound provider config, so loopback requires its own allowPrivateNetwork; redirects still fail closed. providerOutboundPost gains a caller opt-in, allowLocalCleartextPost, used only by the JEV client, that admits http: solely for a literal loopback/localhost/ private address the row allows; public literals and hostnames keep the HTTPS-only POST gate. Also: decisionTimeoutMs (1000..120000, default 4000) for slow or cold self-hosted models; shared combo validation for unknown/non-decision/non-jev use; management saves and GUI round-trips keep both fields; provider deletion is refused while a combo names the row; every jev-decision row is excluded from catalog gathering and gets the bounded decision connection probe; `ocx combo set --decision-provider/--decision-timeout`; docs, README locales and structure docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…low-ups Cleartext POST gate (src/lib/provider-outbound.ts): allowLocalCleartextPost now admits http: only when the row itself sets allowPrivateNetwork: true (a registry default such as `ollama` no longer counts) and the host is exactly `localhost` or an address literal in a narrow allowlist (127/8, ::1, ::ffff:127.0.0.0/104, 10/8, 172.16/12, 192.168/16, fc00::/7). CGNAT, NAT64, benchmark, link-local, metadata and unspecified addresses never qualify. Every resolved answer must stay inside the allowlist, any applicable proxy (global, provider route, or the DNS-failure degradation) is refused, and an injected executor must be given an address literal. Self-hosted decision rows: the baseUrl path must end in /systemone (checked by combo validation and at request time); a row needs its own defaultModel or models[0] (jev-latest is never sent to a self-hosted host); an apiKey referencing TYPESAFE_API_KEY/JEV_API_KEY or another provider's keychain entry makes the row unusable; option counts outside 2..26 fail open without a round-trip (no_choices / invalid). No gate union member was added. Canonical TypeSafe request is byte-identical to before again: a `jev` row's defaultModel/models are ignored. Explicit decisionProvider "jev" validates exactly like omission and is stored sparsely. A retargeted `jev` row is no longer connection-tested through the generic /models probe with its stored key; it reports not-applicable. rewriteProviderReferences follows combos[*].decisionProvider. Probe wording for disabled/unusable custom rows, CLI bounds from shared constants, CLI reference docs, structure docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A JEV Combo's editor (and the add dialog) now shows Decision service (TypeSafe JEV by default, or any configured adapter "jev-decision" row other than jev, with its endpoint) and Decision timeout (ms), both hidden for other strategies. A JEV save sends decisionProvider and decisionTimeoutMs explicitly, null meaning default, so the dashboard can clear them; switching away from jev drops both from the draft and the payload, matching backend validation. The timeout bounds mirror the server constants and a test asserts they match. The Combos overview lists each JEV Combo's decision service, endpoint and timeout. Create JEV Auto is offered for self-hosted jev-decision rows, keyless included (canonical jev still needs its key), and prefills that row as decisionProvider via ?decisionProvider= on the deep link. Strings in all ten locale catalogs; docs note the dashboard controls. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The decision-service picker now annotates and disables rows the server
would reject or the runtime would skip: disabled, a baseUrl not ending in
/systemone, no defaultModel/models[0], a missing row, or a row that is not
a decision service. A stored but unusable id stays selected with its
reason so opening the editor never rewrites it, and validateComboDraft
reports invalidDecisionProvider before Save sends a request. Create JEV
Auto uses the same usability check, and a deep link naming an unusable
row falls back to TypeSafe.
A jev -> other -> jev switch keeps the draft's decision fields; only JEV
compares or sends them. The timeout bounds and default now live in the
import-free src/combos/jev-decision-contract.ts shared by server and GUI.
Timeout errors carry {min}/{max}, the select and timeout input are tied
to their hints with aria-describedby, and the JEV strategy and
model-notes hints no longer name TypeSafe, in all ten locales.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The management API (and so the CLI) accepted a decisionProvider row the runtime would always skip: a disabled row, or one with no defaultModel and no models[0]. The save succeeded and every decision silently fell back. The dashboard already blocked both; the server now matches it via a save-time requireUsableDecisionService option, passed only by the combo PUT route. Config-file load stays lenient so disabling a referenced row never makes the config unloadable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A non-pacing PATCH /api/providers could retarget a row that a JEV combo names as its decisionProvider, e.g. switch its adapter away from jev-decision or move its baseUrl off /systemone. The save succeeded, the combo then failed its load-time checks, and the next reload salvaged the whole combos entry away; until then every decision fell back. The POST and bulk-editor routes already validate the complete candidate config; PATCH only did so when pins were touched. An editor PATCH on a row some combo names as decisionProvider now runs the same full candidate validation, before the await-free commit (400) and again under the mutation lock (409 on a racing change). Disabling the row, pacing-only patches, and the existing pin validation are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A JEV Combo with decisionQuotaSignals: true tells the decision model how much subscription quota each candidate target has left, so it steers away from nearly exhausted accounts. Off (the default) the decision request is byte-identical for TypeSafe and self-hosted services. The signal is read synchronously from the cached provider quota report rows behind `ocx provider quota` (Codex pool aggregate, else the active account or key) and never probes; rows older than 30 minutes, windows whose reset passed, and targets without data send nothing. The worst of the 5h/weekly/monthly meters and a matching model-family window sets the tier (<70 healthy, <90 limited, else nearly exhausted). Self-hosted services get one clause inside each option string plus instructions.quota (the placement measured to move tev1); TypeSafe gets a structured quota object per criterion. A body that overflows the request cap only because of quota is sent without it, and the decision log keeps only per-tier target counts and the picked tier. Validation, sparse storage, management preserve-on-omit and CLI `--decision-quota <on|off|->` follow decisionProvider/decisionTimeoutMs. The JEV section of providers-and-adapters.md moves to providers/jev.md to stay under the structure line budget. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The JEV combo editor gets a "Consider remaining account quota" checkbox next to Decision service and Decision timeout. The dashboard sends decisionQuotaSignals explicitly (false clears it) for JEV combos only, and the Combos overview marks quota-aware JEV rows. Copy in every locale. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A JEV Combo with decisionMode: "level" asks the decision model only to classify how demanding the next call is (trivial, routine, hard, deep, agentic_heavy, agentic_light) and picks the target and effort from a per-level candidate list in decisionLevels. Measured with a self-hosted tev1:4b on a 99-task labelled set and a 60-task held-out set, level mode picked an adequate target/effort 93% of the time on both, against 75%/82% for route mode, with about a third of the input tokens. - The level question is one plain choice question over the configured levels with the eval-validated default descriptions; state is buildJevState without operator notes, and no target or quota text is sent. The transport (endpoint, credential pinning, cleartext policy, deadline, no-redirect, bounded response) is now exchangeJevDecision, shared with route mode, which stays byte-identical. - Selection is synchronous: the level's candidates in order, restricted to the route-mode eligibility set; with decisionQuotaSignals the cached quota tiers order the level stably (healthy or unknown, then limited, then nearly exhausted). No usable candidate tries decisionFallbackLevel (default routine), then fails open; a failed decision fails open with the existing gates. - Validation, sparse normalization, management preserve-on-omit and CLI --decision-mode/--decision-levels/--decision-fallback-level follow the other decision fields. The decision log gains bounded level and levelPath fields. - While a quota-aware level-mode Combo exists, an unref'd, import-free background timer refreshes the cached quota rows about every five minutes through the unforced fetchProviderQuotaReports. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The JEV section gains a Decision mode selector (route or level). Level mode shows the stored per-level candidate lists and the effective fallback level read-only; levels stay editable through the config file, the CLI and the API. The dashboard sends only decisionMode (null for route) and never decisionLevels or decisionFallbackLevel, so the server keeps them across a save; level mode is offered only when levels are stored. The overview marks level-mode Combos, and the quota checkbox hint explains that level mode applies quota in the proxy instead of sending it. New strings are translated in every locale. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Provider id migrations re-point decisionLevels candidates with the targets they name, so a renamed provider no longer leaves a config that fails to load. - The quota warmer now runs every 12 to 15 minutes, serves any quota-aware JEV Combo (route mode reads the same rows), and always re-arms after a tick, including one that joined a flight from before a stop and start. It keeps probing every configured provider: the report publish replaces the whole cached row set, so a narrowed refresh would erase other providers' rows; this is now documented. - A stale level candidate (target removed, effort no longer allowed) is refused with the CLI fix in the message. The dashboard flags such candidates before Save, offers Clear stored levels in route mode, and warns before a save that leaves the JEV strategy and drops the levels. - decisionFallbackLevel is refused without decisionLevels, and a management PUT that clears the levels no longer carries it. - exchangeJevDecision hands prepare only the model and criteria shape. - A golden fixture recorded from 91cc62a4d pins the route-mode request bytes for TypeSafe and self-hosted, with and without quota. - Docs warn that level descriptions reach the decision service and that quota tiers come from the provider's displayed row. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Deterministic PR hygiene checks passed. |
Contributor
⏳ DRAFT
What to do
Review readiness checklist
0/4 boxes ticked. This pull request was already a draft. Its draft status will be preserved after every issue above is resolved. |
3 of 7 tasks
Contributor
Author
Merged
3 tasks done
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional quota signals to JEV decisions (route mode, soft preference) and a level mode where tev classifies one of six demand levels and ocx selects from per-level candidate lists using quota-aware deterministic ordering, configured fallback level, and fail-open behavior. A background quota warmer runs every 12–15 minutes and probes every configured provider. GUI adds a mode selector, quota toggle, read-only levels table, stale-candidate guidance, and clear-level controls. Adds CLI flags and documentation. Route-mode requests remain byte-identical (golden fixture test).
Performance evaluation
Measured against five targets × two efforts (10 options): gpt-6-luna [low,max], gpt-6.1-sol [low,xhigh], gpt-6-astra [low,xhigh], claude-sonnet-5-5 via Cursor [low,xhigh], and claude-opus-5-5 [low,xhigh]. The decision endpoint was a self-hosted Ollama 0.35
tev1:4bendpoint.Evaluation sets: (A) 99 author-labelled tasks spanning six demand levels and five task characters, including tool-result turns, >500-character prompts, delegated-subagent prompts, Korean, and terse prompts; (B) a 60-task held-out set independently written by a separate agent that never saw the level descriptions (10 per level, 12 Korean). Labels are author judgments of which model:effort options are adequate under a “cheapest adequate” policy, not ground truth. A hit means the selected option is in that judged-adequate set.
Route mode on set A selected hard tasks correctly 6/17 and agentic-heavy 4/10; misses were under-provisioning (24/25 selected low effort, p≈0.18–0.34). Level mode reached hard 12/17 and agentic-heavy 10/10; remaining misses were level misclassification (hard work read as routine), not candidate selection. Adding per-task-type capability/cost tiers made the model confidently choose cheap “good” options for hard work (hard 1/17), so that experiment was dropped. Low-confidence escalation threshold was cross-validated over 20 random half splits (tuned 0.30–0.33; held-out 76%→89%); this was motivation, not shipped behavior.
Quota experiment: in route mode, quota text per option plus an instruction moved a hard task from gpt-6-astra to claude-sonnet-5-5 when OpenAI was marked ≥90% used (p 0.30→0.35; weak effect). In level mode with OpenAI nearly exhausted: hard/agentic_heavy selected sonnet:xhigh, deep selected opus:xhigh, and routine selected opus:low; trivial/agentic_light remained luna:low because their lists were OpenAI-only (list order decides), deterministically as designed.
Caveats: labels are policy judgments; evaluation used one decision model (tev1:4b); TypeSafe hosted service was not live-tested with the level question. Quota tiers use the provider's displayed row (active account / pool total), not the per-request routed account.
Verification
bun run test:changed— 29,911 passed, 51 skipped, 22 failed across 1,492 files. All 22 failures are in 8 files (cli-help,native-codex-toggle,native-grok-toggle,serving-runtimes,responses-canonical-nonstream,service-ownership-state,service-sqlite-home,service). Re-run in isolation (bun test <file>) on this branch (47581f7) and on the base feat(combos): self-hosted JEV decision service, level mode with quota-aware selection, configurable decision wording #6302 head (304bd4a), the results are identical: 3 failures remain on both — 2 incli-helpand 1 inservice— the documented systemd-in-container failures from AGENTS.md; the other 19 pass alone on both trees (parallel-run interference).bun run typecheck— passed.bun run structure:check— passed.bun run skill:surface:check— passed.bun run privacy:scan— passed.bun run build:gui— passed.Screenshots (separate assets branch, not committed to this PR):
Checklist
🤖 Generated with Claude Code
Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
Required local validation passed; commands, results, and any full-suite exception are documented.
I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.