Sidecar preflight: shared escalation budget favors catalog-order-early candidates over later ones #1458

Background

Flagged by Devin Review on ContextualWisdomLab/.github#1449 (ADR-0005). Verified against the actual
code in ContextualWisdomLab/.github#1452's _preflight_review_agents
(scripts/ci/contextual_orchestrator_review_launcher.py): candidates are probed in catalog order (a
single for agent in agents: loop), and REVIEW_PREFLIGHT_MAX_ESCALATIONS = 4 is one shared counter
consumed first-come-first-served. Once 4 candidates have used an escalation attempt, every later
candidate that would otherwise qualify (a Trigger B budget-too-small signature) is instead recorded
escalation_budget_exhausted immediately, with no escalation attempt of its own — even if that later
candidate would have succeeded at the escalated budget.

Correction (2026-08-30, a follow-up Devin Review pass on the ADR caught this description was
imprecise): catalog order is deterministic, not random, but it is not purely alphabetical.
Verified directly against build_zdr_prioritized_catalog's actual sort key
(contextual_orchestrator_review_policy.py):

eligible_rows.sort(
    key=lambda row: (
        _COST_EVIDENCE_RANK[_cost_evidence(row)],
        0 if is_zdr_model(str(row["provider"]), model=str(row["model"]), zdr_endpoints=zdr_endpoints) else 1,
        str(row["provider"]),
        str(row["model"]),
    )
)

The real sort key is (cost_evidence_rank, zdr_attested_rank, provider, model): cost-evidence tier
first (constant within orchestrator/free, since every eligible row is already free), ZDR-attested
status second (a ZDR-attested candidate always sorts before a non-attested one, regardless of whether
require_zdr is set), and (provider, model) alphabetically only as the tie-breaker within each
same-cost/same-ZDR-status group. So the practical bias this issue describes is really: ZDR-attested
candidates (and, within a tier, alphabetically-early ones) are systematically favored for the shared
escalation budget over non-attested and alphabetically-later candidates — not simply "alphabetically
early wins," though within one ZDR tier that reduces to the alphabetical case.

Why this isn't a cheap fix

Considered and rejected during triage (verified reasoning, not asserted): the escalation budget being
shared and bounded (REVIEW_PREFLIGHT_MAX_ESCALATIONS = 4, not unbounded "escalate everyone") is
itself required — ADR-0005's own round-3 finding (already fixed) established that an unconditional
"one retry per candidate" blows Layer 1's 180s healthz-readiness budget. Given a fixed, smaller-than-pool
escalation budget, some selection policy must decide which candidates get the available slots — the
current cost/ZDR/alphabetical order, round-robin, random shuffling, priority-by-family, or anything
else. None of these removes the structural trade-off: whichever candidates don't get a slot are, by
construction, unable to prove they would have succeeded with more budget. Reordering changes which
subset of candidates is favored per run, not whether some subset is always favored over the rest — so
it is not a clear correctness improvement, only a change in which arbitrary policy governs slot
allocation, and this org's own ADR-0005 explicitly rejects picking such a policy without real
justification ("no heuristics or rules of thumb"). No real telemetry exists yet on which candidates
actually need escalation more often in practice, which would be the evidence a principled reordering
policy (e.g., weighted by past escalation-success rate, or round-robin across provider families) would
need.

Ask

If real hosted-run telemetry accumulates (the escalation/attempt/trigger fields ADR-0005 already
requires both layers to emit) showing a specific, evidenced ordering or family bias worth correcting,
revisit with that evidence. Options to consider then: round-robin escalation across provider families
instead of strict catalog order, a per-family (not just a global) sub-budget, or increasing
REVIEW_PREFLIGHT_MAX_ESCALATIONS if real data shows the current cap of 4 is too tight relative to how
often Trigger B fires in practice. Not blocking ContextualWisdomLab/.github#1449/#1452 — documented
there as a known, accepted, tracked limitation (same treatment as #1454/#1455).

Cross-ref: ContextualWisdomLab/.github#1449 (ADR-0005), #1452 (implementation), #1454, #1455
(sibling accepted limitations from the same design).