feat(config): consolidate all time knobs into settings.timeouts; make --timeout a real run watchdog

[viraatc]

What does this PR do?

Consolidates every global time knob into settings.timeouts and makes --timeout a real whole-run watchdog.

settings:
  runtime:
    min_duration_ms: null            # ONLY allowed for num-samples-to-issue calculation in POISSON mode 
    max_duration_ms: null            # workload cap: bounds perf-phase ISSUING; normal end, valid report
  timeouts:                          # every global wait/deadline; null = unlimited (0 never means unlimited;
                                     # drain budgets accept an explicit 0 = give up immediately)
    run_timeout_s: null              # --timeout; whole-run watchdog -> INTERRUPTED artifacts + non-zero exit
    service_ready_timeout_s: 30.0    # metrics/event-logger startup
    warmup_drain_timeout_s: 240.0    # per-phase post-issuing drains
    performance_drain_timeout_s: null
    accuracy_drain_timeout_s: null
    metrics_drain_timeout_s: null    # tokenization drain; expiry FAILS the run (complete: false + non-zero exit)
  client:
    worker_initialization_timeout: 60.0   # endpoint-client internals stay on client (unchanged)

Changes

• --timeout / top-level timeout: was consumed nowhere (silent no-op) -> now settings.timeouts.run_timeout_s, a whole-run watchdog armed from setup through the metrics drain; firing writes INTERRUPTED artifacts, skips scoring, exits non-zero.
• settings.drain block, flat settings.service_ready_timeout_s, and every 0 = unlimited sentinel deleted; hard cutover (extra=forbid), no back-compat shims; templates/examples/docs/tests migrated. One convention everywhere: null = unlimited; drain budgets additionally accept an honest 0 (zero budget, give up immediately); run_timeout_s rejects 0.
• min_duration_ms/max_duration_ms are workload durations on settings.runtime, not timeouts. The --duration alias is deleted — --runtime.min-duration-ms is the one spelling (poisson only, requires an explicit target_qps; explicit --num-samples wins; both unset = one dataset pass). The MLPerf ruleset path keeps its internal duration fields.
• Expired metrics_drain_timeout_s now fails the run instead of exiting 0 with partial ISL/OSL buried in the summary; artifacts written first.
• Ctrl-C is part of the same contract, with one behavior: a single process-level SIGINT handler (SigintGovernor, shared by compliance-audit runs) stops the session gracefully — the aggregator drains, artifacts land state: interrupted / complete: false, events.jsonl flushed, exit 130. Teardown is bounded by a fixed 30 s grace: if the metrics drain has not finished, the service children are SIGTERMed (the aggregator writes a best-effort INTERRUPTED snapshot) then SIGKILLed — a wedged drain can never hang the abort, and no second keystroke is needed. Repeat ^C is a logged no-op, so runners that forward the terminal's group SIGINT (uv run delivers a single ^C twice) need no special handling. Escalation is timeout-driven, not keystroke-driven (design cue: aiperf).
• Every ^C delivery window is defined: during setup (immediate abort, exit 130, no artifacts), mid-run (graceful), during the metrics drain (graceful, grace-bounded), during sync finalization (raises immediately — never silently swallowed), between audit phases (refuses to start the next phase), and during post-measurement accuracy scoring (the report is rewritten interrupted/complete: false before it is persisted — an interrupted run is an invalid run; its artifacts only expose the partial metrics).
• Artifact precedence, documented and tested: result_summary.json + the exit code are the run-level truth; metrics/final_snapshot.json records what the aggregator itself observed and can legitimately read state: complete when the abort lands after the session's terminal ENDED (post-ENDED drain window). The aggregator ignores SIGINT (the parent's ENDED path is authoritative); INTERRUPTED is entered via the session's marker event or SIGTERM (run watchdog / teardown grace).
• A run whose measurement was aborted never exits 0 and its result_summary.json is never complete: true (split-brain rewrite guard keyed on report.state, covering the drain-timeout subcase too).
• Docs: run-lifetime timeline of every knob, YAML<->CLI table, Ctrl-C contract + artifact precedence in CLI_QUICK_REFERENCE.md; AGENTS.md aggregator lifecycle updated. CLI aliases otherwise unchanged (only --duration deleted, above).
• Tests: watchdog e2e (mid-run / pre-session / during-drain fire, no-fire), real-subprocess CLI SIGINT (graceful mid-run, grace expiry against a wedged drain, pre-session, one-keystroke-under-uv run, drain-window artifact divergence — exit 130, honest artifacts, no orphans), governor unit suite (graceful + grace arming/expiry/disarm, repeat no-op, no-live-run raise), drain-timeout failure, finalization-window regression, config validation + removed-key tripwires.

Exit codes (interrupt/timeout contract)

exit	meaning	artifacts
0	clean run	state: complete, complete: true
2 / 3	input validation / setup error	none
4	ExecutionError: watchdog (run_timeout_s) fired, metrics_drain_timeout_s expired, or session aborted (e.g. transport closure)	written first; state: interrupted or complete: false
130	user ^C (graceful; teardown bounded by a 30 s grace)	full interrupted set incl. events.jsonl; grace expiry: whatever was already written + best-effort interrupted snapshot

Invariant: an aborted run never exits 0 and never ships a complete: true summary — an interrupted run is an invalid run; its artifacts exist only to expose the partial metrics. One documented, tested edge: final_snapshot.json may read state: complete for a post-ENDED-drain-window abort (the summary stays authoritative).

Follow-ups: #449 (promote warmup to a first-class phase type), #459 (consolidate run-outcome state flags into one abort/outcome model), profiling lifecycle hardening (explicit line-profiler shutdown instead of atexit; non-blocking profile POSTs) as a separate PR after this merges. schema.py module split also deferred to a follow-up MR.

Type of change

• Bug fix
• New feature
• Refactor/cleanup

Testing

• Tests added/updated
• Full unit suite + interrupt/timeout integration suites pass locally
• Manual testing completed (live watchdog + ^C runs at every phase window, incl. real uv run keystroke forwarding)

Checklist

• Code follows project style
• Pre-commit hooks pass

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request refactors the configuration schema by centralizing all global durations, deadlines, and timeouts into a new frozen Pydantic model Timeouts (accessible via settings.timeouts). This separates workload durations from failure-handling deadlines. Additionally, a whole-run watchdog (run_timeout_s) has been introduced to gracefully abort stuck runs, signaling managed subprocesses via SIGTERM to write an interrupted final snapshot before exiting non-zero. All configuration templates, examples, and tests have been updated to align with this new schema, and new integration tests have been added to verify the watchdog behavior. No review comments were provided, so there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[arekay-nv]

I think the schema breakdown make sense and is a lot cleaner.
Regarding the timeouts - two suggestions, and feedback is welcome:

1. Remove the duration field - makes it simpler especially since we are mostly going to be doing concurrency based runs.
2. Modularize the phases with an explicit type of phases and dependencies, but move the per-phase timeouts/drains etc there.
   So a global timeout for everything - and a per-phase config for controlling how a phase behaves. We can have some explicit dependencies such as warmup always goes before performance, reporting comes after accuracy etc.

[arekay-nv]

[review-council: Claude+Codex+Grok] medium — accuracy scoring / finalize is not covered by run_timeout_s; a hung external scorer hangs an otherwise-clean run. RunWatchdog is cancelled here (watchdog.cancel()), then run_benchmark calls finalize_benchmark → score_accuracy → scorer.score() with no live deadline. For SKIP_ENDPOINT_PHASE scorers that delegates whole agent execution + grading to an external service (SWE-bench swebench_service_url), and VBench/LCB spawn uv run subprocesses — any can block indefinitely on a clean run, and only a manual ^C escapes. Acknowledged in CLI_QUICK_REFERENCE.md:166 ("not deadline-bounded") but contradicted by this field's own docstring (schema.py:838 "through every phase and drain") and CLI_QUICK_REFERENCE.md:188 ("the only total-wall-time bound"). Either bound scoring with a dedicated scorer timeout, or reconcile the three statements.

[viraatc]

nit: min_duration to make clear where it should be (duration now applies to

[nvzhihanj]

Wondering if 900 is too short and whether it's worth changing to something like an hour or two

[nvzhihanj]

Dumb question, where is this 21600 timeout set, if not in the config?

[nvzhihanj]

I saw we are removing the CLI timeout in lots of places, why are we not adding the run_timeout_s in the config?

[nvzhihanj]

Cna you help add a bit more comment on how performance_drain interacts with metrics_drain? (Would be helpful to understand whether it's fast-failing like metrics drain) and how it's timed

My guess is both performance drain and metrics drain happen after the last response is returned. Trying to understand how they are controlled differently

[nvzhihanj]

Similar here, need to expand the readme a bit more

[nvzhihanj]

Review Council — Multi-AI Code Review (re-review)

Re-reviewed against the stated design principle: correct answers or no answers — never a plausible-looking wrong number. Posting the critical + high findings only.

[nvzhihanj]

[Claude] critical: Root cause for the drain-abandonment findings below. n_pending_tasks counts pending tokenizations only — it has no notion of a query the load generator gave up waiting for.

A sample abandoned by a skipped or expired response drain never produces a QueryResult, so session.py publishes no COMPLETE and no ERROR. aggregator.py:390-393 increments tracked_samples_failed only on an ERROR event. The sample was already counted at ISSUED. Net result: it lands in tracked_samples_issued and in neither tracked_samples_completed nor tracked_samples_failed.

So issued == completed + failed silently breaks and nothing asserts it. A run that dropped 20% of its samples looks like a run where 20% simply vanished — no error count, no warning, complete: true, exit 0.

Highest-leverage fix: reconcile in Report.from_snapshot (report.py:353-374) — when n_samples_issued != n_samples_completed + n_samples_failed, set complete=False (or add n_samples_dropped and give the run_benchmark ladder an arm for it). One check closes every drain-abandonment path at once, including future ones.

[nvzhihanj]

[Claude] critical: max_duration_ms reaching its cap skips the response drain entirely — performance_drain_timeout_s is never consulted.

stop_current_phase() sets _current_phase_stopped = True, which is the third clause of _drain_inflight's early return (session.py:554-559):

if (phase_issuer.inflight <= 0 or self._stop_requested or self._current_phase_stopped):
    return

Every in-flight response at cap time is abandoned. Because stop_current_phase deliberately does not set _stop_requested, no INTERRUPTED marker is published — the snapshot is a clean COMPLETE, exit 0.

The in-flight set at a cap boundary is systematically the longest requests, and the triggers split: ISL (ISSUED_NS) and TTFT (RECV_FIRST_NS) are recorded for them, latency/OSL/TPOT (COMPLETE_NS) are not. So p99 latency, p99 TPOT and OSL come out optimistically wrong while ISL/TTFT look normal.

This is the highest-volume instance: max_duration_ms is set in ~12 shipped examples. docs/CLI_QUICK_REFERENCE.md:186-187 currently blesses it — "in-flight requests are abandoned (no drain), the report is valid."

Fix: the cap should end issuing and then still drain, as the docs' own two-stage diagram implies. Drop self._current_phase_stopped from the session.py:554-559 guard (keep _stop_requested, the genuine abort); the re-check-after-clear at session.py:561-566 already covers the "cap fired while already inside the wait" case.

[nvzhihanj]

Is this the intended behavior? This is the confusion I had around the different timeouts

[nvzhihanj]

[Claude] critical: TPS/QPS numerator and denominator can truncate independently — the exact failure mode this PR exists to eliminate.

A perf sample abandoned by a skipped/expired drain keeps its row in _in_flight. If an accuracy phase follows, the recv task is still alive, so its COMPLETE arrives minutes or hours later; metrics_table._update_tracked_block then pushes block.last_complete_ns and _loadgen_window_end_ns deep into the accuracy phase. This line re-publishes the window from those extended values at ENDED, and report.py:327-334 divides by it:

qps = (n_completed - 1) / window_s
tps = osl.get("total", 0) / window_s

Accuracy-phase samples contribute nothing to osl.total (their ISSUED arrives with is_tracking == False), so the denominator grows while the numerator gains only the straggler's tokens. One stuck request returning 1700 s into an 1800 s accuracy phase turns a 600 s window into 2300 s — TPS understated ~3.8x, complete: true, exit 0.

Fix: freeze the tracked block at STOP_PERFORMANCE_TRACKING — either evict still-in-flight rows (recording them as dropped) or mark the block closed so _update_tracked_block refuses to extend past the stop event.

[nvzhihanj]

[Claude] critical: When performance_drain_timeout_s expires, the abandoned samples are dropped from the measurement and the run still reports complete: true, exit 0.

_drain_inflight logs one ERROR and returns (session.py:572-578); nothing downstream observes the residual phase_issuer.inflight, so _run_phase proceeds to stop_performance_tracking() and builds a normal PhaseResult. Per the root-cause comment on aggregator.py, those samples are counted in n_samples_issued but in neither completed nor failed.

A user who sets this knob to escape a hang therefore buys a silently-low QPS and optimistically-low latency/TPOT/OSL tails. Under "correct answers or no answers" this is the wrong side: it should be no answer.

Also: the expiry message doesn't say which knob fired — the same line serves warmup, performance and accuracy (execute.py:615/626/681), printing only the value. PhaseConfig already carries phase.name.

Fix: have _drain_inflight return the residual in-flight count; if it is non-zero for the PERFORMANCE phase, mark the report incomplete and give the run_benchmark ladder an arm that raises. Name the phase and the knob in the log line.

[nvzhihanj]

[Claude] critical: The outcome ladder has no arm for state: "live" / "draining", so this path exits 0 with a complete: false report. Confirmed by tracing to main.py — nothing raises, run_benchmark returns normally, exit code 0.

drain_and_build_report falls back to subscriber.latest when final_snapshot.json is missing (pipeline.py:354-363), and the tick task only ever emits LIVE or DRAINING. Reachable without any watchdog or ^C: the aggregator OOM-killed or crashed after STARTED, or the atomic write failing on a full disk. The split-brain guard at execute.py:1187 doesn't help — it is gated on run_timed_out or user_interrupted, both False here.

The artifact is populated from a mid-run tick: HDR-approximate percentiles, counters frozen at the last tick, and a tps/qps over a tracked_duration_ns that was never finalized. CI and && chains see success.

Fix: replace the enumerate-the-states approach with one rule, placed before the report is None arm:

if bench.report is not None and not bench.report.complete:
    raise ExecutionError(f"Report is not complete (state={bench.report.state})")

That subsumes the existing 1325-1342 arm and makes the invariant literally exit 0 iff complete: true.

[nvzhihanj]

[Claude] high: This handler covers SIGINT only — there is no SIGTERM handler, and every abort path in this PR kills the event logger with SIGTERM (pipeline._kill_services → launcher.terminate_all).

So close() never runs. The writer is constructed with flush_interval=100 on top of Python's ~8 KB text buffer, so up to ~99 records plus the buffer tail are lost. _salvage_tmpfs then copies the truncated file to report_dir/events.jsonl with no marker, no line count, no truncated flag.

A truncated events.jsonl is indistinguishable from a complete one — and it is the one artifact that survives a failed run and is most likely to be re-analysed later (scripts/early_stopping_estimate_from_events.py, SWEBenchScorer, agentic turn accounting). Each recomputes over it and gets a plausible wrong answer.

The irony: terminate_module exists precisely because the event logger "would lose buffered records on SIGTERM" (launcher.py:154-156), and abandon_drain SIGTERMs it anyway.

Fix: add a SIGTERM handler that runs service.close() before setting shutdown_event; and have _salvage_tmpfs check the last record is SessionEventType.ENDED, writing an events.jsonl.truncated marker when it isn't.

[nvzhihanj]

[Claude+Quality] high: null is silently reinterpreted as 30 s here, so one field means two different things depending on which abort fired.

interrupted_teardown_grace_s is documented as "(None = never abandon; 0 = abandon immediately)" (schema.py:841) and docs/CLI_QUICK_REFERENCE.md:172 repeats "null = never abandon". On the run_timeout_s path that is false. A user who chose null deliberately — the correct-or-nothing choice, "never truncate my drain" — gets the opposite of what both the field help and the reference table promise, with no log line saying so.

The intent (a null ^C-grace must not soften a hard run bound) is reasonable; the expression isn't. The 30.0 literal also duplicates the schema default and will drift the moment someone changes it.

Fix: give the watchdog its own named field (run_timeout_escalation_grace_s) with its own default, or source the fallback from a module constant shared with Timeouts — and state the exception in the field description and the docs table either way.

[nvzhihanj]

[Claude] high: With interrupted_teardown_grace_s: null the process becomes un-interruptible by Ctrl-C.

The escalation timer is only armed when the grace is non-None, and __call__ makes every subsequent SIGINT a logged no-op (watchdog.py:141-144). So with null — a documented, supported value — a wedged metrics drain parks forever in wait_for_exit(None) and no number of ^C presses escapes; only kill -9 from another terminal will.

docs/CLI_QUICK_REFERENCE.md:205-208 asserts the opposite: "Repeat ^C: logged no-op — the stop is already in flight and the grace bounds the teardown." That is false in exactly this case. test_sigint_grace_expiry_abandons_wedged_drain covers the scenario only with grace=3.0.

Fix: when interrupted_teardown_grace_s is None, either let the second ^C fall through to raise KeyboardInterrupt, or drop None from this field's domain (make it a plain float, ge=0) — "never abandon" has no safe realization while it is also the thing that makes ^C work.

[nvzhihanj]

[Codex] high: bind_task replaces the task and loop but never clears _session.

When run_audit reuses one governor across phases, _session still points at the previous, already-completed BenchmarkSession. A SIGINT during a later phase's service launch or endpoint connect therefore takes the _stop_gracefully path and stops the stale session instead of cancelling the current task — so the new phase keeps running despite Ctrl-C.

Fix: clear _session and _on_grace_expiry when binding each new phase (or take them together in one bind call so the two can't drift out of sync).

[nvzhihanj]

[Claude] high: The default changed from 600000 to None — the dangerous class of break, because a YAML that simply omits the key keeps parsing under extra=forbid and silently runs a different workload.

On origin/main: Field(600000, ge=0). Combined with dropping the synthetic target_qps = 10.0, a poisson config that relied on the default previously issued ceil(target_qps × 600 s × 1.1) samples and now issues one dataset pass. For a 1000-sample dataset at 100 QPS that is 66 000 → 1 000 — a 66x shorter measurement window, reported with a perfectly valid-looking TPS. No error, no warning, no log line saying the sizing rule changed.

config/templates/submission_template.yaml is the proof: it had to drop min_duration_ms: 600000 # 10 minutes because the new _min_duration_requires_qps validator rejects everything but poisson and that template is max_throughput. Two things worth calling out explicitly: (1) max_throughput + min_duration_ms was a shipped combination that now hard-errors with no replacement knob; (2) the reference MLPerf submission config silently lost its 10-minute floor.

Fix: WARN at config-resolve time when min_duration_ms is unset and target_qps is set, naming the old default and the new sizing; and give the submission template an explicit n_samples_to_issue rather than a silent deletion.

[nvzhihanj]

Review Council — Multi-AI Code Review

Reviewed by: Codex + Claude + Code-Quality | Depth: thorough

Re-reviewed against the stated design principle: correct answers or no answers — a benchmark must hang or fail loudly rather than ship a plausible-looking wrong number.

Verdict: the central fix is right, but the guarantee is enforced in only one of the four places it needs to be.

metrics_drain_timeout_s expiry now genuinely fails the run — traced end to end (token_metrics → n_pending_tasks → Report.complete=False → ExecutionError → exit 4), with artifacts written first and no zero-fill anywhere. On main the same condition set n_pending_tasks > 0 and exited 0, rendering TPS from a half-tokenized backlog. That is a real fix for a real bug. run_timeout_s and the ^C teardown grace also verified LOUD_FAILURE.

But the same guarantee is not applied to the response drains. 11 issues posted, most sharing one root cause.

🔴 Must Fix (critical)

#	File	Line	Category	Reviewer(s)	Summary
1	.../metrics_aggregator/aggregator.py	475	data-integrity	Claude	Root cause. Abandoned queries land in issued but neither completed nor failed; issued == completed + failed breaks with nothing asserting it
2	commands/benchmark/execute.py	887	data-integrity	Claude	max_duration_ms cap skips the drain entirely — performance_drain_timeout_s never consulted; ~12 shipped examples set it
3	.../metrics_aggregator/aggregator.py	471	data-integrity	Claude	TPS/QPS window extends into the accuracy phase while the numerator doesn't — rate off by multiples
4	commands/benchmark/execute.py	626	data-integrity	Claude	Perf-drain expiry drops in-flight samples, still reports complete: true, exit 0
5	commands/benchmark/execute.py	1343	data-integrity	Claude	Outcome ladder has no arm for state: "live"/"draining" → exit 0 with complete: false
6	config/schema.py	854	api-contract	Claude	*_drain_timeout_s: 0 accepted with no validator and no warning; one line turns any run into a truncated-but-valid report

🟡 Should Fix (high)

#	File	Line	Category	Reviewer(s)	Summary
7	.../event_logger/__main__.py	200	data-integrity	Claude	SIGINT handler but no SIGTERM handler — every abort path truncates events.jsonl, recorded nowhere
8	commands/benchmark/watchdog.py	258	design	Claude+Quality	null grace silently becomes 30 s on the watchdog path, contradicting the documented null = never abandon
9	commands/benchmark/watchdog.py	121	bug	Claude	interrupted_teardown_grace_s: null makes the process un-interruptible by ^C
10	commands/benchmark/watchdog.py	102	bug	Codex	bind_task never clears _session — ^C during a later audit phase stops the stale session
11	config/schema.py	610	api-contract	Claude	min_duration_ms default 600000 → None silently shrinks the measurement window up to 66x; submission template lost its 10-min floor

The one fix that closes most of #1–#6

Reconcile in Report.from_snapshot (report.py:353-374): when n_samples_issued != n_samples_completed + n_samples_failed, set complete=False (or add n_samples_dropped and give the ladder an arm). Paired with the single-rule ladder from #5 — exit 0 iff complete: true — that makes the invariant structural instead of a list of enumerated states.

What's genuinely right — worth keeping in the PR description

• --timeout went from declared but never read (BenchmarkConfig.timeout on main is referenced nowhere in src/) to a real hard bound that escalates SIGTERM→SIGKILL, so a wedged drain cannot soften it.
• Four scattered blocks (settings.drain.*, flat service_ready_timeout_s, top-level timeout:) collapse into one. metrics_tokenizer_workers — a thread count that was living in drain: — correctly moved out.
• The None/0 convention is stated once and the code actually honours it. On main, drain.metrics_drain_timeout_s: 0 meant unlimited while drain.performance_timeout_s: null also meant unlimited — the same intent spelled two opposite ways.
• Separating workload durations from give-up deadlines is the right cut, and schema.py:1013's validator message (YAML path + CLI flag + rule + alternative, one line) is the bar every other timeout message on this surface should meet.

Filtered by depth — found but not posted inline

Tuning UX (your stated second goal — "easier for the user to tune"): the "Time Knobs" reference table documents --settings.timeouts.* for 6 of 7 rows, but Settings is @cyclopts.Parameter(name="*") so the real flags are --timeouts.* — verified by rendering --help, every documented drain flag errors on copy-paste. Six short aliases were deleted with no replacement (--duration, --{warmup,performance,accuracy,metrics}-drain-timeout, --service-ready-timeout) while run_timeout_s kept its --timeout alias, so the consolidation never required dropping them. grep -rl run_timeout_s examples/ returns 0 — three examples' timeout: values (21600/14400/1800, with rationale comments) were deleted rather than migrated to the field that would finally make them work. And the messages that fire on a hang name no knob: pipeline.py:348 "Waiting for services to finish processing..." precedes an unbounded wait with no heartbeat; watchdog.py:243 names neither knob nor value.

Migration: extra=forbid produces bare extra_forbidden errors that never mention settings.timeouts; max_duration_ms: 0 — main's own template default — now fails with an opaque "Input should be greater than 0".

Code quality: SigintGovernor and RunWatchdog are two hand-rolled copies of the same stop-then-escalate machinery; RunWatchdog has zero unit tests; launcher.py pairs _modules/_procs as parallel lists zipped strict=True; a migration-narrating comment is copy-pasted into 7 example YAMLs (AGENTS.md forbids development-history comments); .pre-commit-config.yaml:61 broadened its regex for the deferred schema split and now fires on 5 unrelated modules.

⚠️ Commit hygiene: 45 commits including 8 apparent fixups. Also, 44b80a04 re-does the datasets==5.0.1 bump that already landed on main as #460 — rebasing drops it and removes the 1091-line uv.lock diff, the largest file in this PR.

[arekay-nv]

suggestion - convert to fixture.

[arekay-nv]

nit : Can we move this to the default value for the function argument intead?

[arekay-nv]

Do we have coverage for streaming responses as well.

[arekay-nv]

Review Council — Multi-AI Code Review

Reviewed by: Codex + Claude (max-effort / thorough). event: COMMENT — no approve/reject.

This PR is already well-reviewed across three prior rounds, so after de-duplicating against existing comments I'm only posting what's new. Everything the council re-derived that was already raised — the cross-phase SigintGovernor stale-_session ^C bug (watchdog.py:102), the 30.0 grace literal duplicating the schema default (watchdog.py:258), the event-logger missing-SIGTERM record-loss gap (event_logger/__main__.py:200), the audit SIGINT-policy path (audit.py:174), and the fired_before_run run-outcome-state sprawl (issue #459) — was left alone as already-covered.

New findings (3)

#	File:line	Severity	Category	Summary
1	docs/CLI_QUICK_REFERENCE.md:167-172	high	documentation	The six timeouts CLI flags are documented as --settings.timeouts.*; cyclopts strips settings (Settings is name="*", Timeouts is not), so the real flags are --timeouts.* / --timeout. Verified live (cyclopts prints Did you mean --timeouts...). The DESIGN.md:65 reply repeats the same wrong form.
2	docs/async_utils/services/metrics_aggregator/DESIGN.md:118	medium	documentation	--drain-timeout row still says 0 (unlimited); the PR flipped the default to None and inverted the sentinel (0 = give-up-immediately). Contradicts the updated AGENTS.md.
3	src/inference_endpoint/commands/audit.py:157	low	api-contract	run_timeout_s is setup-inclusive on the main run but audit phases run setup_benchmark before the deadline is captured, so audit-phase setup isn't counted and each phase gets a fresh full budget after setup.

Verified correct (no action): the None/0 convention across Timeouts; publish_final atomic write + _final_lock serialization; launcher.terminate → terminate_module rename propagation; INTERRUPTED-before-ENDED ordering; template regeneration; no stale old-knob names in committed YAML/code.

⚠️ Commit hygiene: 45 commits incl. ~8 fix* fixups on this branch — expected for a long-lived consolidation; consider squashing before merge.

[arekay-nv]

[review-council: Codex+Claude] high — These six timeouts rows document the CLI flag as --settings.timeouts.<field>, but cyclopts rejects that form. Settings is @cyclopts.Parameter(name="*") (transparent), so the settings segment is stripped, while Timeouts is not transparent — the real flags are --timeouts.<field> (and run_timeout_s is --timeout).

Verified live:

$ inference-endpoint benchmark online ... --settings.timeouts.warmup-drain-timeout-s 10
Error: Unknown option: "--settings.timeouts.warmup-drain-timeout-s". Did you mean "--timeouts.warmup-drain-timeout-s"?

The runtime/client rows in this same table already drop the prefix correctly (--runtime.min-duration-ms, --client.worker-initialization-timeout), so rows 167–172 are internally inconsistent and every copy-paste of a timeouts flag fails.

Heads-up: the reply on docs/config/DESIGN.md:65 states the flag as --settings.timeouts.run-timeout-s — the same broken form — so the mental model is worth reconciling too.

Fix: change the six --settings.timeouts.* cells to --timeouts.*.

[arekay-nv]

[review-council: Codex+Claude] medium — Stale after this PR's None=unlimited / 0=zero-budget convention. This row says --drain-timeout | 0 (unlimited) | ... (0 = unlimited) and line 123 repeats service-side defaults (0 and 4). But metrics_aggregator/__main__.py now sets --drain-timeout default=None (help: “Omit to wait indefinitely (the default); 0 gives up immediately.”) and forwards it verbatim (the old None if arg == 0 translation was removed). So the default is now None, and 0 means give-up-immediately, the exact opposite of this row — and it contradicts the updated AGENTS.md (“None or omitted flag = unlimited, 0 = give up immediately”).

Fix: default → None (unlimited), drop the 0 = unlimited gloss, and change the (0 and 4) phrasing to (None and 4).

[arekay-nv]

[review-council: Codex] low — run_timeout_s is documented as setup-inclusive (“synchronous setup — tokenizer probe, dataset load — counts against the budget”), and the main run honors it: run_benchmark captures the deadline at entry (execute.py:1282) before setup_benchmark. Audit phases don’t — setup_benchmark(...) runs at line 147 and then run_benchmark_async(ctx, sigint=sigint) is called here with no deadline, so run_benchmark_async recomputes a fresh deadline at entry after setup. Net: an audit phase’s dataset/tokenizer load isn’t counted against the budget (unlike the main run), and a phase whose setup alone exceeds run_timeout_s still launches with a fresh full budget. Minor — per-phase fresh budget is intended and audit datasets are small — but the setup-inclusive contract silently doesn’t hold on the audit path.

Fix: capture the deadline before setup_benchmark and pass it into run_benchmark_async, mirroring run_benchmark.

[arekay-nv]

Thanks for this - overall looks great - feel free to merge and create issues for followup as this is likely to cause more merge conflicts.