Skip to content

fix: fail fast on endpoint response stalls - #462

Open
roborluo wants to merge 1 commit into
mlcommons:mainfrom
roborluo:fix-bofengl-no-progress-watchdog
Open

fix: fail fast on endpoint response stalls#462
roborluo wants to merge 1 commit into
mlcommons:mainfrom
roborluo:fix-bofengl-no-progress-watchdog

Conversation

@roborluo

Copy link
Copy Markdown
Collaborator

Summary

  • add an opt-in generic no-progress watchdog for in-flight endpoint requests
  • expose no_progress_timeout_s through YAML and direct CLI aliases
  • cancel the watchdog immediately when the active cohort drains, and document TensorRT-LLM disaggregated guidance

Validation

  • pytest tests/unit/config/test_schema.py tests/unit/commands/test_benchmark.py tests/unit/load_generator/test_async_session.py -q (334 passed)
  • python scripts/regenerate_templates.py --check
  • git diff --check
  • AGA disaggregated held-response canary: job 537769 failed intentionally after 10s with Endpoint made no response progress for 10.0s with 1 request(s) in flight.

@roborluo
roborluo requested a review from a team August 22, 2026 17:29
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Aug 22, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.80488% with 10 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@ea33275). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/inference_endpoint/load_generator/session.py 87.65% 10 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #462   +/-   ##
=======================================
  Coverage        ?   81.07%           
=======================================
  Files           ?      150           
  Lines           ?    20308           
  Branches        ?        0           
=======================================
  Hits            ?    16464           
  Misses          ?     3844           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants