Skip to content

hami: bump gpu-scheduler to v0.1.4 and hami core to v2.6.27 - #4097

Merged
eball merged 1 commit into
mainfrom
fix/gpu-timeslice-lock-images
Sep 5, 2026
Merged

eball merged 1 commit into
mainfrom
fix/gpu-timeslice-lock-images

Conversation

@pengpeng

@pengpeng pengpeng commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

Both images carry the timeslice lock repair from Above-Os/HAMi-core#16 and beclab/HAMi#24.

What was broken

Three states left an owner entry that nothing could clear, with the waiters blocked on a lock_ok that would never be sent:

  • An owner re-requesting a lock the scheduler already believed it held was a no-op. The client only sends lock while its own_lock is 0, so reaching that branch meant the grant never landed — and the client's retry timer assumed the scheduler would handle a duplicate. It discarded it instead, so every retry died and the CUDA threads parked forever. The scheduler now re-sends lock_ok, keeping the original grant timestamp so a lost grant cannot buy extra time quantum.
  • drop: is a request, not a guarantee. Once sent, the timeout loop skipped that owner permanently — no second drop, no disconnect — so an owner that never answered held the lock until its TCP connection happened to die. The scheduler now reclaims after DROP_GRACE_MS (default TIMEOUT_MS * 3) by closing the socket, the one path already proven to erase the owner and promote a waiter.
  • libvgpu only answered a drop while own_lock was 1. That is also 0 after the idle timer releases early, and if the scheduler had not recorded that release it still named the client as owner — precisely why it was dropping a lock the client no longer thought it held. It now answers unconditionally; the scheduler ignores a released from a non-owner, so a duplicate is harmless.

Why the scheduler pin matters twice over

HAMi-core's release job resolved its image tag from whatever tag was reachable from HEAD rather than the ref being built, and under the default shallow checkout it saw no tags at all and fell back to latest. No version tag has been published since v0.1.3, so this pin has been serving code from before even the first half of the fix landed, while the fix itself sat in an unversioned image. That job now publishes the tag it builds, stamps the standard OCI revision labels, and v0.1.4 is the first release to come out of it.

Verification

Deployed to a single-GPU host (RTX 5090, TimeSlice mode) as release candidates before tagging. Two concurrent CUDA workloads in separate pods produced nine clean handoffs over 90 seconds, one per 10s time quantum, in strict alternation:

drop B -> released -> lock
drop A -> released -> lock
...

Eleven lock requests, nine drop, nine released — every drop answered. Neither workload starved (9400 vs 9000 iterations). The three new recovery paths (requeue, re-sent grant, reclaim) never fired, which is what a healthy run should look like; they are covered by three new scenarios in the scheduler's smoke test, all of which fail without the fix.


Note

Medium Risk
Rolling GPU scheduler and HAMi components on CUDA nodes can briefly affect time-sliced GPU workloads during DaemonSet/deployment updates, though the change is a targeted lock-handling fix rather than broad refactors.

Overview
This PR pins newer GPU stack images so deployments pick up the timeslice lock fixes from upstream HAMi/HAMi-core work.

The gpu-scheduler DaemonSet moves from beclab/gpu-scheduler:v0.1.3 to v0.1.4 (first tagged release after the scheduler publish job fix). HAMi chart defaults and the Olares prebuilt manifest move beclab/hami from v2.6.26 to v2.6.27, which flows to scheduler extender and NVIDIA device plugin images via values.yaml version.

Reviewed by Cursor Bugbot for commit 09a20bf. Bugbot is set up for automated code reviews on this repo. Configure here.

Both carry the timeslice lock repair. The scheduler re-sends a grant the owner
never received instead of discarding its retry, and reclaims a lock whose owner
never answered the drop; libvgpu answers a drop even when the lock had already
been released early. Each of those states used to strand an owner entry nobody
could clear, with the waiters blocked on a lock_ok that would never be sent.

The scheduler pin matters twice over. Above-Os/HAMi-core's release job resolved
its image tag from whatever tag was reachable rather than the ref being built,
and under a shallow checkout it saw none and fell back to `latest`. No version
tag has been published since v0.1.3, so this pin has been serving code from
before the first half of the fix landed. That job now publishes the tag it
builds, and v0.1.4 is the first release to come out of it.

Verified on a single-GPU host in TimeSlice mode: two concurrent CUDA workloads
produced nine clean time-quantum handoffs, every drop answered by a released,
and neither workload starved.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vercel

vercel Bot commented Sep 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
olares Ready Ready Preview Sep 5, 2026 1:21pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
olares-docs Ignored Ignored Sep 5, 2026 1:21pm UTC

@eball
eball merged commit 82fee5d into main Sep 5, 2026
17 checks passed

This branch was successfully deployed

1 active deployment
Preview – olares — 09a20bf8 Deployed Sep 5, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants