Repository navigation
hami: bump gpu-scheduler to v0.1.4 and hami core to v2.6.27 - #4097
Merged
Merged
Conversation
Both carry the timeslice lock repair. The scheduler re-sends a grant the owner never received instead of discarding its retry, and reclaims a lock whose owner never answered the drop; libvgpu answers a drop even when the lock had already been released early. Each of those states used to strand an owner entry nobody could clear, with the waiters blocked on a lock_ok that would never be sent. The scheduler pin matters twice over. Above-Os/HAMi-core's release job resolved its image tag from whatever tag was reachable rather than the ref being built, and under a shallow checkout it saw none and fell back to `latest`. No version tag has been published since v0.1.3, so this pin has been serving code from before the first half of the fix landed. That job now publishes the tag it builds, and v0.1.4 is the first release to come out of it. Verified on a single-GPU host in TimeSlice mode: two concurrent CUDA workloads produced nine clean time-quantum handoffs, every drop answered by a released, and neither workload starved. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
eball
approved these changes
Sep 5, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both images carry the timeslice lock repair from Above-Os/HAMi-core#16 and beclab/HAMi#24.
What was broken
Three states left an owner entry that nothing could clear, with the waiters blocked on a
lock_okthat would never be sent:lockwhile itsown_lockis 0, so reaching that branch meant the grant never landed — and the client's retry timer assumed the scheduler would handle a duplicate. It discarded it instead, so every retry died and the CUDA threads parked forever. The scheduler now re-sendslock_ok, keeping the original grant timestamp so a lost grant cannot buy extra time quantum.drop:is a request, not a guarantee. Once sent, the timeout loop skipped that owner permanently — no second drop, no disconnect — so an owner that never answered held the lock until its TCP connection happened to die. The scheduler now reclaims afterDROP_GRACE_MS(defaultTIMEOUT_MS * 3) by closing the socket, the one path already proven to erase the owner and promote a waiter.dropwhileown_lockwas 1. That is also 0 after the idle timer releases early, and if the scheduler had not recorded that release it still named the client as owner — precisely why it was dropping a lock the client no longer thought it held. It now answers unconditionally; the scheduler ignores areleasedfrom a non-owner, so a duplicate is harmless.Why the scheduler pin matters twice over
HAMi-core's release job resolved its image tag from whatever tag was reachable from
HEADrather than the ref being built, and under the default shallow checkout it saw no tags at all and fell back tolatest. No version tag has been published sincev0.1.3, so this pin has been serving code from before even the first half of the fix landed, while the fix itself sat in an unversioned image. That job now publishes the tag it builds, stamps the standard OCI revision labels, andv0.1.4is the first release to come out of it.Verification
Deployed to a single-GPU host (RTX 5090, TimeSlice mode) as release candidates before tagging. Two concurrent CUDA workloads in separate pods produced nine clean handoffs over 90 seconds, one per 10s time quantum, in strict alternation:
Eleven
lockrequests, ninedrop, ninereleased— every drop answered. Neither workload starved (9400 vs 9000 iterations). The three new recovery paths (requeue, re-sent grant, reclaim) never fired, which is what a healthy run should look like; they are covered by three new scenarios in the scheduler's smoke test, all of which fail without the fix.Note
Medium Risk
Rolling GPU scheduler and HAMi components on CUDA nodes can briefly affect time-sliced GPU workloads during DaemonSet/deployment updates, though the change is a targeted lock-handling fix rather than broad refactors.
Overview
This PR pins newer GPU stack images so deployments pick up the timeslice lock fixes from upstream HAMi/HAMi-core work.
The gpu-scheduler DaemonSet moves from
beclab/gpu-scheduler:v0.1.3tov0.1.4(first tagged release after the scheduler publish job fix). HAMi chart defaults and the Olares prebuilt manifest movebeclab/hamifromv2.6.26tov2.6.27, which flows to scheduler extender and NVIDIA device plugin images viavalues.yamlversion.Reviewed by Cursor Bugbot for commit 09a20bf. Bugbot is set up for automated code reviews on this repo. Configure here.