fix: make SM limiter step linear in SM count

[maymust-jonghyunlee]

First, English is not my native language, so I used AI to help with the translation.

Fixes #314

Removes one sm_num factor from the correction step in delta().
The token pool is:

sm_num * max_thread_per_sm * FACTOR

The correction step previously contained:

sm_num * sm_num * max_thread_per_sm * diff / 2560

This caused the step-to-pool ratio to increase with the SM count. The ratio is now diff / 81920 regardless of device size.
This PR does not change g_total_cuda_cores in setspec(). Its int64_t conversion is covered by #303, and the two changes can be merged in either order.

On a 188-SM GPU, the previous correction step could refill the entire pool in a single 120 ms tick. This caused the limiter to alternate between saturation and throttling instead of settling at the configured cap. See #314 for logs and further analysis.

Test environment:

• RTX PRO 6000 Blackwell Server Edition (188 SMs)
• Driver 580.173.02
• CUDA 13.3
• Standalone LD_PRELOAD
• GPU_CORE_UTILIZATION_POLICY=FORCE
• Separate shared cache per process
  Two processes shared one GPU with caps of 30 and 70, running 4096 FP32 matrix multiplication for 240 seconds:

build	A(30) it/s	B(70) it/s	total	device util
no cap	23.1	23.1	46.2	98%
main	21.4	25.2	46.6	98%
this change	13.2	34.9	48.1	98%

With one process capped at 10, main averaged 26.9% device utilization while userutil oscillated between 0 and 83. With this change, userutil converged within approximately 20 seconds and remained between 9 and 13, averaging 10.9%.

Summary by CodeRabbit

• Bug Fixes
  • Corrected CUDA utilization calculations to prevent overestimating core activity on devices with multiple streaming multiprocessors.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maymust-jonghyunlee
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 9e3982be-3b0b-457e-aea0-e299df9a1301

📥 Commits

Reviewing files that changed from the base of the PR and between f01e9f2 and 6b82ed8.

📒 Files selected for processing (1)

• src/multiprocess/multiprocess_utilization_watcher.c

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

The utilization watcher now calculates the CUDA-core increment with one SM-count factor instead of two.

Changes

Utilization correction

Layer / File(s)	Summary
Correct the utilization increment src/multiprocess/multiprocess_utilization_watcher.c	The delta() helper removes the extra SM-count factor from the increment calculation.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to 6b82e

The utilization limiter now adjusts token allocation proportionally to SM count, preventing oversized correction steps on high-SM GPUs. The change is ready to merge.

Suggested labels: enhancement

Suggested reviewers: try-agaaain

Poem

A rabbit checks the CUDA flow,
One SM factor now will show.
No squared count to make it soar,
The watcher measures as before.
Clean increments hop along,
And bounded cores stay steady and strong.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: making the SM limiter correction step linear in SM count.
Linked Issues check	✅ Passed	The one-line change removes the extra SM factor from delta(), which directly implements issue #314. The correction step now scales linearly with SM count and keeps the step-to-token-pool ratio indepen…
Out of Scope Changes check	✅ Passed	The pull request contains only the targeted one-line change in delta(). No unrelated code changes are present.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[maymust-jonghyunlee]

Not like #286 LLM helped posting my replies.
This time i used LLM for translating description.

[mesutoezdil]

this also shrinks the step by sm_num on small cards. did convergence still hold on something like a 40-sm gpu?

[maymust-jonghyunlee]

Since I don't have 40 sm gpu (maybe like 4080?) i tested with gtx 1080.

Measurements were taken on a gtx 1080 with 20 sm (Pascal, max_threads_per_sm=2048, pool size 1,310,720), using driver 580.173.02,CUDA_DEVICE_SM_LIMIT=50,GPU_CORE_UTILIZATION_POLICY=FORCE. Ran continuous matmul for 240 seconds using standalone LD_PRELOAD. Both were built from the same source tree with the same toolchain: main versus main with this patch applied. Watcher samples were collected approximately every 20 seconds.

On main, userutil was 46, 62, 9, 100, 17, 83, 95, 100, 100, 100, 100, 47, unstable. Later in the run, it stayed at 100 for roughly 80 seconds, while share alternated between 0 and approximately 95000. Average device utilization, sampled every 5 seconds with nvidia-smi, was 81%, well above the 50% target.

With the patch, userutil was 40, 41, 43, 55, 56, 68, 51, 67, 50, 43, 42. It was already within the target band at the first sample, 20 seconds into the run, and share remained stable between 95000 and 102000 Average device utilization was 49%, close to the 50% target.