Skip to content

drm/asahi: g17: drop the completion thread's frequency boost while the GPU is the bottleneck - #558

Merged
DjDeveloperr merged 2 commits into
integration/aurorafrom
perf/g17-rx-gpubound-20261028
Oct 28, 2026
Merged

DjDeveloperr merged 2 commits into
integration/aurorafrom
perf/g17-rx-gpubound-20261028

Conversation

@DjDeveloperr

Copy link
Copy Markdown
Contributor

Every G17 completion is delivered by the GPU coprocessor's mailbox. Its receive handler is a threaded IRQ (SCHED_FIFO, since #382). RT threads run with the RT default minimum utilization (sched_util_clamp_min_rt_default, 1024), so schedutil raises CPU0's cluster to its top frequency for each completion. During GPU-bound work that burst gains nothing: on J700 the P-cluster sits at 4.04 GHz in about half of 1 s samples while the CPUs are ~3% busy. Once the SoC reaches its thermal limit (TVD0 107 °C within ~40 s of heavy work), that power comes out of the GPU's clock.

The mailbox driver owns the thread, so it gets the mechanism. asahi only decides the policy.

soc: apple: mailbox

  • apple_mbox_set_rx_util_min(mbox, util_min) takes 0..1024, or -1 for the scheduler default. The request is a single store.
  • The threaded handler applies it to itself with sched_setattr_nocheck() on its next run, keeping its policy and priority.
  • Only mailboxes with a threaded receive handler (G17) support it.
  • Wrapped as apple_rtkit_set_rx_util_min(), RtKit::set_rx_util_min(), and a borrow-free RtKit::rx_util_min_handle().
  • KUnit: request checks, and applying a request to a FIFO thread without changing its policy or priority.

drm/asahi: g17

  • The feed detector also recognizes when the GPU is the bottleneck: ≥8 passes in a row whose refill gaps stay under a quarter of their busy time (the complement of its host-refill-bound test), lingering 8 ms.
  • While that holds, the port asks for 256. Otherwise it asks for the scheduler default: idle, paced work, work bound by host refills, and any completion someone may be waiting on.
  • The port re-evaluates at every publication and completion. After a pause, the publication that ends the GPU-bound state restores the default before its completion arrives. There is no new work item or timer.
  • KUnit: the detector's three outcomes.

Measurements (J700, panel off, one boot, 45 s heat-up, old/new strictly alternated ×5; gpubench µs per submit, lower is better)

row old new
render.heavy fill 2160p d1 2817 2723 −3.3%
render.heavy fill 2160p d16 2655 2577 −2.9%
render.heavy fill 1080p d1 802 780 −2.7%
compute.heavy alu d16 1199 1179 −1.7%
compute.heavy alu d1 1374 1375 one window in five
idle.wake 2 ms (median of 10) 191.2 191.3 tie
idle.wake 20 ms 588.4 589.6 tie
render.light d1/d32, compute.light d1/d8, queues d2, sync.signal binary within noise

On the heavy rows, the P-cluster's mean frequency drops from 2.5 to 1.0 GHz and system power by 0.4–0.5 W.

Alternatives measured and rejected:

  • Lowering the boost whenever the feed isn't streaming. Heavy rows gain the same, but the completion that ends a short pause is slower too: idle.wake 2 ms is +5% even at a floor of 512, and +2.8%/+6.9% on idle.wake 16 ms at 256/0.
  • Lowering it only while the completing engine has more work queued. This keeps idle.wake, but gives up the d1 heavy gains.
  • A floor of 0 instead of 256. It costs compute.heavy alu d1 one completion per 250 ms window.

Qualification

Production kit (integration 87e3a4d + these two commits, no diagnostics) on J700, fresh boot, Mesa wip83:

  • Regression gate: PASS.
  • Targeted deqp-vk (synchronization2 basic and timeline, command buffers, render passes): 4687 pass, 681 not supported, 0 fail, the same as the drm/asahi: retire a G17 render group on its pass record when its stamps stall #555/drm/asahi: G17 kill flag order and keep_graph() doc comment #550 qualifications.
  • Desktop and recovery: desktop alive, 0 recoveries.
  • Stamp-less retirements: the usual one compute command retired on its completion record, and one render group on its pass record. Both are pre-existing fallbacks, also seen on the baseline stack.
  • The completion thread on the Neo: stays SCHED_FIFO at priority 50, with uclamp.min back at the default (1024) when idle or light. During heavy rows the P-cluster runs at ~1 GHz instead of ~2.5 GHz (A/B above).
  • KUnit (arm64 QEMU, 16K pages, the aurora config plus UCLAMP_TASK): 1786 tests, 1782 pass, 4 skips unrelated to this change. This includes apple_mbox_g17_stall.apple_g17_rx_util_min_test and asahi_g17_feed_detector (3 cases).

No omacom mirror: omacom/linux-aurora's integration has no threaded G17 receive handler (#382 isn't there). Its completions are received in hard-IRQ context, so there is no RT minimum-utilization boost to remove, and the patch has nothing to act on there.

…utilization

The G17 mailboxes receive in a threaded IRQ handler, which runs as an RT
thread. RT threads run with the RT default minimum utilization, 1024
unless the system lowers sched_util_clamp_min_rt_default, so schedutil
raises the thread's CPU cluster to its top frequency for every message.
For the GPU completion mailbox that means one top-frequency burst per GPU
completion, also while the GPU itself is the bottleneck and the CPU has
nothing to do quickly.

Add apple_mbox_set_rx_util_min(), which records the minimum utilization a
consumer wants for the receive thread, or -1 for the scheduler's default,
and have the threaded handler apply it to itself (sched_setattr_nocheck()
on current, keeping its policy and priority) the next time it runs. The
request is a single store, so it may be made from any context. Only
mailboxes with a threaded receive handler support it; until a consumer
asks, the thread keeps the default.

Wrap it for RTKit consumers as apple_rtkit_set_rx_util_min() and
RtKit::set_rx_util_min(). RtKit::rx_util_min_handle() returns a handle
that makes the same request without borrowing the RtKit, for consumers
that update it from their submission and completion paths.

A KUnit case covers the request checks, and applying a request to a FIFO
thread without changing its policy or priority.
…e GPU is the bottleneck

Every G17 completion arrives through the primary coprocessor's mailbox,
whose receive thread is an RT thread. With the RT default minimum
utilization, schedutil raises its CPU cluster to the top frequency for
each completion. While the GPU itself is the bottleneck that burst gains
nothing. On J700, during GPU-bound work the P-cluster sits at its top
frequency in about half of 1 s samples although the CPUs are about 3%
busy, and once the SoC reaches its thermal limit (within about 40 s of
heavy work) that power comes out of the GPU's clock.

Have the feed detector also recognize when the GPU is the bottleneck: at
least eight passes in a row whose refill gaps stay under a quarter of
their busy time, the complement of its host-refill-bound test, lingering
8 ms like that state. While it holds, the port asks for a minimum
utilization of 256 for the receive thread, and for the scheduler's
default otherwise. The port re-evaluates at every publication and
completion through a lock-free RTKit handle, so after a pause the
publication that ends the state restores the default before its
completion arrives. There is no new work item or timer.

Lowering the boost whenever the feed is not streaming is not enough. The
completion that ends a short pause then runs slower too: idle.wake with
2 ms gaps took 5% longer even at a floor of 512. A floor of 0 instead of
256 costs single-submit heavy compute about one completion per 250 ms.

Measured on J700 with the panel off, in one boot after a 45 s heat-up,
the old and new behaviour strictly alternated five times (gpubench,
microseconds per submit, mean of the per-round medians; lower is better):

  render.heavy fill 2160p d1     2817 -> 2723   -3.3%
  render.heavy fill 2160p d16    2655 -> 2577   -2.9%
  render.heavy fill 1080p d1      802 ->  780   -2.7%
  compute.heavy alu d16          1199 -> 1179   -1.7%
  compute.heavy alu d1           1374 -> 1375   (one window in five)
  idle.wake 2 ms / 20 ms         191 / 588 -> 191 / 590 (medians)
  render.light d1/d32, compute.light d1/d8, compute.light queues d2,
  sync.signal binary queues      within run-to-run noise

On the heavy rows the P-cluster's mean frequency drops from 2.5 to
1.0 GHz and system power by 0.4-0.5 W.

A KUnit suite covers the detector's three outcomes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant