Skip to content

Monitor: one line per card on a layer split (load, VRAM, temperature, power, PCIe) - #2317

Closed
kennuardo wants to merge 1 commit into
Niko1221:mainfrom
kennuardo:monitor-per-card
Closed

kennuardo wants to merge 1 commit into
Niko1221:mainfrom
kennuardo:monitor-per-card

Conversation

@kennuardo

Copy link
Copy Markdown

Issue: none (related: #2001 is about the Intel reader picking the wrong card; this is about the charts on a layer split)

Summary

On a two-card layer split the Monitor tab's sparklines show one series per tile (mean load, total VRAM and power, the hottest temperature, the first card's PCIe traffic), so the charts cannot say which card is working. With this change the GPU load, VRAM, GPU temp, Power and PCIe tiles draw GPU 0 as today and GPU 1 as a dashed second line on the same scale, and the tiles' subtext names the cards ("RTX 5070 Ti 72% · RTX 5060 Ti 26%"). One card: nothing changes, on the server or on the page.

What changed

  • serve/telemetry.py: with more than one card, _loop also records each card's own gpu<N>_util, gpu<N>_mem_used, gpu<N>_temp, gpu<N>_power, gpu<N>_pcie_rx_mb in the same 60-sample history, and the gpus entries carry pcie_rx_mb / pcie_tx_mb (the reader already read them; they were only summed). Single card: no new keys.
  • serve/web/app.js: sparkPair() draws GPU 0 (solid line + area) and GPU 1 (dashed) on one scale when the per-card series exist, else falls back to spark() exactly as before. The per-card subtext uses short card names split from hardware_static.gpu_name (vendor prefix dropped, falls back to GPU <n> when the names and the card count disagree), one card per side of the tile with the value in bold; the Power and PCIe tiles get the per-card subtext the other tiles already had.
  • serve/web/app.css: the second line in --st-info; the per-card subtext as a flex row.

Extra Notes

  • Measured on an RTX 5070 Ti + RTX 5060 Ti layer split, Linux, engine 0.1.43.1: /metrics grows by five 60-sample series per card (about 3 KB); the sampler does no new reads.
  • Default path: with one card the history keys, the gpus entries and the rendered tiles are the same as before (sparkPair returns to spark() when gpu1_* is absent).
  • Touches the same PCIe tile as Monitor: the PCIe traffic from the bytes the engine moved, not from a driver counter #2272; both apply, the conflict is a few lines in render().
  • Checked: the serve test suite, python -m unittest discover -s serve -p "test_*.py" -t . on this branch and on main at a0116b2: Ran 725 tests ... OK (skipped=11) on both (Linux, Python 3.14). The page was checked by hand on the two-card split (screenshot) and reads the same as before with one card (the per-card series are absent, sparkPair falls back to spark).
image

… power, PCIe)

With a model split across two cards the Monitor tab showed one series per
tile: the cards' mean load, their total VRAM and power, the hottest
temperature and the PCIe traffic of the first card. The tiles' subtext
already listed each card's own value, but the sparklines could not say
which card was working.

serve/telemetry.py: with more than one card, the sampler also records each
card's own series (gpu<N>_util, gpu<N>_mem_used, gpu<N>_temp, gpu<N>_power,
gpu<N>_pcie_rx_mb) in the same 60-sample history, and "gpus" carries each
card's pcie_rx_mb / pcie_tx_mb. One card: nothing is recorded or sent that
was not before.

serve/web/app.js: the GPU load, VRAM, GPU temp, Power and PCIe tiles draw
GPU 0 as today (solid line with its area) and GPU 1 as a dashed line on the
same scale (sparkPair); the subtext names the cards by their short name from
hardware_static.gpu_name ("RTX 5070 Ti", vendor prefix dropped), one card
per side of the tile with the value in bold, and the Power and PCIe tiles
get a per-card subtext as the other tiles had. Without per-card series the
tiles render exactly as before.

serve/web/app.css: the dashed line in --st-info, the per-card subtext row.

Measured on an RTX 5070 Ti + RTX 5060 Ti layer split (Linux, engine
0.1.43.1): /metrics grew by five 60-sample series per card; the sampler's
cost is the same reads it already did.
@shahrokhzargarpour shahrokhzargarpour mentioned this pull request Oct 14, 2026
Closed
@qiuzan2001

Copy link
Copy Markdown

One case the second line misses: serve/telemetry.py stores each card's series under its real index (gpu{g['index']}_util, from the config's GPU list, idx = list(gpu_indices)), but sparkPair() in serve/web/app.js always reads h.gpu0_* and h.gpu1_*. With a split over GPUs 0 and 2 (--gpus 0,2, as in docs/INSTALL.md) or 1 and 2, gpu1_* does not exist, so every tile falls back to the single aggregate line while the subtext names both cards. Taking the two series keys from hw.gpus[0].index and hw.gpus[1].index would cover it. (Read on the branch, not run in a browser; no multi-GPU box here.) Also, the new code comments are marked local:, which reads like a note from a private fork.

@Niko1221

Copy link
Copy Markdown
Owner

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

  1. Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
  2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
  3. Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
  4. Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

@Niko1221 Niko1221 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants