[dkeven]

Problem

Per-container GPU metrics are attributed to the wrong container whenever the GPU container is not the first container in the pod. Dashboards show 0% utilization and 0 GiB memory for a task while the card itself reads 100% / 97% — the device-level numbers come from DCGM and are fine, but the task-level ones never join.

Root cause is a positional contract that gets broken in decode:

1. The scheduler encodes one annotation segment per container, in pod.Spec.Containers order. A pod of [linkerd-proxy, llamacpp, olares-mesh-in-agent] with the GPU on llamacpp produces ;GPU-…,NVIDIA,0,0,2:;; — the device sits at index 1.
2. DecodePodDevices skipped segments that decode to no devices, compacting the slice. The GPU entry moved from index 1 to index 0.
3. GetNextDeviceRequest returned p.Spec.Containers[ctridx] using that compacted index, so it resolved linkerd-proxy.
4. Allocate names the libvgpu shared-cache directory <podUID>_<ctrName> from that container, producing /usr/local/vgpu/containers/<podUID>_linkerd-proxy.
5. vGPUmonitor derives its ctrname label from the directory name, so vGPU_device_memory_usage_in_bytes and Device_utilization_desc_of_container are published under ctrname="linkerd-proxy". Consumers query the real container name and match nothing.

The values themselves were never wrong — only the label. The vGPUmonitor MIG collector (collectPodAndContainerMigInfo) compares ctrIdx != ctrDevIdx and so relies on the same alignment; it was silently broken too.

Any injected sidecar that lands ahead of the workload triggers this, so in a service-mesh cluster it affects every GPU pod. Multi-GPU-container pods were worse: every Allocate resolved index 0, so all of them collapsed onto the same wrong container.

Fix

Keep empty entries in DecodePodDevices so index i stays aligned with pod.Spec.Containers[i]. The encoder terminates every segment with the separator, so decoding yields one extra empty entry past the container count; GetNextDeviceRequest bounds-checks instead of indexing blindly, and logs the container it picked.

This mirrors upstream Project-HAMi/HAMi#1650, minus its init-container offset. Upstream encodes init containers into the annotation and therefore subtracts len(InitContainers); this fork builds the annotation from pod.Spec.Containers alone (k8sutil.Resourcereqs), so applying that offset here would introduce a shift the encoder never produces.

Callers audited: the scheduler (onAddPod → node usage accounting) and the vGPUmonitor MIG collector only range over the decoded slices, so extra empty entries are inert. EraseNextDeviceTypeFromAnnotation re-encodes what it decodes and keeps index alignment across successive Allocate calls.

Test plan

Unit tests — TestGetNextDeviceRequestContainerIndex covers GPU container first / behind a sidecar / last / none / more device entries than containers. Reverting only the decode change makes it fail with exactly the production symptom:

--- FAIL: TestGetNextDeviceRequestContainerIndex/gpu_container_behind_an_injected_sidecar
    container = "linkerd-proxy", want "llamacpp"

TestDecodePodDevicesPreservesContainerIndex guards the index mapping directly.

Real hardware — single-node k3s, RTX 5090 Laptop 24 GB, llama.cpp serving Qwen3.8-27B, containers [linkerd-proxy, llamacpp, olares-mesh-in-agent]:

	before	after
cache directory	<podUID>_linkerd-proxy	<podUID>_llamacpp
decoded annotation	[[{GPU}]]	[[] [{GPU}] [] []]
container resolved	linkerd-proxy	llamacpp at index 1
vGPU_device_memory_usage_in_bytes	22.29 GiB as ctrname="linkerd-proxy"	22.29 GiB as ctrname="llamacpp"
Device_utilization_desc_of_container under load	—	99, against 98-100% from nvidia-smi
hami_container_core_util / hami_container_memory_used	no series	100 / 22824 MB

Note on the test files

pkg/util did not compile before this change — positional ContainerDevice literals predate the ShareMode field, so that package's tests had not been running and its expectations had drifted. They are converted to keyed fields and refreshed, which also brings back Test_DecodePodDevices and Test_EncodeNodeDevices. The package is green now; no other package changes status, and the set of failing tests elsewhere is byte-identical before and after.

Made with Cursor