Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Nimbus318 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Warning Review limit reachedNext included review available in 3 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (18)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
HAMi allocates GPUs to init and sidecar containers, but WebUI only recorded app containers. Such a container showed up as a bare request, and its diagnosis reported that no allocation record was found. Init and sidecar containers with a HAMi record are now allocation rows with their own status. Device, node and cluster totals follow HAMi's Pod accounting (CollapseInitContainerUsage): init containers count at their peak and sidecars add to the app containers, until every ordinary init container has succeeded and HAMi releases the init allocations. The exporter publishes these totals as hami_vgpu_allocated, hami_vcore_allocated and hami_vmemory_allocated. Allocation charts read them and fall back to the per-container sums for older samples. The workload detail labels the container kind and also reads kube-state-metrics' init container limits. Signed-off-by: Nimbus318 <136771156+Nimbus318@users.noreply.github.com>
4be5e4f to
f0c3608
Compare
1 participant
HAMi allocates GPUs to init containers and native sidecars. WebUI only recorded app containers, so such a container appeared in the workload list as a bare request, its scheduling diagnosis said no allocation record was found, and device totals left it out.
Allocation records. Init and sidecar containers with a HAMi record are now allocation rows, read from HAMi's annotation slots (init containers first). Each row shows the container's own memory and cores, its init-container status, and a kind label in the list and on the detail page. The diagnosis links to them like any other allocated container.
Totals. Device, node and cluster totals follow HAMi's Pod accounting (
CollapseInitContainerUsageandSteadyStateDeviceUsageinpkg/device/initContainer.go) instead of adding up containers. Per Pod and device, memory, cores and slots are each:So the rows of one Pod need not add up to its device total. The API totals and the new exporter gauges
hami_vgpu_allocated,hami_vcore_allocatedandhami_vmemory_allocateduse the same function. The overview, node and card allocation charts read those gauges and fall back to the per-container sums for samples scraped before the upgrade. Per-container series and workload rankings are unchanged apart from now including these containers.The workload detail also reads CPU and memory limits from kube-state-metrics'
kube_pod_init_container_*series, where init and sidecar containers are reported. Their GPU usage charts need HAMi's device monitor to report them, which HAMi master does (#2716) and releases up to v2.10.0 do not.Matching HAMi. The accounting tests include the two orderings from HAMi's
Test_calcScore_SidecarInitOrdering. The same seven cases were also run through HAMi's ownCollapseInitContainerUsageat8ca23d0, with identical results. This follows HAMi master; HAMi v2.9 added up every container and v2.10.0 counts sidecars like init containers, so their scheduler figures can be higher or lower for such Pods.make verifyandmake -C server verify(plus-raceon the changed packages) pass.On a single-A10 test cluster, a Pod with a GPU native sidecar (128 MiB) and a GPU app container (384 MiB) now shows the sidecar as an allocation row with its label, the diagnosis lists both allocated containers, and the device counts 9 of 10 slots and 3328 MiB where it previously omitted the sidecar. Ordinary GPU init containers are covered by the unit tests.