Repository navigation
poly, HIP: the expert cache gives VRAM back and grows again on AMD (#13) - #28
Merged
Merged
Conversation
…simplexai-labs#13) On a Linux AMD card the expert cache only ever got smaller: every program that held VRAM a moment cost experts until the server was restarted. Three causes, measured on an RX 9070 XT (16 GB, Fedora, ROCm 7 wheels): - --kv-whole (which setup writes for AMD) turned VMM off for the whole engine, so the expert cache was one allocation, filled again smaller when VRAM ran short and never grown. VMM now stays on for the expert cache; --kv-whole keeps the attention caches as one allocation each (kv_alloc()). - ROCm gives a released handle's VRAM back only when the addresses it was mapped at are freed, and frees only a whole reservation: 320 MiB unmapped and released left 32 MiB more free, the addresses freed all of it, and a partial hipMemAddressFree fails. vmm_unmap() frees the reservation, reserves it again at the same address and maps the kept pieces back (their bytes stay), under STRATA_VMM_RELEASE_WITH_ADDRESSES (hip_compat/cuda.h). - hipMemGetInfo's free figure is KFD accounting, which read 924 MiB while amdgpu had 2,128 MiB free (a 1,500 MiB hipMalloc beside the engine succeeded), and at another time more than amdgpu had. On Linux the free figure is now amdgpu's mem_info_vram_total less mem_info_vram_used for the card's PCI address; STRATA_KERNEL_VRAM_FREE=0 keeps hipMemGetInfo's. With Tiel Coder UD-Q4_K_XL at 128K: another process taking 800 MiB shrank the cache 4523 -> 4005 experts and amdgpu's free VRAM rose by what was cut; when it ended, the next request grew the cache to 4453. A 256-token answer at 110-128 tokens/s and a 4.5K prompt at about 1,820 tokens/s (105.7 and 1,941 before, with 4,758 experts cached). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
- An integrated GPU keeps hipMemGetInfo's free figure: amdgpu's mem_info_vram files count only its BIOS carve-out (a Ryzen 7 9800X3D's Radeon, gfx1036: 512 MiB) while HIP gives it system RAM (15,536 MiB free), so an APU would have lost almost its whole expert cache. - A step of vmm_unmap()'s rebind that fails once the kept pieces have left their addresses ends the engine with a message instead of throwing: the throw reached the request's catch in main.cpp and the next request read VRAM that was no longer mapped. The server starts the engine again for the next request. - vmm_unmap() waits until amdgpu counts the VRAM it gave back as free (2 to 5 ms, at most 100 ms): the log line after a shrink read 248 MiB left where there were 1,012. - The rebind is Linux only, where it was measured; Windows' HIP runtime keeps the plain release. RX 9070 XT, Tiel Coder UD-Q4_K_XL: 800 MiB taken by another process shrank the cache 4694 -> 4180 experts with 1012 MiB left (amdgpu: 1013), and it grew to 4623 when that process ended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
Merged, thanks, and thanks for tracking down all three causes. The CUDA side only changes under |
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a Linux AMD card the strata-poly expert cache only ever got smaller: each time another program held VRAM for a moment, experts were lost until the server was restarted. This fixes that (#13).
Measured on an RX 9070 XT (16 GB, gfx1201), Ryzen 7 9800X3D, 30 GB RAM, Fedora, ROCm 7 from AMD's wheels, Tiel Coder 35B-A3B UD-Q4_K_XL at 131072 tokens with
--kv-whole.Three causes
--kv-wholeturned VMM off for the whole engine. Setup writes this flag for AMD, so the expert cache was one allocation. When VRAM ran short it was refilled smaller, and it never grew. VMM now stays on for the expert cache, and--kv-wholekeeps the attention caches as one allocation each (kv_alloc()).hipMemAddressFreefails.vmm_unmap()now frees the reservation, reserves it again at the same address, and maps the kept pieces back with their contents intact. This is guarded bySTRATA_VMM_RELEASE_WITH_ADDRESSES, defined inhip_compat/cuda.h.hipMemGetInfo's free figure is wrong on Linux. It comes from KFD accounting, which read 924 MiB while amdgpu had 2,128 MiB free; a 1,500 MiBhipMallocnext to the engine still succeeded. At another time it read more than amdgpu had. On Linux the free figure is now amdgpu'smem_info_vram_totalminusmem_info_vram_usedfor the card's PCI address.STRATA_KERNEL_VRAM_FREE=0keepshipMemGetInfo's figure.Result
Another process taking 800 MiB shrank the cache from 4,523 to 4,005 experts, and amdgpu's free VRAM rose by the amount cut. When that process ended, the next request grew the cache back to 4,453 experts.
Speed after the fix:
Testing
bufs_, and with--kv-wholekv_stays empty as before.--kv-wholeno longer turns VMM off for the expert cache.🤖 Generated with Claude Code