Skip to content

AMD/HIP gfx1100 (RX 7900 XTX): engine loads the Coder pack fully then fails in the verify kernel - 'verify: launch: unspecified launch failure' on the first request, or hangs before ready #273

Closed

Description

@rwkeyes

AMD/HIP (gfx1100, RX 7900 XTX): engine loads the Coder pack fully, then verify: launch: unspecified launch failure on the first request — or hangs before ready. No working configuration.

Engine 0.1.29, build from source on Linux, HIP backend, RX 7900 XTX (gfx1100). The build succeeds, the pack prepares, the engine loads the whole model — and it never serves a token.

What happens

Three configurations, all on the same box and pack:

1. --spec 4 (what setup.sh --backend hip wrote). The server comes up and /v1/models reports the model loaded. The first chat request kills the engine:

strata generate: PCIe probe: 7.0 GB/s host->device -> pcie_frac 0.15 (default 0.55)
strata verify: window up to 6 tokens, 74.0 MiB of device buffers
strata verify: captured the 6-token window (upload no error, sync no error)
strata serve: verify: launch: unspecified launch failure

the server then reports the engine stopped unexpectedly (exit code 1), done: 0 tokens in 5 s (0.0 tok/s) (error, cancel=False), and a retry gets the engine stopped unexpectedly (exit code 1) again.

2. --spec 2 (the smallest window the pack accepts). Loads to completion — 644 MiB of VRAM free with everything loaded, session is up (engine 0.1.29) — then never signals ready: the API port stays closed and the log stops.

3. --spec 4 plus the flags docs/AMD_HIP.md prescribes for AMD (--prefill 512 --pool-workers 15 --vram-reserve-mib 1024 --adapt-every 0 --pcie-frac 0). Same hang as (2), with 946 MiB of VRAM free with everything loaded.

--spec 0 cannot be used to test around the verify path: the engine rejects it with

is a native (IQ) pack: it needs --native SHARD1, --spec T (T >= 2) and --prefill CHUNK

so there is no configuration on this hardware that reaches a generated token.

Why it reads as the verify kernel rather than the transfer

The window is allocated and its upload and sync both report success (upload no error, sync no error) on the line immediately before the launch failure, so the fault looks like the kernel launch itself rather than the buffer setup. This release touched exactly that area (#187 QSA block scores read each key block once for all of a verify window's queries, #148 bitwise test of the verify window's multi-token GEMV contract, #154 guards on the verify window's tables), and #228 (QSA select on tensor cores via rocWMMA, opt-in, gfx1100) is in the same neighbourhood — possibly relevant, possibly not.

Environment

  • GPU: AMD Radeon RX 7900 XTX (gfx1100), 24 GB VRAM — the card docs/AMD_HIP.md names
  • ROCm: 7.1.52802 (system install under /opt/rocm, hipcc + hipBLAS + hipBLASLt present), HIP compiler /opt/rocm/llvm/bin/clang++ — i.e. the ROCm version docs/AMD_HIP.md records as the one this backend was tested with
  • OS: Ubuntu 24.04, kernel 7.0.0-34-generic; CPU AMD Ryzen 7 7700X (AVX-512), 16 threads; 46 GB RAM; model data on NVMe
  • Build: cmake -S . -B build-hip -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_ENABLE_CUDA=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1100 → engine/strata 16.9 MB; engine/BUILD.json reports {"source":"local-hip","backend":"hip","version":"0.1.29","archs":["gfx1100"],"vision":"none","lib_dirs":["/opt/rocm/lib"]}
  • Pack: family coder, size IQ1_M, prepared by ./setup.sh --backend hip --family coder --model IQ1_M --vision no --context 4096 --experimental-speed-projection off --low-ram auto
  • Load lines from the engine's own log: experts loaded: 23.42 GiB at 0.19 GiB/s (143 s so far), filling the GPU's expert cache (9367 experts, 17.85 GiB of VRAM)

Reproduce

ROCM_PATH=/opt/rocm ./setup.sh --backend hip --family coder --model IQ1_M --vision no --context 4096
./run-coder-iq1_m.sh
# wait for "ready: http://127.0.0.1:<port>/v1", then send one chat completion

What I did NOT test

  • The other quants (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S need ~48–62 GB system RAM per the installer's own figures; this box has 46 GB, so only the Coder fits)
  • The vision path (disabled), non-gfx1100 AMD cards, Windows+HIP, and multi-GPU
  • Only one GPU on the box is usable for this; the integrated Radeon (gfx1036) is correctly reported as unsupported

Scope

Nothing here suggests the Coder weights, the pack, or the download are at fault — the engine loads 23.42 GiB of experts and exits inside its own GPU kernels before it reads a prompt. Your release notes say AMD was not retested for this release because the AMD test PC was unreachable; this is data from a gfx1100 machine that does have the card and the ROCm version named in docs/AMD_HIP.md, so hopefully it saves a round trip. I can re-test against a fix or a diagnostic build, and I'm happy to run any test you'd like instrumented.

Activity

  1. Maxritz commented on Sep 30, 2026

    @Maxritz
  2. rwkeyes commented on Sep 30, 2026

    @rwkeyes
    ContributorAuthor

    Confirmed fixed on 0.1.30 — same machine, same pack, same failure point as the original report.

    0.1.29: the Coder IQ1_M pack loaded fully (23.42 GiB), then the first request died with verify: launch: unspecified launch failure. No configuration got past it: --spec 4 and --spec 2 both faulted, and --spec 0 is rejected for a native IQ pack ("needs --native SHARD1, --spec T (T >= 2) and --prefill CHUNK"), so the verify path could not be bypassed.

    0.1.30: builds and serves. Verified end to end, twice, on the same pack.

    Environment: RX 7900 XTX (gfx1100), ROCm 7.1.52802 from /opt/rocm (note: the shell exported ROCM_PATH=/opt/rocm-6.2.1, which makes detection fail and pull a TheRock nightly — overriding it to /opt/rocm used the system ROCm), source HIP build (ROCM_PATH=/opt/rocm ./setup.sh --backend hip --build), Coder IQ1_M native pack, --context 65536, --kv int8, Ryzen 7 7700X (16 cores, AVX-512) / 46 GB RAM, HDD-backed weights.

    Measured:

    • ready in ~160 s (23.42 GiB of experts paged in at 0.19 GiB/s off disk)
    • KV streaming engaged automatically at 64K: "the context's KV cache lives in RAM (0.9 GB), more experts fit in VRAM"
    • expert cache 9,144 experts / 17.42 GiB, 93.3–97.1% hit, 23.66 / 23.98 GiB VRAM used
    • prefill 98 tok/s @ 467 tok → 426 @ 1.7K → 560 @ 6.7K → 541 @ 26.8K → 856 @ 49K
    • decode 17.7 tok/s @ 467 tok → 27.5 @ 1.7K → 46.0 @ 6.7K → 60.5 @ 26.8K → 35.4 @ 49K
    • OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) both answer; tool-call smoke passes
    • prompts up to 49,005 tokens (measured, not nominal) served correctly

    The 4K default context was the only thing I had to change for real use; raising it also made it faster here, consistent with your card table.

    Expected on this card, already tracked: the "no hipBLASLt tuning table for gfx1100" warning (dense matmuls fall back to plain hipBLAS) — #170.

    @Maxritz thanks for the fork pointer — upstream 0.1.30 covers my case, so I haven't needed it, but I'll take a look at it for the gfx1100 tuning table.

    Nothing else outstanding from my side — happy to close this if that's useful.

  3. Niko1221 commented on Oct 1, 2026

    @Niko1221
    Owner

    Closing as fixed on 0.1.30 (confirmed above). We added your configuration (low-RAM resident, --pcie-frac > 0, the first verify window) to our AMD smoke tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions