Repository navigation
Conversation
…physical_cores() (Niko1221#14, fixes Niko1221#13 and Niko1221#11) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ad of spinning (Niko1221#9, fixes Niko1221#4) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… engine-queue race between requests (Niko1221#7) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ts through the decode windows (Niko1221#8, Niko1221#10) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… pool fix) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
START-HERE.bat on an installed setup went straight to starting the model, so the MIN_ENGINE check (and the engine download) only ran when setup was run again with its options. Now every start checks the installed engine and replaces it when it is older than MIN_ENGINE. If it cannot (no internet, the model still running, no ready-made engine for this GPU) the installed engine is kept and starts as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
- GPU experts: IQ4_XS gate/up (the IQ3_S file keeps layer 47's routed experts in it), transcribed from llama.cpp's vec_dot_iq4_xs_q8_1; checked on all 48 layers against ggml's float reference (native_expert_parity: 0 failures). - MTP draft head: gather_rows takes rows of any size (the IQ3_S head is Q6_K, 2100-byte rows, not a multiple of 16). - setup: IQ3_S for the original model (Swift 1.5 has none), 64 GB RAM, context capped at 128K like IQ3_XXS. - README / DETAILS: IQ3_S speed (RTX 5070, Ryzen 5 7600, 64 GB): 54 tok/s at 4K, 42 at 128K, 374-397 prompt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
A missing, null, 0 or -1 max_tokens / max_completion_tokens now means the rest of the context (minus the 8-token margin strata --serve keeps), so replies no longer stop at 1024 tokens. prepare() resolves the budget after tokenizing and returns it; an explicit budget is honoured exactly and still rejected when prompt + budget exceeds the context. Adds serve/test_server.py.
Temperature was ignored because the verify window's head sampling was pinned to greedy INSIDE the captured CUDA graph (verify.cpp hardcoded sp.greedy = true), and the GEN protocol carried no sampling keys. - the sampling moves OUT of the captured graph: the graph computes the head logits, and run() applies sample_tokens host-side after the replay, with that request's own parameters and a fresh Philox counter per window (a captured sampler would replay the same draws forever) - the GEN protocol accepts optional key=value pairs before the ids: temperature=, top_p=, top_k=, seed=; absent keys stay greedy, exactly as before - server.py forwards the request's temperature/top_p/top_k/seed - emitted tokens are always the TARGET's own samples (the drafts only fill the verified window), so the output distribution is the target's regardless of what the drafts propose Building on the conversation cache: greedy requests stay bit-identical, sampled requests with the same seed replay identically.
…lt top_k=20 The sampled path requires top_k in 1..64 (the selection loop is O(k*n_vocab) by design); with the engine-default top_k=0 it returned without writing the token, leaking stale buffer contents as output. Sampled requests now default to the sampler's own top_k=20. The Philox draw counter also must persist across a request's verify windows - it was read from a run-local variable that reset every window, so every window redrew the same numbers (identical sampled output on identical requests).
The sampled chain ran in ONE thread per token (64-thread launch): top_k alone was k sequential full-vocabulary scans with an inner sweep over the taken list, and a verify window measured ~1.6 s in the sampler - sampled requests decoded at 1.5 tok/s against 54 greedy. The selection is k argmax rounds, so it parallelises exactly like the greedy kernel: one block per token, the rounds back to back inside it, each a two-level (warp, then block) reduction. Ties resolve to the lowest index (the serial scan's strict '>'), so the kept sequence - set and order - is unchanged, and the top_p cut, temperature and Philox draw read that order as before. sampler_parity passes; a 248,320-vocab GPU-vs-CPU draw check matches the serial reference on five parameter sets including injected duplicate logits. Measured (RTX 4070 TS, IQ3_XXS, 128-token generations): sampled decode 1.50 -> 42.7 tok/s; greedy decode unchanged (53.3 vs 54.3 tok/s median, within run noise; greedy launches the untouched kernel).
StrataEngine.generate accepted the request's sampling parameters and dropped them, so every served request decoded greedy whatever the client asked - the README's 'temperature / top_p / top_k / seed are honored per request' was not true over the HTTP API, and the engine-side per-request sampling was unreachable from it. The keys ride the GEN line in the engine's own spelling; an absent temperature keeps the engine's greedy default and temperature=0 is not forwarded (it means the same thing); top_k outside the sampled path's 1..64 is dropped rather than sending a value the engine would refuse. Verified end to end over the HTTP server (static residency for the seeded leg): temperature=0 and absent agree token for token; a temperature=0.8 request differs from greedy; two temperature=0.8 seed=99 requests return identical text.
It contradicted the paragraph next to it, and since the sampling keys now ride the GEN line (and the block-parallel sampler makes a sampled window cost about what a greedy one does), it was simply false.
The run config's optional `sampling` block (temperature, top_p, top_k, seed) sets defaults for the fields a request leaves out, so a client that sends no sampling keys gets configured sampling instead of greedy. The request's own fields always win: an explicit temperature=0 still means greedy, and a field set to null falls back to the default. Keys the sampled path does not implement yet (min_p, presence_penalty, repetition_penalty, frequency_penalty) are named at startup and ignored rather than silently dropped; unknown keys are reported the same way. An invalid value (top_k outside 1..64, a non-positive seed, a negative temperature) refuses to start the server - a typo'd config should not quietly change sampling. Verified with a spy engine through Service.run: defaults apply when the request omits the fields, the request's own values win, a null field falls back to the default, and a server without the block forwards the request unchanged; the config validator rejects out-of-range values and warns on the not-yet-implemented keys.
The sampled chain's last two gaps close: `min_p` and the llama.cpp penalty set (repeat/freq/present over a history window) are live end to end - GEN keys, per-request params, and the token history the penalties count over. - GEN accepts min_p=F penalty_last_n=N penalty_repeat=F penalty_freq=F penalty_present=F between max_new and the ids (the server's spelling); the engine clamps defensively (min_p to 0..1, a negative last_n to 0) and skips what it does not know, as before. - min_p cuts the top_k survivors to the prefix with p >= min_p * p_max - in logits, sel_logit >= sel_logit[0] + logf(min_p) - before top_p, so the two cuts compose; 0 disables, and the head always survives, so the kept count never reaches zero. - the penalties need the token history: the serve loop uploads the last penalty_last_n tokens - the consumed tail plus the fed-back head, which joins consumed_seq only after the window commits - into one device row before every window. A request without penalties hands the sampler a null row and takes the byte-for-byte neutral path (no upload, nothing handed over). Verified: sampler_parity grows a full-chain host reference (the Philox draw transcribed host-side) and fixtures for the sampled chain with penalties and for min_p at 0/0.5/0.9 - the kernel matches the reference on all of them, and each fixture first asserts it can SEE the feature before asserting the kernel matches. On the IQ3_XXS artifact, same seed: a request with neutral penalty keys is token-identical to one without them; a six-copy prompt that loops 24/24 on one token clean produces 24 distinct tokens with repeat 1.5 + present 0.5; min_p 0.9 changes the pick.
min_p, presence_penalty, repetition_penalty and frequency_penalty move from "warned and ignored" to the sampled path, and penalty_last_n sets the window they count over - the server sends the engine's default of 64 when a penalty is set and the client left the window out. sampling_keys forwards each field only when it is not neutral, so a client (or a config) that leaves them out costs nothing. The validator range-checks the new fields and still refuses to start the server on a value it cannot honour. Verified: the validator accepts and range-checks min_p/presence/ frequency/repetition/penalty_last_n; the GEN line spells them the engine's way and omits neutral values; the last-64 default appears only when a penalty is actually set.
The penalties made the sampled path pay O(k x n_vocab x hlen) integer compares per token: every top_k argmax round recounted every candidate's history occurrences (k=20 rounds, hlen=64, ~318M compares on the 248,320 vocabulary) - measured 45.2 -> 30.8 tok/s on a real chat workload. But the history touches at most hlen DISTINCT tokens, so membership is a bit: each block now builds a shared bitmap from its history row (atomicOr, one pass) and a candidate pays the O(hlen) count scan only when its bit is set - at most hlen x k scans per token instead of k x n_vocab. Neutral values and the no-history path skip the bitmap entirely. The counts - and therefore every penalised value, kept token and draw - are exactly what the per-candidate scan produced; sampler_parity passes unchanged, plus a new fixture pinning the window clamp (an 8-entry history at penalty_last_n=4 must punish only the tail half).
From 64K context the KV cache lives in pinned RAM and a VRAM block cache holds the part the sparse attention reads (--kv-resident N; setup turns it on with 32768 when RAM allows). Blocks the indexer selects are made resident on the device inside the captured graphs (clock eviction, zero-copy reads of misses); writers write through, so the attention reads exactly the same values. The freed VRAM goes to the expert cache: Q2_0 at 262K decodes 62.6 tok/s instead of 50.9. The MTP drafter keeps its window in a ring. New ctest kv_stream_parity. Also wires the opt-in suffix drafter (--suffix-draft) into the engine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… is left (engine 0.1.6)
The MTP drafter's ring over a host copy is set up after the expert arena has pinned what it could; when that
pinned allocation fails the drafter now falls back to its fully resident KV instead of stopping the engine
("mtp: state init failed", seen with Swift 1.5 at 262K). setup requires engine 0.1.6.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ne 0.1.7) - Prompt lookup on by default: the suffix drafter's windows are taken only where a learned policy (measured acceptance per match length, measured window cost) expects more tokens per ms than the MTP window; lookup windows up to --spec + 2. Code edits +6-11%, other text unchanged, verified output identical. --suffix-draft 0 = off. - Per-request sampling (Niko1221#19) keeps greedy requests on the in-graph sampler (no extra launch per window), and draws row t of a window with Philox(seed, pos0 + t), so a seed replays the same text whatever the drafts were. - tools: a shallow install (G:\Strata) no longer crashes the MTP packer and the pack tools (IndexError, Niko1221#20). - setup: says at step 1 when the PC has less RAM than the smallest model needs; requires engine 0.1.7. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
- Implement Q4_0 KV quantization and dequantization kernels with scale/offset - Implement fast Walsh-Hadamard transform (FWHT-256) rotation for outlier suppression - Add Q4_0 attention decode support in qsa_decode_attn - Add kv_q4_parity parity test verifying CPU vs CUDA numerical accuracy - Support --kv q4_0 in generate CLI and runtime layer allocation - Update .gitignore for build and repository hygiene
- Implement Q4_0 KV quantization and dequantization kernels with scale/offset - Implement fast Walsh-Hadamard transform (FWHT-256) rotation for outlier suppression - Add Q4_0 attention decode support in qsa_decode_attn - Add kv_q4_parity parity test verifying CPU vs CUDA numerical accuracy - Support --kv q4_0 in generate CLI and runtime layer allocation - Update .gitignore for build and repository hygiene
…egrated with KV streaming; accurate expert-cache note (Niko1221#23) PR Niko1221#21 (code-martin): a 4-bit K/V format (orthonormal FWHT-256 rotation of K, V and Q, then q4_0 blocks). On merge: - Q4 is a full third format of the KV-streaming code (host copy, block residency, prompt-path staging, MTP ring, STATE_HASH): with --kv-resident the PR wrote K/V through the residency map's -1 entries. kv_stream_parity now checks Q4 streamed vs resident, bitwise. - The per-token path ran attention twice (once before the blocks were resident); once now. - The q4 group maximum breaks |x| ties the same way in every lane (one scale per block). - Dropped: the duplicate path fixes (in 0.1.7), personal .gitignore entries, the re-encoded CMakeLists. - Measured (bench/results/2026-09-27-kv-q4): int8 matches fp16; q4_0 costs precision on long documents (perplexity +8-12%), needles 5/5, +3.7% decode at 128K. Opt-in: setup asks above 8K context (int8 recommended), START-HERE.bat --kv q4_0, or --kv q4_0 in the config. Niko1221#23: the start-up warning "the GPU hit path is NOT CORRECT" dated from a long-fixed fault. Measured teacher-forced on 2,557 tokens (bench/results/2026-09-27-cache-parity): cache on vs off, 95-98% same top-1 and equal perplexity (-0.005 +- 0.005 nats); the warning is now one accurate line. Co-Authored-By: code-martin <code-martin@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
The browser page at http://127.0.0.1:8080 is now the Strata app, in the Strata design language (serve/web, same origin, no CDN): - Chat: streaming answers, collapsible thinking, Markdown with code copy, images when enabled, sampling/thinking settings, export, new chat. - Monitor: live model state (reading with progress / generating at tok/s), GPU load, VRAM, temperature, power, PCIe traffic, CPU, RAM, disk, the context fill and the last requests, each with a minute of history. - About: the model and engine settings, this PC, the API addresses. Server: GET /metrics (API key when set), /web/* app files, /fonts/* only when installed locally (not shipped; the page falls back to the system font). serve/telemetry.py reads NVML via ctypes and psutil (new setup dependency) with fallbacks. The engine prints an INFO line before READY (kv, expert slots, speculation); older servers ignore it and the server works with older engines. Based on the dashboard idea in PR Niko1221#22 by code-martin. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…il picked; engine 0.1.8 - serve/web/fonts: Outfit as one variable font (latin + latin-ext, weights 100-900) with its OFL.txt, in place of the design kit's commercial font. - The page no longer saves the system's theme on first load (which froze it); only a click on the theme button or toggle is remembered, and the page follows system changes until then. - setup: requires engine 0.1.8 (--kv q4_0, the INFO line). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…took NVIDIA without mentioning the Radeon)
…pt pass in batches); setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…experts.bin; setup uses it when the model's experts do not fit the RAM beside the OS (a 32 GB PC with a big GPU: the experts come from the file cache, the GPU holds the most-used)
…ment Measured engine knobs (STRATA_HIPBLASLT_TUNING, STRATA_PF_EMBED_BATCH, STRATA_PREFILL_RING) previously lived only in one systemd unit's command line and vanished on any restart. The config's 'env' now wins over the inherited environment, so a promoted A/B state is expressed entirely in the config file. Verified on the gfx1100 rollout: the promoted resident serves with 'prefill gemm: hipBLASLt tuning enabled (30 rows, gfx1100, version 100100)' read from the config-driven env; paired A/B (results/tune-20260929T175811Z, T184449Z): long-prompt prefill 416->971 tok/s, decode -0.8% (inside the 3% protected cap), winner-vs-original decode +38.3%.
…K than 0.1.22) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
The tensor-core prompt-attention kernel is already refused at runtime on AMD (line 695), but the file unconditionally included <math_constants.h> for CUDART_INF_F, which is not present on a ROCm-only build. Replace it with the portable INFINITY macro from <cmath> so the HIP build succeeds without a CUDA toolkit installed.
Rebased onto main (0.1.22): the portable pre-sm_80 QSA scorer and the sm_80 guard on the tensor-core prompt attention are already in main, so this is the rest of the port. - the runtime floor: device_info refuses anything older than compute capability 7 (main's runtime check still requires sm_120, so a binary built for sm_75 would never start), - fused_gr: the down kernel carries the tokens in slices that fit the card's shared-memory opt-in (Turing, 64 KB: 6-token slices; 4 tokens when the card reports no opt-in, under the 48 KB default), merged on top of main's per-device opt-in - full 8-token windows no longer fail on Turing with "invalid argument", - the setup/CMake floor is 7.5 (RTX 20 / 30 / 40 / 50) without an experimental flag: CMake's arch refusals, setup.py's gpu_problem and the GPU hints and error texts, - docs/MULTI_GPU.md says what a Turing card actually runs. Verified end-to-end on RTX 2070 (sm_75, 0.1.20 base): model loads, OpenAI/ Anthropic API serving works, greedy output identical to the sm_120 build. Merged (PR Niko1221#87) with two changes: the runtime floor is 7.5 exactly (major*10+minor < 75; the PR let Volta 7.0/7.2 through), kept beside main's HIP check; and a card reporting no shared-memory opt-in slices the tokens to what fits the 48 KB default for its own tile (4 tokens on CUDA, all 8 with HIP's smaller tile, so gfx1100 keeps one launch). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…me text in a message stays text (Niko1221#150) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…the engine is compiled again for a card it has no code for, at setup and at start (Niko1221#128) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…ts added (Niko1221#137); setup replaces an older shipped subset at setup and at start, a hand-made one is kept Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… compiled here when it has no code for the card (BUILD.json vision_archs) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
… Korean tokens (40,525 -> 106,299 ids; Niko1221#137). Q2_0 on CJK answers +15-38% decode, English -1-2%, the same expert slots on 12 GB Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
…anese and Korean), RTX 20 cards, the HIP build without CUDA headers; setup requires it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
# Conflicts: # CMakeLists.txt # README.md # docs/AMD_HIP.md # docs/AMD_HIP_PERFORMANCE.md # include/strata/hip_compat/cuda_runtime.h # src/core/device.cu # src/kernels/cuda/qsa_prompt_attn.cu # src/kernels/cuda/qsa_select.cu # src/prefill/prefill.cpp # tests/hip/prefill_native_batch.cpp # tests/platform/direct_file_async.cpp
Re-application of d7a0d6b after the 0.1.27 merge (the upstream tree still has both defects): qsa_prompt_attn_parity links CUDA::cudart unconditionally, and native_mmvq_multi names bench/micro/native_mmvq_multi.cpp, a source in no commit of this repository or upstream. Verified by a HIP-only configure with STRATA_BUILD_TESTS=ON and the CTest run.
telemetry.py reads the GPU through NVML, so on an AMD box (no libnvidia-ml) every GPU series is absent: the Monitor's GPU load, VRAM, temperature, power and PCIe cards show "-", and hardware_static.gpu_name is null. Where psutil is not installed, the disk rate shows "needs psutil (setup installs it)" and the physical-core count is missing. _AmdSysfs reads the same fields from amdgpu's sysfs: the card is found by scanning /sys/class/drm/card*/device for DRIVER=amdgpu (the number is not fixed - a box with another DRM device can have the AMD card at card1), the temperature is the hwmon sensor labelled "edge" (what NVML calls the GPU temperature), power comes from power1_average/power1_cap, and the link from current/max_link_speed plus current_link_width. Every reading is a small file read, so a sample cannot block the server, and anything the kernel does not expose stays None rather than being invented. amdgpu has no PCIe throughput counter, so the link is reported and the MB/s series stays empty; the web app already renders "Gen4 x16" from those fields. The patch also adds /proc/diskstats (the source psutil itself reads; whole disks only, never partitions and never loop/zram/dm, which would count a device-mapper layer twice) and /proc/cpuinfo's (physical id, core id) pairs for the psutil-less case. NVML still wins whenever it loads, and the psutil path is untouched, so an NVIDIA install behaves exactly as before. Tests: serve/test_telemetry.py needs no GPU - the pure helpers, the no-card case, and a sample that must still carry every series the Monitor plots. (Rebased onto the multi-card Monitor f7262d0: the AMD backend now feeds the per-card readings and their total/mean/hottest aggregation of issue Niko1221#112 too. NVML still wins when it loads - the follow-up commit changes that for AMD.)
Two rules for boxes where the amdgpu scan or NVML alone picked the wrong card: - The _AmdSysfs card is the one at the model device's PCI slot (HIP ordinal -> PCI slot via libamdhip64, matched to the DRM card's uevent), not 'the index-th amdgpu' - a Ryzen iGPU is amdgpu too and takes index 0 of the card scan. - When the model runs on the AMD card the readings come from amdgpu's sysfs even where NVML loads (it ships with the NVIDIA driver and used to win), because 0.1.26's setup lets a mixed PC choose AMD. 'Runs on AMD' is the config's backend key (setup's own signal) or, without one, the engine binary's own linkage to libamdhip64 (a string-table read; the binary is not executed). On NVIDIA-only boxes nothing changes: NVML still wins there. Tests: serve/test_telemetry.py 8/8 with a fake sysfs (iGPU vs dGPU selection), fake backends (AMD wins over a loading NVML exactly when the model is on AMD) and the backend decision cases. docs/AMD_HIP.md no longer says the Monitor has no GPU statistics.
The fork's clock64() stamp variant conflicted with upstream's wall_clock64() rework at the merge (the splice did not compile). The wall-clock form is kept: it reports true nanoseconds, which is what the verify-window profiler's span labels assume; the clock64 form's 'relative spans only' caveat is a downgrade. This is the exact tree the 0.1.27 gates and the version A/B ran on.
xyzzing
added a commit
to xyzzing/Strata
that referenced
this pull request
Oct 4, 2026
…d opt-in switches, and the status of the two open performance PRs (Niko1221#755 table, Niko1221#744 autotune)
Owner
|
Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new Sorry for the noise, and thank you for the contribution. |
This was referenced Oct 7, 2026
Contributor
Author
|
Re-filed as #1297 — shipping the ported + measured version (not the 0.1.27-era patch this branch carried); re-based onto the new main with |
Owner
|
Thank you, xyzzing. The decode autotune found the shipped defaults already win (deltas under 1%), and the release workflow with contents: write is not something we can take, so this stays closed. The one useful result, --spec-min-p 0.70 at +10%, is a setting; a small PR or issue with that A/B is welcome. The PR was closed by the history rewrite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On-device decode autotuning for the AMD HIP backend (gfx1100): per-shape rows-per-block tuning for the decode expert kernels and
iq_mmvq(bitwise-exact variants only), a guarded tuning-table format, an on-device tuner, and an orchestrator that sweeps kernels, draft floor, CPU workers, and drafting windows with the user's own prompts. Settings-layer sweeps (--spec,--suffix-draft,--spec-min-p, workers) now work on HIP too, andcalibrateis HIP-aware (keeps the config's--pcie-frac).Both layers keep answers exactly the same: kernel variants are byte-compared on-device and end-to-end; the settings layer is greedy-verification-exact.
Measured on the target hardware (RX 7900 XTX 24 GB, IQ3_S, ROCm 7.1.1)
--spec-min-p 0.70= 63.1 tok/s vs baseline (~+10%), token-identity verified end-to-end — adopted in our serving config.strata-<model>.autotune.json) carries every measurement.Notes
4da9c59); a rebase onto current upstream main is queued (the files the patch touches are untouched by 0.1.36–0.1.38's kernel changes, so the rebase should be mechanical).