Repository navigation
Experiment: decode attention at deep context (two cells per step, a combine that skips empty chunks) #40
Description
Idea from Strata by @Niko1221 (MIT): STRATA_ATTN_DAILY (Strata ggml-org#2075). Decode attention processes two cells per step, and the merge of the split-KV partial results skips empty chunks. It is bit-exact. Decode after a 30K-token prompt went +3.2% on an RTX 3060 (5/5 pairs) and +0.7% on an RTX A4000 (DETAILS.md, "CUDA opt-in switches measured for 0.1.45").
Why it matters here
Agent conversations run at 30-100k tokens of context. At that depth the attention of the 16 full-attention layers of Qwen3.8 (and the QSA layers of qwen4exp) is a growing share of a decode step. The production cache is q5_1, read natively (PR 85).
With the unified cache lists (#32) there are padded, fully masked chunks at the end of a list; the split-KV combine still reads their partial results.
Experiment
- nsys profile of decode at 32k and 96k context (production config, MTP): the time of the FA kernels (vec / MMA for 3-4 query rows) and of the combine kernel per step.
- Combine: skip chunks whose rows were all masked (record them in the partial-results buffer, or bound them by
KV_maxper tile). - Vec kernel: process two KV cells per iteration for the quantized-native path (q5_1) and compare.
Measure
Decode t/s at 32k / 96k, interleaved pairs; test-backend-ops FLASH_ATTN_EXT (native quant types) bit-exact or within the current tolerance.
Activity
Piggidragon commentedon Oct 19, 2026
Profile at 29k context (2026-10-19), production config, MTP depth 2:
-as 1,3, and 0.72 ms on the 4070, of a 40.1 ms step.At this depth a 2x faster decode attention would save ~1.1 ms (~3%); the combine skip is worth <0.1 ms. The next check is 96k, where the share is about 3x larger.
Piggidragon commentedon Oct 19, 2026
98.6k context (2026-10-19)
Same setup, one slot at 98.6k tokens. The step takes 47.1 ms (36.8 at short context).
flash_attn_ext_f16<256,256,4,8,..., q5_1>), 374 us per layer for the 3060's 3 of 4 KV heads (-as 1,3). That reads ~114 MB of q5_1 K/V per layer, ~304 GB/s or 84% of the 3060's bandwidth.So the decode attention is bandwidth-bound. Two cells per step and skipping empty chunks in the combine can gain at most a few percent of the FA time, ~0.3-0.4 ms per step (<1%). Balancing the heads 2:2 would save ~2 ms per step, but
-as 1,3is needed to fit 2 x 131k context.The bigger cost at this depth is the host. The GPUs idle 6.8-7.3 ms per step, against 3.6 ms at short context:
--compact-attn-mask(#32) builds that mask on the device; a measurement of it at 96k follows.Piggidragon commentedon Oct 19, 2026
--compact-attn-mask(#32) at 98.6k, same setup:The attention kernels are unchanged (FA 6.94 ms per step on the 3060). So the deep-context cost that is easy to remove is the host-built mask, and the flag removes it. Recommended for production with per-slot streams; with
kv-unified-gatherthe mask ranges come from the cell list anyway.Piggidragon commentedon Oct 20, 2026
Closed: the decode attention at 98.6k is bandwidth-bound (84% of the 3060), so two cells per step and a combine that skips empty chunks could gain <1%. The deep-context cost that was easy to remove was the host-built KQ mask:
--compact-attn-mask(#32), -5.2% step time at 98.6k, deployed in production since 2026-10-19.🤖 Generated with Claude Code