Skip to content

[0.1.43.1] Blackwell sm_120 (RTX PRO 6000): first CUDA-side measures of the opt-in switch season - GDN_CHUNKED and the S25/S26 fused stack measure neutral on the fully-resident tier; greedy outputs drift on 2/12 probes under the stack #2268

Closed as not planned

Description

@beastrobin

What this measures

First CUDA-side (sm_120) measurements I can find for the S25/S26 opt-in switch season, on the fully-resident tier. All switches applied as engine-subprocess environment only; nothing else in the config was touched between paired runs. Two findings: speed is neutral on this tier, and the "bitwise equal" claims do not survive end-to-end here - 2 of 12 greedy probes drift, while the untouched baseline is byte-reproducible run to run.

Hardware / software (same machine and configuration as #2261): NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB, sm_120), Ultra 9 285K, Windows 11 (26200), driver 610.74, power cap locked at 300 W. Model qwen3.8-flash-next-iq3_s, all 24576 experts VRAM resident (~100 % hit rate, zero-doorbell graph path), --kv int8 --kv-resident 0 --max-context 262144 --spec-min-p 0.70 --lookup-chain 4 --batch 2. Engine and server 0.1.43.1.

How the env vars were injected (reproducible)

Top-level "env" field in the serve config; child_env() forwards it to the engine subprocess only (scoped, no system environment). A restart per arm (~90 s with everything cached).

Experiment 1 - STRATA_GDN_CHUNKED=1: neutral here

RTX 5090 recurrence gains did not carry over. Runs=2, same day but a different noise regime than the evening pair below, so read these as "no visible gain", not a precise delta: chat decode 189.6 (162.8-251.4) vs 196.5 on the unchanged-config baseline earlier that day; 16K prompt 4191.6 (3192.6-5190.5) vs ~5086 stable; the in-arm spread exploded the way the paired runs later showed for the environment itself. I kept it off. Docs already say full-prompt time barely moves because attention still dominates; on sm_120 there is simply less recurrence share of prefill to win. README attached at the bottom.

Experiment 2 - fused-stack pair (same session window, minutes apart, runs=3 each)

Arm S env: STRATA_EXPERT_V2=1 STRATA_TSUM=1 STRATA_ATTN_LANECELL=1 STRATA_PLE_BATCH=1 STRATA_MMVF_ROWS=1. The five are CUDA-reachable opt-ins (getenv sites live in shared core / CUDA kernels; the arch table itself is never consulted by a CUDA build, so they default off unless set - except PLE_BATCH, see the asymmetry note). EXPERT_V2's shape gate (gu_type==21 && d_type==20 && n_embd==2560 && n_ff==640) matches this IQ3_S/IQ4_NL pack exactly.

metric fusion stack ON baseline (minutes apart) delta
chat decode tok/s 187.4 (146.4-242.0) 187.3 (148.8-240.3) ~0 %
agent decode tok/s 174.3 (116.2-211.4) 176.8 (111.6-212.2) within noise
4K fresh prompt tok/s 4140.7 4110.7 +0.7 %
16K fresh prompt tok/s 5055.2 5017.2 +0.8 %
32K fresh prompt tok/s 5083.1 5050.3 +0.7 %

Consistent but sub-percent prefill medians, flat decode: neutral on the fully-resident Blackwell tier - which fits the design intent (these fusions cut launches and shared-memory pressure where waves and LDS are scarce; this card has plenty of both). Both arms show wide in-run spread that evening; an earlier same-day untouched-baseline run was tight (ranges within a few percent), so treat this night's environment as noisy, not the switches. I kept them off. READMEs attached below.

The more interesting result: end-to-end greedy drift under the stack

A fixed 12-question set at temperature=0, 96 tokens, thinking off, captured while the stack was live and again after reverting to baseline: 10/12 byte-identical. The two that differ do so deep inside the generation (first difference around character 155-198): $3$ vs **3** formatting choice, one heading wording change - semantically equivalent answers, different token streams from roughly token 40 onward.

Why I attribute it to the switches and not server non-determinism: repeated identical probes against the reverted baseline reproduce its own capture byte-for-byte. So greedy decoding is stable per engine state on this machine, and the drift tracks the switch combination. This does not contradict the S26 harness (which memcmp-compares expert-kernel outputs for T=1..3 blobs); it says that across full prompt phases, cache states and shapes actually seen in service, some part of the five-switch combination perturbs logits enough for near-ties to flip. Given how #783 turned on exactly this question, the datapoint seemed worth putting somewhere public. Both probe transcripts attached verbatim.

Defaults asymmetry noticed while reading the code (not a bug claim)

STRATA_PLE_BATCH: unset counts as ON at src/prefill/prefill.cpp (~line 2114: v == nullptr || atoi(v) != 0) but OFF at a branch in verify.cpp (~line 971), so "=1" is partly a no-op depending on path. STRATA_SH_STREAM already defaults ON in CUDA builds (sh_stream_on(), HIP-only else-branch). Since arch_defaults.cpp's season table is AMD-gfx by design, these switches are effectively undiscoverable from a CUDA install unless documented - an arch-table note or docs line (sm_120: measured neutral here) might save others the restart cycle.

Not tested

EXPERT_V2K (UD-Q4_K_XL path), QFUSE / LFUSE / Q8_PACKED arms, non-fully-resident tiers, multi-GPU, thinking-on profiles; n=2-3 runs, one machine, power-capped. Happy to run further arms with ~1 minute windows each if useful.

README: GDN arm (runs 2)

Community benchmark on NVIDIA RTX PRO 6000 Blackwell Workstation Edition

Measured on 2026-10-13 with tools/strata_bench.py (suite 1, 2 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

  • GPU and VRAM: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GiB VRAM
  • CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 253.4 GiB; storage: not measured
  • OS: Windows 11 (10.0.26200); GPU driver: 610.74; CUDA: not read
  • Strata: project 0.1.43.1, commit not a git checkout; engine 0.1.43.1
  • Background workloads and power limits: not recorded

Model and configuration

  • Model and quantization: qwen3.8-flash-next-iq3_s
  • Context: 262144 tokens; parallel slots: 2
  • Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py --long --runs 2 against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

  • short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 2 runs
  • 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 2 fresh prompts each
  • agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 2 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
  • concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Configuration Actual prompt tokens Reused tokens Generated tokens Runs Prompt tok/s median and range Decode tok/s median and range TTFT seconds median and range
short chat 46 (43-51) 0 100 (100-100) 6 not measured 189.6 (162.8-251.4) 0.24 (0.22-0.33)
4K prompt, fresh 4019 (4019-4019) 0 (0-0) 24 cap 2 3112.8 (1955.9-4269.6) not measured 1.52 (0.95-2.09)
16K prompt, fresh 16058 (16058-16058) 0 (0-0) 24 cap 2 4191.6 (3192.6-5190.5) not measured 4.09 (3.12-5.06)
32K prompt, fresh 32049 (32049-32049) 0 (0-0) 24 cap 2 5042.1 (4828.8-5255.5) not measured 6.42 (6.15-6.69)
agent session, follow-up turns 14493 (14484-14502) at the last turn see below up to 64 2 see below 187.8 (125.7-246.8) 1.12 (0.59-2.73)

Ranges are the lowest and highest of the runs; Runs counts requests that returned a token.

Agent session (conversation cache)

Cache hit over the follow-up turns: 68.6 % (68.4-68.8); earlier-turn tokens read again per session: 102 (3-200) (earlier turns that the cache did not hold; the 13633 (13633-13633) new tokens of the files are read once); first turn (nothing cached) first token: 0.80 s (0.71-0.88).

Turn (run 1) Prompt tokens Reused Read Prompt tok/s First token (s) Decode tok/s
0 609 0 609 712.6 0.882 171.3
1 3714 634 3080 1141.2 2.726 200.4
2 5247 3707 1540 1226.9 1.293 175.6
3 8691 5240 3451 1409.7 2.486 140.5
4 11607 8747 2860 1204.2 2.404 200.1
5 14502 11600 2902 1381.5 2.133 125.7

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load and power limits are not recorded. Total time of the suite: 53 s.

README: fusion-stack arm S (runs 3)

Community benchmark on NVIDIA RTX PRO 6000 Blackwell Workstation Edition

Measured on 2026-10-13 with tools/strata_bench.py (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

  • GPU and VRAM: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GiB VRAM
  • CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 253.4 GiB; storage: not measured
  • OS: Windows 11 (10.0.26200); GPU driver: 610.74; CUDA: not read
  • Strata: project 0.1.43.1, commit not a git checkout; engine 0.1.43.1
  • Background workloads and power limits: not recorded

Model and configuration

  • Model and quantization: qwen3.8-flash-next-iq3_s
  • Context: 262144 tokens; parallel slots: 2
  • Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py --long against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

  • short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 3 runs
  • 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
  • agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
  • concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Configuration Actual prompt tokens Reused tokens Generated tokens Runs Prompt tok/s median and range Decode tok/s median and range TTFT seconds median and range
short chat 46 (43-51) 0 100 (100-100) 9 not measured 187.4 (146.4-242.0) 0.23 (0.21-0.25)
4K prompt, fresh 4019 (4019-4019) 0 (0-0) 24 cap 3 4140.7 (1935.8-4173.8) not measured 0.99 (0.99-2.11)
16K prompt, fresh 16058 (16058-16058) 0 (0-0) 24 cap 3 5055.2 (3206.9-5060.2) not measured 3.22 (3.22-5.04)
32K prompt, fresh 32049 (32049-32049) 0 (0-0) 24 cap 3 5083.1 (4679.0-5103.8) not measured 6.35 (6.33-6.90)
agent session, follow-up turns 14521 (14468-14523) at the last turn see below up to 64 3 see below 174.3 (116.2-211.4) 0.89 (0.57-2.84)

Ranges are the lowest and highest of the runs; Runs counts requests that returned a token.

Agent session (conversation cache)

Cache hit over the follow-up turns: 68.8 % (68.7-68.9); earlier-turn tokens read again per session: 5 (2-74) (earlier turns that the cache did not hold; the 13635 (13632-13635) new tokens of the files are read once); first turn (nothing cached) first token: 0.70 s (0.70-0.76).

Turn (run 1) Prompt tokens Reused Read Prompt tok/s First token (s) Decode tok/s
0 609 0 609 840.3 0.758 169.3
1 3717 634 3083 1099.3 2.839 192.9
2 5250 3780 1470 978.4 1.527 150.8
3 8694 5313 3381 1375.8 2.49 146.4
4 11593 8733 2860 1196.5 2.409 211.4
5 14468 11622 2846 1522.0 1.904 149.7

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load and power limits are not recorded. Total time of the suite: 73 s.

README: paired baseline (runs 3)

Community benchmark on NVIDIA RTX PRO 6000 Blackwell Workstation Edition

Measured on 2026-10-13 with tools/strata_bench.py (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

Hardware and software

  • GPU and VRAM: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GiB VRAM
  • CPU: Intel(R) Core(TM) Ultra 9 285K (24 cores, 24 threads); installed RAM: 253.4 GiB; storage: not measured
  • OS: Windows 11 (10.0.26200); GPU driver: 610.74; CUDA: not read
  • Strata: project 0.1.43.1, commit not a git checkout; engine 0.1.43.1
  • Background workloads and power limits: not recorded

Model and configuration

  • Model and quantization: qwen3.8-flash-next-iq3_s
  • Context: 262144 tokens; parallel slots: 2
  • Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

Method

Run by python tools/strata_bench.py --long against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

  • short chat: the three questions of tools/ab_engine.py, up to 100 new tokens, 3 runs
  • 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
  • agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (cached_tokens) and timings (prompt_n)
  • concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

Results

Configuration Actual prompt tokens Reused tokens Generated tokens Runs Prompt tok/s median and range Decode tok/s median and range TTFT seconds median and range
short chat 46 (43-51) 0 100 (100-100) 9 not measured 187.3 (148.8-240.3) 0.23 (0.22-0.30)
4K prompt, fresh 4019 (4019-4019) 0 (0-0) 24 cap 3 4110.7 (1933.1-4124.2) not measured 1.01 (0.99-2.11)
16K prompt, fresh 16058 (16058-16058) 0 (0-0) 24 cap 3 5017.2 (3164.9-5031.3) not measured 3.24 (3.23-5.09)
32K prompt, fresh 32049 (32049-32049) 0 (0-0) 24 cap 3 5050.3 (4655.9-5065.5) not measured 6.39 (6.37-6.92)
agent session, follow-up turns 14492 (14478-14515) at the last turn see below up to 64 3 see below 176.8 (111.6-212.2) 0.95 (0.57-2.97)

Ranges are the lowest and highest of the runs; Runs counts requests that returned a token.

Agent session (conversation cache)

Cache hit over the follow-up turns: 68.8 % (68.5-68.8); earlier-turn tokens read again per session: 4 (4-143) (earlier turns that the cache did not hold; the 13634 (13633-13634) new tokens of the files are read once); first turn (nothing cached) first token: 0.70 s (0.70-0.71).

Turn (run 1) Prompt tokens Reused Read Prompt tok/s First token (s) Decode tok/s
0 609 0 609 879.5 0.715 160.3
1 3714 634 3080 1048.2 2.973 191.0
2 5247 3777 1470 971.1 1.529 171.9
3 8691 5310 3381 1372.8 2.5 148.6
4 11608 8748 2860 1175.7 2.451 209.2
5 14478 11632 2846 1470.3 1.976 137.7

Per-request data: results.json. Engine log lines from the start: engine-log-excerpt.txt (when this run started the server).

Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, nvidia-smi/rocm-smi and the OS; background load and power limits are not recorded. Total time of the suite: 74 s.

Greedy probe transcripts, stack live / baseline (full text)
[
 {
  "q": "What is 173-99+56?",
  "a": "To solve the expression $173 - 99 + 56$, we can follow these steps:\n\n**Step 1: Perform the subtraction first (left to right)**\n$$173 - 99 = 74$$\n\n*Calculation detail:*\n$173 - 100 = 73$, and since we subtracted one too many, add 1 back: $73 + 1 = 7"
 },
 {
  "q": "A jar has 240 grams of flour. A baker uses one fifth of the flour, then adds 14 more grams. How many grams are in the jar now?",
  "a": "Here is the step-by-step calculation to determine the final amount of flour in the jar:\n\n**Step 1: Calculate the amount of flour used.**\nThe baker uses one-fifth ($\\frac{1}{5}$) of the initial $240$ grams.\n$$ \\text{Flour used} = 240 \\times \\frac{1}{5} = 48 \\text{ grams} $$\n\n**Step 2:"
 },
 {
  "q": "Which number is larger, 3.11 or 3.8?",
  "a": "To determine which number is larger between **3.11** and **3.8**, we can compare them digit by digit from left to right:\n\n1.  **Compare the whole numbers:** Both numbers have a whole number part of $3$. Since they are equal, we move to the decimal parts.\n2.  **Compare the tenths place:**\n    *   The tenths digit of $3.11$ is $1$.\n   "
 },
 {
  "q": "If a ball costs $1.10 and a bat costs $1.00 more than the ball, how much does the bat cost?",
  "a": "To determine the cost of the bat, we can break down the problem using simple algebra or logical deduction.\n\n### Step-by-Step Deduction\n\n1.  **Identify the cost of the ball:**\n    The problem states that the ball costs $\\$1.10$.\n\n2.  **Determine the relationship between the bat and the ball:**\n    The problem states that the bat costs $\\$1.00$ **more** than the ball"
 },
 {
  "q": "How many r's are there in the word strawberry?",
  "a": "To determine how many times the letter \"r\" appears in the word **strawberry**, let's break down the spelling of the word:\n\n1.  s\n2.  t\n3.  **r** (1st occurrence)\n4.  a\n5.  w\n6.  b\n7.  e\n8.  **r** (2nd occurrence)\n9.  **r** (3rd occurrence)\n"
 },
 {
  "q": "You have an 8x8x8 cube painted on all sides and cut into unit cubes. How many unit cubes have no paint on any face? Give just the count.",
  "a": "To determine how many unit cubes have no paint on any face, we need to identify the interior of the large cube that is not exposed to the outside surface.\n\n1.  **Understand the Structure**: The large cube has dimensions $8 \\times 8 \\times 8$. It is composed of $8^3 = 512$ unit cubes.\n2.  **Identify Painted Cubes**: A unit cube has paint if it lies on"
 },
 {
  "q": "List every prime number between 50 and 70 separated by commas.",
  "a": "To find the prime numbers between 50 and 70, we examine each integer in this range to determine if it has any divisors other than 1 and itself.\n\n1.  **50**: Even number (divisible by 2). Not prime.\n2.  **51**: Sum of digits is $5+1=6$, so divisible by 3 ($51 = 3 \\times 17$). Not prime."
 },
 {
  "q": "Reproduce exactly this sentence and nothing else: The quick brown fox jumps over the lazy dog near the eight mile river.",
  "a": "The quick brown fox jumps over the lazy dog near the eight mile river."
 },
 {
  "q": "Tom is older than Jess. Jess is older than Alex. Who is youngest?",
  "a": "Based on the information provided:\n\n1. Tom is older than Jess ($Tom > Jess$).\n2. Jess is older than Alex ($Jess > Alex$).\n\nCombining these statements, we get the order of age from oldest to youngest:\n$$Tom > Jess > Alex$$\n\nTherefore, **Alex** is the youngest."
 },
 {
  "q": "Convert 1011 (base two) to decimal. Give just the number.",
  "a": "11"
 },
 {
  "q": "I put on my socks at 7:02, my shoes at 7:06, my coat at 7:11, and left at 7:15. Which did I put on first?",
  "a": "Based on the timeline you provided, here is the order of events:\n\n1. **Socks**: 7:02\n2. **Shoes**: 7:06\n3. **Coat**: 7:11\n4. **Left**: 7:15\n\nYou put on your **socks** first."
 },
 {
  "q": "What is one third of 576 plus 41? Give just the result as a whole number.",
  "a": "233"
 }
]
[
 {
  "q": "What is 173-99+56?",
  "a": "To solve the expression $173 - 99 + 56$, we can follow these steps:\n\n**Step 1: Perform the subtraction first (left to right)**\n$$173 - 99 = 74$$\n\n*Calculation detail:*\n$173 - 100 = 73$, and since we subtracted one too many, add 1 back: $73 + 1 = 7"
 },
 {
  "q": "A jar has 240 grams of flour. A baker uses one fifth of the flour, then adds 14 more grams. How many grams are in the jar now?",
  "a": "Here is the step-by-step calculation to determine the final amount of flour in the jar:\n\n**Step 1: Calculate the amount of flour used.**\nThe baker uses one-fifth ($\\frac{1}{5}$) of the initial $240$ grams.\n$$ \\text{Flour used} = 240 \\times \\frac{1}{5} = 48 \\text{ grams} $$\n\n**Step 2:"
 },
 {
  "q": "Which number is larger, 3.11 or 3.8?",
  "a": "To determine which number is larger between **3.11** and **3.8**, we can compare them digit by digit from left to right:\n\n1.  **Compare the whole numbers:** Both numbers have a whole number part of **3**. Since they are equal, we move to the decimal parts.\n2.  **Compare the tenths place:**\n    *   The tenths digit of 3.11 is **1**.\n    *"
 },
 {
  "q": "If a ball costs $1.10 and a bat costs $1.00 more than the ball, how much does the bat cost?",
  "a": "To determine the cost of the bat, we can break down the problem using simple algebra or logical deduction.\n\n### Step-by-Step Deduction\n\n1.  **Identify the known values:**\n    *   Cost of the ball = $\\$1.10$\n    *   The bat costs $\\$1.00$ **more** than the ball.\n\n2.  **Set up the equation for the bat's cost:**\n    $$ \\"
 },
 {
  "q": "How many r's are there in the word strawberry?",
  "a": "To determine how many times the letter \"r\" appears in the word **strawberry**, let's break down the spelling of the word:\n\n1.  s\n2.  t\n3.  **r** (1st occurrence)\n4.  a\n5.  w\n6.  b\n7.  e\n8.  **r** (2nd occurrence)\n9.  **r** (3rd occurrence)\n"
 },
 {
  "q": "You have an 8x8x8 cube painted on all sides and cut into unit cubes. How many unit cubes have no paint on any face? Give just the count.",
  "a": "To determine how many unit cubes have no paint on any face, we need to identify the interior of the large cube that is not exposed to the outside surface.\n\n1.  **Understand the Structure**: The large cube has dimensions $8 \\times 8 \\times 8$. It is composed of $8^3 = 512$ unit cubes.\n2.  **Identify Painted Cubes**: A unit cube has paint if it lies on"
 },
 {
  "q": "List every prime number between 50 and 70 separated by commas.",
  "a": "To find the prime numbers between 50 and 70, we examine each integer in this range to determine if it has any divisors other than 1 and itself.\n\n1.  **50**: Even number (divisible by 2). Not prime.\n2.  **51**: Sum of digits is $5+1=6$, so divisible by 3 ($51 = 3 \\times 17$). Not prime."
 },
 {
  "q": "Reproduce exactly this sentence and nothing else: The quick brown fox jumps over the lazy dog near the eight mile river.",
  "a": "The quick brown fox jumps over the lazy dog near the eight mile river."
 },
 {
  "q": "Tom is older than Jess. Jess is older than Alex. Who is youngest?",
  "a": "Based on the information provided:\n\n1. Tom is older than Jess ($Tom > Jess$).\n2. Jess is older than Alex ($Jess > Alex$).\n\nCombining these statements, we get the order of age from oldest to youngest:\n$$Tom > Jess > Alex$$\n\nTherefore, **Alex** is the youngest."
 },
 {
  "q": "Convert 1011 (base two) to decimal. Give just the number.",
  "a": "11"
 },
 {
  "q": "I put on my socks at 7:02, my shoes at 7:06, my coat at 7:11, and left at 7:15. Which did I put on first?",
  "a": "Based on the timeline you provided, here is the order of events:\n\n1. **Socks**: 7:02\n2. **Shoes**: 7:06\n3. **Coat**: 7:11\n4. **Left**: 7:15\n\nYou put on your **socks** first."
 },
 {
  "q": "What is one third of 576 plus 41? Give just the result as a whole number.",
  "a": "233"
 }
]

Activity

  1. beastrobin commented on Oct 13, 2026

    @beastrobin
    Author

    DUMMY_NODES probe closes the fused-kernel question for this tier: node overhead here is <= ~0.5 us, so fusing everything leaves only a few percent at best.

    To decide whether the #783-style fused decode kernels (+9-10 % claimed on fully-resident 2x3090 rigs) justify a build-toolchain project on the RTX PRO 6000 96 GB fully-resident arm of this box, I used the probe hook v0.1.43.1 already ships: STRATA_DUMMY_NODES=k adds k empty one-thread kernel nodes per layer inside the verify window (verify.cpp, next to the QFUSE read group). One env, two engine restarts, benchmark suite paired back to back.

    Setup: k = 32 over 48 layers = 1,536 extra graph nodes per verify window. The real chain runs roughly 10-20 launches per layer (~500-1,000 nodes/window), so this inflates the node count by 1.5-3x - if node cost mattered at all, it has somewhere to show. Arms alternated through the config "env" field (injected into the engine subprocess by serve/server.py child_env); same-day quiet evening regime; tools/strata_bench.py --runs 3 --long; suite index paired across arms because suite #1 after every fresh engine start pays a ~2x decode cold tax and was dropped from both sides (dummy steady n=27 chat / 54 agent, clean steady n=18 / 36).

    group dummy arm med clean arm med delta
    short chat 183.8 178.6 +2.9 %
    agent follow-up turns 178.9 184.8 -3.1 %
    16K prompt decode 164.3 169.3 -3.0 %
    32K prompt decode 139.0 145.8 -4.7 %
    4K prompt decode 142.9 118.7 +20 % (n=6, not believable)

    Signs flip between groups and the two high-n groups cancel: 1,536 extra nodes per window are indistinguishable from this machine's noise (ceiling about +/-5 % at these sample sizes). That puts a dependent kernel-node at <= ~0.3-0.5 us in the CUDA PDL chain - against the ~2.4 us quoted for one node on gfx1201 in verify_kernels.hpp. The absolute ceiling for fusing away every real per-layer node is then roughly 48 x 15 x 0.5 us ~ 0.4 ms/window ~ 2-3 % decode, and any realistic fusion keeps most of the work fused-kernel side, so well under that.

    Read: #783 stays dead at the fully-resident Blackwell tier; there is no 9-10 % there without changing what the kernels do per token, only how many times they start. The graph-launch-reduction parts can still matter where residency forces host traffic (the pull/doorbell paths), which is a different cost than node count. Not tested here: partial residency, concurrency arms (--clients off), HIP side, or engine builds other than 0.1.43.1 release.

    Baseline suite for this box went in as #2280; probe method and raw suites are noted in our local log if a maintainer ever wants them transcribed.

  2. qni-live commented on Oct 17, 2026

    @qni-live
    Contributor

    Adding a second sm_120 data point, on Linux with two cards: 2x RTX PRO 4500 Blackwell (GB203, 200 W, memory at 14,000 MHz), layer split 25, Swift IQ3_XXS, all experts resident, --kv k8v4, v0.1.45 (no local patches), solo.

    The three bit-exact or near-exact switches from the 0.1.45 table, decode after a 32K prompt (bench-style: fresh engine per arm, 3 runs per arm, 3 arms per variant in rotating order, so 9 values per variant; medians):

    Switch decode tok/s vs default
    default 174.5 (range 144.7-179.2)
    STRATA_ATTN_DAILY=1 177.7 (165.7-199.3) +1.8 %
    STRATA_QSA_SPLIT_ATTN=1 171.7 (161.3-180.5) -1.6 %
    STRATA_EXPDEC=1 169.9 (158.1-179.2) -2.6 %

    All inside the run-to-run spread; prompt reading is 5,267 to 5,272 tok/s in every variant, draft acceptance 68 to 70 %. So the switches are neutral here, in line with the finding in this issue on the RTX PRO 6000.
    Answers: 10 greedy prompts of 120 tokens (thinking off) give the same hash in all 12 arms, including STRATA_QSA_SPLIT_ATTN=1, which the table lists as rounding-level. The process environment of each arm was checked in /proc/<pid>/environ.

    Also for the record, logit_bias with 65,929 IDs on 0.1.45 costs +20 to +23 ms per request on this machine (0.1.44.1 with the old patch: +20 to +22 ms), and it works on the two-GPU layer split (25 CJK characters without the ban, 0 with it, same seed).

    Not measured: other context lengths, --batch, the other switches of the table.

    Written with AI assistance (Claude); numbers reviewed by me.

  3. Niko1221 commented on Oct 19, 2026

    @Niko1221
    Owner

    Thanks for taking the time to report this.

    We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

    To reopen it takes about two minutes:

    1. Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
    2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
    3. Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
    4. Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

    The rules are in CONTRIBUTING.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions