Skip to content

bench: community report, 4x RTX 3090 layer split, stock 0.1.44 against 0.1.43 with a patch series, 1 to 16 users #2364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
havok2-htwo wants to merge 1 commit into from
Closed

bench: community report, 4x RTX 3090 layer split, stock 0.1.44 against 0.1.43 with a patch series, 1 to 16 users #2364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
378 changes: 378 additions & 0 deletions bench/results/2026-10-14-community-4x-rtx-3090-0144/README.md
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 30 additions & 0 deletions bench/results/2026-10-14-community-4x-rtx-3090-0144/bench-output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
[cb144] production down 04:12:10
# cb144-ours-short: prompt~100 tok, max_tokens 512, 2 request(s) per user, temperature 0.0
users ok/req wall agg out dec p50 dec p5 TTFT p50 TTFT p95 TTFT max e2e p50 e2e p95 steady
1 2/2 8.8 116.2 121.5 121.3 0.20 0.20 0.20 4.4 4.4 118.9
8 16/16 20.5 399.4 54.5 50.5 0.65 1.66 1.84 10.2 10.9 421.1
16 32/32 31.7 516.7 34.4 32.8 0.89 3.10 3.47 15.8 17.9 548.9
# gpu means over 72 samples (1 s): GPU0 1858 MHz 290 W 66 C (max 75) 70 % | GPU1 1872 MHz 286 W 65 C (max 74) 58 % | GPU2 1897 MHz 283 W 59 C (max 68) 56 % | GPU3 1842 MHz 290 W 67 C (max 76) 64 %
users ok/req wall agg out dec p50 dec p5 TTFT p50 TTFT p95 TTFT max e2e p50 e2e p95 steady
16 32/32 40.3 406.5 28.3 26.6 1.91 6.61 7.29 20.0 24.8 454.3
# gpu means over 43 samples (1 s): GPU0 1868 MHz 318 W 73 C (max 78) 74 % | GPU1 1883 MHz 321 W 72 C (max 76) 72 % | GPU2 1924 MHz 316 W 66 C (max 71) 62 % | GPU3 1870 MHz 314 W 74 C (max 78) 69 %
1 11131 0 15.25 28.80 10.8
2 15589 11440 1.82 5.38 16.2
3 20017 15863 0.98 3.61 22.2
wall 87s, output 13202 tok = 151.9 tok/s, prompt tokens sent 747829, follow-up TTFT p50 1.52 s
[cb144] arm ours took 343 s
# cb144-stock-short: prompt~100 tok, max_tokens 512, 2 request(s) per user, temperature 0.0
users ok/req wall agg out dec p50 dec p5 TTFT p50 TTFT p95 TTFT max e2e p50 e2e p95 steady
1 2/2 9.2 111.7 120.8 120.8 0.35 0.35 0.35 4.6 4.6 116.2
8 16/16 26.1 314.4 46.0 40.4 1.68 3.60 4.00 13.0 14.2 343.3
16 32/32 55.9 292.9 45.9 40.3 8.69 29.47 41.54 18.9 39.6 -
# gpu means over 102 samples (1 s): GPU0 1856 MHz 274 W 68 C (max 76) 59 % | GPU1 1870 MHz 274 W 67 C (max 74) 58 % | GPU2 1891 MHz 265 W 61 C (max 68) 48 % | GPU3 1825 MHz 276 W 69 C (max 77) 63 %
users ok/req wall agg out dec p50 dec p5 TTFT p50 TTFT p95 TTFT max e2e p50 e2e p95 steady
16 32/32 68.8 238.1 38.7 31.4 6.96 39.91 42.50 18.9 52.1 -
# gpu means over 71 samples (1 s): GPU0 1890 MHz 291 W 71 C (max 77) 69 % | GPU1 1912 MHz 295 W 69 C (max 74) 55 % | GPU2 1949 MHz 289 W 63 C (max 67) 45 % | GPU3 1874 MHz 292 W 71 C (max 77) 62 %
1 11131 0 28.02 82.98 12.0
2 15587 11131 14.43 51.40 14.1
3 20019 15608 4.68 76.65 16.5
wall 146s, output 13453 tok = 92.0 tok/s, prompt tokens sent 747809, follow-up TTFT p50 9.91 s
[cb144] arm stock took 401 s
[cb144] production started again 04:25:06
86 changes: 86 additions & 0 deletions bench/results/2026-10-14-community-4x-rtx-3090-0144/configs/ours-CB144-ours-prod-text.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
{
"exe": "<work>/Strata/engine/strata-g3-143v2",
"args": [
"--pack",
"<work>/Strata-data/packs/unsloth-ud-iq4_xs",
"--native",
"<models>/Strata-GGUF/unsloth-UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf",
"--expert-profile",
"<work>/Strata/data/expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"512",
"--spec",
"4",
"--spec-min-p",
"0.5",
"--mtp",
"<work>/Strata-data/mtp/rt",
"--max-context",
"262144",
"--kv",
"int8",
"--resident-experts",
"--trim-stage-weights",
"--ple-io",
"ram",
"--aux-cpus",
"65-127",
"--conversation-cache-mib",
"98304",
"--conversation-cache-slots",
"320",
"--kv-resident",
"32768",
"--conversation-cache-min-free-mib",
"16384",
"--conversation-cache-min-tokens",
"1024"
],
"cwd": "<work>/Strata",
"tokenizer": "<work>/Strata-data/packs/unsloth-ud-iq4_xs/tokenizer",
"model_name": "qwen3.8-flash-next-unsloth-ud-iq4_xs",
"log": "<work>/Strata/strata-cb144-ours.log",
"lib_dirs": [
"/usr/local/cuda-13.0/bin",
"/usr/local/cuda-13.0/lib64"
],
"port": 8080,
"gpu": [
0,
1,
2,
3
],
"gpus_asked": true,
"layer_split": "12,25,37",
"open_browser": false,
"parallel": 16,
"env": {
"STRATA_MMQ_DIRECT": "1",
"STRATA_SLOT_COPY_DIRECT": "1",
"STRATA_PIPE_BYIELD_NODRAIN": "1",
"STRATA_ADMIT_MERGE_FIRST": "1",
"STRATA_CKPT_POOL": "1",
"STRATA_CKPT_THREADS": "1",
"STRATA_SLOT_CKPT_MOVE": "1",
"STRATA_SLOT_COPY_BATCHSYNC": "1",
"STRATA_SPLIT_MTP_BATCH": "1",
"STRATA_MMQ_GROUP_SORT": "1",
"STRATA_BATCH_SELF_COMMIT": "1",
"STRATA_NAN_GUARD": "0",
"STRATA_QSA_SLOTS": "1",
"STRATA_HANDOFF_PIPE": "8",
"STRATA_PARK_ASYNC": "1",
"STRATA_SLOT_PARK": "1",
"STRATA_CONV_TIER_DIR": "<data>/strata-kvspill-bench",
"STRATA_KV_PREFETCH": "1",
"STRATA_CONV_TIER_WARM": "1",
"MALLOC_MMAP_THRESHOLD_": "16777216",
"MALLOC_TRIM_THRESHOLD_": "67108864",
"STRATA_CONV_TIER_MIN_TOKENS": "8192",
"STRATA_CONV_TIER_WRITE_GB_DAY": "300",
"STRATA_ADMIT_SHORT_FIRST": "8192"
}
}
67 changes: 67 additions & 0 deletions bench/results/2026-10-14-community-4x-rtx-3090-0144/configs/stock-CB144-stock-0.1.44-p8.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
{
"exe": "<work>/Strata-144/build-144/strata",
"args": [
"--pack",
"<work>/Strata-data/packs/unsloth-ud-iq4_xs",
"--native",
"<models>/Strata-GGUF/unsloth-UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf",
"--expert-profile",
"<work>/Strata/data/expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"512",
"--spec",
"4",
"--spec-min-p",
"0.5",
"--mtp",
"<work>/Strata-data/mtp/rt",
"--max-context",
"262144",
"--kv",
"int8",
"--resident-experts",
"--trim-stage-weights",
"--ple-io",
"ram",
"--aux-cpus",
"65-127",
"--conversation-cache-mib",
"98304",
"--conversation-cache-slots",
"320",
"--kv-resident",
"32768",
"--conversation-cache-min-free-mib",
"16384",
"--conversation-cache-min-tokens",
"1024"
],
"cwd": "<work>/Strata",
"tokenizer": "<work>/Strata-data/packs/unsloth-ud-iq4_xs/tokenizer",
"model_name": "qwen3.8-flash-next-unsloth-ud-iq4_xs",
"log": "<work>/Strata/strata-cb144-stock.log",
"lib_dirs": [
"/usr/local/cuda-13.0/bin",
"/usr/local/cuda-13.0/lib64"
],
"port": 8080,
"gpu": [
0,
1,
2,
3
],
"gpus_asked": true,
"layer_split": "12,25,37",
"open_browser": false,
"parallel": 8,
"env": {
"STRATA_SPLIT_MTP_BATCH": "1",
"STRATA_NAN_GUARD": "0",
"STRATA_KV_PREFETCH": "1",
"MALLOC_MMAP_THRESHOLD_": "16777216",
"MALLOC_TRIM_THRESHOLD_": "67108864"
}
}
173 changes: 173 additions & 0 deletions bench/results/2026-10-14-community-4x-rtx-3090-0144/gpu/gpu-ours-long.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
timestamp, index, clocks.current.sm [MHz], power.draw [W], temperature.gpu, utilization.gpu [%]
2026/10/14 04:15:50.921, 0, 1785, 133.84, 64, 4
2026/10/14 04:15:50.925, 1, 1785, 148.10, 64, 0
2026/10/14 04:15:50.928, 2, 1785, 139.05, 59, 0
2026/10/14 04:15:50.930, 3, 1785, 141.41, 64, 0
2026/10/14 04:15:51.933, 0, 1875, 182.53, 69, 86
2026/10/14 04:15:51.933, 1, 1785, 154.62, 68, 99
2026/10/14 04:15:51.933, 2, 1965, 145.07, 59, 3
2026/10/14 04:15:51.933, 3, 1935, 148.49, 69, 73
2026/10/14 04:15:52.933, 0, 1905, 266.16, 69, 95
2026/10/14 04:15:52.933, 1, 1950, 252.69, 67, 58
2026/10/14 04:15:52.933, 2, 1860, 241.45, 64, 56
2026/10/14 04:15:52.933, 3, 1935, 236.70, 67, 14
2026/10/14 04:15:53.933, 0, 1905, 242.62, 67, 5
2026/10/14 04:15:53.934, 1, 1950, 250.32, 66, 13
2026/10/14 04:15:53.934, 2, 1965, 262.40, 62, 72
2026/10/14 04:15:53.934, 3, 1860, 225.19, 66, 0
2026/10/14 04:15:54.934, 0, 1875, 248.00, 71, 22
2026/10/14 04:15:54.934, 1, 1860, 257.79, 70, 88
2026/10/14 04:15:54.934, 2, 1965, 244.86, 61, 11
2026/10/14 04:15:54.934, 3, 1905, 252.90, 72, 95
2026/10/14 04:15:55.934, 0, 1905, 268.45, 70, 94
2026/10/14 04:15:55.934, 1, 1950, 273.79, 67, 55
2026/10/14 04:15:55.934, 2, 1875, 243.59, 64, 51
2026/10/14 04:15:55.934, 3, 1935, 243.54, 68, 14
2026/10/14 04:15:56.934, 0, 1905, 244.11, 66, 7
2026/10/14 04:15:56.934, 1, 1950, 250.19, 67, 15
2026/10/14 04:15:56.934, 2, 1965, 264.77, 64, 86
2026/10/14 04:15:56.934, 3, 1860, 226.23, 66, 0
2026/10/14 04:15:57.934, 0, 1860, 243.13, 70, 18
2026/10/14 04:15:57.935, 1, 1860, 256.12, 70, 64
2026/10/14 04:15:57.935, 2, 1965, 249.61, 61, 16
2026/10/14 04:15:57.935, 3, 1860, 247.89, 72, 95
2026/10/14 04:15:58.935, 0, 1875, 323.76, 73, 100
2026/10/14 04:15:58.935, 1, 1905, 294.14, 72, 91
2026/10/14 04:15:58.935, 2, 1950, 314.15, 67, 82
2026/10/14 04:15:58.935, 3, 1860, 315.89, 74, 91
2026/10/14 04:15:59.935, 0, 1875, 377.30, 74, 100
2026/10/14 04:15:59.935, 1, 1890, 378.47, 73, 89
2026/10/14 04:15:59.935, 2, 1935, 374.30, 67, 81
2026/10/14 04:15:59.935, 3, 1860, 373.70, 75, 88
2026/10/14 04:16:00.935, 0, 1875, 378.41, 75, 100
2026/10/14 04:16:00.935, 1, 1890, 377.95, 73, 91
2026/10/14 04:16:00.935, 2, 1935, 375.66, 68, 81
2026/10/14 04:16:00.935, 3, 1860, 372.84, 75, 88
2026/10/14 04:16:01.935, 0, 1875, 376.79, 75, 100
2026/10/14 04:16:01.936, 1, 1875, 378.01, 73, 90
2026/10/14 04:16:01.936, 2, 1920, 375.79, 68, 82
2026/10/14 04:16:01.936, 3, 1860, 372.87, 76, 90
2026/10/14 04:16:02.936, 0, 1875, 377.24, 75, 100
2026/10/14 04:16:02.936, 1, 1890, 377.26, 74, 91
2026/10/14 04:16:02.936, 2, 1935, 374.61, 68, 79
2026/10/14 04:16:02.936, 3, 1860, 373.47, 76, 89
2026/10/14 04:16:03.936, 0, 1860, 377.10, 75, 100
2026/10/14 04:16:03.936, 1, 1875, 377.76, 74, 90
2026/10/14 04:16:03.936, 2, 1920, 375.72, 68, 83
2026/10/14 04:16:03.936, 3, 1860, 373.35, 76, 90
2026/10/14 04:16:04.936, 0, 1860, 378.47, 75, 100
2026/10/14 04:16:04.936, 1, 1890, 377.75, 74, 90
2026/10/14 04:16:04.936, 2, 1935, 376.68, 68, 81
2026/10/14 04:16:04.936, 3, 1860, 373.07, 76, 89
2026/10/14 04:16:05.936, 0, 1860, 377.08, 75, 100
2026/10/14 04:16:05.937, 1, 1875, 378.33, 74, 91
2026/10/14 04:16:05.937, 2, 1920, 375.45, 69, 82
2026/10/14 04:16:05.937, 3, 1845, 374.54, 76, 91
2026/10/14 04:16:06.937, 0, 1860, 378.31, 76, 100
2026/10/14 04:16:06.937, 1, 1875, 378.33, 75, 91
2026/10/14 04:16:06.937, 2, 1920, 376.14, 69, 82
2026/10/14 04:16:06.937, 3, 1845, 374.67, 77, 91
2026/10/14 04:16:07.937, 0, 1860, 377.72, 76, 100
2026/10/14 04:16:07.937, 1, 1890, 377.68, 75, 91
2026/10/14 04:16:07.937, 2, 1920, 375.91, 69, 82
2026/10/14 04:16:07.937, 3, 1860, 373.75, 77, 89
2026/10/14 04:16:08.937, 0, 1860, 378.38, 76, 100
2026/10/14 04:16:08.937, 1, 1875, 378.10, 75, 90
2026/10/14 04:16:08.937, 2, 1920, 375.88, 69, 82
2026/10/14 04:16:08.937, 3, 1845, 375.58, 77, 91
2026/10/14 04:16:09.937, 0, 1860, 378.09, 76, 100
2026/10/14 04:16:09.938, 1, 1875, 378.28, 75, 90
2026/10/14 04:16:09.938, 2, 1920, 377.86, 69, 83
2026/10/14 04:16:09.938, 3, 1845, 374.94, 77, 90
2026/10/14 04:16:10.938, 0, 1860, 378.08, 76, 100
2026/10/14 04:16:10.938, 1, 1890, 378.04, 75, 90
2026/10/14 04:16:10.938, 2, 1920, 376.03, 69, 82
2026/10/14 04:16:10.938, 3, 1845, 375.30, 77, 91
2026/10/14 04:16:11.938, 0, 1890, 282.06, 70, 4
2026/10/14 04:16:11.938, 1, 1830, 319.42, 69, 33
2026/10/14 04:16:11.938, 2, 1950, 283.62, 64, 31
2026/10/14 04:16:11.938, 3, 1845, 286.37, 74, 37
2026/10/14 04:16:12.938, 0, 1875, 242.50, 72, 21
2026/10/14 04:16:12.938, 1, 1830, 267.83, 73, 75
2026/10/14 04:16:12.938, 2, 1950, 238.72, 63, 18
2026/10/14 04:16:12.938, 3, 1905, 253.96, 74, 94
2026/10/14 04:16:13.939, 0, 1860, 261.63, 73, 84
2026/10/14 04:16:13.939, 1, 1920, 275.90, 73, 100
2026/10/14 04:16:13.939, 2, 1860, 237.40, 62, 3
2026/10/14 04:16:13.939, 3, 1935, 258.64, 72, 59
2026/10/14 04:16:14.939, 0, 1890, 263.00, 71, 94
2026/10/14 04:16:14.939, 1, 1935, 261.94, 69, 22
2026/10/14 04:16:14.939, 2, 1845, 245.17, 66, 67
2026/10/14 04:16:14.939, 3, 1935, 241.42, 69, 17
2026/10/14 04:16:15.939, 0, 1890, 241.25, 68, 13
2026/10/14 04:16:15.939, 1, 1950, 245.39, 68, 10
2026/10/14 04:16:15.939, 2, 1935, 261.87, 66, 100
2026/10/14 04:16:15.939, 3, 1935, 229.26, 68, 0
2026/10/14 04:16:16.939, 0, 1890, 231.89, 68, 4
2026/10/14 04:16:16.939, 1, 1845, 240.73, 70, 6
2026/10/14 04:16:16.939, 2, 1965, 253.04, 63, 17
2026/10/14 04:16:16.939, 3, 1860, 237.20, 72, 59
2026/10/14 04:16:17.960, 0, 1860, 247.15, 72, 34
2026/10/14 04:16:17.960, 1, 1860, 260.73, 72, 99
2026/10/14 04:16:17.960, 2, 1965, 235.60, 62, 12
2026/10/14 04:16:17.960, 3, 1845, 257.27, 74, 94
2026/10/14 04:16:18.960, 0, 1875, 321.00, 75, 100
2026/10/14 04:16:18.960, 1, 1890, 294.37, 73, 90
2026/10/14 04:16:18.960, 2, 1920, 316.95, 68, 80
2026/10/14 04:16:18.960, 3, 1860, 305.81, 75, 89
2026/10/14 04:16:19.960, 0, 1860, 377.96, 76, 100
2026/10/14 04:16:19.961, 1, 1875, 377.48, 74, 90
2026/10/14 04:16:19.961, 2, 1920, 377.98, 69, 82
2026/10/14 04:16:19.961, 3, 1860, 372.98, 76, 91
2026/10/14 04:16:20.961, 0, 1860, 377.43, 76, 100
2026/10/14 04:16:20.961, 1, 1875, 378.11, 75, 91
2026/10/14 04:16:20.961, 2, 1920, 377.91, 69, 79
2026/10/14 04:16:20.961, 3, 1860, 373.96, 77, 89
2026/10/14 04:16:21.961, 0, 1860, 377.67, 76, 100
2026/10/14 04:16:21.961, 1, 1890, 378.00, 75, 90
2026/10/14 04:16:21.961, 2, 1920, 377.18, 69, 83
2026/10/14 04:16:21.982, 3, 1860, 373.49, 77, 90
2026/10/14 04:16:22.982, 0, 1860, 377.81, 76, 100
2026/10/14 04:16:22.982, 1, 1875, 377.77, 75, 91
2026/10/14 04:16:22.982, 2, 1920, 378.17, 70, 83
2026/10/14 04:16:22.982, 3, 1845, 375.32, 77, 91
2026/10/14 04:16:23.982, 0, 1860, 377.86, 77, 100
2026/10/14 04:16:23.982, 1, 1875, 378.09, 75, 91
2026/10/14 04:16:23.982, 2, 1920, 377.77, 70, 83
2026/10/14 04:16:23.982, 3, 1845, 373.47, 77, 91
2026/10/14 04:16:24.982, 0, 1860, 378.11, 77, 100
2026/10/14 04:16:24.982, 1, 1860, 378.05, 76, 90
2026/10/14 04:16:24.982, 2, 1920, 378.37, 70, 83
2026/10/14 04:16:24.982, 3, 1845, 373.20, 78, 91
2026/10/14 04:16:25.982, 0, 1860, 377.54, 77, 100
2026/10/14 04:16:25.982, 1, 1875, 377.75, 76, 91
2026/10/14 04:16:25.982, 2, 1920, 377.83, 70, 80
2026/10/14 04:16:25.982, 3, 1860, 373.59, 78, 89
2026/10/14 04:16:26.983, 0, 1860, 378.20, 77, 100
2026/10/14 04:16:26.983, 1, 1860, 377.85, 76, 91
2026/10/14 04:16:26.983, 2, 1920, 377.62, 70, 82
2026/10/14 04:16:26.983, 3, 1845, 375.05, 78, 90
2026/10/14 04:16:27.983, 0, 1860, 378.39, 77, 100
2026/10/14 04:16:27.983, 1, 1875, 377.91, 76, 90
2026/10/14 04:16:27.983, 2, 1920, 377.39, 70, 80
2026/10/14 04:16:27.983, 3, 1860, 374.60, 78, 88
2026/10/14 04:16:28.983, 0, 1860, 377.93, 77, 100
2026/10/14 04:16:28.983, 1, 1860, 377.83, 76, 91
2026/10/14 04:16:28.983, 2, 1920, 377.79, 70, 83
2026/10/14 04:16:28.983, 3, 1845, 373.95, 78, 91
2026/10/14 04:16:29.983, 0, 1860, 378.55, 78, 100
2026/10/14 04:16:29.983, 1, 1860, 378.00, 76, 90
2026/10/14 04:16:29.983, 2, 1920, 378.06, 70, 81
2026/10/14 04:16:29.983, 3, 1845, 374.62, 78, 88
2026/10/14 04:16:30.984, 0, 1860, 378.40, 78, 100
2026/10/14 04:16:30.984, 1, 1875, 377.60, 76, 90
2026/10/14 04:16:30.984, 2, 1920, 378.15, 71, 81
2026/10/14 04:16:30.984, 3, 1845, 374.99, 78, 88
2026/10/14 04:16:31.984, 0, 1905, 209.64, 69, 7
2026/10/14 04:16:31.984, 1, 1935, 285.64, 69, 0
2026/10/14 04:16:31.984, 2, 1950, 209.40, 63, 0
2026/10/14 04:16:31.984, 3, 1935, 242.17, 69, 0
2026/10/14 04:16:32.984, 0, 1785, 149.40, 68, 4
2026/10/14 04:16:32.984, 1, 1950, 160.37, 68, 0
2026/10/14 04:16:32.984, 2, 1950, 149.61, 62, 0
2026/10/14 04:16:32.984, 3, 1950, 158.14, 68, 0
Loading