diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/PRACTICAL-EVALUATION.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/PRACTICAL-EVALUATION.md new file mode 100644 index 00000000..f1c4cbd8 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/PRACTICAL-EVALUATION.md @@ -0,0 +1,70 @@ +# Practical evaluation, separately labeled (not the speed suite) + +This file is **not** part of the official benchmark suite and is not +comparable to it. It records what this machine's campaign measured beyond +speed, for readers deciding between models on 12 GB RDNA2 hardware. +Methodology and caveats per section; no general accuracy claim is made. + +## A. Reliability: 6-hour day-desktop soak (2026-10-21) + +The ctx32768 Qwen server (the daily driver) served **483 health checks +and traffic over 6 h under normal desktop use** (browser, editor) with +**zero failures**, GPU telemetry live (`backend: hip`). Summary: +`soak-summary.json` next to this file. The summary's `max_server_rss_kB` +tracks the Python wrapper process, not the engine (engine RSS is tens of +GB, mostly reclaimable page cache) — ignore that field. This is a single +soak, not a statistical claim. + +## B. Blind agent screens (12-task easy set, 2026-10-20) + +An agent client (Nanobot, thinking on, temp 0.6, 20-tool-call cap) +completed a 12-task office/research/synthesis suite twice, once per +model, blind: rows and workspaces merged under symlinked names, graded +against fixture-derived ground truths, grade reasons sealed by SHA before +unblinding. + +| | Total | coding | research | synthesis | drafting | +| --- | --- | --- | --- | --- | --- | +| Qwen3.6 Q4_K_XL (poly) | **10/12** | 3/3 | 2/3 | **2/3** | 3/3 | +| LFM2.5-24B Q4_K_XL (poly) | 8/12 | 3/3 | 2/3 | 0/3 | 3/3 | + +Both failed the same two research tasks (a procurement-revision conflict +and a count-preservation check — the failure class: a fact wired to the +wrong slot). LFM's synthesis failures included invented measurement +values. Caveats: 1 repetition per task; same fixtures for both arms; +graders saw neither model identity nor arm. + +## C. Hard-set screen (6 discriminating tasks, 2026-10-21/22) + +Six tasks built to discriminate (generator-derived ground truths: fixed +totals, ID sets, a windowed timeline, a revision conflict, a sandboxed +code task), screened blind the same way. The easy set's models all +saturated it except Gemma: + +| Model | hard set | probe | gate | +| --- | --- | --- | --- | +| Qwen3.6 Q4_K_XL (driver) | **6/6** | 4/4 | 12/12 | +| Tiel Coder Q3_K_XL | **6/6** | 4/4 | 11/12 | +| Gemma-4 QAT (speed arm) | **3/6** | 4/4 | 10/12 | + +Gemma's failures: an arithmetic total (211 vs 216), an unwritten answer +file, the sandbox code task, and a recovery sequence (twice). Caveats: +1 repetition, 6 tasks, one failure was a degenerate generation; four +independent signals agree on direction. + +## D. Same-day llama.cpp controls (methodology check) + +The matched harness (identical prompts across engines) ran same-day +llama.cpp controls on this box: the current llama.cpp default +(IQ3_XXS arm) measured 60.2/66.1 tok/s synth/prose against +strata-poly's Qwen 75.4/73.5 on the same prompts; a fresh control the +next day drifted ~2%, bounding day-to-day noise. These are custom +matched probes, not suite numbers, and are reported only to show the +speed ordering is not a harness artifact. + +## What was not tested + +Three repetitions (screens are 1x); other agent clients; other +quantizations of the same models on this card; the draft/spec state +separately (engine default); `--tail-skip 6`; hipBLASLt tuning for +gfx1031; concurrency beyond the suite's 2-client stage. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/README.md new file mode 100644 index 00000000..f27d2157 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/README.md @@ -0,0 +1,160 @@ +# Community benchmark on AMD Radeon RX 6700 XT 12 GB (gfx1031), Ryzen 5 5600X + +Measured 2026-10-20 to 2026-10-22 by mkw432. The official suite on four +models (Qwen3.6-35B UD-Q4_K_XL, LFM2.5-24B UD-Q4_K_XL, Tiel Coder +UD-Q3_K_XL, Gemma-4-26B-A4B QAT UD-Q4_K_XL) at 32K and 64K contexts, plus +a same-machine re-run of the Qwen suites at current main (e97c8aa) against +the original 0.1.45-c54f0eaf runs, a Gemma draft-pairing A/B, and a +separately labeled practical evaluation appendix. This is the second +report from an RX 6700 XT (after +[2026-10-19-community-rx6700xt-polystrata](../2026-10-19-community-rx6700xt-polystrata/README.md), +a Ryzen 5 3600 machine); the Qwen3.6 and LFM2.5 numbers reproduce that +report's rows within a few percent on different RAM and CPU. + +Main limitation: single box, three days, one repetition of the optional +correctness screens (not three); the quality appendix is not a benchmark +suite and makes no claim beyond this hardware. + +## Hardware and software + +- GPU and VRAM: AMD Radeon RX 6700 XT 12 GB (gfx1031, PCI 1002:73DF, + display-attached); 12272 MiB VRAM reported by amdgpu sysfs; PCIe gen4 x16 +- CPU: AMD Ryzen 5 5600X (6 cores / 12 threads); installed RAM: 32 GB + (31.2 GiB reported); storage: NVMe SSD (models on a second internal + NVMe, ext4) +- OS: Ubuntu, Linux 7.0.0-38-generic; GPU driver: amdgpu (in-kernel); + ROCm: AMD's Python wheels inside the project venv (no system rocm-smi) +- PolyStrata: release engine 0.1.45 built locally for gfx1031 (HIP); the + Oct-20/21 runs used project commit c54f0eaf, the Oct-22 re-runs used + e97c8aa (the engine binary is unchanged; the newer commits touch CUDA + kernels and docs only) +- Background workloads: the Oct-20 LFM and Qwen suites ran while a large + model download wrote to a second disk (labeled; see limitations); other + runs idle-ish desktop; no power limits set + +## Models and configurations + +All GGUFs SHA-256 verified the same day (hashes in +[provenance.json](provenance.json)): + +| Model | File | Repository | +| --- | --- | --- | +| Qwen3.6-35B-A3B UD-Q4_K_XL | `Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | unsloth/Qwen3.6-35B-A3B-MTP-GGUF | +| LFM2.5-24B-A2B UD-Q4_K_XL | `LFM2.5-24B-A2B-UD-Q4_K_XL.gguf` | unsloth/LFM2.5-24B-A2B-GGUF | +| Tiel Coder 35B-A3B MTP UD-Q3_K_XL | `Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf` | peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP | +| Gemma-4-26B-A4B it QAT UD-Q4_K_XL | `gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf` | unsloth/gemma-4-26B-A4B-it-GGUF | + +Common launch settings (the community-parity profile): `--kv-whole` +`--conversation-cache-mib 0` `--vram-reserve-mib 1024`, `backend: hip` in +the run config. LFM adds `--prompt-batch 2048` and its setup-written +expert profile; Tiel adds `--threads 8` and its profile; Gemma adds +`--threads 8` and a draft model (see the A/B below); Qwen adds +`--tail-skip 0`. Contexts 32768 and 65536. + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024 +``` + +## Method + +`python tools/strata_bench.py --runs 3` through the HTTP API of a running +server (the Oct-22 64K re-run started the server itself; its +`engine-log-excerpt.txt` is included). Suite 1 as the guide describes: +thinking off, temperature 0, unique prompt tags, warm-up discarded, 3 runs +per test, medians and ranges. No other engine on the card during any run +(one engine at a time; the daily-driver server was stopped for the re-run +that started its own). + +## Results + +Full tables in each run folder's README; medians: + +| Profile | Stage | Date / commit | Short chat decode | 4K prompt | 16K prompt | 32K prompt | +| --- | --- | --- | ---: | ---: | ---: | ---: | +| Qwen3.6 Q4_K_XL, ctx32768 | official-standard | 10-20 / c54f0eaf | 79.7 | 1053.1 | 999.7 | — | +| Qwen3.6 Q4_K_XL, ctx32768 | official-standard | 10-22 / e97c8aa | 82.2 | 1016.7 | 999.7 | — | +| Qwen3.6 Q4_K_XL, ctx65536 | official-long | 10-20 / c54f0eaf | 81.4 | 1000.9 | 997.5 | 775.0 | +| Qwen3.6 Q4_K_XL, ctx65536 | official-long | 10-22 / e97c8aa | 84.5 | 998.2 | 993.7 | 773.2 | +| LFM2.5 Q4_K_XL, ctx32768 | official-standard | 10-20 / c54f0eaf | 88.5 | 1778.8 | 1414.0 | — | +| LFM2.5 Q4_K_XL, ctx65536 | official-long | 10-20 / c54f0eaf | 86.4 | 1616.4 | 1407.0 | 1043.7 | +| Tiel Coder Q3_K_XL, ctx32768 | official-standard | 10-21 / c54f0eaf | 73.1 | 1241.5 | 1031.6 | — | +| Gemma-4 QAT Q4_K_XL, ctx32768, QAT-paired draft | official-standard | 10-21 / c54f0eaf | 87.7 | 1172.0 | 1009.3 | — | +| Gemma-4 QAT Q4_K_XL, ctx32768, repo Q8_0 draft | official-control | 10-21 / c54f0eaf | 81.4 | 1186.5 | 1024.3 | — | + +All tok/s, engine-timed, medians of 3 runs (chat: median of 9 requests). +Agent-session stages and per-turn tables are in each run README; LFM's +32K agent stage did not complete (its Python-list tool-call format is +rejected by the stage, same gap as the 10-19 report). + +Observations: + +1. **Reproduction of the 10-19 RX 6700 XT report** (different CPU and + RAM, same card): Qwen short-chat 79.7-84.5 vs their Q4_K_M control + rows in the 80s class; LFM 86-89 matches the reported LFM2.5 row + class; all four of our Qwen/LFM 32K and 64K rows land within a few + percent of that report. +2. **Version delta on the same box, same prompts:** c54f0eaf → e97c8aa + changed nothing measurable on HIP (32K prompt 999.7 = 999.7; 32K-row + long 775.0 vs 773.2; chat deltas inside the run-to-run range). +3. **Gemma draft pairing:** swapping the QAT-paired 252 MB draft for the + repo's Q8_0 draft (461 MB, the standard-model pairing) costs + −7.2% short-chat decode (87.7 → 81.4) with prompt throughput + unchanged (control ✓). Same model, same settings, only `--draft-model` + differs. QAT users should keep the QAT-paired draft. +4. **Concurrent clients:** 2 clients total 65.0 tok/s on Qwen at ctx65536 + (below one client; `--batch` was not enabled on these servers). + +## Correctness and limitations + +See [PRACTICAL-EVALUATION.md](PRACTICAL-EVALUATION.md) for a separately +labeled, non-speed evaluation (blind agent screens and a day-desktop +soak) — it is not part of the official suite and no general accuracy +claim is made from it. + +- The suite is a speed check; correctness is not measured by it. +- Oct-20 LFM and Qwen 32K suites ran while a large model download wrote + to a second disk; prompt rates at 4K/16K match the idle re-run + (1053.1/999.7 vs 1016.7/999.7), so the interference is not visible in + the reported medians, but the condition is disclosed. +- Single machine; three days; day-to-day drift on this box measured at + ~1-2% on a same-configuration llama.cpp control. +- GPU naming: this kernel's amdgpu sysfs exposes no `product_name`, so + the server and `tools/strata_bench.py` (which now read sysfs) report + the generic "AMD Radeon", and the report folder names say `unknown` + (renamed to rx6700xt here; original generated names kept inside). + The card is verified externally as an RX 6700 XT: PCI ID 1002:73DF, + 12272 MiB VRAM, gfx1031 engine build. +- No power limits, no clock capture; the card is display-attached. +- Failures: none in the official suites; one skipped test (the LFM 32K + agent stage, above). + +## Notes for RX 6700 XT / RDNA2 / 32 GB users + +Collected during the same campaign; all on this hardware only: + +- `--vram-reserve-mib 1024` matters: with the default reserve the expert + cache grows into the last free MiB and a later request can fault in + ROCm's buffer fill (the failure reported in #52's review thread). All + rows here used an explicit reserve; zero memory faults across nine + suite runs including 64K contexts. +- Qwen3.6 **Q6_K_XL is not usable on a 12/32 GB box**: native Strata + cannot place it (32.6 GB file, placement timed out at >900 s even with + all background load stripped — nominal fit is not placement slack), + and strata-poly loads it (19 s, 2,344/10,496 experts resident) but + faults with `illegal memory access` during a ~4.8K-token prefill. + Short prompts generate fine (~25-37 tok/s). Q4_K_XL remains the pick. +- Qwen3.6 MTP lives **inside** the GGUF (~768 MB of the file); files + from the plain `Qwen3.6-35B-A3B-GGUF` repo lack + `nextn_predict_layers` and the engine refuses them — use the + `Qwen3.6-35B-A3B-MTP-GGUF` repo. +- The web UI's GPU tiles need `"backend": "hip"` in the run config; + setup-written configs lacked it and the tiles fell back to NVML + ("not readable"). +- No hipBLASLt tuning table ships for gfx1031; low-RAM-mode streaming + models (IQ2/IQ1-class Flash-Next files) show prompt throughput 4-10x + below the Q4-class rows here. `tools/hip/tune_hipblaslt` on this card + is untested territory. +- Native Strata pins ~19-29 GB host RAM at load (32 GB boxes: bench it + solo or watch the kernel OOM-killer); strata-poly's experts are + page-cache based and tolerate background load — that difference is + why every tabled number here is a poly number. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/index.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/index.json new file mode 100644 index 00000000..afe59f9d --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/index.json @@ -0,0 +1,73 @@ +{ + "source_commits": { + "original_runs": "c54f0eaf6200a9207f13f900a5fc772d437d6f7a", + "rerun_runs": "e97c8aa47ca2c01025accdd32d07a01536d794ce" + }, + "runs": [ + { + "profile": "gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard" + }, + { + "profile": "gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768", + "stage": "official-control", + "status": "completed", + "reports": 1, + "path": "runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control" + }, + { + "profile": "lfm25-ud-q4_k_xl-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard" + }, + { + "profile": "lfm25-ud-q4_k_xl-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long" + }, + { + "profile": "qwen36-ud-q4_k_xl-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard" + }, + { + "profile": "qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard" + }, + { + "profile": "qwen36-ud-q4_k_xl-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long" + }, + { + "profile": "qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long" + }, + { + "profile": "tiel-ud-q3_k_xl-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard" + } + ], + "tool_sha256": "bd65126bed1f8e5ff3648f5d65c230ea7566219a443272115c1d622735d02be5", + "tool_sha_note": "sha256 of tools/strata_bench.py at e97c8aa (rerun runs); original runs used the same file at c54f0ea (runner-side AMD naming fix present in both)" +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/provenance.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/provenance.json new file mode 100644 index 00000000..826cfc85 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/provenance.json @@ -0,0 +1,82 @@ +{ + "source_commit_note": "original suites (Oct-20/21) at c54f0eaf6200a9207f13f900a5fc772d437d6f7a; Qwen re-runs (Oct-22) at e97c8aa47ca2c01025accdd32d07a01536d794ce; the engine binary (0.1.45, gfx1031, built from c54f0eaf) is identical across both \u2014 the delta commits are CUDA-only", + "build": { + "source": "local-hip", + "backend": "hip", + "version": "0.1.45", + "archs": [ + "gfx1031" + ], + "vision": "none", + "rocm": "AMD Python wheels in the project venv (rocm SDK gfx103X libs)" + }, + "models": [ + { + "file": "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf", + "repo": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF", + "bytes": 22853663008, + "sha256": "55983c5a75a1ab969824077b3bb3de4146e82a9234072b48ad4e8f92ad3fe9f1" + }, + { + "file": "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "repo": "unsloth/LFM2.5-24B-A2B-GGUF", + "sha256": "a235d23638cc5549cb1ff61da7522c8a7e97f056f27093462a1e56f35aba91d8", + "bytes": "not recorded (file deleted in post-campaign cleanup; sha256 verified same day)", + "sha_note": "sha256 recorded same day (2026-10-20) by the campaign harness before the file was deleted in a post-campaign disk cleanup" + }, + { + "file": "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "repo": "peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP", + "bytes": 17234935648, + "sha256": "46b8fbeba6e888c9a101e49cc717baa9bb8dfa0d203ec7b3185288159543ec14" + }, + { + "file": "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf", + "repo": "unsloth/gemma-4-26B-A4B-it-GGUF", + "sha256": "a7c5bc715f5ff8e99a3e8901ce7d2b42b402c669bf24f7c5250747633d0f5891", + "bytes": 14249047104 + }, + { + "file": "mtp-gemma-4-26B-A4B-it.gguf (QAT-paired draft, 252 MB)", + "repo": "unsloth/gemma-4-26B-A4B-it-GGUF", + "sha256": "7272d97595f0d4c74bd7b623492b7dbdaafd8b7c72f329a8270ba4eca68f768a", + "bytes": 251939328 + }, + { + "file": "MTP/mtp-gemma-4-26B-A4B-it-Q8_0.gguf (repo draft, 461 MB)", + "repo": "unsloth/gemma-4-26B-A4B-it-GGUF", + "sha256": "6326fb9f5e487aa8dcdd313a091e3c67724cb2a666ec3b7d2895b5b26d93ed1b", + "bytes": 461766816 + } + ], + "settings_common": [ + "--kv-whole", + "--conversation-cache-mib 0", + "--vram-reserve-mib 1024", + "backend hip" + ], + "settings_extra": { + "qwen36": [ + "--tail-skip 0" + ], + "lfm25": [ + "--prompt-batch 2048", + "setup-written expert-profile.txt" + ], + "tiel": [ + "--threads 8", + "setup-written expert-profile.txt" + ], + "gemma": [ + "--threads 8", + "--draft-model " + ] + }, + "gpu_identity": { + "reported_by_tool": "AMD Radeon (generic; no amdgpu product_name sysfs in this kernel)", + "verified_externally": "AMD Radeon RX 6700 XT, PCI 1002:73DF, 12272 MiB VRAM, gfx1031", + "folder_rename_note": "report folders renamed *-unknown-* -> rx6700xt-5600x; the inner generated names and READMEs are unmodified" + }, + "reproduce": "python tools/strata_bench.py --url http://127.0.0.1:8091 --config --runs 3 [--long --clients 2 for the 64K stages]", + "campaign_reference": "Full campaign record (matched probes, llama.cpp controls, quality screens, incidents) kept in the contributor's local campaign notes; the distilled version is in this folder's README.md and PRACTICAL-EVALUATION.md" +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md new file mode 100644 index 00000000..f9ff29b4 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: gemma-4-26b-a4b-qat-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged (hip backend); thinking off for the suite; temperature 0 + +```text +strata-poly --model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --max-context 32768 --draft-model mtp-gemma-4-26B-A4B-it.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 48 (45-53) | 0 | 100 (100-100) | 9 | not measured | 87.7 (78.0-110.2) | 0.38 (0.38-0.50) | +| 4K prompt, fresh | 4101 (4101-4101) | 0 (0-0) | 24 cap | 3 | 1172.0 (1120.0-1181.5) | not measured | 3.51 (3.50-3.67) | +| 16K prompt, fresh | 16390 (16390-16390) | 0 (0-0) | 24 cap | 3 | 1009.3 (920.7-1011.6) | not measured | 16.42 (16.21-17.81) | +| agent session, follow-up turns | 15434 (15434-15437) at the last turn | see below | up to 64 | 3 | see below | 76.4 (54.1-81.5) | 4.29 (1.99-5.50) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.1 % (68.1-68.1); earlier-turn tokens read again per session: 267 (267-270) (earlier turns that the cache did not hold; the 14806 (14806-14806) new tokens of the files are read once); first turn (nothing cached) first token: 0.80 s (0.80-1.16). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 343.2 | 1.158 | 107.5 | +| 1 | 4162 | 389 | 3773 | 1002.8 | 3.777 | 80.1 | +| 2 | 5872 | 4155 | 1717 | 860.4 | 2.007 | 81.5 | +| 3 | 9365 | 5865 | 3500 | 738.4 | 4.756 | 61.8 | +| 4 | 12379 | 9358 | 3021 | 551.3 | 5.5 | 75.8 | +| 5 | 15434 | 12372 | 3062 | 687.3 | 4.473 | 75.9 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 154 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json new file mode 100644 index 00000000..a41c28d1 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-qatdraft-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-21", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 154.47732305526733, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--draft-model", + "mtp-gemma-4-26B-A4B-it.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "display_name": "Gemma 4 26B-A4B QAT UD-Q4_K_XL (poly)", + "backend": "hip", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --max-context 32768 --draft-model mtp-gemma-4-26B-A4B-it.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 2264, + "expert_cache_mib": 7223, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1009, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.375, + "total_s": 1.331, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 142.0, + "prompt_tps_source": "engine", + "decode_tps": 104.7, + "decode_tps_source": "engine", + "prompt_ms": 373.3, + "decode_ms": 955.5, + "drafts_offered": 84, + "drafts_accepted": 71, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.384, + "total_s": 1.57, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 117.8, + "prompt_tps_source": "engine", + "decode_tps": 84.3, + "decode_tps_source": "engine", + "prompt_ms": 382.0, + "decode_ms": 1186.1, + "drafts_offered": 96, + "drafts_accepted": 58, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.378, + "total_s": 1.605, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 127.7, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 376.0, + "decode_ms": 1227.5, + "drafts_offered": 96, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.382, + "total_s": 1.328, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 139.4, + "prompt_tps_source": "engine", + "decode_tps": 105.7, + "decode_tps_source": "engine", + "prompt_ms": 380.1, + "decode_ms": 946.4, + "drafts_offered": 84, + "drafts_accepted": 71, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.375, + "total_s": 1.515, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 120.7, + "prompt_tps_source": "engine", + "decode_tps": 87.7, + "decode_tps_source": "engine", + "prompt_ms": 372.8, + "decode_ms": 1140.4, + "drafts_offered": 90, + "drafts_accepted": 58, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.381, + "total_s": 1.663, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 126.5, + "prompt_tps_source": "engine", + "decode_tps": 78.0, + "decode_tps_source": "engine", + "prompt_ms": 379.3, + "decode_ms": 1282.1, + "drafts_offered": 100, + "drafts_accepted": 55, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.4, + "total_s": 1.308, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 133.0, + "prompt_tps_source": "engine", + "decode_tps": 110.2, + "decode_tps_source": "engine", + "prompt_ms": 398.5, + "decode_ms": 907.7, + "drafts_offered": 78, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.496, + "total_s": 1.587, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 91.1, + "prompt_tps_source": "engine", + "decode_tps": 91.7, + "decode_tps_source": "engine", + "prompt_ms": 493.8, + "decode_ms": 1091.0, + "drafts_offered": 92, + "drafts_accepted": 63, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.383, + "total_s": 1.635, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 125.9, + "prompt_tps_source": "engine", + "decode_tps": 79.8, + "decode_tps_source": "engine", + "prompt_ms": 381.3, + "decode_ms": 1252.4, + "drafts_offered": 100, + "drafts_accepted": 55, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 87.7, + "min": 78.0, + "max": 110.2, + "n": 9 + }, + "ttft_s": { + "median": 0.382, + "min": 0.375, + "max": 0.496, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 48, + "min": 45, + "max": 53, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.51, + "total_s": 3.874, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1181.5, + "prompt_tps_source": "engine", + "decode_tps": 66.1, + "decode_tps_source": "engine", + "prompt_ms": 3470.9, + "decode_ms": 363.3, + "drafts_offered": 24, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.502, + "total_s": 3.971, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1172.0, + "prompt_tps_source": "engine", + "decode_tps": 51.1, + "decode_tps_source": "engine", + "prompt_ms": 3499.1, + "decode_ms": 469.3, + "drafts_offered": 27, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.665, + "total_s": 3.992, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1120.0, + "prompt_tps_source": "engine", + "decode_tps": 73.2, + "decode_tps_source": "engine", + "prompt_ms": 3661.6, + "decode_ms": 327.9, + "drafts_offered": 20, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 1172.0, + "min": 1120.0, + "max": 1181.5, + "n": 3 + }, + "ttft_s": { + "median": 3.51, + "min": 3.502, + "max": 3.665, + "n": 3 + }, + "prompt_tokens": { + "median": 4101, + "min": 4101, + "max": 4101, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.418, + "total_s": 16.774, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 1009.3, + "prompt_tps_source": "engine", + "decode_tps": 67.4, + "decode_tps_source": "engine", + "prompt_ms": 16238.5, + "decode_ms": 356.3, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.208, + "total_s": 16.504, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 1011.6, + "prompt_tps_source": "engine", + "decode_tps": 80.9, + "decode_tps_source": "engine", + "prompt_ms": 16201.3, + "decode_ms": 296.6, + "drafts_offered": 17, + "drafts_accepted": 14, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 17.808, + "total_s": 18.148, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 920.7, + "prompt_tps_source": "engine", + "decode_tps": 70.5, + "decode_tps_source": "engine", + "prompt_ms": 17801.3, + "decode_ms": 340.4, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1009.3, + "min": 920.7, + "max": 1011.6, + "n": 3 + }, + "ttft_s": { + "median": 16.418, + "min": 16.208, + "max": 17.808, + "n": 3 + }, + "prompt_tokens": { + "median": 16390, + "min": 16390, + "max": 16390, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.158, + "total_s": 1.362, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 343.2, + "prompt_tps_source": "engine", + "decode_tps": 107.5, + "decode_tps_source": "engine", + "prompt_ms": 1153.9, + "decode_ms": 204.7, + "drafts_offered": 14, + "drafts_accepted": 13, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.777, + "total_s": 4.401, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 50, + "prompt_tps": 1002.8, + "prompt_tps_source": "engine", + "decode_tps": 80.1, + "decode_tps_source": "engine", + "prompt_ms": 3762.5, + "decode_ms": 624.1, + "drafts_offered": 39, + "drafts_accepted": 27, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.007, + "total_s": 2.681, + "prompt_tokens": 5872, + "reused_tokens": 4155, + "read_tokens": 1717, + "generated_tokens": 55, + "prompt_tps": 860.4, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 1995.5, + "decode_ms": 674.7, + "drafts_offered": 44, + "drafts_accepted": 32, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.756, + "total_s": 5.566, + "prompt_tokens": 9365, + "reused_tokens": 5865, + "read_tokens": 3500, + "generated_tokens": 50, + "prompt_tps": 738.4, + "prompt_tps_source": "engine", + "decode_tps": 61.8, + "decode_tps_source": "engine", + "prompt_ms": 4740.2, + "decode_ms": 809.5, + "drafts_offered": 41, + "drafts_accepted": 26, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.5, + "total_s": 6.225, + "prompt_tokens": 12379, + "reused_tokens": 9358, + "read_tokens": 3021, + "generated_tokens": 55, + "prompt_tps": 551.3, + "prompt_tps_source": "engine", + "decode_tps": 75.8, + "decode_tps_source": "engine", + "prompt_ms": 5480.0, + "decode_ms": 725.2, + "drafts_offered": 32, + "drafts_accepted": 23, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.473, + "total_s": 5.316, + "prompt_tokens": 15434, + "reused_tokens": 12372, + "read_tokens": 3062, + "generated_tokens": 64, + "prompt_tps": 687.3, + "prompt_tps_source": "engine", + "decode_tps": 75.9, + "decode_tps_source": "engine", + "prompt_ms": 4455.4, + "decode_ms": 843.2, + "drafts_offered": 37, + "drafts_accepted": 29, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.801, + "total_s": 1.036, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 495.7, + "prompt_tps_source": "engine", + "decode_tps": 93.6, + "decode_tps_source": "engine", + "prompt_ms": 798.8, + "decode_ms": 235.1, + "drafts_offered": 13, + "drafts_accepted": 12, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.039, + "total_s": 4.687, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 50, + "prompt_tps": 934.9, + "prompt_tps_source": "engine", + "decode_tps": 77.1, + "decode_tps_source": "engine", + "prompt_ms": 4035.6, + "decode_ms": 648.7, + "drafts_offered": 31, + "drafts_accepted": 21, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.201, + "total_s": 2.911, + "prompt_tokens": 5872, + "reused_tokens": 4155, + "read_tokens": 1717, + "generated_tokens": 55, + "prompt_tps": 781.6, + "prompt_tps_source": "engine", + "decode_tps": 77.4, + "decode_tps_source": "engine", + "prompt_ms": 2196.8, + "decode_ms": 710.2, + "drafts_offered": 32, + "drafts_accepted": 24, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.584, + "total_s": 5.508, + "prompt_tokens": 9365, + "reused_tokens": 5865, + "read_tokens": 3500, + "generated_tokens": 50, + "prompt_tps": 764.3, + "prompt_tps_source": "engine", + "decode_tps": 54.1, + "decode_tps_source": "engine", + "prompt_ms": 4579.4, + "decode_ms": 923.8, + "drafts_offered": 22, + "drafts_accepted": 12, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.273, + "total_s": 4.992, + "prompt_tokens": 12379, + "reused_tokens": 9358, + "read_tokens": 3021, + "generated_tokens": 55, + "prompt_tps": 708.2, + "prompt_tps_source": "engine", + "decode_tps": 76.4, + "decode_tps_source": "engine", + "prompt_ms": 4265.5, + "decode_ms": 719.5, + "drafts_offered": 46, + "drafts_accepted": 32, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.491, + "total_s": 5.403, + "prompt_tokens": 15434, + "reused_tokens": 12372, + "read_tokens": 3062, + "generated_tokens": 64, + "prompt_tps": 682.8, + "prompt_tps_source": "engine", + "decode_tps": 70.2, + "decode_tps_source": "engine", + "prompt_ms": 4484.6, + "decode_ms": 912.3, + "drafts_offered": 45, + "drafts_accepted": 37, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.804, + "total_s": 1.036, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 493.6, + "prompt_tps_source": "engine", + "decode_tps": 95.0, + "decode_tps_source": "engine", + "prompt_ms": 802.2, + "decode_ms": 231.5, + "drafts_offered": 17, + "drafts_accepted": 15, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.077, + "total_s": 4.776, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 55, + "prompt_tps": 926.1, + "prompt_tps_source": "engine", + "decode_tps": 78.7, + "decode_tps_source": "engine", + "prompt_ms": 4073.9, + "decode_ms": 698.8, + "drafts_offered": 44, + "drafts_accepted": 31, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.992, + "total_s": 2.703, + "prompt_tokens": 5877, + "reused_tokens": 4155, + "read_tokens": 1722, + "generated_tokens": 55, + "prompt_tps": 866.1, + "prompt_tps_source": "engine", + "decode_tps": 77.3, + "decode_tps_source": "engine", + "prompt_ms": 1988.2, + "decode_ms": 711.2, + "drafts_offered": 41, + "drafts_accepted": 33, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.763, + "total_s": 5.413, + "prompt_tokens": 9370, + "reused_tokens": 5870, + "read_tokens": 3500, + "generated_tokens": 48, + "prompt_tps": 735.7, + "prompt_tps_source": "engine", + "decode_tps": 73.7, + "decode_tps_source": "engine", + "prompt_ms": 4757.4, + "decode_ms": 650.9, + "drafts_offered": 40, + "drafts_accepted": 26, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.292, + "total_s": 4.987, + "prompt_tokens": 12382, + "reused_tokens": 9363, + "read_tokens": 3019, + "generated_tokens": 55, + "prompt_tps": 704.5, + "prompt_tps_source": "engine", + "decode_tps": 79.1, + "decode_tps_source": "engine", + "prompt_ms": 4285.2, + "decode_ms": 695.3, + "drafts_offered": 41, + "drafts_accepted": 31, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.496, + "total_s": 5.272, + "prompt_tokens": 15437, + "reused_tokens": 12375, + "read_tokens": 3062, + "generated_tokens": 56, + "prompt_tps": 682.2, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 4488.1, + "decode_ms": 777.5, + "drafts_offered": 46, + "drafts_accepted": 31, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.074, + "min": 68.074, + "max": 68.078, + "n": 3 + }, + "reread_tokens": { + "median": 267, + "min": 267, + "max": 270, + "n": 3 + }, + "new_tokens": { + "median": 14806, + "min": 14806, + "max": 14806, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.292, + "min": 1.992, + "max": 5.5, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15434, + "min": 15434, + "max": 15437, + "n": 3 + }, + "turn_decode_tps": { + "median": 76.4, + "min": 54.1, + "max": 81.5, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 0.804, + "min": 0.801, + "max": 1.158, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md new file mode 100644 index 00000000..6b2b8f1f --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: gemma-4-26b-a4b-qat-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged (hip backend); thinking off for the suite; temperature 0 + +```text +strata-poly --model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --max-context 32768 --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 48 (45-53) | 0 | 100 (100-100) | 9 | not measured | 81.4 (72.5-100.6) | 0.37 (0.37-0.38) | +| 4K prompt, fresh | 4101 (4101-4101) | 0 (0-0) | 24 cap | 3 | 1186.5 (1186.4-1196.0) | not measured | 3.46 (3.46-3.46) | +| 16K prompt, fresh | 16390 (16390-16390) | 0 (0-0) | 24 cap | 3 | 1024.3 (1020.6-1025.3) | not measured | 16.01 (15.99-16.23) | +| agent session, follow-up turns | 15440 (15435-15442) at the last turn | see below | up to 64 | 3 | see below | 71.9 (56.7-84.7) | 3.98 (1.90-4.42) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.1 % (68.1-68.1); earlier-turn tokens read again per session: 273 (268-275) (earlier turns that the cache did not hold; the 14806 (14806-14806) new tokens of the files are read once); first turn (nothing cached) first token: 0.76 s (0.75-0.77). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 524.0 | 0.76 | 73.6 | +| 1 | 4162 | 389 | 3773 | 1049.5 | 3.608 | 76.7 | +| 2 | 5879 | 4155 | 1724 | 913.5 | 1.898 | 71.9 | +| 3 | 9373 | 5872 | 3501 | 795.2 | 4.419 | 56.7 | +| 4 | 12386 | 9366 | 3020 | 764.2 | 3.969 | 67.6 | +| 5 | 15442 | 12379 | 3063 | 726.5 | 4.233 | 68.7 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 148 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json new file mode 100644 index 00000000..5e590b38 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/gemma4-qat-ud-q4_k_xl-repodraft-reserve1024-ctx32768/official-control/2026-10-21-community-unknown-gemma-4-26b-a4b-qat-ud-q4-k-xl/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-21", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 147.9081826210022, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--draft-model", + "mtp-gemma-4-26B-A4B-it-Q8_0.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "display_name": "Gemma 4 26B-A4B QAT UD-Q4_K_XL (poly)", + "backend": "hip", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --max-context 32768 --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "gemma-4-26b-a4b-qat-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 2241, + "expert_cache_mib": 7150, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1009, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.367, + "total_s": 1.442, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 145.0, + "prompt_tps_source": "engine", + "decode_tps": 93.0, + "decode_tps_source": "engine", + "prompt_ms": 365.6, + "decode_ms": 1075.1, + "drafts_offered": 75, + "drafts_accepted": 60, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.379, + "total_s": 1.608, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 119.2, + "prompt_tps_source": "engine", + "decode_tps": 81.4, + "decode_tps_source": "engine", + "prompt_ms": 377.4, + "decode_ms": 1229.2, + "drafts_offered": 83, + "drafts_accepted": 51, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.37, + "total_s": 1.749, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 130.4, + "prompt_tps_source": "engine", + "decode_tps": 72.5, + "decode_tps_source": "engine", + "prompt_ms": 368.1, + "decode_ms": 1379.2, + "drafts_offered": 96, + "drafts_accepted": 45, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.377, + "total_s": 1.371, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 141.3, + "prompt_tps_source": "engine", + "decode_tps": 100.6, + "decode_tps_source": "engine", + "prompt_ms": 375.1, + "decode_ms": 994.2, + "drafts_offered": 81, + "drafts_accepted": 67, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.368, + "total_s": 1.511, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 122.9, + "prompt_tps_source": "engine", + "decode_tps": 87.5, + "decode_tps_source": "engine", + "prompt_ms": 366.1, + "decode_ms": 1143.1, + "drafts_offered": 78, + "drafts_accepted": 54, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.381, + "total_s": 1.618, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 126.7, + "prompt_tps_source": "engine", + "decode_tps": 80.8, + "decode_tps_source": "engine", + "prompt_ms": 378.8, + "decode_ms": 1237.5, + "drafts_offered": 89, + "drafts_accepted": 51, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.381, + "total_s": 1.437, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 139.7, + "prompt_tps_source": "engine", + "decode_tps": 94.7, + "decode_tps_source": "engine", + "prompt_ms": 379.4, + "decode_ms": 1055.8, + "drafts_offered": 85, + "drafts_accepted": 64, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.368, + "total_s": 1.666, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 123.0, + "prompt_tps_source": "engine", + "decode_tps": 77.0, + "decode_tps_source": "engine", + "prompt_ms": 365.8, + "decode_ms": 1298.9, + "drafts_offered": 92, + "drafts_accepted": 50, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.374, + "total_s": 1.672, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 128.9, + "prompt_tps_source": "engine", + "decode_tps": 77.0, + "decode_tps_source": "engine", + "prompt_ms": 372.5, + "decode_ms": 1298.1, + "drafts_offered": 89, + "drafts_accepted": 46, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 81.4, + "min": 72.5, + "max": 100.6, + "n": 9 + }, + "ttft_s": { + "median": 0.374, + "min": 0.367, + "max": 0.381, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 48, + "min": 45, + "max": 53, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.465, + "total_s": 3.878, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1196.0, + "prompt_tps_source": "engine", + "decode_tps": 58.1, + "decode_tps_source": "engine", + "prompt_ms": 3429.0, + "decode_ms": 413.2, + "drafts_offered": 22, + "drafts_accepted": 7, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.459, + "total_s": 3.86, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1186.4, + "prompt_tps_source": "engine", + "decode_tps": 59.8, + "decode_tps_source": "engine", + "prompt_ms": 3456.8, + "decode_ms": 401.1, + "drafts_offered": 15, + "drafts_accepted": 4, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.459, + "total_s": 3.804, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 1186.5, + "prompt_tps_source": "engine", + "decode_tps": 69.5, + "decode_tps_source": "engine", + "prompt_ms": 3456.4, + "decode_ms": 345.5, + "drafts_offered": 5, + "drafts_accepted": 3, + "run": 2 + } + ], + "prompt_tps": { + "median": 1186.5, + "min": 1186.4, + "max": 1196.0, + "n": 3 + }, + "ttft_s": { + "median": 3.459, + "min": 3.459, + "max": 3.465, + "n": 3 + }, + "prompt_tokens": { + "median": 4101, + "min": 4101, + "max": 4101, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.234, + "total_s": 16.668, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 1020.6, + "prompt_tps_source": "engine", + "decode_tps": 55.2, + "decode_tps_source": "engine", + "prompt_ms": 16058.6, + "decode_ms": 434.6, + "drafts_offered": 20, + "drafts_accepted": 8, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.991, + "total_s": 16.372, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 1025.3, + "prompt_tps_source": "engine", + "decode_tps": 62.9, + "decode_tps_source": "engine", + "prompt_ms": 15985.1, + "decode_ms": 381.5, + "drafts_offered": 4, + "drafts_accepted": 2, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.008, + "total_s": 16.402, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 1024.3, + "prompt_tps_source": "engine", + "decode_tps": 60.8, + "decode_tps_source": "engine", + "prompt_ms": 16001.1, + "decode_ms": 394.9, + "drafts_offered": 8, + "drafts_accepted": 4, + "run": 2 + } + ], + "prompt_tps": { + "median": 1024.3, + "min": 1020.6, + "max": 1025.3, + "n": 3 + }, + "ttft_s": { + "median": 16.008, + "min": 15.991, + "max": 16.234, + "n": 3 + }, + "prompt_tokens": { + "median": 16390, + "min": 16390, + "max": 16390, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 0.76, + "total_s": 1.058, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 524.0, + "prompt_tps_source": "engine", + "decode_tps": 73.6, + "decode_tps_source": "engine", + "prompt_ms": 755.7, + "decode_ms": 298.9, + "drafts_offered": 4, + "drafts_accepted": 2, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.608, + "total_s": 4.351, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 57, + "prompt_tps": 1049.5, + "prompt_tps_source": "engine", + "decode_tps": 76.7, + "decode_tps_source": "engine", + "prompt_ms": 3595.2, + "decode_ms": 743.4, + "drafts_offered": 49, + "drafts_accepted": 31, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.898, + "total_s": 2.677, + "prompt_tokens": 5879, + "reused_tokens": 4155, + "read_tokens": 1724, + "generated_tokens": 56, + "prompt_tps": 913.5, + "prompt_tps_source": "engine", + "decode_tps": 71.9, + "decode_tps_source": "engine", + "prompt_ms": 1887.2, + "decode_ms": 779.0, + "drafts_offered": 55, + "drafts_accepted": 31, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.419, + "total_s": 5.283, + "prompt_tokens": 9373, + "reused_tokens": 5872, + "read_tokens": 3501, + "generated_tokens": 49, + "prompt_tps": 795.2, + "prompt_tps_source": "engine", + "decode_tps": 56.7, + "decode_tps_source": "engine", + "prompt_ms": 4402.7, + "decode_ms": 864.5, + "drafts_offered": 39, + "drafts_accepted": 12, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.969, + "total_s": 4.797, + "prompt_tokens": 12386, + "reused_tokens": 9366, + "read_tokens": 3020, + "generated_tokens": 56, + "prompt_tps": 764.2, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 3951.9, + "decode_ms": 828.4, + "drafts_offered": 29, + "drafts_accepted": 18, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.233, + "total_s": 5.164, + "prompt_tokens": 15442, + "reused_tokens": 12379, + "read_tokens": 3063, + "generated_tokens": 64, + "prompt_tps": 726.5, + "prompt_tps_source": "engine", + "decode_tps": 68.7, + "decode_tps_source": "engine", + "prompt_ms": 4216.2, + "decode_ms": 931.4, + "drafts_offered": 55, + "drafts_accepted": 33, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.77, + "total_s": 1.003, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 515.6, + "prompt_tps_source": "engine", + "decode_tps": 94.3, + "decode_tps_source": "engine", + "prompt_ms": 768.1, + "decode_ms": 233.4, + "drafts_offered": 17, + "drafts_accepted": 13, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.629, + "total_s": 4.231, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 51, + "prompt_tps": 1040.5, + "prompt_tps_source": "engine", + "decode_tps": 84.7, + "decode_tps_source": "engine", + "prompt_ms": 3626.0, + "decode_ms": 602.1, + "drafts_offered": 41, + "drafts_accepted": 29, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.902, + "total_s": 2.68, + "prompt_tokens": 5873, + "reused_tokens": 4155, + "read_tokens": 1718, + "generated_tokens": 55, + "prompt_tps": 905.5, + "prompt_tps_source": "engine", + "decode_tps": 70.7, + "decode_tps_source": "engine", + "prompt_ms": 1897.2, + "decode_ms": 778.4, + "drafts_offered": 48, + "drafts_accepted": 27, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.4, + "total_s": 5.242, + "prompt_tokens": 9366, + "reused_tokens": 5866, + "read_tokens": 3500, + "generated_tokens": 50, + "prompt_tps": 796.4, + "prompt_tps_source": "engine", + "decode_tps": 59.4, + "decode_tps_source": "engine", + "prompt_ms": 4395.0, + "decode_ms": 842.1, + "drafts_offered": 57, + "drafts_accepted": 24, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.978, + "total_s": 4.712, + "prompt_tokens": 12380, + "reused_tokens": 9359, + "read_tokens": 3021, + "generated_tokens": 55, + "prompt_tps": 760.7, + "prompt_tps_source": "engine", + "decode_tps": 74.9, + "decode_tps_source": "engine", + "prompt_ms": 3971.6, + "decode_ms": 734.2, + "drafts_offered": 45, + "drafts_accepted": 31, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.233, + "total_s": 5.098, + "prompt_tokens": 15435, + "reused_tokens": 12373, + "read_tokens": 3062, + "generated_tokens": 64, + "prompt_tps": 724.5, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 4226.2, + "decode_ms": 865.2, + "drafts_offered": 54, + "drafts_accepted": 37, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.751, + "total_s": 1.005, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 528.9, + "prompt_tps_source": "engine", + "decode_tps": 86.5, + "decode_tps_source": "engine", + "prompt_ms": 748.7, + "decode_ms": 254.3, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.652, + "total_s": 4.326, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 52, + "prompt_tps": 1034.0, + "prompt_tps_source": "engine", + "decode_tps": 77.2, + "decode_tps_source": "engine", + "prompt_ms": 3648.9, + "decode_ms": 674.0, + "drafts_offered": 45, + "drafts_accepted": 29, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.906, + "total_s": 2.629, + "prompt_tokens": 5874, + "reused_tokens": 4155, + "read_tokens": 1719, + "generated_tokens": 55, + "prompt_tps": 903.8, + "prompt_tps_source": "engine", + "decode_tps": 76.1, + "decode_tps_source": "engine", + "prompt_ms": 1901.9, + "decode_ms": 722.8, + "drafts_offered": 45, + "drafts_accepted": 29, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.415, + "total_s": 5.276, + "prompt_tokens": 9367, + "reused_tokens": 5867, + "read_tokens": 3500, + "generated_tokens": 53, + "prompt_tps": 793.7, + "prompt_tps_source": "engine", + "decode_tps": 61.6, + "decode_tps_source": "engine", + "prompt_ms": 4410.0, + "decode_ms": 860.8, + "drafts_offered": 53, + "drafts_accepted": 24, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.989, + "total_s": 4.815, + "prompt_tokens": 12384, + "reused_tokens": 9360, + "read_tokens": 3024, + "generated_tokens": 56, + "prompt_tps": 759.2, + "prompt_tps_source": "engine", + "decode_tps": 67.8, + "decode_tps_source": "engine", + "prompt_ms": 3983.3, + "decode_ms": 826.4, + "drafts_offered": 52, + "drafts_accepted": 29, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.237, + "total_s": 5.109, + "prompt_tokens": 15440, + "reused_tokens": 12377, + "read_tokens": 3063, + "generated_tokens": 64, + "prompt_tps": 724.2, + "prompt_tps_source": "engine", + "decode_tps": 73.3, + "decode_tps_source": "engine", + "prompt_ms": 4229.7, + "decode_ms": 872.9, + "drafts_offered": 53, + "drafts_accepted": 35, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.074, + "min": 68.071, + "max": 68.077, + "n": 3 + }, + "reread_tokens": { + "median": 273, + "min": 268, + "max": 275, + "n": 3 + }, + "new_tokens": { + "median": 14806, + "min": 14806, + "max": 14806, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.978, + "min": 1.898, + "max": 4.419, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15440, + "min": 15435, + "max": 15442, + "n": 3 + }, + "turn_decode_tps": { + "median": 71.9, + "min": 56.7, + "max": 84.7, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 0.76, + "min": 0.751, + "max": 0.77, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..2e0b8b80 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-20 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: lfm2.5-24b-a2b-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --vram-reserve-mib 1024 --prompt-batch 2048 --kv-whole +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 39 (36-45) | 0 | 100 (100-100) | 9 | not measured | 88.5 (76.1-95.2) | 0.19 (0.17-0.22) | +| 4K prompt, fresh | 4126 (4126-4126) | 0 (0-0) | 24 cap | 3 | 1778.8 (1769.1-1782.8) | not measured | 2.33 (2.33-2.35) | +| 16K prompt, fresh | 16490 (16490-16490) | 0 (0-0) | 24 cap | 3 | 1414.0 (1406.5-1414.9) | not measured | 11.69 (11.68-11.75) | +| agent session, follow-up turns | not measured at the last turn | see below | up to 64 | 0 | see below | not measured | not measured | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: not measured; earlier-turn tokens read again per session: 0 (0-0) (earlier turns that the cache did not hold; the 0 (0-0) new tokens of the files are read once); first turn (nothing cached) first token: not measured. + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | + +### Skipped and failed + +- failed agent: no token came back +- failed agent: no token came back +- failed agent: no token came back + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 61 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..300b3f4d --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json @@ -0,0 +1,570 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-20", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 60.79493188858032, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024", + "--prompt-batch", + "2048", + "--kv-whole" + ], + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "open_browser": false, + "host": "127.0.0.1" + }, + "launch": "strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --vram-reserve-mib 1024 --prompt-batch 2048 --kv-whole", + "engine_info": { + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 1468, + "expert_cache_mib": 8073, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1006, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.222, + "total_s": 1.525, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 214.9, + "prompt_tps_source": "engine", + "decode_tps": 76.1, + "decode_tps_source": "engine", + "prompt_ms": 209.4, + "decode_ms": 1313.7, + "drafts_offered": 7, + "drafts_accepted": 3, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.206, + "total_s": 1.353, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 186.9, + "prompt_tps_source": "engine", + "decode_tps": 86.3, + "decode_tps_source": "engine", + "prompt_ms": 192.6, + "decode_ms": 1159.0, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.186, + "total_s": 1.374, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 225.4, + "prompt_tps_source": "engine", + "decode_tps": 83.4, + "decode_tps_source": "engine", + "prompt_ms": 173.0, + "decode_ms": 1199.1, + "drafts_offered": 6, + "drafts_accepted": 0, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.174, + "total_s": 1.294, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 277.4, + "prompt_tps_source": "engine", + "decode_tps": 88.5, + "decode_tps_source": "engine", + "prompt_ms": 162.2, + "decode_ms": 1130.4, + "drafts_offered": 6, + "drafts_accepted": 5, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.188, + "total_s": 1.228, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 203.9, + "prompt_tps_source": "engine", + "decode_tps": 95.2, + "decode_tps_source": "engine", + "prompt_ms": 176.6, + "decode_ms": 1050.0, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.177, + "total_s": 1.252, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 236.7, + "prompt_tps_source": "engine", + "decode_tps": 92.1, + "decode_tps_source": "engine", + "prompt_ms": 164.8, + "decode_ms": 1085.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.171, + "total_s": 1.31, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 282.1, + "prompt_tps_source": "engine", + "decode_tps": 87.1, + "decode_tps_source": "engine", + "prompt_ms": 159.5, + "decode_ms": 1148.7, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.189, + "total_s": 1.241, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 204.0, + "prompt_tps_source": "engine", + "decode_tps": 94.0, + "decode_tps_source": "engine", + "prompt_ms": 176.5, + "decode_ms": 1063.3, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.166, + "total_s": 1.254, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 252.8, + "prompt_tps_source": "engine", + "decode_tps": 91.0, + "decode_tps_source": "engine", + "prompt_ms": 154.3, + "decode_ms": 1098.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 88.5, + "min": 76.1, + "max": 95.2, + "n": 9 + }, + "ttft_s": { + "median": 0.186, + "min": 0.166, + "max": 0.222, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 39, + "min": 36, + "max": 45, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 2.347, + "total_s": 2.617, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1769.1, + "prompt_tps_source": "engine", + "decode_tps": 85.3, + "decode_tps_source": "engine", + "prompt_ms": 2332.3, + "decode_ms": 281.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.335, + "total_s": 2.619, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1778.8, + "prompt_tps_source": "engine", + "decode_tps": 80.9, + "decode_tps_source": "engine", + "prompt_ms": 2319.6, + "decode_ms": 296.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.33, + "total_s": 2.595, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1782.8, + "prompt_tps_source": "engine", + "decode_tps": 86.7, + "decode_tps_source": "engine", + "prompt_ms": 2314.4, + "decode_ms": 276.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1778.8, + "min": 1769.1, + "max": 1782.8, + "n": 3 + }, + "ttft_s": { + "median": 2.335, + "min": 2.33, + "max": 2.347, + "n": 3 + }, + "prompt_tokens": { + "median": 4126, + "min": 4126, + "max": 4126, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.752, + "total_s": 12.092, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1406.5, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 11724.4, + "decode_ms": 355.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.679, + "total_s": 12.005, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1414.9, + "prompt_tps_source": "engine", + "decode_tps": 70.5, + "decode_tps_source": "engine", + "prompt_ms": 11654.2, + "decode_ms": 340.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.688, + "total_s": 12.009, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1414.0, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 11661.9, + "decode_ms": 336.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1414.0, + "min": 1406.5, + "max": 1414.9, + "n": 3 + }, + "ttft_s": { + "median": 11.688, + "min": 11.679, + "max": 11.752, + "n": 3 + }, + "prompt_tokens": { + "median": 16490, + "min": 16490, + "max": 16490, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.776, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 812.5, + "prompt_tps_source": "engine", + "decode_tps": 63.0, + "decode_tps_source": "engine", + "prompt_ms": 487.4, + "decode_ms": 285.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + } + ], + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.725, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 858.6, + "prompt_tps_source": "engine", + "decode_tps": 68.6, + "decode_tps_source": "engine", + "prompt_ms": 461.2, + "decode_ms": 262.2, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 0 + } + ], + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.709, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 846.3, + "prompt_tps_source": "engine", + "decode_tps": 75.2, + "decode_tps_source": "engine", + "prompt_ms": 467.9, + "decode_ms": 239.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + } + ] + ], + "hit_pct": null, + "reread_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + }, + "new_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + }, + "turn_ttft_s": null, + "last_turn_prompt_tokens": null, + "turn_decode_tps": null, + "first_turn_ttft_s": null + }, + "clients": {}, + "skipped": {}, + "failures": [ + { + "where": "agent", + "error": "no token came back" + }, + { + "where": "agent", + "error": "no token came back" + }, + { + "where": "agent", + "error": "no token came back" + } + ] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..c5ecb780 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/README.md @@ -0,0 +1,69 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-20 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: lfm2.5-24b-a2b-ud-q4_k_xl +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --vram-reserve-mib 1024 --prompt-batch 2048 --kv-whole +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 39 (36-45) | 0 | 100 (100-100) | 9 | not measured | 86.4 (79.7-92.1) | 0.19 (0.18-0.21) | +| 4K prompt, fresh | 4126 (4126-4126) | 0 (0-0) | 24 cap | 3 | 1616.4 (1597.7-1618.2) | not measured | 2.57 (2.56-2.60) | +| 16K prompt, fresh | 16490 (16490-16490) | 0 (0-0) | 24 cap | 3 | 1407.0 (1391.5-1413.7) | not measured | 11.75 (11.69-11.88) | +| 32K prompt, fresh | 32875 (32875-32875) | 0 (0-0) | 24 cap | 3 | 1043.7 (1031.1-1044.0) | not measured | 31.54 (31.53-31.93) | +| agent session, follow-up turns | 14438 (14429-14448) at the last turn | see below | up to 64 | 2 | see below | 76.7 (68.5-83.9) | 2.71 (1.66-3.04) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 67.8 % (67.8-67.8); earlier-turn tokens read again per session: 310 (0-312) (earlier turns that the cache did not hold; the 13748 (0-13765) new tokens of the files are read once); first turn (nothing cached) first token: 0.54 s (0.54-0.54). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 758.3 | 0.538 | 67.5 | +| 1 | 3730 | 391 | 3339 | 1650.7 | 2.161 | 69.5 | +| 2 | 5402 | 3725 | 1677 | 1016.5 | 1.699 | 75.7 | +| 3 | 8568 | 5397 | 3171 | 1179.6 | 2.75 | 77.6 | +| 4 | 11595 | 8563 | 3032 | 1123.8 | 2.756 | 78.9 | +| 5 | 14448 | 11590 | 2858 | 949.7 | 3.039 | 68.5 | + +### Concurrency: 2 clients + +Total 77.7 tok/s (69.9-80.0); first token p50 1.074 s, p95 2.033 s; request latency p50 2.493 s, p95 3.57 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +### Skipped and failed + +- failed agent: no token came back + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 206 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..5337ba62 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-lfm2-5-24b-a2b-ud-q4-k-xl/results.json @@ -0,0 +1,936 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-20", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 206.12482452392578, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024", + "--prompt-batch", + "2048", + "--kv-whole" + ], + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "open_browser": false, + "host": "127.0.0.1" + }, + "launch": "strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --vram-reserve-mib 1024 --prompt-batch 2048 --kv-whole", + "engine_info": { + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 1357, + "expert_cache_mib": 7431, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1001, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.176, + "total_s": 1.327, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 276.8, + "prompt_tps_source": "engine", + "decode_tps": 86.0, + "decode_tps_source": "engine", + "prompt_ms": 162.6, + "decode_ms": 1162.6, + "drafts_offered": 6, + "drafts_accepted": 3, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.215, + "total_s": 1.405, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 179.6, + "prompt_tps_source": "engine", + "decode_tps": 83.2, + "decode_tps_source": "engine", + "prompt_ms": 200.5, + "decode_ms": 1202.1, + "drafts_offered": 1, + "drafts_accepted": 1, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.193, + "total_s": 1.436, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 217.6, + "prompt_tps_source": "engine", + "decode_tps": 79.7, + "decode_tps_source": "engine", + "prompt_ms": 179.2, + "decode_ms": 1254.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.176, + "total_s": 1.329, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 273.6, + "prompt_tps_source": "engine", + "decode_tps": 86.0, + "decode_tps_source": "engine", + "prompt_ms": 164.5, + "decode_ms": 1163.4, + "drafts_offered": 2, + "drafts_accepted": 2, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.195, + "total_s": 1.269, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 197.5, + "prompt_tps_source": "engine", + "decode_tps": 92.1, + "decode_tps_source": "engine", + "prompt_ms": 182.3, + "decode_ms": 1085.6, + "drafts_offered": 1, + "drafts_accepted": 1, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.192, + "total_s": 1.337, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 217.8, + "prompt_tps_source": "engine", + "decode_tps": 86.4, + "decode_tps_source": "engine", + "prompt_ms": 179.1, + "decode_ms": 1157.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.176, + "total_s": 1.305, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 275.2, + "prompt_tps_source": "engine", + "decode_tps": 87.7, + "decode_tps_source": "engine", + "prompt_ms": 163.5, + "decode_ms": 1140.2, + "drafts_offered": 11, + "drafts_accepted": 7, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.198, + "total_s": 1.291, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 194.1, + "prompt_tps_source": "engine", + "decode_tps": 90.6, + "decode_tps_source": "engine", + "prompt_ms": 185.5, + "decode_ms": 1103.7, + "drafts_offered": 4, + "drafts_accepted": 1, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.182, + "total_s": 1.316, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 229.8, + "prompt_tps_source": "engine", + "decode_tps": 87.4, + "decode_tps_source": "engine", + "prompt_ms": 169.7, + "decode_ms": 1144.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 86.4, + "min": 79.7, + "max": 92.1, + "n": 9 + }, + "ttft_s": { + "median": 0.192, + "min": 0.176, + "max": 0.215, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 39, + "min": 36, + "max": 45, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 2.601, + "total_s": 2.888, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1597.7, + "prompt_tps_source": "engine", + "decode_tps": 80.3, + "decode_tps_source": "engine", + "prompt_ms": 2582.5, + "decode_ms": 298.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.568, + "total_s": 2.84, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1616.4, + "prompt_tps_source": "engine", + "decode_tps": 84.7, + "decode_tps_source": "engine", + "prompt_ms": 2552.6, + "decode_ms": 283.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.565, + "total_s": 2.84, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1618.2, + "prompt_tps_source": "engine", + "decode_tps": 83.8, + "decode_tps_source": "engine", + "prompt_ms": 2549.8, + "decode_ms": 286.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1616.4, + "min": 1597.7, + "max": 1618.2, + "n": 3 + }, + "ttft_s": { + "median": 2.568, + "min": 2.565, + "max": 2.601, + "n": 3 + }, + "prompt_tokens": { + "median": 4126, + "min": 4126, + "max": 4126, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.752, + "total_s": 12.101, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1407.0, + "prompt_tps_source": "engine", + "decode_tps": 65.8, + "decode_tps_source": "engine", + "prompt_ms": 11720.0, + "decode_ms": 364.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.691, + "total_s": 12.025, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1413.7, + "prompt_tps_source": "engine", + "decode_tps": 68.6, + "decode_tps_source": "engine", + "prompt_ms": 11664.3, + "decode_ms": 349.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.877, + "total_s": 12.213, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1391.5, + "prompt_tps_source": "engine", + "decode_tps": 68.2, + "decode_tps_source": "engine", + "prompt_ms": 11850.2, + "decode_ms": 351.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1407.0, + "min": 1391.5, + "max": 1413.7, + "n": 3 + }, + "ttft_s": { + "median": 11.752, + "min": 11.691, + "max": 11.877, + "n": 3 + }, + "prompt_tokens": { + "median": 16490, + "min": 16490, + "max": 16490, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 31.932, + "total_s": 32.376, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1031.1, + "prompt_tps_source": "engine", + "decode_tps": 51.5, + "decode_tps_source": "engine", + "prompt_ms": 31884.7, + "decode_ms": 465.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 31.531, + "total_s": 31.954, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1044.0, + "prompt_tps_source": "engine", + "decode_tps": 54.1, + "decode_tps_source": "engine", + "prompt_ms": 31490.8, + "decode_ms": 443.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 31.54, + "total_s": 31.953, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1043.7, + "prompt_tps_source": "engine", + "decode_tps": 55.3, + "decode_tps_source": "engine", + "prompt_ms": 31499.7, + "decode_ms": 433.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1043.7, + "min": 1031.1, + "max": 1044.0, + "n": 3 + }, + "ttft_s": { + "median": 31.54, + "min": 31.531, + "max": 31.932, + "n": 3 + }, + "prompt_tokens": { + "median": 32875, + "min": 32875, + "max": 32875, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 0.538, + "total_s": 0.998, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 31, + "prompt_tps": 758.3, + "prompt_tps_source": "engine", + "decode_tps": 67.5, + "decode_tps_source": "engine", + "prompt_ms": 522.2, + "decode_ms": 459.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.161, + "total_s": 3.07, + "prompt_tokens": 3730, + "reused_tokens": 391, + "read_tokens": 3339, + "generated_tokens": 64, + "prompt_tps": 1650.7, + "prompt_tps_source": "engine", + "decode_tps": 69.5, + "decode_tps_source": "engine", + "prompt_ms": 2022.8, + "decode_ms": 921.5, + "drafts_offered": 24, + "drafts_accepted": 16, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.699, + "total_s": 2.544, + "prompt_tokens": 5402, + "reused_tokens": 3725, + "read_tokens": 1677, + "generated_tokens": 64, + "prompt_tps": 1016.5, + "prompt_tps_source": "engine", + "decode_tps": 75.7, + "decode_tps_source": "engine", + "prompt_ms": 1649.8, + "decode_ms": 845.1, + "drafts_offered": 4, + "drafts_accepted": 1, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.75, + "total_s": 3.574, + "prompt_tokens": 8568, + "reused_tokens": 5397, + "read_tokens": 3171, + "generated_tokens": 64, + "prompt_tps": 1179.6, + "prompt_tps_source": "engine", + "decode_tps": 77.6, + "decode_tps_source": "engine", + "prompt_ms": 2688.3, + "decode_ms": 824.7, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.756, + "total_s": 3.566, + "prompt_tokens": 11595, + "reused_tokens": 8563, + "read_tokens": 3032, + "generated_tokens": 64, + "prompt_tps": 1123.8, + "prompt_tps_source": "engine", + "decode_tps": 78.9, + "decode_tps_source": "engine", + "prompt_ms": 2698.0, + "decode_ms": 810.9, + "drafts_offered": 15, + "drafts_accepted": 11, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.039, + "total_s": 3.973, + "prompt_tokens": 14448, + "reused_tokens": 11590, + "read_tokens": 2858, + "generated_tokens": 64, + "prompt_tps": 949.7, + "prompt_tps_source": "engine", + "decode_tps": 68.5, + "decode_tps_source": "engine", + "prompt_ms": 3009.5, + "decode_ms": 934.1, + "drafts_offered": 17, + "drafts_accepted": 7, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.845, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 718.3, + "prompt_tps_source": "engine", + "decode_tps": 61.7, + "decode_tps_source": "engine", + "prompt_ms": 551.3, + "decode_ms": 291.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 0 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.543, + "total_s": 0.922, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 30, + "prompt_tps": 732.0, + "prompt_tps_source": "engine", + "decode_tps": 79.1, + "decode_tps_source": "engine", + "prompt_ms": 541.0, + "decode_ms": 379.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.051, + "total_s": 2.832, + "prompt_tokens": 3730, + "reused_tokens": 391, + "read_tokens": 3339, + "generated_tokens": 63, + "prompt_tps": 1639.9, + "prompt_tps_source": "engine", + "decode_tps": 79.5, + "decode_tps_source": "engine", + "prompt_ms": 2036.1, + "decode_ms": 792.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.662, + "total_s": 2.424, + "prompt_tokens": 5383, + "reused_tokens": 3725, + "read_tokens": 1658, + "generated_tokens": 64, + "prompt_tps": 1000.4, + "prompt_tps_source": "engine", + "decode_tps": 83.9, + "decode_tps_source": "engine", + "prompt_ms": 1657.3, + "decode_ms": 763.0, + "drafts_offered": 4, + "drafts_accepted": 1, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.695, + "total_s": 3.476, + "prompt_tokens": 8549, + "reused_tokens": 5378, + "read_tokens": 3171, + "generated_tokens": 64, + "prompt_tps": 1178.8, + "prompt_tps_source": "engine", + "decode_tps": 81.9, + "decode_tps_source": "engine", + "prompt_ms": 2690.1, + "decode_ms": 781.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.734, + "total_s": 3.6, + "prompt_tokens": 11576, + "reused_tokens": 8544, + "read_tokens": 3032, + "generated_tokens": 64, + "prompt_tps": 1115.4, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 2718.2, + "decode_ms": 866.7, + "drafts_offered": 8, + "drafts_accepted": 3, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.045, + "total_s": 3.944, + "prompt_tokens": 14429, + "reused_tokens": 11571, + "read_tokens": 2858, + "generated_tokens": 64, + "prompt_tps": 944.9, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 3024.7, + "decode_ms": 900.0, + "drafts_offered": 16, + "drafts_accepted": 7, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 67.813, + "min": 67.806, + "max": 67.819, + "n": 2 + }, + "reread_tokens": { + "median": 310, + "min": 0, + "max": 312, + "n": 3 + }, + "new_tokens": { + "median": 13748, + "min": 0, + "max": 13765, + "n": 3 + }, + "turn_ttft_s": { + "median": 2.715, + "min": 1.662, + "max": 3.045, + "n": 10 + }, + "last_turn_prompt_tokens": { + "median": 14438.5, + "min": 14429, + "max": 14448, + "n": 2 + }, + "turn_decode_tps": { + "median": 76.65, + "min": 68.5, + "max": 83.9, + "n": 10 + }, + "first_turn_ttft_s": { + "median": 0.54, + "min": 0.538, + "max": 0.543, + "n": 2 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 3.66, + "tokens": 256, + "total_tps": 69.89, + "first_s": [ + 0.27207550799357705, + 2.065375217003748 + ], + "latency_s": [ + 1.7844965999975102, + 3.6624543130019447 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 3.29, + "tokens": 256, + "total_tps": 77.74, + "first_s": [ + 0.2722602790017845, + 1.9372688540024683 + ], + "latency_s": [ + 1.6726300529990112, + 3.292794099994353 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 3.2, + "tokens": 256, + "total_tps": 79.95, + "first_s": [ + 0.27188608900178224, + 1.874987537987181 + ], + "latency_s": [ + 1.617615843002568, + 3.2014343529881444 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 77.74, + "min": 69.89, + "max": 79.95, + "n": 3 + }, + "ttft_p50_s": 1.074, + "ttft_p95_s": 2.033, + "latency_p50_s": 2.493, + "latency_p95_s": 3.57, + "errors": 0 + }, + "skipped": {}, + "failures": [ + { + "where": "agent", + "error": "no token came back" + } + ] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..41339f51 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-22 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit e97c8aa4; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged (hip backend); thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 82.2 (58.6-102.9) | 0.33 (0.30-0.46) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1016.7 (1005.8-1017.6) | not measured | 3.96 (3.95-4.00) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 999.7 (998.4-1000.2) | not measured | 16.07 (16.07-16.10) | +| agent session, follow-up turns | 15135 (15125-15141) at the last turn | see below | up to 64 | 3 | see below | 76.8 (68.3-84.2) | 4.31 (2.38-5.31) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.7-68.8); earlier-turn tokens read again per session: 247 (237-253) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.12 s (1.12-1.23). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 528.8 | 1.231 | 89.8 | +| 1 | 4196 | 602 | 3594 | 846.7 | 4.374 | 83.5 | +| 2 | 5830 | 4189 | 1641 | 682.2 | 2.474 | 78.2 | +| 3 | 9255 | 5823 | 3432 | 682.3 | 5.089 | 81.5 | +| 4 | 12207 | 9248 | 2959 | 695.0 | 4.314 | 78.5 | +| 5 | 15141 | 12200 | 2941 | 556.2 | 5.31 | 72.3 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 160 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..6b3c5e7c --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768-rerun-e97c8aa/official-standard/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json @@ -0,0 +1,881 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-22", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 160.37374711036682, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "e97c8aa4", + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "backend": "hip", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_xl", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_XL (strata-poly)", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 3682, + "expert_cache_mib": 6733, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.333, + "total_s": 2.041, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 154.3, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 330.6, + "decode_ms": 1707.7, + "drafts_offered": 75, + "drafts_accepted": 61, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.464, + "total_s": 2.068, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 95.7, + "prompt_tps_source": "engine", + "decode_tps": 62.3, + "decode_tps_source": "engine", + "prompt_ms": 449.1, + "decode_ms": 1604.0, + "drafts_offered": 67, + "drafts_accepted": 52, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.451, + "total_s": 1.963, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 104.5, + "prompt_tps_source": "engine", + "decode_tps": 66.2, + "decode_tps_source": "engine", + "prompt_ms": 440.3, + "decode_ms": 1511.6, + "drafts_offered": 69, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.394, + "total_s": 1.611, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 129.9, + "prompt_tps_source": "engine", + "decode_tps": 82.2, + "decode_tps_source": "engine", + "prompt_ms": 392.6, + "decode_ms": 1217.2, + "drafts_offered": 71, + "drafts_accepted": 66, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.329, + "total_s": 1.546, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 131.2, + "prompt_tps_source": "engine", + "decode_tps": 82.2, + "decode_tps_source": "engine", + "prompt_ms": 327.7, + "decode_ms": 1217.0, + "drafts_offered": 68, + "drafts_accepted": 55, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.33, + "total_s": 1.567, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 140.0, + "prompt_tps_source": "engine", + "decode_tps": 80.8, + "decode_tps_source": "engine", + "prompt_ms": 328.5, + "decode_ms": 1237.1, + "drafts_offered": 78, + "drafts_accepted": 57, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.33, + "total_s": 1.302, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 155.1, + "prompt_tps_source": "engine", + "decode_tps": 102.9, + "decode_tps_source": "engine", + "prompt_ms": 328.8, + "decode_ms": 971.9, + "drafts_offered": 75, + "drafts_accepted": 71, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.301, + "total_s": 1.467, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 143.5, + "prompt_tps_source": "engine", + "decode_tps": 85.8, + "decode_tps_source": "engine", + "prompt_ms": 299.6, + "decode_ms": 1166.0, + "drafts_offered": 69, + "drafts_accepted": 56, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.31, + "total_s": 1.486, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 149.0, + "prompt_tps_source": "engine", + "decode_tps": 85.0, + "decode_tps_source": "engine", + "prompt_ms": 308.7, + "decode_ms": 1175.8, + "drafts_offered": 77, + "drafts_accepted": 58, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 82.2, + "min": 58.6, + "max": 102.9, + "n": 9 + }, + "ttft_s": { + "median": 0.33, + "min": 0.301, + "max": 0.464, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.003, + "total_s": 4.321, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1005.8, + "prompt_tps_source": "engine", + "decode_tps": 75.4, + "decode_tps_source": "engine", + "prompt_ms": 3995.8, + "decode_ms": 318.1, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.956, + "total_s": 4.26, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1016.7, + "prompt_tps_source": "engine", + "decode_tps": 79.1, + "decode_tps_source": "engine", + "prompt_ms": 3953.0, + "decode_ms": 303.5, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.953, + "total_s": 4.259, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1017.6, + "prompt_tps_source": "engine", + "decode_tps": 78.5, + "decode_tps_source": "engine", + "prompt_ms": 3949.3, + "decode_ms": 305.7, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1016.7, + "min": 1005.8, + "max": 1017.6, + "n": 3 + }, + "ttft_s": { + "median": 3.956, + "min": 3.953, + "max": 4.003, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.103, + "total_s": 16.434, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 998.4, + "prompt_tps_source": "engine", + "decode_tps": 72.4, + "decode_tps_source": "engine", + "prompt_ms": 16083.9, + "decode_ms": 331.5, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.065, + "total_s": 16.401, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1000.2, + "prompt_tps_source": "engine", + "decode_tps": 71.3, + "decode_tps_source": "engine", + "prompt_ms": 16054.1, + "decode_ms": 336.5, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.073, + "total_s": 16.41, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 999.7, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 16062.1, + "decode_ms": 337.4, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 999.7, + "min": 998.4, + "max": 1000.2, + "n": 3 + }, + "ttft_s": { + "median": 16.073, + "min": 16.065, + "max": 16.103, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.231, + "total_s": 1.599, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 33, + "prompt_tps": 528.8, + "prompt_tps_source": "engine", + "decode_tps": 89.8, + "decode_tps_source": "engine", + "prompt_ms": 1151.6, + "decode_ms": 367.3, + "drafts_offered": 26, + "drafts_accepted": 20, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.374, + "total_s": 4.877, + "prompt_tokens": 4196, + "reused_tokens": 602, + "read_tokens": 3594, + "generated_tokens": 42, + "prompt_tps": 846.7, + "prompt_tps_source": "engine", + "decode_tps": 83.5, + "decode_tps_source": "engine", + "prompt_ms": 4244.7, + "decode_ms": 503.1, + "drafts_offered": 35, + "drafts_accepted": 26, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.474, + "total_s": 3.215, + "prompt_tokens": 5830, + "reused_tokens": 4189, + "read_tokens": 1641, + "generated_tokens": 58, + "prompt_tps": 682.2, + "prompt_tps_source": "engine", + "decode_tps": 78.2, + "decode_tps_source": "engine", + "prompt_ms": 2405.3, + "decode_ms": 742.0, + "drafts_offered": 46, + "drafts_accepted": 33, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.089, + "total_s": 5.518, + "prompt_tokens": 9255, + "reused_tokens": 5823, + "read_tokens": 3432, + "generated_tokens": 35, + "prompt_tps": 682.3, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 5029.8, + "decode_ms": 429.6, + "drafts_offered": 21, + "drafts_accepted": 19, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.314, + "total_s": 4.951, + "prompt_tokens": 12207, + "reused_tokens": 9248, + "read_tokens": 2959, + "generated_tokens": 50, + "prompt_tps": 695.0, + "prompt_tps_source": "engine", + "decode_tps": 78.5, + "decode_tps_source": "engine", + "prompt_ms": 4257.8, + "decode_ms": 637.3, + "drafts_offered": 43, + "drafts_accepted": 31, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.31, + "total_s": 5.905, + "prompt_tokens": 15141, + "reused_tokens": 12200, + "read_tokens": 2941, + "generated_tokens": 43, + "prompt_tps": 556.2, + "prompt_tps_source": "engine", + "decode_tps": 72.3, + "decode_tps_source": "engine", + "prompt_ms": 5287.7, + "decode_ms": 594.9, + "drafts_offered": 29, + "drafts_accepted": 23, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.124, + "total_s": 1.404, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 26, + "prompt_tps": 542.6, + "prompt_tps_source": "engine", + "decode_tps": 92.9, + "decode_tps_source": "engine", + "prompt_ms": 1122.3, + "decode_ms": 279.8, + "drafts_offered": 23, + "drafts_accepted": 17, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.197, + "total_s": 4.85, + "prompt_tokens": 4189, + "reused_tokens": 602, + "read_tokens": 3587, + "generated_tokens": 55, + "prompt_tps": 855.4, + "prompt_tps_source": "engine", + "decode_tps": 84.2, + "decode_tps_source": "engine", + "prompt_ms": 4193.3, + "decode_ms": 653.4, + "drafts_offered": 46, + "drafts_accepted": 34, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.397, + "total_s": 3.052, + "prompt_tokens": 5836, + "reused_tokens": 4182, + "read_tokens": 1654, + "generated_tokens": 50, + "prompt_tps": 691.1, + "prompt_tps_source": "engine", + "decode_tps": 76.3, + "decode_tps_source": "engine", + "prompt_ms": 2393.3, + "decode_ms": 655.5, + "drafts_offered": 36, + "drafts_accepted": 26, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.054, + "total_s": 5.562, + "prompt_tokens": 9253, + "reused_tokens": 5829, + "read_tokens": 3424, + "generated_tokens": 39, + "prompt_tps": 678.2, + "prompt_tps_source": "engine", + "decode_tps": 76.8, + "decode_tps_source": "engine", + "prompt_ms": 5048.4, + "decode_ms": 507.7, + "drafts_offered": 28, + "drafts_accepted": 21, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.249, + "total_s": 4.805, + "prompt_tokens": 12209, + "reused_tokens": 9246, + "read_tokens": 2963, + "generated_tokens": 42, + "prompt_tps": 698.3, + "prompt_tps_source": "engine", + "decode_tps": 75.4, + "decode_tps_source": "engine", + "prompt_ms": 4243.3, + "decode_ms": 556.8, + "drafts_offered": 33, + "drafts_accepted": 24, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.293, + "total_s": 5.945, + "prompt_tokens": 15135, + "reused_tokens": 12202, + "read_tokens": 2933, + "generated_tokens": 49, + "prompt_tps": 554.9, + "prompt_tps_source": "engine", + "decode_tps": 75.1, + "decode_tps_source": "engine", + "prompt_ms": 5285.9, + "decode_ms": 652.1, + "drafts_offered": 32, + "drafts_accepted": 27, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.119, + "total_s": 1.235, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 545.3, + "prompt_tps_source": "engine", + "decode_tps": 94.3, + "decode_tps_source": "engine", + "prompt_ms": 1116.9, + "decode_ms": 116.6, + "drafts_offered": 8, + "drafts_accepted": 6, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.193, + "total_s": 4.732, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 44, + "prompt_tps": 852.8, + "prompt_tps_source": "engine", + "decode_tps": 81.6, + "decode_tps_source": "engine", + "prompt_ms": 4188.8, + "decode_ms": 539.3, + "drafts_offered": 31, + "drafts_accepted": 24, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.376, + "total_s": 3.062, + "prompt_tokens": 5810, + "reused_tokens": 4167, + "read_tokens": 1643, + "generated_tokens": 53, + "prompt_tps": 692.8, + "prompt_tps_source": "engine", + "decode_tps": 77.1, + "decode_tps_source": "engine", + "prompt_ms": 2371.5, + "decode_ms": 687.0, + "drafts_offered": 35, + "drafts_accepted": 27, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.933, + "total_s": 5.553, + "prompt_tokens": 9230, + "reused_tokens": 5803, + "read_tokens": 3427, + "generated_tokens": 45, + "prompt_tps": 695.4, + "prompt_tps_source": "engine", + "decode_tps": 72.6, + "decode_tps_source": "engine", + "prompt_ms": 4927.8, + "decode_ms": 620.0, + "drafts_offered": 29, + "drafts_accepted": 22, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.248, + "total_s": 4.899, + "prompt_tokens": 12192, + "reused_tokens": 9223, + "read_tokens": 2969, + "generated_tokens": 49, + "prompt_tps": 700.0, + "prompt_tps_source": "engine", + "decode_tps": 75.2, + "decode_tps_source": "engine", + "prompt_ms": 4241.3, + "decode_ms": 651.9, + "drafts_offered": 36, + "drafts_accepted": 27, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.303, + "total_s": 6.02, + "prompt_tokens": 15125, + "reused_tokens": 12185, + "read_tokens": 2940, + "generated_tokens": 49, + "prompt_tps": 555.4, + "prompt_tps_source": "engine", + "decode_tps": 68.3, + "decode_tps_source": "engine", + "prompt_ms": 5293.9, + "decode_ms": 717.9, + "drafts_offered": 35, + "drafts_accepted": 25, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.76, + "min": 68.728, + "max": 68.768, + "n": 3 + }, + "reread_tokens": { + "median": 247, + "min": 237, + "max": 253, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.314, + "min": 2.376, + "max": 5.31, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15135, + "min": 15125, + "max": 15141, + "n": 3 + }, + "turn_decode_tps": { + "median": 76.8, + "min": 68.3, + "max": 84.2, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.124, + "min": 1.119, + "max": 1.231, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..9edc2f8f --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-20 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 79.7 (71.1-87.8) | 0.31 (0.30-0.32) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1053.1 (1051.1-1053.1) | not measured | 3.82 (3.82-3.83) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 999.7 (998.0-999.9) | not measured | 16.07 (16.07-16.10) | +| agent session, follow-up turns | 15125 (15122-15152) at the last turn | see below | up to 64 | 3 | see below | 76.5 (65.5-82.6) | 4.25 (2.38-5.30) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.7-68.8); earlier-turn tokens read again per session: 237 (234-264) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.11 s (1.11-1.14). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 537.4 | 1.136 | 93.9 | +| 1 | 4185 | 602 | 3583 | 857.6 | 4.184 | 79.6 | +| 2 | 5819 | 4178 | 1641 | 691.2 | 2.379 | 82.6 | +| 3 | 9244 | 5812 | 3432 | 687.5 | 4.999 | 74.9 | +| 4 | 12196 | 9237 | 2959 | 699.1 | 4.24 | 74.7 | +| 5 | 15125 | 12189 | 2936 | 557.7 | 5.271 | 74.0 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 157 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..f2573a9e --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json @@ -0,0 +1,880 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-20", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 156.7476029396057, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_xl", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_XL (strata-poly)", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 3672, + "expert_cache_mib": 6714, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.303, + "total_s": 1.442, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 168.9, + "prompt_tps_source": "engine", + "decode_tps": 87.8, + "decode_tps_source": "engine", + "prompt_ms": 301.9, + "decode_ms": 1138.6, + "drafts_offered": 72, + "drafts_accepted": 67, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.315, + "total_s": 1.722, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 137.1, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 313.6, + "decode_ms": 1406.5, + "drafts_offered": 70, + "drafts_accepted": 52, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.317, + "total_s": 1.704, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 145.9, + "prompt_tps_source": "engine", + "decode_tps": 72.1, + "decode_tps_source": "engine", + "prompt_ms": 315.2, + "decode_ms": 1387.6, + "drafts_offered": 60, + "drafts_accepted": 44, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.313, + "total_s": 1.53, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 163.8, + "prompt_tps_source": "engine", + "decode_tps": 82.2, + "decode_tps_source": "engine", + "prompt_ms": 311.3, + "decode_ms": 1217.1, + "drafts_offered": 62, + "drafts_accepted": 56, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.304, + "total_s": 1.533, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 142.3, + "prompt_tps_source": "engine", + "decode_tps": 81.3, + "decode_tps_source": "engine", + "prompt_ms": 302.2, + "decode_ms": 1229.4, + "drafts_offered": 54, + "drafts_accepted": 46, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.303, + "total_s": 1.69, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 152.4, + "prompt_tps_source": "engine", + "decode_tps": 72.1, + "decode_tps_source": "engine", + "prompt_ms": 301.8, + "decode_ms": 1387.0, + "drafts_offered": 60, + "drafts_accepted": 40, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.322, + "total_s": 1.576, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 159.0, + "prompt_tps_source": "engine", + "decode_tps": 79.7, + "decode_tps_source": "engine", + "prompt_ms": 320.7, + "decode_ms": 1254.2, + "drafts_offered": 54, + "drafts_accepted": 50, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.309, + "total_s": 1.54, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 139.9, + "prompt_tps_source": "engine", + "decode_tps": 81.2, + "decode_tps_source": "engine", + "prompt_ms": 307.4, + "decode_ms": 1231.3, + "drafts_offered": 71, + "drafts_accepted": 55, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.302, + "total_s": 1.562, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 153.0, + "prompt_tps_source": "engine", + "decode_tps": 79.4, + "decode_tps_source": "engine", + "prompt_ms": 300.6, + "decode_ms": 1259.4, + "drafts_offered": 70, + "drafts_accepted": 53, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 79.7, + "min": 71.1, + "max": 87.8, + "n": 9 + }, + "ttft_s": { + "median": 0.309, + "min": 0.302, + "max": 0.322, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.828, + "total_s": 4.15, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1051.1, + "prompt_tps_source": "engine", + "decode_tps": 74.6, + "decode_tps_source": "engine", + "prompt_ms": 3823.7, + "decode_ms": 321.9, + "drafts_offered": 17, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.82, + "total_s": 4.161, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1053.1, + "prompt_tps_source": "engine", + "decode_tps": 70.4, + "decode_tps_source": "engine", + "prompt_ms": 3816.2, + "decode_ms": 341.0, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.821, + "total_s": 4.149, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1053.1, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 3816.4, + "decode_ms": 328.2, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1053.1, + "min": 1051.1, + "max": 1053.1, + "n": 3 + }, + "ttft_s": { + "median": 3.821, + "min": 3.82, + "max": 3.828, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.101, + "total_s": 16.521, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 998.0, + "prompt_tps_source": "engine", + "decode_tps": 57.0, + "decode_tps_source": "engine", + "prompt_ms": 16090.3, + "decode_ms": 420.9, + "drafts_offered": 19, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.074, + "total_s": 16.437, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 999.7, + "prompt_tps_source": "engine", + "decode_tps": 66.0, + "decode_tps_source": "engine", + "prompt_ms": 16063.4, + "decode_ms": 363.4, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.071, + "total_s": 16.461, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 999.9, + "prompt_tps_source": "engine", + "decode_tps": 61.4, + "decode_tps_source": "engine", + "prompt_ms": 16060.2, + "decode_ms": 390.8, + "drafts_offered": 16, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 999.7, + "min": 998.0, + "max": 999.9, + "n": 3 + }, + "ttft_s": { + "median": 16.074, + "min": 16.071, + "max": 16.101, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.136, + "total_s": 1.37, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 22, + "prompt_tps": 537.4, + "prompt_tps_source": "engine", + "decode_tps": 93.9, + "decode_tps_source": "engine", + "prompt_ms": 1133.3, + "decode_ms": 234.2, + "drafts_offered": 13, + "drafts_accepted": 12, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.184, + "total_s": 4.711, + "prompt_tokens": 4185, + "reused_tokens": 602, + "read_tokens": 3583, + "generated_tokens": 42, + "prompt_tps": 857.6, + "prompt_tps_source": "engine", + "decode_tps": 79.6, + "decode_tps_source": "engine", + "prompt_ms": 4178.1, + "decode_ms": 527.8, + "drafts_offered": 32, + "drafts_accepted": 24, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.379, + "total_s": 3.081, + "prompt_tokens": 5819, + "reused_tokens": 4178, + "read_tokens": 1641, + "generated_tokens": 58, + "prompt_tps": 691.2, + "prompt_tps_source": "engine", + "decode_tps": 82.6, + "decode_tps_source": "engine", + "prompt_ms": 2374.3, + "decode_ms": 702.2, + "drafts_offered": 42, + "drafts_accepted": 34, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.999, + "total_s": 5.466, + "prompt_tokens": 9244, + "reused_tokens": 5812, + "read_tokens": 3432, + "generated_tokens": 35, + "prompt_tps": 687.5, + "prompt_tps_source": "engine", + "decode_tps": 74.9, + "decode_tps_source": "engine", + "prompt_ms": 4992.2, + "decode_ms": 467.2, + "drafts_offered": 22, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.24, + "total_s": 4.843, + "prompt_tokens": 12196, + "reused_tokens": 9237, + "read_tokens": 2959, + "generated_tokens": 45, + "prompt_tps": 699.1, + "prompt_tps_source": "engine", + "decode_tps": 74.7, + "decode_tps_source": "engine", + "prompt_ms": 4232.7, + "decode_ms": 602.7, + "drafts_offered": 37, + "drafts_accepted": 26, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.271, + "total_s": 5.907, + "prompt_tokens": 15125, + "reused_tokens": 12189, + "read_tokens": 2936, + "generated_tokens": 47, + "prompt_tps": 557.7, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 5264.3, + "decode_ms": 635.5, + "drafts_offered": 35, + "drafts_accepted": 27, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.111, + "total_s": 1.509, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 33, + "prompt_tps": 549.1, + "prompt_tps_source": "engine", + "decode_tps": 82.9, + "decode_tps_source": "engine", + "prompt_ms": 1109.0, + "decode_ms": 398.0, + "drafts_offered": 23, + "drafts_accepted": 17, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.189, + "total_s": 4.833, + "prompt_tokens": 4196, + "reused_tokens": 602, + "read_tokens": 3594, + "generated_tokens": 53, + "prompt_tps": 858.8, + "prompt_tps_source": "engine", + "decode_tps": 82.2, + "decode_tps_source": "engine", + "prompt_ms": 4184.9, + "decode_ms": 644.8, + "drafts_offered": 39, + "drafts_accepted": 30, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.375, + "total_s": 3.104, + "prompt_tokens": 5841, + "reused_tokens": 4189, + "read_tokens": 1652, + "generated_tokens": 53, + "prompt_tps": 696.8, + "prompt_tps_source": "engine", + "decode_tps": 72.6, + "decode_tps_source": "engine", + "prompt_ms": 2371.0, + "decode_ms": 729.7, + "drafts_offered": 44, + "drafts_accepted": 28, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.018, + "total_s": 5.597, + "prompt_tokens": 9261, + "reused_tokens": 5834, + "read_tokens": 3427, + "generated_tokens": 46, + "prompt_tps": 683.6, + "prompt_tps_source": "engine", + "decode_tps": 79.4, + "decode_tps_source": "engine", + "prompt_ms": 5013.0, + "decode_ms": 579.4, + "drafts_offered": 31, + "drafts_accepted": 26, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.257, + "total_s": 4.852, + "prompt_tokens": 12224, + "reused_tokens": 9254, + "read_tokens": 2970, + "generated_tokens": 44, + "prompt_tps": 698.8, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 4250.3, + "decode_ms": 596.0, + "drafts_offered": 33, + "drafts_accepted": 24, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.301, + "total_s": 5.915, + "prompt_tokens": 15152, + "reused_tokens": 12217, + "read_tokens": 2935, + "generated_tokens": 47, + "prompt_tps": 554.3, + "prompt_tps_source": "engine", + "decode_tps": 76.5, + "decode_tps_source": "engine", + "prompt_ms": 5294.8, + "decode_ms": 614.2, + "drafts_offered": 36, + "drafts_accepted": 28, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.115, + "total_s": 1.231, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 547.2, + "prompt_tps_source": "engine", + "decode_tps": 94.2, + "decode_tps_source": "engine", + "prompt_ms": 1112.9, + "decode_ms": 116.8, + "drafts_offered": 8, + "drafts_accepted": 6, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.179, + "total_s": 4.747, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 44, + "prompt_tps": 855.5, + "prompt_tps_source": "engine", + "decode_tps": 77.5, + "decode_tps_source": "engine", + "prompt_ms": 4175.3, + "decode_ms": 567.6, + "drafts_offered": 31, + "drafts_accepted": 23, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.378, + "total_s": 3.025, + "prompt_tokens": 5810, + "reused_tokens": 4167, + "read_tokens": 1643, + "generated_tokens": 53, + "prompt_tps": 692.1, + "prompt_tps_source": "engine", + "decode_tps": 81.9, + "decode_tps_source": "engine", + "prompt_ms": 2374.0, + "decode_ms": 647.5, + "drafts_offered": 32, + "drafts_accepted": 28, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.923, + "total_s": 5.498, + "prompt_tokens": 9230, + "reused_tokens": 5803, + "read_tokens": 3427, + "generated_tokens": 43, + "prompt_tps": 696.8, + "prompt_tps_source": "engine", + "decode_tps": 74.8, + "decode_tps_source": "engine", + "prompt_ms": 4918.2, + "decode_ms": 575.0, + "drafts_offered": 26, + "drafts_accepted": 21, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.249, + "total_s": 4.834, + "prompt_tokens": 12190, + "reused_tokens": 9223, + "read_tokens": 2967, + "generated_tokens": 48, + "prompt_tps": 699.2, + "prompt_tps_source": "engine", + "decode_tps": 82.1, + "decode_tps_source": "engine", + "prompt_ms": 4243.2, + "decode_ms": 584.6, + "drafts_offered": 34, + "drafts_accepted": 29, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.287, + "total_s": 6.019, + "prompt_tokens": 15122, + "reused_tokens": 12183, + "read_tokens": 2939, + "generated_tokens": 48, + "prompt_tps": 556.6, + "prompt_tps_source": "engine", + "decode_tps": 65.5, + "decode_tps_source": "engine", + "prompt_ms": 5280.3, + "decode_ms": 732.7, + "drafts_offered": 36, + "drafts_accepted": 24, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.754, + "min": 68.731, + "max": 68.766, + "n": 3 + }, + "reread_tokens": { + "median": 237, + "min": 234, + "max": 264, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.249, + "min": 2.375, + "max": 5.301, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15125, + "min": 15122, + "max": 15152, + "n": 3 + }, + "turn_decode_tps": { + "median": 76.5, + "min": 65.5, + "max": 82.6, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.115, + "min": 1.111, + "max": 1.136, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..828dad04 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md @@ -0,0 +1,65 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-22 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit e97c8aa4; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_xl +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged (hip backend); thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 65536 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (started by the benchmark from the install configuration); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 84.5 (71.8-100.6) | 0.32 (0.31-0.34) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 998.2 (997.6-1001.9) | not measured | 4.03 (4.01-4.03) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 993.7 (992.3-993.9) | not measured | 16.17 (16.17-16.19) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 773.2 (771.4-773.4) | not measured | 41.47 (41.46-41.57) | +| agent session, follow-up turns | 15145 (15143-15160) at the last turn | see below | up to 64 | 3 | see below | 77.3 (66.6-81.1) | 4.31 (2.46-5.37) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.7-68.8); earlier-turn tokens read again per session: 257 (255-272) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.16 s (1.16-1.19). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 513.7 | 1.189 | 95.4 | +| 1 | 4196 | 602 | 3594 | 843.5 | 4.267 | 77.3 | +| 2 | 5843 | 4189 | 1654 | 671.1 | 2.47 | 79.8 | +| 3 | 9273 | 5836 | 3437 | 675.5 | 5.095 | 73.5 | +| 4 | 12230 | 9266 | 2964 | 689.4 | 4.307 | 68.2 | +| 5 | 15160 | 12223 | 2937 | 548.7 | 5.361 | 80.3 | + +### Concurrency: 2 clients + +Total 65.0 tok/s (63.7-65.7); first token p50 1.314 s, p95 2.391 s; request latency p50 2.984 s, p95 3.998 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 298 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/engine-log-excerpt.txt b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/engine-log-excerpt.txt new file mode 100644 index 00000000..b4d24683 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/engine-log-excerpt.txt @@ -0,0 +1 @@ +ready: http://127.0.0.1:8091/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 65536 tokens) diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..5402a21d --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536-rerun-e97c8aa/official-long/2026-10-22-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json @@ -0,0 +1,1030 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-22", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": true, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 297.67433524131775, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "e97c8aa4", + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "backend": "hip", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf", + "--max-context", + "65536", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_xl", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_XL (strata-poly)", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 65536 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 3297, + "expert_cache_mib": 6029, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.318, + "total_s": 1.409, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 161.2, + "prompt_tps_source": "engine", + "decode_tps": 91.6, + "decode_tps_source": "engine", + "prompt_ms": 316.4, + "decode_ms": 1091.4, + "drafts_offered": 75, + "drafts_accepted": 67, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.312, + "total_s": 1.705, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 138.4, + "prompt_tps_source": "engine", + "decode_tps": 71.8, + "decode_tps_source": "engine", + "prompt_ms": 310.7, + "decode_ms": 1392.5, + "drafts_offered": 77, + "drafts_accepted": 56, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.337, + "total_s": 1.632, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 137.1, + "prompt_tps_source": "engine", + "decode_tps": 77.2, + "decode_tps_source": "engine", + "prompt_ms": 335.5, + "decode_ms": 1294.7, + "drafts_offered": 72, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.325, + "total_s": 1.498, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 157.6, + "prompt_tps_source": "engine", + "decode_tps": 85.3, + "decode_tps_source": "engine", + "prompt_ms": 323.7, + "decode_ms": 1172.5, + "drafts_offered": 71, + "drafts_accepted": 59, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.306, + "total_s": 1.525, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 141.1, + "prompt_tps_source": "engine", + "decode_tps": 82.0, + "decode_tps_source": "engine", + "prompt_ms": 304.7, + "decode_ms": 1219.2, + "drafts_offered": 70, + "drafts_accepted": 55, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.323, + "total_s": 1.569, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 143.2, + "prompt_tps_source": "engine", + "decode_tps": 80.2, + "decode_tps_source": "engine", + "prompt_ms": 321.2, + "decode_ms": 1246.2, + "drafts_offered": 78, + "drafts_accepted": 55, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.337, + "total_s": 1.331, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 152.0, + "prompt_tps_source": "engine", + "decode_tps": 100.6, + "decode_tps_source": "engine", + "prompt_ms": 335.5, + "decode_ms": 994.2, + "drafts_offered": 74, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.308, + "total_s": 1.492, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 140.3, + "prompt_tps_source": "engine", + "decode_tps": 84.5, + "decode_tps_source": "engine", + "prompt_ms": 306.4, + "decode_ms": 1184.0, + "drafts_offered": 71, + "drafts_accepted": 56, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.315, + "total_s": 1.493, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 146.9, + "prompt_tps_source": "engine", + "decode_tps": 84.8, + "decode_tps_source": "engine", + "prompt_ms": 313.2, + "decode_ms": 1178.9, + "drafts_offered": 73, + "drafts_accepted": 58, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 84.5, + "min": 71.8, + "max": 100.6, + "n": 9 + }, + "ttft_s": { + "median": 0.318, + "min": 0.306, + "max": 0.337, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.033, + "total_s": 4.359, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 997.6, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 4028.8, + "decode_ms": 326.6, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.03, + "total_s": 4.336, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 998.2, + "prompt_tps_source": "engine", + "decode_tps": 78.2, + "decode_tps_source": "engine", + "prompt_ms": 4026.1, + "decode_ms": 307.0, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.015, + "total_s": 4.37, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1001.9, + "prompt_tps_source": "engine", + "decode_tps": 67.7, + "decode_tps_source": "engine", + "prompt_ms": 4011.4, + "decode_ms": 354.6, + "drafts_offered": 16, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 998.2, + "min": 997.6, + "max": 1001.9, + "n": 3 + }, + "ttft_s": { + "median": 4.03, + "min": 4.015, + "max": 4.033, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.193, + "total_s": 16.533, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 992.3, + "prompt_tps_source": "engine", + "decode_tps": 70.4, + "decode_tps_source": "engine", + "prompt_ms": 16182.0, + "decode_ms": 340.8, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.17, + "total_s": 16.507, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 993.7, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 16159.8, + "decode_ms": 337.3, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.168, + "total_s": 16.522, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 993.9, + "prompt_tps_source": "engine", + "decode_tps": 67.8, + "decode_tps_source": "engine", + "prompt_ms": 16157.3, + "decode_ms": 354.1, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 993.7, + "min": 992.3, + "max": 993.9, + "n": 3 + }, + "ttft_s": { + "median": 16.17, + "min": 16.168, + "max": 16.193, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 41.57, + "total_s": 42.098, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 771.4, + "prompt_tps_source": "engine", + "decode_tps": 45.4, + "decode_tps_source": "engine", + "prompt_ms": 41548.6, + "decode_ms": 528.6, + "drafts_offered": 22, + "drafts_accepted": 9, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 41.46, + "total_s": 41.919, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 773.4, + "prompt_tps_source": "engine", + "decode_tps": 52.3, + "decode_tps_source": "engine", + "prompt_ms": 41441.2, + "decode_ms": 459.3, + "drafts_offered": 19, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 41.47, + "total_s": 41.908, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 773.2, + "prompt_tps_source": "engine", + "decode_tps": 54.8, + "decode_tps_source": "engine", + "prompt_ms": 41450.8, + "decode_ms": 438.3, + "drafts_offered": 18, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 773.2, + "min": 771.4, + "max": 773.4, + "n": 3 + }, + "ttft_s": { + "median": 41.47, + "min": 41.46, + "max": 41.57, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.189, + "total_s": 1.534, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 33, + "prompt_tps": 513.7, + "prompt_tps_source": "engine", + "decode_tps": 95.4, + "decode_tps_source": "engine", + "prompt_ms": 1185.5, + "decode_ms": 345.9, + "drafts_offered": 25, + "drafts_accepted": 21, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.267, + "total_s": 4.978, + "prompt_tokens": 4196, + "reused_tokens": 602, + "read_tokens": 3594, + "generated_tokens": 55, + "prompt_tps": 843.5, + "prompt_tps_source": "engine", + "decode_tps": 77.3, + "decode_tps_source": "engine", + "prompt_ms": 4261.0, + "decode_ms": 711.1, + "drafts_offered": 47, + "drafts_accepted": 32, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.47, + "total_s": 3.259, + "prompt_tokens": 5843, + "reused_tokens": 4189, + "read_tokens": 1654, + "generated_tokens": 63, + "prompt_tps": 671.1, + "prompt_tps_source": "engine", + "decode_tps": 79.8, + "decode_tps_source": "engine", + "prompt_ms": 2464.5, + "decode_ms": 789.3, + "drafts_offered": 47, + "drafts_accepted": 36, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.095, + "total_s": 5.639, + "prompt_tokens": 9273, + "reused_tokens": 5836, + "read_tokens": 3437, + "generated_tokens": 40, + "prompt_tps": 675.5, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 5088.1, + "decode_ms": 544.2, + "drafts_offered": 30, + "drafts_accepted": 21, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.307, + "total_s": 4.982, + "prompt_tokens": 12230, + "reused_tokens": 9266, + "read_tokens": 2964, + "generated_tokens": 46, + "prompt_tps": 689.4, + "prompt_tps_source": "engine", + "decode_tps": 68.2, + "decode_tps_source": "engine", + "prompt_ms": 4299.6, + "decode_ms": 674.8, + "drafts_offered": 44, + "drafts_accepted": 26, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.361, + "total_s": 6.082, + "prompt_tokens": 15160, + "reused_tokens": 12223, + "read_tokens": 2937, + "generated_tokens": 58, + "prompt_tps": 548.7, + "prompt_tps_source": "engine", + "decode_tps": 80.3, + "decode_tps_source": "engine", + "prompt_ms": 5353.1, + "decode_ms": 722.2, + "drafts_offered": 39, + "drafts_accepted": 36, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.164, + "total_s": 1.288, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 524.3, + "prompt_tps_source": "engine", + "decode_tps": 88.2, + "decode_tps_source": "engine", + "prompt_ms": 1161.6, + "decode_ms": 124.7, + "drafts_offered": 9, + "drafts_accepted": 6, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.247, + "total_s": 4.917, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 52, + "prompt_tps": 841.8, + "prompt_tps_source": "engine", + "decode_tps": 77.6, + "decode_tps_source": "engine", + "prompt_ms": 4243.4, + "decode_ms": 669.8, + "drafts_offered": 38, + "drafts_accepted": 29, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.468, + "total_s": 3.226, + "prompt_tokens": 5818, + "reused_tokens": 4167, + "read_tokens": 1651, + "generated_tokens": 60, + "prompt_tps": 669.9, + "prompt_tps_source": "engine", + "decode_tps": 79.2, + "decode_tps_source": "engine", + "prompt_ms": 2464.5, + "decode_ms": 757.9, + "drafts_offered": 40, + "drafts_accepted": 32, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.085, + "total_s": 5.722, + "prompt_tokens": 9245, + "reused_tokens": 5811, + "read_tokens": 3434, + "generated_tokens": 48, + "prompt_tps": 676.1, + "prompt_tps_source": "engine", + "decode_tps": 75.3, + "decode_tps_source": "engine", + "prompt_ms": 5079.4, + "decode_ms": 637.8, + "drafts_offered": 36, + "drafts_accepted": 26, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.3, + "total_s": 4.929, + "prompt_tokens": 12210, + "reused_tokens": 9238, + "read_tokens": 2972, + "generated_tokens": 51, + "prompt_tps": 692.0, + "prompt_tps_source": "engine", + "decode_tps": 81.1, + "decode_tps_source": "engine", + "prompt_ms": 4294.5, + "decode_ms": 628.8, + "drafts_offered": 37, + "drafts_accepted": 31, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.371, + "total_s": 6.256, + "prompt_tokens": 15145, + "reused_tokens": 12203, + "read_tokens": 2942, + "generated_tokens": 59, + "prompt_tps": 548.4, + "prompt_tps_source": "engine", + "decode_tps": 66.6, + "decode_tps_source": "engine", + "prompt_ms": 5364.7, + "decode_ms": 885.6, + "drafts_offered": 49, + "drafts_accepted": 31, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.164, + "total_s": 1.502, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 30, + "prompt_tps": 524.2, + "prompt_tps_source": "engine", + "decode_tps": 88.7, + "decode_tps_source": "engine", + "prompt_ms": 1161.8, + "decode_ms": 338.2, + "drafts_offered": 22, + "drafts_accepted": 17, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.276, + "total_s": 4.96, + "prompt_tokens": 4193, + "reused_tokens": 602, + "read_tokens": 3591, + "generated_tokens": 55, + "prompt_tps": 840.7, + "prompt_tps_source": "engine", + "decode_tps": 80.4, + "decode_tps_source": "engine", + "prompt_ms": 4271.3, + "decode_ms": 684.4, + "drafts_offered": 45, + "drafts_accepted": 32, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.464, + "total_s": 3.17, + "prompt_tokens": 5840, + "reused_tokens": 4186, + "read_tokens": 1654, + "generated_tokens": 53, + "prompt_tps": 672.2, + "prompt_tps_source": "engine", + "decode_tps": 75.1, + "decode_tps_source": "engine", + "prompt_ms": 2460.6, + "decode_ms": 705.5, + "drafts_offered": 37, + "drafts_accepted": 27, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.093, + "total_s": 5.602, + "prompt_tokens": 9260, + "reused_tokens": 5833, + "read_tokens": 3427, + "generated_tokens": 40, + "prompt_tps": 673.6, + "prompt_tps_source": "engine", + "decode_tps": 78.4, + "decode_tps_source": "engine", + "prompt_ms": 5087.4, + "decode_ms": 510.0, + "drafts_offered": 28, + "drafts_accepted": 22, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.307, + "total_s": 4.886, + "prompt_tokens": 12217, + "reused_tokens": 9253, + "read_tokens": 2964, + "generated_tokens": 42, + "prompt_tps": 689.1, + "prompt_tps_source": "engine", + "decode_tps": 72.5, + "decode_tps_source": "engine", + "prompt_ms": 4301.2, + "decode_ms": 579.3, + "drafts_offered": 36, + "drafts_accepted": 24, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.365, + "total_s": 6.111, + "prompt_tokens": 15143, + "reused_tokens": 12210, + "read_tokens": 2933, + "generated_tokens": 52, + "prompt_tps": 547.3, + "prompt_tps_source": "engine", + "decode_tps": 69.7, + "decode_tps_source": "engine", + "prompt_ms": 5359.0, + "decode_ms": 746.4, + "drafts_offered": 41, + "drafts_accepted": 29, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.768, + "min": 68.726, + "max": 68.772, + "n": 3 + }, + "reread_tokens": { + "median": 257, + "min": 255, + "max": 272, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.307, + "min": 2.464, + "max": 5.371, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15145, + "min": 15143, + "max": 15160, + "n": 3 + }, + "turn_decode_tps": { + "median": 77.3, + "min": 66.6, + "max": 81.1, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.164, + "min": 1.164, + "max": 1.189, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 4.02, + "tokens": 256, + "total_tps": 63.72, + "first_s": [ + 0.37801819600281306, + 2.2743712489900645 + ], + "latency_s": [ + 1.9029239530209452, + 4.017055921984138 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 3.94, + "tokens": 256, + "total_tps": 64.96, + "first_s": [ + 0.378448252013186, + 2.4293799310107715 + ], + "latency_s": [ + 2.070366432017181, + 3.9404914999904577 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 3.9, + "tokens": 256, + "total_tps": 65.69, + "first_s": [ + 0.36614990199450403, + 2.249222241982352 + ], + "latency_s": [ + 1.8699755140114576, + 3.8969837320037186 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 64.96, + "min": 63.72, + "max": 65.69, + "n": 3 + }, + "ttft_p50_s": 1.314, + "ttft_p95_s": 2.391, + "latency_p50_s": 2.984, + "latency_p95_s": 3.998, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md new file mode 100644 index 00000000..9deb6c55 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/README.md @@ -0,0 +1,65 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-20 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_xl +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 65536 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 81.4 (74.4-98.2) | 0.33 (0.31-0.34) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1000.9 (999.4-1002.7) | not measured | 4.02 (4.01-4.03) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 997.5 (994.2-998.5) | not measured | 16.11 (16.09-16.16) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 775.0 (772.7-776.5) | not measured | 41.38 (41.30-41.49) | +| agent session, follow-up turns | 15161 (15153-15162) at the last turn | see below | up to 64 | 3 | see below | 74.4 (66.1-85.0) | 4.33 (2.46-5.37) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.8-68.8); earlier-turn tokens read again per session: 273 (265-274) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.19 s (1.15-1.24). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 491.7 | 1.242 | 90.4 | +| 1 | 4198 | 602 | 3596 | 832.0 | 4.328 | 75.4 | +| 2 | 5832 | 4191 | 1641 | 658.2 | 2.499 | 74.4 | +| 3 | 9257 | 5825 | 3432 | 671.8 | 5.117 | 68.2 | +| 4 | 12221 | 9250 | 2971 | 687.5 | 4.33 | 72.8 | +| 5 | 15153 | 12214 | 2939 | 549.6 | 5.355 | 66.1 | + +### Concurrency: 2 clients + +Total 63.2 tok/s (62.7-65.7); first token p50 1.412 s, p95 2.533 s; request latency p50 3.039 s, p95 4.076 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 304 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json new file mode 100644 index 00000000..6d74536c --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/qwen36-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-20-community-unknown-qwen3-6-35b-a3b-ud-q4-k-xl/results.json @@ -0,0 +1,1029 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-20", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 304.2440233230591, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf", + "--max-context", + "65536", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_xl", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_XL (strata-poly)", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 65536 --tail-skip 0 --kv-whole --conversation-cache-mib 0 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_xl", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 3287, + "expert_cache_mib": 6011, + "pool_workers": 12, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.334, + "total_s": 1.579, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 153.5, + "prompt_tps_source": "engine", + "decode_tps": 80.3, + "decode_tps_source": "engine", + "prompt_ms": 332.3, + "decode_ms": 1245.3, + "drafts_offered": 62, + "drafts_accepted": 57, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.323, + "total_s": 1.666, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 133.7, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 321.7, + "decode_ms": 1343.3, + "drafts_offered": 63, + "drafts_accepted": 54, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.341, + "total_s": 1.679, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 135.7, + "prompt_tps_source": "engine", + "decode_tps": 74.7, + "decode_tps_source": "engine", + "prompt_ms": 338.9, + "decode_ms": 1338.3, + "drafts_offered": 69, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.327, + "total_s": 1.394, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 157.0, + "prompt_tps_source": "engine", + "decode_tps": 93.7, + "decode_tps_source": "engine", + "prompt_ms": 324.9, + "decode_ms": 1067.7, + "drafts_offered": 76, + "drafts_accepted": 69, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.31, + "total_s": 1.514, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 139.2, + "prompt_tps_source": "engine", + "decode_tps": 83.1, + "decode_tps_source": "engine", + "prompt_ms": 308.8, + "decode_ms": 1203.6, + "drafts_offered": 62, + "drafts_accepted": 53, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.327, + "total_s": 1.523, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 141.5, + "prompt_tps_source": "engine", + "decode_tps": 83.6, + "decode_tps_source": "engine", + "prompt_ms": 325.1, + "decode_ms": 1196.4, + "drafts_offered": 73, + "drafts_accepted": 58, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.341, + "total_s": 1.359, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 150.5, + "prompt_tps_source": "engine", + "decode_tps": 98.2, + "decode_tps_source": "engine", + "prompt_ms": 338.9, + "decode_ms": 1018.5, + "drafts_offered": 74, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.313, + "total_s": 1.541, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 138.3, + "prompt_tps_source": "engine", + "decode_tps": 81.4, + "decode_tps_source": "engine", + "prompt_ms": 311.0, + "decode_ms": 1229.0, + "drafts_offered": 69, + "drafts_accepted": 55, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.319, + "total_s": 1.559, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 145.0, + "prompt_tps_source": "engine", + "decode_tps": 80.6, + "decode_tps_source": "engine", + "prompt_ms": 317.3, + "decode_ms": 1240.1, + "drafts_offered": 74, + "drafts_accepted": 55, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 81.4, + "min": 74.4, + "max": 98.2, + "n": 9 + }, + "ttft_s": { + "median": 0.327, + "min": 0.31, + "max": 0.341, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.026, + "total_s": 4.379, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 999.4, + "prompt_tps_source": "engine", + "decode_tps": 67.9, + "decode_tps_source": "engine", + "prompt_ms": 4021.6, + "decode_ms": 353.6, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.012, + "total_s": 4.319, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1002.7, + "prompt_tps_source": "engine", + "decode_tps": 78.0, + "decode_tps_source": "engine", + "prompt_ms": 4008.1, + "decode_ms": 307.5, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.019, + "total_s": 4.328, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1000.9, + "prompt_tps_source": "engine", + "decode_tps": 77.6, + "decode_tps_source": "engine", + "prompt_ms": 4015.3, + "decode_ms": 309.3, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1000.9, + "min": 999.4, + "max": 1002.7, + "n": 3 + }, + "ttft_s": { + "median": 4.019, + "min": 4.012, + "max": 4.026, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.109, + "total_s": 16.638, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 997.5, + "prompt_tps_source": "engine", + "decode_tps": 45.4, + "decode_tps_source": "engine", + "prompt_ms": 16098.4, + "decode_ms": 529.2, + "drafts_offered": 27, + "drafts_accepted": 8, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.162, + "total_s": 16.523, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 994.2, + "prompt_tps_source": "engine", + "decode_tps": 66.5, + "decode_tps_source": "engine", + "prompt_ms": 16151.2, + "decode_ms": 361.1, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.092, + "total_s": 16.45, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 998.5, + "prompt_tps_source": "engine", + "decode_tps": 66.9, + "decode_tps_source": "engine", + "prompt_ms": 16081.4, + "decode_ms": 358.7, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 997.5, + "min": 994.2, + "max": 998.5, + "n": 3 + }, + "ttft_s": { + "median": 16.109, + "min": 16.092, + "max": 16.162, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 41.494, + "total_s": 41.905, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 772.7, + "prompt_tps_source": "engine", + "decode_tps": 58.3, + "decode_tps_source": "engine", + "prompt_ms": 41474.9, + "decode_ms": 411.8, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 41.377, + "total_s": 41.824, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 775.0, + "prompt_tps_source": "engine", + "decode_tps": 53.5, + "decode_tps_source": "engine", + "prompt_ms": 41356.1, + "decode_ms": 448.2, + "drafts_offered": 17, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 41.297, + "total_s": 41.758, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 776.5, + "prompt_tps_source": "engine", + "decode_tps": 51.9, + "decode_tps_source": "engine", + "prompt_ms": 41276.3, + "decode_ms": 462.1, + "drafts_offered": 15, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 775.0, + "min": 772.7, + "max": 776.5, + "n": 3 + }, + "ttft_s": { + "median": 41.377, + "min": 41.297, + "max": 41.494, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.242, + "total_s": 1.629, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 35, + "prompt_tps": 491.7, + "prompt_tps_source": "engine", + "decode_tps": 90.4, + "decode_tps_source": "engine", + "prompt_ms": 1238.5, + "decode_ms": 387.3, + "drafts_offered": 28, + "drafts_accepted": 22, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.328, + "total_s": 4.885, + "prompt_tokens": 4198, + "reused_tokens": 602, + "read_tokens": 3596, + "generated_tokens": 42, + "prompt_tps": 832.0, + "prompt_tps_source": "engine", + "decode_tps": 75.4, + "decode_tps_source": "engine", + "prompt_ms": 4322.0, + "decode_ms": 557.2, + "drafts_offered": 36, + "drafts_accepted": 24, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.499, + "total_s": 3.278, + "prompt_tokens": 5832, + "reused_tokens": 4191, + "read_tokens": 1641, + "generated_tokens": 58, + "prompt_tps": 658.2, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 2493.3, + "decode_ms": 779.4, + "drafts_offered": 42, + "drafts_accepted": 34, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.117, + "total_s": 5.806, + "prompt_tokens": 9257, + "reused_tokens": 5825, + "read_tokens": 3432, + "generated_tokens": 47, + "prompt_tps": 671.8, + "prompt_tps_source": "engine", + "decode_tps": 68.2, + "decode_tps_source": "engine", + "prompt_ms": 5108.9, + "decode_ms": 689.5, + "drafts_offered": 33, + "drafts_accepted": 23, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.33, + "total_s": 4.99, + "prompt_tokens": 12221, + "reused_tokens": 9250, + "read_tokens": 2971, + "generated_tokens": 48, + "prompt_tps": 687.5, + "prompt_tps_source": "engine", + "decode_tps": 72.8, + "decode_tps_source": "engine", + "prompt_ms": 4321.7, + "decode_ms": 659.7, + "drafts_offered": 34, + "drafts_accepted": 25, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.355, + "total_s": 6.233, + "prompt_tokens": 15153, + "reused_tokens": 12214, + "read_tokens": 2939, + "generated_tokens": 58, + "prompt_tps": 549.6, + "prompt_tps_source": "engine", + "decode_tps": 66.1, + "decode_tps_source": "engine", + "prompt_ms": 5347.9, + "decode_ms": 877.9, + "drafts_offered": 42, + "drafts_accepted": 28, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.146, + "total_s": 1.44, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 28, + "prompt_tps": 532.3, + "prompt_tps_source": "engine", + "decode_tps": 95.1, + "decode_tps_source": "engine", + "prompt_ms": 1144.0, + "decode_ms": 294.5, + "drafts_offered": 22, + "drafts_accepted": 18, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.304, + "total_s": 5.027, + "prompt_tokens": 4191, + "reused_tokens": 602, + "read_tokens": 3589, + "generated_tokens": 55, + "prompt_tps": 834.7, + "prompt_tps_source": "engine", + "decode_tps": 76.1, + "decode_tps_source": "engine", + "prompt_ms": 4299.7, + "decode_ms": 723.2, + "drafts_offered": 48, + "drafts_accepted": 32, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.46, + "total_s": 3.207, + "prompt_tokens": 5838, + "reused_tokens": 4184, + "read_tokens": 1654, + "generated_tokens": 63, + "prompt_tps": 673.5, + "prompt_tps_source": "engine", + "decode_tps": 84.3, + "decode_tps_source": "engine", + "prompt_ms": 2455.9, + "decode_ms": 747.0, + "drafts_offered": 48, + "drafts_accepted": 38, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.09, + "total_s": 5.619, + "prompt_tokens": 9268, + "reused_tokens": 5831, + "read_tokens": 3437, + "generated_tokens": 45, + "prompt_tps": 676.0, + "prompt_tps_source": "engine", + "decode_tps": 85.0, + "decode_tps_source": "engine", + "prompt_ms": 5084.3, + "decode_ms": 529.6, + "drafts_offered": 27, + "drafts_accepted": 26, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.298, + "total_s": 4.961, + "prompt_tokens": 12230, + "reused_tokens": 9261, + "read_tokens": 2969, + "generated_tokens": 48, + "prompt_tps": 691.7, + "prompt_tps_source": "engine", + "decode_tps": 72.4, + "decode_tps_source": "engine", + "prompt_ms": 4292.4, + "decode_ms": 663.2, + "drafts_offered": 42, + "drafts_accepted": 28, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.361, + "total_s": 6.153, + "prompt_tokens": 15162, + "reused_tokens": 12223, + "read_tokens": 2939, + "generated_tokens": 57, + "prompt_tps": 548.9, + "prompt_tps_source": "engine", + "decode_tps": 71.9, + "decode_tps_source": "engine", + "prompt_ms": 5354.1, + "decode_ms": 792.4, + "drafts_offered": 44, + "drafts_accepted": 32, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.186, + "total_s": 1.467, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 29, + "prompt_tps": 514.4, + "prompt_tps_source": "engine", + "decode_tps": 103.0, + "decode_tps_source": "engine", + "prompt_ms": 1184.0, + "decode_ms": 281.6, + "drafts_offered": 22, + "drafts_accepted": 19, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.271, + "total_s": 4.941, + "prompt_tokens": 4192, + "reused_tokens": 602, + "read_tokens": 3590, + "generated_tokens": 55, + "prompt_tps": 841.4, + "prompt_tps_source": "engine", + "decode_tps": 82.1, + "decode_tps_source": "engine", + "prompt_ms": 4266.8, + "decode_ms": 670.1, + "drafts_offered": 43, + "drafts_accepted": 32, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.466, + "total_s": 3.206, + "prompt_tokens": 5839, + "reused_tokens": 4185, + "read_tokens": 1654, + "generated_tokens": 59, + "prompt_tps": 671.7, + "prompt_tps_source": "engine", + "decode_tps": 79.7, + "decode_tps_source": "engine", + "prompt_ms": 2462.5, + "decode_ms": 740.4, + "drafts_offered": 41, + "drafts_accepted": 32, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.139, + "total_s": 5.767, + "prompt_tokens": 9265, + "reused_tokens": 5832, + "read_tokens": 3433, + "generated_tokens": 44, + "prompt_tps": 668.8, + "prompt_tps_source": "engine", + "decode_tps": 69.9, + "decode_tps_source": "engine", + "prompt_ms": 5133.1, + "decode_ms": 629.1, + "drafts_offered": 27, + "drafts_accepted": 20, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.294, + "total_s": 4.914, + "prompt_tokens": 12226, + "reused_tokens": 9258, + "read_tokens": 2968, + "generated_tokens": 51, + "prompt_tps": 692.3, + "prompt_tps_source": "engine", + "decode_tps": 82.0, + "decode_tps_source": "engine", + "prompt_ms": 4287.1, + "decode_ms": 621.6, + "drafts_offered": 39, + "drafts_accepted": 32, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.372, + "total_s": 6.17, + "prompt_tokens": 15161, + "reused_tokens": 12219, + "read_tokens": 2942, + "generated_tokens": 58, + "prompt_tps": 548.3, + "prompt_tps_source": "engine", + "decode_tps": 72.6, + "decode_tps_source": "engine", + "prompt_ms": 5365.6, + "decode_ms": 798.4, + "drafts_offered": 39, + "drafts_accepted": 31, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.755, + "min": 68.753, + "max": 68.755, + "n": 3 + }, + "reread_tokens": { + "median": 273, + "min": 265, + "max": 274, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.328, + "min": 2.46, + "max": 5.372, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15161, + "min": 15153, + "max": 15162, + "n": 3 + }, + "turn_decode_tps": { + "median": 74.4, + "min": 66.1, + "max": 85.0, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.186, + "min": 1.146, + "max": 1.242, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 4.08, + "tokens": 256, + "total_tps": 62.67, + "first_s": [ + 0.40562668099300936, + 2.5694049849989824 + ], + "latency_s": [ + 2.1828930049960036, + 4.084232332010288 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 3.9, + "tokens": 256, + "total_tps": 65.71, + "first_s": [ + 0.37878235698735807, + 2.4247594870103057 + ], + "latency_s": [ + 2.0177453349897405, + 3.895597277005436 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 4.05, + "tokens": 256, + "total_tps": 63.21, + "first_s": [ + 0.37335063400678337, + 2.4173959890031256 + ], + "latency_s": [ + 2.029798349001794, + 4.049449420010205 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 63.21, + "min": 62.67, + "max": 65.71, + "n": 3 + }, + "ttft_p50_s": 1.412, + "ttft_p95_s": 2.533, + "latency_p50_s": 3.039, + "latency_p95_s": 4.076, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/README.md b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/README.md new file mode 100644 index 00000000..78548188 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on AMD Radeon + +Measured on 2026-10-21 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: AMD Radeon, 12 GiB VRAM +- CPU: AMD Ryzen 5 5600X 6-Core Processor (6 cores, 12 threads); installed RAM: 31.2 GiB; storage: not measured +- OS: Linux 7.0.0-38-generic; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit c54f0eaf; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: tiel-coder-35b-a3b-ud-q3_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged (hip backend); thinking off for the suite; temperature 0 + +```text +strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 224 (221-229) | 0 | 100 (100-100) | 9 | not measured | 73.1 (69.0-91.8) | 0.52 (0.50-0.59) | +| 4K prompt, fresh | 4025 (4025-4025) | 0 (0-0) | 24 cap | 3 | 1241.5 (1240.1-1241.5) | not measured | 3.25 (3.25-3.25) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1031.6 (1028.5-1036.6) | not measured | 15.58 (15.50-15.62) | +| agent session, follow-up turns | 15391 (15348-15396) at the last turn | see below | up to 64 | 3 | see below | 66.5 (57.3-77.6) | 4.11 (1.90-4.62) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 70.0 % (69.9-70.0); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 14334 (14334-14335) new tokens of the files are read once); first turn (nothing cached) first token: 1.04 s (1.04-1.04). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 835 | 0 | 835 | 807.4 | 1.037 | 64.3 | +| 1 | 4403 | 844 | 3559 | 976.8 | 3.649 | 74.5 | +| 2 | 6041 | 4444 | 1597 | 843.9 | 1.897 | 68.4 | +| 3 | 9477 | 6104 | 3373 | 734.4 | 4.6 | 62.4 | +| 4 | 12455 | 9533 | 2922 | 712.2 | 4.111 | 69.4 | +| 5 | 15396 | 12507 | 2889 | 666.1 | 4.345 | 66.5 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 150 s. diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/results.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/results.json new file mode 100644 index 00000000..bf93da97 --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-21-community-unknown-tiel-coder-35b-a3b-ud-q3-k-xl/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-21", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 150.45646357536316, + "machine": { + "gpu": [ + "AMD Radeon" + ], + "gpu_vram_mib": 12272, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 5600X 6-Core Processor", + "cpu_cores": 6, + "cpu_threads": 12, + "ram_gib": 31.2, + "os": "Linux 7.0.0-38-generic", + "python": "3.14.4", + "strata_version": "0.1.45", + "strata_commit": "c54f0eaf", + "engine_version": "0.1.45", + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "quant": "UD-Q3_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "backend": "hip", + "args": [ + "--model", + "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "tiel-coder-35b-a3b-ud-q3_k_xl", + "display_name": "Tiel Coder 35B-A3B UD-Q3_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.14/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.14/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8091, + "host": "127.0.0.1", + "open_browser": false + }, + "launch": "strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 5221, + "expert_cache_mib": 6966, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.499, + "total_s": 1.839, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 460.9, + "prompt_tps_source": "engine", + "decode_tps": 74.6, + "decode_tps_source": "engine", + "prompt_ms": 496.9, + "decode_ms": 1340.7, + "drafts_offered": 65, + "drafts_accepted": 60, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.505, + "total_s": 1.87, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 438.8, + "prompt_tps_source": "engine", + "decode_tps": 73.3, + "decode_tps_source": "engine", + "prompt_ms": 503.6, + "decode_ms": 1364.7, + "drafts_offered": 53, + "drafts_accepted": 45, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.59, + "total_s": 2.038, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 380.6, + "prompt_tps_source": "engine", + "decode_tps": 69.0, + "decode_tps_source": "engine", + "prompt_ms": 588.5, + "decode_ms": 1448.3, + "drafts_offered": 51, + "drafts_accepted": 42, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.52, + "total_s": 1.744, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 441.6, + "prompt_tps_source": "engine", + "decode_tps": 81.7, + "decode_tps_source": "engine", + "prompt_ms": 518.6, + "decode_ms": 1223.8, + "drafts_offered": 53, + "drafts_accepted": 52, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.51, + "total_s": 1.881, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 435.1, + "prompt_tps_source": "engine", + "decode_tps": 72.9, + "decode_tps_source": "engine", + "prompt_ms": 507.9, + "decode_ms": 1371.1, + "drafts_offered": 63, + "drafts_accepted": 46, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.519, + "total_s": 1.924, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 433.2, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 517.1, + "decode_ms": 1405.0, + "drafts_offered": 68, + "drafts_accepted": 49, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.519, + "total_s": 1.609, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 442.4, + "prompt_tps_source": "engine", + "decode_tps": 91.8, + "decode_tps_source": "engine", + "prompt_ms": 517.6, + "decode_ms": 1089.2, + "drafts_offered": 73, + "drafts_accepted": 67, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.522, + "total_s": 1.89, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 425.0, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 520.0, + "decode_ms": 1368.4, + "drafts_offered": 64, + "drafts_accepted": 48, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.517, + "total_s": 1.916, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 434.1, + "prompt_tps_source": "engine", + "decode_tps": 71.5, + "decode_tps_source": "engine", + "prompt_ms": 516.0, + "decode_ms": 1398.3, + "drafts_offered": 67, + "drafts_accepted": 49, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 73.1, + "min": 69.0, + "max": 91.8, + "n": 9 + }, + "ttft_s": { + "median": 0.519, + "min": 0.499, + "max": 0.59, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 224, + "min": 221, + "max": 229, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.25, + "total_s": 3.6, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1240.1, + "prompt_tps_source": "engine", + "decode_tps": 68.6, + "decode_tps_source": "engine", + "prompt_ms": 3245.7, + "decode_ms": 350.1, + "drafts_offered": 16, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.246, + "total_s": 3.583, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1241.5, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 3242.1, + "decode_ms": 337.6, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.246, + "total_s": 3.569, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1241.5, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 3242.1, + "decode_ms": 322.7, + "drafts_offered": 16, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1241.5, + "min": 1240.1, + "max": 1241.5, + "n": 3 + }, + "ttft_s": { + "median": 3.246, + "min": 3.246, + "max": 3.25, + "n": 3 + }, + "prompt_tokens": { + "median": 4025, + "min": 4025, + "max": 4025, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 15.578, + "total_s": 15.991, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1031.6, + "prompt_tps_source": "engine", + "decode_tps": 58.0, + "decode_tps_source": "engine", + "prompt_ms": 15566.7, + "decode_ms": 413.5, + "drafts_offered": 18, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.504, + "total_s": 15.894, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1036.6, + "prompt_tps_source": "engine", + "decode_tps": 61.5, + "decode_tps_source": "engine", + "prompt_ms": 15491.7, + "decode_ms": 390.2, + "drafts_offered": 12, + "drafts_accepted": 9, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.624, + "total_s": 15.953, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1028.5, + "prompt_tps_source": "engine", + "decode_tps": 73.0, + "decode_tps_source": "engine", + "prompt_ms": 15613.1, + "decode_ms": 328.7, + "drafts_offered": 15, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1031.6, + "min": 1028.5, + "max": 1036.6, + "n": 3 + }, + "ttft_s": { + "median": 15.578, + "min": 15.504, + "max": 15.624, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.037, + "total_s": 1.193, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 807.4, + "prompt_tps_source": "engine", + "decode_tps": 64.3, + "decode_tps_source": "engine", + "prompt_ms": 1034.2, + "decode_ms": 155.5, + "drafts_offered": 11, + "drafts_accepted": 5, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.649, + "total_s": 4.212, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 42, + "prompt_tps": 976.8, + "prompt_tps_source": "engine", + "decode_tps": 74.5, + "decode_tps_source": "engine", + "prompt_ms": 3643.6, + "decode_ms": 563.6, + "drafts_offered": 30, + "drafts_accepted": 23, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.897, + "total_s": 2.833, + "prompt_tokens": 6041, + "reused_tokens": 4444, + "read_tokens": 1597, + "generated_tokens": 64, + "prompt_tps": 843.9, + "prompt_tps_source": "engine", + "decode_tps": 68.4, + "decode_tps_source": "engine", + "prompt_ms": 1892.3, + "decode_ms": 935.7, + "drafts_offered": 48, + "drafts_accepted": 33, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.6, + "total_s": 5.514, + "prompt_tokens": 9477, + "reused_tokens": 6104, + "read_tokens": 3373, + "generated_tokens": 57, + "prompt_tps": 734.4, + "prompt_tps_source": "engine", + "decode_tps": 62.4, + "decode_tps_source": "engine", + "prompt_ms": 4592.8, + "decode_ms": 913.5, + "drafts_offered": 40, + "drafts_accepted": 25, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.111, + "total_s": 4.874, + "prompt_tokens": 12455, + "reused_tokens": 9533, + "read_tokens": 2922, + "generated_tokens": 53, + "prompt_tps": 712.2, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 4102.5, + "decode_ms": 763.3, + "drafts_offered": 38, + "drafts_accepted": 29, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.345, + "total_s": 5.307, + "prompt_tokens": 15396, + "reused_tokens": 12507, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 666.1, + "prompt_tps_source": "engine", + "decode_tps": 66.5, + "decode_tps_source": "engine", + "prompt_ms": 4337.2, + "decode_ms": 962.2, + "drafts_offered": 46, + "drafts_accepted": 36, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.037, + "total_s": 1.164, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 806.7, + "prompt_tps_source": "engine", + "decode_tps": 78.7, + "decode_tps_source": "engine", + "prompt_ms": 1035.1, + "decode_ms": 127.1, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.719, + "total_s": 4.365, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 44, + "prompt_tps": 958.0, + "prompt_tps_source": "engine", + "decode_tps": 68.1, + "decode_tps_source": "engine", + "prompt_ms": 3715.2, + "decode_ms": 646.1, + "drafts_offered": 29, + "drafts_accepted": 20, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.896, + "total_s": 2.909, + "prompt_tokens": 6043, + "reused_tokens": 4446, + "read_tokens": 1597, + "generated_tokens": 60, + "prompt_tps": 844.2, + "prompt_tps_source": "engine", + "decode_tps": 59.2, + "decode_tps_source": "engine", + "prompt_ms": 1891.7, + "decode_ms": 1013.8, + "drafts_offered": 44, + "drafts_accepted": 23, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.611, + "total_s": 5.606, + "prompt_tokens": 9474, + "reused_tokens": 6102, + "read_tokens": 3372, + "generated_tokens": 57, + "prompt_tps": 732.2, + "prompt_tps_source": "engine", + "decode_tps": 57.3, + "decode_tps_source": "engine", + "prompt_ms": 4605.4, + "decode_ms": 995.0, + "drafts_offered": 39, + "drafts_accepted": 21, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.117, + "total_s": 4.871, + "prompt_tokens": 12452, + "reused_tokens": 9530, + "read_tokens": 2922, + "generated_tokens": 51, + "prompt_tps": 710.9, + "prompt_tps_source": "engine", + "decode_tps": 67.5, + "decode_tps_source": "engine", + "prompt_ms": 4110.1, + "decode_ms": 755.0, + "drafts_offered": 35, + "drafts_accepted": 26, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.269, + "total_s": 5.28, + "prompt_tokens": 15391, + "reused_tokens": 12502, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 677.8, + "prompt_tps_source": "engine", + "decode_tps": 63.3, + "decode_tps_source": "engine", + "prompt_ms": 4262.3, + "decode_ms": 1011.3, + "drafts_offered": 53, + "drafts_accepted": 35, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.041, + "total_s": 1.174, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 803.5, + "prompt_tps_source": "engine", + "decode_tps": 75.4, + "decode_tps_source": "engine", + "prompt_ms": 1039.2, + "decode_ms": 132.6, + "drafts_offered": 9, + "drafts_accepted": 5, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.687, + "total_s": 4.202, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 40, + "prompt_tps": 966.4, + "prompt_tps_source": "engine", + "decode_tps": 77.6, + "decode_tps_source": "engine", + "prompt_ms": 3682.9, + "decode_ms": 515.2, + "drafts_offered": 28, + "drafts_accepted": 22, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.904, + "total_s": 2.605, + "prompt_tokens": 6039, + "reused_tokens": 4442, + "read_tokens": 1597, + "generated_tokens": 49, + "prompt_tps": 840.7, + "prompt_tps_source": "engine", + "decode_tps": 69.8, + "decode_tps_source": "engine", + "prompt_ms": 1899.6, + "decode_ms": 701.6, + "drafts_offered": 33, + "drafts_accepted": 24, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.625, + "total_s": 5.169, + "prompt_tokens": 9459, + "reused_tokens": 6087, + "read_tokens": 3372, + "generated_tokens": 36, + "prompt_tps": 730.0, + "prompt_tps_source": "engine", + "decode_tps": 66.0, + "decode_tps_source": "engine", + "prompt_ms": 4619.1, + "decode_ms": 545.2, + "drafts_offered": 25, + "drafts_accepted": 17, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.09, + "total_s": 4.803, + "prompt_tokens": 12416, + "reused_tokens": 9494, + "read_tokens": 2922, + "generated_tokens": 44, + "prompt_tps": 715.6, + "prompt_tps_source": "engine", + "decode_tps": 61.6, + "decode_tps_source": "engine", + "prompt_ms": 4083.3, + "decode_ms": 713.8, + "drafts_offered": 30, + "drafts_accepted": 19, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.126, + "total_s": 5.092, + "prompt_tokens": 15348, + "reused_tokens": 12459, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 701.4, + "prompt_tps_source": "engine", + "decode_tps": 66.2, + "decode_tps_source": "engine", + "prompt_ms": 4118.8, + "decode_ms": 966.8, + "drafts_offered": 47, + "drafts_accepted": 34, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 69.979, + "min": 69.917, + "max": 69.982, + "n": 3 + }, + "reread_tokens": { + "median": 5, + "min": 5, + "max": 5, + "n": 3 + }, + "new_tokens": { + "median": 14334, + "min": 14334, + "max": 14335, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.111, + "min": 1.896, + "max": 4.625, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15391, + "min": 15348, + "max": 15396, + "n": 3 + }, + "turn_decode_tps": { + "median": 66.5, + "min": 57.3, + "max": 77.6, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.037, + "min": 1.037, + "max": 1.041, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/soak-summary.json b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/soak-summary.json new file mode 100644 index 00000000..033c990f --- /dev/null +++ b/bench/results/2026-10-22-community-rx6700xt-5600x-polystrata/soak-summary.json @@ -0,0 +1,10 @@ +{ + "finished_at": 1792615294.8604295, + "hours": 6, + "health_ok": 483, + "health_fails": 0, + "hourly_ok": 8, + "hourly_failed": 0, + "max_server_rss_kB": 169568, + "note": "day-desktop soak (user active in Chrome/webchat); summary rebuilt from soak.log (writer argv bug fixed)" +} \ No newline at end of file