diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/README.md new file mode 100644 index 000000000..c71b3151f --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/README.md @@ -0,0 +1,61 @@ +# RX 6700 XT 12 GB: official PolyStrata speed suite + +Measured on 2026-10-19 from `26e4cb5df7996edfb5ff549a63b82d8581e1e658`, with the native HIP `gfx1031` engine 0.1.45. This report contains only the upstream `tools/strata_bench.py` suite. No custom quality tests are included. + +## Hardware and environment + +- GPU: AMD Radeon RX 6700 XT, 12 GB dedicated VRAM; one card used. +- CPU: AMD Ryzen 5 3600, 6 physical cores / 12 logical CPUs; eight engine workers. +- RAM: 32 GB installed. Unprivileged Debian 13 LXC limited to 28 GiB, no guest swap. +- Host kernel: 7.0.14-24-pve, existing amdgpu driver. No host driver change. +- HIP userspace: ROCm 7.13.0a20260515. Native gfx1031 source build; no architecture override. +- Storage: Intel SSDPEKNW010T8 NVMe SSD. Observed PCIe GPU link: 16.0 GT/s x16. The exposed power1_cap was 211 W at the metadata check; no power-limit change was made. +- Inference Docker: nonroot, offline, 26 GiB RAM/no swap, nine-CPU quota. Client: one CPU / 1.5 GiB. +- Other user LXCs were stopped. Model loading is outside the suite timings. + +The benchmark client did not discover the AMD GPU through its hardware detector: generated directory names can say `cpu`, GPU fields can be empty, and RAM can reflect the host rather than the cgroup limit. These are retained as emitted. The native HIP build and external hardware identification above describe the actual device; no CPU-only execution is implied. + +## Method and settings + +Each context starts a fresh server, performs two short readiness requests, and then runs the upstream suite three times per test. Expert/cache adaptation within a running suite is retained. 32K uses the standard suite; 64K adds `--long --clients 2`. See every generated report for per-request timings, actual token counts, medians, ranges, agent-cache reuse and concurrency results. + +With the configured server already running, the measured command is: + +```sh +python tools/strata_bench.py --url http://127.0.0.1:8080 --config CONFIG.json --runs 3 --out OUTPUT +# For the 65536-token profile, also add: --long --clients 2 +``` + +Each run has `settings.json` with the actual engine arguments and model hashes; `provenance.json` identifies the native build, installer revisions and available static tokenizer/pack hashes. Static expert-profile files were absent at post-run capture; their hashes are explicitly unavailable. Adapted output profiles are not substituted for missing initial hashes. Replace `` with the installation directory. Conversation cache is disabled, the reasoning budget is 512 tokens, and explicit VRAM reserve is 1024 MiB for the main profiles. Upstream draft, tail-skip and batching defaults are retained unless the recorded arguments override them. `benchmark_mtp=0` is an adapter sentinel and does not disable engine speculation. Vision was disabled. + +The Qwen UD-Q4_K_M control disables tail-skip and records its weight SHA comparison with a previous same-quant run. This is not an isolated engine comparison: runtimes, draft implementation and other defaults can differ. No historical throughput or private quality scores are included here. + +## Successful suite summaries + +Short-chat decode speed is not generation speed after a full context. Read the linked reports for sample counts and ranges. + +| Profile | Short-chat median, tok/s | 16K prompt median, tok/s | Official report | +| --- | ---: | ---: | --- | +| qwen36-ud-q3_k_m-reserve1024-ctx32768 | 75.8 | 1104.0 | [report](runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md) | +| qwen36-ud-q3_k_m-reserve1024-ctx65536 | 79.9 | 1103.4 | [report](runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md) | +| ornith-apex-i-compact-reserve1024-ctx32768 | 68.5 | 989.2 | [report](runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md) | +| ornith-apex-i-compact-reserve1024-ctx65536 | 69.2 | 991.5 | [report](runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md) | +| tiel-ud-q3_k_xl-reserve1024-ctx32768 | 71.0 | 1077.7 | [report](runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md) | +| tiel-ud-q3_k_xl-reserve1024-ctx65536 | 68.7 | 1088.3 | [report](runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md) | +| gemma4-ud-q4_k_m-reserve1024-ctx32768 | 73.5 | 988.3 | [report](runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md) | +| gemma4-ud-q4_k_m-reserve1024-ctx65536 | 73.9 | 985.3 | [report](runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md) | +| minicpmv-mix_q3gud_im-reserve1024-ctx32768 | 59.2 | 1008.2 | [report](runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md) | +| minicpmv-mix_q3gud_im-reserve1024-ctx65536 | 59.2 | 1006.1 | [report](runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md) | +| lfm25-ud-q4_k_xl-reserve1024-ctx32768 | 84.0 | 1501.2 | [report](runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md) | +| lfm25-ud-q4_k_xl-reserve1024-ctx65536 | 80.5 | 1490.1 | [report](runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md) | +| qwen36-ud-q4_k_m-control-reserve1024-ctx32768 | 80.7 | 1019.4 | [report](runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md) | +| qwen36-ud-q4_k_m-control-reserve1024-ctx65536 | 79.7 | 1021.3 | [report](runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md) | +| qwen36-ud-q3_k_m-ctx32768 | 78.2 | 1103.6 | [report](runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md) | + +## Failures and limits + +The initial Qwen profile without an explicit reserve passed its 32K suite, retained separately as `qwen36-ud-q3_k_m`. Its 64K profile encountered an AMD memory-access fault during readiness, before the official suite started; there is no official 64K result for that attempt. Readiness and diagnostic outputs are not part of this official-results submission. The 1 GiB reserve was selected after a separate diagnostic, which is not presented as an independent official result. Inspect any included failed official stage receipts and client output alongside successful runs. + +These measurements do not establish general model accuracy, production reliability, vision support or successful recall at the configured context limit. The upstream agent session is a speed/cache workload, not an external tool-execution accuracy test. No llama.cpp baseline was run for this report. + +`index.json` enumerates included official stages and the measured tool hash. Copied upstream reports and per-run JSON are unchanged; settings metadata is separately labeled. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/index.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/index.json new file mode 100644 index 000000000..0e605bf8d --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/index.json @@ -0,0 +1,111 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "runs": [ + { + "profile": "qwen36-ud-q3_k_m-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard" + }, + { + "profile": "qwen36-ud-q3_k_m-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long" + }, + { + "profile": "ornith-apex-i-compact-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard" + }, + { + "profile": "ornith-apex-i-compact-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long" + }, + { + "profile": "tiel-ud-q3_k_xl-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard" + }, + { + "profile": "tiel-ud-q3_k_xl-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long" + }, + { + "profile": "gemma4-ud-q4_k_m-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard" + }, + { + "profile": "gemma4-ud-q4_k_m-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long" + }, + { + "profile": "minicpmv-mix_q3gud_im-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard" + }, + { + "profile": "minicpmv-mix_q3gud_im-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long" + }, + { + "profile": "lfm25-ud-q4_k_xl-reserve1024-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard" + }, + { + "profile": "lfm25-ud-q4_k_xl-reserve1024-ctx65536", + "stage": "official-long", + "status": "completed", + "reports": 1, + "path": "runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long" + }, + { + "profile": "qwen36-ud-q4_k_m-control-reserve1024-ctx32768", + "stage": "official-control", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control" + }, + { + "profile": "qwen36-ud-q4_k_m-control-reserve1024-ctx65536", + "stage": "official-control", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control" + }, + { + "profile": "qwen36-ud-q3_k_m-ctx32768", + "stage": "official-standard", + "status": "completed", + "reports": 1, + "path": "runs/qwen36-ud-q3_k_m-ctx32768/official-standard" + } + ], + "tool_sha256": "58f2c493d940501401fa34b250cd4c982fcce6f5078e97725270519a3f08ee55" +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/provenance.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/provenance.json new file mode 100644 index 000000000..5fd3b691a --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/provenance.json @@ -0,0 +1,309 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "build": { + "source": "local-hip", + "backend": "hip", + "version": "0.1.45", + "archs": [ + "gfx1031" + ], + "vision": "none", + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "src": "ad576bed60fbf2b5" + }, + "installer_hf_revisions": { + "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF": "ed59f92082b1e93c0e96d60a8b11aab089b52f09", + "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF": "b22d729eae29b5796f76fb70f91aef549b9fc52c", + "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF": "5348543e0147355ac9cbcb031184a3546350988e", + "unsloth/Qwen3.8-Flash-Next-GGUF": "38bb39ee97821de2c9009abb7e93950eec396e66", + "neuralll/GLM-5.3-Flash-GSQ-RCO-3.0bit-Q4Kattn-GGUF": "7feedfeb5ee924bfbe0064c63b124728b443ddc0", + "patrickbdevaney/GLM-5.3-Flash-REAP50-GGUF": "46cb1bb887721d919e978d2f3eb18da245fecd77", + "unsloth/GLM-5.3-Flash-GGUF": "621d456e93e926e4b52f85cff5f634358c1828f9", + "mudler/Ornith-1.5-35B-A3B-APEX-MTP-GGUF": "4c326713c52d13fdb40983a2a09a46cca8e88739", + "unsloth/gemma-4-26B-A4B-it-GGUF": "c099eb48e663fd284577b04978a94ffccb261841", + "peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF-MTP": "bbe9e566f39e4fc9652ac66b71968289a03c520a", + "unsloth/Qwen3.6-35B-A3B-MTP-GGUF": "5bc3e238d916f48a861bac2f8a1990a0e9b7e98d", + "unsloth/DeepSeek-V4-Flash-0731-GGUF": "fbbb5b93fb787c21338159b0af3318bb3f4d9768", + "unsloth/LFM2.5-24B-A2B-GGUF": "faa77113e93335b63423e9e8d33395f86eaf8946", + "LiquidAI/LFM2.5-24B-A2B-GGUF": "0912d33141f77e10705345a6732bfd83f0252fdc", + "openbmb/MiniCPM-V-4.7-35B-A3B-gguf": "77f6d8cb08b2b115ce27d2e2ac1359e126921689" + }, + "profiles": [ + { + "profile": "qwen36-ud-q3_k_m-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/chat_template.jinja", + "bytes": 8057, + "sha256": "55d4931433fe502b794226ee7f4d206a6bdd436ac9f80eb7d8ebb4c639f9ea0c" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/token_type.json", + "bytes": 744960, + "sha256": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/tokenizer.json", + "bytes": 589, + "sha256": "5e89589848ca155b6d1583b8b4ec52670558d1bf368b0ece589201290898c874" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/vocab.json", + "bytes": 5737005, + "sha256": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4" + } + ], + "missing_at_capture": [ + "data/packs/qwen36-ud-q3_k_m/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "ornith-apex-i-compact-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/ornith-apex-i-compact/tokenizer/chat_template.jinja", + "bytes": 7764, + "sha256": "e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259" + }, + { + "path": "data/packs/ornith-apex-i-compact/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/ornith-apex-i-compact/tokenizer/token_type.json", + "bytes": 744960, + "sha256": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b" + }, + { + "path": "data/packs/ornith-apex-i-compact/tokenizer/tokenizer.json", + "bytes": 589, + "sha256": "0f07678e69a0f3eab2d69875163e602b8f9efb3fae836967d173104e3534b7ee" + }, + { + "path": "data/packs/ornith-apex-i-compact/tokenizer/vocab.json", + "bytes": 5737005, + "sha256": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4" + } + ], + "missing_at_capture": [ + "data/packs/ornith-apex-i-compact/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "tiel-ud-q3_k_xl-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/tiel-ud-q3_k_xl/tokenizer/chat_template.jinja", + "bytes": 30517, + "sha256": "a41d3a6495c7531e85a45b665ae99c0251a68eaef5fada66cfa283827d3f6422" + }, + { + "path": "data/packs/tiel-ud-q3_k_xl/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/tiel-ud-q3_k_xl/tokenizer/token_type.json", + "bytes": 744960, + "sha256": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b" + }, + { + "path": "data/packs/tiel-ud-q3_k_xl/tokenizer/tokenizer.json", + "bytes": 589, + "sha256": "0f07678e69a0f3eab2d69875163e602b8f9efb3fae836967d173104e3534b7ee" + }, + { + "path": "data/packs/tiel-ud-q3_k_xl/tokenizer/vocab.json", + "bytes": 5737005, + "sha256": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4" + } + ], + "missing_at_capture": [ + "data/packs/tiel-ud-q3_k_xl/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "gemma4-ud-q4_k_m-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/gemma4-ud-q4_k_m/tokenizer/chat_template.jinja", + "bytes": 18924, + "sha256": "845f1ee48e39fc942fe190da9df6a1c5db229e17a96ea08966ad1c9274e73d1b" + }, + { + "path": "data/packs/gemma4-ud-q4_k_m/tokenizer/merges.json", + "bytes": 7215682, + "sha256": "5559bcf26deec637ff8107e4379b90ae35ad59469b25e0b3bad610ae6c7c6bc4" + }, + { + "path": "data/packs/gemma4-ud-q4_k_m/tokenizer/token_type.json", + "bytes": 786432, + "sha256": "5a1960565004f4ec12210cc671c9dfd2cbe8ce30c575469128d342dc4db26cbd" + }, + { + "path": "data/packs/gemma4-ud-q4_k_m/tokenizer/tokenizer.json", + "bytes": 487, + "sha256": "869d2eeee353e0af375d6b0f1bde01350da5652d241312e71b1b387961db9b4c" + }, + { + "path": "data/packs/gemma4-ud-q4_k_m/tokenizer/vocab.json", + "bytes": 5353679, + "sha256": "680c1dc49f8c7ed3a76006aea67942f6c6359413e03098d02b87de30ef77944a" + } + ], + "missing_at_capture": [ + "data/packs/gemma4-ud-q4_k_m/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "minicpmv-mix_q3gud_im-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/minicpmv-mix_q3gud_im/tokenizer/chat_template.jinja", + "bytes": 7250, + "sha256": "c4531cacbf2a0ab65170433d7a4f371d11b7599cf96c04b192e6372432636c70" + }, + { + "path": "data/packs/minicpmv-mix_q3gud_im/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/minicpmv-mix_q3gud_im/tokenizer/token_type.json", + "bytes": 744432, + "sha256": "c4bfd0a6b37ab167f0de06b4461515a228da06aa929fba6d4dbcfed159e7a6c0" + }, + { + "path": "data/packs/minicpmv-mix_q3gud_im/tokenizer/tokenizer.json", + "bytes": 634, + "sha256": "98d1716071b00f9c6d86d126c133868044ba8e2b52ab9a839aed7af03bae5129" + }, + { + "path": "data/packs/minicpmv-mix_q3gud_im/tokenizer/vocab.json", + "bytes": 5732971, + "sha256": "9e395f65168a6e3130b33949e0e311f787115e3f90cd7826b2004b71ed8f9348" + } + ], + "missing_at_capture": [ + "data/packs/minicpmv-mix_q3gud_im/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "lfm25-ud-q4_k_xl-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/lfm25-ud-q4_k_xl/tokenizer/chat_template.jinja", + "bytes": 5443, + "sha256": "ea663864491de7ade391839479860ca95541f892f72665c73251fbd4643b1bef" + }, + { + "path": "data/packs/lfm25-ud-q4_k_xl/tokenizer/merges.txt", + "bytes": 3478678, + "sha256": "5470d099a8f5dfa5a52567c73afeea16d4a018b85ee93242d92d7025bec772a1" + }, + { + "path": "data/packs/lfm25-ud-q4_k_xl/tokenizer/token_type.json", + "bytes": 384000, + "sha256": "f8ef9886ad75cfbea968c64222f35d68fb3cd039a35ddabe261b784d0937a246" + }, + { + "path": "data/packs/lfm25-ud-q4_k_xl/tokenizer/tokenizer.json", + "bytes": 569, + "sha256": "327e7eabb6f40b32a729a6ee34ecd4593e12d8c9e73ede1b7b5be9d8f2873c33" + }, + { + "path": "data/packs/lfm25-ud-q4_k_xl/tokenizer/vocab.json", + "bytes": 2574329, + "sha256": "32db6899e2fe640a025ac47ac09da122d85d56b60a47441fe98750d71ee0a031" + } + ], + "missing_at_capture": [ + "data/packs/lfm25-ud-q4_k_xl/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "qwen36-ud-q4_k_m-control-reserve1024", + "initial_readonly_pack_files": [ + { + "path": "data/packs/qwen36-ud-q4_k_m/tokenizer/chat_template.jinja", + "bytes": 8057, + "sha256": "55d4931433fe502b794226ee7f4d206a6bdd436ac9f80eb7d8ebb4c639f9ea0c" + }, + { + "path": "data/packs/qwen36-ud-q4_k_m/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/qwen36-ud-q4_k_m/tokenizer/token_type.json", + "bytes": 744960, + "sha256": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b" + }, + { + "path": "data/packs/qwen36-ud-q4_k_m/tokenizer/tokenizer.json", + "bytes": 589, + "sha256": "5e89589848ca155b6d1583b8b4ec52670558d1bf368b0ece589201290898c874" + }, + { + "path": "data/packs/qwen36-ud-q4_k_m/tokenizer/vocab.json", + "bytes": 5737005, + "sha256": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4" + } + ], + "missing_at_capture": [ + "data/packs/qwen36-ud-q4_k_m/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + }, + { + "profile": "qwen36-ud-q3_k_m", + "initial_readonly_pack_files": [ + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/chat_template.jinja", + "bytes": 8057, + "sha256": "55d4931433fe502b794226ee7f4d206a6bdd436ac9f80eb7d8ebb4c639f9ea0c" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/merges.txt", + "bytes": 3353258, + "sha256": "4ba698529766c7a2f5052083ea10cdf2485577f5237a332ee291054eb11e3332" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/token_type.json", + "bytes": 744960, + "sha256": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/tokenizer.json", + "bytes": 589, + "sha256": "5e89589848ca155b6d1583b8b4ec52670558d1bf368b0ece589201290898c874" + }, + { + "path": "data/packs/qwen36-ud-q3_k_m/tokenizer/vocab.json", + "bytes": 5737005, + "sha256": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4" + } + ], + "missing_at_capture": [ + "data/packs/qwen36-ud-q3_k_m/expert-profile.txt" + ], + "missing_note": "No hash is available for an absent static expert profile. Do not substitute the adapted output profile or infer an initial seed hash." + } + ], + "note": "Hashes identify the static readonly installer packs used to seed fresh profiles, not the adapted output profiles after later workloads. Captured after timed work." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md new file mode 100644 index 000000000..4f48be15f --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: gemma-4-26b-a4b-ud-q4_k_m +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 48 (45-53) | 0 | 100 (100-100) | 9 | not measured | 73.5 (63.8-92.8) | 0.51 (0.45-0.54) | +| 4K prompt, fresh | 4101 (4101-4101) | 0 (0-0) | 24 cap | 3 | 930.7 (928.0-957.9) | not measured | 4.42 (4.28-4.45) | +| 16K prompt, fresh | 16390 (16390-16390) | 0 (0-0) | 24 cap | 3 | 988.3 (983.6-990.2) | not measured | 16.59 (16.56-16.92) | +| agent session, follow-up turns | 15428 (15422-15434) at the last turn | see below | up to 64 | 3 | see below | 69.9 (54.6-76.3) | 4.53 (2.24-5.32) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.1 % (68.1-68.1); earlier-turn tokens read again per session: 261 (255-267) (earlier turns that the cache did not hold; the 14806 (14806-14806) new tokens of the files are read once); first turn (nothing cached) first token: 1.01 s (1.01-1.02). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 391.1 | 1.017 | 76.6 | +| 1 | 4162 | 389 | 3773 | 887.6 | 4.266 | 67.6 | +| 2 | 5870 | 4155 | 1715 | 730.1 | 2.362 | 71.3 | +| 3 | 9362 | 5863 | 3499 | 677.9 | 5.18 | 58.6 | +| 4 | 12374 | 9355 | 3019 | 669.5 | 4.531 | 69.9 | +| 5 | 15428 | 12367 | 3061 | 675.4 | 4.552 | 58.1 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 166 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json new file mode 100644 index 000000000..a18d5971e --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json @@ -0,0 +1,885 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 165.73256945610046, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "gemma-4-26b-a4b-ud-q4_k_m", + "quant": "UD-Q4_K_M", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--draft-model", + "mtp-gemma-4-26B-A4B-it-Q8_0.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "gemma-4-26b-a4b-ud-q4_k_m", + "display_name": "Gemma 4 26B-A4B UD-Q4_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "gemma-4-26b-a4b-ud-q4_k_m", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 1686, + "expert_cache_mib": 6010, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1011, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.453, + "total_s": 1.7, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 117.4, + "prompt_tps_source": "engine", + "decode_tps": 80.2, + "decode_tps_source": "engine", + "prompt_ms": 451.4, + "decode_ms": 1247.0, + "drafts_offered": 72, + "drafts_accepted": 61, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.492, + "total_s": 2.059, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 91.9, + "prompt_tps_source": "engine", + "decode_tps": 63.8, + "decode_tps_source": "engine", + "prompt_ms": 489.6, + "decode_ms": 1567.9, + "drafts_offered": 82, + "drafts_accepted": 51, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.531, + "total_s": 1.892, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 90.7, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 529.3, + "decode_ms": 1361.3, + "drafts_offered": 78, + "drafts_accepted": 56, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.514, + "total_s": 1.591, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 103.5, + "prompt_tps_source": "engine", + "decode_tps": 92.8, + "decode_tps_source": "engine", + "prompt_ms": 512.3, + "decode_ms": 1077.2, + "drafts_offered": 82, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.493, + "total_s": 1.894, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 91.7, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 490.8, + "decode_ms": 1401.4, + "drafts_offered": 106, + "drafts_accepted": 56, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.536, + "total_s": 1.993, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 89.9, + "prompt_tps_source": "engine", + "decode_tps": 68.6, + "decode_tps_source": "engine", + "prompt_ms": 533.7, + "decode_ms": 1456.9, + "drafts_offered": 102, + "drafts_accepted": 55, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.523, + "total_s": 1.664, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 101.7, + "prompt_tps_source": "engine", + "decode_tps": 87.6, + "decode_tps_source": "engine", + "prompt_ms": 521.2, + "decode_ms": 1141.2, + "drafts_offered": 86, + "drafts_accepted": 69, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.506, + "total_s": 1.865, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 89.4, + "prompt_tps_source": "engine", + "decode_tps": 73.6, + "decode_tps_source": "engine", + "prompt_ms": 503.5, + "decode_ms": 1359.5, + "drafts_offered": 107, + "drafts_accepted": 59, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.513, + "total_s": 1.946, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 94.0, + "prompt_tps_source": "engine", + "decode_tps": 69.8, + "decode_tps_source": "engine", + "prompt_ms": 510.9, + "decode_ms": 1433.6, + "drafts_offered": 107, + "drafts_accepted": 57, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 73.5, + "min": 63.8, + "max": 92.8, + "n": 9 + }, + "ttft_s": { + "median": 0.513, + "min": 0.453, + "max": 0.536, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 48, + "min": 45, + "max": 53, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.452, + "total_s": 4.842, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 930.7, + "prompt_tps_source": "engine", + "decode_tps": 61.6, + "decode_tps_source": "engine", + "prompt_ms": 4406.5, + "decode_ms": 389.9, + "drafts_offered": 24, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.284, + "total_s": 4.684, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 957.9, + "prompt_tps_source": "engine", + "decode_tps": 60.0, + "decode_tps_source": "engine", + "prompt_ms": 4281.1, + "decode_ms": 399.8, + "drafts_offered": 26, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.422, + "total_s": 4.803, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 928.0, + "prompt_tps_source": "engine", + "decode_tps": 63.1, + "decode_tps_source": "engine", + "prompt_ms": 4419.2, + "decode_ms": 380.5, + "drafts_offered": 25, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 930.7, + "min": 928.0, + "max": 957.9, + "n": 3 + }, + "ttft_s": { + "median": 4.422, + "min": 4.284, + "max": 4.452, + "n": 3 + }, + "prompt_tokens": { + "median": 4101, + "min": 4101, + "max": 4101, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.918, + "total_s": 17.333, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 983.6, + "prompt_tps_source": "engine", + "decode_tps": 57.7, + "decode_tps_source": "engine", + "prompt_ms": 16663.2, + "decode_ms": 415.6, + "drafts_offered": 25, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.559, + "total_s": 16.944, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 990.2, + "prompt_tps_source": "engine", + "decode_tps": 62.3, + "decode_tps_source": "engine", + "prompt_ms": 16551.5, + "decode_ms": 385.2, + "drafts_offered": 24, + "drafts_accepted": 14, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.592, + "total_s": 16.968, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 988.3, + "prompt_tps_source": "engine", + "decode_tps": 63.8, + "decode_tps_source": "engine", + "prompt_ms": 16583.8, + "decode_ms": 376.2, + "drafts_offered": 23, + "drafts_accepted": 14, + "run": 2 + } + ], + "prompt_tps": { + "median": 988.3, + "min": 983.6, + "max": 990.2, + "n": 3 + }, + "ttft_s": { + "median": 16.592, + "min": 16.559, + "max": 16.918, + "n": 3 + }, + "prompt_tokens": { + "median": 16390, + "min": 16390, + "max": 16390, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.017, + "total_s": 1.304, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 391.1, + "prompt_tps_source": "engine", + "decode_tps": 76.6, + "decode_tps_source": "engine", + "prompt_ms": 1012.5, + "decode_ms": 287.3, + "drafts_offered": 21, + "drafts_accepted": 15, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.266, + "total_s": 4.976, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 48, + "prompt_tps": 887.6, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 4250.6, + "decode_ms": 710.1, + "drafts_offered": 49, + "drafts_accepted": 29, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.362, + "total_s": 3.119, + "prompt_tokens": 5870, + "reused_tokens": 4155, + "read_tokens": 1715, + "generated_tokens": 54, + "prompt_tps": 730.1, + "prompt_tps_source": "engine", + "decode_tps": 71.3, + "decode_tps_source": "engine", + "prompt_ms": 2348.9, + "decode_ms": 757.2, + "drafts_offered": 50, + "drafts_accepted": 34, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.18, + "total_s": 5.998, + "prompt_tokens": 9362, + "reused_tokens": 5863, + "read_tokens": 3499, + "generated_tokens": 48, + "prompt_tps": 677.9, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 5161.7, + "decode_ms": 819.2, + "drafts_offered": 52, + "drafts_accepted": 25, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.531, + "total_s": 5.303, + "prompt_tokens": 12374, + "reused_tokens": 9355, + "read_tokens": 3019, + "generated_tokens": 54, + "prompt_tps": 669.5, + "prompt_tps_source": "engine", + "decode_tps": 69.9, + "decode_tps_source": "engine", + "prompt_ms": 4509.4, + "decode_ms": 772.2, + "drafts_offered": 50, + "drafts_accepted": 34, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.552, + "total_s": 5.654, + "prompt_tokens": 15428, + "reused_tokens": 12367, + "read_tokens": 3061, + "generated_tokens": 64, + "prompt_tps": 675.4, + "prompt_tps_source": "engine", + "decode_tps": 58.1, + "decode_tps_source": "engine", + "prompt_ms": 4532.4, + "decode_ms": 1102.3, + "drafts_offered": 67, + "drafts_accepted": 35, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.006, + "total_s": 1.26, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 394.5, + "prompt_tps_source": "engine", + "decode_tps": 86.5, + "decode_tps_source": "engine", + "prompt_ms": 1003.9, + "decode_ms": 254.2, + "drafts_offered": 21, + "drafts_accepted": 15, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.297, + "total_s": 4.939, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 48, + "prompt_tps": 878.7, + "prompt_tps_source": "engine", + "decode_tps": 74.8, + "decode_tps_source": "engine", + "prompt_ms": 4293.6, + "decode_ms": 641.7, + "drafts_offered": 46, + "drafts_accepted": 30, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.244, + "total_s": 2.978, + "prompt_tokens": 5870, + "reused_tokens": 4155, + "read_tokens": 1715, + "generated_tokens": 56, + "prompt_tps": 765.8, + "prompt_tps_source": "engine", + "decode_tps": 76.3, + "decode_tps_source": "engine", + "prompt_ms": 2239.5, + "decode_ms": 734.3, + "drafts_offered": 53, + "drafts_accepted": 36, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.274, + "total_s": 6.171, + "prompt_tokens": 9364, + "reused_tokens": 5863, + "read_tokens": 3501, + "generated_tokens": 56, + "prompt_tps": 664.6, + "prompt_tps_source": "engine", + "decode_tps": 62.5, + "decode_tps_source": "engine", + "prompt_ms": 5267.7, + "decode_ms": 896.5, + "drafts_offered": 60, + "drafts_accepted": 32, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.517, + "total_s": 5.219, + "prompt_tokens": 12384, + "reused_tokens": 9357, + "read_tokens": 3027, + "generated_tokens": 50, + "prompt_tps": 671.3, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 4509.4, + "decode_ms": 701.8, + "drafts_offered": 49, + "drafts_accepted": 32, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.585, + "total_s": 5.626, + "prompt_tokens": 15434, + "reused_tokens": 12377, + "read_tokens": 3057, + "generated_tokens": 64, + "prompt_tps": 668.0, + "prompt_tps_source": "engine", + "decode_tps": 61.4, + "decode_tps_source": "engine", + "prompt_ms": 4576.2, + "decode_ms": 1042.1, + "drafts_offered": 70, + "drafts_accepted": 38, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.014, + "total_s": 1.285, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 391.4, + "prompt_tps_source": "engine", + "decode_tps": 81.1, + "decode_tps_source": "engine", + "prompt_ms": 1011.7, + "decode_ms": 271.3, + "drafts_offered": 21, + "drafts_accepted": 14, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.308, + "total_s": 4.962, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 47, + "prompt_tps": 876.5, + "prompt_tps_source": "engine", + "decode_tps": 71.8, + "decode_tps_source": "engine", + "prompt_ms": 4304.7, + "decode_ms": 654.3, + "drafts_offered": 48, + "drafts_accepted": 29, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.286, + "total_s": 3.023, + "prompt_tokens": 5869, + "reused_tokens": 4155, + "read_tokens": 1714, + "generated_tokens": 54, + "prompt_tps": 751.5, + "prompt_tps_source": "engine", + "decode_tps": 73.2, + "decode_tps_source": "engine", + "prompt_ms": 2280.8, + "decode_ms": 738.1, + "drafts_offered": 52, + "drafts_accepted": 34, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.319, + "total_s": 6.18, + "prompt_tokens": 9361, + "reused_tokens": 5862, + "read_tokens": 3499, + "generated_tokens": 47, + "prompt_tps": 658.5, + "prompt_tps_source": "engine", + "decode_tps": 54.6, + "decode_tps_source": "engine", + "prompt_ms": 5313.4, + "decode_ms": 861.2, + "drafts_offered": 55, + "drafts_accepted": 23, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.629, + "total_s": 5.331, + "prompt_tokens": 12372, + "reused_tokens": 9354, + "read_tokens": 3018, + "generated_tokens": 50, + "prompt_tps": 653.1, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 4621.1, + "decode_ms": 701.8, + "drafts_offered": 49, + "drafts_accepted": 32, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.556, + "total_s": 5.539, + "prompt_tokens": 15422, + "reused_tokens": 12365, + "read_tokens": 3057, + "generated_tokens": 63, + "prompt_tps": 672.2, + "prompt_tps_source": "engine", + "decode_tps": 64.0, + "decode_tps_source": "engine", + "prompt_ms": 4547.7, + "decode_ms": 984.0, + "drafts_offered": 63, + "drafts_accepted": 36, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.076, + "min": 68.075, + "max": 68.082, + "n": 3 + }, + "reread_tokens": { + "median": 261, + "min": 255, + "max": 267, + "n": 3 + }, + "new_tokens": { + "median": 14806, + "min": 14806, + "max": 14806, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.531, + "min": 2.244, + "max": 5.319, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15428, + "min": 15422, + "max": 15434, + "n": 3 + }, + "turn_decode_tps": { + "median": 69.9, + "min": 54.6, + "max": 76.3, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.014, + "min": 1.006, + "max": 1.017, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..2802de5ed --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.00 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 80.2 tok/s, first token 0.453 s, 100 tokens + run 1 q2: decode 63.8 tok/s, first token 0.492 s, 100 tokens + run 1 q3: decode 73.5 tok/s, first token 0.531 s, 100 tokens + run 2 q1: decode 92.8 tok/s, first token 0.514 s, 100 tokens + run 2 q2: decode 71.4 tok/s, first token 0.493 s, 100 tokens + run 2 q3: decode 68.6 tok/s, first token 0.536 s, 100 tokens + run 3 q1: decode 87.6 tok/s, first token 0.523 s, 100 tokens + run 3 q2: decode 73.6 tok/s, first token 0.506 s, 100 tokens + run 3 q3: decode 69.8 tok/s, first token 0.513 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4101 tokens, prompt 930.7 tok/s (0 reused), first token 4.452 s + run 2: 4101 tokens, prompt 957.9 tok/s (0 reused), first token 4.284 s + run 3: 4101 tokens, prompt 928.0 tok/s (0 reused), first token 4.422 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16390 tokens, prompt 983.6 tok/s (0 reused), first token 16.918 s + run 2: 16390 tokens, prompt 990.2 tok/s (0 reused), first token 16.559 s + run 3: 16390 tokens, prompt 988.3 tok/s (0 reused), first token 16.592 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 396, reused 0, read 396, first token 1.017 s + run 1 turn 1: prompt 4162, reused 389, read 3773, first token 4.266 s + run 1 turn 2: prompt 5870, reused 4155, read 1715, first token 2.362 s + run 1 turn 3: prompt 9362, reused 5863, read 3499, first token 5.18 s + run 1 turn 4: prompt 12374, reused 9355, read 3019, first token 4.531 s + run 1 turn 5: prompt 15428, reused 12367, read 3061, first token 4.552 s + run 2 turn 0: prompt 396, reused 0, read 396, first token 1.006 s + run 2 turn 1: prompt 4162, reused 389, read 3773, first token 4.297 s + run 2 turn 2: prompt 5870, reused 4155, read 1715, first token 2.244 s + run 2 turn 3: prompt 9364, reused 5863, read 3501, first token 5.274 s + run 2 turn 4: prompt 12384, reused 9357, read 3027, first token 4.517 s + run 2 turn 5: prompt 15434, reused 12377, read 3057, first token 4.585 s + run 3 turn 0: prompt 396, reused 0, read 396, first token 1.014 s + run 3 turn 1: prompt 4162, reused 389, read 3773, first token 4.308 s + run 3 turn 2: prompt 5869, reused 4155, read 1714, first token 2.286 s + run 3 turn 3: prompt 9361, reused 5862, read 3499, first token 5.319 s + run 3 turn 4: prompt 12372, reused 9354, read 3018, first token 4.629 s + run 3 turn 5: prompt 15422, reused 12365, read 3057, first token 4.556 s + +Results (median, range over the runs): + chat decode tok/s 73.50 (63.80 - 92.80, n=9) + chat first token (s) 0.51 (0.45 - 0.54, n=9) + 4K prompt tok/s 930.70 (928.00 - 957.90, n=3) + 4K first token (s) 4.42 (4.28 - 4.45, n=3) + 16K prompt tok/s 988.30 (983.60 - 990.20, n=3) + 16K first token (s) 16.59 (16.56 - 16.92, n=3) + agent cache hit % 68.08 (68.08 - 68.08, n=3) + agent re-read tokens/session 261.00 (255.00 - 267.00, n=3) + agent turn first token (s) 4.53 (2.24 - 5.32, n=15) + agent decode tok/s 69.90 (54.60 - 76.30, n=15) + +Report: /opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..a4abe1365 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,55 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/gemma4-UD-Q4_K_M/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/gemma4-ud-q4_k_m-reserve1024-ctx32768/out/expert-profile.txt", + "--draft-model", + "/data/models/gemma4-UD-Q4_K_M/mtp-gemma-4-26B-A4B-it-Q8_0.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/gemma4-ud-q4_k_m/tokenizer", + "model_name": "gemma-4-26b-a4b-ud-q4_k_m", + "display_name": "Gemma 4 26B-A4B UD-Q4_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 16947541728, + "sha256": "f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f", + "filename": "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf" + }, + { + "bytes": 461766816, + "sha256": "6326fb9f5e487aa8dcdd313a091e3c67724cb2a666ec3b7d2895b5b26d93ed1b", + "filename": "mtp-gemma-4-26B-A4B-it-Q8_0.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..08f8747f5 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 165.9723191719968 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md new file mode 100644 index 000000000..053173904 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: gemma-4-26b-a4b-ud-q4_k_m +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 48 (45-53) | 0 | 100 (100-100) | 9 | not measured | 73.9 (59.0-91.8) | 0.56 (0.45-0.57) | +| 4K prompt, fresh | 4101 (4101-4101) | 0 (0-0) | 24 cap | 3 | 933.8 (932.6-936.0) | not measured | 4.40 (4.38-4.44) | +| 16K prompt, fresh | 16390 (16390-16390) | 0 (0-0) | 24 cap | 3 | 985.3 (976.0-985.5) | not measured | 16.64 (16.64-17.05) | +| 32K prompt, fresh | 32774 (32774-32774) | 0 (0-0) | 24 cap | 3 | 819.1 (815.2-821.1) | not measured | 40.02 (39.93-40.85) | +| agent session, follow-up turns | 15420 (15418-15420) at the last turn | see below | up to 64 | 3 | see below | 71.5 (55.4-75.7) | 4.64 (2.37-5.39) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.1 % (68.1-68.1); earlier-turn tokens read again per session: 253 (251-253) (earlier turns that the cache did not hold; the 14806 (14806-14806) new tokens of the files are read once); first turn (nothing cached) first token: 1.03 s (1.01-1.10). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 362.2 | 1.098 | 77.5 | +| 1 | 4162 | 389 | 3773 | 864.9 | 4.378 | 71.5 | +| 2 | 5869 | 4155 | 1714 | 726.6 | 2.372 | 73.0 | +| 3 | 9358 | 5862 | 3496 | 650.2 | 5.395 | 58.5 | +| 4 | 12370 | 9351 | 3019 | 656.0 | 4.625 | 71.4 | +| 5 | 15420 | 12363 | 3057 | 661.8 | 4.639 | 55.4 | + +### Concurrency: 2 clients + +Total 49.8 tok/s (48.0-51.4); first token p50 1.878 s, p95 3.402 s; request latency p50 3.952 s, p95 5.289 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 305 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json new file mode 100644 index 000000000..340de18d6 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/results.json @@ -0,0 +1,1034 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 305.4844799041748, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "gemma-4-26b-a4b-ud-q4_k_m", + "quant": "UD-Q4_K_M", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--draft-model", + "mtp-gemma-4-26B-A4B-it-Q8_0.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "gemma-4-26b-a4b-ud-q4_k_m", + "display_name": "Gemma 4 26B-A4B UD-Q4_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --draft-model mtp-gemma-4-26B-A4B-it-Q8_0.gguf --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "gemma-4-26b-a4b-ud-q4_k_m", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 1507, + "expert_cache_mib": 5372, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1012, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.454, + "total_s": 1.628, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 117.2, + "prompt_tps_source": "engine", + "decode_tps": 85.2, + "decode_tps_source": "engine", + "prompt_ms": 452.3, + "decode_ms": 1173.6, + "drafts_offered": 71, + "drafts_accepted": 65, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.526, + "total_s": 2.019, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 85.9, + "prompt_tps_source": "engine", + "decode_tps": 66.9, + "decode_tps_source": "engine", + "prompt_ms": 523.8, + "decode_ms": 1493.8, + "drafts_offered": 91, + "drafts_accepted": 60, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.555, + "total_s": 2.248, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 86.8, + "prompt_tps_source": "engine", + "decode_tps": 59.0, + "decode_tps_source": "engine", + "prompt_ms": 552.9, + "decode_ms": 1693.5, + "drafts_offered": 108, + "drafts_accepted": 52, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.556, + "total_s": 1.645, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 95.7, + "prompt_tps_source": "engine", + "decode_tps": 91.8, + "decode_tps_source": "engine", + "prompt_ms": 554.0, + "decode_ms": 1089.1, + "drafts_offered": 79, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.545, + "total_s": 1.846, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 82.8, + "prompt_tps_source": "engine", + "decode_tps": 76.9, + "decode_tps_source": "engine", + "prompt_ms": 543.2, + "decode_ms": 1301.1, + "drafts_offered": 94, + "drafts_accepted": 60, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.566, + "total_s": 2.018, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 85.2, + "prompt_tps_source": "engine", + "decode_tps": 68.8, + "decode_tps_source": "engine", + "prompt_ms": 563.6, + "decode_ms": 1453.2, + "drafts_offered": 107, + "drafts_accepted": 58, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.571, + "total_s": 1.699, + "prompt_tokens": 53, + "reused_tokens": 0, + "read_tokens": 53, + "generated_tokens": 100, + "prompt_tps": 93.1, + "prompt_tps_source": "engine", + "decode_tps": 88.6, + "decode_tps_source": "engine", + "prompt_ms": 569.1, + "decode_ms": 1128.5, + "drafts_offered": 80, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.527, + "total_s": 1.88, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 85.8, + "prompt_tps_source": "engine", + "decode_tps": 73.9, + "decode_tps_source": "engine", + "prompt_ms": 524.6, + "decode_ms": 1353.5, + "drafts_offered": 92, + "drafts_accepted": 55, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.558, + "total_s": 1.95, + "prompt_tokens": 48, + "reused_tokens": 0, + "read_tokens": 48, + "generated_tokens": 100, + "prompt_tps": 86.3, + "prompt_tps_source": "engine", + "decode_tps": 71.8, + "decode_tps_source": "engine", + "prompt_ms": 556.2, + "decode_ms": 1392.3, + "drafts_offered": 95, + "drafts_accepted": 57, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 73.9, + "min": 59.0, + "max": 91.8, + "n": 9 + }, + "ttft_s": { + "median": 0.555, + "min": 0.454, + "max": 0.571, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 48, + "min": 45, + "max": 53, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.439, + "total_s": 4.845, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 933.8, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 4391.8, + "decode_ms": 406.4, + "drafts_offered": 24, + "drafts_accepted": 14, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.385, + "total_s": 4.843, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 936.0, + "prompt_tps_source": "engine", + "decode_tps": 52.4, + "decode_tps_source": "engine", + "prompt_ms": 4381.4, + "decode_ms": 457.9, + "drafts_offered": 28, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.401, + "total_s": 4.799, + "prompt_tokens": 4101, + "reused_tokens": 0, + "read_tokens": 4101, + "generated_tokens": 24, + "prompt_tps": 932.6, + "prompt_tps_source": "engine", + "decode_tps": 60.2, + "decode_tps_source": "engine", + "prompt_ms": 4397.3, + "decode_ms": 398.9, + "drafts_offered": 28, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 933.8, + "min": 932.6, + "max": 936.0, + "n": 3 + }, + "ttft_s": { + "median": 4.401, + "min": 4.385, + "max": 4.439, + "n": 3 + }, + "prompt_tokens": { + "median": 4101, + "min": 4101, + "max": 4101, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 17.048, + "total_s": 17.399, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 976.0, + "prompt_tps_source": "engine", + "decode_tps": 68.3, + "decode_tps_source": "engine", + "prompt_ms": 16793.0, + "decode_ms": 351.3, + "drafts_offered": 22, + "drafts_accepted": 15, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.643, + "total_s": 17.109, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 985.3, + "prompt_tps_source": "engine", + "decode_tps": 51.5, + "decode_tps_source": "engine", + "prompt_ms": 16634.8, + "decode_ms": 466.1, + "drafts_offered": 29, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.639, + "total_s": 17.041, + "prompt_tokens": 16390, + "reused_tokens": 0, + "read_tokens": 16390, + "generated_tokens": 24, + "prompt_tps": 985.5, + "prompt_tps_source": "engine", + "decode_tps": 59.7, + "decode_tps_source": "engine", + "prompt_ms": 16631.3, + "decode_ms": 401.8, + "drafts_offered": 26, + "drafts_accepted": 14, + "run": 2 + } + ], + "prompt_tps": { + "median": 985.3, + "min": 976.0, + "max": 985.5, + "n": 3 + }, + "ttft_s": { + "median": 16.643, + "min": 16.639, + "max": 17.048, + "n": 3 + }, + "prompt_tokens": { + "median": 16390, + "min": 16390, + "max": 16390, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 40.853, + "total_s": 41.377, + "prompt_tokens": 32774, + "reused_tokens": 0, + "read_tokens": 32774, + "generated_tokens": 24, + "prompt_tps": 815.2, + "prompt_tps_source": "engine", + "decode_tps": 45.7, + "decode_tps_source": "engine", + "prompt_ms": 40203.9, + "decode_ms": 524.8, + "drafts_offered": 27, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 39.928, + "total_s": 40.39, + "prompt_tokens": 32774, + "reused_tokens": 0, + "read_tokens": 32774, + "generated_tokens": 24, + "prompt_tps": 821.1, + "prompt_tps_source": "engine", + "decode_tps": 51.8, + "decode_tps_source": "engine", + "prompt_ms": 39913.3, + "decode_ms": 463.3, + "drafts_offered": 21, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 40.025, + "total_s": 40.422, + "prompt_tokens": 32774, + "reused_tokens": 0, + "read_tokens": 32774, + "generated_tokens": 24, + "prompt_tps": 819.1, + "prompt_tps_source": "engine", + "decode_tps": 60.2, + "decode_tps_source": "engine", + "prompt_ms": 40010.1, + "decode_ms": 398.5, + "drafts_offered": 22, + "drafts_accepted": 14, + "run": 2 + } + ], + "prompt_tps": { + "median": 819.1, + "min": 815.2, + "max": 821.1, + "n": 3 + }, + "ttft_s": { + "median": 40.025, + "min": 39.928, + "max": 40.853, + "n": 3 + }, + "prompt_tokens": { + "median": 32774, + "min": 32774, + "max": 32774, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.098, + "total_s": 1.382, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 362.2, + "prompt_tps_source": "engine", + "decode_tps": 77.5, + "decode_tps_source": "engine", + "prompt_ms": 1093.2, + "decode_ms": 284.0, + "drafts_offered": 21, + "drafts_accepted": 15, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.378, + "total_s": 5.035, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 47, + "prompt_tps": 864.9, + "prompt_tps_source": "engine", + "decode_tps": 71.5, + "decode_tps_source": "engine", + "prompt_ms": 4362.4, + "decode_ms": 657.8, + "drafts_offered": 43, + "drafts_accepted": 30, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.372, + "total_s": 3.07, + "prompt_tokens": 5869, + "reused_tokens": 4155, + "read_tokens": 1714, + "generated_tokens": 51, + "prompt_tps": 726.6, + "prompt_tps_source": "engine", + "decode_tps": 73.0, + "decode_tps_source": "engine", + "prompt_ms": 2358.9, + "decode_ms": 698.8, + "drafts_offered": 49, + "drafts_accepted": 33, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.395, + "total_s": 6.214, + "prompt_tokens": 9358, + "reused_tokens": 5862, + "read_tokens": 3496, + "generated_tokens": 48, + "prompt_tps": 650.2, + "prompt_tps_source": "engine", + "decode_tps": 58.5, + "decode_tps_source": "engine", + "prompt_ms": 5376.6, + "decode_ms": 819.9, + "drafts_offered": 53, + "drafts_accepted": 25, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.625, + "total_s": 5.324, + "prompt_tokens": 12370, + "reused_tokens": 9351, + "read_tokens": 3019, + "generated_tokens": 50, + "prompt_tps": 656.0, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 4602.0, + "decode_ms": 699.8, + "drafts_offered": 48, + "drafts_accepted": 32, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.639, + "total_s": 5.794, + "prompt_tokens": 15420, + "reused_tokens": 12363, + "read_tokens": 3057, + "generated_tokens": 64, + "prompt_tps": 661.8, + "prompt_tps_source": "engine", + "decode_tps": 55.4, + "decode_tps_source": "engine", + "prompt_ms": 4619.3, + "decode_ms": 1155.5, + "drafts_offered": 72, + "drafts_accepted": 35, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.032, + "total_s": 1.291, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 384.5, + "prompt_tps_source": "engine", + "decode_tps": 85.0, + "decode_tps_source": "engine", + "prompt_ms": 1029.9, + "decode_ms": 258.9, + "drafts_offered": 21, + "drafts_accepted": 15, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.41, + "total_s": 5.05, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 47, + "prompt_tps": 856.2, + "prompt_tps_source": "engine", + "decode_tps": 73.4, + "decode_tps_source": "engine", + "prompt_ms": 4406.7, + "decode_ms": 640.3, + "drafts_offered": 49, + "drafts_accepted": 30, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.39, + "total_s": 3.108, + "prompt_tokens": 5869, + "reused_tokens": 4155, + "read_tokens": 1714, + "generated_tokens": 54, + "prompt_tps": 718.5, + "prompt_tps_source": "engine", + "decode_tps": 75.2, + "decode_tps_source": "engine", + "prompt_ms": 2385.4, + "decode_ms": 717.9, + "drafts_offered": 51, + "drafts_accepted": 34, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.323, + "total_s": 6.165, + "prompt_tokens": 9361, + "reused_tokens": 5862, + "read_tokens": 3499, + "generated_tokens": 47, + "prompt_tps": 658.2, + "prompt_tps_source": "engine", + "decode_tps": 55.7, + "decode_tps_source": "engine", + "prompt_ms": 5316.4, + "decode_ms": 843.2, + "drafts_offered": 60, + "drafts_accepted": 25, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.662, + "total_s": 5.329, + "prompt_tokens": 12372, + "reused_tokens": 9354, + "read_tokens": 3018, + "generated_tokens": 48, + "prompt_tps": 648.4, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 4654.4, + "decode_ms": 667.1, + "drafts_offered": 43, + "drafts_accepted": 31, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.715, + "total_s": 5.687, + "prompt_tokens": 15420, + "reused_tokens": 12365, + "read_tokens": 3055, + "generated_tokens": 63, + "prompt_tps": 649.1, + "prompt_tps_source": "engine", + "decode_tps": 64.8, + "decode_tps_source": "engine", + "prompt_ms": 4706.4, + "decode_ms": 972.2, + "drafts_offered": 60, + "drafts_accepted": 36, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.008, + "total_s": 1.265, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 22, + "prompt_tps": 394.0, + "prompt_tps_source": "engine", + "decode_tps": 85.3, + "decode_tps_source": "engine", + "prompt_ms": 1005.2, + "decode_ms": 257.9, + "drafts_offered": 21, + "drafts_accepted": 15, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.448, + "total_s": 5.07, + "prompt_tokens": 4162, + "reused_tokens": 389, + "read_tokens": 3773, + "generated_tokens": 47, + "prompt_tps": 848.9, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 4444.6, + "decode_ms": 622.2, + "drafts_offered": 43, + "drafts_accepted": 29, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.39, + "total_s": 3.069, + "prompt_tokens": 5869, + "reused_tokens": 4155, + "read_tokens": 1714, + "generated_tokens": 51, + "prompt_tps": 718.6, + "prompt_tps_source": "engine", + "decode_tps": 75.0, + "decode_tps_source": "engine", + "prompt_ms": 2385.2, + "decode_ms": 679.9, + "drafts_offered": 48, + "drafts_accepted": 32, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.257, + "total_s": 6.086, + "prompt_tokens": 9358, + "reused_tokens": 5862, + "read_tokens": 3496, + "generated_tokens": 48, + "prompt_tps": 665.7, + "prompt_tps_source": "engine", + "decode_tps": 57.9, + "decode_tps_source": "engine", + "prompt_ms": 5251.6, + "decode_ms": 828.8, + "drafts_offered": 53, + "drafts_accepted": 25, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.636, + "total_s": 5.269, + "prompt_tokens": 12370, + "reused_tokens": 9351, + "read_tokens": 3019, + "generated_tokens": 48, + "prompt_tps": 652.3, + "prompt_tps_source": "engine", + "decode_tps": 75.7, + "decode_tps_source": "engine", + "prompt_ms": 4628.3, + "decode_ms": 633.8, + "drafts_offered": 41, + "drafts_accepted": 31, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.674, + "total_s": 5.665, + "prompt_tokens": 15418, + "reused_tokens": 12363, + "read_tokens": 3055, + "generated_tokens": 64, + "prompt_tps": 654.8, + "prompt_tps_source": "engine", + "decode_tps": 64.6, + "decode_tps_source": "engine", + "prompt_ms": 4665.2, + "decode_ms": 991.4, + "drafts_offered": 65, + "drafts_accepted": 39, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.084, + "min": 68.081, + "max": 68.085, + "n": 3 + }, + "reread_tokens": { + "median": 253, + "min": 251, + "max": 253, + "n": 3 + }, + "new_tokens": { + "median": 14806, + "min": 14806, + "max": 14806, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.636, + "min": 2.372, + "max": 5.395, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15420, + "min": 15418, + "max": 15420, + "n": 3 + }, + "turn_decode_tps": { + "median": 71.5, + "min": 55.4, + "max": 75.7, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.032, + "min": 1.008, + "max": 1.098, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 5.34, + "tokens": 256, + "total_tps": 47.96, + "first_s": [ + 0.730472268012818, + 3.4962051709881052 + ], + "latency_s": [ + 2.9259578460187186, + 5.336894079984631 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 5.14, + "tokens": 256, + "total_tps": 49.76, + "first_s": [ + 0.5798614739906043, + 3.1203454920032527 + ], + "latency_s": [ + 2.4961857850139495, + 5.1439942459983286 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 4.98, + "tokens": 256, + "total_tps": 51.43, + "first_s": [ + 0.5972120610240381, + 3.024710869998671 + ], + "latency_s": [ + 2.418796102021588, + 4.977674977009883 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 49.76, + "min": 47.96, + "max": 51.43, + "n": 3 + }, + "ttft_p50_s": 1.878, + "ttft_p95_s": 3.402, + "latency_p50_s": 3.952, + "latency_p95_s": 5.289, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..81df76dd8 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.00 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 85.2 tok/s, first token 0.454 s, 100 tokens + run 1 q2: decode 66.9 tok/s, first token 0.526 s, 100 tokens + run 1 q3: decode 59.0 tok/s, first token 0.555 s, 100 tokens + run 2 q1: decode 91.8 tok/s, first token 0.556 s, 100 tokens + run 2 q2: decode 76.9 tok/s, first token 0.545 s, 100 tokens + run 2 q3: decode 68.8 tok/s, first token 0.566 s, 100 tokens + run 3 q1: decode 88.6 tok/s, first token 0.571 s, 100 tokens + run 3 q2: decode 73.9 tok/s, first token 0.527 s, 100 tokens + run 3 q3: decode 71.8 tok/s, first token 0.558 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4101 tokens, prompt 933.8 tok/s (0 reused), first token 4.439 s + run 2: 4101 tokens, prompt 936.0 tok/s (0 reused), first token 4.385 s + run 3: 4101 tokens, prompt 932.6 tok/s (0 reused), first token 4.401 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16390 tokens, prompt 976.0 tok/s (0 reused), first token 17.048 s + run 2: 16390 tokens, prompt 985.3 tok/s (0 reused), first token 16.643 s + run 3: 16390 tokens, prompt 985.5 tok/s (0 reused), first token 16.639 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 396, reused 0, read 396, first token 1.098 s + run 1 turn 1: prompt 4162, reused 389, read 3773, first token 4.378 s + run 1 turn 2: prompt 5869, reused 4155, read 1714, first token 2.372 s + run 1 turn 3: prompt 9358, reused 5862, read 3496, first token 5.395 s + run 1 turn 4: prompt 12370, reused 9351, read 3019, first token 4.625 s + run 1 turn 5: prompt 15420, reused 12363, read 3057, first token 4.639 s + run 2 turn 0: prompt 396, reused 0, read 396, first token 1.032 s + run 2 turn 1: prompt 4162, reused 389, read 3773, first token 4.41 s + run 2 turn 2: prompt 5869, reused 4155, read 1714, first token 2.39 s + run 2 turn 3: prompt 9361, reused 5862, read 3499, first token 5.323 s + run 2 turn 4: prompt 12372, reused 9354, read 3018, first token 4.662 s + run 2 turn 5: prompt 15420, reused 12365, read 3055, first token 4.715 s + run 3 turn 0: prompt 396, reused 0, read 396, first token 1.008 s + run 3 turn 1: prompt 4162, reused 389, read 3773, first token 4.448 s + run 3 turn 2: prompt 5869, reused 4155, read 1714, first token 2.39 s + run 3 turn 3: prompt 9358, reused 5862, read 3496, first token 5.257 s + run 3 turn 4: prompt 12370, reused 9351, read 3019, first token 4.636 s + run 3 turn 5: prompt 15418, reused 12363, read 3055, first token 4.674 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 5.3 s = 48.0 tok/s total + round 2: 256 tokens in 5.1 s = 49.8 tok/s total + round 3: 256 tokens in 5.0 s = 51.4 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32774 tokens, prompt 815.2 tok/s (0 reused), first token 40.853 s + run 2: 32774 tokens, prompt 821.1 tok/s (0 reused), first token 39.928 s + run 3: 32774 tokens, prompt 819.1 tok/s (0 reused), first token 40.025 s + +Results (median, range over the runs): + chat decode tok/s 73.90 (59.00 - 91.80, n=9) + chat first token (s) 0.56 (0.45 - 0.57, n=9) + 4K prompt tok/s 933.80 (932.60 - 936.00, n=3) + 4K first token (s) 4.40 (4.38 - 4.44, n=3) + 16K prompt tok/s 985.30 (976.00 - 985.50, n=3) + 16K first token (s) 16.64 (16.64 - 17.05, n=3) + 32K prompt tok/s 819.10 (815.20 - 821.10, n=3) + 32K first token (s) 40.02 (39.93 - 40.85, n=3) + agent cache hit % 68.08 (68.08 - 68.08, n=3) + agent re-read tokens/session 253.00 (251.00 - 253.00, n=3) + agent turn first token (s) 4.64 (2.37 - 5.39, n=15) + agent decode tok/s 71.50 (55.40 - 75.70, n=15) + 2 clients total tok/s 49.76 (47.96 - 51.43, n=3) + +Report: /opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-gemma-4-26b-a4b-ud-q4-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..4dc504eb8 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,55 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/gemma4-UD-Q4_K_M/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/gemma4-ud-q4_k_m-reserve1024-ctx65536/out/expert-profile.txt", + "--draft-model", + "/data/models/gemma4-UD-Q4_K_M/mtp-gemma-4-26B-A4B-it-Q8_0.gguf", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/gemma4-ud-q4_k_m/tokenizer", + "model_name": "gemma-4-26b-a4b-ud-q4_k_m", + "display_name": "Gemma 4 26B-A4B UD-Q4_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 16947541728, + "sha256": "f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f", + "filename": "gemma-4-26B-A4B-it-UD-Q4_K_M.gguf" + }, + { + "bytes": 461766816, + "sha256": "6326fb9f5e487aa8dcdd313a091e3c67724cb2a666ec3b7d2895b5b26d93ed1b", + "filename": "mtp-gemma-4-26B-A4B-it-Q8_0.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..9491d5f4b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/gemma4-ud-q4_k_m-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 305.69801035901764 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md new file mode 100644 index 000000000..e504a0c22 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: lfm2.5-24b-a2b-ud-q4_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --prompt-batch 2048 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 39 (36-45) | 0 | 100 (100-100) | 9 | not measured | 84.0 (74.2-91.4) | 0.19 (0.17-0.20) | +| 4K prompt, fresh | 4126 (4126-4126) | 0 (0-0) | 24 cap | 3 | 1638.8 (1613.9-1664.2) | not measured | 2.54 (2.50-2.57) | +| 16K prompt, fresh | 16490 (16490-16490) | 0 (0-0) | 24 cap | 3 | 1501.2 (1484.8-1503.5) | not measured | 11.01 (11.00-11.13) | +| agent session, follow-up turns | 14430 (14430-14430) at the last turn | see below | up to 64 | 1 | see below | 72.6 (62.3-75.1) | 2.66 (1.82-3.06) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 67.8 % (67.8-67.8); earlier-turn tokens read again per session: 0 (0-311) (earlier turns that the cache did not hold; the 0 (0-13748) new tokens of the files are read once); first turn (nothing cached) first token: 0.62 s (0.62-0.62). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 396 | 0 | 396 | 637.6 | 0.623 | 64.3 | +| 1 | 3730 | 391 | 3339 | 1646.4 | 2.047 | 62.3 | +| 2 | 5384 | 3725 | 1659 | 914.8 | 1.819 | 74.7 | +| 3 | 8550 | 5379 | 3171 | 1148.4 | 2.775 | 72.6 | +| 4 | 11577 | 8545 | 3032 | 1147.1 | 2.66 | 75.1 | +| 5 | 14430 | 11572 | 2858 | 939.1 | 3.062 | 69.2 | + +### Skipped and failed + +- failed agent: no token came back +- failed agent: no token came back + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 79 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json new file mode 100644 index 000000000..76b59eae7 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json @@ -0,0 +1,694 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 78.70480728149414, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--prompt-batch", + "2048", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --prompt-batch 2048 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 1493, + "expert_cache_mib": 8211, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1006, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.191, + "total_s": 1.453, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 251.1, + "prompt_tps_source": "engine", + "decode_tps": 78.6, + "decode_tps_source": "engine", + "prompt_ms": 179.2, + "decode_ms": 1272.0, + "drafts_offered": 3, + "drafts_accepted": 3, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.199, + "total_s": 1.509, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 194.2, + "prompt_tps_source": "engine", + "decode_tps": 75.6, + "decode_tps_source": "engine", + "prompt_ms": 185.4, + "decode_ms": 1322.1, + "drafts_offered": 6, + "drafts_accepted": 1, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.184, + "total_s": 1.521, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 227.7, + "prompt_tps_source": "engine", + "decode_tps": 74.2, + "decode_tps_source": "engine", + "prompt_ms": 171.3, + "decode_ms": 1348.3, + "drafts_offered": 9, + "drafts_accepted": 7, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.173, + "total_s": 1.353, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 279.0, + "prompt_tps_source": "engine", + "decode_tps": 84.0, + "decode_tps_source": "engine", + "prompt_ms": 161.3, + "decode_ms": 1190.6, + "drafts_offered": 8, + "drafts_accepted": 4, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.184, + "total_s": 1.268, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 208.8, + "prompt_tps_source": "engine", + "decode_tps": 91.4, + "decode_tps_source": "engine", + "prompt_ms": 172.4, + "decode_ms": 1094.1, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.18, + "total_s": 1.369, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 232.8, + "prompt_tps_source": "engine", + "decode_tps": 83.4, + "decode_tps_source": "engine", + "prompt_ms": 167.5, + "decode_ms": 1199.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.195, + "total_s": 1.34, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 246.2, + "prompt_tps_source": "engine", + "decode_tps": 86.6, + "decode_tps_source": "engine", + "prompt_ms": 182.8, + "decode_ms": 1155.3, + "drafts_offered": 4, + "drafts_accepted": 3, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.194, + "total_s": 1.346, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 199.0, + "prompt_tps_source": "engine", + "decode_tps": 86.0, + "decode_tps_source": "engine", + "prompt_ms": 180.9, + "decode_ms": 1163.2, + "drafts_offered": 4, + "drafts_accepted": 2, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.193, + "total_s": 1.332, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 215.8, + "prompt_tps_source": "engine", + "decode_tps": 86.9, + "decode_tps_source": "engine", + "prompt_ms": 180.7, + "decode_ms": 1150.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 84.0, + "min": 74.2, + "max": 91.4, + "n": 9 + }, + "ttft_s": { + "median": 0.191, + "min": 0.173, + "max": 0.199, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 39, + "min": 36, + "max": 45, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 2.573, + "total_s": 2.861, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1613.9, + "prompt_tps_source": "engine", + "decode_tps": 80.2, + "decode_tps_source": "engine", + "prompt_ms": 2556.6, + "decode_ms": 299.2, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.535, + "total_s": 2.839, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1638.8, + "prompt_tps_source": "engine", + "decode_tps": 75.7, + "decode_tps_source": "engine", + "prompt_ms": 2517.7, + "decode_ms": 316.9, + "drafts_offered": 1, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.498, + "total_s": 2.779, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1664.2, + "prompt_tps_source": "engine", + "decode_tps": 81.4, + "decode_tps_source": "engine", + "prompt_ms": 2479.3, + "decode_ms": 294.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1638.8, + "min": 1613.9, + "max": 1664.2, + "n": 3 + }, + "ttft_s": { + "median": 2.535, + "min": 2.498, + "max": 2.573, + "n": 3 + }, + "prompt_tokens": { + "median": 4126, + "min": 4126, + "max": 4126, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.135, + "total_s": 11.469, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1484.8, + "prompt_tps_source": "engine", + "decode_tps": 68.7, + "decode_tps_source": "engine", + "prompt_ms": 11106.1, + "decode_ms": 349.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 10.996, + "total_s": 11.331, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1503.5, + "prompt_tps_source": "engine", + "decode_tps": 68.8, + "decode_tps_source": "engine", + "prompt_ms": 10968.0, + "decode_ms": 349.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.013, + "total_s": 11.339, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1501.2, + "prompt_tps_source": "engine", + "decode_tps": 70.5, + "decode_tps_source": "engine", + "prompt_ms": 10984.5, + "decode_ms": 340.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1501.2, + "min": 1484.8, + "max": 1503.5, + "n": 3 + }, + "ttft_s": { + "median": 11.013, + "min": 10.996, + "max": 11.135, + "n": 3 + }, + "prompt_tokens": { + "median": 16490, + "min": 16490, + "max": 16490, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 0.623, + "total_s": 1.109, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 30, + "prompt_tps": 637.6, + "prompt_tps_source": "engine", + "decode_tps": 64.3, + "decode_tps_source": "engine", + "prompt_ms": 621.1, + "decode_ms": 466.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.047, + "total_s": 3.061, + "prompt_tokens": 3730, + "reused_tokens": 391, + "read_tokens": 3339, + "generated_tokens": 64, + "prompt_tps": 1646.4, + "prompt_tps_source": "engine", + "decode_tps": 62.3, + "decode_tps_source": "engine", + "prompt_ms": 2028.1, + "decode_ms": 1026.5, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.819, + "total_s": 2.677, + "prompt_tokens": 5384, + "reused_tokens": 3725, + "read_tokens": 1659, + "generated_tokens": 64, + "prompt_tps": 914.8, + "prompt_tps_source": "engine", + "decode_tps": 74.7, + "decode_tps_source": "engine", + "prompt_ms": 1813.5, + "decode_ms": 857.3, + "drafts_offered": 6, + "drafts_accepted": 1, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.775, + "total_s": 3.657, + "prompt_tokens": 8550, + "reused_tokens": 5379, + "read_tokens": 3171, + "generated_tokens": 64, + "prompt_tps": 1148.4, + "prompt_tps_source": "engine", + "decode_tps": 72.6, + "decode_tps_source": "engine", + "prompt_ms": 2761.3, + "decode_ms": 881.1, + "drafts_offered": 15, + "drafts_accepted": 6, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.66, + "total_s": 3.512, + "prompt_tokens": 11577, + "reused_tokens": 8545, + "read_tokens": 3032, + "generated_tokens": 64, + "prompt_tps": 1147.1, + "prompt_tps_source": "engine", + "decode_tps": 75.1, + "decode_tps_source": "engine", + "prompt_ms": 2643.1, + "decode_ms": 852.5, + "drafts_offered": 20, + "drafts_accepted": 8, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.062, + "total_s": 3.986, + "prompt_tokens": 14430, + "reused_tokens": 11572, + "read_tokens": 2858, + "generated_tokens": 64, + "prompt_tps": 939.1, + "prompt_tps_source": "engine", + "decode_tps": 69.2, + "decode_tps_source": "engine", + "prompt_ms": 3043.3, + "decode_ms": 924.6, + "drafts_offered": 12, + "drafts_accepted": 4, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.938, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 657.9, + "prompt_tps_source": "engine", + "decode_tps": 53.8, + "decode_tps_source": "engine", + "prompt_ms": 601.9, + "decode_ms": 334.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 0 + } + ], + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 0.884, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 13, + "prompt_tps": 588.7, + "prompt_tps_source": "engine", + "decode_tps": 62.0, + "decode_tps_source": "engine", + "prompt_ms": 672.7, + "decode_ms": 209.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + } + ] + ], + "hit_pct": { + "median": 67.807, + "min": 67.807, + "max": 67.807, + "n": 1 + }, + "reread_tokens": { + "median": 0, + "min": 0, + "max": 311, + "n": 3 + }, + "new_tokens": { + "median": 0, + "min": 0, + "max": 13748, + "n": 3 + }, + "turn_ttft_s": { + "median": 2.66, + "min": 1.819, + "max": 3.062, + "n": 5 + }, + "last_turn_prompt_tokens": { + "median": 14430, + "min": 14430, + "max": 14430, + "n": 1 + }, + "turn_decode_tps": { + "median": 72.6, + "min": 62.3, + "max": 75.1, + "n": 5 + }, + "first_turn_ttft_s": { + "median": 0.623, + "min": 0.623, + "max": 0.623, + "n": 1 + } + }, + "clients": {}, + "skipped": {}, + "failures": [ + { + "where": "agent", + "error": "no token came back" + }, + { + "where": "agent", + "error": "no token came back" + } + ] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..a92aeb89d --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,49 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.30 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 78.6 tok/s, first token 0.191 s, 100 tokens + run 1 q2: decode 75.6 tok/s, first token 0.199 s, 100 tokens + run 1 q3: decode 74.2 tok/s, first token 0.184 s, 100 tokens + run 2 q1: decode 84.0 tok/s, first token 0.173 s, 100 tokens + run 2 q2: decode 91.4 tok/s, first token 0.184 s, 100 tokens + run 2 q3: decode 83.4 tok/s, first token 0.18 s, 100 tokens + run 3 q1: decode 86.6 tok/s, first token 0.195 s, 100 tokens + run 3 q2: decode 86.0 tok/s, first token 0.194 s, 100 tokens + run 3 q3: decode 86.9 tok/s, first token 0.193 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4126 tokens, prompt 1613.9 tok/s (0 reused), first token 2.573 s + run 2: 4126 tokens, prompt 1638.8 tok/s (0 reused), first token 2.535 s + run 3: 4126 tokens, prompt 1664.2 tok/s (0 reused), first token 2.498 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16490 tokens, prompt 1484.8 tok/s (0 reused), first token 11.135 s + run 2: 16490 tokens, prompt 1503.5 tok/s (0 reused), first token 10.996 s + run 3: 16490 tokens, prompt 1501.2 tok/s (0 reused), first token 11.013 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 396, reused 0, read 396, first token 0.623 s + run 1 turn 1: prompt 3730, reused 391, read 3339, first token 2.047 s + run 1 turn 2: prompt 5384, reused 3725, read 1659, first token 1.819 s + run 1 turn 3: prompt 8550, reused 5379, read 3171, first token 2.775 s + run 1 turn 4: prompt 11577, reused 8545, read 3032, first token 2.66 s + run 1 turn 5: prompt 14430, reused 11572, read 2858, first token 3.062 s + FAILED: no token came back + FAILED: no token came back + +Results (median, range over the runs): + chat decode tok/s 84.00 (74.20 - 91.40, n=9) + chat first token (s) 0.19 (0.17 - 0.20, n=9) + 4K prompt tok/s 1638.80 (1613.90 - 1664.20, n=3) + 4K first token (s) 2.54 (2.50 - 2.57, n=3) + 16K prompt tok/s 1501.20 (1484.80 - 1503.50, n=3) + 16K first token (s) 11.01 (11.00 - 11.13, n=3) + agent cache hit % 67.81 (67.81 - 67.81, n=1) + agent re-read tokens/session 0.00 (0.00 - 311.00, n=3) + agent turn first token (s) 2.66 (1.82 - 3.06, n=5) + agent decode tok/s 72.60 (62.30 - 75.10, n=5) + (2 failed request(s), 0 skipped test(s): see the report) + +Report: /opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..fc34bf1e0 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,50 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/lfm25-UD-Q4_K_XL/LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/lfm25-ud-q4_k_xl-reserve1024-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--prompt-batch", + "2048", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/lfm25-ud-q4_k_xl/tokenizer", + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 14876901632, + "sha256": "a235d23638cc5549cb1ff61da7522c8a7e97f056f27093462a1e56f35aba91d8", + "filename": "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..f91b61b06 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 78.94333643899881 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md new file mode 100644 index 000000000..cbdc2d836 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md @@ -0,0 +1,63 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: lfm2.5-24b-a2b-ud-q4_k_xl +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --prompt-batch 2048 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 39 (36-45) | 0 | 100 (100-100) | 9 | not measured | 80.5 (71.2-89.3) | 0.21 (0.20-0.22) | +| 4K prompt, fresh | 4126 (4126-4126) | 0 (0-0) | 24 cap | 3 | 1571.6 (1528.6-1585.4) | not measured | 2.64 (2.62-2.72) | +| 16K prompt, fresh | 16490 (16490-16490) | 0 (0-0) | 24 cap | 3 | 1490.1 (1469.5-1495.3) | not measured | 11.10 (11.06-11.25) | +| 32K prompt, fresh | 32875 (32875-32875) | 0 (0-0) | 24 cap | 3 | 1120.6 (1113.9-1121.7) | not measured | 29.39 (29.36-29.56) | +| agent session, follow-up turns | 14457 (14442-14472) at the last turn | see below | up to 64 | 2 | see below | 72.2 (56.6-84.7) | 2.76 (1.94-3.17) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 67.8 % (67.8-67.9); earlier-turn tokens read again per session: 311 (0-334) (earlier turns that the cache did not hold; the 13760 (0-13767) new tokens of the files are read once); first turn (nothing cached) first token: 0.66 s (0.64-0.68). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | + +### Concurrency: 2 clients + +Total 73.2 tok/s (65.6-75.0); first token p50 1.154 s, p95 2.289 s; request latency p50 2.721 s, p95 3.802 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +### Skipped and failed + +- failed agent: no token came back + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 199 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json new file mode 100644 index 000000000..caf9bb50c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/results.json @@ -0,0 +1,939 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 199.18644881248474, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "quant": "UD-Q4_K_XL", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--prompt-batch", + "2048", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model LFM2.5-24B-A2B-UD-Q4_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --prompt-batch 2048 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "lfm2.5-24b-a2b-ud-q4_k_xl", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 1376, + "expert_cache_mib": 7568, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1007, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.206, + "total_s": 1.538, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 232.4, + "prompt_tps_source": "engine", + "decode_tps": 74.5, + "decode_tps_source": "engine", + "prompt_ms": 193.6, + "decode_ms": 1342.5, + "drafts_offered": 5, + "drafts_accepted": 3, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.208, + "total_s": 1.539, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 185.4, + "prompt_tps_source": "engine", + "decode_tps": 74.5, + "decode_tps_source": "engine", + "prompt_ms": 194.2, + "decode_ms": 1343.0, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.198, + "total_s": 1.59, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 212.0, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 184.0, + "decode_ms": 1404.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.207, + "total_s": 1.438, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 232.0, + "prompt_tps_source": "engine", + "decode_tps": 80.5, + "decode_tps_source": "engine", + "prompt_ms": 194.0, + "decode_ms": 1242.1, + "drafts_offered": 4, + "drafts_accepted": 3, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.223, + "total_s": 1.379, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 171.2, + "prompt_tps_source": "engine", + "decode_tps": 85.7, + "decode_tps_source": "engine", + "prompt_ms": 210.3, + "decode_ms": 1167.2, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.217, + "total_s": 1.458, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 191.7, + "prompt_tps_source": "engine", + "decode_tps": 79.8, + "decode_tps_source": "engine", + "prompt_ms": 203.4, + "decode_ms": 1252.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.223, + "total_s": 1.432, + "prompt_tokens": 45, + "reused_tokens": 0, + "read_tokens": 45, + "generated_tokens": 100, + "prompt_tps": 214.7, + "prompt_tps_source": "engine", + "decode_tps": 81.9, + "decode_tps_source": "engine", + "prompt_ms": 209.6, + "decode_ms": 1221.0, + "drafts_offered": 4, + "drafts_accepted": 4, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.217, + "total_s": 1.326, + "prompt_tokens": 36, + "reused_tokens": 0, + "read_tokens": 36, + "generated_tokens": 100, + "prompt_tps": 176.6, + "prompt_tps_source": "engine", + "decode_tps": 89.3, + "decode_tps_source": "engine", + "prompt_ms": 203.8, + "decode_ms": 1120.4, + "drafts_offered": 1, + "drafts_accepted": 1, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.207, + "total_s": 1.406, + "prompt_tokens": 39, + "reused_tokens": 0, + "read_tokens": 39, + "generated_tokens": 100, + "prompt_tps": 200.5, + "prompt_tps_source": "engine", + "decode_tps": 82.6, + "decode_tps_source": "engine", + "prompt_ms": 194.5, + "decode_ms": 1210.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 80.5, + "min": 71.2, + "max": 89.3, + "n": 9 + }, + "ttft_s": { + "median": 0.208, + "min": 0.198, + "max": 0.223, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 39, + "min": 36, + "max": 45, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 2.717, + "total_s": 3.029, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1528.6, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 2699.2, + "decode_ms": 325.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.642, + "total_s": 2.932, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1571.6, + "prompt_tps_source": "engine", + "decode_tps": 79.5, + "decode_tps_source": "engine", + "prompt_ms": 2625.4, + "decode_ms": 301.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.62, + "total_s": 2.923, + "prompt_tokens": 4126, + "reused_tokens": 0, + "read_tokens": 4126, + "generated_tokens": 24, + "prompt_tps": 1585.4, + "prompt_tps_source": "engine", + "decode_tps": 76.0, + "decode_tps_source": "engine", + "prompt_ms": 2602.5, + "decode_ms": 315.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1571.6, + "min": 1528.6, + "max": 1585.4, + "n": 3 + }, + "ttft_s": { + "median": 2.642, + "min": 2.62, + "max": 2.717, + "n": 3 + }, + "prompt_tokens": { + "median": 4126, + "min": 4126, + "max": 4126, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 11.251, + "total_s": 11.593, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1469.5, + "prompt_tps_source": "engine", + "decode_tps": 67.1, + "decode_tps_source": "engine", + "prompt_ms": 11221.2, + "decode_ms": 357.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.095, + "total_s": 11.433, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1490.1, + "prompt_tps_source": "engine", + "decode_tps": 67.9, + "decode_tps_source": "engine", + "prompt_ms": 11066.3, + "decode_ms": 353.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 11.057, + "total_s": 11.413, + "prompt_tokens": 16490, + "reused_tokens": 0, + "read_tokens": 16490, + "generated_tokens": 24, + "prompt_tps": 1495.3, + "prompt_tps_source": "engine", + "decode_tps": 64.6, + "decode_tps_source": "engine", + "prompt_ms": 11027.9, + "decode_ms": 371.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1490.1, + "min": 1469.5, + "max": 1495.3, + "n": 3 + }, + "ttft_s": { + "median": 11.095, + "min": 11.057, + "max": 11.251, + "n": 3 + }, + "prompt_tokens": { + "median": 16490, + "min": 16490, + "max": 16490, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 29.564, + "total_s": 30.019, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1113.9, + "prompt_tps_source": "engine", + "decode_tps": 50.1, + "decode_tps_source": "engine", + "prompt_ms": 29513.9, + "decode_ms": 479.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 29.358, + "total_s": 29.801, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1121.7, + "prompt_tps_source": "engine", + "decode_tps": 51.4, + "decode_tps_source": "engine", + "prompt_ms": 29309.3, + "decode_ms": 466.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 29.386, + "total_s": 29.818, + "prompt_tokens": 32875, + "reused_tokens": 0, + "read_tokens": 32875, + "generated_tokens": 24, + "prompt_tps": 1120.6, + "prompt_tps_source": "engine", + "decode_tps": 52.7, + "decode_tps_source": "engine", + "prompt_ms": 29337.6, + "decode_ms": 455.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1120.6, + "min": 1113.9, + "max": 1121.7, + "n": 3 + }, + "ttft_s": { + "median": 29.386, + "min": 29.358, + "max": 29.564, + "n": 3 + }, + "prompt_tokens": { + "median": 32875, + "min": 32875, + "max": 32875, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": false, + "err": null, + "ttft_s": null, + "total_s": 1.031, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 18, + "prompt_tps": 608.9, + "prompt_tps_source": "engine", + "decode_tps": 47.6, + "decode_tps_source": "engine", + "prompt_ms": 650.3, + "decode_ms": 378.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.641, + "total_s": 1.42, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 53, + "prompt_tps": 620.3, + "prompt_tps_source": "engine", + "decode_tps": 68.0, + "decode_tps_source": "engine", + "prompt_ms": 638.4, + "decode_ms": 779.6, + "drafts_offered": 4, + "drafts_accepted": 2, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.143, + "total_s": 3.26, + "prompt_tokens": 3754, + "reused_tokens": 391, + "read_tokens": 3363, + "generated_tokens": 64, + "prompt_tps": 1583.4, + "prompt_tps_source": "engine", + "decode_tps": 56.6, + "decode_tps_source": "engine", + "prompt_ms": 2123.9, + "decode_ms": 1130.0, + "drafts_offered": 25, + "drafts_accepted": 16, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.938, + "total_s": 2.816, + "prompt_tokens": 5426, + "reused_tokens": 3749, + "read_tokens": 1677, + "generated_tokens": 64, + "prompt_tps": 867.9, + "prompt_tps_source": "engine", + "decode_tps": 72.9, + "decode_tps_source": "engine", + "prompt_ms": 1932.3, + "decode_ms": 877.8, + "drafts_offered": 6, + "drafts_accepted": 2, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.875, + "total_s": 3.776, + "prompt_tokens": 8592, + "reused_tokens": 5421, + "read_tokens": 3171, + "generated_tokens": 64, + "prompt_tps": 1106.2, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 2866.6, + "decode_ms": 901.7, + "drafts_offered": 11, + "drafts_accepted": 3, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.751, + "total_s": 3.66, + "prompt_tokens": 11619, + "reused_tokens": 8587, + "read_tokens": 3032, + "generated_tokens": 64, + "prompt_tps": 1105.8, + "prompt_tps_source": "engine", + "decode_tps": 70.4, + "decode_tps_source": "engine", + "prompt_ms": 2741.8, + "decode_ms": 909.0, + "drafts_offered": 13, + "drafts_accepted": 7, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.168, + "total_s": 4.131, + "prompt_tokens": 14472, + "reused_tokens": 11614, + "read_tokens": 2858, + "generated_tokens": 64, + "prompt_tps": 904.9, + "prompt_tps_source": "engine", + "decode_tps": 66.4, + "decode_tps_source": "engine", + "prompt_ms": 3158.5, + "decode_ms": 963.4, + "drafts_offered": 11, + "drafts_accepted": 5, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 0.676, + "total_s": 1.127, + "prompt_tokens": 396, + "reused_tokens": 0, + "read_tokens": 396, + "generated_tokens": 30, + "prompt_tps": 588.1, + "prompt_tps_source": "engine", + "decode_tps": 66.5, + "decode_tps_source": "engine", + "prompt_ms": 673.3, + "decode_ms": 451.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.215, + "total_s": 2.988, + "prompt_tokens": 3730, + "reused_tokens": 391, + "read_tokens": 3339, + "generated_tokens": 64, + "prompt_tps": 1519.1, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 2198.0, + "decode_ms": 785.5, + "drafts_offered": 1, + "drafts_accepted": 0, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.953, + "total_s": 2.708, + "prompt_tokens": 5396, + "reused_tokens": 3725, + "read_tokens": 1671, + "generated_tokens": 64, + "prompt_tps": 857.9, + "prompt_tps_source": "engine", + "decode_tps": 84.7, + "decode_tps_source": "engine", + "prompt_ms": 1947.8, + "decode_ms": 755.2, + "drafts_offered": 1, + "drafts_accepted": 0, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.885, + "total_s": 3.708, + "prompt_tokens": 8562, + "reused_tokens": 5391, + "read_tokens": 3171, + "generated_tokens": 64, + "prompt_tps": 1101.7, + "prompt_tps_source": "engine", + "decode_tps": 77.7, + "decode_tps_source": "engine", + "prompt_ms": 2878.2, + "decode_ms": 823.5, + "drafts_offered": 3, + "drafts_accepted": 0, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.763, + "total_s": 3.629, + "prompt_tokens": 11589, + "reused_tokens": 8557, + "read_tokens": 3032, + "generated_tokens": 64, + "prompt_tps": 1100.8, + "prompt_tps_source": "engine", + "decode_tps": 73.9, + "decode_tps_source": "engine", + "prompt_ms": 2754.4, + "decode_ms": 866.2, + "drafts_offered": 3, + "drafts_accepted": 0, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.154, + "total_s": 4.051, + "prompt_tokens": 14442, + "reused_tokens": 11584, + "read_tokens": 2858, + "generated_tokens": 64, + "prompt_tps": 908.7, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 3145.0, + "decode_ms": 896.6, + "drafts_offered": 10, + "drafts_accepted": 6, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 67.834, + "min": 67.815, + "max": 67.852, + "n": 2 + }, + "reread_tokens": { + "median": 311, + "min": 0, + "max": 334, + "n": 3 + }, + "new_tokens": { + "median": 13760, + "min": 0, + "max": 13767, + "n": 3 + }, + "turn_ttft_s": { + "median": 2.757, + "min": 1.938, + "max": 3.168, + "n": 10 + }, + "last_turn_prompt_tokens": { + "median": 14457.0, + "min": 14442, + "max": 14472, + "n": 2 + }, + "turn_decode_tps": { + "median": 72.15, + "min": 56.6, + "max": 84.7, + "n": 10 + }, + "first_turn_ttft_s": { + "median": 0.659, + "min": 0.641, + "max": 0.676, + "n": 2 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 3.9, + "tokens": 256, + "total_tps": 65.56, + "first_s": [ + 0.33750370697816834, + 2.3609953650156967 + ], + "latency_s": [ + 2.0281291829887778, + 3.904149576992495 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 3.5, + "tokens": 256, + "total_tps": 73.2, + "first_s": [ + 0.3160019680217374, + 2.0745247620216105 + ], + "latency_s": [ + 1.7463452630036045, + 3.4968300400068983 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 3.41, + "tokens": 256, + "total_tps": 74.97, + "first_s": [ + 0.2971693619911093, + 1.970161963981809 + ], + "latency_s": [ + 1.6572078479803167, + 3.4146058309997898 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 73.2, + "min": 65.56, + "max": 74.97, + "n": 3 + }, + "ttft_p50_s": 1.154, + "ttft_p95_s": 2.289, + "latency_p50_s": 2.721, + "latency_p95_s": 3.802, + "errors": 0 + }, + "skipped": {}, + "failures": [ + { + "where": "agent", + "error": "no token came back" + } + ] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..6b06cbe9b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,66 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.30 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 74.5 tok/s, first token 0.206 s, 100 tokens + run 1 q2: decode 74.5 tok/s, first token 0.208 s, 100 tokens + run 1 q3: decode 71.2 tok/s, first token 0.198 s, 100 tokens + run 2 q1: decode 80.5 tok/s, first token 0.207 s, 100 tokens + run 2 q2: decode 85.7 tok/s, first token 0.223 s, 100 tokens + run 2 q3: decode 79.8 tok/s, first token 0.217 s, 100 tokens + run 3 q1: decode 81.9 tok/s, first token 0.223 s, 100 tokens + run 3 q2: decode 89.3 tok/s, first token 0.217 s, 100 tokens + run 3 q3: decode 82.6 tok/s, first token 0.207 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4126 tokens, prompt 1528.6 tok/s (0 reused), first token 2.717 s + run 2: 4126 tokens, prompt 1571.6 tok/s (0 reused), first token 2.642 s + run 3: 4126 tokens, prompt 1585.4 tok/s (0 reused), first token 2.62 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16490 tokens, prompt 1469.5 tok/s (0 reused), first token 11.251 s + run 2: 16490 tokens, prompt 1490.1 tok/s (0 reused), first token 11.095 s + run 3: 16490 tokens, prompt 1495.3 tok/s (0 reused), first token 11.057 s +agent: a session of 6 turns with four tools, x 3 runs + FAILED: no token came back + run 2 turn 0: prompt 396, reused 0, read 396, first token 0.641 s + run 2 turn 1: prompt 3754, reused 391, read 3363, first token 2.143 s + run 2 turn 2: prompt 5426, reused 3749, read 1677, first token 1.938 s + run 2 turn 3: prompt 8592, reused 5421, read 3171, first token 2.875 s + run 2 turn 4: prompt 11619, reused 8587, read 3032, first token 2.751 s + run 2 turn 5: prompt 14472, reused 11614, read 2858, first token 3.168 s + run 3 turn 0: prompt 396, reused 0, read 396, first token 0.676 s + run 3 turn 1: prompt 3730, reused 391, read 3339, first token 2.215 s + run 3 turn 2: prompt 5396, reused 3725, read 1671, first token 1.953 s + run 3 turn 3: prompt 8562, reused 5391, read 3171, first token 2.885 s + run 3 turn 4: prompt 11589, reused 8557, read 3032, first token 2.763 s + run 3 turn 5: prompt 14442, reused 11584, read 2858, first token 3.154 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 3.9 s = 65.6 tok/s total + round 2: 256 tokens in 3.5 s = 73.2 tok/s total + round 3: 256 tokens in 3.4 s = 75.0 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32875 tokens, prompt 1113.9 tok/s (0 reused), first token 29.564 s + run 2: 32875 tokens, prompt 1121.7 tok/s (0 reused), first token 29.358 s + run 3: 32875 tokens, prompt 1120.6 tok/s (0 reused), first token 29.386 s + +Results (median, range over the runs): + chat decode tok/s 80.50 (71.20 - 89.30, n=9) + chat first token (s) 0.21 (0.20 - 0.22, n=9) + 4K prompt tok/s 1571.60 (1528.60 - 1585.40, n=3) + 4K first token (s) 2.64 (2.62 - 2.72, n=3) + 16K prompt tok/s 1490.10 (1469.50 - 1495.30, n=3) + 16K first token (s) 11.10 (11.06 - 11.25, n=3) + 32K prompt tok/s 1120.60 (1113.90 - 1121.70, n=3) + 32K first token (s) 29.39 (29.36 - 29.56, n=3) + agent cache hit % 67.83 (67.81 - 67.85, n=2) + agent re-read tokens/session 311.00 (0.00 - 334.00, n=3) + agent turn first token (s) 2.76 (1.94 - 3.17, n=10) + agent decode tok/s 72.15 (56.60 - 84.70, n=10) + 2 clients total tok/s 73.20 (65.56 - 74.97, n=3) + (1 failed request(s), 0 skipped test(s): see the report) + +Report: /opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-lfm2-5-24b-a2b-ud-q4-k-xl (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..41a456c18 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,50 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/lfm25-UD-Q4_K_XL/LFM2.5-24B-A2B-UD-Q4_K_XL.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/lfm25-ud-q4_k_xl-reserve1024-ctx65536/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--prompt-batch", + "2048", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/lfm25-ud-q4_k_xl/tokenizer", + "model_name": "lfm2.5-24b-a2b-ud-q4_k_xl", + "display_name": "LFM2.5-24B-A2B UD-Q4_K_XL", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 14876901632, + "sha256": "a235d23638cc5549cb1ff61da7522c8a7e97f056f27093462a1e56f35aba91d8", + "filename": "LFM2.5-24B-A2B-UD-Q4_K_XL.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..0de03a655 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/lfm25-ud-q4_k_xl-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 199.41146662799292 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md new file mode 100644 index 000000000..422dd8fa1 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: minicpm-v-4.7-35b-a3b-mix_q3gud_im +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 59.2 (58.4-59.4) | 0.36 (0.34-0.39) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1132.5 (1129.7-1142.2) | not measured | 3.55 (3.52-3.56) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1008.2 (1003.4-1008.3) | not measured | 15.94 (15.94-16.02) | +| agent session, follow-up turns | 15157 (15157-15159) at the last turn | see below | up to 64 | 3 | see below | 54.6 (45.5-57.3) | 3.97 (1.99-4.92) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 69.2 % (69.2-69.2); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 14334 (14334-14334) new tokens of the files are read once); first turn (nothing cached) first token: 1.02 s (1.01-1.07). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 571.7 | 1.069 | 64.3 | +| 1 | 4178 | 619 | 3559 | 904.1 | 3.944 | 54.5 | +| 2 | 5824 | 4227 | 1597 | 794.4 | 2.017 | 55.7 | +| 3 | 9252 | 5880 | 3372 | 707.5 | 4.776 | 53.4 | +| 4 | 12215 | 9293 | 2922 | 737.8 | 3.971 | 52.4 | +| 5 | 15157 | 12268 | 2889 | 654.8 | 4.423 | 51.4 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 161 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json new file mode 100644 index 000000000..0342948d0 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 160.6695795059204, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "quant": null, + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "display_name": "MiniCPM-V 4.7 35B-A3B mix_q3gud_im", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 6179, + "expert_cache_mib": 7967, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.344, + "total_s": 2.055, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 149.1, + "prompt_tps_source": "engine", + "decode_tps": 58.4, + "decode_tps_source": "engine", + "prompt_ms": 342.1, + "decode_ms": 1711.7, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.361, + "total_s": 2.068, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 119.6, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 359.6, + "decode_ms": 1706.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.36, + "total_s": 2.06, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 128.3, + "prompt_tps_source": "engine", + "decode_tps": 58.8, + "decode_tps_source": "engine", + "prompt_ms": 358.5, + "decode_ms": 1700.0, + "drafts_offered": 1, + "drafts_accepted": 0, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.377, + "total_s": 2.074, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 135.9, + "prompt_tps_source": "engine", + "decode_tps": 58.9, + "decode_tps_source": "engine", + "prompt_ms": 375.4, + "decode_ms": 1697.3, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.38, + "total_s": 2.071, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 113.5, + "prompt_tps_source": "engine", + "decode_tps": 59.2, + "decode_tps_source": "engine", + "prompt_ms": 378.9, + "decode_ms": 1690.2, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.355, + "total_s": 2.039, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 130.2, + "prompt_tps_source": "engine", + "decode_tps": 59.4, + "decode_tps_source": "engine", + "prompt_ms": 353.3, + "decode_ms": 1684.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.366, + "total_s": 2.05, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 139.9, + "prompt_tps_source": "engine", + "decode_tps": 59.4, + "decode_tps_source": "engine", + "prompt_ms": 364.5, + "decode_ms": 1683.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.362, + "total_s": 2.05, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 119.1, + "prompt_tps_source": "engine", + "decode_tps": 59.3, + "decode_tps_source": "engine", + "prompt_ms": 360.9, + "decode_ms": 1687.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.385, + "total_s": 2.074, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 120.1, + "prompt_tps_source": "engine", + "decode_tps": 59.2, + "decode_tps_source": "engine", + "prompt_ms": 383.0, + "decode_ms": 1689.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 59.2, + "min": 58.4, + "max": 59.4, + "n": 9 + }, + "ttft_s": { + "median": 0.362, + "min": 0.344, + "max": 0.385, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.554, + "total_s": 3.965, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1132.5, + "prompt_tps_source": "engine", + "decode_tps": 58.4, + "decode_tps_source": "engine", + "prompt_ms": 3548.7, + "decode_ms": 411.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.563, + "total_s": 3.971, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1129.7, + "prompt_tps_source": "engine", + "decode_tps": 58.7, + "decode_tps_source": "engine", + "prompt_ms": 3557.7, + "decode_ms": 408.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.524, + "total_s": 3.932, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1142.2, + "prompt_tps_source": "engine", + "decode_tps": 58.8, + "decode_tps_source": "engine", + "prompt_ms": 3518.5, + "decode_ms": 408.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1132.5, + "min": 1129.7, + "max": 1142.2, + "n": 3 + }, + "ttft_s": { + "median": 3.554, + "min": 3.524, + "max": 3.563, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.021, + "total_s": 16.755, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1003.4, + "prompt_tps_source": "engine", + "decode_tps": 32.7, + "decode_tps_source": "engine", + "prompt_ms": 16004.0, + "decode_ms": 734.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.941, + "total_s": 16.417, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1008.3, + "prompt_tps_source": "engine", + "decode_tps": 50.5, + "decode_tps_source": "engine", + "prompt_ms": 15925.2, + "decode_ms": 475.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.943, + "total_s": 16.47, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1008.2, + "prompt_tps_source": "engine", + "decode_tps": 45.5, + "decode_tps_source": "engine", + "prompt_ms": 15926.8, + "decode_ms": 527.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1008.2, + "min": 1003.4, + "max": 1008.3, + "n": 3 + }, + "ttft_s": { + "median": 15.943, + "min": 15.941, + "max": 16.021, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.069, + "total_s": 1.24, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 571.7, + "prompt_tps_source": "engine", + "decode_tps": 64.3, + "decode_tps_source": "engine", + "prompt_ms": 1065.2, + "decode_ms": 171.2, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.944, + "total_s": 4.861, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 904.1, + "prompt_tps_source": "engine", + "decode_tps": 54.5, + "decode_tps_source": "engine", + "prompt_ms": 3936.5, + "decode_ms": 916.8, + "drafts_offered": 1, + "drafts_accepted": 0, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.017, + "total_s": 3.041, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 57, + "prompt_tps": 794.4, + "prompt_tps_source": "engine", + "decode_tps": 55.7, + "decode_tps_source": "engine", + "prompt_ms": 2010.4, + "decode_ms": 1023.9, + "drafts_offered": 4, + "drafts_accepted": 1, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.776, + "total_s": 5.562, + "prompt_tokens": 9252, + "reused_tokens": 5880, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 707.5, + "prompt_tps_source": "engine", + "decode_tps": 53.4, + "decode_tps_source": "engine", + "prompt_ms": 4766.2, + "decode_ms": 785.8, + "drafts_offered": 5, + "drafts_accepted": 2, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.971, + "total_s": 5.002, + "prompt_tokens": 12215, + "reused_tokens": 9293, + "read_tokens": 2922, + "generated_tokens": 54, + "prompt_tps": 737.8, + "prompt_tps_source": "engine", + "decode_tps": 52.4, + "decode_tps_source": "engine", + "prompt_ms": 3960.6, + "decode_ms": 1031.3, + "drafts_offered": 9, + "drafts_accepted": 5, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.423, + "total_s": 5.667, + "prompt_tokens": 15157, + "reused_tokens": 12268, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 654.8, + "prompt_tps_source": "engine", + "decode_tps": 51.4, + "decode_tps_source": "engine", + "prompt_ms": 4412.2, + "decode_ms": 1244.4, + "drafts_offered": 6, + "drafts_accepted": 2, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.018, + "total_s": 1.189, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 599.6, + "prompt_tps_source": "engine", + "decode_tps": 64.0, + "decode_tps_source": "engine", + "prompt_ms": 1015.6, + "decode_ms": 171.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.137, + "total_s": 5.009, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 861.4, + "prompt_tps_source": "engine", + "decode_tps": 57.3, + "decode_tps_source": "engine", + "prompt_ms": 4131.8, + "decode_ms": 872.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.989, + "total_s": 2.998, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 57, + "prompt_tps": 804.7, + "prompt_tps_source": "engine", + "decode_tps": 56.5, + "decode_tps_source": "engine", + "prompt_ms": 1984.7, + "decode_ms": 1008.8, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.917, + "total_s": 5.679, + "prompt_tokens": 9252, + "reused_tokens": 5880, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 686.7, + "prompt_tps_source": "engine", + "decode_tps": 55.1, + "decode_tps_source": "engine", + "prompt_ms": 4910.2, + "decode_ms": 762.6, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.96, + "total_s": 4.956, + "prompt_tokens": 12215, + "reused_tokens": 9293, + "read_tokens": 2922, + "generated_tokens": 54, + "prompt_tps": 739.4, + "prompt_tps_source": "engine", + "decode_tps": 54.2, + "decode_tps_source": "engine", + "prompt_ms": 3952.0, + "decode_ms": 995.8, + "drafts_offered": 2, + "drafts_accepted": 2, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.886, + "total_s": 6.115, + "prompt_tokens": 15157, + "reused_tokens": 12268, + "read_tokens": 2889, + "generated_tokens": 56, + "prompt_tps": 592.4, + "prompt_tps_source": "engine", + "decode_tps": 45.5, + "decode_tps_source": "engine", + "prompt_ms": 4876.8, + "decode_ms": 1229.8, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.012, + "total_s": 1.184, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 602.9, + "prompt_tps_source": "engine", + "decode_tps": 64.0, + "decode_tps_source": "engine", + "prompt_ms": 1010.2, + "decode_ms": 172.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.94, + "total_s": 4.812, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 904.6, + "prompt_tps_source": "engine", + "decode_tps": 57.3, + "decode_tps_source": "engine", + "prompt_ms": 3934.4, + "decode_ms": 872.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.988, + "total_s": 3.006, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 57, + "prompt_tps": 805.1, + "prompt_tps_source": "engine", + "decode_tps": 56.0, + "decode_tps_source": "engine", + "prompt_ms": 1983.7, + "decode_ms": 1018.3, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.8, + "total_s": 5.569, + "prompt_tokens": 9252, + "reused_tokens": 5880, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 703.5, + "prompt_tps_source": "engine", + "decode_tps": 54.6, + "decode_tps_source": "engine", + "prompt_ms": 4793.5, + "decode_ms": 768.6, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.95, + "total_s": 4.975, + "prompt_tokens": 12215, + "reused_tokens": 9293, + "read_tokens": 2922, + "generated_tokens": 56, + "prompt_tps": 741.3, + "prompt_tps_source": "engine", + "decode_tps": 54.6, + "decode_tps_source": "engine", + "prompt_ms": 3941.7, + "decode_ms": 1026.0, + "drafts_offered": 2, + "drafts_accepted": 2, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.392, + "total_s": 5.616, + "prompt_tokens": 15159, + "reused_tokens": 12270, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 659.2, + "prompt_tps_source": "engine", + "decode_tps": 52.3, + "decode_tps_source": "engine", + "prompt_ms": 4382.8, + "decode_ms": 1224.5, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 69.247, + "min": 69.247, + "max": 69.248, + "n": 3 + }, + "reread_tokens": { + "median": 5, + "min": 5, + "max": 5, + "n": 3 + }, + "new_tokens": { + "median": 14334, + "min": 14334, + "max": 14334, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.971, + "min": 1.988, + "max": 4.917, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15157, + "min": 15157, + "max": 15159, + "n": 3 + }, + "turn_decode_tps": { + "median": 54.6, + "min": 45.5, + "max": 57.3, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.018, + "min": 1.012, + "max": 1.069, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..4b4945d0b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 58.4 tok/s, first token 0.344 s, 100 tokens + run 1 q2: decode 58.6 tok/s, first token 0.361 s, 100 tokens + run 1 q3: decode 58.8 tok/s, first token 0.36 s, 100 tokens + run 2 q1: decode 58.9 tok/s, first token 0.377 s, 100 tokens + run 2 q2: decode 59.2 tok/s, first token 0.38 s, 100 tokens + run 2 q3: decode 59.4 tok/s, first token 0.355 s, 100 tokens + run 3 q1: decode 59.4 tok/s, first token 0.366 s, 100 tokens + run 3 q2: decode 59.3 tok/s, first token 0.362 s, 100 tokens + run 3 q3: decode 59.2 tok/s, first token 0.385 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1132.5 tok/s (0 reused), first token 3.554 s + run 2: 4019 tokens, prompt 1129.7 tok/s (0 reused), first token 3.563 s + run 3: 4019 tokens, prompt 1142.2 tok/s (0 reused), first token 3.524 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1003.4 tok/s (0 reused), first token 16.021 s + run 2: 16058 tokens, prompt 1008.3 tok/s (0 reused), first token 15.941 s + run 3: 16058 tokens, prompt 1008.2 tok/s (0 reused), first token 15.943 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.069 s + run 1 turn 1: prompt 4178, reused 619, read 3559, first token 3.944 s + run 1 turn 2: prompt 5824, reused 4227, read 1597, first token 2.017 s + run 1 turn 3: prompt 9252, reused 5880, read 3372, first token 4.776 s + run 1 turn 4: prompt 12215, reused 9293, read 2922, first token 3.971 s + run 1 turn 5: prompt 15157, reused 12268, read 2889, first token 4.423 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.018 s + run 2 turn 1: prompt 4178, reused 619, read 3559, first token 4.137 s + run 2 turn 2: prompt 5824, reused 4227, read 1597, first token 1.989 s + run 2 turn 3: prompt 9252, reused 5880, read 3372, first token 4.917 s + run 2 turn 4: prompt 12215, reused 9293, read 2922, first token 3.96 s + run 2 turn 5: prompt 15157, reused 12268, read 2889, first token 4.886 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.012 s + run 3 turn 1: prompt 4178, reused 619, read 3559, first token 3.94 s + run 3 turn 2: prompt 5824, reused 4227, read 1597, first token 1.988 s + run 3 turn 3: prompt 9252, reused 5880, read 3372, first token 4.8 s + run 3 turn 4: prompt 12215, reused 9293, read 2922, first token 3.95 s + run 3 turn 5: prompt 15159, reused 12270, read 2889, first token 4.392 s + +Results (median, range over the runs): + chat decode tok/s 59.20 (58.40 - 59.40, n=9) + chat first token (s) 0.36 (0.34 - 0.39, n=9) + 4K prompt tok/s 1132.50 (1129.70 - 1142.20, n=3) + 4K first token (s) 3.55 (3.52 - 3.56, n=3) + 16K prompt tok/s 1008.20 (1003.40 - 1008.30, n=3) + 16K first token (s) 15.94 (15.94 - 16.02, n=3) + agent cache hit % 69.25 (69.25 - 69.25, n=3) + agent re-read tokens/session 5.00 (5.00 - 5.00, n=3) + agent turn first token (s) 3.97 (1.99 - 4.92, n=15) + agent decode tok/s 54.60 (45.50 - 57.30, n=15) + +Report: /opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..ef3ece87f --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/minicpmv-mix_q3gud_im/MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/minicpmv-mix_q3gud_im-reserve1024-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/minicpmv-mix_q3gud_im/tokenizer", + "model_name": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "display_name": "MiniCPM-V 4.7 35B-A3B mix_q3gud_im", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 15503944928, + "sha256": "6bfeefbfc723ab6fe557fcea28b31253e14128efe8d8d5774e3256a547b3a946", + "filename": "MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..c9e666f13 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 160.90037102301721 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md new file mode 100644 index 000000000..450600aa2 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: minicpm-v-4.7-35b-a3b-mix_q3gud_im +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 59.2 (58.2-59.6) | 0.37 (0.34-0.42) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1120.2 (1117.0-1120.5) | not measured | 3.59 (3.59-3.60) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1006.1 (996.4-1009.7) | not measured | 15.98 (15.92-16.13) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 790.5 (783.5-790.8) | not measured | 40.57 (40.56-40.94) | +| agent session, follow-up turns | 15157 (15155-15157) at the last turn | see below | up to 64 | 3 | see below | 54.7 (49.7-57.3) | 4.03 (2.00-9.62) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 69.2 % (69.2-69.2); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 14334 (14334-14334) new tokens of the files are read once); first turn (nothing cached) first token: 1.09 s (1.06-1.13). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 541.2 | 1.129 | 64.1 | +| 1 | 4178 | 619 | 3559 | 901.7 | 3.955 | 56.6 | +| 2 | 5824 | 4227 | 1597 | 799.9 | 2.004 | 54.5 | +| 3 | 9252 | 5880 | 3372 | 678.9 | 4.977 | 54.7 | +| 4 | 12215 | 9293 | 2922 | 671.5 | 4.362 | 53.7 | +| 5 | 15157 | 12268 | 2889 | 653.4 | 4.432 | 50.7 | + +### Concurrency: 2 clients + +Total 49.3 tok/s (44.3-49.7); first token p50 1.742 s, p95 3.468 s; request latency p50 4.167 s, p95 5.633 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 306 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json new file mode 100644 index 000000000..9d7e43c70 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/results.json @@ -0,0 +1,1032 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 305.94040298461914, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "quant": null, + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "display_name": "MiniCPM-V 4.7 35B-A3B mix_q3gud_im", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 5683, + "expert_cache_mib": 7328, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.344, + "total_s": 2.061, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 149.1, + "prompt_tps_source": "engine", + "decode_tps": 58.2, + "decode_tps_source": "engine", + "prompt_ms": 342.1, + "decode_ms": 1717.5, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.343, + "total_s": 2.061, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 126.2, + "prompt_tps_source": "engine", + "decode_tps": 58.2, + "decode_tps_source": "engine", + "prompt_ms": 340.8, + "decode_ms": 1718.0, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.378, + "total_s": 2.07, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 122.3, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 376.1, + "decode_ms": 1691.8, + "drafts_offered": 2, + "drafts_accepted": 0, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.379, + "total_s": 2.065, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 135.2, + "prompt_tps_source": "engine", + "decode_tps": 59.3, + "decode_tps_source": "engine", + "prompt_ms": 377.2, + "decode_ms": 1686.4, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.349, + "total_s": 2.041, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 123.8, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 347.3, + "decode_ms": 1691.8, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.34, + "total_s": 2.017, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 136.2, + "prompt_tps_source": "engine", + "decode_tps": 59.6, + "decode_tps_source": "engine", + "prompt_ms": 337.8, + "decode_ms": 1677.4, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.409, + "total_s": 2.086, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 125.2, + "prompt_tps_source": "engine", + "decode_tps": 59.6, + "decode_tps_source": "engine", + "prompt_ms": 407.5, + "decode_ms": 1676.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.369, + "total_s": 2.058, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 117.0, + "prompt_tps_source": "engine", + "decode_tps": 59.2, + "decode_tps_source": "engine", + "prompt_ms": 367.5, + "decode_ms": 1689.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.419, + "total_s": 2.097, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 110.3, + "prompt_tps_source": "engine", + "decode_tps": 59.6, + "decode_tps_source": "engine", + "prompt_ms": 417.0, + "decode_ms": 1678.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 59.2, + "min": 58.2, + "max": 59.6, + "n": 9 + }, + "ttft_s": { + "median": 0.369, + "min": 0.34, + "max": 0.419, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.593, + "total_s": 4.006, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1120.2, + "prompt_tps_source": "engine", + "decode_tps": 58.1, + "decode_tps_source": "engine", + "prompt_ms": 3587.6, + "decode_ms": 412.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.592, + "total_s": 4.0, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1120.5, + "prompt_tps_source": "engine", + "decode_tps": 58.8, + "decode_tps_source": "engine", + "prompt_ms": 3586.8, + "decode_ms": 408.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.604, + "total_s": 4.013, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1117.0, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 3598.1, + "decode_ms": 409.3, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1120.2, + "min": 1117.0, + "max": 1120.5, + "n": 3 + }, + "ttft_s": { + "median": 3.593, + "min": 3.592, + "max": 3.604, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 15.977, + "total_s": 16.429, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1006.1, + "prompt_tps_source": "engine", + "decode_tps": 53.0, + "decode_tps_source": "engine", + "prompt_ms": 15960.4, + "decode_ms": 452.5, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.132, + "total_s": 16.585, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 996.4, + "prompt_tps_source": "engine", + "decode_tps": 52.9, + "decode_tps_source": "engine", + "prompt_ms": 16115.5, + "decode_ms": 454.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.921, + "total_s": 16.374, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1009.7, + "prompt_tps_source": "engine", + "decode_tps": 52.9, + "decode_tps_source": "engine", + "prompt_ms": 15904.3, + "decode_ms": 453.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 1006.1, + "min": 996.4, + "max": 1009.7, + "n": 3 + }, + "ttft_s": { + "median": 15.977, + "min": 15.921, + "max": 16.132, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 40.937, + "total_s": 41.444, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 783.5, + "prompt_tps_source": "engine", + "decode_tps": 47.3, + "decode_tps_source": "engine", + "prompt_ms": 40906.1, + "decode_ms": 507.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 40.558, + "total_s": 41.066, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 790.8, + "prompt_tps_source": "engine", + "decode_tps": 47.2, + "decode_tps_source": "engine", + "prompt_ms": 40527.2, + "decode_ms": 508.1, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 40.573, + "total_s": 41.083, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 790.5, + "prompt_tps_source": "engine", + "decode_tps": 47.0, + "decode_tps_source": "engine", + "prompt_ms": 40541.6, + "decode_ms": 510.8, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2 + } + ], + "prompt_tps": { + "median": 790.5, + "min": 783.5, + "max": 790.8, + "n": 3 + }, + "ttft_s": { + "median": 40.573, + "min": 40.558, + "max": 40.937, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.129, + "total_s": 1.3, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 541.2, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 1125.2, + "decode_ms": 171.6, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.955, + "total_s": 4.838, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 901.7, + "prompt_tps_source": "engine", + "decode_tps": 56.6, + "decode_tps_source": "engine", + "prompt_ms": 3947.2, + "decode_ms": 882.9, + "drafts_offered": 2, + "drafts_accepted": 0, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.004, + "total_s": 3.05, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 57, + "prompt_tps": 799.9, + "prompt_tps_source": "engine", + "decode_tps": 54.5, + "decode_tps_source": "engine", + "prompt_ms": 1996.6, + "decode_ms": 1046.4, + "drafts_offered": 5, + "drafts_accepted": 1, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.977, + "total_s": 5.744, + "prompt_tokens": 9252, + "reused_tokens": 5880, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 678.9, + "prompt_tps_source": "engine", + "decode_tps": 54.7, + "decode_tps_source": "engine", + "prompt_ms": 4967.1, + "decode_ms": 767.4, + "drafts_offered": 5, + "drafts_accepted": 2, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.362, + "total_s": 5.367, + "prompt_tokens": 12215, + "reused_tokens": 9293, + "read_tokens": 2922, + "generated_tokens": 54, + "prompt_tps": 671.5, + "prompt_tps_source": "engine", + "decode_tps": 53.7, + "decode_tps_source": "engine", + "prompt_ms": 4351.2, + "decode_ms": 1005.4, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.432, + "total_s": 5.695, + "prompt_tokens": 15157, + "reused_tokens": 12268, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 653.4, + "prompt_tps_source": "engine", + "decode_tps": 50.7, + "decode_tps_source": "engine", + "prompt_ms": 4421.6, + "decode_ms": 1263.1, + "drafts_offered": 11, + "drafts_accepted": 3, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.06, + "total_s": 1.231, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 575.8, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 1057.7, + "decode_ms": 171.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.952, + "total_s": 4.826, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 901.7, + "prompt_tps_source": "engine", + "decode_tps": 57.2, + "decode_tps_source": "engine", + "prompt_ms": 3946.9, + "decode_ms": 874.0, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.006, + "total_s": 2.989, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 55, + "prompt_tps": 797.9, + "prompt_tps_source": "engine", + "decode_tps": 55.9, + "decode_tps_source": "engine", + "prompt_ms": 2001.4, + "decode_ms": 983.1, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 9.62, + "total_s": 10.387, + "prompt_tokens": 9250, + "reused_tokens": 5878, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 350.8, + "prompt_tps_source": "engine", + "decode_tps": 54.7, + "decode_tps_source": "engine", + "prompt_ms": 9613.0, + "decode_ms": 767.5, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.033, + "total_s": 5.12, + "prompt_tokens": 12213, + "reused_tokens": 9291, + "read_tokens": 2922, + "generated_tokens": 54, + "prompt_tps": 726.1, + "prompt_tps_source": "engine", + "decode_tps": 49.7, + "decode_tps_source": "engine", + "prompt_ms": 4024.3, + "decode_ms": 1087.5, + "drafts_offered": 2, + "drafts_accepted": 2, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.958, + "total_s": 6.195, + "prompt_tokens": 15155, + "reused_tokens": 12266, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 583.8, + "prompt_tps_source": "engine", + "decode_tps": 51.7, + "decode_tps_source": "engine", + "prompt_ms": 4948.7, + "decode_ms": 1237.3, + "drafts_offered": 5, + "drafts_accepted": 2, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.093, + "total_s": 1.265, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 558.2, + "prompt_tps_source": "engine", + "decode_tps": 64.0, + "decode_tps_source": "engine", + "prompt_ms": 1091.1, + "decode_ms": 171.9, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.988, + "total_s": 4.861, + "prompt_tokens": 4178, + "reused_tokens": 619, + "read_tokens": 3559, + "generated_tokens": 50, + "prompt_tps": 893.5, + "prompt_tps_source": "engine", + "decode_tps": 57.3, + "decode_tps_source": "engine", + "prompt_ms": 3983.3, + "decode_ms": 872.7, + "drafts_offered": 0, + "drafts_accepted": 0, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.013, + "total_s": 3.031, + "prompt_tokens": 5824, + "reused_tokens": 4227, + "read_tokens": 1597, + "generated_tokens": 57, + "prompt_tps": 795.4, + "prompt_tps_source": "engine", + "decode_tps": 55.9, + "decode_tps_source": "engine", + "prompt_ms": 2007.8, + "decode_ms": 1019.2, + "drafts_offered": 3, + "drafts_accepted": 1, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.939, + "total_s": 5.692, + "prompt_tokens": 9252, + "reused_tokens": 5880, + "read_tokens": 3372, + "generated_tokens": 42, + "prompt_tps": 683.7, + "prompt_tps_source": "engine", + "decode_tps": 55.8, + "decode_tps_source": "engine", + "prompt_ms": 4932.1, + "decode_ms": 753.0, + "drafts_offered": 1, + "drafts_accepted": 1, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.003, + "total_s": 5.009, + "prompt_tokens": 12215, + "reused_tokens": 9293, + "read_tokens": 2922, + "generated_tokens": 54, + "prompt_tps": 731.5, + "prompt_tps_source": "engine", + "decode_tps": 53.6, + "decode_tps_source": "engine", + "prompt_ms": 3994.3, + "decode_ms": 1006.6, + "drafts_offered": 2, + "drafts_accepted": 1, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.444, + "total_s": 5.671, + "prompt_tokens": 15157, + "reused_tokens": 12268, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 651.5, + "prompt_tps_source": "engine", + "decode_tps": 52.2, + "decode_tps_source": "engine", + "prompt_ms": 4434.4, + "decode_ms": 1227.2, + "drafts_offered": 1, + "drafts_accepted": 1, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 69.247, + "min": 69.243, + "max": 69.247, + "n": 3 + }, + "reread_tokens": { + "median": 5, + "min": 5, + "max": 5, + "n": 3 + }, + "new_tokens": { + "median": 14334, + "min": 14334, + "max": 14334, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.033, + "min": 2.004, + "max": 9.62, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15157, + "min": 15155, + "max": 15157, + "n": 3 + }, + "turn_decode_tps": { + "median": 54.7, + "min": 49.7, + "max": 57.3, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.093, + "min": 1.06, + "max": 1.129, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 5.78, + "tokens": 256, + "total_tps": 44.27, + "first_s": [ + 0.5034954560105689, + 3.6171913750004023 + ], + "latency_s": [ + 3.1840875440102536, + 5.78189775600913 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 5.19, + "tokens": 256, + "total_tps": 49.34, + "first_s": [ + 0.4193754779989831, + 3.0220336340134963 + ], + "latency_s": [ + 2.5845785599958617, + 5.188020997011336 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 5.15, + "tokens": 256, + "total_tps": 49.71, + "first_s": [ + 0.40712625699234195, + 2.9802002549986355 + ], + "latency_s": [ + 2.575250911002513, + 5.149369971011765 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 49.34, + "min": 44.27, + "max": 49.71, + "n": 3 + }, + "ttft_p50_s": 1.742, + "ttft_p95_s": 3.468, + "latency_p50_s": 4.167, + "latency_p95_s": 5.633, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..47b48a0cb --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 58.2 tok/s, first token 0.344 s, 100 tokens + run 1 q2: decode 58.2 tok/s, first token 0.343 s, 100 tokens + run 1 q3: decode 59.1 tok/s, first token 0.378 s, 100 tokens + run 2 q1: decode 59.3 tok/s, first token 0.379 s, 100 tokens + run 2 q2: decode 59.1 tok/s, first token 0.349 s, 100 tokens + run 2 q3: decode 59.6 tok/s, first token 0.34 s, 100 tokens + run 3 q1: decode 59.6 tok/s, first token 0.409 s, 100 tokens + run 3 q2: decode 59.2 tok/s, first token 0.369 s, 100 tokens + run 3 q3: decode 59.6 tok/s, first token 0.419 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1120.2 tok/s (0 reused), first token 3.593 s + run 2: 4019 tokens, prompt 1120.5 tok/s (0 reused), first token 3.592 s + run 3: 4019 tokens, prompt 1117.0 tok/s (0 reused), first token 3.604 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1006.1 tok/s (0 reused), first token 15.977 s + run 2: 16058 tokens, prompt 996.4 tok/s (0 reused), first token 16.132 s + run 3: 16058 tokens, prompt 1009.7 tok/s (0 reused), first token 15.921 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.129 s + run 1 turn 1: prompt 4178, reused 619, read 3559, first token 3.955 s + run 1 turn 2: prompt 5824, reused 4227, read 1597, first token 2.004 s + run 1 turn 3: prompt 9252, reused 5880, read 3372, first token 4.977 s + run 1 turn 4: prompt 12215, reused 9293, read 2922, first token 4.362 s + run 1 turn 5: prompt 15157, reused 12268, read 2889, first token 4.432 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.06 s + run 2 turn 1: prompt 4178, reused 619, read 3559, first token 3.952 s + run 2 turn 2: prompt 5824, reused 4227, read 1597, first token 2.006 s + run 2 turn 3: prompt 9250, reused 5878, read 3372, first token 9.62 s + run 2 turn 4: prompt 12213, reused 9291, read 2922, first token 4.033 s + run 2 turn 5: prompt 15155, reused 12266, read 2889, first token 4.958 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.093 s + run 3 turn 1: prompt 4178, reused 619, read 3559, first token 3.988 s + run 3 turn 2: prompt 5824, reused 4227, read 1597, first token 2.013 s + run 3 turn 3: prompt 9252, reused 5880, read 3372, first token 4.939 s + run 3 turn 4: prompt 12215, reused 9293, read 2922, first token 4.003 s + run 3 turn 5: prompt 15157, reused 12268, read 2889, first token 4.444 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 5.8 s = 44.3 tok/s total + round 2: 256 tokens in 5.2 s = 49.3 tok/s total + round 3: 256 tokens in 5.1 s = 49.7 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32049 tokens, prompt 783.5 tok/s (0 reused), first token 40.937 s + run 2: 32049 tokens, prompt 790.8 tok/s (0 reused), first token 40.558 s + run 3: 32049 tokens, prompt 790.5 tok/s (0 reused), first token 40.573 s + +Results (median, range over the runs): + chat decode tok/s 59.20 (58.20 - 59.60, n=9) + chat first token (s) 0.37 (0.34 - 0.42, n=9) + 4K prompt tok/s 1120.20 (1117.00 - 1120.50, n=3) + 4K first token (s) 3.59 (3.59 - 3.60, n=3) + 16K prompt tok/s 1006.10 (996.40 - 1009.70, n=3) + 16K first token (s) 15.98 (15.92 - 16.13, n=3) + 32K prompt tok/s 790.50 (783.50 - 790.80, n=3) + 32K first token (s) 40.57 (40.56 - 40.94, n=3) + agent cache hit % 69.25 (69.24 - 69.25, n=3) + agent re-read tokens/session 5.00 (5.00 - 5.00, n=3) + agent turn first token (s) 4.03 (2.00 - 9.62, n=15) + agent decode tok/s 54.70 (49.70 - 57.30, n=15) + 2 clients total tok/s 49.34 (44.27 - 49.71, n=3) + +Report: /opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-minicpm-v-4-7-35b-a3b-mix-q3gud-im (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..c3a5623a8 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/minicpmv-mix_q3gud_im/MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/minicpmv-mix_q3gud_im-reserve1024-ctx65536/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/minicpmv-mix_q3gud_im/tokenizer", + "model_name": "minicpm-v-4.7-35b-a3b-mix_q3gud_im", + "display_name": "MiniCPM-V 4.7 35B-A3B mix_q3gud_im", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 15503944928, + "sha256": "6bfeefbfc723ab6fe557fcea28b31253e14128efe8d8d5774e3256a547b3a946", + "filename": "MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..070f5a23b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/minicpmv-mix_q3gud_im-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 306.2016549629916 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md new file mode 100644 index 000000000..9a7f0028c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: ornith-1.5-35b-a3b-apex-i-compact +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 68.5 (66.1-79.2) | 0.37 (0.34-0.38) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1067.7 (1059.1-1071.4) | not measured | 3.77 (3.76-3.80) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 989.2 (986.8-992.4) | not measured | 16.25 (16.20-16.29) | +| agent session, follow-up turns | 15147 (15134-15152) at the last turn | see below | up to 64 | 3 | see below | 64.6 (57.4-67.7) | 4.15 (2.16-5.38) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.7 % (68.7-68.8); earlier-turn tokens read again per session: 259 (246-264) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.11 s (1.07-1.18). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 552.1 | 1.106 | 75.4 | +| 1 | 4182 | 602 | 3580 | 883.1 | 4.061 | 65.7 | +| 2 | 5820 | 4175 | 1645 | 764.9 | 2.157 | 67.4 | +| 3 | 9244 | 5813 | 3431 | 681.4 | 5.044 | 58.6 | +| 4 | 12213 | 9237 | 2976 | 723.6 | 4.123 | 64.6 | +| 5 | 15147 | 12206 | 2941 | 548.1 | 5.375 | 57.4 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 161 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json new file mode 100644 index 000000000..087b2beb4 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 161.41572618484497, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "ornith-1.5-35b-a3b-apex-i-compact", + "quant": null, + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "ornith-1.5-35b-a3b-apex-i-compact", + "display_name": "Ornith 1.5 35B-A3B APEX-I-COMPACT", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "ornith-1.5-35b-a3b-apex-i-compact", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 5412, + "expert_cache_mib": 7754, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1022, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.365, + "total_s": 1.878, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 140.3, + "prompt_tps_source": "engine", + "decode_tps": 66.1, + "decode_tps_source": "engine", + "prompt_ms": 363.4, + "decode_ms": 1513.7, + "drafts_offered": 54, + "drafts_accepted": 43, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.338, + "total_s": 1.797, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 127.7, + "prompt_tps_source": "engine", + "decode_tps": 68.5, + "decode_tps_source": "engine", + "prompt_ms": 336.6, + "decode_ms": 1459.1, + "drafts_offered": 41, + "drafts_accepted": 36, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.381, + "total_s": 1.878, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 121.1, + "prompt_tps_source": "engine", + "decode_tps": 66.8, + "decode_tps_source": "engine", + "prompt_ms": 379.9, + "decode_ms": 1496.8, + "drafts_offered": 35, + "drafts_accepted": 30, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.367, + "total_s": 1.631, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 139.5, + "prompt_tps_source": "engine", + "decode_tps": 79.1, + "decode_tps_source": "engine", + "prompt_ms": 365.6, + "decode_ms": 1264.2, + "drafts_offered": 52, + "drafts_accepted": 50, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.348, + "total_s": 1.757, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 124.0, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 346.8, + "decode_ms": 1408.3, + "drafts_offered": 41, + "drafts_accepted": 34, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.366, + "total_s": 1.86, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 126.2, + "prompt_tps_source": "engine", + "decode_tps": 66.9, + "decode_tps_source": "engine", + "prompt_ms": 364.6, + "decode_ms": 1493.9, + "drafts_offered": 33, + "drafts_accepted": 27, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.381, + "total_s": 1.643, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 134.2, + "prompt_tps_source": "engine", + "decode_tps": 79.2, + "decode_tps_source": "engine", + "prompt_ms": 379.9, + "decode_ms": 1261.9, + "drafts_offered": 50, + "drafts_accepted": 48, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.362, + "total_s": 1.802, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 119.2, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 360.8, + "decode_ms": 1440.1, + "drafts_offered": 40, + "drafts_accepted": 32, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.371, + "total_s": 1.85, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 124.4, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 369.9, + "decode_ms": 1478.6, + "drafts_offered": 38, + "drafts_accepted": 30, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 68.5, + "min": 66.1, + "max": 79.2, + "n": 9 + }, + "ttft_s": { + "median": 0.366, + "min": 0.338, + "max": 0.381, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.8, + "total_s": 4.163, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1059.1, + "prompt_tps_source": "engine", + "decode_tps": 66.0, + "decode_tps_source": "engine", + "prompt_ms": 3794.9, + "decode_ms": 363.7, + "drafts_offered": 10, + "drafts_accepted": 7, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.756, + "total_s": 4.119, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1071.4, + "prompt_tps_source": "engine", + "decode_tps": 66.1, + "decode_tps_source": "engine", + "prompt_ms": 3751.1, + "decode_ms": 362.9, + "drafts_offered": 7, + "drafts_accepted": 6, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.769, + "total_s": 4.136, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1067.7, + "prompt_tps_source": "engine", + "decode_tps": 65.3, + "decode_tps_source": "engine", + "prompt_ms": 3764.1, + "decode_ms": 367.7, + "drafts_offered": 10, + "drafts_accepted": 7, + "run": 2 + } + ], + "prompt_tps": { + "median": 1067.7, + "min": 1059.1, + "max": 1071.4, + "n": 3 + }, + "ttft_s": { + "median": 3.769, + "min": 3.756, + "max": 3.8, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.247, + "total_s": 16.653, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 989.2, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 16233.1, + "decode_ms": 406.1, + "drafts_offered": 8, + "drafts_accepted": 7, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.195, + "total_s": 16.608, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 992.4, + "prompt_tps_source": "engine", + "decode_tps": 58.0, + "decode_tps_source": "engine", + "prompt_ms": 16180.7, + "decode_ms": 413.6, + "drafts_offered": 6, + "drafts_accepted": 5, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.288, + "total_s": 16.689, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 986.8, + "prompt_tps_source": "engine", + "decode_tps": 59.8, + "decode_tps_source": "engine", + "prompt_ms": 16273.5, + "decode_ms": 401.6, + "drafts_offered": 7, + "drafts_accepted": 6, + "run": 2 + } + ], + "prompt_tps": { + "median": 989.2, + "min": 986.8, + "max": 992.4, + "n": 3 + }, + "ttft_s": { + "median": 16.247, + "min": 16.195, + "max": 16.288, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.106, + "total_s": 1.358, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 552.1, + "prompt_tps_source": "engine", + "decode_tps": 75.4, + "decode_tps_source": "engine", + "prompt_ms": 1103.1, + "decode_ms": 252.1, + "drafts_offered": 10, + "drafts_accepted": 8, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.061, + "total_s": 4.762, + "prompt_tokens": 4182, + "reused_tokens": 602, + "read_tokens": 3580, + "generated_tokens": 46, + "prompt_tps": 883.1, + "prompt_tps_source": "engine", + "decode_tps": 65.7, + "decode_tps_source": "engine", + "prompt_ms": 4053.8, + "decode_ms": 700.6, + "drafts_offered": 22, + "drafts_accepted": 16, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.157, + "total_s": 3.003, + "prompt_tokens": 5820, + "reused_tokens": 4175, + "read_tokens": 1645, + "generated_tokens": 57, + "prompt_tps": 764.9, + "prompt_tps_source": "engine", + "decode_tps": 67.4, + "decode_tps_source": "engine", + "prompt_ms": 2150.7, + "decode_ms": 845.7, + "drafts_offered": 29, + "drafts_accepted": 22, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.044, + "total_s": 5.931, + "prompt_tokens": 9244, + "reused_tokens": 5813, + "read_tokens": 3431, + "generated_tokens": 52, + "prompt_tps": 681.4, + "prompt_tps_source": "engine", + "decode_tps": 58.6, + "decode_tps_source": "engine", + "prompt_ms": 5035.1, + "decode_ms": 887.6, + "drafts_offered": 21, + "drafts_accepted": 14, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.123, + "total_s": 4.896, + "prompt_tokens": 12213, + "reused_tokens": 9237, + "read_tokens": 2976, + "generated_tokens": 50, + "prompt_tps": 723.6, + "prompt_tps_source": "engine", + "decode_tps": 64.6, + "decode_tps_source": "engine", + "prompt_ms": 4112.8, + "decode_ms": 773.6, + "drafts_offered": 25, + "drafts_accepted": 20, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.375, + "total_s": 6.489, + "prompt_tokens": 15147, + "reused_tokens": 12206, + "read_tokens": 2941, + "generated_tokens": 64, + "prompt_tps": 548.1, + "prompt_tps_source": "engine", + "decode_tps": 57.4, + "decode_tps_source": "engine", + "prompt_ms": 5365.6, + "decode_ms": 1114.4, + "drafts_offered": 34, + "drafts_accepted": 24, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.184, + "total_s": 1.448, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 515.4, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 1181.7, + "decode_ms": 264.0, + "drafts_offered": 10, + "drafts_accepted": 7, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.126, + "total_s": 4.791, + "prompt_tokens": 4182, + "reused_tokens": 602, + "read_tokens": 3580, + "generated_tokens": 45, + "prompt_tps": 868.8, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 4120.6, + "decode_ms": 665.4, + "drafts_offered": 23, + "drafts_accepted": 17, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.166, + "total_s": 2.972, + "prompt_tokens": 5819, + "reused_tokens": 4175, + "read_tokens": 1644, + "generated_tokens": 52, + "prompt_tps": 760.6, + "prompt_tps_source": "engine", + "decode_tps": 64.5, + "decode_tps_source": "engine", + "prompt_ms": 2161.4, + "decode_ms": 806.4, + "drafts_offered": 23, + "drafts_accepted": 17, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.928, + "total_s": 5.81, + "prompt_tokens": 9238, + "reused_tokens": 5812, + "read_tokens": 3426, + "generated_tokens": 56, + "prompt_tps": 696.1, + "prompt_tps_source": "engine", + "decode_tps": 63.5, + "decode_tps_source": "engine", + "prompt_ms": 4921.5, + "decode_ms": 881.6, + "drafts_offered": 22, + "drafts_accepted": 18, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.121, + "total_s": 5.002, + "prompt_tokens": 12211, + "reused_tokens": 9231, + "read_tokens": 2980, + "generated_tokens": 57, + "prompt_tps": 724.4, + "prompt_tps_source": "engine", + "decode_tps": 64.7, + "decode_tps_source": "engine", + "prompt_ms": 4113.5, + "decode_ms": 881.3, + "drafts_offered": 27, + "drafts_accepted": 23, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.314, + "total_s": 6.367, + "prompt_tokens": 15152, + "reused_tokens": 12204, + "read_tokens": 2948, + "generated_tokens": 64, + "prompt_tps": 555.6, + "prompt_tps_source": "engine", + "decode_tps": 60.8, + "decode_tps_source": "engine", + "prompt_ms": 5305.7, + "decode_ms": 1052.6, + "drafts_offered": 28, + "drafts_accepted": 22, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.074, + "total_s": 1.534, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 33, + "prompt_tps": 567.9, + "prompt_tps_source": "engine", + "decode_tps": 71.7, + "decode_tps_source": "engine", + "prompt_ms": 1072.4, + "decode_ms": 460.0, + "drafts_offered": 20, + "drafts_accepted": 14, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.152, + "total_s": 4.753, + "prompt_tokens": 4196, + "reused_tokens": 602, + "read_tokens": 3594, + "generated_tokens": 39, + "prompt_tps": 866.7, + "prompt_tps_source": "engine", + "decode_tps": 64.8, + "decode_tps_source": "engine", + "prompt_ms": 4146.6, + "decode_ms": 601.6, + "drafts_offered": 16, + "drafts_accepted": 12, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.166, + "total_s": 2.934, + "prompt_tokens": 5827, + "reused_tokens": 4189, + "read_tokens": 1638, + "generated_tokens": 52, + "prompt_tps": 757.8, + "prompt_tps_source": "engine", + "decode_tps": 67.7, + "decode_tps_source": "engine", + "prompt_ms": 2161.5, + "decode_ms": 768.0, + "drafts_offered": 26, + "drafts_accepted": 20, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.031, + "total_s": 5.741, + "prompt_tokens": 9246, + "reused_tokens": 5820, + "read_tokens": 3426, + "generated_tokens": 44, + "prompt_tps": 681.9, + "prompt_tps_source": "engine", + "decode_tps": 61.9, + "decode_tps_source": "engine", + "prompt_ms": 5023.9, + "decode_ms": 711.2, + "drafts_offered": 20, + "drafts_accepted": 14, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.157, + "total_s": 4.816, + "prompt_tokens": 12207, + "reused_tokens": 9239, + "read_tokens": 2968, + "generated_tokens": 43, + "prompt_tps": 715.3, + "prompt_tps_source": "engine", + "decode_tps": 65.2, + "decode_tps_source": "engine", + "prompt_ms": 4149.1, + "decode_ms": 659.6, + "drafts_offered": 20, + "drafts_accepted": 17, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.345, + "total_s": 6.189, + "prompt_tokens": 15134, + "reused_tokens": 12200, + "read_tokens": 2934, + "generated_tokens": 51, + "prompt_tps": 549.8, + "prompt_tps_source": "engine", + "decode_tps": 60.4, + "decode_tps_source": "engine", + "prompt_ms": 5336.7, + "decode_ms": 844.1, + "drafts_offered": 20, + "drafts_accepted": 16, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.731, + "min": 68.718, + "max": 68.762, + "n": 3 + }, + "reread_tokens": { + "median": 259, + "min": 246, + "max": 264, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.152, + "min": 2.157, + "max": 5.375, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15147, + "min": 15134, + "max": 15152, + "n": 3 + }, + "turn_decode_tps": { + "median": 64.6, + "min": 57.4, + "max": 67.7, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.106, + "min": 1.074, + "max": 1.184, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..b736c468c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 66.1 tok/s, first token 0.365 s, 100 tokens + run 1 q2: decode 68.5 tok/s, first token 0.338 s, 100 tokens + run 1 q3: decode 66.8 tok/s, first token 0.381 s, 100 tokens + run 2 q1: decode 79.1 tok/s, first token 0.367 s, 100 tokens + run 2 q2: decode 71.0 tok/s, first token 0.348 s, 100 tokens + run 2 q3: decode 66.9 tok/s, first token 0.366 s, 100 tokens + run 3 q1: decode 79.2 tok/s, first token 0.381 s, 100 tokens + run 3 q2: decode 69.4 tok/s, first token 0.362 s, 100 tokens + run 3 q3: decode 67.6 tok/s, first token 0.371 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1059.1 tok/s (0 reused), first token 3.8 s + run 2: 4019 tokens, prompt 1071.4 tok/s (0 reused), first token 3.756 s + run 3: 4019 tokens, prompt 1067.7 tok/s (0 reused), first token 3.769 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 989.2 tok/s (0 reused), first token 16.247 s + run 2: 16058 tokens, prompt 992.4 tok/s (0 reused), first token 16.195 s + run 3: 16058 tokens, prompt 986.8 tok/s (0 reused), first token 16.288 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.106 s + run 1 turn 1: prompt 4182, reused 602, read 3580, first token 4.061 s + run 1 turn 2: prompt 5820, reused 4175, read 1645, first token 2.157 s + run 1 turn 3: prompt 9244, reused 5813, read 3431, first token 5.044 s + run 1 turn 4: prompt 12213, reused 9237, read 2976, first token 4.123 s + run 1 turn 5: prompt 15147, reused 12206, read 2941, first token 5.375 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.184 s + run 2 turn 1: prompt 4182, reused 602, read 3580, first token 4.126 s + run 2 turn 2: prompt 5819, reused 4175, read 1644, first token 2.166 s + run 2 turn 3: prompt 9238, reused 5812, read 3426, first token 4.928 s + run 2 turn 4: prompt 12211, reused 9231, read 2980, first token 4.121 s + run 2 turn 5: prompt 15152, reused 12204, read 2948, first token 5.314 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.074 s + run 3 turn 1: prompt 4196, reused 602, read 3594, first token 4.152 s + run 3 turn 2: prompt 5827, reused 4189, read 1638, first token 2.166 s + run 3 turn 3: prompt 9246, reused 5820, read 3426, first token 5.031 s + run 3 turn 4: prompt 12207, reused 9239, read 2968, first token 4.157 s + run 3 turn 5: prompt 15134, reused 12200, read 2934, first token 5.345 s + +Results (median, range over the runs): + chat decode tok/s 68.50 (66.10 - 79.20, n=9) + chat first token (s) 0.37 (0.34 - 0.38, n=9) + 4K prompt tok/s 1067.70 (1059.10 - 1071.40, n=3) + 4K first token (s) 3.77 (3.76 - 3.80, n=3) + 16K prompt tok/s 989.20 (986.80 - 992.40, n=3) + 16K first token (s) 16.25 (16.20 - 16.29, n=3) + agent cache hit % 68.73 (68.72 - 68.76, n=3) + agent re-read tokens/session 259.00 (246.00 - 264.00, n=3) + agent turn first token (s) 4.15 (2.16 - 5.38, n=15) + agent decode tok/s 64.60 (57.40 - 67.70, n=15) + +Report: /opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..b1e9989f9 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/ornith-APEX-I-COMPACT/Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/ornith-apex-i-compact-reserve1024-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/ornith-apex-i-compact/tokenizer", + "model_name": "ornith-1.5-35b-a3b-apex-i-compact", + "display_name": "Ornith 1.5 35B-A3B APEX-I-COMPACT", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17437861408, + "sha256": "67e6cdff678e67180d3cd4eef55277277563319057de287b8855c4d1fbff4165", + "filename": "Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..cae116aaf --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 161.6403184319788 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md new file mode 100644 index 000000000..606a7682b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: ornith-1.5-35b-a3b-apex-i-compact +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 69.2 (62.6-77.8) | 0.36 (0.35-0.40) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1047.5 (1040.0-1047.5) | not measured | 3.84 (3.84-3.87) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 991.5 (988.3-991.7) | not measured | 16.21 (16.21-16.26) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 781.9 (776.0-782.8) | not measured | 41.02 (40.97-41.33) | +| agent session, follow-up turns | 15155 (15128-15164) at the last turn | see below | up to 64 | 3 | see below | 64.5 (54.7-71.0) | 4.18 (2.17-5.39) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.7 % (68.7-68.8); earlier-turn tokens read again per session: 267 (240-276) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.15 s (1.14-1.26). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 539.8 | 1.138 | 69.3 | +| 1 | 4187 | 602 | 3585 | 857.3 | 4.19 | 67.7 | +| 2 | 5825 | 4180 | 1645 | 760.0 | 2.171 | 68.2 | +| 3 | 9249 | 5818 | 3431 | 668.1 | 5.145 | 60.4 | +| 4 | 12218 | 9242 | 2976 | 724.4 | 4.118 | 66.7 | +| 5 | 15155 | 12211 | 2944 | 552.1 | 5.342 | 54.7 | + +### Concurrency: 2 clients + +Total 52.8 tok/s (52.4-53.1); first token p50 1.674 s, p95 2.908 s; request latency p50 3.665 s, p95 4.878 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 302 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json new file mode 100644 index 000000000..596a8ff2c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/results.json @@ -0,0 +1,1032 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 302.3506438732147, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "ornith-1.5-35b-a3b-apex-i-compact", + "quant": null, + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "ornith-1.5-35b-a3b-apex-i-compact", + "display_name": "Ornith 1.5 35B-A3B APEX-I-COMPACT", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "ornith-1.5-35b-a3b-apex-i-compact", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 4920, + "expert_cache_mib": 7049, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1022, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.355, + "total_s": 1.8, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 144.3, + "prompt_tps_source": "engine", + "decode_tps": 69.2, + "decode_tps_source": "engine", + "prompt_ms": 353.4, + "decode_ms": 1445.1, + "drafts_offered": 55, + "drafts_accepted": 46, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.357, + "total_s": 1.924, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 121.0, + "prompt_tps_source": "engine", + "decode_tps": 63.8, + "decode_tps_source": "engine", + "prompt_ms": 355.5, + "decode_ms": 1566.8, + "drafts_offered": 36, + "drafts_accepted": 29, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.365, + "total_s": 1.961, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 126.6, + "prompt_tps_source": "engine", + "decode_tps": 62.6, + "decode_tps_source": "engine", + "prompt_ms": 363.4, + "decode_ms": 1596.2, + "drafts_offered": 38, + "drafts_accepted": 27, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.37, + "total_s": 1.692, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 138.6, + "prompt_tps_source": "engine", + "decode_tps": 75.6, + "decode_tps_source": "engine", + "prompt_ms": 368.0, + "decode_ms": 1322.6, + "drafts_offered": 55, + "drafts_accepted": 46, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.357, + "total_s": 1.776, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 121.0, + "prompt_tps_source": "engine", + "decode_tps": 70.5, + "decode_tps_source": "engine", + "prompt_ms": 355.3, + "decode_ms": 1419.3, + "drafts_offered": 43, + "drafts_accepted": 35, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.372, + "total_s": 1.875, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 124.1, + "prompt_tps_source": "engine", + "decode_tps": 66.5, + "decode_tps_source": "engine", + "prompt_ms": 370.6, + "decode_ms": 1502.7, + "drafts_offered": 38, + "drafts_accepted": 29, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.402, + "total_s": 1.688, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 127.5, + "prompt_tps_source": "engine", + "decode_tps": 77.8, + "decode_tps_source": "engine", + "prompt_ms": 400.1, + "decode_ms": 1286.1, + "drafts_offered": 52, + "drafts_accepted": 49, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.351, + "total_s": 1.793, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 122.9, + "prompt_tps_source": "engine", + "decode_tps": 69.3, + "decode_tps_source": "engine", + "prompt_ms": 349.8, + "decode_ms": 1442.0, + "drafts_offered": 38, + "drafts_accepted": 31, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.379, + "total_s": 1.923, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 121.8, + "prompt_tps_source": "engine", + "decode_tps": 64.8, + "decode_tps_source": "engine", + "prompt_ms": 377.6, + "decode_ms": 1544.0, + "drafts_offered": 33, + "drafts_accepted": 25, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 69.2, + "min": 62.6, + "max": 77.8, + "n": 9 + }, + "ttft_s": { + "median": 0.365, + "min": 0.351, + "max": 0.402, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.87, + "total_s": 4.217, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1040.0, + "prompt_tps_source": "engine", + "decode_tps": 69.0, + "decode_tps_source": "engine", + "prompt_ms": 3864.6, + "decode_ms": 348.0, + "drafts_offered": 10, + "drafts_accepted": 8, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.842, + "total_s": 4.198, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1047.5, + "prompt_tps_source": "engine", + "decode_tps": 67.4, + "decode_tps_source": "engine", + "prompt_ms": 3836.9, + "decode_ms": 356.1, + "drafts_offered": 6, + "drafts_accepted": 6, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.842, + "total_s": 4.18, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1047.5, + "prompt_tps_source": "engine", + "decode_tps": 70.8, + "decode_tps_source": "engine", + "prompt_ms": 3836.8, + "decode_ms": 338.8, + "drafts_offered": 11, + "drafts_accepted": 9, + "run": 2 + } + ], + "prompt_tps": { + "median": 1047.5, + "min": 1040.0, + "max": 1047.5, + "n": 3 + }, + "ttft_s": { + "median": 3.842, + "min": 3.842, + "max": 3.87, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 16.262, + "total_s": 16.656, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 988.3, + "prompt_tps_source": "engine", + "decode_tps": 60.9, + "decode_tps_source": "engine", + "prompt_ms": 16247.7, + "decode_ms": 394.0, + "drafts_offered": 9, + "drafts_accepted": 7, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.207, + "total_s": 16.633, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 991.7, + "prompt_tps_source": "engine", + "decode_tps": 56.2, + "decode_tps_source": "engine", + "prompt_ms": 16191.7, + "decode_ms": 426.8, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 16.212, + "total_s": 16.607, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 991.5, + "prompt_tps_source": "engine", + "decode_tps": 60.5, + "decode_tps_source": "engine", + "prompt_ms": 16196.3, + "decode_ms": 396.5, + "drafts_offered": 9, + "drafts_accepted": 7, + "run": 2 + } + ], + "prompt_tps": { + "median": 991.5, + "min": 988.3, + "max": 991.7, + "n": 3 + }, + "ttft_s": { + "median": 16.212, + "min": 16.207, + "max": 16.262, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 41.328, + "total_s": 41.773, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 776.0, + "prompt_tps_source": "engine", + "decode_tps": 53.8, + "decode_tps_source": "engine", + "prompt_ms": 41300.5, + "decode_ms": 446.2, + "drafts_offered": 9, + "drafts_accepted": 7, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 41.018, + "total_s": 41.451, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 781.9, + "prompt_tps_source": "engine", + "decode_tps": 55.4, + "decode_tps_source": "engine", + "prompt_ms": 40991.0, + "decode_ms": 433.4, + "drafts_offered": 10, + "drafts_accepted": 8, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 40.969, + "total_s": 41.405, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 782.8, + "prompt_tps_source": "engine", + "decode_tps": 54.9, + "decode_tps_source": "engine", + "prompt_ms": 40941.2, + "decode_ms": 437.1, + "drafts_offered": 10, + "drafts_accepted": 9, + "run": 2 + } + ], + "prompt_tps": { + "median": 781.9, + "min": 776.0, + "max": 782.8, + "n": 3 + }, + "ttft_s": { + "median": 41.018, + "min": 40.969, + "max": 41.328, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.138, + "total_s": 1.484, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 539.8, + "prompt_tps_source": "engine", + "decode_tps": 69.3, + "decode_tps_source": "engine", + "prompt_ms": 1128.2, + "decode_ms": 346.5, + "drafts_offered": 12, + "drafts_accepted": 9, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.19, + "total_s": 4.869, + "prompt_tokens": 4187, + "reused_tokens": 602, + "read_tokens": 3585, + "generated_tokens": 46, + "prompt_tps": 857.3, + "prompt_tps_source": "engine", + "decode_tps": 67.7, + "decode_tps_source": "engine", + "prompt_ms": 4181.9, + "decode_ms": 679.6, + "drafts_offered": 22, + "drafts_accepted": 17, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.171, + "total_s": 3.006, + "prompt_tokens": 5825, + "reused_tokens": 4180, + "read_tokens": 1645, + "generated_tokens": 57, + "prompt_tps": 760.0, + "prompt_tps_source": "engine", + "decode_tps": 68.2, + "decode_tps_source": "engine", + "prompt_ms": 2164.5, + "decode_ms": 835.5, + "drafts_offered": 25, + "drafts_accepted": 21, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.145, + "total_s": 6.005, + "prompt_tokens": 9249, + "reused_tokens": 5818, + "read_tokens": 3431, + "generated_tokens": 52, + "prompt_tps": 668.1, + "prompt_tps_source": "engine", + "decode_tps": 60.4, + "decode_tps_source": "engine", + "prompt_ms": 5135.5, + "decode_ms": 861.1, + "drafts_offered": 23, + "drafts_accepted": 16, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.118, + "total_s": 4.912, + "prompt_tokens": 12218, + "reused_tokens": 9242, + "read_tokens": 2976, + "generated_tokens": 53, + "prompt_tps": 724.4, + "prompt_tps_source": "engine", + "decode_tps": 66.7, + "decode_tps_source": "engine", + "prompt_ms": 4108.1, + "decode_ms": 794.3, + "drafts_offered": 27, + "drafts_accepted": 22, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.342, + "total_s": 6.512, + "prompt_tokens": 15155, + "reused_tokens": 12211, + "read_tokens": 2944, + "generated_tokens": 64, + "prompt_tps": 552.1, + "prompt_tps_source": "engine", + "decode_tps": 54.7, + "decode_tps_source": "engine", + "prompt_ms": 5332.8, + "decode_ms": 1170.7, + "drafts_offered": 35, + "drafts_accepted": 22, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.148, + "total_s": 1.476, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 24, + "prompt_tps": 531.8, + "prompt_tps_source": "engine", + "decode_tps": 73.0, + "decode_tps_source": "engine", + "prompt_ms": 1145.2, + "decode_ms": 328.9, + "drafts_offered": 12, + "drafts_accepted": 9, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.144, + "total_s": 4.691, + "prompt_tokens": 4187, + "reused_tokens": 602, + "read_tokens": 3585, + "generated_tokens": 38, + "prompt_tps": 866.3, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 4138.5, + "decode_ms": 547.6, + "drafts_offered": 20, + "drafts_accepted": 15, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.195, + "total_s": 2.902, + "prompt_tokens": 5817, + "reused_tokens": 4180, + "read_tokens": 1637, + "generated_tokens": 48, + "prompt_tps": 747.5, + "prompt_tps_source": "engine", + "decode_tps": 67.8, + "decode_tps_source": "engine", + "prompt_ms": 2189.9, + "decode_ms": 708.1, + "drafts_offered": 21, + "drafts_accepted": 17, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.041, + "total_s": 5.811, + "prompt_tokens": 9232, + "reused_tokens": 5810, + "read_tokens": 3422, + "generated_tokens": 45, + "prompt_tps": 679.8, + "prompt_tps_source": "engine", + "decode_tps": 58.3, + "decode_tps_source": "engine", + "prompt_ms": 5033.8, + "decode_ms": 771.3, + "drafts_offered": 21, + "drafts_accepted": 12, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.119, + "total_s": 4.956, + "prompt_tokens": 12194, + "reused_tokens": 9225, + "read_tokens": 2969, + "generated_tokens": 50, + "prompt_tps": 722.1, + "prompt_tps_source": "engine", + "decode_tps": 59.7, + "decode_tps_source": "engine", + "prompt_ms": 4111.8, + "decode_ms": 838.0, + "drafts_offered": 24, + "drafts_accepted": 16, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.376, + "total_s": 6.123, + "prompt_tokens": 15128, + "reused_tokens": 12187, + "read_tokens": 2941, + "generated_tokens": 46, + "prompt_tps": 547.9, + "prompt_tps_source": "engine", + "decode_tps": 61.5, + "decode_tps_source": "engine", + "prompt_ms": 5367.8, + "decode_ms": 747.4, + "drafts_offered": 19, + "drafts_accepted": 15, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.259, + "total_s": 1.831, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 39, + "prompt_tps": 490.5, + "prompt_tps_source": "engine", + "decode_tps": 68.1, + "decode_tps_source": "engine", + "prompt_ms": 1241.6, + "decode_ms": 572.6, + "drafts_offered": 22, + "drafts_accepted": 15, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.179, + "total_s": 4.827, + "prompt_tokens": 4202, + "reused_tokens": 602, + "read_tokens": 3600, + "generated_tokens": 46, + "prompt_tps": 862.5, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 4174.1, + "decode_ms": 647.6, + "drafts_offered": 20, + "drafts_accepted": 18, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.203, + "total_s": 3.009, + "prompt_tokens": 5840, + "reused_tokens": 4195, + "read_tokens": 1645, + "generated_tokens": 52, + "prompt_tps": 748.4, + "prompt_tps_source": "engine", + "decode_tps": 64.5, + "decode_tps_source": "engine", + "prompt_ms": 2198.1, + "decode_ms": 806.4, + "drafts_offered": 21, + "drafts_accepted": 16, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.253, + "total_s": 6.056, + "prompt_tokens": 9259, + "reused_tokens": 5833, + "read_tokens": 3426, + "generated_tokens": 51, + "prompt_tps": 653.0, + "prompt_tps_source": "engine", + "decode_tps": 63.5, + "decode_tps_source": "engine", + "prompt_ms": 5246.5, + "decode_ms": 803.7, + "drafts_offered": 22, + "drafts_accepted": 17, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.128, + "total_s": 4.943, + "prompt_tokens": 12227, + "reused_tokens": 9252, + "read_tokens": 2975, + "generated_tokens": 53, + "prompt_tps": 722.1, + "prompt_tps_source": "engine", + "decode_tps": 65.0, + "decode_tps_source": "engine", + "prompt_ms": 4119.9, + "decode_ms": 815.8, + "drafts_offered": 25, + "drafts_accepted": 21, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.395, + "total_s": 6.544, + "prompt_tokens": 15164, + "reused_tokens": 12220, + "read_tokens": 2944, + "generated_tokens": 64, + "prompt_tps": 546.6, + "prompt_tps_source": "engine", + "decode_tps": 55.7, + "decode_tps_source": "engine", + "prompt_ms": 5386.5, + "decode_ms": 1149.2, + "drafts_offered": 31, + "drafts_accepted": 20, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.74, + "min": 68.733, + "max": 68.753, + "n": 3 + }, + "reread_tokens": { + "median": 267, + "min": 240, + "max": 276, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.179, + "min": 2.171, + "max": 5.395, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15155, + "min": 15128, + "max": 15164, + "n": 3 + }, + "turn_decode_tps": { + "median": 64.5, + "min": 54.7, + "max": 71.0, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.148, + "min": 1.138, + "max": 1.259, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 4.89, + "tokens": 256, + "total_tps": 52.38, + "first_s": [ + 0.47872909301077016, + 2.8879800259892363 + ], + "latency_s": [ + 2.487306553986855, + 4.887098115985282 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 4.82, + "tokens": 256, + "total_tps": 53.07, + "first_s": [ + 0.4728545160032809, + 2.866529890015954 + ], + "latency_s": [ + 2.41560353001114, + 4.8235634110169485 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 4.85, + "tokens": 256, + "total_tps": 52.77, + "first_s": [ + 0.4817309769860003, + 2.9150642169988714 + ], + "latency_s": [ + 2.506889346987009, + 4.85097673299606 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 52.77, + "min": 52.38, + "max": 53.07, + "n": 3 + }, + "ttft_p50_s": 1.674, + "ttft_p95_s": 2.908, + "latency_p50_s": 3.665, + "latency_p95_s": 4.878, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..7fcc0ac20 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 69.2 tok/s, first token 0.355 s, 100 tokens + run 1 q2: decode 63.8 tok/s, first token 0.357 s, 100 tokens + run 1 q3: decode 62.6 tok/s, first token 0.365 s, 100 tokens + run 2 q1: decode 75.6 tok/s, first token 0.37 s, 100 tokens + run 2 q2: decode 70.5 tok/s, first token 0.357 s, 100 tokens + run 2 q3: decode 66.5 tok/s, first token 0.372 s, 100 tokens + run 3 q1: decode 77.8 tok/s, first token 0.402 s, 100 tokens + run 3 q2: decode 69.3 tok/s, first token 0.351 s, 100 tokens + run 3 q3: decode 64.8 tok/s, first token 0.379 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1040.0 tok/s (0 reused), first token 3.87 s + run 2: 4019 tokens, prompt 1047.5 tok/s (0 reused), first token 3.842 s + run 3: 4019 tokens, prompt 1047.5 tok/s (0 reused), first token 3.842 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 988.3 tok/s (0 reused), first token 16.262 s + run 2: 16058 tokens, prompt 991.7 tok/s (0 reused), first token 16.207 s + run 3: 16058 tokens, prompt 991.5 tok/s (0 reused), first token 16.212 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.138 s + run 1 turn 1: prompt 4187, reused 602, read 3585, first token 4.19 s + run 1 turn 2: prompt 5825, reused 4180, read 1645, first token 2.171 s + run 1 turn 3: prompt 9249, reused 5818, read 3431, first token 5.145 s + run 1 turn 4: prompt 12218, reused 9242, read 2976, first token 4.118 s + run 1 turn 5: prompt 15155, reused 12211, read 2944, first token 5.342 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.148 s + run 2 turn 1: prompt 4187, reused 602, read 3585, first token 4.144 s + run 2 turn 2: prompt 5817, reused 4180, read 1637, first token 2.195 s + run 2 turn 3: prompt 9232, reused 5810, read 3422, first token 5.041 s + run 2 turn 4: prompt 12194, reused 9225, read 2969, first token 4.119 s + run 2 turn 5: prompt 15128, reused 12187, read 2941, first token 5.376 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.259 s + run 3 turn 1: prompt 4202, reused 602, read 3600, first token 4.179 s + run 3 turn 2: prompt 5840, reused 4195, read 1645, first token 2.203 s + run 3 turn 3: prompt 9259, reused 5833, read 3426, first token 5.253 s + run 3 turn 4: prompt 12227, reused 9252, read 2975, first token 4.128 s + run 3 turn 5: prompt 15164, reused 12220, read 2944, first token 5.395 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 4.9 s = 52.4 tok/s total + round 2: 256 tokens in 4.8 s = 53.1 tok/s total + round 3: 256 tokens in 4.9 s = 52.8 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32049 tokens, prompt 776.0 tok/s (0 reused), first token 41.328 s + run 2: 32049 tokens, prompt 781.9 tok/s (0 reused), first token 41.018 s + run 3: 32049 tokens, prompt 782.8 tok/s (0 reused), first token 40.969 s + +Results (median, range over the runs): + chat decode tok/s 69.20 (62.60 - 77.80, n=9) + chat first token (s) 0.36 (0.35 - 0.40, n=9) + 4K prompt tok/s 1047.50 (1040.00 - 1047.50, n=3) + 4K first token (s) 3.84 (3.84 - 3.87, n=3) + 16K prompt tok/s 991.50 (988.30 - 991.70, n=3) + 16K first token (s) 16.21 (16.21 - 16.26, n=3) + 32K prompt tok/s 781.90 (776.00 - 782.80, n=3) + 32K first token (s) 41.02 (40.97 - 41.33, n=3) + agent cache hit % 68.74 (68.73 - 68.75, n=3) + agent re-read tokens/session 267.00 (240.00 - 276.00, n=3) + agent turn first token (s) 4.18 (2.17 - 5.39, n=15) + agent decode tok/s 64.50 (54.70 - 71.00, n=15) + 2 clients total tok/s 52.77 (52.38 - 53.07, n=3) + +Report: /opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/ornith-apex-i-compact-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-ornith-1-5-35b-a3b-apex-i-compact (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..89a0bd488 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/ornith-APEX-I-COMPACT/Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/ornith-apex-i-compact-reserve1024-ctx65536/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/ornith-apex-i-compact/tokenizer", + "model_name": "ornith-1.5-35b-a3b-apex-i-compact", + "display_name": "Ornith 1.5 35B-A3B APEX-I-COMPACT", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17437861408, + "sha256": "67e6cdff678e67180d3cd4eef55277277563319057de287b8855c4d1fbff4165", + "filename": "Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..be253c450 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/ornith-apex-i-compact-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 302.56188049999764 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md new file mode 100644 index 000000000..1744b1a74 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q3_k_m +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 78.2 (64.4-97.5) | 0.39 (0.34-0.41) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1211.9 (1205.6-1216.4) | not measured | 3.32 (3.31-3.34) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1103.6 (1092.4-1104.2) | not measured | 14.56 (14.56-14.72) | +| agent session, follow-up turns | 15139 (15126-15147) at the last turn | see below | up to 64 | 3 | see below | 71.1 (59.9-74.4) | 3.73 (1.90-4.87) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.7-68.8); earlier-turn tokens read again per session: 251 (238-259) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.08 s (1.08-1.09). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 562.6 | 1.086 | 71.4 | +| 1 | 4189 | 602 | 3587 | 989.5 | 3.632 | 72.3 | +| 2 | 5825 | 4182 | 1643 | 861.3 | 1.914 | 69.8 | +| 3 | 9245 | 5818 | 3427 | 775.2 | 4.43 | 71.7 | +| 4 | 12197 | 9238 | 2959 | 789.3 | 3.758 | 74.4 | +| 5 | 15126 | 12190 | 2936 | 609.1 | 4.829 | 64.4 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 145 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json new file mode 100644 index 000000000..97e44a032 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 145.34016704559326, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "quant": "UD-Q3_K_M", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8" + ], + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 5766, + "expert_cache_mib": 7691, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 382, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.389, + "total_s": 1.574, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 131.9, + "prompt_tps_source": "engine", + "decode_tps": 84.3, + "decode_tps_source": "engine", + "prompt_ms": 386.8, + "decode_ms": 1185.6, + "drafts_offered": 77, + "drafts_accepted": 68, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.343, + "total_s": 1.896, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 125.8, + "prompt_tps_source": "engine", + "decode_tps": 64.4, + "decode_tps_source": "engine", + "prompt_ms": 341.7, + "decode_ms": 1552.6, + "drafts_offered": 70, + "drafts_accepted": 48, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.39, + "total_s": 1.714, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 118.4, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 388.6, + "decode_ms": 1324.1, + "drafts_offered": 66, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.404, + "total_s": 1.467, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 126.6, + "prompt_tps_source": "engine", + "decode_tps": 94.1, + "decode_tps_source": "engine", + "prompt_ms": 402.9, + "decode_ms": 1062.3, + "drafts_offered": 73, + "drafts_accepted": 69, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.39, + "total_s": 1.646, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 110.8, + "prompt_tps_source": "engine", + "decode_tps": 79.6, + "decode_tps_source": "engine", + "prompt_ms": 388.0, + "decode_ms": 1256.1, + "drafts_offered": 69, + "drafts_accepted": 59, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.366, + "total_s": 1.661, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 126.1, + "prompt_tps_source": "engine", + "decode_tps": 77.2, + "decode_tps_source": "engine", + "prompt_ms": 364.7, + "decode_ms": 1295.3, + "drafts_offered": 69, + "drafts_accepted": 56, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.396, + "total_s": 1.422, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 129.3, + "prompt_tps_source": "engine", + "decode_tps": 97.5, + "decode_tps_source": "engine", + "prompt_ms": 394.3, + "decode_ms": 1025.8, + "drafts_offered": 75, + "drafts_accepted": 73, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.407, + "total_s": 1.685, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 106.0, + "prompt_tps_source": "engine", + "decode_tps": 78.2, + "decode_tps_source": "engine", + "prompt_ms": 405.5, + "decode_ms": 1278.2, + "drafts_offered": 69, + "drafts_accepted": 57, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.401, + "total_s": 1.689, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 115.3, + "prompt_tps_source": "engine", + "decode_tps": 77.6, + "decode_tps_source": "engine", + "prompt_ms": 399.0, + "decode_ms": 1288.4, + "drafts_offered": 67, + "drafts_accepted": 56, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 78.2, + "min": 64.4, + "max": 97.5, + "n": 9 + }, + "ttft_s": { + "median": 0.39, + "min": 0.343, + "max": 0.407, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.339, + "total_s": 3.717, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1205.6, + "prompt_tps_source": "engine", + "decode_tps": 63.4, + "decode_tps_source": "engine", + "prompt_ms": 3333.6, + "decode_ms": 378.3, + "drafts_offered": 20, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.321, + "total_s": 3.637, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1211.9, + "prompt_tps_source": "engine", + "decode_tps": 75.9, + "decode_tps_source": "engine", + "prompt_ms": 3316.3, + "decode_ms": 316.4, + "drafts_offered": 17, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.309, + "total_s": 3.621, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1216.4, + "prompt_tps_source": "engine", + "decode_tps": 76.9, + "decode_tps_source": "engine", + "prompt_ms": 3304.1, + "decode_ms": 312.2, + "drafts_offered": 16, + "drafts_accepted": 14, + "run": 2 + } + ], + "prompt_tps": { + "median": 1211.9, + "min": 1205.6, + "max": 1216.4, + "n": 3 + }, + "ttft_s": { + "median": 3.321, + "min": 3.309, + "max": 3.339, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 14.725, + "total_s": 15.156, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1092.4, + "prompt_tps_source": "engine", + "decode_tps": 55.7, + "decode_tps_source": "engine", + "prompt_ms": 14700.3, + "decode_ms": 430.9, + "drafts_offered": 19, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.557, + "total_s": 14.981, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1104.2, + "prompt_tps_source": "engine", + "decode_tps": 56.6, + "decode_tps_source": "engine", + "prompt_ms": 14543.2, + "decode_ms": 423.9, + "drafts_offered": 18, + "drafts_accepted": 11, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.565, + "total_s": 15.013, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1103.6, + "prompt_tps_source": "engine", + "decode_tps": 53.6, + "decode_tps_source": "engine", + "prompt_ms": 14551.0, + "decode_ms": 447.8, + "drafts_offered": 21, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 1103.6, + "min": 1092.4, + "max": 1104.2, + "n": 3 + }, + "ttft_s": { + "median": 14.565, + "min": 14.557, + "max": 14.725, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.086, + "total_s": 1.45, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 26, + "prompt_tps": 562.6, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 1082.5, + "decode_ms": 364.1, + "drafts_offered": 23, + "drafts_accepted": 14, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.632, + "total_s": 4.241, + "prompt_tokens": 4189, + "reused_tokens": 602, + "read_tokens": 3587, + "generated_tokens": 44, + "prompt_tps": 989.5, + "prompt_tps_source": "engine", + "decode_tps": 72.3, + "decode_tps_source": "engine", + "prompt_ms": 3624.9, + "decode_ms": 608.9, + "drafts_offered": 33, + "drafts_accepted": 24, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.914, + "total_s": 2.673, + "prompt_tokens": 5825, + "reused_tokens": 4182, + "read_tokens": 1643, + "generated_tokens": 53, + "prompt_tps": 861.3, + "prompt_tps_source": "engine", + "decode_tps": 69.8, + "decode_tps_source": "engine", + "prompt_ms": 1907.5, + "decode_ms": 759.0, + "drafts_offered": 40, + "drafts_accepted": 29, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.43, + "total_s": 4.917, + "prompt_tokens": 9245, + "reused_tokens": 5818, + "read_tokens": 3427, + "generated_tokens": 35, + "prompt_tps": 775.2, + "prompt_tps_source": "engine", + "decode_tps": 71.7, + "decode_tps_source": "engine", + "prompt_ms": 4421.0, + "decode_ms": 488.0, + "drafts_offered": 21, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.758, + "total_s": 4.363, + "prompt_tokens": 12197, + "reused_tokens": 9238, + "read_tokens": 2959, + "generated_tokens": 45, + "prompt_tps": 789.3, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 3748.7, + "decode_ms": 605.0, + "drafts_offered": 30, + "drafts_accepted": 26, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.829, + "total_s": 5.729, + "prompt_tokens": 15126, + "reused_tokens": 12190, + "read_tokens": 2936, + "generated_tokens": 58, + "prompt_tps": 609.1, + "prompt_tps_source": "engine", + "decode_tps": 64.4, + "decode_tps_source": "engine", + "prompt_ms": 4819.9, + "decode_ms": 900.5, + "drafts_offered": 41, + "drafts_accepted": 30, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.085, + "total_s": 1.217, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 562.5, + "prompt_tps_source": "engine", + "decode_tps": 83.0, + "decode_tps_source": "engine", + "prompt_ms": 1082.7, + "decode_ms": 132.5, + "drafts_offered": 8, + "drafts_accepted": 6, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.652, + "total_s": 4.323, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 47, + "prompt_tps": 979.3, + "prompt_tps_source": "engine", + "decode_tps": 70.1, + "decode_tps_source": "engine", + "prompt_ms": 3647.4, + "decode_ms": 670.4, + "drafts_offered": 29, + "drafts_accepted": 22, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.9, + "total_s": 2.745, + "prompt_tokens": 5813, + "reused_tokens": 4167, + "read_tokens": 1646, + "generated_tokens": 61, + "prompt_tps": 868.2, + "prompt_tps_source": "engine", + "decode_tps": 72.2, + "decode_tps_source": "engine", + "prompt_ms": 1895.8, + "decode_ms": 844.6, + "drafts_offered": 39, + "drafts_accepted": 32, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.472, + "total_s": 5.194, + "prompt_tokens": 9241, + "reused_tokens": 5806, + "read_tokens": 3435, + "generated_tokens": 52, + "prompt_tps": 769.2, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 4465.8, + "decode_ms": 722.4, + "drafts_offered": 33, + "drafts_accepted": 28, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.727, + "total_s": 4.511, + "prompt_tokens": 12210, + "reused_tokens": 9234, + "read_tokens": 2976, + "generated_tokens": 53, + "prompt_tps": 800.0, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 3720.1, + "decode_ms": 784.3, + "drafts_offered": 37, + "drafts_accepted": 28, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.858, + "total_s": 5.649, + "prompt_tokens": 15147, + "reused_tokens": 12203, + "read_tokens": 2944, + "generated_tokens": 49, + "prompt_tps": 607.0, + "prompt_tps_source": "engine", + "decode_tps": 62.0, + "decode_tps_source": "engine", + "prompt_ms": 4850.2, + "decode_ms": 790.6, + "drafts_offered": 37, + "drafts_accepted": 25, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.081, + "total_s": 1.498, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 34, + "prompt_tps": 564.6, + "prompt_tps_source": "engine", + "decode_tps": 81.3, + "decode_tps_source": "engine", + "prompt_ms": 1078.6, + "decode_ms": 418.0, + "drafts_offered": 25, + "drafts_accepted": 20, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.705, + "total_s": 4.31, + "prompt_tokens": 4197, + "reused_tokens": 602, + "read_tokens": 3595, + "generated_tokens": 42, + "prompt_tps": 971.6, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 3700.1, + "decode_ms": 605.3, + "drafts_offered": 35, + "drafts_accepted": 23, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.923, + "total_s": 2.71, + "prompt_tokens": 5831, + "reused_tokens": 4190, + "read_tokens": 1641, + "generated_tokens": 57, + "prompt_tps": 855.3, + "prompt_tps_source": "engine", + "decode_tps": 72.4, + "decode_tps_source": "engine", + "prompt_ms": 1918.7, + "decode_ms": 787.0, + "drafts_offered": 39, + "drafts_accepted": 31, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.581, + "total_s": 5.102, + "prompt_tokens": 9255, + "reused_tokens": 5824, + "read_tokens": 3431, + "generated_tokens": 37, + "prompt_tps": 749.9, + "prompt_tps_source": "engine", + "decode_tps": 71.1, + "decode_tps_source": "engine", + "prompt_ms": 4575.1, + "decode_ms": 520.7, + "drafts_offered": 23, + "drafts_accepted": 19, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.718, + "total_s": 4.343, + "prompt_tokens": 12209, + "reused_tokens": 9248, + "read_tokens": 2961, + "generated_tokens": 46, + "prompt_tps": 797.9, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 3711.0, + "decode_ms": 625.7, + "drafts_offered": 31, + "drafts_accepted": 27, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.869, + "total_s": 5.67, + "prompt_tokens": 15139, + "reused_tokens": 12202, + "read_tokens": 2937, + "generated_tokens": 48, + "prompt_tps": 604.2, + "prompt_tps_source": "engine", + "decode_tps": 59.9, + "decode_tps_source": "engine", + "prompt_ms": 4860.8, + "decode_ms": 801.6, + "drafts_offered": 37, + "drafts_accepted": 23, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.76, + "min": 68.717, + "max": 68.765, + "n": 3 + }, + "reread_tokens": { + "median": 251, + "min": 238, + "max": 259, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.727, + "min": 1.9, + "max": 4.869, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15139, + "min": 15126, + "max": 15147, + "n": 3 + }, + "turn_decode_tps": { + "median": 71.1, + "min": 59.9, + "max": 74.4, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.085, + "min": 1.081, + "max": 1.086, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/client.log new file mode 100644 index 000000000..1b96edba5 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/out,dst=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 84.3 tok/s, first token 0.389 s, 100 tokens + run 1 q2: decode 64.4 tok/s, first token 0.343 s, 100 tokens + run 1 q3: decode 75.5 tok/s, first token 0.39 s, 100 tokens + run 2 q1: decode 94.1 tok/s, first token 0.404 s, 100 tokens + run 2 q2: decode 79.6 tok/s, first token 0.39 s, 100 tokens + run 2 q3: decode 77.2 tok/s, first token 0.366 s, 100 tokens + run 3 q1: decode 97.5 tok/s, first token 0.396 s, 100 tokens + run 3 q2: decode 78.2 tok/s, first token 0.407 s, 100 tokens + run 3 q3: decode 77.6 tok/s, first token 0.401 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1205.6 tok/s (0 reused), first token 3.339 s + run 2: 4019 tokens, prompt 1211.9 tok/s (0 reused), first token 3.321 s + run 3: 4019 tokens, prompt 1216.4 tok/s (0 reused), first token 3.309 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1092.4 tok/s (0 reused), first token 14.725 s + run 2: 16058 tokens, prompt 1104.2 tok/s (0 reused), first token 14.557 s + run 3: 16058 tokens, prompt 1103.6 tok/s (0 reused), first token 14.565 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.086 s + run 1 turn 1: prompt 4189, reused 602, read 3587, first token 3.632 s + run 1 turn 2: prompt 5825, reused 4182, read 1643, first token 1.914 s + run 1 turn 3: prompt 9245, reused 5818, read 3427, first token 4.43 s + run 1 turn 4: prompt 12197, reused 9238, read 2959, first token 3.758 s + run 1 turn 5: prompt 15126, reused 12190, read 2936, first token 4.829 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.085 s + run 2 turn 1: prompt 4174, reused 602, read 3572, first token 3.652 s + run 2 turn 2: prompt 5813, reused 4167, read 1646, first token 1.9 s + run 2 turn 3: prompt 9241, reused 5806, read 3435, first token 4.472 s + run 2 turn 4: prompt 12210, reused 9234, read 2976, first token 3.727 s + run 2 turn 5: prompt 15147, reused 12203, read 2944, first token 4.858 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.081 s + run 3 turn 1: prompt 4197, reused 602, read 3595, first token 3.705 s + run 3 turn 2: prompt 5831, reused 4190, read 1641, first token 1.923 s + run 3 turn 3: prompt 9255, reused 5824, read 3431, first token 4.581 s + run 3 turn 4: prompt 12209, reused 9248, read 2961, first token 3.718 s + run 3 turn 5: prompt 15139, reused 12202, read 2937, first token 4.869 s + +Results (median, range over the runs): + chat decode tok/s 78.20 (64.40 - 97.50, n=9) + chat first token (s) 0.39 (0.34 - 0.41, n=9) + 4K prompt tok/s 1211.90 (1205.60 - 1216.40, n=3) + 4K first token (s) 3.32 (3.31 - 3.34, n=3) + 16K prompt tok/s 1103.60 (1092.40 - 1104.20, n=3) + 16K first token (s) 14.56 (14.56 - 14.72, n=3) + agent cache hit % 68.76 (68.72 - 68.77, n=3) + agent re-read tokens/session 251.00 (238.00 - 259.00, n=3) + agent turn first token (s) 3.73 (1.90 - 4.87, n=15) + agent decode tok/s 71.10 (59.90 - 74.40, n=15) + +Report: /opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/out/official/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/qwen36-ud-q3_k_m-ctx32768/out/official/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..e5280a50f --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/qwen36-UD-Q3_K_M/Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/qwen36-ud-q3_k_m-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/qwen36-ud-q3_k_m/tokenizer", + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17104402720, + "sha256": "8966dd0cd8c543c4228490a2a8b0e0814fc4f1e6a8e199ceed4de6754ae7b8e1", + "filename": "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..27b9edd45 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 145.5607028079976 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md new file mode 100644 index 000000000..3f2281e55 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q3_k_m +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 75.8 (66.0-98.2) | 0.39 (0.38-0.41) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1177.4 (1168.3-1188.1) | not measured | 3.42 (3.39-3.44) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1104.0 (1098.3-1104.7) | not measured | 14.56 (14.55-14.64) | +| agent session, follow-up turns | 15121 (15120-15127) at the last turn | see below | up to 64 | 3 | see below | 68.2 (57.7-75.5) | 3.79 (1.94-5.02) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.7 % (68.7-68.8); earlier-turn tokens read again per session: 233 (232-239) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.16 s (1.15-1.21). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 531.6 | 1.149 | 83.3 | +| 1 | 4192 | 602 | 3590 | 948.1 | 3.794 | 65.7 | +| 2 | 5826 | 4185 | 1641 | 848.7 | 1.941 | 74.9 | +| 3 | 9244 | 5819 | 3425 | 740.7 | 4.634 | 68.2 | +| 4 | 12198 | 9237 | 2961 | 795.1 | 3.734 | 72.4 | +| 5 | 15127 | 12191 | 2936 | 599.1 | 4.91 | 57.7 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 147 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json new file mode 100644 index 000000000..63bd2daf2 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json @@ -0,0 +1,885 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 146.8426218032837, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "quant": "UD-Q3_K_M", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 5285, + "expert_cache_mib": 7049, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.383, + "total_s": 1.523, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 133.6, + "prompt_tps_source": "engine", + "decode_tps": 87.7, + "decode_tps_source": "engine", + "prompt_ms": 381.8, + "decode_ms": 1140.2, + "drafts_offered": 76, + "drafts_accepted": 69, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.396, + "total_s": 1.912, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 108.9, + "prompt_tps_source": "engine", + "decode_tps": 66.0, + "decode_tps_source": "engine", + "prompt_ms": 394.7, + "decode_ms": 1515.6, + "drafts_offered": 67, + "drafts_accepted": 49, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.387, + "total_s": 1.747, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 119.5, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 385.0, + "decode_ms": 1359.9, + "drafts_offered": 70, + "drafts_accepted": 53, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.399, + "total_s": 1.422, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 128.3, + "prompt_tps_source": "engine", + "decode_tps": 97.7, + "decode_tps_source": "engine", + "prompt_ms": 397.5, + "decode_ms": 1023.4, + "drafts_offered": 76, + "drafts_accepted": 73, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.38, + "total_s": 1.793, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 113.8, + "prompt_tps_source": "engine", + "decode_tps": 70.8, + "decode_tps_source": "engine", + "prompt_ms": 378.0, + "decode_ms": 1413.3, + "drafts_offered": 73, + "drafts_accepted": 51, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.395, + "total_s": 1.714, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 116.9, + "prompt_tps_source": "engine", + "decode_tps": 75.8, + "decode_tps_source": "engine", + "prompt_ms": 393.5, + "decode_ms": 1318.8, + "drafts_offered": 70, + "drafts_accepted": 55, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.392, + "total_s": 1.411, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 130.6, + "prompt_tps_source": "engine", + "decode_tps": 98.2, + "decode_tps_source": "engine", + "prompt_ms": 390.5, + "decode_ms": 1018.8, + "drafts_offered": 75, + "drafts_accepted": 73, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.379, + "total_s": 1.819, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 114.0, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 377.2, + "decode_ms": 1440.1, + "drafts_offered": 72, + "drafts_accepted": 49, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.408, + "total_s": 1.727, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 113.1, + "prompt_tps_source": "engine", + "decode_tps": 75.8, + "decode_tps_source": "engine", + "prompt_ms": 406.6, + "decode_ms": 1318.5, + "drafts_offered": 67, + "drafts_accepted": 54, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 75.8, + "min": 66.0, + "max": 98.2, + "n": 9 + }, + "ttft_s": { + "median": 0.392, + "min": 0.379, + "max": 0.408, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.445, + "total_s": 3.818, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1168.3, + "prompt_tps_source": "engine", + "decode_tps": 64.3, + "decode_tps_source": "engine", + "prompt_ms": 3439.9, + "decode_ms": 373.2, + "drafts_offered": 19, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.418, + "total_s": 3.76, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1177.4, + "prompt_tps_source": "engine", + "decode_tps": 70.2, + "decode_tps_source": "engine", + "prompt_ms": 3413.4, + "decode_ms": 341.7, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.387, + "total_s": 3.712, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1188.1, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 3382.8, + "decode_ms": 324.5, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1177.4, + "min": 1168.3, + "max": 1188.1, + "n": 3 + }, + "ttft_s": { + "median": 3.418, + "min": 3.387, + "max": 3.445, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 14.636, + "total_s": 15.007, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1098.3, + "prompt_tps_source": "engine", + "decode_tps": 64.6, + "decode_tps_source": "engine", + "prompt_ms": 14621.3, + "decode_ms": 371.6, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.55, + "total_s": 14.915, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1104.7, + "prompt_tps_source": "engine", + "decode_tps": 65.7, + "decode_tps_source": "engine", + "prompt_ms": 14536.1, + "decode_ms": 365.3, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.559, + "total_s": 14.915, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1104.0, + "prompt_tps_source": "engine", + "decode_tps": 67.3, + "decode_tps_source": "engine", + "prompt_ms": 14544.9, + "decode_ms": 356.8, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1104.0, + "min": 1098.3, + "max": 1104.7, + "n": 3 + }, + "ttft_s": { + "median": 14.559, + "min": 14.55, + "max": 14.636, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.149, + "total_s": 1.497, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 29, + "prompt_tps": 531.6, + "prompt_tps_source": "engine", + "decode_tps": 83.3, + "decode_tps_source": "engine", + "prompt_ms": 1145.5, + "decode_ms": 348.0, + "drafts_offered": 23, + "drafts_accepted": 18, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.794, + "total_s": 4.433, + "prompt_tokens": 4192, + "reused_tokens": 602, + "read_tokens": 3590, + "generated_tokens": 42, + "prompt_tps": 948.1, + "prompt_tps_source": "engine", + "decode_tps": 65.7, + "decode_tps_source": "engine", + "prompt_ms": 3786.6, + "decode_ms": 639.7, + "drafts_offered": 39, + "drafts_accepted": 23, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.941, + "total_s": 2.622, + "prompt_tokens": 5826, + "reused_tokens": 4185, + "read_tokens": 1641, + "generated_tokens": 51, + "prompt_tps": 848.7, + "prompt_tps_source": "engine", + "decode_tps": 74.9, + "decode_tps_source": "engine", + "prompt_ms": 1933.6, + "decode_ms": 681.3, + "drafts_offered": 32, + "drafts_accepted": 27, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.634, + "total_s": 5.176, + "prompt_tokens": 9244, + "reused_tokens": 5819, + "read_tokens": 3425, + "generated_tokens": 37, + "prompt_tps": 740.7, + "prompt_tps_source": "engine", + "decode_tps": 68.2, + "decode_tps_source": "engine", + "prompt_ms": 4624.3, + "decode_ms": 542.6, + "drafts_offered": 25, + "drafts_accepted": 19, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.734, + "total_s": 4.355, + "prompt_tokens": 12198, + "reused_tokens": 9237, + "read_tokens": 2961, + "generated_tokens": 45, + "prompt_tps": 795.1, + "prompt_tps_source": "engine", + "decode_tps": 72.4, + "decode_tps_source": "engine", + "prompt_ms": 3724.0, + "decode_ms": 621.2, + "drafts_offered": 32, + "drafts_accepted": 26, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.91, + "total_s": 5.741, + "prompt_tokens": 15127, + "reused_tokens": 12191, + "read_tokens": 2936, + "generated_tokens": 48, + "prompt_tps": 599.1, + "prompt_tps_source": "engine", + "decode_tps": 57.7, + "decode_tps_source": "engine", + "prompt_ms": 4900.6, + "decode_ms": 832.2, + "drafts_offered": 38, + "drafts_accepted": 23, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.155, + "total_s": 1.26, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 528.4, + "prompt_tps_source": "engine", + "decode_tps": 103.8, + "decode_tps_source": "engine", + "prompt_ms": 1152.5, + "decode_ms": 106.0, + "drafts_offered": 7, + "drafts_accepted": 7, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.675, + "total_s": 4.393, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 47, + "prompt_tps": 973.2, + "prompt_tps_source": "engine", + "decode_tps": 65.5, + "decode_tps_source": "engine", + "prompt_ms": 3670.3, + "decode_ms": 717.9, + "drafts_offered": 35, + "drafts_accepted": 22, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.944, + "total_s": 2.607, + "prompt_tokens": 5813, + "reused_tokens": 4167, + "read_tokens": 1646, + "generated_tokens": 50, + "prompt_tps": 848.4, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 1940.2, + "decode_ms": 662.6, + "drafts_offered": 29, + "drafts_accepted": 26, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.592, + "total_s": 5.256, + "prompt_tokens": 9230, + "reused_tokens": 5806, + "read_tokens": 3424, + "generated_tokens": 44, + "prompt_tps": 746.6, + "prompt_tps_source": "engine", + "decode_tps": 66.2, + "decode_tps_source": "engine", + "prompt_ms": 4586.1, + "decode_ms": 664.2, + "drafts_offered": 32, + "drafts_accepted": 22, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.732, + "total_s": 4.394, + "prompt_tokens": 12191, + "reused_tokens": 9223, + "read_tokens": 2968, + "generated_tokens": 46, + "prompt_tps": 796.7, + "prompt_tps_source": "engine", + "decode_tps": 69.5, + "decode_tps_source": "engine", + "prompt_ms": 3725.3, + "decode_ms": 662.0, + "drafts_offered": 35, + "drafts_accepted": 26, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.968, + "total_s": 5.781, + "prompt_tokens": 15121, + "reused_tokens": 12184, + "read_tokens": 2937, + "generated_tokens": 51, + "prompt_tps": 592.1, + "prompt_tps_source": "engine", + "decode_tps": 62.7, + "decode_tps_source": "engine", + "prompt_ms": 4960.0, + "decode_ms": 813.4, + "drafts_offered": 38, + "drafts_accepted": 26, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.21, + "total_s": 1.483, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 19, + "prompt_tps": 504.3, + "prompt_tps_source": "engine", + "decode_tps": 69.4, + "decode_tps_source": "engine", + "prompt_ms": 1207.7, + "decode_ms": 273.6, + "drafts_offered": 13, + "drafts_accepted": 8, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.739, + "total_s": 4.383, + "prompt_tokens": 4182, + "reused_tokens": 602, + "read_tokens": 3580, + "generated_tokens": 42, + "prompt_tps": 958.7, + "prompt_tps_source": "engine", + "decode_tps": 65.2, + "decode_tps_source": "engine", + "prompt_ms": 3734.4, + "decode_ms": 644.5, + "drafts_offered": 35, + "drafts_accepted": 21, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.981, + "total_s": 2.629, + "prompt_tokens": 5816, + "reused_tokens": 4175, + "read_tokens": 1641, + "generated_tokens": 48, + "prompt_tps": 830.4, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 1976.1, + "decode_ms": 648.6, + "drafts_offered": 31, + "drafts_accepted": 26, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.617, + "total_s": 5.205, + "prompt_tokens": 9231, + "reused_tokens": 5809, + "read_tokens": 3422, + "generated_tokens": 43, + "prompt_tps": 742.1, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 4611.0, + "decode_ms": 587.9, + "drafts_offered": 28, + "drafts_accepted": 24, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.948, + "total_s": 4.582, + "prompt_tokens": 12191, + "reused_tokens": 9224, + "read_tokens": 2967, + "generated_tokens": 45, + "prompt_tps": 752.9, + "prompt_tps_source": "engine", + "decode_tps": 70.9, + "decode_tps_source": "engine", + "prompt_ms": 3941.0, + "decode_ms": 634.5, + "drafts_offered": 31, + "drafts_accepted": 25, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.023, + "total_s": 5.777, + "prompt_tokens": 15120, + "reused_tokens": 12184, + "read_tokens": 2936, + "generated_tokens": 48, + "prompt_tps": 585.5, + "prompt_tps_source": "engine", + "decode_tps": 63.6, + "decode_tps_source": "engine", + "prompt_ms": 5014.4, + "decode_ms": 755.1, + "drafts_offered": 34, + "drafts_accepted": 24, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.745, + "min": 68.736, + "max": 68.762, + "n": 3 + }, + "reread_tokens": { + "median": 233, + "min": 232, + "max": 239, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.794, + "min": 1.941, + "max": 5.023, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15121, + "min": 15120, + "max": 15127, + "n": 3 + }, + "turn_decode_tps": { + "median": 68.2, + "min": 57.7, + "max": 75.5, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.155, + "min": 1.149, + "max": 1.21, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..151b2c23d --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 87.7 tok/s, first token 0.383 s, 100 tokens + run 1 q2: decode 66.0 tok/s, first token 0.396 s, 100 tokens + run 1 q3: decode 73.5 tok/s, first token 0.387 s, 100 tokens + run 2 q1: decode 97.7 tok/s, first token 0.399 s, 100 tokens + run 2 q2: decode 70.8 tok/s, first token 0.38 s, 100 tokens + run 2 q3: decode 75.8 tok/s, first token 0.395 s, 100 tokens + run 3 q1: decode 98.2 tok/s, first token 0.392 s, 100 tokens + run 3 q2: decode 69.4 tok/s, first token 0.379 s, 100 tokens + run 3 q3: decode 75.8 tok/s, first token 0.408 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1168.3 tok/s (0 reused), first token 3.445 s + run 2: 4019 tokens, prompt 1177.4 tok/s (0 reused), first token 3.418 s + run 3: 4019 tokens, prompt 1188.1 tok/s (0 reused), first token 3.387 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1098.3 tok/s (0 reused), first token 14.636 s + run 2: 16058 tokens, prompt 1104.7 tok/s (0 reused), first token 14.55 s + run 3: 16058 tokens, prompt 1104.0 tok/s (0 reused), first token 14.559 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.149 s + run 1 turn 1: prompt 4192, reused 602, read 3590, first token 3.794 s + run 1 turn 2: prompt 5826, reused 4185, read 1641, first token 1.941 s + run 1 turn 3: prompt 9244, reused 5819, read 3425, first token 4.634 s + run 1 turn 4: prompt 12198, reused 9237, read 2961, first token 3.734 s + run 1 turn 5: prompt 15127, reused 12191, read 2936, first token 4.91 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.155 s + run 2 turn 1: prompt 4174, reused 602, read 3572, first token 3.675 s + run 2 turn 2: prompt 5813, reused 4167, read 1646, first token 1.944 s + run 2 turn 3: prompt 9230, reused 5806, read 3424, first token 4.592 s + run 2 turn 4: prompt 12191, reused 9223, read 2968, first token 3.732 s + run 2 turn 5: prompt 15121, reused 12184, read 2937, first token 4.968 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.21 s + run 3 turn 1: prompt 4182, reused 602, read 3580, first token 3.739 s + run 3 turn 2: prompt 5816, reused 4175, read 1641, first token 1.981 s + run 3 turn 3: prompt 9231, reused 5809, read 3422, first token 4.617 s + run 3 turn 4: prompt 12191, reused 9224, read 2967, first token 3.948 s + run 3 turn 5: prompt 15120, reused 12184, read 2936, first token 5.023 s + +Results (median, range over the runs): + chat decode tok/s 75.80 (66.00 - 98.20, n=9) + chat first token (s) 0.39 (0.38 - 0.41, n=9) + 4K prompt tok/s 1177.40 (1168.30 - 1188.10, n=3) + 4K first token (s) 3.42 (3.39 - 3.44, n=3) + 16K prompt tok/s 1104.00 (1098.30 - 1104.70, n=3) + 16K first token (s) 14.56 (14.55 - 14.64, n=3) + agent cache hit % 68.75 (68.74 - 68.76, n=3) + agent re-read tokens/session 233.00 (232.00 - 239.00, n=3) + agent turn first token (s) 3.79 (1.94 - 5.02, n=15) + agent decode tok/s 68.20 (57.70 - 75.50, n=15) + +Report: /opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..2e48baaac --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,50 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/qwen36-UD-Q3_K_M/Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/qwen36-ud-q3_k_m-reserve1024-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/qwen36-ud-q3_k_m/tokenizer", + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17104402720, + "sha256": "8966dd0cd8c543c4228490a2a8b0e0814fc4f1e6a8e199ceed4de6754ae7b8e1", + "filename": "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..28c994f5c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 147.0612616330036 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md new file mode 100644 index 000000000..1131f121e --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q3_k_m +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 79.9 (68.9-99.0) | 0.40 (0.37-0.43) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 1144.6 (1138.6-1149.0) | not measured | 3.52 (3.50-3.54) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1103.4 (1094.8-1104.6) | not measured | 14.57 (14.55-14.68) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 850.9 (843.1-851.1) | not measured | 37.69 (37.69-38.04) | +| agent session, follow-up turns | 15124 (15122-15132) at the last turn | see below | up to 64 | 3 | see below | 72.7 (60.5-78.8) | 3.82 (2.01-5.00) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.8-68.8); earlier-turn tokens read again per session: 236 (234-244) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.23 s (1.23-1.26). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 496.0 | 1.232 | 85.5 | +| 1 | 4192 | 602 | 3590 | 958.3 | 3.754 | 69.6 | +| 2 | 5824 | 4185 | 1639 | 817.7 | 2.011 | 77.1 | +| 3 | 9249 | 5817 | 3432 | 726.6 | 4.733 | 65.8 | +| 4 | 12203 | 9242 | 2961 | 783.7 | 3.788 | 67.0 | +| 5 | 15132 | 12196 | 2936 | 588.3 | 5.0 | 61.1 | + +### Concurrency: 2 clients + +Total 58.4 tok/s (57.0-60.3); first token p50 1.509 s, p95 2.746 s; request latency p50 3.309 s, p95 4.464 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 275 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json new file mode 100644 index 000000000..25fa4c6e3 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/results.json @@ -0,0 +1,1034 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 275.43845653533936, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "quant": "UD-Q3_K_M", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q3_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --tail-skip 6 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q3_k_m", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 4757, + "expert_cache_mib": 6346, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.384, + "total_s": 1.559, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 133.4, + "prompt_tps_source": "engine", + "decode_tps": 85.1, + "decode_tps_source": "engine", + "prompt_ms": 382.4, + "decode_ms": 1174.8, + "drafts_offered": 77, + "drafts_accepted": 69, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.418, + "total_s": 1.87, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 103.3, + "prompt_tps_source": "engine", + "decode_tps": 68.9, + "decode_tps_source": "engine", + "prompt_ms": 416.3, + "decode_ms": 1451.6, + "drafts_offered": 61, + "drafts_accepted": 50, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.37, + "total_s": 1.694, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 125.1, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 367.7, + "decode_ms": 1324.7, + "drafts_offered": 69, + "drafts_accepted": 58, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.412, + "total_s": 1.446, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 124.5, + "prompt_tps_source": "engine", + "decode_tps": 96.6, + "decode_tps_source": "engine", + "prompt_ms": 409.6, + "decode_ms": 1034.7, + "drafts_offered": 77, + "drafts_accepted": 73, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.377, + "total_s": 1.627, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 114.7, + "prompt_tps_source": "engine", + "decode_tps": 79.9, + "decode_tps_source": "engine", + "prompt_ms": 374.8, + "decode_ms": 1250.9, + "drafts_offered": 64, + "drafts_accepted": 56, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.403, + "total_s": 1.703, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 114.6, + "prompt_tps_source": "engine", + "decode_tps": 76.9, + "decode_tps_source": "engine", + "prompt_ms": 401.3, + "decode_ms": 1300.5, + "drafts_offered": 69, + "drafts_accepted": 55, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.43, + "total_s": 1.439, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 119.2, + "prompt_tps_source": "engine", + "decode_tps": 99.0, + "decode_tps_source": "engine", + "prompt_ms": 427.8, + "decode_ms": 1009.8, + "drafts_offered": 75, + "drafts_accepted": 73, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.368, + "total_s": 1.581, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 117.3, + "prompt_tps_source": "engine", + "decode_tps": 82.4, + "decode_tps_source": "engine", + "prompt_ms": 366.6, + "decode_ms": 1213.0, + "drafts_offered": 63, + "drafts_accepted": 57, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.397, + "total_s": 1.652, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 116.5, + "prompt_tps_source": "engine", + "decode_tps": 79.6, + "decode_tps_source": "engine", + "prompt_ms": 394.9, + "decode_ms": 1255.7, + "drafts_offered": 67, + "drafts_accepted": 56, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 79.9, + "min": 68.9, + "max": 99.0, + "n": 9 + }, + "ttft_s": { + "median": 0.397, + "min": 0.368, + "max": 0.43, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.535, + "total_s": 3.888, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1138.6, + "prompt_tps_source": "engine", + "decode_tps": 68.0, + "decode_tps_source": "engine", + "prompt_ms": 3529.9, + "decode_ms": 353.2, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.503, + "total_s": 3.842, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1149.0, + "prompt_tps_source": "engine", + "decode_tps": 70.7, + "decode_tps_source": "engine", + "prompt_ms": 3497.8, + "decode_ms": 339.6, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.517, + "total_s": 3.883, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 1144.6, + "prompt_tps_source": "engine", + "decode_tps": 65.4, + "decode_tps_source": "engine", + "prompt_ms": 3511.4, + "decode_ms": 367.2, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1144.6, + "min": 1138.6, + "max": 1149.0, + "n": 3 + }, + "ttft_s": { + "median": 3.517, + "min": 3.503, + "max": 3.535, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 14.682, + "total_s": 15.063, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1094.8, + "prompt_tps_source": "engine", + "decode_tps": 62.9, + "decode_tps_source": "engine", + "prompt_ms": 14667.5, + "decode_ms": 381.7, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.552, + "total_s": 14.901, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1104.6, + "prompt_tps_source": "engine", + "decode_tps": 68.6, + "decode_tps_source": "engine", + "prompt_ms": 14537.1, + "decode_ms": 350.0, + "drafts_offered": 17, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.567, + "total_s": 14.917, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1103.4, + "prompt_tps_source": "engine", + "decode_tps": 68.5, + "decode_tps_source": "engine", + "prompt_ms": 14552.9, + "decode_ms": 350.3, + "drafts_offered": 17, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 1103.4, + "min": 1094.8, + "max": 1104.6, + "n": 3 + }, + "ttft_s": { + "median": 14.567, + "min": 14.552, + "max": 14.682, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 38.043, + "total_s": 38.48, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 843.1, + "prompt_tps_source": "engine", + "decode_tps": 54.9, + "decode_tps_source": "engine", + "prompt_ms": 38015.5, + "decode_ms": 437.5, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 37.692, + "total_s": 38.138, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 850.9, + "prompt_tps_source": "engine", + "decode_tps": 53.7, + "decode_tps_source": "engine", + "prompt_ms": 37665.6, + "decode_ms": 446.9, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 37.685, + "total_s": 38.095, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 851.1, + "prompt_tps_source": "engine", + "decode_tps": 58.4, + "decode_tps_source": "engine", + "prompt_ms": 37657.1, + "decode_ms": 411.2, + "drafts_offered": 19, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 850.9, + "min": 843.1, + "max": 851.1, + "n": 3 + }, + "ttft_s": { + "median": 37.692, + "min": 37.685, + "max": 38.043, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.232, + "total_s": 1.57, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 29, + "prompt_tps": 496.0, + "prompt_tps_source": "engine", + "decode_tps": 85.5, + "decode_tps_source": "engine", + "prompt_ms": 1227.9, + "decode_ms": 339.2, + "drafts_offered": 23, + "drafts_accepted": 19, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.754, + "total_s": 4.328, + "prompt_tokens": 4192, + "reused_tokens": 602, + "read_tokens": 3590, + "generated_tokens": 40, + "prompt_tps": 958.3, + "prompt_tps_source": "engine", + "decode_tps": 69.6, + "decode_tps_source": "engine", + "prompt_ms": 3746.2, + "decode_ms": 574.4, + "drafts_offered": 34, + "drafts_accepted": 23, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.011, + "total_s": 2.763, + "prompt_tokens": 5824, + "reused_tokens": 4185, + "read_tokens": 1639, + "generated_tokens": 58, + "prompt_tps": 817.7, + "prompt_tps_source": "engine", + "decode_tps": 77.1, + "decode_tps_source": "engine", + "prompt_ms": 2004.4, + "decode_ms": 752.1, + "drafts_offered": 40, + "drafts_accepted": 34, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.733, + "total_s": 5.294, + "prompt_tokens": 9249, + "reused_tokens": 5817, + "read_tokens": 3432, + "generated_tokens": 37, + "prompt_tps": 726.6, + "prompt_tps_source": "engine", + "decode_tps": 65.8, + "decode_tps_source": "engine", + "prompt_ms": 4723.4, + "decode_ms": 561.9, + "drafts_offered": 25, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.788, + "total_s": 4.459, + "prompt_tokens": 12203, + "reused_tokens": 9242, + "read_tokens": 2961, + "generated_tokens": 45, + "prompt_tps": 783.7, + "prompt_tps_source": "engine", + "decode_tps": 67.0, + "decode_tps_source": "engine", + "prompt_ms": 3778.4, + "decode_ms": 671.6, + "drafts_offered": 34, + "drafts_accepted": 24, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.0, + "total_s": 5.784, + "prompt_tokens": 15132, + "reused_tokens": 12196, + "read_tokens": 2936, + "generated_tokens": 48, + "prompt_tps": 588.3, + "prompt_tps_source": "engine", + "decode_tps": 61.1, + "decode_tps_source": "engine", + "prompt_ms": 4990.4, + "decode_ms": 785.0, + "drafts_offered": 38, + "drafts_accepted": 24, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.228, + "total_s": 1.548, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 29, + "prompt_tps": 496.7, + "prompt_tps_source": "engine", + "decode_tps": 90.6, + "decode_tps_source": "engine", + "prompt_ms": 1226.0, + "decode_ms": 320.1, + "drafts_offered": 22, + "drafts_accepted": 19, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.97, + "total_s": 4.538, + "prompt_tokens": 4192, + "reused_tokens": 602, + "read_tokens": 3590, + "generated_tokens": 40, + "prompt_tps": 905.4, + "prompt_tps_source": "engine", + "decode_tps": 70.4, + "decode_tps_source": "engine", + "prompt_ms": 3965.2, + "decode_ms": 567.8, + "drafts_offered": 34, + "drafts_accepted": 23, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.055, + "total_s": 2.716, + "prompt_tokens": 5824, + "reused_tokens": 4185, + "read_tokens": 1639, + "generated_tokens": 50, + "prompt_tps": 799.4, + "prompt_tps_source": "engine", + "decode_tps": 75.5, + "decode_tps_source": "engine", + "prompt_ms": 2050.2, + "decode_ms": 661.9, + "drafts_offered": 34, + "drafts_accepted": 28, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.685, + "total_s": 5.183, + "prompt_tokens": 9241, + "reused_tokens": 5817, + "read_tokens": 3424, + "generated_tokens": 37, + "prompt_tps": 731.8, + "prompt_tps_source": "engine", + "decode_tps": 74.2, + "decode_tps_source": "engine", + "prompt_ms": 4678.9, + "decode_ms": 498.5, + "drafts_offered": 23, + "drafts_accepted": 20, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.779, + "total_s": 4.394, + "prompt_tokens": 12195, + "reused_tokens": 9234, + "read_tokens": 2961, + "generated_tokens": 45, + "prompt_tps": 785.1, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 3771.5, + "decode_ms": 616.0, + "drafts_offered": 31, + "drafts_accepted": 26, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.0, + "total_s": 5.73, + "prompt_tokens": 15124, + "reused_tokens": 12188, + "read_tokens": 2936, + "generated_tokens": 46, + "prompt_tps": 588.2, + "prompt_tps_source": "engine", + "decode_tps": 63.0, + "decode_tps_source": "engine", + "prompt_ms": 4991.8, + "decode_ms": 730.4, + "drafts_offered": 33, + "drafts_accepted": 23, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.263, + "total_s": 1.568, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 26, + "prompt_tps": 483.1, + "prompt_tps_source": "engine", + "decode_tps": 85.2, + "decode_tps_source": "engine", + "prompt_ms": 1260.5, + "decode_ms": 305.2, + "drafts_offered": 19, + "drafts_accepted": 16, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.776, + "total_s": 4.353, + "prompt_tokens": 4189, + "reused_tokens": 602, + "read_tokens": 3587, + "generated_tokens": 42, + "prompt_tps": 951.3, + "prompt_tps_source": "engine", + "decode_tps": 72.7, + "decode_tps_source": "engine", + "prompt_ms": 3770.8, + "decode_ms": 578.1, + "drafts_offered": 34, + "drafts_accepted": 24, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.006, + "total_s": 2.64, + "prompt_tokens": 5823, + "reused_tokens": 4182, + "read_tokens": 1641, + "generated_tokens": 47, + "prompt_tps": 820.0, + "prompt_tps_source": "engine", + "decode_tps": 74.2, + "decode_tps_source": "engine", + "prompt_ms": 2001.3, + "decode_ms": 633.8, + "drafts_offered": 30, + "drafts_accepted": 25, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.661, + "total_s": 5.155, + "prompt_tokens": 9237, + "reused_tokens": 5816, + "read_tokens": 3421, + "generated_tokens": 39, + "prompt_tps": 735.0, + "prompt_tps_source": "engine", + "decode_tps": 78.8, + "decode_tps_source": "engine", + "prompt_ms": 4654.3, + "decode_ms": 494.7, + "drafts_offered": 22, + "drafts_accepted": 22, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.821, + "total_s": 4.425, + "prompt_tokens": 12193, + "reused_tokens": 9230, + "read_tokens": 2963, + "generated_tokens": 45, + "prompt_tps": 777.0, + "prompt_tps_source": "engine", + "decode_tps": 74.4, + "decode_tps_source": "engine", + "prompt_ms": 3813.6, + "decode_ms": 604.5, + "drafts_offered": 31, + "drafts_accepted": 27, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.999, + "total_s": 5.775, + "prompt_tokens": 15122, + "reused_tokens": 12186, + "read_tokens": 2936, + "generated_tokens": 47, + "prompt_tps": 588.2, + "prompt_tps_source": "engine", + "decode_tps": 60.5, + "decode_tps_source": "engine", + "prompt_ms": 4991.1, + "decode_ms": 776.5, + "drafts_offered": 35, + "drafts_accepted": 23, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.76, + "min": 68.757, + "max": 68.761, + "n": 3 + }, + "reread_tokens": { + "median": 236, + "min": 234, + "max": 244, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 3.821, + "min": 2.006, + "max": 5.0, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15124, + "min": 15122, + "max": 15132, + "n": 3 + }, + "turn_decode_tps": { + "median": 72.7, + "min": 60.5, + "max": 78.8, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.232, + "min": 1.228, + "max": 1.263, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 4.49, + "tokens": 256, + "total_tps": 56.99, + "first_s": [ + 0.48641130700707436, + 2.803317843005061 + ], + "latency_s": [ + 2.369768368982477, + 4.4917282560199965 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 4.38, + "tokens": 256, + "total_tps": 58.41, + "first_s": [ + 0.44266226899344474, + 2.531317371991463 + ], + "latency_s": [ + 2.0628959450114053, + 4.382355737005128 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 4.25, + "tokens": 256, + "total_tps": 60.26, + "first_s": [ + 0.43584367001312785, + 2.575519539997913 + ], + "latency_s": [ + 2.1342665370029863, + 4.24806363901007 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 58.41, + "min": 56.99, + "max": 60.26, + "n": 3 + }, + "ttft_p50_s": 1.509, + "ttft_p95_s": 2.746, + "latency_p50_s": 3.309, + "latency_p95_s": 4.464, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..adccfd0b8 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 85.1 tok/s, first token 0.384 s, 100 tokens + run 1 q2: decode 68.9 tok/s, first token 0.418 s, 100 tokens + run 1 q3: decode 75.5 tok/s, first token 0.37 s, 100 tokens + run 2 q1: decode 96.6 tok/s, first token 0.412 s, 100 tokens + run 2 q2: decode 79.9 tok/s, first token 0.377 s, 100 tokens + run 2 q3: decode 76.9 tok/s, first token 0.403 s, 100 tokens + run 3 q1: decode 99.0 tok/s, first token 0.43 s, 100 tokens + run 3 q2: decode 82.4 tok/s, first token 0.368 s, 100 tokens + run 3 q3: decode 79.6 tok/s, first token 0.397 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 1138.6 tok/s (0 reused), first token 3.535 s + run 2: 4019 tokens, prompt 1149.0 tok/s (0 reused), first token 3.503 s + run 3: 4019 tokens, prompt 1144.6 tok/s (0 reused), first token 3.517 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1094.8 tok/s (0 reused), first token 14.682 s + run 2: 16058 tokens, prompt 1104.6 tok/s (0 reused), first token 14.552 s + run 3: 16058 tokens, prompt 1103.4 tok/s (0 reused), first token 14.567 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.232 s + run 1 turn 1: prompt 4192, reused 602, read 3590, first token 3.754 s + run 1 turn 2: prompt 5824, reused 4185, read 1639, first token 2.011 s + run 1 turn 3: prompt 9249, reused 5817, read 3432, first token 4.733 s + run 1 turn 4: prompt 12203, reused 9242, read 2961, first token 3.788 s + run 1 turn 5: prompt 15132, reused 12196, read 2936, first token 5.0 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.228 s + run 2 turn 1: prompt 4192, reused 602, read 3590, first token 3.97 s + run 2 turn 2: prompt 5824, reused 4185, read 1639, first token 2.055 s + run 2 turn 3: prompt 9241, reused 5817, read 3424, first token 4.685 s + run 2 turn 4: prompt 12195, reused 9234, read 2961, first token 3.779 s + run 2 turn 5: prompt 15124, reused 12188, read 2936, first token 5.0 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.263 s + run 3 turn 1: prompt 4189, reused 602, read 3587, first token 3.776 s + run 3 turn 2: prompt 5823, reused 4182, read 1641, first token 2.006 s + run 3 turn 3: prompt 9237, reused 5816, read 3421, first token 4.661 s + run 3 turn 4: prompt 12193, reused 9230, read 2963, first token 3.821 s + run 3 turn 5: prompt 15122, reused 12186, read 2936, first token 4.999 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 4.5 s = 57.0 tok/s total + round 2: 256 tokens in 4.4 s = 58.4 tok/s total + round 3: 256 tokens in 4.2 s = 60.3 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32049 tokens, prompt 843.1 tok/s (0 reused), first token 38.043 s + run 2: 32049 tokens, prompt 850.9 tok/s (0 reused), first token 37.692 s + run 3: 32049 tokens, prompt 851.1 tok/s (0 reused), first token 37.685 s + +Results (median, range over the runs): + chat decode tok/s 79.90 (68.90 - 99.00, n=9) + chat first token (s) 0.40 (0.37 - 0.43, n=9) + 4K prompt tok/s 1144.60 (1138.60 - 1149.00, n=3) + 4K first token (s) 3.52 (3.50 - 3.54, n=3) + 16K prompt tok/s 1103.40 (1094.80 - 1104.60, n=3) + 16K first token (s) 14.57 (14.55 - 14.68, n=3) + 32K prompt tok/s 850.90 (843.10 - 851.10, n=3) + 32K first token (s) 37.69 (37.69 - 38.04, n=3) + agent cache hit % 68.76 (68.76 - 68.76, n=3) + agent re-read tokens/session 236.00 (234.00 - 244.00, n=3) + agent turn first token (s) 3.82 (2.01 - 5.00, n=15) + agent decode tok/s 72.70 (60.50 - 78.80, n=15) + 2 clients total tok/s 58.41 (56.99 - 60.26, n=3) + +Report: /opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q3-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..8e527c648 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,50 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/qwen36-UD-Q3_K_M/Qwen3.6-35B-A3B-UD-Q3_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/qwen36-ud-q3_k_m-reserve1024-ctx65536/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--tail-skip", + "6", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/qwen36-ud-q3_k_m/tokenizer", + "model_name": "qwen3.6-35b-a3b-ud-q3_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q3_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17104402720, + "sha256": "8966dd0cd8c543c4228490a2a8b0e0814fc4f1e6a8e199ceed4de6754ae7b8e1", + "filename": "Qwen3.6-35B-A3B-UD-Q3_K_M.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..276096865 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q3_k_m-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 275.65166014200076 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md new file mode 100644 index 000000000..08c0e6f74 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_m +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --tail-skip 0 --kv-whole --conversation-cache-mib 0 --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 80.7 (68.8-94.9) | 0.41 (0.37-0.46) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 921.4 (873.0-924.8) | not measured | 4.37 (4.35-4.61) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1019.4 (1009.5-1026.4) | not measured | 15.77 (15.66-15.92) | +| agent session, follow-up turns | 15129 (15124-15160) at the last turn | see below | up to 64 | 3 | see below | 78.7 (63.6-84.1) | 4.60 (2.73-5.61) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.7 % (68.7-68.8); earlier-turn tokens read again per session: 241 (236-272) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.58 s (1.50-1.63). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 385.7 | 1.582 | 88.7 | +| 1 | 4192 | 602 | 3590 | 782.3 | 4.597 | 78.7 | +| 2 | 5839 | 4185 | 1654 | 604.7 | 2.742 | 78.3 | +| 3 | 9265 | 5832 | 3433 | 642.7 | 5.351 | 63.6 | +| 4 | 12226 | 9258 | 2968 | 659.9 | 4.508 | 79.6 | +| 5 | 15160 | 12219 | 2941 | 525.2 | 5.609 | 69.1 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 167 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json new file mode 100644 index 000000000..46d62b3a6 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json @@ -0,0 +1,885 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 166.78402423858643, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_m", + "quant": "UD-Q4_K_M", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --max-context 32768 --expert-profile expert-profile.txt --tail-skip 0 --kv-whole --conversation-cache-mib 0 --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_m", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 3860, + "expert_cache_mib": 7034, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.41, + "total_s": 1.602, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 124.8, + "prompt_tps_source": "engine", + "decode_tps": 83.9, + "decode_tps_source": "engine", + "prompt_ms": 408.5, + "decode_ms": 1191.8, + "drafts_offered": 73, + "drafts_accepted": 66, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.372, + "total_s": 1.826, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 116.0, + "prompt_tps_source": "engine", + "decode_tps": 68.8, + "decode_tps_source": "engine", + "prompt_ms": 370.6, + "decode_ms": 1453.8, + "drafts_offered": 71, + "drafts_accepted": 59, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.45, + "total_s": 1.852, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 102.5, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 448.7, + "decode_ms": 1401.4, + "drafts_offered": 71, + "drafts_accepted": 55, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.397, + "total_s": 1.475, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 129.1, + "prompt_tps_source": "engine", + "decode_tps": 92.8, + "decode_tps_source": "engine", + "prompt_ms": 395.0, + "decode_ms": 1077.7, + "drafts_offered": 78, + "drafts_accepted": 70, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.393, + "total_s": 1.633, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 109.9, + "prompt_tps_source": "engine", + "decode_tps": 80.7, + "decode_tps_source": "engine", + "prompt_ms": 391.2, + "decode_ms": 1239.9, + "drafts_offered": 74, + "drafts_accepted": 56, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.443, + "total_s": 1.701, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 104.1, + "prompt_tps_source": "engine", + "decode_tps": 79.5, + "decode_tps_source": "engine", + "prompt_ms": 441.9, + "decode_ms": 1257.4, + "drafts_offered": 74, + "drafts_accepted": 57, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.455, + "total_s": 1.509, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 112.4, + "prompt_tps_source": "engine", + "decode_tps": 94.9, + "decode_tps_source": "engine", + "prompt_ms": 453.8, + "decode_ms": 1053.9, + "drafts_offered": 74, + "drafts_accepted": 70, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.459, + "total_s": 1.7, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 94.1, + "prompt_tps_source": "engine", + "decode_tps": 80.6, + "decode_tps_source": "engine", + "prompt_ms": 456.9, + "decode_ms": 1241.1, + "drafts_offered": 70, + "drafts_accepted": 54, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.414, + "total_s": 1.641, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 111.6, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 412.2, + "decode_ms": 1226.9, + "drafts_offered": 73, + "drafts_accepted": 57, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 80.7, + "min": 68.8, + "max": 94.9, + "n": 9 + }, + "ttft_s": { + "median": 0.414, + "min": 0.372, + "max": 0.459, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.609, + "total_s": 4.95, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 873.0, + "prompt_tps_source": "engine", + "decode_tps": 70.2, + "decode_tps_source": "engine", + "prompt_ms": 4603.7, + "decode_ms": 341.8, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.351, + "total_s": 4.69, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 924.8, + "prompt_tps_source": "engine", + "decode_tps": 70.8, + "decode_tps_source": "engine", + "prompt_ms": 4345.9, + "decode_ms": 338.9, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.367, + "total_s": 4.695, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 921.4, + "prompt_tps_source": "engine", + "decode_tps": 73.2, + "decode_tps_source": "engine", + "prompt_ms": 4362.0, + "decode_ms": 328.0, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 921.4, + "min": 873.0, + "max": 924.8, + "n": 3 + }, + "ttft_s": { + "median": 4.367, + "min": 4.351, + "max": 4.609, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 15.768, + "total_s": 16.165, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1019.4, + "prompt_tps_source": "engine", + "decode_tps": 60.5, + "decode_tps_source": "engine", + "prompt_ms": 15753.1, + "decode_ms": 397.0, + "drafts_offered": 18, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.659, + "total_s": 16.047, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1026.4, + "prompt_tps_source": "engine", + "decode_tps": 61.8, + "decode_tps_source": "engine", + "prompt_ms": 15644.8, + "decode_ms": 388.3, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.921, + "total_s": 16.312, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1009.5, + "prompt_tps_source": "engine", + "decode_tps": 61.4, + "decode_tps_source": "engine", + "prompt_ms": 15907.1, + "decode_ms": 391.1, + "drafts_offered": 18, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1019.4, + "min": 1009.5, + "max": 1026.4, + "n": 3 + }, + "ttft_s": { + "median": 15.768, + "min": 15.659, + "max": 15.921, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.582, + "total_s": 1.909, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 29, + "prompt_tps": 385.7, + "prompt_tps_source": "engine", + "decode_tps": 88.7, + "decode_tps_source": "engine", + "prompt_ms": 1578.8, + "decode_ms": 326.8, + "drafts_offered": 24, + "drafts_accepted": 19, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.597, + "total_s": 5.295, + "prompt_tokens": 4192, + "reused_tokens": 602, + "read_tokens": 3590, + "generated_tokens": 55, + "prompt_tps": 782.3, + "prompt_tps_source": "engine", + "decode_tps": 78.7, + "decode_tps_source": "engine", + "prompt_ms": 4589.2, + "decode_ms": 698.7, + "drafts_offered": 38, + "drafts_accepted": 31, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.742, + "total_s": 3.495, + "prompt_tokens": 5839, + "reused_tokens": 4185, + "read_tokens": 1654, + "generated_tokens": 59, + "prompt_tps": 604.7, + "prompt_tps_source": "engine", + "decode_tps": 78.3, + "decode_tps_source": "engine", + "prompt_ms": 2735.3, + "decode_ms": 753.3, + "drafts_offered": 41, + "drafts_accepted": 32, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.351, + "total_s": 6.043, + "prompt_tokens": 9265, + "reused_tokens": 5832, + "read_tokens": 3433, + "generated_tokens": 44, + "prompt_tps": 642.7, + "prompt_tps_source": "engine", + "decode_tps": 63.6, + "decode_tps_source": "engine", + "prompt_ms": 5341.7, + "decode_ms": 691.8, + "drafts_offered": 32, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.508, + "total_s": 5.135, + "prompt_tokens": 12226, + "reused_tokens": 9258, + "read_tokens": 2968, + "generated_tokens": 50, + "prompt_tps": 659.9, + "prompt_tps_source": "engine", + "decode_tps": 79.6, + "decode_tps_source": "engine", + "prompt_ms": 4497.7, + "decode_ms": 627.9, + "drafts_offered": 37, + "drafts_accepted": 30, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.609, + "total_s": 6.506, + "prompt_tokens": 15160, + "reused_tokens": 12219, + "read_tokens": 2941, + "generated_tokens": 62, + "prompt_tps": 525.2, + "prompt_tps_source": "engine", + "decode_tps": 69.1, + "decode_tps_source": "engine", + "prompt_ms": 5599.3, + "decode_ms": 897.3, + "drafts_offered": 44, + "drafts_accepted": 33, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.629, + "total_s": 1.751, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 374.4, + "prompt_tps_source": "engine", + "decode_tps": 89.9, + "decode_tps_source": "engine", + "prompt_ms": 1626.5, + "decode_ms": 122.4, + "drafts_offered": 8, + "drafts_accepted": 6, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.596, + "total_s": 5.175, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 42, + "prompt_tps": 778.0, + "prompt_tps_source": "engine", + "decode_tps": 72.5, + "decode_tps_source": "engine", + "prompt_ms": 4591.0, + "decode_ms": 579.1, + "drafts_offered": 33, + "drafts_accepted": 21, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.73, + "total_s": 3.464, + "prompt_tokens": 5808, + "reused_tokens": 4167, + "read_tokens": 1641, + "generated_tokens": 58, + "prompt_tps": 602.1, + "prompt_tps_source": "engine", + "decode_tps": 79.0, + "decode_tps_source": "engine", + "prompt_ms": 2725.5, + "decode_ms": 734.1, + "drafts_offered": 39, + "drafts_accepted": 31, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.283, + "total_s": 5.799, + "prompt_tokens": 9233, + "reused_tokens": 5801, + "read_tokens": 3432, + "generated_tokens": 41, + "prompt_tps": 650.4, + "prompt_tps_source": "engine", + "decode_tps": 79.4, + "decode_tps_source": "engine", + "prompt_ms": 5276.8, + "decode_ms": 516.1, + "drafts_offered": 24, + "drafts_accepted": 22, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.506, + "total_s": 5.172, + "prompt_tokens": 12191, + "reused_tokens": 9226, + "read_tokens": 2965, + "generated_tokens": 54, + "prompt_tps": 659.0, + "prompt_tps_source": "engine", + "decode_tps": 81.0, + "decode_tps_source": "engine", + "prompt_ms": 4498.9, + "decode_ms": 666.4, + "drafts_offered": 39, + "drafts_accepted": 33, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.571, + "total_s": 6.333, + "prompt_tokens": 15129, + "reused_tokens": 12184, + "read_tokens": 2945, + "generated_tokens": 53, + "prompt_tps": 529.5, + "prompt_tps_source": "engine", + "decode_tps": 69.5, + "decode_tps_source": "engine", + "prompt_ms": 5562.3, + "decode_ms": 762.3, + "drafts_offered": 40, + "drafts_accepted": 29, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.497, + "total_s": 1.616, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 11, + "prompt_tps": 407.5, + "prompt_tps_source": "engine", + "decode_tps": 91.7, + "decode_tps_source": "engine", + "prompt_ms": 1494.5, + "decode_ms": 120.0, + "drafts_offered": 8, + "drafts_accepted": 6, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.863, + "total_s": 5.372, + "prompt_tokens": 4174, + "reused_tokens": 602, + "read_tokens": 3572, + "generated_tokens": 42, + "prompt_tps": 735.3, + "prompt_tps_source": "engine", + "decode_tps": 82.4, + "decode_tps_source": "engine", + "prompt_ms": 4857.9, + "decode_ms": 509.9, + "drafts_offered": 30, + "drafts_accepted": 24, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.727, + "total_s": 3.416, + "prompt_tokens": 5808, + "reused_tokens": 4167, + "read_tokens": 1641, + "generated_tokens": 58, + "prompt_tps": 602.7, + "prompt_tps_source": "engine", + "decode_tps": 84.1, + "decode_tps_source": "engine", + "prompt_ms": 2722.7, + "decode_ms": 689.6, + "drafts_offered": 40, + "drafts_accepted": 34, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.243, + "total_s": 5.852, + "prompt_tokens": 9233, + "reused_tokens": 5801, + "read_tokens": 3432, + "generated_tokens": 45, + "prompt_tps": 655.4, + "prompt_tps_source": "engine", + "decode_tps": 73.9, + "decode_tps_source": "engine", + "prompt_ms": 5236.3, + "decode_ms": 609.0, + "drafts_offered": 29, + "drafts_accepted": 23, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.504, + "total_s": 5.041, + "prompt_tokens": 12195, + "reused_tokens": 9226, + "read_tokens": 2969, + "generated_tokens": 45, + "prompt_tps": 660.3, + "prompt_tps_source": "engine", + "decode_tps": 83.7, + "decode_tps_source": "engine", + "prompt_ms": 4496.2, + "decode_ms": 537.5, + "drafts_offered": 32, + "drafts_accepted": 28, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.578, + "total_s": 6.24, + "prompt_tokens": 15124, + "reused_tokens": 12188, + "read_tokens": 2936, + "generated_tokens": 47, + "prompt_tps": 527.2, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 5569.3, + "decode_ms": 662.3, + "drafts_offered": 33, + "drafts_accepted": 25, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.733, + "min": 68.722, + "max": 68.755, + "n": 3 + }, + "reread_tokens": { + "median": 241, + "min": 236, + "max": 272, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.597, + "min": 2.727, + "max": 5.609, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15129, + "min": 15124, + "max": 15160, + "n": 3 + }, + "turn_decode_tps": { + "median": 78.7, + "min": 63.6, + "max": 84.1, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.582, + "min": 1.497, + "max": 1.629, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/client.log new file mode 100644 index 000000000..ed73b80a3 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-control", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out/official-control"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 83.9 tok/s, first token 0.41 s, 100 tokens + run 1 q2: decode 68.8 tok/s, first token 0.372 s, 100 tokens + run 1 q3: decode 71.4 tok/s, first token 0.45 s, 100 tokens + run 2 q1: decode 92.8 tok/s, first token 0.397 s, 100 tokens + run 2 q2: decode 80.7 tok/s, first token 0.393 s, 100 tokens + run 2 q3: decode 79.5 tok/s, first token 0.443 s, 100 tokens + run 3 q1: decode 94.9 tok/s, first token 0.455 s, 100 tokens + run 3 q2: decode 80.6 tok/s, first token 0.459 s, 100 tokens + run 3 q3: decode 81.5 tok/s, first token 0.414 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 873.0 tok/s (0 reused), first token 4.609 s + run 2: 4019 tokens, prompt 924.8 tok/s (0 reused), first token 4.351 s + run 3: 4019 tokens, prompt 921.4 tok/s (0 reused), first token 4.367 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1019.4 tok/s (0 reused), first token 15.768 s + run 2: 16058 tokens, prompt 1026.4 tok/s (0 reused), first token 15.659 s + run 3: 16058 tokens, prompt 1009.5 tok/s (0 reused), first token 15.921 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.582 s + run 1 turn 1: prompt 4192, reused 602, read 3590, first token 4.597 s + run 1 turn 2: prompt 5839, reused 4185, read 1654, first token 2.742 s + run 1 turn 3: prompt 9265, reused 5832, read 3433, first token 5.351 s + run 1 turn 4: prompt 12226, reused 9258, read 2968, first token 4.508 s + run 1 turn 5: prompt 15160, reused 12219, read 2941, first token 5.609 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.629 s + run 2 turn 1: prompt 4174, reused 602, read 3572, first token 4.596 s + run 2 turn 2: prompt 5808, reused 4167, read 1641, first token 2.73 s + run 2 turn 3: prompt 9233, reused 5801, read 3432, first token 5.283 s + run 2 turn 4: prompt 12191, reused 9226, read 2965, first token 4.506 s + run 2 turn 5: prompt 15129, reused 12184, read 2945, first token 5.571 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.497 s + run 3 turn 1: prompt 4174, reused 602, read 3572, first token 4.863 s + run 3 turn 2: prompt 5808, reused 4167, read 1641, first token 2.727 s + run 3 turn 3: prompt 9233, reused 5801, read 3432, first token 5.243 s + run 3 turn 4: prompt 12195, reused 9226, read 2969, first token 4.504 s + run 3 turn 5: prompt 15124, reused 12188, read 2936, first token 5.578 s + +Results (median, range over the runs): + chat decode tok/s 80.70 (68.80 - 94.90, n=9) + chat first token (s) 0.41 (0.37 - 0.46, n=9) + 4K prompt tok/s 921.40 (873.00 - 924.80, n=3) + 4K first token (s) 4.37 (4.35 - 4.61, n=3) + 16K prompt tok/s 1019.40 (1009.50 - 1026.40, n=3) + 16K first token (s) 15.77 (15.66 - 15.92, n=3) + agent cache hit % 68.73 (68.72 - 68.75, n=3) + agent re-read tokens/session 241.00 (236.00 - 272.00, n=3) + agent turn first token (s) 4.60 (2.73 - 5.61, n=15) + agent decode tok/s 78.70 (63.60 - 84.10, n=15) + +Report: /opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/settings.json new file mode 100644 index 000000000..9ed023348 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/settings.json @@ -0,0 +1,44 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/qwen36-UD-Q4_K_M/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/out/expert-profile.txt", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/qwen36-ud-q4_k_m/tokenizer", + "model_name": "qwen3.6-35b-a3b-ud-q4_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [], + "weight_sha256": "0b21525e972670ed59e1812e170b27c26355381f0656ecc4e25617ece7dac58b", + "weight_filename": "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "weight_bytes": 22663387424, + "historical_weight_sha256": "0b21525e972670ed59e1812e170b27c26355381f0656ecc4e25617ece7dac58b", + "byte_identical_to_historical_weights": true, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/stage.json new file mode 100644 index 000000000..d554b0749 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx32768/official-control/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 167.04158919400652 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md new file mode 100644 index 000000000..fbb520db9 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: qwen3.6-35b-a3b-ud-q4_k_m +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --tail-skip 0 --kv-whole --conversation-cache-mib 0 --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 79.7 (63.7-87.0) | 0.44 (0.42-0.49) | +| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 912.1 (880.7-915.5) | not measured | 4.41 (4.39-4.57) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1021.3 (1017.4-1025.1) | not measured | 15.74 (15.68-15.80) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 816.6 (814.9-819.0) | not measured | 39.28 (39.16-39.35) | +| agent session, follow-up turns | 15128 (15115-15155) at the last turn | see below | up to 64 | 3 | see below | 76.5 (68.5-88.1) | 4.66 (2.77-5.65) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 68.8 % (68.8-68.8); earlier-turn tokens read again per session: 240 (227-267) (earlier turns that the cache did not hold; the 14314 (14314-14314) new tokens of the files are read once); first turn (nothing cached) first token: 1.67 s (1.67-1.72). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 609 | 0 | 609 | 355.9 | 1.715 | 88.7 | +| 1 | 4185 | 602 | 3583 | 736.3 | 4.874 | 68.5 | +| 2 | 5817 | 4178 | 1639 | 592.4 | 2.774 | 75.7 | +| 3 | 9235 | 5810 | 3425 | 639.3 | 5.367 | 76.5 | +| 4 | 12187 | 9228 | 2959 | 660.9 | 4.487 | 74.2 | +| 5 | 15115 | 12180 | 2935 | 523.5 | 5.616 | 73.1 | + +### Concurrency: 2 clients + +Total 60.8 tok/s (58.3-61.0); first token p50 1.587 s, p95 2.764 s; request latency p50 3.245 s, p95 4.345 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 300 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json new file mode 100644 index 000000000..a6adba126 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/results.json @@ -0,0 +1,1034 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 300.2464425563812, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "qwen3.6-35b-a3b-ud-q4_k_m", + "quant": "UD-Q4_K_M", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "qwen3.6-35b-a3b-ud-q4_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_M", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --max-context 65536 --expert-profile expert-profile.txt --tail-skip 0 --kv-whole --conversation-cache-mib 0 --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "qwen3.6-35b-a3b-ud-q4_k_m", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 3474, + "expert_cache_mib": 6331, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.427, + "total_s": 1.577, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 119.9, + "prompt_tps_source": "engine", + "decode_tps": 87.0, + "decode_tps_source": "engine", + "prompt_ms": 425.4, + "decode_ms": 1149.8, + "drafts_offered": 75, + "drafts_accepted": 70, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.44, + "total_s": 2.008, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 98.2, + "prompt_tps_source": "engine", + "decode_tps": 63.7, + "decode_tps_source": "engine", + "prompt_ms": 438.1, + "decode_ms": 1568.7, + "drafts_offered": 60, + "drafts_accepted": 50, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.46, + "total_s": 1.952, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 100.3, + "prompt_tps_source": "engine", + "decode_tps": 67.0, + "decode_tps_source": "engine", + "prompt_ms": 458.5, + "decode_ms": 1492.0, + "drafts_offered": 66, + "drafts_accepted": 51, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.46, + "total_s": 1.629, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 111.4, + "prompt_tps_source": "engine", + "decode_tps": 85.5, + "decode_tps_source": "engine", + "prompt_ms": 458.0, + "decode_ms": 1169.2, + "drafts_offered": 76, + "drafts_accepted": 67, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.423, + "total_s": 1.827, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 102.1, + "prompt_tps_source": "engine", + "decode_tps": 71.2, + "decode_tps_source": "engine", + "prompt_ms": 421.3, + "decode_ms": 1404.0, + "drafts_offered": 66, + "drafts_accepted": 46, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.46, + "total_s": 1.789, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 100.4, + "prompt_tps_source": "engine", + "decode_tps": 75.2, + "decode_tps_source": "engine", + "prompt_ms": 458.0, + "decode_ms": 1329.2, + "drafts_offered": 71, + "drafts_accepted": 54, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.489, + "total_s": 1.655, + "prompt_tokens": 51, + "reused_tokens": 0, + "read_tokens": 51, + "generated_tokens": 100, + "prompt_tps": 104.7, + "prompt_tps_source": "engine", + "decode_tps": 85.8, + "decode_tps_source": "engine", + "prompt_ms": 487.0, + "decode_ms": 1166.1, + "drafts_offered": 75, + "drafts_accepted": 69, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.439, + "total_s": 1.692, + "prompt_tokens": 43, + "reused_tokens": 0, + "read_tokens": 43, + "generated_tokens": 100, + "prompt_tps": 98.4, + "prompt_tps_source": "engine", + "decode_tps": 79.7, + "decode_tps_source": "engine", + "prompt_ms": 436.8, + "decode_ms": 1254.1, + "drafts_offered": 67, + "drafts_accepted": 54, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.421, + "total_s": 1.653, + "prompt_tokens": 46, + "reused_tokens": 0, + "read_tokens": 46, + "generated_tokens": 100, + "prompt_tps": 109.8, + "prompt_tps_source": "engine", + "decode_tps": 81.1, + "decode_tps_source": "engine", + "prompt_ms": 419.1, + "decode_ms": 1232.3, + "drafts_offered": 68, + "drafts_accepted": 54, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 79.7, + "min": 63.7, + "max": 87.0, + "n": 9 + }, + "ttft_s": { + "median": 0.44, + "min": 0.421, + "max": 0.489, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 46, + "min": 43, + "max": 51, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 4.411, + "total_s": 4.784, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 912.1, + "prompt_tps_source": "engine", + "decode_tps": 64.5, + "decode_tps_source": "engine", + "prompt_ms": 4406.4, + "decode_ms": 372.2, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.569, + "total_s": 4.922, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 880.7, + "prompt_tps_source": "engine", + "decode_tps": 67.8, + "decode_tps_source": "engine", + "prompt_ms": 4563.6, + "decode_ms": 353.9, + "drafts_offered": 17, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.395, + "total_s": 4.729, + "prompt_tokens": 4019, + "reused_tokens": 0, + "read_tokens": 4019, + "generated_tokens": 24, + "prompt_tps": 915.5, + "prompt_tps_source": "engine", + "decode_tps": 71.7, + "decode_tps_source": "engine", + "prompt_ms": 4390.1, + "decode_ms": 334.6, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 2 + } + ], + "prompt_tps": { + "median": 912.1, + "min": 880.7, + "max": 915.5, + "n": 3 + }, + "ttft_s": { + "median": 4.411, + "min": 4.395, + "max": 4.569, + "n": 3 + }, + "prompt_tokens": { + "median": 4019, + "min": 4019, + "max": 4019, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 15.798, + "total_s": 16.169, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1017.4, + "prompt_tps_source": "engine", + "decode_tps": 64.7, + "decode_tps_source": "engine", + "prompt_ms": 15783.6, + "decode_ms": 370.9, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.737, + "total_s": 16.103, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1021.3, + "prompt_tps_source": "engine", + "decode_tps": 65.5, + "decode_tps_source": "engine", + "prompt_ms": 15723.0, + "decode_ms": 366.5, + "drafts_offered": 18, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 15.68, + "total_s": 16.059, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1025.1, + "prompt_tps_source": "engine", + "decode_tps": 63.2, + "decode_tps_source": "engine", + "prompt_ms": 15665.2, + "decode_ms": 379.5, + "drafts_offered": 15, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 1021.3, + "min": 1017.4, + "max": 1025.1, + "n": 3 + }, + "ttft_s": { + "median": 15.737, + "min": 15.68, + "max": 15.798, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 39.354, + "total_s": 39.943, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 814.9, + "prompt_tps_source": "engine", + "decode_tps": 40.6, + "decode_tps_source": "engine", + "prompt_ms": 39326.4, + "decode_ms": 590.6, + "drafts_offered": 24, + "drafts_accepted": 8, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 39.157, + "total_s": 39.662, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 819.0, + "prompt_tps_source": "engine", + "decode_tps": 47.4, + "decode_tps_source": "engine", + "prompt_ms": 39129.6, + "decode_ms": 505.9, + "drafts_offered": 22, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 39.276, + "total_s": 39.737, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 816.6, + "prompt_tps_source": "engine", + "decode_tps": 51.9, + "decode_tps_source": "engine", + "prompt_ms": 39249.0, + "decode_ms": 462.3, + "drafts_offered": 19, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 816.6, + "min": 814.9, + "max": 819.0, + "n": 3 + }, + "ttft_s": { + "median": 39.276, + "min": 39.157, + "max": 39.354, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.715, + "total_s": 1.963, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 22, + "prompt_tps": 355.9, + "prompt_tps_source": "engine", + "decode_tps": 88.7, + "decode_tps_source": "engine", + "prompt_ms": 1711.3, + "decode_ms": 248.0, + "drafts_offered": 12, + "drafts_accepted": 12, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.874, + "total_s": 5.457, + "prompt_tokens": 4185, + "reused_tokens": 602, + "read_tokens": 3583, + "generated_tokens": 40, + "prompt_tps": 736.3, + "prompt_tps_source": "engine", + "decode_tps": 68.5, + "decode_tps_source": "engine", + "prompt_ms": 4866.5, + "decode_ms": 583.8, + "drafts_offered": 32, + "drafts_accepted": 22, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.774, + "total_s": 3.447, + "prompt_tokens": 5817, + "reused_tokens": 4178, + "read_tokens": 1639, + "generated_tokens": 51, + "prompt_tps": 592.4, + "prompt_tps_source": "engine", + "decode_tps": 75.7, + "decode_tps_source": "engine", + "prompt_ms": 2766.7, + "decode_ms": 674.0, + "drafts_offered": 36, + "drafts_accepted": 28, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.367, + "total_s": 5.823, + "prompt_tokens": 9235, + "reused_tokens": 5810, + "read_tokens": 3425, + "generated_tokens": 35, + "prompt_tps": 639.3, + "prompt_tps_source": "engine", + "decode_tps": 76.5, + "decode_tps_source": "engine", + "prompt_ms": 5357.1, + "decode_ms": 457.3, + "drafts_offered": 21, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.487, + "total_s": 5.08, + "prompt_tokens": 12187, + "reused_tokens": 9228, + "read_tokens": 2959, + "generated_tokens": 44, + "prompt_tps": 660.9, + "prompt_tps_source": "engine", + "decode_tps": 74.2, + "decode_tps_source": "engine", + "prompt_ms": 4477.5, + "decode_ms": 593.1, + "drafts_offered": 34, + "drafts_accepted": 25, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.616, + "total_s": 6.203, + "prompt_tokens": 15115, + "reused_tokens": 12180, + "read_tokens": 2935, + "generated_tokens": 43, + "prompt_tps": 523.5, + "prompt_tps_source": "engine", + "decode_tps": 73.1, + "decode_tps_source": "engine", + "prompt_ms": 5606.8, + "decode_ms": 588.0, + "drafts_offered": 28, + "drafts_accepted": 23, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.67, + "total_s": 2.382, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 58, + "prompt_tps": 365.1, + "prompt_tps_source": "engine", + "decode_tps": 81.5, + "decode_tps_source": "engine", + "prompt_ms": 1668.1, + "decode_ms": 712.0, + "drafts_offered": 44, + "drafts_accepted": 34, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.659, + "total_s": 5.186, + "prompt_tokens": 4221, + "reused_tokens": 602, + "read_tokens": 3619, + "generated_tokens": 40, + "prompt_tps": 777.7, + "prompt_tps_source": "engine", + "decode_tps": 75.8, + "decode_tps_source": "engine", + "prompt_ms": 4653.5, + "decode_ms": 527.5, + "drafts_offered": 35, + "drafts_accepted": 23, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.774, + "total_s": 3.445, + "prompt_tokens": 5853, + "reused_tokens": 4214, + "read_tokens": 1639, + "generated_tokens": 53, + "prompt_tps": 591.7, + "prompt_tps_source": "engine", + "decode_tps": 78.9, + "decode_tps_source": "engine", + "prompt_ms": 2769.8, + "decode_ms": 671.5, + "drafts_offered": 37, + "drafts_accepted": 29, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.381, + "total_s": 5.813, + "prompt_tokens": 9273, + "reused_tokens": 5846, + "read_tokens": 3427, + "generated_tokens": 35, + "prompt_tps": 637.6, + "prompt_tps_source": "engine", + "decode_tps": 81.0, + "decode_tps_source": "engine", + "prompt_ms": 5375.0, + "decode_ms": 432.3, + "drafts_offered": 22, + "drafts_accepted": 19, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.498, + "total_s": 5.019, + "prompt_tokens": 12225, + "reused_tokens": 9266, + "read_tokens": 2959, + "generated_tokens": 46, + "prompt_tps": 658.9, + "prompt_tps_source": "engine", + "decode_tps": 88.1, + "decode_tps_source": "engine", + "prompt_ms": 4490.6, + "decode_ms": 521.9, + "drafts_offered": 30, + "drafts_accepted": 29, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.653, + "total_s": 6.297, + "prompt_tokens": 15155, + "reused_tokens": 12218, + "read_tokens": 2937, + "generated_tokens": 47, + "prompt_tps": 520.3, + "prompt_tps_source": "engine", + "decode_tps": 72.9, + "decode_tps_source": "engine", + "prompt_ms": 5644.3, + "decode_ms": 644.3, + "drafts_offered": 33, + "drafts_accepted": 26, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.674, + "total_s": 1.962, + "prompt_tokens": 609, + "reused_tokens": 0, + "read_tokens": 609, + "generated_tokens": 26, + "prompt_tps": 364.4, + "prompt_tps_source": "engine", + "decode_tps": 90.2, + "decode_tps_source": "engine", + "prompt_ms": 1671.2, + "decode_ms": 288.4, + "drafts_offered": 21, + "drafts_accepted": 16, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.627, + "total_s": 5.136, + "prompt_tokens": 4189, + "reused_tokens": 602, + "read_tokens": 3587, + "generated_tokens": 40, + "prompt_tps": 776.0, + "prompt_tps_source": "engine", + "decode_tps": 78.6, + "decode_tps_source": "engine", + "prompt_ms": 4622.2, + "decode_ms": 508.8, + "drafts_offered": 34, + "drafts_accepted": 23, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.952, + "total_s": 3.696, + "prompt_tokens": 5821, + "reused_tokens": 4182, + "read_tokens": 1639, + "generated_tokens": 59, + "prompt_tps": 556.1, + "prompt_tps_source": "engine", + "decode_tps": 79.3, + "decode_tps_source": "engine", + "prompt_ms": 2947.4, + "decode_ms": 744.3, + "drafts_offered": 44, + "drafts_accepted": 34, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.413, + "total_s": 5.866, + "prompt_tokens": 9247, + "reused_tokens": 5814, + "read_tokens": 3433, + "generated_tokens": 35, + "prompt_tps": 635.1, + "prompt_tps_source": "engine", + "decode_tps": 77.1, + "decode_tps_source": "engine", + "prompt_ms": 5405.8, + "decode_ms": 454.2, + "drafts_offered": 22, + "drafts_accepted": 18, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.498, + "total_s": 5.055, + "prompt_tokens": 12199, + "reused_tokens": 9240, + "read_tokens": 2959, + "generated_tokens": 45, + "prompt_tps": 658.9, + "prompt_tps_source": "engine", + "decode_tps": 80.7, + "decode_tps_source": "engine", + "prompt_ms": 4490.8, + "decode_ms": 557.6, + "drafts_offered": 32, + "drafts_accepted": 27, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.635, + "total_s": 6.274, + "prompt_tokens": 15128, + "reused_tokens": 12192, + "read_tokens": 2936, + "generated_tokens": 47, + "prompt_tps": 521.8, + "prompt_tps_source": "engine", + "decode_tps": 73.5, + "decode_tps_source": "engine", + "prompt_ms": 5626.5, + "decode_ms": 639.6, + "drafts_offered": 37, + "drafts_accepted": 28, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 68.758, + "min": 68.755, + "max": 68.795, + "n": 3 + }, + "reread_tokens": { + "median": 240, + "min": 227, + "max": 267, + "n": 3 + }, + "new_tokens": { + "median": 14314, + "min": 14314, + "max": 14314, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.659, + "min": 2.774, + "max": 5.653, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15128, + "min": 15115, + "max": 15155, + "n": 3 + }, + "turn_decode_tps": { + "median": 76.5, + "min": 68.5, + "max": 88.1, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.674, + "min": 1.67, + "max": 1.715, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 4.39, + "tokens": 256, + "total_tps": 58.33, + "first_s": [ + 0.5585873180243652, + 2.8005811129987705 + ], + "latency_s": [ + 2.2956133230181877, + 4.388507377996575 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 4.19, + "tokens": 256, + "total_tps": 61.04, + "first_s": [ + 0.4709917109867092, + 2.6533884149976075 + ], + "latency_s": [ + 2.1517698469979223, + 4.1933869479980785 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 4.21, + "tokens": 256, + "total_tps": 60.75, + "first_s": [ + 0.4986253659881186, + 2.6162478619953617 + ], + "latency_s": [ + 2.103186975000426, + 4.2133163859834895 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 60.75, + "min": 58.33, + "max": 61.04, + "n": 3 + }, + "ttft_p50_s": 1.587, + "ttft_p95_s": 2.764, + "latency_p50_s": 3.245, + "latency_p95_s": 4.345, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/client.log new file mode 100644 index 000000000..d3371c1ad --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-control", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out/official-control", "--long", "--clients", "2"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 87.0 tok/s, first token 0.427 s, 100 tokens + run 1 q2: decode 63.7 tok/s, first token 0.44 s, 100 tokens + run 1 q3: decode 67.0 tok/s, first token 0.46 s, 100 tokens + run 2 q1: decode 85.5 tok/s, first token 0.46 s, 100 tokens + run 2 q2: decode 71.2 tok/s, first token 0.423 s, 100 tokens + run 2 q3: decode 75.2 tok/s, first token 0.46 s, 100 tokens + run 3 q1: decode 85.8 tok/s, first token 0.489 s, 100 tokens + run 3 q2: decode 79.7 tok/s, first token 0.439 s, 100 tokens + run 3 q3: decode 81.1 tok/s, first token 0.421 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4019 tokens, prompt 912.1 tok/s (0 reused), first token 4.411 s + run 2: 4019 tokens, prompt 880.7 tok/s (0 reused), first token 4.569 s + run 3: 4019 tokens, prompt 915.5 tok/s (0 reused), first token 4.395 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1017.4 tok/s (0 reused), first token 15.798 s + run 2: 16058 tokens, prompt 1021.3 tok/s (0 reused), first token 15.737 s + run 3: 16058 tokens, prompt 1025.1 tok/s (0 reused), first token 15.68 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 609, reused 0, read 609, first token 1.715 s + run 1 turn 1: prompt 4185, reused 602, read 3583, first token 4.874 s + run 1 turn 2: prompt 5817, reused 4178, read 1639, first token 2.774 s + run 1 turn 3: prompt 9235, reused 5810, read 3425, first token 5.367 s + run 1 turn 4: prompt 12187, reused 9228, read 2959, first token 4.487 s + run 1 turn 5: prompt 15115, reused 12180, read 2935, first token 5.616 s + run 2 turn 0: prompt 609, reused 0, read 609, first token 1.67 s + run 2 turn 1: prompt 4221, reused 602, read 3619, first token 4.659 s + run 2 turn 2: prompt 5853, reused 4214, read 1639, first token 2.774 s + run 2 turn 3: prompt 9273, reused 5846, read 3427, first token 5.381 s + run 2 turn 4: prompt 12225, reused 9266, read 2959, first token 4.498 s + run 2 turn 5: prompt 15155, reused 12218, read 2937, first token 5.653 s + run 3 turn 0: prompt 609, reused 0, read 609, first token 1.674 s + run 3 turn 1: prompt 4189, reused 602, read 3587, first token 4.627 s + run 3 turn 2: prompt 5821, reused 4182, read 1639, first token 2.952 s + run 3 turn 3: prompt 9247, reused 5814, read 3433, first token 5.413 s + run 3 turn 4: prompt 12199, reused 9240, read 2959, first token 4.498 s + run 3 turn 5: prompt 15128, reused 12192, read 2936, first token 5.635 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 4.4 s = 58.3 tok/s total + round 2: 256 tokens in 4.2 s = 61.0 tok/s total + round 3: 256 tokens in 4.2 s = 60.8 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32049 tokens, prompt 814.9 tok/s (0 reused), first token 39.354 s + run 2: 32049 tokens, prompt 819.0 tok/s (0 reused), first token 39.157 s + run 3: 32049 tokens, prompt 816.6 tok/s (0 reused), first token 39.276 s + +Results (median, range over the runs): + chat decode tok/s 79.70 (63.70 - 87.00, n=9) + chat first token (s) 0.44 (0.42 - 0.49, n=9) + 4K prompt tok/s 912.10 (880.70 - 915.50, n=3) + 4K first token (s) 4.41 (4.39 - 4.57, n=3) + 16K prompt tok/s 1021.30 (1017.40 - 1025.10, n=3) + 16K first token (s) 15.74 (15.68 - 15.80, n=3) + 32K prompt tok/s 816.60 (814.90 - 819.00, n=3) + 32K first token (s) 39.28 (39.16 - 39.35, n=3) + agent cache hit % 68.76 (68.75 - 68.80, n=3) + agent re-read tokens/session 240.00 (227.00 - 267.00, n=3) + agent turn first token (s) 4.66 (2.77 - 5.65, n=15) + agent decode tok/s 76.50 (68.50 - 88.10, n=15) + 2 clients total tok/s 60.75 (58.33 - 61.04, n=3) + +Report: /opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out/official-control/2026-10-19-community-cpu-qwen3-6-35b-a3b-ud-q4-k-m (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/settings.json new file mode 100644 index 000000000..45342708f --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/settings.json @@ -0,0 +1,44 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/qwen36-UD-Q4_K_M/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/out/expert-profile.txt", + "--tail-skip", + "0", + "--kv-whole", + "--conversation-cache-mib", + "0", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/qwen36-ud-q4_k_m/tokenizer", + "model_name": "qwen3.6-35b-a3b-ud-q4_k_m", + "display_name": "Qwen3.6-35B-A3B UD-Q4_K_M", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [], + "weight_sha256": "0b21525e972670ed59e1812e170b27c26355381f0656ecc4e25617ece7dac58b", + "weight_filename": "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + "weight_bytes": 22663387424, + "historical_weight_sha256": "0b21525e972670ed59e1812e170b27c26355381f0656ecc4e25617ece7dac58b", + "byte_identical_to_historical_weights": true, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/stage.json new file mode 100644 index 000000000..debb4c2e6 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/qwen36-ud-q4_k_m-control-reserve1024-ctx65536/official-control/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 300.470634953992 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md new file mode 100644 index 000000000..8de4c6b51 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md @@ -0,0 +1,60 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: tiel-coder-35b-a3b-ud-q3_k_xl +- Context: 32768 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 224 (221-229) | 0 | 100 (100-100) | 9 | not measured | 71.0 (60.5-84.4) | 0.65 (0.64-0.72) | +| 4K prompt, fresh | 4025 (4025-4025) | 0 (0-0) | 24 cap | 3 | 1159.7 (1158.8-1161.2) | not measured | 3.48 (3.47-3.48) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1077.7 (1067.5-1082.8) | not measured | 14.91 (14.84-15.06) | +| agent session, follow-up turns | 15373 (15314-15392) at the last turn | see below | up to 64 | 3 | see below | 63.1 (54.2-74.5) | 4.07 (1.99-5.20) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 70.0 % (69.9-70.0); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 14334 (14334-14334) new tokens of the files are read once); first turn (nothing cached) first token: 1.28 s (1.23-1.31). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 835 | 0 | 835 | 655.2 | 1.278 | 60.4 | +| 1 | 4403 | 844 | 3559 | 958.8 | 3.719 | 67.0 | +| 2 | 6043 | 4446 | 1597 | 797.5 | 2.009 | 63.1 | +| 3 | 9466 | 6094 | 3372 | 669.5 | 5.046 | 62.7 | +| 4 | 12432 | 9510 | 2922 | 701.2 | 4.177 | 67.6 | +| 5 | 15373 | 12484 | 2889 | 712.5 | 4.065 | 66.7 | + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 154 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json new file mode 100644 index 000000000..8c6192e2b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json @@ -0,0 +1,883 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 0, + "long": false, + "started_server": false, + "command": "python tools/strata_bench.py" + }, + "elapsed_s": 154.03256607055664, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "quant": "UD-Q3_K_XL", + "context": 32768, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "tiel-coder-35b-a3b-ud-q3_k_xl", + "display_name": "Tiel Coder 35B-A3B UD-Q3_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 32768 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "max_context": 32768, + "images": false, + "version": "0.1.45", + "context": 32768, + "expert_slots": 5310, + "expert_cache_mib": 7084, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.718, + "total_s": 2.046, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 319.9, + "prompt_tps_source": "engine", + "decode_tps": 75.3, + "decode_tps_source": "engine", + "prompt_ms": 715.9, + "decode_ms": 1328.7, + "drafts_offered": 65, + "drafts_accepted": 62, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.676, + "total_s": 2.286, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 327.8, + "prompt_tps_source": "engine", + "decode_tps": 62.1, + "decode_tps_source": "engine", + "prompt_ms": 674.2, + "decode_ms": 1610.1, + "drafts_offered": 65, + "drafts_accepted": 50, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.65, + "total_s": 2.302, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 345.4, + "prompt_tps_source": "engine", + "decode_tps": 60.5, + "decode_tps_source": "engine", + "prompt_ms": 648.5, + "decode_ms": 1652.0, + "drafts_offered": 67, + "drafts_accepted": 42, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.655, + "total_s": 1.899, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 350.6, + "prompt_tps_source": "engine", + "decode_tps": 80.4, + "decode_tps_source": "engine", + "prompt_ms": 653.1, + "decode_ms": 1243.8, + "drafts_offered": 74, + "drafts_accepted": 66, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.64, + "total_s": 2.049, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 346.1, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 638.6, + "decode_ms": 1408.6, + "drafts_offered": 61, + "drafts_accepted": 47, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.65, + "total_s": 2.136, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 345.8, + "prompt_tps_source": "engine", + "decode_tps": 67.3, + "decode_tps_source": "engine", + "prompt_ms": 647.8, + "decode_ms": 1486.8, + "drafts_offered": 65, + "drafts_accepted": 46, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.641, + "total_s": 1.825, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 358.1, + "prompt_tps_source": "engine", + "decode_tps": 84.4, + "decode_tps_source": "engine", + "prompt_ms": 639.4, + "decode_ms": 1184.3, + "drafts_offered": 73, + "drafts_accepted": 67, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.651, + "total_s": 1.992, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 340.7, + "prompt_tps_source": "engine", + "decode_tps": 74.5, + "decode_tps_source": "engine", + "prompt_ms": 648.7, + "decode_ms": 1342.0, + "drafts_offered": 60, + "drafts_accepted": 49, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.672, + "total_s": 2.18, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 334.1, + "prompt_tps_source": "engine", + "decode_tps": 66.3, + "decode_tps_source": "engine", + "prompt_ms": 670.4, + "decode_ms": 1508.3, + "drafts_offered": 67, + "drafts_accepted": 46, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 71.0, + "min": 60.5, + "max": 84.4, + "n": 9 + }, + "ttft_s": { + "median": 0.651, + "min": 0.64, + "max": 0.718, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 224, + "min": 221, + "max": 229, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.471, + "total_s": 3.846, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1161.2, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 3466.2, + "decode_ms": 374.4, + "drafts_offered": 14, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.478, + "total_s": 3.8, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1158.8, + "prompt_tps_source": "engine", + "decode_tps": 74.7, + "decode_tps_source": "engine", + "prompt_ms": 3473.3, + "decode_ms": 321.5, + "drafts_offered": 15, + "drafts_accepted": 13, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.475, + "total_s": 3.818, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1159.7, + "prompt_tps_source": "engine", + "decode_tps": 70.0, + "decode_tps_source": "engine", + "prompt_ms": 3470.6, + "decode_ms": 343.0, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 2 + } + ], + "prompt_tps": { + "median": 1159.7, + "min": 1158.8, + "max": 1161.2, + "n": 3 + }, + "ttft_s": { + "median": 3.475, + "min": 3.471, + "max": 3.478, + "n": 3 + }, + "prompt_tokens": { + "median": 4025, + "min": 4025, + "max": 4025, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 15.057, + "total_s": 15.46, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1067.5, + "prompt_tps_source": "engine", + "decode_tps": 59.5, + "decode_tps_source": "engine", + "prompt_ms": 15042.3, + "decode_ms": 403.1, + "drafts_offered": 16, + "drafts_accepted": 11, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.844, + "total_s": 15.297, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1082.8, + "prompt_tps_source": "engine", + "decode_tps": 53.0, + "decode_tps_source": "engine", + "prompt_ms": 14829.4, + "decode_ms": 452.8, + "drafts_offered": 18, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.915, + "total_s": 15.335, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1077.7, + "prompt_tps_source": "engine", + "decode_tps": 57.2, + "decode_tps_source": "engine", + "prompt_ms": 14900.3, + "decode_ms": 419.9, + "drafts_offered": 15, + "drafts_accepted": 10, + "run": 2 + } + ], + "prompt_tps": { + "median": 1077.7, + "min": 1067.5, + "max": 1082.8, + "n": 3 + }, + "ttft_s": { + "median": 14.915, + "min": 14.844, + "max": 15.057, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.278, + "total_s": 1.444, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 655.2, + "prompt_tps_source": "engine", + "decode_tps": 60.4, + "decode_tps_source": "engine", + "prompt_ms": 1274.5, + "decode_ms": 165.6, + "drafts_offered": 10, + "drafts_accepted": 5, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.719, + "total_s": 4.376, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 44, + "prompt_tps": 958.8, + "prompt_tps_source": "engine", + "decode_tps": 67.0, + "decode_tps_source": "engine", + "prompt_ms": 3712.0, + "decode_ms": 656.9, + "drafts_offered": 35, + "drafts_accepted": 25, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.009, + "total_s": 2.834, + "prompt_tokens": 6043, + "reused_tokens": 4446, + "read_tokens": 1597, + "generated_tokens": 52, + "prompt_tps": 797.5, + "prompt_tps_source": "engine", + "decode_tps": 63.1, + "decode_tps_source": "engine", + "prompt_ms": 2002.5, + "decode_ms": 824.7, + "drafts_offered": 43, + "drafts_accepted": 26, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.046, + "total_s": 5.764, + "prompt_tokens": 9466, + "reused_tokens": 6094, + "read_tokens": 3372, + "generated_tokens": 45, + "prompt_tps": 669.5, + "prompt_tps_source": "engine", + "decode_tps": 62.7, + "decode_tps_source": "engine", + "prompt_ms": 5036.6, + "decode_ms": 717.9, + "drafts_offered": 32, + "drafts_accepted": 21, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.177, + "total_s": 4.961, + "prompt_tokens": 12432, + "reused_tokens": 9510, + "read_tokens": 2922, + "generated_tokens": 53, + "prompt_tps": 701.2, + "prompt_tps_source": "engine", + "decode_tps": 67.6, + "decode_tps_source": "engine", + "prompt_ms": 4167.2, + "decode_ms": 783.9, + "drafts_offered": 40, + "drafts_accepted": 29, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.065, + "total_s": 4.724, + "prompt_tokens": 15373, + "reused_tokens": 12484, + "read_tokens": 2889, + "generated_tokens": 44, + "prompt_tps": 712.5, + "prompt_tps_source": "engine", + "decode_tps": 66.7, + "decode_tps_source": "engine", + "prompt_ms": 4054.8, + "decode_ms": 659.6, + "drafts_offered": 30, + "drafts_accepted": 23, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.23, + "total_s": 1.367, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 680.2, + "prompt_tps_source": "engine", + "decode_tps": 72.7, + "decode_tps_source": "engine", + "prompt_ms": 1227.6, + "decode_ms": 137.6, + "drafts_offered": 9, + "drafts_accepted": 5, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.689, + "total_s": 4.212, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 39, + "prompt_tps": 966.1, + "prompt_tps_source": "engine", + "decode_tps": 74.5, + "decode_tps_source": "engine", + "prompt_ms": 3683.7, + "decode_ms": 523.5, + "drafts_offered": 27, + "drafts_accepted": 21, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.989, + "total_s": 2.675, + "prompt_tokens": 6038, + "reused_tokens": 4441, + "read_tokens": 1597, + "generated_tokens": 44, + "prompt_tps": 804.7, + "prompt_tps_source": "engine", + "decode_tps": 64.1, + "decode_tps_source": "engine", + "prompt_ms": 1984.7, + "decode_ms": 686.2, + "drafts_offered": 32, + "drafts_accepted": 20, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.979, + "total_s": 5.412, + "prompt_tokens": 9453, + "reused_tokens": 6081, + "read_tokens": 3372, + "generated_tokens": 24, + "prompt_tps": 678.2, + "prompt_tps_source": "engine", + "decode_tps": 55.4, + "decode_tps_source": "engine", + "prompt_ms": 4972.0, + "decode_ms": 433.0, + "drafts_offered": 20, + "drafts_accepted": 9, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.1, + "total_s": 4.548, + "prompt_tokens": 12398, + "reused_tokens": 9476, + "read_tokens": 2922, + "generated_tokens": 28, + "prompt_tps": 714.0, + "prompt_tps_source": "engine", + "decode_tps": 62.5, + "decode_tps_source": "engine", + "prompt_ms": 4092.7, + "decode_ms": 448.3, + "drafts_offered": 24, + "drafts_accepted": 14, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.055, + "total_s": 4.94, + "prompt_tokens": 15314, + "reused_tokens": 12425, + "read_tokens": 2889, + "generated_tokens": 48, + "prompt_tps": 714.0, + "prompt_tps_source": "engine", + "decode_tps": 54.2, + "decode_tps_source": "engine", + "prompt_ms": 4046.1, + "decode_ms": 885.2, + "drafts_offered": 43, + "drafts_accepted": 23, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.309, + "total_s": 1.457, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 638.9, + "prompt_tps_source": "engine", + "decode_tps": 67.7, + "decode_tps_source": "engine", + "prompt_ms": 1307.0, + "decode_ms": 147.8, + "drafts_offered": 9, + "drafts_accepted": 5, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.725, + "total_s": 4.301, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 42, + "prompt_tps": 956.7, + "prompt_tps_source": "engine", + "decode_tps": 72.9, + "decode_tps_source": "engine", + "prompt_ms": 3719.9, + "decode_ms": 576.3, + "drafts_offered": 29, + "drafts_accepted": 22, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 1.985, + "total_s": 3.013, + "prompt_tokens": 6041, + "reused_tokens": 4444, + "read_tokens": 1597, + "generated_tokens": 61, + "prompt_tps": 806.5, + "prompt_tps_source": "engine", + "decode_tps": 59.3, + "decode_tps_source": "engine", + "prompt_ms": 1980.2, + "decode_ms": 1028.4, + "drafts_offered": 40, + "drafts_accepted": 22, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.202, + "total_s": 6.2, + "prompt_tokens": 9473, + "reused_tokens": 6101, + "read_tokens": 3372, + "generated_tokens": 59, + "prompt_tps": 649.1, + "prompt_tps_source": "engine", + "decode_tps": 59.1, + "decode_tps_source": "engine", + "prompt_ms": 5195.2, + "decode_ms": 997.8, + "drafts_offered": 42, + "drafts_accepted": 24, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.166, + "total_s": 4.884, + "prompt_tokens": 12453, + "reused_tokens": 9531, + "read_tokens": 2922, + "generated_tokens": 51, + "prompt_tps": 702.7, + "prompt_tps_source": "engine", + "decode_tps": 71.0, + "decode_tps_source": "engine", + "prompt_ms": 4158.5, + "decode_ms": 718.3, + "drafts_offered": 32, + "drafts_accepted": 28, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.324, + "total_s": 5.346, + "prompt_tokens": 15392, + "reused_tokens": 12503, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 669.5, + "prompt_tps_source": "engine", + "decode_tps": 62.6, + "decode_tps_source": "engine", + "prompt_ms": 4315.4, + "decode_ms": 1022.3, + "drafts_offered": 49, + "drafts_accepted": 35, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 69.95, + "min": 69.88, + "max": 69.978, + "n": 3 + }, + "reread_tokens": { + "median": 5, + "min": 5, + "max": 5, + "n": 3 + }, + "new_tokens": { + "median": 14334, + "min": 14334, + "max": 14334, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.065, + "min": 1.985, + "max": 5.202, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15373, + "min": 15314, + "max": 15392, + "n": 3 + }, + "turn_decode_tps": { + "median": 63.1, + "min": 54.2, + "max": 74.5, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.278, + "min": 1.23, + "max": 1.309, + "n": 3 + } + }, + "clients": {}, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/client.log new file mode 100644 index 000000000..de37569d4 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/client.log @@ -0,0 +1,58 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-standard", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out,dst=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--out", "/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out/official"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 75.3 tok/s, first token 0.718 s, 100 tokens + run 1 q2: decode 62.1 tok/s, first token 0.676 s, 100 tokens + run 1 q3: decode 60.5 tok/s, first token 0.65 s, 100 tokens + run 2 q1: decode 80.4 tok/s, first token 0.655 s, 100 tokens + run 2 q2: decode 71.0 tok/s, first token 0.64 s, 100 tokens + run 2 q3: decode 67.3 tok/s, first token 0.65 s, 100 tokens + run 3 q1: decode 84.4 tok/s, first token 0.641 s, 100 tokens + run 3 q2: decode 74.5 tok/s, first token 0.651 s, 100 tokens + run 3 q3: decode 66.3 tok/s, first token 0.672 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4025 tokens, prompt 1161.2 tok/s (0 reused), first token 3.471 s + run 2: 4025 tokens, prompt 1158.8 tok/s (0 reused), first token 3.478 s + run 3: 4025 tokens, prompt 1159.7 tok/s (0 reused), first token 3.475 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1067.5 tok/s (0 reused), first token 15.057 s + run 2: 16058 tokens, prompt 1082.8 tok/s (0 reused), first token 14.844 s + run 3: 16058 tokens, prompt 1077.7 tok/s (0 reused), first token 14.915 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 835, reused 0, read 835, first token 1.278 s + run 1 turn 1: prompt 4403, reused 844, read 3559, first token 3.719 s + run 1 turn 2: prompt 6043, reused 4446, read 1597, first token 2.009 s + run 1 turn 3: prompt 9466, reused 6094, read 3372, first token 5.046 s + run 1 turn 4: prompt 12432, reused 9510, read 2922, first token 4.177 s + run 1 turn 5: prompt 15373, reused 12484, read 2889, first token 4.065 s + run 2 turn 0: prompt 835, reused 0, read 835, first token 1.23 s + run 2 turn 1: prompt 4403, reused 844, read 3559, first token 3.689 s + run 2 turn 2: prompt 6038, reused 4441, read 1597, first token 1.989 s + run 2 turn 3: prompt 9453, reused 6081, read 3372, first token 4.979 s + run 2 turn 4: prompt 12398, reused 9476, read 2922, first token 4.1 s + run 2 turn 5: prompt 15314, reused 12425, read 2889, first token 4.055 s + run 3 turn 0: prompt 835, reused 0, read 835, first token 1.309 s + run 3 turn 1: prompt 4403, reused 844, read 3559, first token 3.725 s + run 3 turn 2: prompt 6041, reused 4444, read 1597, first token 1.985 s + run 3 turn 3: prompt 9473, reused 6101, read 3372, first token 5.202 s + run 3 turn 4: prompt 12453, reused 9531, read 2922, first token 4.166 s + run 3 turn 5: prompt 15392, reused 12503, read 2889, first token 4.324 s + +Results (median, range over the runs): + chat decode tok/s 71.00 (60.50 - 84.40, n=9) + chat first token (s) 0.65 (0.64 - 0.72, n=9) + 4K prompt tok/s 1159.70 (1158.80 - 1161.20, n=3) + 4K first token (s) 3.48 (3.47 - 3.48, n=3) + 16K prompt tok/s 1077.70 (1067.50 - 1082.80, n=3) + 16K first token (s) 14.91 (14.84 - 15.06, n=3) + agent cache hit % 69.95 (69.88 - 69.98, n=3) + agent re-read tokens/session 5.00 (5.00 - 5.00, n=3) + agent turn first token (s) 4.07 (1.99 - 5.20, n=15) + agent decode tok/s 63.10 (54.20 - 74.50, n=15) + +Report: /opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out/official/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/settings.json new file mode 100644 index 000000000..95254ce2b --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/tiel-UD-Q3_K_XL/Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "--max-context", + "32768", + "--expert-profile", + "/results/tiel-ud-q3_k_xl-reserve1024-ctx32768/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/tiel-ud-q3_k_xl/tokenizer", + "model_name": "tiel-coder-35b-a3b-ud-q3_k_xl", + "display_name": "Tiel Coder 35B-A3B UD-Q3_K_XL", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17234935648, + "sha256": "46b8fbeba6e888c9a101e49cc717baa9bb8dfa0d203ec7b3185288159543ec14", + "filename": "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/stage.json new file mode 100644 index 000000000..2662c43e0 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx32768/official-standard/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 154.28853936999803 +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md new file mode 100644 index 000000000..aba53ebdb --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md @@ -0,0 +1,65 @@ +# Community benchmark on CPU only + +Measured on 2026-10-19 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run. + +## Hardware and software + +- GPU and VRAM: CPU only, VRAM not read +- CPU: AMD Ryzen 5 3600 6-Core Processor (? cores, 9 threads); installed RAM: 31.3 GiB; storage: not measured +- OS: Linux 7.0.14-24-pve; GPU driver: not read; CUDA: not read +- PolyStrata: project 0.1.45, commit not a git checkout; engine 0.1.45 +- Background workloads and power limits: not recorded + +## Model and configuration + +- Model and quantization: tiel-coder-35b-a3b-ud-q3_k_xl +- Context: 65536 tokens; parallel slots: 1 +- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0 + +```text +strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024 +``` + +## Method + +Run by `python tools/strata_bench.py --clients 2 --long` against the HTTP API of a PolyStrata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests. + +- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs +- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each +- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`) +- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests + +## Results + +| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range | +| --- | ---: | ---: | ---: | ---: | --- | --- | --- | +| short chat | 224 (221-229) | 0 | 100 (100-100) | 9 | not measured | 68.7 (60.9-82.6) | 0.73 (0.70-0.78) | +| 4K prompt, fresh | 4025 (4025-4025) | 0 (0-0) | 24 cap | 3 | 1136.2 (1073.0-1138.6) | not measured | 3.55 (3.54-3.76) | +| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 1088.3 (1087.1-1089.2) | not measured | 14.77 (14.76-14.79) | +| 32K prompt, fresh | 32049 (32049-32049) | 0 (0-0) | 24 cap | 3 | 838.7 (836.1-841.7) | not measured | 38.24 (38.10-38.36) | +| agent session, follow-up turns | 15360 (15339-15365) at the last turn | see below | up to 64 | 3 | see below | 63.2 (51.9-72.0) | 4.13 (2.01-8.21) | + +Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token. + +### Agent session (conversation cache) + +Cache hit over the follow-up turns: 69.9 % (69.9-69.9); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 14334 (14334-14334) new tokens of the files are read once); first turn (nothing cached) first token: 1.34 s (1.34-1.36). + +| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 | 835 | 0 | 835 | 625.4 | 1.339 | 63.7 | +| 1 | 4403 | 844 | 3559 | 954.0 | 3.738 | 66.7 | +| 2 | 6041 | 4444 | 1597 | 788.3 | 2.033 | 56.1 | +| 3 | 9455 | 6083 | 3372 | 411.0 | 8.215 | 58.8 | +| 4 | 12420 | 9498 | 2922 | 695.9 | 4.209 | 62.0 | +| 5 | 15339 | 12450 | 2889 | 705.7 | 4.104 | 63.7 | + +### Concurrency: 2 clients + +Total 49.0 tok/s (46.2-49.4); first token p50 2.022 s, p95 3.578 s; request latency p50 4.052 s, p95 5.459 s; errors 0. Server parallel slots: 1 (requests beyond that wait their turn). + +Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server). + +## Correctness and limitations + +Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details come from the server, `nvidia-smi`/`rocm-smi` and the OS; background load and power limits are not recorded. Total time of the suite: 291 s. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json new file mode 100644 index 000000000..add48cfc0 --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/results.json @@ -0,0 +1,1032 @@ +{ + "schema": 1, + "tool": "tools/strata_bench.py", + "date": "2026-10-19", + "params": { + "runs": 3, + "think": false, + "clients": 2, + "long": true, + "started_server": false, + "command": "python tools/strata_bench.py --clients 2 --long" + }, + "elapsed_s": 291.4453032016754, + "machine": { + "gpu": [], + "gpu_vram_mib": null, + "gpu_driver": null, + "cuda": null, + "cpu": "AMD Ryzen 5 3600 6-Core Processor", + "cpu_cores": null, + "cpu_threads": 9, + "ram_gib": 31.3, + "os": "Linux 7.0.14-24-pve", + "python": "3.13.5", + "strata_version": "0.1.45", + "strata_commit": null, + "engine_version": "0.1.45", + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "quant": "UD-Q3_K_XL", + "context": 65536, + "parallel_slots": 1, + "config": { + "exe": "strata-poly", + "args": [ + "--model", + "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "--max-context", + "65536", + "--expert-profile", + "expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "model_name": "tiel-coder-35b-a3b-ud-q3_k_xl", + "display_name": "Tiel Coder 35B-A3B UD-Q3_K_XL", + "lib_dirs": [ + "/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ], + "port": 8080, + "gpu": 0, + "gpus_asked": true, + "host": "127.0.0.1", + "open_browser": false, + "benchmark_mtp": 0 + }, + "launch": "strata-poly --model Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf --max-context 65536 --expert-profile expert-profile.txt --conversation-cache-mib 0 --kv-whole --threads 8 --vram-reserve-mib 1024", + "engine_info": { + "model": "tiel-coder-35b-a3b-ud-q3_k_xl", + "max_context": 65536, + "images": false, + "version": "0.1.45", + "context": 65536, + "expert_slots": 4782, + "expert_cache_mib": 6380, + "pool_workers": 8, + "conversation_cache_mib": 0, + "conversation_cache_slots": 4, + "conversation_cache_min_free_mib": 2560, + "vram_free_mib": 1023, + "vram_elastic": 1, + "batch_slots": 0, + "engine": "0.1.45" + }, + "vram_after": { + "elastic": true + } + }, + "results": { + "chat": { + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 0.719, + "total_s": 2.151, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 319.2, + "prompt_tps_source": "engine", + "decode_tps": 69.8, + "decode_tps_source": "engine", + "prompt_ms": 717.5, + "decode_ms": 1431.8, + "drafts_offered": 73, + "drafts_accepted": 63, + "run": 0, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.778, + "total_s": 2.408, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 285.0, + "prompt_tps_source": "engine", + "decode_tps": 61.3, + "decode_tps_source": "engine", + "prompt_ms": 775.5, + "decode_ms": 1630.8, + "drafts_offered": 55, + "drafts_accepted": 44, + "run": 0, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.73, + "total_s": 2.37, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 307.9, + "prompt_tps_source": "engine", + "decode_tps": 60.9, + "decode_tps_source": "engine", + "prompt_ms": 727.6, + "decode_ms": 1640.9, + "drafts_offered": 65, + "drafts_accepted": 46, + "run": 0, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.703, + "total_s": 1.913, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 326.7, + "prompt_tps_source": "engine", + "decode_tps": 82.6, + "decode_tps_source": "engine", + "prompt_ms": 700.9, + "decode_ms": 1210.3, + "drafts_offered": 73, + "drafts_accepted": 66, + "run": 1, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.732, + "total_s": 2.082, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 302.8, + "prompt_tps_source": "engine", + "decode_tps": 74.0, + "decode_tps_source": "engine", + "prompt_ms": 729.9, + "decode_ms": 1350.8, + "drafts_offered": 62, + "drafts_accepted": 50, + "run": 1, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.761, + "total_s": 2.344, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 295.3, + "prompt_tps_source": "engine", + "decode_tps": 63.1, + "decode_tps_source": "engine", + "prompt_ms": 758.6, + "decode_ms": 1584.0, + "drafts_offered": 74, + "drafts_accepted": 45, + "run": 1, + "prompt": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.719, + "total_s": 1.953, + "prompt_tokens": 229, + "reused_tokens": 0, + "read_tokens": 229, + "generated_tokens": 100, + "prompt_tps": 319.3, + "prompt_tps_source": "engine", + "decode_tps": 81.0, + "decode_tps_source": "engine", + "prompt_ms": 717.1, + "decode_ms": 1234.5, + "drafts_offered": 73, + "drafts_accepted": 65, + "run": 2, + "prompt": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.715, + "total_s": 2.177, + "prompt_tokens": 221, + "reused_tokens": 0, + "read_tokens": 221, + "generated_tokens": 100, + "prompt_tps": 309.8, + "prompt_tps_source": "engine", + "decode_tps": 68.4, + "decode_tps_source": "engine", + "prompt_ms": 713.4, + "decode_ms": 1462.2, + "drafts_offered": 64, + "drafts_accepted": 44, + "run": 2, + "prompt": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 0.781, + "total_s": 2.235, + "prompt_tokens": 224, + "reused_tokens": 0, + "read_tokens": 224, + "generated_tokens": 100, + "prompt_tps": 287.7, + "prompt_tps_source": "engine", + "decode_tps": 68.7, + "decode_tps_source": "engine", + "prompt_ms": 778.6, + "decode_ms": 1455.0, + "drafts_offered": 67, + "drafts_accepted": 50, + "run": 2, + "prompt": 2 + } + ], + "decode_tps": { + "median": 68.7, + "min": 60.9, + "max": 82.6, + "n": 9 + }, + "ttft_s": { + "median": 0.73, + "min": 0.703, + "max": 0.781, + "n": 9 + }, + "generated_tokens": { + "median": 100, + "min": 100, + "max": 100, + "n": 9 + }, + "prompt_tokens": { + "median": 224, + "min": 221, + "max": 229, + "n": 9 + } + }, + "prompts": { + "4K": { + "target_tokens": 4096, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 3.548, + "total_s": 3.92, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1136.2, + "prompt_tps_source": "engine", + "decode_tps": 64.5, + "decode_tps_source": "engine", + "prompt_ms": 3542.6, + "decode_ms": 372.1, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.54, + "total_s": 3.865, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1138.6, + "prompt_tps_source": "engine", + "decode_tps": 73.8, + "decode_tps_source": "engine", + "prompt_ms": 3535.1, + "decode_ms": 325.0, + "drafts_offered": 15, + "drafts_accepted": 12, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.756, + "total_s": 4.092, + "prompt_tokens": 4025, + "reused_tokens": 0, + "read_tokens": 4025, + "generated_tokens": 24, + "prompt_tps": 1073.0, + "prompt_tps_source": "engine", + "decode_tps": 71.4, + "decode_tps_source": "engine", + "prompt_ms": 3751.3, + "decode_ms": 336.1, + "drafts_offered": 14, + "drafts_accepted": 11, + "run": 2 + } + ], + "prompt_tps": { + "median": 1136.2, + "min": 1073.0, + "max": 1138.6, + "n": 3 + }, + "ttft_s": { + "median": 3.548, + "min": 3.54, + "max": 3.756, + "n": 3 + }, + "prompt_tokens": { + "median": 4025, + "min": 4025, + "max": 4025, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "16K": { + "target_tokens": 16384, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 14.787, + "total_s": 15.246, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1087.1, + "prompt_tps_source": "engine", + "decode_tps": 52.3, + "decode_tps_source": "engine", + "prompt_ms": 14771.6, + "decode_ms": 459.3, + "drafts_offered": 18, + "drafts_accepted": 10, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.758, + "total_s": 15.194, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1089.2, + "prompt_tps_source": "engine", + "decode_tps": 54.9, + "decode_tps_source": "engine", + "prompt_ms": 14742.5, + "decode_ms": 437.3, + "drafts_offered": 18, + "drafts_accepted": 10, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 14.771, + "total_s": 15.19, + "prompt_tokens": 16058, + "reused_tokens": 0, + "read_tokens": 16058, + "generated_tokens": 24, + "prompt_tps": 1088.3, + "prompt_tps_source": "engine", + "decode_tps": 57.1, + "decode_tps_source": "engine", + "prompt_ms": 14755.2, + "decode_ms": 420.4, + "drafts_offered": 14, + "drafts_accepted": 9, + "run": 2 + } + ], + "prompt_tps": { + "median": 1088.3, + "min": 1087.1, + "max": 1089.2, + "n": 3 + }, + "ttft_s": { + "median": 14.771, + "min": 14.758, + "max": 14.787, + "n": 3 + }, + "prompt_tokens": { + "median": 16058, + "min": 16058, + "max": 16058, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + }, + "32K": { + "target_tokens": 32768, + "requests": [ + { + "ok": true, + "err": null, + "ttft_s": 38.36, + "total_s": 38.88, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 836.1, + "prompt_tps_source": "engine", + "decode_tps": 46.0, + "decode_tps_source": "engine", + "prompt_ms": 38330.2, + "decode_ms": 521.4, + "drafts_offered": 16, + "drafts_accepted": 9, + "run": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 38.105, + "total_s": 38.6, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 841.7, + "prompt_tps_source": "engine", + "decode_tps": 48.4, + "decode_tps_source": "engine", + "prompt_ms": 38076.3, + "decode_ms": 496.3, + "drafts_offered": 16, + "drafts_accepted": 9, + "run": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 38.242, + "total_s": 38.741, + "prompt_tokens": 32049, + "reused_tokens": 0, + "read_tokens": 32049, + "generated_tokens": 24, + "prompt_tps": 838.7, + "prompt_tps_source": "engine", + "decode_tps": 48.0, + "decode_tps_source": "engine", + "prompt_ms": 38213.6, + "decode_ms": 499.6, + "drafts_offered": 16, + "drafts_accepted": 9, + "run": 2 + } + ], + "prompt_tps": { + "median": 838.7, + "min": 836.1, + "max": 841.7, + "n": 3 + }, + "ttft_s": { + "median": 38.242, + "min": 38.105, + "max": 38.36, + "n": 3 + }, + "prompt_tokens": { + "median": 32049, + "min": 32049, + "max": 32049, + "n": 3 + }, + "reused_tokens": { + "median": 0, + "min": 0, + "max": 0, + "n": 3 + } + } + }, + "agent": { + "sessions": [ + [ + { + "ok": true, + "err": null, + "ttft_s": 1.339, + "total_s": 1.496, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 625.4, + "prompt_tps_source": "engine", + "decode_tps": 63.7, + "decode_tps_source": "engine", + "prompt_ms": 1335.1, + "decode_ms": 157.1, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 0, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.738, + "total_s": 4.367, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 42, + "prompt_tps": 954.0, + "prompt_tps_source": "engine", + "decode_tps": 66.7, + "decode_tps_source": "engine", + "prompt_ms": 3730.6, + "decode_ms": 629.6, + "drafts_offered": 29, + "drafts_accepted": 22, + "run": 0, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.033, + "total_s": 2.799, + "prompt_tokens": 6041, + "reused_tokens": 4444, + "read_tokens": 1597, + "generated_tokens": 43, + "prompt_tps": 788.3, + "prompt_tps_source": "engine", + "decode_tps": 56.1, + "decode_tps_source": "engine", + "prompt_ms": 2026.0, + "decode_ms": 766.4, + "drafts_offered": 35, + "drafts_accepted": 18, + "run": 0, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 8.215, + "total_s": 8.963, + "prompt_tokens": 9455, + "reused_tokens": 6083, + "read_tokens": 3372, + "generated_tokens": 44, + "prompt_tps": 411.0, + "prompt_tps_source": "engine", + "decode_tps": 58.8, + "decode_tps_source": "engine", + "prompt_ms": 8205.2, + "decode_ms": 748.4, + "drafts_offered": 28, + "drafts_accepted": 18, + "run": 0, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.209, + "total_s": 4.708, + "prompt_tokens": 12420, + "reused_tokens": 9498, + "read_tokens": 2922, + "generated_tokens": 31, + "prompt_tps": 695.9, + "prompt_tps_source": "engine", + "decode_tps": 62.0, + "decode_tps_source": "engine", + "prompt_ms": 4198.6, + "decode_ms": 500.2, + "drafts_offered": 23, + "drafts_accepted": 15, + "run": 0, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.104, + "total_s": 5.108, + "prompt_tokens": 15339, + "reused_tokens": 12450, + "read_tokens": 2889, + "generated_tokens": 64, + "prompt_tps": 705.7, + "prompt_tps_source": "engine", + "decode_tps": 63.7, + "decode_tps_source": "engine", + "prompt_ms": 4093.8, + "decode_ms": 1004.4, + "drafts_offered": 45, + "drafts_accepted": 35, + "run": 0, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.342, + "total_s": 1.47, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 623.3, + "prompt_tps_source": "engine", + "decode_tps": 77.8, + "decode_tps_source": "engine", + "prompt_ms": 1339.7, + "decode_ms": 128.5, + "drafts_offered": 8, + "drafts_accepted": 5, + "run": 1, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.748, + "total_s": 4.303, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 40, + "prompt_tps": 950.9, + "prompt_tps_source": "engine", + "decode_tps": 72.0, + "decode_tps_source": "engine", + "prompt_ms": 3742.7, + "decode_ms": 555.5, + "drafts_offered": 29, + "drafts_accepted": 21, + "run": 1, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.014, + "total_s": 3.038, + "prompt_tokens": 6039, + "reused_tokens": 4442, + "read_tokens": 1597, + "generated_tokens": 58, + "prompt_tps": 795.1, + "prompt_tps_source": "engine", + "decode_tps": 56.6, + "decode_tps_source": "engine", + "prompt_ms": 2008.5, + "decode_ms": 1025.4, + "drafts_offered": 48, + "drafts_accepted": 24, + "run": 1, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.085, + "total_s": 5.771, + "prompt_tokens": 9468, + "reused_tokens": 6096, + "read_tokens": 3372, + "generated_tokens": 41, + "prompt_tps": 664.1, + "prompt_tps_source": "engine", + "decode_tps": 59.7, + "decode_tps_source": "engine", + "prompt_ms": 5077.9, + "decode_ms": 686.2, + "drafts_offered": 28, + "drafts_accepted": 17, + "run": 1, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.213, + "total_s": 4.878, + "prompt_tokens": 12430, + "reused_tokens": 9508, + "read_tokens": 2922, + "generated_tokens": 42, + "prompt_tps": 694.8, + "prompt_tps_source": "engine", + "decode_tps": 63.2, + "decode_tps_source": "engine", + "prompt_ms": 4205.4, + "decode_ms": 664.9, + "drafts_offered": 31, + "drafts_accepted": 21, + "run": 1, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.13, + "total_s": 5.112, + "prompt_tokens": 15360, + "reused_tokens": 12471, + "read_tokens": 2889, + "generated_tokens": 51, + "prompt_tps": 701.0, + "prompt_tps_source": "engine", + "decode_tps": 51.9, + "decode_tps_source": "engine", + "prompt_ms": 4121.1, + "decode_ms": 983.0, + "drafts_offered": 39, + "drafts_accepted": 19, + "run": 1, + "turn": 5 + } + ], + [ + { + "ok": true, + "err": null, + "ttft_s": 1.356, + "total_s": 1.478, + "prompt_tokens": 835, + "reused_tokens": 0, + "read_tokens": 835, + "generated_tokens": 10, + "prompt_tps": 617.0, + "prompt_tps_source": "engine", + "decode_tps": 81.4, + "decode_tps_source": "engine", + "prompt_ms": 1353.4, + "decode_ms": 122.9, + "drafts_offered": 7, + "drafts_accepted": 5, + "run": 2, + "turn": 0 + }, + { + "ok": true, + "err": null, + "ttft_s": 3.763, + "total_s": 4.335, + "prompt_tokens": 4403, + "reused_tokens": 844, + "read_tokens": 3559, + "generated_tokens": 39, + "prompt_tps": 947.0, + "prompt_tps_source": "engine", + "decode_tps": 68.1, + "decode_tps_source": "engine", + "prompt_ms": 3758.1, + "decode_ms": 572.5, + "drafts_offered": 26, + "drafts_accepted": 18, + "run": 2, + "turn": 1 + }, + { + "ok": true, + "err": null, + "ttft_s": 2.058, + "total_s": 2.898, + "prompt_tokens": 6038, + "reused_tokens": 4441, + "read_tokens": 1597, + "generated_tokens": 58, + "prompt_tps": 777.8, + "prompt_tps_source": "engine", + "decode_tps": 69.0, + "decode_tps_source": "engine", + "prompt_ms": 2053.2, + "decode_ms": 840.4, + "drafts_offered": 43, + "drafts_accepted": 30, + "run": 2, + "turn": 2 + }, + { + "ok": true, + "err": null, + "ttft_s": 5.125, + "total_s": 5.897, + "prompt_tokens": 9467, + "reused_tokens": 6095, + "read_tokens": 3372, + "generated_tokens": 47, + "prompt_tps": 658.9, + "prompt_tps_source": "engine", + "decode_tps": 60.9, + "decode_tps_source": "engine", + "prompt_ms": 5117.9, + "decode_ms": 772.0, + "drafts_offered": 34, + "drafts_accepted": 21, + "run": 2, + "turn": 3 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.277, + "total_s": 4.911, + "prompt_tokens": 12435, + "reused_tokens": 9513, + "read_tokens": 2922, + "generated_tokens": 42, + "prompt_tps": 684.5, + "prompt_tps_source": "engine", + "decode_tps": 66.2, + "decode_tps_source": "engine", + "prompt_ms": 4268.7, + "decode_ms": 634.8, + "drafts_offered": 32, + "drafts_accepted": 23, + "run": 2, + "turn": 4 + }, + { + "ok": true, + "err": null, + "ttft_s": 4.228, + "total_s": 4.934, + "prompt_tokens": 15365, + "reused_tokens": 12476, + "read_tokens": 2889, + "generated_tokens": 47, + "prompt_tps": 684.7, + "prompt_tps_source": "engine", + "decode_tps": 66.5, + "decode_tps_source": "engine", + "prompt_ms": 4219.3, + "decode_ms": 706.8, + "drafts_offered": 32, + "drafts_accepted": 25, + "run": 2, + "turn": 5 + } + ] + ], + "hit_pct": { + "median": 69.939, + "min": 69.913, + "max": 69.944, + "n": 3 + }, + "reread_tokens": { + "median": 5, + "min": 5, + "max": 5, + "n": 3 + }, + "new_tokens": { + "median": 14334, + "min": 14334, + "max": 14334, + "n": 3 + }, + "turn_ttft_s": { + "median": 4.13, + "min": 2.014, + "max": 8.215, + "n": 15 + }, + "last_turn_prompt_tokens": { + "median": 15360, + "min": 15339, + "max": 15365, + "n": 3 + }, + "turn_decode_tps": { + "median": 63.2, + "min": 51.9, + "max": 72.0, + "n": 15 + }, + "first_turn_ttft_s": { + "median": 1.342, + "min": 1.339, + "max": 1.356, + "n": 3 + } + }, + "clients": { + "clients": 2, + "rounds": [ + { + "round": 0, + "wall_s": 5.54, + "tokens": 256, + "total_tps": 46.24, + "first_s": [ + 0.7674229329859372, + 3.6395431970013306 + ], + "latency_s": [ + 2.9259920949989464, + 5.535748152993619 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 1, + "wall_s": 5.23, + "tokens": 256, + "total_tps": 48.97, + "first_s": [ + 0.7398774659959599, + 3.3939770360011607 + ], + "latency_s": [ + 2.664912618987728, + 5.227599042991642 + ], + "errors": [], + "error_count": 0 + }, + { + "round": 2, + "wall_s": 5.18, + "tokens": 256, + "total_tps": 49.44, + "first_s": [ + 0.7161726249905769, + 3.277270596008748 + ], + "latency_s": [ + 2.5517363430117257, + 5.177324885997223 + ], + "errors": [], + "error_count": 0 + } + ], + "total_tps": { + "median": 48.97, + "min": 46.24, + "max": 49.44, + "n": 3 + }, + "ttft_p50_s": 2.022, + "ttft_p95_s": 3.578, + "latency_p50_s": 4.052, + "latency_p95_s": 5.459, + "errors": 0 + }, + "skipped": {}, + "failures": [] + } +} \ No newline at end of file diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/client.log b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/client.log new file mode 100644 index 000000000..0a573750c --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/client.log @@ -0,0 +1,70 @@ + +COMMAND ["docker", "run", "--rm", "--name", "poly-client-official-long", "--network", "container:polystrata-host6-inference", "--read-only", "--user", "65534:65534", "--cap-drop", "ALL", "--security-opt", "no-new-privileges:true", "--env", "PYTHONDONTWRITEBYTECODE=1", "--env", "HOME=/tmp", "--pids-limit", "128", "--memory", "1536m", "--memory-swap", "1536m", "--cpus", "1", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m", "--mount", "type=bind,src=/opt/polystrata-host6/upstream,dst=/opt/polystrata-host6/upstream,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/data,dst=/opt/polystrata-host6/data,readonly", "--mount", "type=bind,src=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out,dst=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out", "--mount", "type=bind,src=/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/config.json,dst=/config.json,readonly", "sha256:3273611332bfcb24e8421ac4d9cf7822b3ef638795f87b91083706e5d62e4336", "/opt/polystrata-host6/upstream/tools/strata_bench.py", "--url", "http://127.0.0.1:8080", "--config", "/config.json", "--runs", "3", "--long", "--clients", "2", "--out", "/opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out/official-long"] +Using the PolyStrata server already running at http://127.0.0.1:8080. +Note: the server runs 1 request(s) at a time (--batch), so 2 clients will partly wait their turn. +Running the suite (about 3-10 minutes) ... +warming up (not measured) ... + calibrated: about 1.08 tokens per word +chat: 3 short questions x 3 runs, up to 100 new tokens + run 1 q1: decode 69.8 tok/s, first token 0.719 s, 100 tokens + run 1 q2: decode 61.3 tok/s, first token 0.778 s, 100 tokens + run 1 q3: decode 60.9 tok/s, first token 0.73 s, 100 tokens + run 2 q1: decode 82.6 tok/s, first token 0.703 s, 100 tokens + run 2 q2: decode 74.0 tok/s, first token 0.732 s, 100 tokens + run 2 q3: decode 63.1 tok/s, first token 0.761 s, 100 tokens + run 3 q1: decode 81.0 tok/s, first token 0.719 s, 100 tokens + run 3 q2: decode 68.4 tok/s, first token 0.715 s, 100 tokens + run 3 q3: decode 68.7 tok/s, first token 0.781 s, 100 tokens +4K prompt: 3 fresh prompts of about 4096 tokens, 24 new tokens + run 1: 4025 tokens, prompt 1136.2 tok/s (0 reused), first token 3.548 s + run 2: 4025 tokens, prompt 1138.6 tok/s (0 reused), first token 3.54 s + run 3: 4025 tokens, prompt 1073.0 tok/s (0 reused), first token 3.756 s +16K prompt: 3 fresh prompts of about 16384 tokens, 24 new tokens + run 1: 16058 tokens, prompt 1087.1 tok/s (0 reused), first token 14.787 s + run 2: 16058 tokens, prompt 1089.2 tok/s (0 reused), first token 14.758 s + run 3: 16058 tokens, prompt 1088.3 tok/s (0 reused), first token 14.771 s +agent: a session of 6 turns with four tools, x 3 runs + run 1 turn 0: prompt 835, reused 0, read 835, first token 1.339 s + run 1 turn 1: prompt 4403, reused 844, read 3559, first token 3.738 s + run 1 turn 2: prompt 6041, reused 4444, read 1597, first token 2.033 s + run 1 turn 3: prompt 9455, reused 6083, read 3372, first token 8.215 s + run 1 turn 4: prompt 12420, reused 9498, read 2922, first token 4.209 s + run 1 turn 5: prompt 15339, reused 12450, read 2889, first token 4.104 s + run 2 turn 0: prompt 835, reused 0, read 835, first token 1.342 s + run 2 turn 1: prompt 4403, reused 844, read 3559, first token 3.748 s + run 2 turn 2: prompt 6039, reused 4442, read 1597, first token 2.014 s + run 2 turn 3: prompt 9468, reused 6096, read 3372, first token 5.085 s + run 2 turn 4: prompt 12430, reused 9508, read 2922, first token 4.213 s + run 2 turn 5: prompt 15360, reused 12471, read 2889, first token 4.13 s + run 3 turn 0: prompt 835, reused 0, read 835, first token 1.356 s + run 3 turn 1: prompt 4403, reused 844, read 3559, first token 3.763 s + run 3 turn 2: prompt 6038, reused 4441, read 1597, first token 2.058 s + run 3 turn 3: prompt 9467, reused 6095, read 3372, first token 5.125 s + run 3 turn 4: prompt 12435, reused 9513, read 2922, first token 4.277 s + run 3 turn 5: prompt 15365, reused 12476, read 2889, first token 4.228 s +concurrency: 2 requests at once x 3 rounds, 128 new tokens each + round 1: 256 tokens in 5.5 s = 46.2 tok/s total + round 2: 256 tokens in 5.2 s = 49.0 tok/s total + round 3: 256 tokens in 5.2 s = 49.4 tok/s total +32K prompt: 3 fresh prompts of about 32768 tokens, 24 new tokens + run 1: 32049 tokens, prompt 836.1 tok/s (0 reused), first token 38.36 s + run 2: 32049 tokens, prompt 841.7 tok/s (0 reused), first token 38.105 s + run 3: 32049 tokens, prompt 838.7 tok/s (0 reused), first token 38.242 s + +Results (median, range over the runs): + chat decode tok/s 68.70 (60.90 - 82.60, n=9) + chat first token (s) 0.73 (0.70 - 0.78, n=9) + 4K prompt tok/s 1136.20 (1073.00 - 1138.60, n=3) + 4K first token (s) 3.55 (3.54 - 3.76, n=3) + 16K prompt tok/s 1088.30 (1087.10 - 1089.20, n=3) + 16K first token (s) 14.77 (14.76 - 14.79, n=3) + 32K prompt tok/s 838.70 (836.10 - 841.70, n=3) + 32K first token (s) 38.24 (38.10 - 38.36, n=3) + agent cache hit % 69.94 (69.91 - 69.94, n=3) + agent re-read tokens/session 5.00 (5.00 - 5.00, n=3) + agent turn first token (s) 4.13 (2.01 - 8.21, n=15) + agent decode tok/s 63.20 (51.90 - 72.00, n=15) + 2 clients total tok/s 48.97 (46.24 - 49.44, n=3) + +Report: /opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl/README.md +Share it: open a pull request that adds the folder /opt/polystrata-host6/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out/official-long/2026-10-19-community-cpu-tiel-coder-35b-a3b-ud-q3-k-xl (see docs/COMMUNITY_BENCHMARKS.md), or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; read it once before you share it. diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/settings.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/settings.json new file mode 100644 index 000000000..9bba1a58a --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/settings.json @@ -0,0 +1,48 @@ +{ + "source_commit": "26e4cb5df7996edfb5ff549a63b82d8581e1e658", + "config": { + "exe": "/upstream/engine-poly/strata-poly", + "args": [ + "--model", + "/data/models/tiel-UD-Q3_K_XL/Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf", + "--max-context", + "65536", + "--expert-profile", + "/results/tiel-ud-q3_k_xl-reserve1024-ctx65536/out/expert-profile.txt", + "--conversation-cache-mib", + "0", + "--kv-whole", + "--threads", + "8", + "--vram-reserve-mib", + "1024" + ], + "cwd": "/upstream", + "tokenizer": "/data/packs/tiel-ud-q3_k_xl/tokenizer", + "model_name": "tiel-coder-35b-a3b-ud-q3_k_xl", + "display_name": "Tiel Coder 35B-A3B UD-Q3_K_XL", + "reasoning_budget_tokens": 512, + "benchmark_mtp": 0, + "host": "127.0.0.1", + "port": 8080, + "lib_dirs": [ + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib", + "/upstream/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx103X_all/lib" + ] + }, + "engine_sha256": "36bd496dbeb88b9715a4d88480929981d421eb23ce23d8172a8cd358688c9a37", + "weights": [ + { + "bytes": 17234935648, + "sha256": "46b8fbeba6e888c9a101e49cc717baa9bb8dfa0d203ec7b3185288159543ec14", + "filename": "Tiel-Coder-35B-A3B-MTP-UD-Q3_K_XL.gguf" + } + ], + "weight_sha256": null, + "weight_filename": null, + "weight_bytes": null, + "historical_weight_sha256": null, + "byte_identical_to_historical_weights": null, + "draft_note": "benchmark_mtp=0 is an adapter sentinel, not evidence that engine speculation is disabled.", + "scope": "Official tools/strata_bench.py only; no custom quality or retrieval results." +} diff --git a/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/stage.json b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/stage.json new file mode 100644 index 000000000..7a0d962ea --- /dev/null +++ b/bench/results/2026-10-19-community-rx6700xt-polystrata/runs/tiel-ud-q3_k_xl-reserve1024-ctx65536/official-long/stage.json @@ -0,0 +1,4 @@ +{ + "status": "completed", + "wall_s": 291.6794993689982 +} \ No newline at end of file diff --git a/docs/COMMUNITY_BENCHMARKS.md b/docs/COMMUNITY_BENCHMARKS.md index f0cd59cdd..29f6f5ce7 100644 --- a/docs/COMMUNITY_BENCHMARKS.md +++ b/docs/COMMUNITY_BENCHMARKS.md @@ -38,6 +38,10 @@ To add yours, open an issue at [VecSzn/PolyStrata](https://github.com/VecSzn/Pol [What to record](#what-to-record) lists. `python polystrata.py benchmark --model --prompt-tokens 24000` gives the two speeds the README uses ([what it sends and prints](BACKENDS.md#benchmark)). +An upstream-suite report from an [RX 6700 XT 12 GB, Ryzen 5 3600 and 32 GB RAM](../bench/results/2026-10-19-community-rx6700xt-polystrata/README.md) +includes Qwen3.6, Ornith, Tiel Coder, Gemma 4, MiniCPM-V and LFM2.5 at 32K and 64K, +with per-request data, exact settings and a separate Qwen UD-Q4_K_M speed control. + ### Comparing with llama.cpp `bench/bench_vs_llama.py` is the script Strata used for