Repository navigation
[Feature]: run the image encoder on a second PC #1404
Description
Goal: read pictures at GPU speed without spending ~1.4 GB of VRAM on the image encoder.
What I use today: --vision cpu, because the troubleshooting table's other option costs the
VRAM. It works, but a 1024x1024 picture takes 15.3 s on this machine (Ryzen 7 9700X, RTX 5080
16 GB), measured the way a client sees it.
What I did: strata-vision is already a separate process behind a small line protocol, and it is
not tied to the server's machine — so I run it on a second PC (i7-11800H + RTX 3060 Laptop 12 GB)
over ssh. Same config, same pictures, only the encoder different; median of 5 runs each:
| encoder | one 1024x1024 picture | local VRAM | expert cache |
|---|---|---|---|
| this GPU | 1.63 s | 1383 MiB | 3112 experts, 5.96 GiB |
| another PC | 2.83 s | 0 | 3641 experts, 6.96 GiB |
| this CPU (8 threads) | 15.27 s | 0 | 3641 experts, 6.96 GiB |
That is 5.4x faster than the CPU encoder, and 1.00 GiB more expert cache than the GPU one (from the
engine's own startup line). It is 1.2 s a picture slower than a local GPU encoder, which is the price
of not spending the VRAM.
What it does not do: make text faster. The expert cache does grow (3112 to 3641 experts), but
long-context text is identical in the three modes — 3.27/3.30/3.31 K prompt tok/s at 257 K tokens and
3.03/3.05/3.04 K at 463 K, within run-to-run noise. ~1 GiB against a ~47 GiB expert set is ~2%
coverage. It is a picture-latency and VRAM trade, not a text-speed one.
I have it working behind an off-by-default flag, with tests and docs. Do you want it? Or is
--vision cpu the answer you would rather give people? If you do want it: config key
("remote": true) or --vision remote in setup?
(It needs two narrow server-side changes: skip the request FIFO when the encoder is remote — a local
encoder shares the GPU and must be serialised, a remote one shares nothing, and holding the FIFO
across a slow upload froze the whole API, text included — and start serving text when the encoder is
not there, so a vision host that is off does not cost the server. Detail and measurements are ready
for the PR.)
Activity
talisp commentedon Oct 7, 2026
Nice idea, and really clean measurements. Thanks for sharing!
It made me think of a variant that would be cool for anyone who runs Strata/Qwen on their own server (even if the "server" is just a desktop with a gaming GPU): do the encoding on the client machine. In my case the agent (Hermes) runs on a Windows workstation with an RTX A4000 16 GB, next to Unreal, and about 70% of that VRAM sits idle all the time. If the client encoded the picture there and sent only the result, the server would never spend VRAM on vision and would never need to call another machine. It would need the API to accept pre-computed image embeddings, and the encoder on the client would have to match the server's model.
For me it is not worth it, though: my server has a 48 GB GPU, the expert cache already hits ~97% here, and as your table shows, ~1.4 GB more does not make text faster. The encoded result is also several times bigger than the picture, which matters over an ssh tunnel. So vision stays on the server GPU for me.
Still a great idea for people with a 16 GB card on the server. And congrats on Strata, it's a great project! 🙂
TraceRecursion commentedon Oct 7, 2026
Thanks — and that client-side variant is the more interesting idea of the two, for exactly the reason you give: the VRAM that sits idle is usually on the machine the agent already runs on, not on a spare box.
The two differ in what has to change. Yours needs the API to accept embeddings, and a client encoder numerically identical to the server's. Mine needs only an ssh-reachable host with a resident encoder, because the existing
ENC <image> <out>protocol already passes paths, not data. Yours wins when the client is behind NAT, when there are many clients, or when you would rather not keep an encoder resident anywhere; mine wins when there is one dedicated encoder host and you would rather not touch the API.One piece of this work transfers to your variant:
make_vocab_stub.py. A client encoder does not need the 51 GB text model —strata-visionopens it withvocab_only = true, so an 11 MB vocabulary-only stub is enough. That removes the worst objection to running the encoder on a workstation.You are right about the transfer, and it is the honest limit of the design: the embeddings are 10.0 MB for 1024 tokens (1024 x 2560 x 4, F32) against ~0.1-0.4 MB for the pictures I used. On the 10 GbE link I measured that is ~150 ms, and base64 adds 36 ms on top — so it assumes a fast link, and would be poor over a slow WAN.
For your hardware I agree with your conclusion: a 48 GB card with a ~97% cache hit rate has nothing to gain, and my own numbers say the same (long-context text is identical in all three modes). This is for a machine like mine, where the server's card is 16 GB.
@Niko1221 — is either variant something you want? Or is
--vision cputhe answer you would rather give people? If you want the ssh one, I have it working behind an off-by-default flag with tests and docs, and I am glad to open a PR. If the client-side one sounds better, that is a different change, and worth deciding before anyone writes it.Niko1221 commentedon Oct 8, 2026
Clear measurement: 2.83 s on a second PC against 1.63 s on the local GPU and 15.3 s on the CPU, with 1 GiB more expert cache.
strata-visionis a separate process behind a line protocol, as you say, so running it over ssh works today. We have not built it into setup or the server (no setting for a remote encoder, no check of the link), and since the ENC lines carry paths, the image files must be reachable on the encoder's machine; so it is a recipe for now, not a feature. A short docs page would be welcome: the exact command, the config line, and what happens when the second PC is down. If you write it as a PR we will review it. talisp's variant, encoding on the client, needs an API that accepts image embeddings, which the OpenAI-compatible API does not have.TraceRecursion commentedon Oct 8, 2026
Thanks — a docs page it is. One thing first, because the answer to your third question is what decides the page's shape.
What happens when the second PC is down.
Vision(...)is built at startup with no try around it (serve/server.py:5695), and_start()raisesRuntimeError("the vision encoder did not start")(:1871). So with avisionblock configured and the host absent, the server does not come up at all. Withlazy_loadit comes up and the first image fails, and nothing retries — the host can come back and images stay broken until the server is restarted.And the encode holds the request FIFO (
:3233, and:3198for pictures a tool returned; the comment says why — a local encoder shares the GPU with the engine, so it takes its turn in the queue). For a remote encoder that is the wrong queue: I measured a remote encode on a starved host freezing text requests too, and one stalled encode left every later request queued behind it.Both are small, and both were in the patch I mentioned:
"remote": truein thevisionblock skips the FIFO — the wrapper serialises its own encodes, so taking them out of the queue loses nothing;Service.ensure_visionkeeps serving text when the encoder is absent and retries on the first image, answering400 the image encoder is not available: ...instead of dropping the connection.So: should the page document today's behaviour as it stands — the server will not start if the host is down, and an encode blocks text — or shall I write the recipe page and include those two changes so the recipe is usable? Either is fine with me; I would just rather not publish a page that recommends a configuration taking the API down with it.
The command and the config line are settled either way:
cfg["vision"]["exe"](serve/server.py:5692) already points the encoder at a wrapper, and the wrapper istools/remote-vision/from my branch.TraceRecursion commentedon Oct 11, 2026
PR is up: #1942.
It takes the third option from my last comment, for the reason I gave there: a page that documents today's
behaviour as it stands would recommend a configuration that takes the API down with it when the encoder's host is
absent.
Both server-side changes are gated on
"remote": true, so the default path is untouched - including the startuptolerance: a LOCAL encoder that fails to start is still fatal and still names the cause (#1445), because that is a
broken install and not an absent host. Only a remote one starts text-only and asks for the encoder again on the
first image (a
400carrying the reason, instead of a dropped connection).The page carries the measurements with the engine version and the date (0.1.40.2, 2026-10-07/08): the three modes
were measured the same day on one build, which is what the deltas need. I have not re-run the table on 0.1.42 - I
am glad to if you want it there.
I have also said in the PR body that I am happy to split the
serve/server.pybehaviour and its tests into theirown PR if that is easier to review; the docs page and
tools/remote-vision/need each other to be usable.TraceRecursion commentedon Oct 13, 2026
The PR is updated: #1942 is rebased on
a0116b27and no longer conflicts. The one conflict was with your #1999fix - the engine's turn is a cancellable
Service._turn(cancel)now, so the resolution keeps all of that and onlyskips the turn for a remote encoder (a local one still waits in it, and the encoder gets the client's cancel event
either way).
Since my last comment I also ran both behaviours on a real pair of machines (engine 0.1.43.1). The numbers are in
the PR body rather than repeated here: a text request sent during an encode that held the turn waited 13.05 s,
against 0.14-0.26 s with
"remote": true(baseline 0.13-0.15 s; the encode was a local CPU one, about 15 s, sothe window is unambiguous). With the encoder's host unreachable, the server starts and serves text, and the first
image answers
400with the reason instead of dropping the connection.The question from my last comment is still the open one - the page plus the two changes, or the page alone - and I
am still happy to split the
serve/server.pypart into its own PR if that is easier to review.Niko1221 commentedon Oct 13, 2026
Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.
If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.