flm serve: /v1/completions and other routes run outside the NPU queue and race with model swaps #759

Everything below comes from reading the source on main at 39ff855. None of it has been measured.

What looks wrong

requires_npu_access() decides which requests go through the single NPU queue (server.cpp:160-170). It lists /api/generate, /api/chat, /v1/chat/completions, /v1/audio/transcriptions and /v1/embeddings. Every other route runs directly on whichever of the 10 I/O threads accepted it (main.cpp:624), concurrently with whatever holds the NPU. Three of those routes touch state that the queued routes change:

1. POST /v1/completions is missing from the list, but it does inference. handle_openai_completion calls ensure_model_loaded(), insert() and generate() on the shared auto_chat_engine (rest_handler.cpp:1423-1490). A /v1/completions request that arrives while a /v1/chat/completions request is decoding would therefore run on the same engine and the same NPU at the same time. If it names a different model, it would reset the engine that the other request is using (:385-388).

2. POST /api/embeddings is also missing, while its twin /v1/embeddings is listed. Both call the same handle_embeddings, which runs the embedding model on the NPU.

3. GET /api/ps reads current_model_tag (:1017-1022). Meanwhile, ensure_model_loaded assigns that std::string on a queued request's thread (:411, :418). A concurrent read of a std::string during assignment is a data race, and undefined behaviour in C++.

   We have measured one consequence on our patched build. The tag stays at the model-faker sentinel until a first load finishes. Polling /api/ps while that load ran hit the sentinel lookup (a separate bug: undefined behaviour in rectify_model_tag) on 19 of 23 polls.

Expected

Anything that uses or swaps auto_chat_engine or the embedding engine goes through the NPU queue. For a read-only route like /api/ps, a snapshot of (engine loaded, tag) published under a mutex, or atomically, would be enough.

Suggested fix

• Add /v1/completions and /api/embeddings to requires_npu_access().
• Give /api/ps a synchronised view of the loaded tag.

We have not written a patch for this yet. We found it while fixing /api/ps downstream. That fix (https://github.com/noamsto/nix-amd-ai/blob/46d7a73c1044435cec0ff08410e1a3bc87cbc544/pkgs/fastflowlm/patches/ps-loaded-models.patch) avoids the sentinel lookup, but it does not add synchronisation.

The heap-corruption reports in #608 involve concurrent requests. We haven't connected them to this, so this is only a pointer.

Environment

Found by reading the source; checked against main at 39ff855 and v1.0.6.