Repository navigation
Conversation
andrej
force-pushed
the
fastflowlm-explicit-compile
branch
from
October 30, 2026 20:11
e1591cf to
4d9af30
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The registered embedding_gemma2_multimodal model fails to load through the server because its inherited checkpoint loader accesses a missing class-level configuration, causing an AttributeError.
1 open issue
What changed in this PR
This PR connects IRON models to FastFlowLM’s command-line interface through a binary-protocol server and an example integration pinned to a FastFlowLM fork.
Changes:
- Adds model registration and server commands for loading, logits, reset, and embeddings.
- Provides build instructions and tooling for FastFlowLM’s IRON backend.
- Adds protocol tests, an NPU2 end-to-end chat test, and a required container dependency.
| File | Description |
|---|---|
tests/applications/fastflowlm_iron/test.py |
Tests a Llama chat request through FastFlowLM on NPU2. |
iron/tests/test_lm_server.py |
Tests registration, protocol exchanges, capability negotiation, and diagnostic isolation. |
iron/lm_server.py |
Implements the binary-protocol model server. |
iron/applications/fastflowlm_iron/README.md |
Documents setup, usage, and integration-test weights. |
iron/applications/fastflowlm_iron/Makefile |
Builds the pinned FastFlowLM fork with IRON support. |
iron/applications/__init__.py |
Adds lazy model-class lookup through a registry. |
ci/docker-based/Dockerfile |
Adds the libedit-dev build dependency. |
Review effort: Balanced
Comment on lines
+13
to
+16
| "embedding_gemma2_multimodal": ( | ||
| "iron.applications.embedding_gemma2", | ||
| "Multimodal", | ||
| ), |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds some small infrastructure (
lm_server.py) as well as an example application (applications/fastflowlm_iron) that enable and demonstrate calling IRON model implementations from within the FastFlowLMflmcommand-line interface.This is implemented as a small server application (
lm_server.py) that spawns the model (jit compiled on first call) and handles request from theflmengine.Includes an end-to-end test that builds FastFlowLM with the IRON backend enabled and makes sure a simple request to Llama 3.2 1B produces tokens. Other models are already tested on their own, so this only tests integration with FastFlowLM based on one model.
Currently, for this, we are pinning a fork/branch of FastFlowLM that includes the IRON backend; once the backend gets merged upstream, we can update this.