Skip to content

Local-model serving: optional chat-completions transport and constrained ID fields - #2

Open
const-apex wants to merge 1 commit into
lamm-mit:mainfrom
const-apex:local-model-serving
Open

const-apex wants to merge 1 commit into
lamm-mit:mainfrom
const-apex:local-model-serving

Conversation

@const-apex

Copy link
Copy Markdown

Summary

This adds two opt-in configuration options for running SwarmWorld against open-weight servers (vLLM, llama.cpp) and one diagnostic improvement.

All defaults reproduce current behaviour; the paper profiles are unaffected, and the prompt and action-schema hashes recorded in trace headers are unchanged unless a flag is set.

Changes:

llm.transport: chat_completions

(default: responses).

llama-server's /v1/responses converts requests to chat completions internally and drops text.format, so JSON comes back unconstrained (18 of 33 plans failed local validation in our test).

With this option:

  • the provider posts to {base_url}/chat/completions with response_format: {type: json_schema, json_schema: {name, schema, strict}}, the same schema object the Responses path sends.
  • max_tokens, temperature and reasoning_effort map one-to-one;
  • usage keys are mapped back to input_tokens / output_tokens / total_tokens so traces and the compute audit stay uniform;
  • finish_reason: length raises the same incomplete error;
  • a leading <think>...</think> block is stripped when a server returns reasoning inline.
  • llm.extra_body merges arbitrary request fields into either transport (used for chat_template_kwargs.enable_thinking: false with Qwen on vLLM).

llm.strict_id_patterns

(default false).

Small models copy the prompt's "use NONE when irrelevant" into reply_to and target_artifact_id (29 and 22 of 42 actions with Gemma 4 E4B), and the engine rejects every such action.

With the flag on:

  • the three free-string ID fields get a pattern in the schema (^(message_[0-9]+)?$, ^(agent_[0-9]+)?$, ^(artifact_[0-9]+)?$), so constrained decoding cannot emit placeholders;
  • the reproducibility hash includes the flag, so traces are self-labelled.
  • Requires a backend that honours pattern (vLLM: guidance; xgrammar rejects it).

Diagnostics

On finish_reason: length the error string now records the length, head and tail of the truncated reply, so runaways can be classified from the trace.

Tests

Tested:

  • ruff check src tests clean;
  • 154 tests pass in the engine-only environment (the 25 deselected ones need the analysis extras and are untouched), including the 7 new ones;
  • with the flags off, the action schema is byte-identical to the unpatched module (same SHA-256), and prompt_sha256 in trace headers equals the released traces' value.
  • In use: ~50,000 model calls across Gemma 4 E4B, Qwen3-235B (llama.cpp RPC) and Qwen3.8-27B (vLLM, three quantisations), including a complete N=50 row of the paper's protocol.

Related: #1 (late state-digest divergence when replaying the released traces on revision 9); this PR does not touch replay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant