SwarmWorld experiment reproduced with Muse Glimmer 30B and Ornith 1.5 #3

Hi, and thanks for releasing SwarmWorld with traces and the analysis pipeline. That made this easy to do.

We ran the technology_ecology study with two open-weight models, using the same protocol as the Qwen3.8-27B replication (#1, #2 here, and apexfusion/swarmworld-replication-data):

• Muse Glimmer 30B (Meta, Apache-2.0);
• Ornith 1.5 35B-A3B (ornith-ai, MIT; a mixture-of-experts model with ~3B active parameters).

The protocol: N=50; conditions full and independent-search; seeds 3201-3204; 800 ticks; held-out seeds 9201-9208. Each model ran two arms that differ in one system-prompt paragraph. One result looks relevant to the paper's central comparison, so here it is with the data.

Setup and deviations

• Engine: the repo at 6af7ae9, plus PR Local-model serving: optional chat-completions transport and constrained ID fields #2 (chat-completions transport, strict_id_patterns). The first arm of each model runs the authors' prompt unchanged.

• Hardware and serving: one RTX 5090, llama.cpp (a fork build, gs-1003), Q4 GGUF quants, 8 slots × 32K context, 8 cells in parallel. Each arm took 5.5-12 h.

• Glimmer:

  • Reasoning at "low", since Glimmer has no thinking-off switch.
  • The schema is enforced through a forced tool call validated with jsonschema, via a small proxy (glimmer_adapter.py, in the dataset). With llama.cpp, /v1/responses silently ignores text.format (fixed by server: fix /v1/responses text.format not translated to response_format ggml-org/llama.cpp#29346, now approved), and on our fork build the chat path ignored the schema for Glimmer's template.
  • All 12,800 plan calls were valid on the first attempt.

• Ornith: the replication's own path. PR Local-model serving: optional chat-completions transport and constrained ID fields #2's chat-completions transport straight to the server, thinking off, the schema enforced by the server's grammar, no proxy.

• The second arm, "fill-the-plan" (our addition, behind an env switch, off by default), adds one paragraph to the system prompt:

  Actions cost nothing in this world, and your queue is refilled only at your next macroturn: every unused plan slot is an idle tick. Queue the full 12 actions every macroturn.

  Everything else is identical: the same server and the same seeds.

Why the second arm

In this profile, actions have no cost and there's no action budget, so plan length is purely the model's choice. Mean actions per plan, across all macroturns:

With the authors' prompt, both models leave about 70% of every 12-slot plan empty, so most of their agent-ticks are WAITs. We wanted to know how much of the model comparison, and of the society-vs-solo comparison, is just activity.

Results (society arms, 4 seeds summed, at tick 800)

• Glimmer: the prompt raised milestones (31 → 43). Early in the run the extra actions went mostly to MOVE and INSPECT, and production came later.
• Ornith: the prompt nearly took it to the ceiling (38 → 61 milestones), with builds on every seed. With the authors' prompt it looked like a non-builder.
• At comparable plan lengths, Qwen still fabricates and builds several times as much as either. So activity explains part of the model gap, but not most of it.

The society-vs-solo comparison (analyze-study, unmodified)

cultural-gain.csv: the society minus the best-of-50 independent envelope, with bootstrap 95% CIs over 4 seed pairs:

Glimmer: every outcome gain moved against the society.

• The society's peak artifact performance barely moved (0.134 → 0.127), while the best-of-50 envelope nearly tripled (0.079 → 0.209; validated inventions 0.25 → 0.75).
• A maximum over 50 independent searchers rewards per-agent activity most directly. Here the solo agents also filled their plans more than the society members did (9.9 vs 8.7).

Ornith: the activity lifted both sides, and the verdict now depends on the endpoint.

• Peak artifact performance rose about 4x for the society (0.072 → 0.307) and about 5x for the envelope (0.073 → 0.401).
• The discovery-frontier gain flipped from clearly favouring the best solo agent to clearly favouring the society, and validated inventions now clearly favour the society. In both cases the CI excludes zero.
• Plan lengths are matched between society and solo here (11.2 vs 11.0).

In both models the society side collaborated far more with a full plan:

What we take from it (4 seeds per arm: directions, not results)

1. With zero action cost, how much each agent acts is a free variable, set by the model (3.2-3.6 vs Qwen's 7.8 actions per plan) and by the prompt.
2. In both models, changing only per-agent activity moved the society-vs-best-of-50 comparison. The direction depends on the model and the endpoint. For a given model, the verdict may partly reflect its default plan length.
3. So it might be worth reporting mean plan length (or WAIT share) next to the cultural gain, or running a fixed-activity control (e.g. a minimum plan length, or a small action cost with a budget).
4. The society-only collaboration measures rose with activity in both models, including where the society's edge over the best solo agent shrank.

Serving notes for llama.cpp users

• With the default Responses transport, plans are unconstrained (llama.cpp#29346).
• Grammar enforcement on the chat path isn't quite enough. For Ornith, the engine's own validation still rejected 0.7% of plans in the first arm and 2.5% in the fill-the-plan arm. Another 0.7% of fill-the-plan calls ran out of the 4,096-token output budget with 12 actions. Such calls give the agent no new actions until its next planning turn.

We're happy to run other probes on this setup if useful.

Data

Both datasets are Apache-2.0, in the layout of lamm-mit/swarmworld-data and apexfusion/swarmworld-replication-data:

• MattiAstedrone/swarmworld-muse-glimmer;
• MattiAstedrone/swarmworld-ornith-1.5.

Each holds:

• all 16 cells (society traces, 50 solo member traces per seed, study summaries);
• the analyze-study outputs with figures;
• the run scripts and patches (and, for Glimmer, the adapter).

Every trace replays bit-exactly on the released engine: all 816, with every stored state snapshot verified.