[crazyaimachine]

Two small fixes for --batch N --batch-groups G on a layer split. Two concurrent requests ran at about half the speed of the 4-request case per request, sometimes worse.

1. A group window holds only the slots up to its last active one (generate.cpp)

Every pipelined group window ran all GS slots of its group. An idle slot ran a pad row through every layer of every stage. The server spreads requests over the groups first (slot_order). So with --batch 4 --batch-groups 2 and two requests, every window was half pad rows. Now PGroup::S is the group's last active slot + 1, and the window, its picks and batch_launch() cover only those slots.

This fix was in the #559 series (ca74f92 there, "pipelined group windows only up to the last active slot"). It did not come along into 0.1.39, where the non-pipelined path got the same idea.

2. pick_slot() spreads requests over the groups (serve/server.py)

When no slot held the prompt's start, pick_slot() took the least recently used free slot. Once every slot held an old conversation, two new requests could land in one group while the other group's stage sat idle. Now the rule is:

1. A held prefix of at least 512 tokens still wins, because reading a long conversation again costs more.
2. Otherwise the free slot in the group with the fewest busy slots is taken.
3. Then an empty slot.
4. Then the least recently used one.

Measured

Setup:

• 2× TITAN RTX (Turing, PCIe 3.0), UD-Q4_K_XL with the BF16 PLE table.
• --batch 4 --batch-groups 2 --trim-stage-weights, layer split 22, images on.
• Through the HTTP server, T = 0.7, 400 tokens per answer, total tok/s.

Concurrent	0.1.40.1	with these two commits
1	48–58	47–59
2	36–38 (each request ~19 tok/s)	52–82
4	118–122	117–123

Without --batch-groups (one group), 0.1.40.1 gave 50 / 68 / 94 on the same machine.

Not solved

Even with both fixes, a round of two requests now and then still runs at ~19 tok/s each, with the requests in slots 0 and 2 (different groups). In six back-to-back rounds the worst was 52 tok/s total, but it does happen. I have not found the cause yet. If you have an idea where to look (doorbells? the stage hand-off?), I am happy to test.

The two commits are on 0.1.40.1 and apply to current main. Written by my agent, measured on my machine.

[blange48]

This overlaps with #1179 (opened a few hours earlier, the replacement of #793 after the history rewrite), so here are notes to line the two up.

Commit 1 (group window up to the last active slot) is the same change as #1179's first commit (your 8119a7f / ca74f92 from #559). One of the two PRs can carry it.

Commit 2 (slot choice) solves the same problem with a different rule. #1179's pick_slot (from @rhgo1749's report on #793) works like this with pipelined groups (the engine reports batch_groups= on its INFO line):

1. it fills a free slot below a busy slot of the same group first (repairs [_ B] after a departure);
2. then it follows slot_order (which already spreads over groups).

It ignores held prefixes there, because a pipelined slot is never kept as a conversation cache (BDONE clears cached), so the 512-token prefix rule in your step 1 may never fire on a pipelined split. Worth checking on your side. serve/test_slot_alloc.py has 2,000 random arrivals/departures asserting no pick leaves a free slot below it in its group.

Measured on 4× RTX 5080, --batch 8 --batch-groups 4 --trim-stage-weights, IQ3_S, T=0.7, 300 tokens, total tok/s, same image without/with the allocator part: 1: 120.6 → 122.9, 2: 149.1 → 150.6, 3: 141.8 → 212.1, 4: 222.2 → 251.3, 8: 372 → 372.

On your "not solved" case (two requests in slots 0 and 2, still ~19 tok/s each): I have not reproduced it on 4 groups of 2. One thing to rule out is a stale [_ B] shape in one group left by an earlier round, which rule 1 above repairs and an LRU / fewest-busy rule does not. Logging the slot index of each admission plus the window sizes (PGroup::S) per round would show it. Happy to run your repro on our 4-GPU box if you post the client script.

[Niko1221]

Thank you, crazyaimachine. The batch-group fixes are in 0.1.40.2 (Co-authored-by credit). One change: the slot choice and the 512-token prefix threshold apply only with --batch-groups, so the default slot choice is exactly as before.