Skip to content

Raise committor capacity and use idle executors with bounded admission #1804

Closed

Description

@snawaz

AppliesTo : master / 42ac9f2

Goal

Improve committor utilization on the release branch in two focused PRs: raise the blocked-intent limit to 3,000, then allow bounded additional admission when execution slots are available.

Why

RCA shared by @GabrielePicco (rephrased):

Head-of-line blocking in intent_execution_engine.rs::next_scheduled_intent. At 1,000 blocked intents, can_receive() returns false, stopping admission even for unrelated accounts while 49 of 50 execution slots may be idle.

The cap was first reached on Oct 13 at 09:41. mbv_committor_intent_backlog_count grew from 0 to 12,054 between Oct 13 20:00 and Oct 14 07:00. At 06:19, intent 164147 had about 11,000 intents ahead of it.

Queued payloads already occupy memory upstream on this branch. Moving them into the scheduler primarily adds dependency-index overhead; unrestricted admission would still allow that metadata and its processing cost to grow without a scheduler bound.

Scope

PR1: raise the limit

Increase the blocked-intent limit from 1,000 to 3,000. Keep execution concurrency at 50.

PR2: bounded admission using the existing scheduler (approach 1)

Use a normal blocked-intent limit of 3,000 and an additional reserve of 3,000 while execution slots are available. The hard blocked-intent limit is 6,000.

The admission policy is:

blocked < 3,000 OR (execution capacity available AND blocked < 6,000)

This is a policy, not a sufficient implementation by itself. Observe actual semaphore capacity and wake admission when retries release execution permits; the number of tracked executor tasks includes retrying tasks and is not the available-slot count.

Reuse the existing per-account queues and duplicate checks. Keep dependency ordering across the channel, backlog, and scheduler. When completion makes multiple intents eligible, dispatch eligible work into available slots without requiring another completion for each successor. Bound processing per turn, avoid busy polling, and retain the 50-execution limit.

Preserve result delivery, retry behavior, and persistence/recovery. Do not drop accepted intents or clone account payloads merely to inspect or move them.

Why choose approach 1 for master?

The limit increase in PR1 is an immediate mitigation. For the scheduling improvement in PR2, we considered two approaches:

  • Approach 1 — bounded admission, selected: a bounded admission reserve reuses existing dependency bookkeeping and keeps the release-branch change focused. The unrestricted alternative of receiving whenever any slot is free is excluded.
  • Approach 2 — look-ahead, deferred: look ahead through pending intents to find independent work while earlier conflicting intents remain queued. This requires dependency tracking across those earlier intents. A fixed look-ahead window can still hide independent work beyond its boundary. Look-ahead without a fixed depth cutoff can use dependency indexes and a ready queue to avoid repeated full scans, but the larger refactor belongs on dev.

The 6,000 hard limit can still leave independent work waiting farther back. This PR improves utilization within a bounded reserve; it does not solve total backlog growth.

Deferred dev follow-up: #1805. That work starts much later, after both PRs here are implemented on master and dev is rebased onto those changes.

Acceptance criteria

  • Verify the normal and hard admission boundaries and independent work executing beyond the normal limit.
  • Preserve direct and transitive account ordering, with no lost or duplicate scheduling.
  • One completion can dispatch multiple newly eligible intents into free slots.
  • Retry permit release wakes admission; active execution never exceeds 50.
  • An entirely conflicting backlog does not cause busy polling or unbounded scheduler growth.
  • Measure scheduler cost and allocation behavior for both one hot account and workloads spanning many distinct accounts.

Activity

  1. self-assigned this
    on Oct 14, 2026
  2. added theissue type on Oct 14, 2026
  3. changed the title [-]Raise the committor limit to 5,000 and use idle executors with look-ahead[/-] [+]Raise committor capacity and use idle executors with bounded admission[/+] on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions