Skip to content

[Bug]: Load balancing reads the host's CPUs and memory inside a container, so a limited container always wins #28228

Closed

Description

@TylerJNewman

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Run t3 start (0.1.5) in a Linux container with cgroup v2 limits. Mine is a Railway service limited to 16 vCPU / 24 GB on a 48-CPU host.
  2. Save it as an environment on a desktop client and enable load balancing alongside a Mac.
  3. Saturate the container (8 parallel tsc runs) and start a new draft.

Expected behavior

The container reports its own limits (16 CPUs, 24 GB) and its own CPU use, so the picker sends new drafts elsewhere once the container is busy.

Actual behavior

HostResources reads os.cpus(), os.totalmem() and /proc/meminfo, which inside the container describe the host: 48 CPUs, 346 GB, and host-wide CPU use. The score (weight × CPUs × idle share × free-memory share) stays the highest while the container is throttled (51 s of CPU throttling during 8 parallel typechecks), so every draft keeps landing on it and overflow never happens.

Impact

Major degradation or frequent failure

Version or commit

0.1.5. apps/server/src/resourceTelemetry/HostResources.ts is unchanged on main at 15fe283.

Environment

Railway, Debian bookworm, Node 24.22.2, cgroup v2 with a private cgroup namespace (/proc/self/cgroup is 0::/). Desktop client on macOS.

Logs or stack traces

From inside the container:

os.cpus().length            48
os.availableParallelism()   16
process.constrainedMemory() 24000000000
os.totalmem()               346488958976
/sys/fs/cgroup/cpu.max      1600000 100000

Workaround

None through weights: a weight can offset the CPU count but not CPU use the server never sees.

I have a small patch ready: when /sys/fs/cgroup/cpu.max and memory.max hold limits, report CPUs as min(quota, host CPUs), CPU use from cpu.stat usage_usec over the sample, and memory from memory.max minus (memory.current − inactive_file); hosts without limits keep today's values. It has parsing tests, passes vp run check:changed, and on the container reports 16 CPUs and 23.4 of 24.0 GB available. Happy to open a PR if this is accepted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions