[Bug]: Opening a thread freezes the desktop renderer for ~1 min when there are ~13k threads: environment-quiet-thread-shell-list rebuilds ~14k times over every thread #25823

Activity

1. reed-yang commented on Oct 28, 2026

   

   reed-yang

   on Oct 28, 2026

   More actions

   Independent reproduction on v0.1.3 (d2c440312e5f39d59827ed466838a623516454d6), plus before/after validation of #25853. A single snapshot update can reproduce a long synchronous stall; thousands of WebSocket frames are not required.

   Reproduction and measurements

   The attached script imports the real createEnvironmentThreadShellAtoms, the existing synthetic fixtures, PrimaryConnectionTarget, and the real shell reducer. It mounts the quiet list as a subscriber, then updates synthetic thread titles. It does not use a server, provider, credentials, or private conversations. Timings cover registry.set and synchronous subscriber delivery, excluding fixture construction and initial mount.

   macOS 26.7, Apple Silicon, Node 26.0.0, Effect 4.0.5, release lockfile. All rows change in the bulk cases; the five-row case changes only five existing objects in a single published snapshot. These are state-layer timings, not whole-app frame-rate measurements.

   Threads / changes in one snapshot	v0.1.3	v0.1.3 + #25853	Subscriber deliveries before → after
   4,000 / all	3,667 ms	11.3–12.6 ms	4,000 → 1
   12,000 / all	47,070 ms	17.8–30.1 ms	12,000 → 1
   26,138 / five	106.8–118.2 ms	3.5–10.2 ms	5 → 1
   26,138 / all	not run on the old quiet list	28.8–38.1 ms	— → 1

   The patched ranges are three consecutive updates in a fresh process. The 12k baseline ran with Node CPU profiling enabled; the other timing rows did not. These are a few local measurements, not a statistical benchmark. As a control, the existing non-quiet threadShellsAtom handles the 26,138/all case in approximately 33 ms both before and after.

   The synthetic 12k CPU profile also matches this issue's reported hot path: about 37.9% self time in MutableHashMap lookup, 25.4% in readTracked, and 18.7% in Atom.family lookup. The quiet list at threadShell.ts:394 is above these calls. A 4k profiled run reproduced the same stall in 3,526 ms.

   Why this produces a freeze

   Each changed per-thread atom invalidates the aggregate quiet list while the shared snapshot is still propagating. Each rebuild walks all thread references. One snapshot changing K threads therefore does roughly K × N list work and publishes intermediate, partially updated lists. For a full replacement this becomes quadratic. Because registry.set completes this synchronously, the renderer cannot process input during that work.

   #25853 fixes this by building the quiet list directly from each environment's thread array and keeping the previous quiet shell per thread, instead of subscribing the aggregate to every per-thread atom. It preserves the streaming hold behavior while publishing the completed list once. This is validation of the existing PR, not a competing implementation.

   Regression validation

   Applied only #25853's test first: delivers one quiet shell list per snapshot update that changes several threads fails on v0.1.3, receiving three lists for one three-thread update, including two partially updated states. After applying its production change:

   cd packages/client-runtime
   node_modules/.bin/vp test run src/state/threadShell.test.ts src/state/shellReducer.test.ts
   # 2 files passed, 24 tests passed
   node_modules/.bin/vp run typecheck
   # passed

   At inspected main commit 0894156d9a38eeff8abfd9c266f3f57b8e524217, the aggregate still loops over threadRefsAtom and reads quietThreadShellAtomFamily per thread. I did not run the full benchmark or typecheck on that newer main.

   Runnable synthetic case

   In a disposable checkout, with the script below saved outside it as repro-quiet-shell.mjs:

   git clone --filter=blob:none --single-branch --branch v0.1.3 https://github.com/pingdotgg/t3code.git t3-quiet-repro
   cd t3-quiet-repro
   npx --yes pnpm@12.14.0 install --filter @t3tools/client-runtime... --filter @t3tools/monorepo --frozen-lockfile --ignore-scripts
   node /path/to/repro-quiet-shell.mjs "$PWD" 4000 all 1
   node /path/to/repro-quiet-shell.mjs "$PWD" 12000 all 1
   node /path/to/repro-quiet-shell.mjs "$PWD" 26138 5 3
   # Control:
   node /path/to/repro-quiet-shell.mjs "$PWD" 26138 all 1 normal

   Apply #25853's two-file diff (tested PR head da9db8b08b16130745600e1aa682bcdb24850884) to this disposable checkout and repeat. No running T3 install needs modification.

   Complete reproduction script

   import assert from "node:assert/strict";
   import { createRequire } from "node:module";
   import { resolve } from "node:path";
   import { pathToFileURL } from "node:url";
   
   const [sourceRoot, countArg = "1000", changedArg = "5", framesArg = "3", listKind = "quiet"] =
     process.argv.slice(2);
   assert.ok(sourceRoot, "Pass the T3 source checkout path first");
   const count = Number(countArg);
   const changed = changedArg === "all" ? count : Number(changedArg);
   const frames = Number(framesArg);
   assert.ok(count > 0 && changed > 0 && changed <= count && frames > 0);
   const packageRoot = resolve(sourceRoot, "packages/client-runtime");
   const require = createRequire(resolve(packageRoot, "package.json"));
   const fromPackage = (name) => import(pathToFileURL(require.resolve(name)).href);
   const fromSource = (name) => import(pathToFileURL(resolve(packageRoot, "src", name)).href);
   const [{ Atom, AtomRegistry }, Option, contracts, fixtures, shells, connections, reducer] =
     await Promise.all([
       fromPackage("effect/reactivity"),
       fromPackage("effect/Option"),
       import(pathToFileURL(resolve(sourceRoot, "packages/contracts/src/index.ts")).href),
       fromSource("state/orchestrationV2TestFixtures.ts"),
       fromSource("state/threadShell.ts"),
       fromSource("connection/model.ts"),
       fromSource("state/shellReducer.ts"),
     ]);
   const { EnvironmentId, ThreadId } = contracts;
   const environmentId = EnvironmentId.make("synthetic-environment");
   let snapshot = {
     ...fixtures.v2ShellSnapshot,
     threads: Array.from({ length: count }, (_, index) => {
       const id = ThreadId.make(`synthetic-thread-${index}`);
       return {
         ...fixtures.v2ThreadShell,
         id,
         title: `Synthetic ${index}`,
         lineage: { ...fixtures.v2ThreadShell.lineage, rootThreadId: id },
       };
     }),
   };
   const snapshotAtom = Atom.make(snapshot);
   const catalogValueAtom = Atom.make({
     isReady: true,
     entries: new Map([
       [environmentId, {
         target: new connections.PrimaryConnectionTarget({
           environmentId,
           label: "Synthetic environment",
           httpBaseUrl: "https://example.test",
           wsBaseUrl: "wss://example.test",
         }),
         profile: Option.none(),
         enabled: true,
       }],
     ]),
   });
   const registry = AtomRegistry.make();
   const atoms = shells.createEnvironmentThreadShellAtoms({
     catalogValueAtom,
     snapshotAtom: () => snapshotAtom,
   });
   const listAtom = listKind === "quiet" ? atoms.quietThreadShellsAtom : atoms.threadShellsAtom;
   let deliveries = 0;
   let lastList;
   const emit = (value) => process.stdout.write(JSON.stringify(value) + "\n");
   emit({ phase: "start", node: process.version, count, changed, frames, listKind });
   const mountStart = performance.now();
   const unsubscribe = registry.subscribe(listAtom, (list) => {
     deliveries += 1;
     lastList = list;
   }, { immediate: true });
   emit({ phase: "mounted", ms: performance.now() - mountStart, deliveries });
   try {
     for (let frame = 1; frame <= frames; frame += 1) {
       if (changed <= 10) {
         for (let index = 0; index < changed; index += 1) {
           snapshot = reducer.applyShellStreamEvent(snapshot, {
             kind: "thread.updated",
             location: "active",
             sequence: snapshot.snapshotSequence + 1,
             thread: { ...snapshot.threads[index], title: `Changed ${frame} ${index}` },
           });
         }
       } else {
         snapshot = {
           ...snapshot,
           snapshotSequence: snapshot.snapshotSequence + 1,
           threads: snapshot.threads.map((thread, index) =>
             index < changed ? { ...thread, title: `Changed ${frame} ${index}` } : thread),
         };
       }
       deliveries = 0;
       emit({ phase: "updating", frame });
       const start = performance.now();
       registry.set(snapshotAtom, snapshot);
       const ms = performance.now() - start;
       assert.equal(lastList.length, count);
       assert.equal(lastList[changed - 1].title, `Changed ${frame} ${changed - 1}`);
       emit({ phase: "updated", frame, ms, deliveries, rss: process.memoryUsage().rss });
     }
   } finally {
     unsubscribe();
     registry.dispose();
   }

   Relation to the original live incident

   The motivating incident was a signed macOS 0.1.3 desktop whose renderer stayed near one CPU core while the main process and local/remote backend descriptors remained responsive. The window eventually recovered without a reload. Its remote database has about 26k thread rows, but the active client shell cardinality was not measured and shell filtering can reduce it. The live native sample did not resolve JavaScript function names. Thus this confirms the source defect and the candidate patch, but does not prove this path was the sole cause of that particular live freeze. No private logs or conversation content are attached, and a patched production desktop was not deployed.

   Prepared and verified with OpenAI Codex (GPT-6), through the Codex desktop harness.
2. reed-yang commented on Oct 28, 2026

   

   reed-yang

   on Oct 28, 2026

   More actions

   Follow-up: a second live macOS 0.1.3 freeze is now localized to this exact atom, rather than inferred only from the synthetic case.

   DevTools paused the busy main renderer at runtime-78TAXKYj.js:2:91914. Reading the matching asset from the installed signed release places that column at l(Z(n)) in:

   _=T(e=>{let t=[];for(let n of e(p)){let r=e(l(Z(n)));r!==null&&t.push(r)}return Ns(g,t)?g:(g=t,t)}).pipe(j(`environment-quiet-thread-shell-list`))

   The installed asset SHA256 is 4adda3fdf68d109ca74762de1e87c20bc0e3ae93df91920a09168f32e4cc58b9. The selected frame's partially built output array already held 9,505 entries; that is not the total list count. The surrounding stack repeatedly alternates build → value → propagate → invalidate → invalidateChildren → setValue; below it are setSelf, notification delivery, Effect resume and runLoop. Private reference IDs were omitted.

   The renderer was at about 99–100% CPU while main/backend remained responsive. A frontend Force Reload completed but immediately returned to the same high-CPU condition. Terminating only the verified main renderer let the existing official recovery handler recreate it; main and backend PIDs were unchanged, the new UI loaded, and renderer CPU was 2.6% after 26 seconds. This was temporary recovery, not deployment of #25853. No production bundle or database was modified.

   This live pause confirms that the quiet-shell aggregate is involved in the recurrent freeze. It does not yet quantify its exclusive share of CPU or exclude other contributing paths. The user confirmed that an earlier occurrence recovered by itself without reloading.
3. NoorChasib commented on Nov 2, 2026

   

   NoorChasib

   on Nov 2, 2026

   More actions

   Another reproduction on v0.1.6 (stable), at larger scale

   • Surface: macOS desktop app 0.1.6 (Electron 44) connected to a remote Linux x64 server 0.1.6 (boot service, Node 26.12.1), statev2.sqlite about 84 GB.
   • Thread: one long-running orchestration thread with 21,015 descendant subagent threads, 14,061 of them not archived (environment total: 38,049 threads, 16,790 active). It also has 13,106 messages, 40,646 turn items, 3,774 runs, 1,406 subagent rows and 155 linked PRs.
   • Repro: switch to that thread from any other thread. The whole desktop window freezes for about a minute, then recovers. It happens every time, and other threads are unaffected.
   • Server side is fast: ws.orchestrationV2.subscribeThread ≤1.4 s, ws.rpc.orchestration.getThreadFamily ≤5.9 s, ShellStream.loadShellSnapshotParts ≤6.5 s. The freeze is client-side, consistent with the environment-quiet-thread-shell-list analysis above.
   • Contributing factor: getThreadFamily returns every non-archived descendant with no limit (ProjectionStore.ts family query, v0.1.6). The client then merges them all into the environment thread list. Archive does not cascade from the parent, and settling doesn't remove children from the family read.
   • Code check: the quiet-list loop is unchanged on main as of 2026-11-02, so this still reproduces on current nightlies by code inspection. fix(client-runtime): opening a thread no longer freezes with thousands of threads #25853 is still open.

   Produced by Claude Code (Claude Opus 5.5) via t3 triage.