[Bug]: Repository identity and VCS detection spawn git per project on every read, stalling shellSnapshot

[willsheldon]

Area

apps/server

Steps to reproduce

1. Add enough projects that projection_projects holds a few dozen active rows. Mine has 34.
2. Put the machine under real load, for example several concurrent agent sessions.
3. Reconnect a client, or start a turn, so getShellSnapshot and VCS refreshes run.
4. Watch RepositoryIdentityResolver.resolve and VcsDriverRegistry.detect in server.trace.ndjson.

Expected behavior

Resolving which repository a project belongs to is cached. A workspace root maps to the same git top level for the life of the checkout, so a snapshot read should not spawn git once per project every time it runs.

Actual behavior

Two independent caching gaps make snapshot reads spawn one or more git processes per project on every call.

1. RepositoryIdentityResolver.resolve runs an uncached git rev-parse.

resolve(cwd) has two steps. Step two, git remote -v, is cached behind repositoryIdentityCache. Step one derives that cache's key by running git rev-parse --show-toplevel, and is not cached at all:

https://github.com/pingdotgg/t3code/blob/31c1c5996/apps/server/src/project/RepositoryIdentityResolver.ts#L160-L166

https://github.com/pingdotgg/t3code/blob/31c1c5996/apps/server/src/project/RepositoryIdentityResolver.ts#L90-L113

ProjectionSnapshotQuery.resolveRepositoryIdentitiesForProjects resolves every active project on every call, at concurrency 4:

https://github.com/pingdotgg/t3code/blob/31c1c5996/apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.ts#L359-L375

So the cache never prevents a spawn. 34 projects means 34 git processes per snapshot read, however warm the cache is. DEFAULT_POSITIVE_CACHE_TTL is also Duration.minutes(1), so step two re-spawns for every root at least once a minute.

2. VcsDriverRegistry caches repository detection for 2 seconds.

https://github.com/pingdotgg/t3code/blob/31c1c5996/apps/server/src/vcs/VcsDriverRegistry.ts#L15

GitVcsDriver.detectRepository spawns three git processes per cwd: rev-parse --is-inside-work-tree, rev-parse --show-toplevel, and rev-parse --git-common-dir. Any VCS call more than two seconds after the previous one re-runs all three for the same checkout.

Two seconds is an outlier in this layer. Every neighbouring cache in GitVcsDriverCore.ts is measured in minutes:

REPOSITORY_PATHS_CACHE_TTL      = Duration.minutes(10)
LIST_REFS_SNAPSHOT_CACHE_TTL    = Duration.minutes(2)
STATUS_DEFAULT_BRANCH_CACHE_TTL = Duration.minutes(5)
STATUS_ORIGIN_EXISTS_CACHE_TTL  = Duration.minutes(5)

Individually a git spawn is cheap. In aggregate, with these all queueing on a loaded machine, each resolve was observed at 4 to 6.5 s and VcsDriverRegistry.detect at 4.8 to 5.9 s.

Impact

environment.orchestration.shellSnapshot took 32.9 s on my install, which fires the 15s slow-request toast. It is on the client bootstrap and reconnect path.

Trace breakdown of that request:

33755ms  http.server GET
32954ms  environment.orchestration.shellSnapshot
32499ms    ProjectionSnapshotQuery.resolveRepositoryIdentitiesForProjects
 6508ms      RepositoryIdentityResolver.resolve
 5841ms      RepositoryIdentityResolver.resolve
 5838ms      RepositoryIdentityResolver.resolve
 ...  34 of these, at concurrency 4
 4070ms        RepositoryIdentityResolver.resolveFromCacheKey
 4057ms          processRunner.runProcessCore

Worth stating plainly, because it is the obvious suspect and it is wrong: this is not SQL. In the same window, across 117,031 sql.execute spans against a 2.9 GB state.sqlite, p50 was 0.04 ms, p99 was 16 ms, and not one query exceeded 1 second.

The cost scales with the number of projects, so it gets worse as an install grows.

Version or commit

main at 31c1c59. Both constants are unchanged since well before that; also present in 0.0.36.

Environment

macOS, desktop app hosting its own server, client connected over 127.0.0.1. 34 active projects.

Logs or stack traces

The clearest signal is the span ratio. Before any fix, RepositoryIdentityResolver.resolve and its inner git rev-parse occur 1:1, and VcsDriverRegistry.detect and detectRepository do too, because neither result survives to the next call.

After caching both, on the same install:

n=63   RepositoryIdentityResolver.resolve
n=34   RepositoryIdentityResolver.resolveRepositoryRoot     <- one per distinct root, cold fill only
n=291  VcsDriverRegistry.detect
n=160  detectRepository

and shellSnapshot drops from 32,954 ms to 1,744 ms, which is itself the cold startup run.

Workaround

None from the client side. Reducing the number of projects reduces the spawn count proportionally.