[Bug]: SSH command timeout leaves a configured proxy helper running #35763

Summary

An SSH command can finish with a timeout error while a local proxy helper it spawned continues running. A reproduction using real OpenSSH and the production command helper leaves a proxy alive when it ignores the initial termination signal; the otherwise identical control exits cleanly. Timed-out device checks should release their owned local SSH processes so retries do not accumulate helpers.

Before submitting

• I searched existing issues and did not find an exact duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Use the stated checkout with its pinned dependencies installed on a POSIX system with OpenSSH.
2. Run the script below, which creates two isolated SSH configurations and local ProxyCommand helpers: one accepts SIGTERM and one ignores it.
3. Both commands must report the intended timeout. The normal helper must be gone; the helper ignoring SIGTERM remains alive after the timeout result.

The final direct reproduction failed in two runs. An earlier attempt that relied on a PATH shim did not start the proxy and was inconclusive; the final script passes its isolated configuration explicitly through -F. The script checks process state with ps so a zombie PID does not count as a live helper, and forcibly stops only a PID written by its own fixture during cleanup.

Save the following as repro-proxy-cleanup.ts outside the checkout and run node repro-proxy-cleanup.ts /path/to/checkout. It needs the installed checkout dependencies, Node with native TypeScript support, Git, OpenSSH, and system POSIX ps, sh, and sleep. It creates isolated HOME, TMPDIR, and T3 state directories, opens no SSH network connection, removes its files and captured helper processes, and leaves the checkout unchanged. Exit 1 means the proxy leaked, 0 means both timeout cases clean up their helpers, and 2 means setup or a control failed.

import { createRequire } from 'node:module';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir, platform, release } from 'node:os';
import { resolve, join } from 'node:path';
import { spawnSync } from 'node:child_process';
const checkout = resolve(process.argv[2] ?? '.');
const requireFromRepo = createRequire(join(checkout, 'apps/server/package.json'));
const scratch = mkdtempSync(join(tmpdir(), 't3-ssh-cleanup-'));
const ownedPids = new Set<number>();
let status = 2;
try {
  console.log('checkout:', spawnSync('git', ['-C', checkout, 'rev-parse', 'HEAD'], {encoding:'utf8'}).stdout.trim());
  console.log('Node:', process.version);
  console.log(spawnSync('git', ['--version'], {encoding:'utf8'}).stdout.trim());
  console.log(spawnSync('/usr/bin/ssh', ['-V'], {encoding:'utf8'}).stderr.trim());
  console.log('system ps/sh/sleep:', platform(), release());
  for (const name of ['home', 'tmp', 'state']) mkdirSync(join(scratch, name));
  process.env.HOME = join(scratch, 'home');
  process.env.TMPDIR = join(scratch, 'tmp');
  process.env.T3CODE_HOME = join(scratch, 'state');
  // Use the installed OpenSSH with an explicit isolated config.
  const Effect = await import(requireFromRepo.resolve('effect/Effect'));
  const References = await import(requireFromRepo.resolve('effect/References'));
  const NodeServices = await import(requireFromRepo.resolve('@effect/platform-node/NodeServices'));
  const { runSshCommand } = await import(join(checkout, 'packages/ssh/src/command.ts'));
  const rows = [];
  for (const ignoreTerm of [false, true]) {
    const name = ignoreTerm ? 'ignores-TERM' : 'normal';
    const pidFile = join(scratch, name + '.pid');
    const proxy = join(scratch, name + '.sh');
    const config = join(scratch, name + '.config');
    writeFileSync(proxy, `#!/bin/sh\n${ignoreTerm ? "trap '' TERM\n" : ''}echo $$ > '${pidFile}'\nexec sleep 120\n`, {mode:0o700});
    writeFileSync(config, `Host fixture.invalid\n  HostName fixture.invalid\n  RequestTTY no\n  ProxyCommand ${proxy}\n`);
    const result = await Effect.runPromise(runSshCommand(
      {alias:'fixture.invalid', hostname:'fixture.invalid', username:null, port:null},
      {preHostArgs:['-F', config], remoteCommandArgs:['true'], timeoutMs:200},
    ).pipe(Effect.result, Effect.provideService(References.MinimumLogLevel, 'None'), Effect.provide(NodeServices.layer)));
    const pid = Number(readFileSync(pidFile, 'utf8').trim());
    if (!Number.isSafeInteger(pid) || pid <= 1) throw new Error('Invalid owned proxy PID');
    ownedPids.add(pid);
    const processState = spawnSync('/bin/ps', ['-p', String(pid), '-o', 'stat=,comm='], {encoding:'utf8'}).stdout.trim();
    const liveProxy = processState.length > 0 && !processState.startsWith('Z');
    rows.push({name, timedOut:result._tag === 'Failure' && result.failure.message === 'SSH command timed out after 200ms.', liveProxy});
    console.log(JSON.stringify(rows.at(-1)));
    if (liveProxy) process.kill(pid, 'SIGKILL');
    ownedPids.delete(pid);
  }
  status = !rows[0].timedOut || rows[0].liveProxy || !rows[1].timedOut ? 2 : rows[1].liveProxy ? 1 : 0;
} catch (error) { console.error(String(error)); }
finally {
  for (const pid of ownedPids) { try { process.kill(pid, 'SIGKILL'); } catch {} }
  rmSync(scratch, {recursive:true,force:true});
  console.log(status === 1 ? 'RESULT: defect reproduced' : status === 0 ? 'RESULT: expected behavior' : 'RESULT: inconclusive');
  process.exitCode = status;
}

Expected behavior

After a failed SSH operation finishes its timeout cleanup, local processes owned by that attempt should no longer remain active. This is the resource-ownership contract implied by the command's scoped subprocess in packages/ssh/src/command.ts#L272-L284 and scoped timeout in packages/ssh/src/command.ts#L366-L393. OpenSSH ssh_config(5), ProxyCommand establishes that a configured proxy is a local process started by SSH. Device bootstrap intentionally relies on this helper's deadline in apps/server/src/device/SshDeviceHost.ts#L75-L90. The expected result concerns cleanup after failure, not making an unavailable SSH endpoint succeed.

Actual behavior

The production runSshCommand returns SshCommandError with SSH command timed out after 200ms. while a real OpenSSH ProxyCommand process remains alive in a non-zombie state. The fixture's proxy writes its own PID, ignores SIGTERM, and runs sleep; it performs no network operation. An otherwise identical proxy without the ignored signal is gone after the same timeout. The helper is reachable from device-host Test connection and bootstrap through SshDeviceHost.probe and bootstrap; those supported calls use the same scoped SSH execution with a longer deadline. This reproduction invokes that shared command boundary directly with a short deadline rather than waiting for the device UI's 45-second probe budget.

Impact

Minor bug or occasional failure

Impact: P2 - Failed SSH attempts can leave local helpers consuming processes and their other resources after T3 has reported timeout. Retrying repeats the leak for the tested signal-resistant proxy configuration.

The observed scope is a POSIX OpenSSH ProxyCommand helper that ignores SIGTERM, under the pinned Effect subprocess implementation. The normal termination control works. No production cloud proxy or Windows runtime was tested, and there is no claim that every proxy leaks. The reproducer's verified cleanup is to send SIGKILL to the captured fixture PID; using a helper that accepts termination also avoids the leak in the control. Resource exhaustion from repeated real UI checks was not exercised.

Version or commit

main @ 0bb6eaa546c91915efbd3f092088c694f3949709 (the inspected checkout); the latest published release was v0.1.9, but the packaged release was not run.

Environment

Node v26.12.1; Git 2.56.0; OpenSSH_10.5p1 with LibreSSL 3.3.6; Darwin 27.2.0, arm64. Dependencies are the checkout's lockfile versions, including Effect 4.1.3, @effect/platform-node 4.1.2, and @effect/platform-node-shared 4.1.2.

Interface: UI/API: SSH device-host Test connection/bootstrap; the shared runSshCommand boundary reproduced directly.

Earliest failure stage: Cleanup or lifecycle.

Logs or stack traces

Evidence provenance: observed (direct module reproductions), with static source tracing for the UI/RPC call paths. Local verification: reproduced. Reproduction completeness: complete at the tested service boundary. Missing fact: packaged-release and end-to-end GUI behavior were not run; other platforms were not tested.

Expected source: the cited existing T3 behavior contract. Failure source: the exact script and output below, with the cited production call paths.

Observed output from the final isolated script, with no personal data:

checkout: 0bb6eaa546c91915efbd3f092088c694f3949709
Node: v26.12.1
git version 2.56.0
OpenSSH_10.5p1, LibreSSL 3.3.6
system ps/sh/sleep: darwin 27.2.0
{"name":"normal","timedOut":true,"liveProxy":false}
{"name":"ignores-TERM","timedOut":true,"liveProxy":true}
RESULT: defect reproduced

The shared helper creates its owned subprocess without a forced-termination deadline in packages/ssh/src/command.ts#L272-L284, then reports the timeout after scope cleanup in packages/ssh/src/command.ts#L366-L393. The pinned subprocess library's NodeChildProcessSpawner.ts states that without forceKillAfter, cleanup signals the group, waits for the leader and gives remaining members a grace period without escalation. The observed surviving process is a running sleep, not a zombie. Device bootstrap calls this helper in apps/server/src/device/SshDeviceHost.ts#L75-L90 and the supported probe entry point reaches it through apps/server/src/device/DeviceService.ts#L1390-L1396.

Restoration check

The normal and SIGTERM-resistant proxy rows must both report timedOut:true and liveProxy:false. A timeout result must not require the user to find and terminate leftover helpers manually.

Workaround

A proxy helper that accepts SIGTERM is cleaned up in the control. For the deliberately signal-resistant fixture, sending SIGKILL to the helper PID captured by the fixture stops it; the script performs that cleanup. No production proxy-specific workaround was tested.

References

• [Bug]: A child that ignores SIGTERM never lets ProcessRunner's timeout return, leaking a VcsProcess permit; after 8 leaks every git call and every new turn hangs forever #20419 — concerned a different subprocess runner whose timeout never returned; this report covers the shared SSH helper returning a timeout while its proxy descendant survives.
• Server self-update leaks the old server's SSH device-host tunnel #17438 — concerned SSH device tunnels surviving server self-update; this reproduction needs neither an update nor a long-lived tunnel and exercises command timeout cleanup.
• fix(server): keep helper processes from outliving the backend #12513 — fixes forced termination in ProcessRunner and text-generation helpers; its changed files do not include packages/ssh/src/command.ts.