Repository navigation
Task reports phase=Running and Ready=True indefinitely after its command has exited #346
Description
Summary
A Task reports phase: Running with Ready: True and the message "Task is running and its workspace is ready" indefinitely after its command has exited. This holds for a clean non-zero exit, for a process that died on an unhandled exception, and for a task whose Substrate actor has moved to ACTOR_STATE_CRASHED.
TaskStatus carries no exit code and no condition describing the command, so there is no field that distinguishes "working" from "exited an hour ago".
AX v0.3.0 (d8ed0fe), Agent Substrate on a 2-node kind cluster.
Reproduction
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
name: exitprobe
atespace: default
spec:
image: "<your image>"
command: ["bash", "-lc", "echo I-AM-ABOUT-TO-DIE; exit 42"]
gateway: {name: some-gateway}At t+60s, ax describe task exitprobe:
Conditions:
TYPE STATUS REASON MESSAGE
GatewayReady True PoliciesApplied Network policies active
WorkspaceReady True SetupComplete Workspace setup completed at 10.244.1.65
Ready True TaskRunning Task is running and its workspace is ready
status.phase is Running. The exit code 42 appears nowhere.
Two further cases
A command that died on an exception. A python process exited with a traceback; six minutes later ax get task still reported Running/Ready=True. ps aux inside the sandbox showed only PID 1 /usr/local/bin/ax-task-runner and a defunct [git] — no application process at all.
An actor Substrate already knows has crashed. For another task, ax ssh refused with:
actor default/killtest unavailable: AssignWorker prerequisite not met for Actor:
default/killtest (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)
At the same moment ax get task killtest reported phase: Running, Ready: True. So Substrate had the correct state and the Task resource did not reflect it. That last case looks like the most clearly actionable one — the information already exists a layer down.
Why this is being reported rather than worked around
docs/concepts.md is explicit that AX deliberately does not model the agent's shape, and I think that boundary is right — we are not asking for AX to supervise the agent. The gap is narrower: the conditions table documents Ready as "the task is running and WorkspaceReady is True", and ax watch/ax describe are presented as the way to follow a task, but neither can answer whether the command is still alive. A long-running task that dies quietly is indistinguishable from one doing slow work, for as long as you care to wait.
Some options, in what I would guess is increasing order of cost:
- surface the actor state when it is
ACTOR_STATE_CRASHED, since Substrate already reports it - a terminal
phase(Succeeded/Failed) plus the exit code inTaskStatuswhen the command exits - a
CommandRunningcondition, leavingReadymeaning what it means now
Even the first would be enough to stop a dead task reading as a healthy one.
Happy to test a patch on this cluster.
Activity
Skip white space only text part. (google#346)
foma-agent commentedon Sep 21, 2026
docs/runner.mdon v0.3.0 already says the controller never startsspec.commandas PID 1. It always launches/usr/local/bin/ax-task-runner, the command arrives only throughAX_TASK_YAML, and "the control plane does not currently read the command's exit status back from the container." Stay-up-after-exit is required so/readyzandax sshkeep working after the child dies.Your
ACTOR_STATE_CRASHEDvsphase: Runningcase is the extra one: Substrate already has a terminal actor state, and the Task resource still prints Ready. I read the runner page atd8ed0fe; I did not run ax.arbazkhan971 commentedon Sep 22, 2026
Fix ready: implemented on my fork (branch
fix/346-task-phase-after-command-exit, regression test included,make testgreen) and ready to PR — but this repo'scollaborators_onlyPR creation policy rejects PRs from non-collaborators (see #358), so I cannot open it. If PR creation opens for external contributors, I will open the PR immediately. Happy to adjust scope per maintainer feedback.