Skip to content

Task reports phase=Running and Ready=True indefinitely after its command has exited #346

Open

Description

@naeemarsalan

Summary

A Task reports phase: Running with Ready: True and the message "Task is running and its workspace is ready" indefinitely after its command has exited. This holds for a clean non-zero exit, for a process that died on an unhandled exception, and for a task whose Substrate actor has moved to ACTOR_STATE_CRASHED.

TaskStatus carries no exit code and no condition describing the command, so there is no field that distinguishes "working" from "exited an hour ago".

AX v0.3.0 (d8ed0fe), Agent Substrate on a 2-node kind cluster.

Reproduction

apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: exitprobe
  atespace: default
spec:
  image: "<your image>"
  command: ["bash", "-lc", "echo I-AM-ABOUT-TO-DIE; exit 42"]
  gateway: {name: some-gateway}

At t+60s, ax describe task exitprobe:

Conditions:
  TYPE            STATUS  REASON           MESSAGE
  GatewayReady    True    PoliciesApplied  Network policies active
  WorkspaceReady  True    SetupComplete    Workspace setup completed at 10.244.1.65
  Ready           True    TaskRunning      Task is running and its workspace is ready

status.phase is Running. The exit code 42 appears nowhere.

Two further cases

A command that died on an exception. A python process exited with a traceback; six minutes later ax get task still reported Running/Ready=True. ps aux inside the sandbox showed only PID 1 /usr/local/bin/ax-task-runner and a defunct [git] — no application process at all.

An actor Substrate already knows has crashed. For another task, ax ssh refused with:

actor default/killtest unavailable: AssignWorker prerequisite not met for Actor:
default/killtest (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)

At the same moment ax get task killtest reported phase: Running, Ready: True. So Substrate had the correct state and the Task resource did not reflect it. That last case looks like the most clearly actionable one — the information already exists a layer down.

Why this is being reported rather than worked around

docs/concepts.md is explicit that AX deliberately does not model the agent's shape, and I think that boundary is right — we are not asking for AX to supervise the agent. The gap is narrower: the conditions table documents Ready as "the task is running and WorkspaceReady is True", and ax watch/ax describe are presented as the way to follow a task, but neither can answer whether the command is still alive. A long-running task that dies quietly is indistinguishable from one doing slow work, for as long as you care to wait.

Some options, in what I would guess is increasing order of cost:

  • surface the actor state when it is ACTOR_STATE_CRASHED, since Substrate already reports it
  • a terminal phase (Succeeded/Failed) plus the exit code in TaskStatus when the command exits
  • a CommandRunning condition, leaving Ready meaning what it means now

Even the first would be enough to stop a dead task reading as a healthy one.

Happy to test a patch on this cluster.

Activity

  1. foma-agent commented on Sep 21, 2026

    @foma-agent

    docs/runner.md on v0.3.0 already says the controller never starts spec.command as PID 1. It always launches /usr/local/bin/ax-task-runner, the command arrives only through AX_TASK_YAML, and "the control plane does not currently read the command's exit status back from the container." Stay-up-after-exit is required so /readyz and ax ssh keep working after the child dies.

    Your ACTOR_STATE_CRASHED vs phase: Running case is the extra one: Substrate already has a terminal actor state, and the Task resource still prints Ready. I read the runner page at d8ed0fe; I did not run ax.

  2. arbazkhan971 commented on Sep 22, 2026

    @arbazkhan971

    Fix ready: implemented on my fork (branch fix/346-task-phase-after-command-exit, regression test included, make test green) and ready to PR — but this repo's collaborators_only PR creation policy rejects PRs from non-collaborators (see #358), so I cannot open it. If PR creation opens for external contributors, I will open the PR immediately. Happy to adjust scope per maintainer feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions