Skip to content

Explain OpenResearch runs lost before the remote runner starts (OR-634) - #804

Open
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1793131476-or634-lost-run-reason
Open

devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/1793131476-or634-lost-run-reason

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes OR-634.

Summary

On a leased OpenResearch rental, rental::launch started orx rental-supervise in the background and returned right away. If that remote supervisor exits before it claims the launch (on Massed Compute it hits the unwritable ~/.config, see OR-589 / #766), nothing activates the lease. The run then shows running with an empty log until the provider releases the box. After that it ends lost, and orx exp status gives no reason, because Run::failure_detail only printed reasons for failed runs.

  • src/jobs/rental.rs: the launch command now writes supervisor.pid. Launch then waits up to ~10 s for launch_started. If the supervisor has exited with output and never claimed the launch, launch fails with SUPERVISOR_EXITED plus the tail of supervisor.log, and the normal launch retries run. A retry is safe because no launch_started marker exists yet. A supervisor that is still running, or one that exited silently because another supervisor holds the lock, still counts as launched.
  • src/commands/supervise.rs: if a leased run ends lost and its last launch attempt failed with SUPERVISOR_EXITED, the run records Outcome unavailable: the run never started on box <id>: <error>. Other launch errors, such as SSH drops where the outcome is unknown, keep the watcher's reason.
  • src/plane.rs: failure_detail now prints reason: for lost runs as well as failed ones. orx exp status, orx exp wait and orx runs all use it.

Not in this PR: the fix for the remote config-dir failure itself (#766 / OR-589) and the unreachable-host deadline (#796 / OR-628).

Recordings

Both runs used the same flow. Claude Code in the dashboard ran orx exp run <id> --backend openresearch --flavor rtx_a6000 --disk 80 --timeout 2h against a box whose ~/.config is unwritable.

On main, the run went starting → running → lost, with no reason: line and no log:

Bug reproduction on main

After the fix, the launch retries report the dead supervisor. When the box is released, orx exp status and orx runs print:

reason: Outcome unavailable: the run never started on box sb_1793131924: Remote command failed (exit 1): remote supervisor exited before starting the run:
Permission denied (os error 13)

Post-fix functionality

Test plan

  • cargo fmt --check, cargo clippy --locked -- -D warnings
  • cargo test --locked. New tests: startup_check_reports_a_supervisor_that_died_before_launch and lost_run_reports_its_recorded_reason. Three local::projects tests also fail on unmodified main in this VM, because of its git environment.
  • Post-fix dashboard run on Linux, recorded above on commit 8fef931. The follow-up commit only narrows when the "never started" reason is used. The recorded run took that path.

Repro environment: Linux; a local SSH box with an unwritable /home/Ubuntu/.config; a local stand-in for the openresearch.sh sandbox/lease API that releases unactivated boxes after 5 minutes; the real Claude Code agent. No real GPU provider was used.

Automation improvements

  • The Codex preflight failed with an expired CODEX_AUTH_JSON ("Provided authentication token is expired"). Refreshing that secret would let Codex-based repros run.

Link to Devin session: https://app.devin.ai/sessions/93dec1fe602c46ec8b92a1e5f1c86ec3
Open in Devin Desktop: https://app.devin.ai/desktop/session/93dec1fe602c46ec8b92a1e5f1c86ec3?variant=devin

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@greptile-apps

greptile-apps Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the follow-up fixes the false startup claim.

Summary

The PR explains leased runs lost before their remote supervisor starts.

  • Rental launches report supervisors that exit before claiming a run.
  • Lost runs show the reason saved for them.

The follow-up addresses the previous finding: an unknown SSH outcome no longer becomes a claim that the run never started.

Reviews (2) · Last reviewed commit: "Only claim a lost run never started when..." · Reviewed by Greptile

Comment thread src/commands/supervise.rs Outdated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai review — re-review after f4c0ac5 (only claim 'never started' when the supervisor was seen dying).

@devin-ai-integration devin-ai-integration Bot added the awaiting review Devin bugfix PR is open and needs human review before merge label Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review Devin bugfix PR is open and needs human review before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant