Skip to content

factory runtime failure: lifecycle manage pr_frontier_failed #52

Closed as not planned

Description

@mikeroysoft-agent

Fingerprint: b4b6c83c4478

factory watch observed a runtime failure of the factory itself from local evidence: signal lifecycle, stage manage, pr_frontier_failed. Filed automatically; no human has looked at it yet.

Evidence

Journal rows (1 newest)

{"dispatcher_run_id":"86a5d5d8-b99d-4d52-88eb-0744758280f2","root_execution_id":"f8bbd1ab-86c0-47c8-9ac8-cb8c94c26dbc","execution_id":"0363b876-9648-4d7e-9958-18564831dbc6","parent_execution_id":"f8bbd1ab-86c0-47c8-9ac8-cb8c94c26dbc","ticket":16,"attempt":null,"review_round":null,"stage":"manage","event":"lifecycle","schema_version":1,"event_id":"4283d1b9-0ca6-4175-85b1-6aed60ef2442","kind":"exit","at":"2026-10-02T13:25:58.078074Z","outcome":"mechanism_failure","reason":"pr_frontier_failed","process":{"pid":2610495,"boot_id":"8799e8c2-3387-4aa7-a525-c71ce4d444c7","start_ticks":23173027,"pid_namespace":"pid:[4026531836]","uid":1000,"state":"R","ppid":1567},"locks":[],"sequence":8}

Reproduce

zcat -f .factory/events.jsonl* | grep -F '"reason":"pr_frontier_failed"'
factory watch --dry-run

Expected behavior

The factory pass completes without pr_frontier_failed at stage manage, and factory watch --dry-run no longer reports fingerprint b4b6c83c4478.

Activity

  1. mikeroysoft-agent commented on Oct 17, 2026

    @mikeroysoft-agent
    CollaboratorAuthor

    Triage: This is a fully specified bug report: it has a clear problem statement (factory runtime failure at the manage stage with mechanism failure pr_frontier_failed, fingerprint b4b6c83c4478), concrete reproduction steps, and an observable done-condition (factory pass completes without pr_frontier_failed and factory watch --dry-run no longer reports the fingerprint). It is contained within this repository and does not touch release, signing, or security policy, so it does not require human authorization or design judgment. An agent can investigate the manage-stage mechanism failure, fix the root cause, and verify via the provided commands.

    Agent brief: The factory's manage stage exits with outcome mechanism_failure and reason pr_frontier_failed (fingerprint b4b6c83c4478), detected automatically via factory watch. Investigate the root cause of the manage-stage mechanism failure and fix it so the factory pass completes cleanly without that failure. Verification: zcat -f .factory/events.jsonl* | grep -F '"reason":"pr_frontier_failed"' returns nothing, and factory watch --dry-run no longer reports fingerprint b4b6c83c4478.

  2. added
    ready-for-agentFully specified and ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 17, 2026
  3. added and removed
    ready-for-agentFully specified and ready for an AFK agent
    on Oct 17, 2026
  4. removed their assignment
    on Oct 17, 2026
  5. mikeroysoft-agent commented on Oct 17, 2026

    @mikeroysoft-agent
    CollaboratorAuthor

    Factory dispatcher escalating: agent/52: PR not published (no commits over main or existing PR target mismatch); inspect dispatcher log.

    Escalation packet: /home/mike/.local/share/district/roots/rocm-cli/.factory/escalations/52.md

    Worker logs: /home/mike/.local/share/district/roots/rocm-cli/.factory/logs/52-attempt-1.log

    Worker handoff notes:

    Handoff: #52 factory runtime failure: lifecycle manage pr_frontier_failed

    Root cause

    Not a rocm-cli bug. One manage execution on 2026-10-02T13:25:57Z for ticket 16
    raised inside factory/manage.py::frontier_pass: its first gh child
    (gh issue view 16 --json …, pid 2612588) exited non-zero after 0.39 s and the
    handler at manage.py:746-749 set mechanism_failure / pr_frontier_failed.
    The next pass completed; the retained journal has 7009 completed manage exits
    for ticket 16 and this single failure, so the gh error was transient. Its
    text was only ever printed to the dispatcher log (str(CalledProcessError)
    carries no stderr), and the journal for that boot has rotated out, so the
    specific GitHub error is unrecoverable.

    The factory defect that made this a distinct ticket: the frontier was the only
    handler minting its own reason. pr_frontier_failed is absent from
    factory/runtime_events.py::REASONS (so dashboard --runtime-json sanitises
    it to null), while the viability, escalation, handoff and triage handlers all
    journal the same failure class as github_command_failed — the fingerprint
    factory watch already tracks for it.

    What changed

    Fix lives in the factory repo, not this one — nothing to commit on agent/52.

    • Repo: ~/.local/share/district/roots/factory, worktree
      .factory/wt-rocm-cli-52, branch agent/rocm-cli-52, one signed-off commit
      on origin/main (51f9e03, == released factory 0.4.1).
    • factory/manage.py:748: reason pr_frontier_failed → github_command_failed.
    • tests/test_frontier.py::test_failed_gh_read_journals_the_shared_github_command_failed_reason:
      a failing gh issue view in the frontier journals ("mechanism_failure", "github_command_failed") and that reason is in runtime_events.REASONS.
      Failed before (observed pr_frontier_failed), passes after.
    • CHANGELOG.md Unreleased line.

    Verified

    • PYTHONPATH=. …/factory/bin/python -P -m unittest tests.test_frontier tests.test_watch tests.test_handoff tests.test_lifecycle_consumers → 52 OK.
    • uvx ruff@0.16.10 check factory/manage.py tests/test_frontier.py → clean.
    • Against the real rocm-cli journal with the patched code on PYTHONPATH:
      factory watch --dry-run does not report b4b6c83c4478 (it also does not
      with the installed tool, because factory runtime failure: lifecycle manage pr_frontier_failed #52 is the filing for it and no newer
      evidence exists).
    • rocm-cli gate: see .factory/gate-report-52.md.
    • Full factory suite (unittest discover -s tests -t .): 588 run, 2 errors —
      tests.test_mcp and tests.test_plan fail to import (No module named 'test_evidence'); identical on a detached origin/main worktree, so
      pre-existing and unrelated.

    Unverified / not done

    • The installed uv tool factory was not reinstalled; the running dispatcher
      keeps the old reason until the factory branch merges and the tool upgrades.
    • zcat -f .factory/events.jsonl* | grep -F '"reason":"pr_frontier_failed"'
      still returns the one 2026-10-02 row: that is the journal, not code, and was
      left intact. After the upgrade no new row with that reason can be written.
    • All four gh-failure handlers log str(exc) without exc.stderr, so the next
      transient gh failure will again be undiagnosable once the journal rotates.
      Out of scope here; one-line improvement if it recurs.

    Next

    1. Push agent/rocm-cli-52 in the factory repo and open a PR there.
    2. After merge: upgrade the installed tool, then factory watch --dry-run in
      rocm-cli to confirm with the installed code.
    3. Close this ticket as fixed-upstream; no rocm-cli change.
  6. mikeroysoft-agent commented on Oct 17, 2026

    @mikeroysoft-agent
    CollaboratorAuthor

    Factory manager: The escalation is a repository mismatch, not a failed rocm-cli gate: the worker changed the factory repository and left no commits on agent/52, so the dispatcher had nothing to publish here.

    The external worktree’s factory/manage.py:748 uses github_command_failed. This normalizes the journal reason; it does not prevent GitHub command failures. The worker reports a transient failure, but its underlying stderr is unavailable. Absence of the fingerprint from watch --dry-run is insufficient proof: the handoff says the unpatched tool also omits it because #52 already tracks the evidence.

    Human action: review and publish agent/rocm-cli-52 from the factory repository, verify its checks, then merge and upgrade the installed factory. Confirm the installed behavior before closing #52. Publication, merge, and installation status were not verified here. Retrying the rocm-cli worker would target the wrong repository again.

  7. mikeroysoft-agent commented on Oct 17, 2026

    @mikeroysoft-agent
    CollaboratorAuthor

    Factory handoff request 52/1: a human decision is needed on #52.

    No decision owner could be determined (unassigned); a maintainer should claim it.

    Question
    How should #52 proceed? It escalated with: agent/52: PR not published (no commits over main or existing PR target mismatch); inspect dispatcher log. Automatic recovery stopped because the manager asked for a human decision.

    Evidence (bounded links; worker logs and the escalation packet stay on the runner)

    Proposed next step
    Read the evidence, then amend the ticket and relabel ready-for-agent for another attempt, fix the branch by hand, request changes on the PR, or close it. Replying here is context for humans; nothing here retries, approves or merges.

    Routing

    • decision_owner: absent — ticket body has no **Decision owner** section
    • initiative: skipped — initiative owner only routes requirements, not implementation
    • configuration: absent — no [collaboration] section in .factory.toml; legacy behaviour unchanged
      To claim or reassign this decision, set a **Decision owner** section (a GitHub login) in the issue body; the next pass reflects it and never overwrites it.
  8. mikeroySoft commented on Oct 19, 2026

    @mikeroySoft
    Owner

    Duplicate of mikeroySoft/factory#264 (manager gh failures; host gh failures are no longer filed since ROCm#281). Stopping watch from filing engine bugs here is tracked in mikeroySoft/factory#311. Engine bugs are tracked in mikeroySoft/factory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-humanRequires human implementation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions