Skip to content

tools/shepherd: W7 if: always() session-log steps record verdict='ok' on failed workflow runs #4499

Description

@romanfjose

Summary

Across .github/workflows/shepherd-board.yml, .github/workflows/shepherd-dup.yml, and .github/workflows/shepherd-rebase.yml, the final W7 session log step runs under if: always() and invokes python3 tools/shepherd.py log with hardcoded completion arguments (--verdict "ok" --note "board + labels refreshed", --note "phase A/B pass complete", --note "rebase shepherd tick") without binding ${{ job.status }}.

When an earlier workflow step fails or is cancelled, if: always() still executes W7 session log and appends a false completion record (verdict: "ok" / completion note) to logs/SESSION.jsonl on automation/session-log even though the scan, rewrite, sync, duplicate pass, or rebase pass never ran.

Live Production Evidence (main @ b1627e8)

Observed in workflow run 43752174585 (shepherd-board, commit b1627e83b437bfc948292f17a8ce4183cef23130):

  1. Step 4 (Scan) crashed with exit code 1 (urllib.error.HTTPError: HTTP Error 403: rate limit exceeded on /repos/limitlessprojects/Go-Hybrid/pulls?state=open&per_page=100&page=3).
  2. Step 5 (Rewrite board) and Step 6 (Sync tier labels) were skipped (conclusion: skipped).
  3. Step 7 (W7 session log) ran via if: always() and executed:
    python3 tools/shepherd.py --token-env GITHUB_TOKEN log --event board-refresh --verdict "ok" --note "board + labels refreshed"
    recording a falsified verdict: "ok" and "board + labels refreshed" audit entry for a failed run where neither the board nor labels were touched.

The same structural defect exists in:

  • .github/workflows/shepherd-board.yml (--event board-refresh --verdict "ok" --note "board + labels refreshed")
  • .github/workflows/shepherd-dup.yml (--event dup-pass --note "phase A/B pass complete")
  • .github/workflows/shepherd-rebase.yml (--event rebase-tick --note "rebase shepherd tick")

Root Cause

tools/shepherd.py (cmd_log) has no --job-status binding, and the three if: always() workflow steps pass static completion strings rather than forwarding GitHub Actions ${{ job.status }}. Because if: always() runs after step failures and cancellations, static --verdict "ok" / completion --note arguments turn failed automation runs into false-positive audit records in logs/SESSION.jsonl.

Reproduction

python3 -m unittest discover -s tools -p test_shepherd_gate.py -v

Before the fix:

  • shepherd.py log --event board-refresh --verdict ok --note "board + labels refreshed" --job-status failure is unsupported, and shepherd-{board,dup,rebase}.yml hardcode static completion notes/verdicts under if: always().

Expected Fix

  1. Add resolve_log_outcome(verdict, note, job_status=None) and --job-status to tools/shepherd.py (cmd_log):
    • Direct CLI calls without --job-status preserve caller verdict and note unchanged.
    • When --job-status is "success", preserve verdict (defaulting None to "ok") and note.
    • When --job-status is non-"success" (e.g. "failure" or "cancelled"), override optimistic verdicts with the normalized job status and rewrite note to "job <status> before completion (intended: <note>)".
  2. Pass --verdict "ok" and --job-status "${{ job.status }}" in all if: always() W7 session log steps across shepherd-board.yml, shepherd-dup.yml, and shepherd-rebase.yml.
  3. Pin both resolve_log_outcome behavior and all if: always() workflow shepherd.py ... log invocations in tools/test_shepherd_gate.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions