Skip to content

Agent UX: pm test --run never says whether the run was recorded on the item (off by default, and reports changed:false even when it records) #1328

Closed

Description

@unbraind

Summary

pm test <id> --run never tells an agent whether the run became evidence on the item:

  1. With the default test-result-tracking policy (disabled), the run executes and passes, but nothing is written to the item's test_runs or history. The output is changed: false, with no hint that recording is off or how to turn it on. Background runs (--background, with linked-test-adapters installed) also finish passed in the runtime store and never reach the item.
  2. With pm config project set test-result-tracking enabled, the same command appends a test_run_track history entry and a test_runs row, yet the output still reports changed: false.

pm 2026.9.27, fresh pm init defaults.

Reproduction

D=$(mktemp -d) && cd "$D" && git init -q && export PM_AUTHOR=repro && pm init
id=$(pm create --type Issue --title "tracking repro" --description d --json | jq -r '.item.id // .id')
pm claim "$id"
pm test "$id" --add command="node -e 0"

pm test "$id" --run | grep changed          # changed: false
pm history "$id" --compact | grep 'op:'     # create, claim, tests_add; no test_run_track

pm config project set test-result-tracking enabled
pm test "$id" --run | grep changed          # changed: false  (but...)
pm history "$id" --compact | grep 'op:'     # ... now ends with test_run_track

Expected

  • The run result says whether it was recorded, for example recorded: false plus reason: test-result-tracking disabled and the exact command that enables it. The same recovery-bundle style pm already uses elsewhere would work.
  • When a test_run_track entry is appended, changed is true (or a separate recorded: true makes the item mutation explicit).
  • Optionally, the agent quickstart that tells agents to run pm test <id> --run --progress before pm close mentions the policy, because the close evidence it implies is not written by default.

Why it matters for agents

The documented loop is: link tests, pm test --run, pm close with evidence. An agent reads ok: true and assumes the passing run is on the item for reviewers and --validate-close. By default it is not, and when it is, the output says nothing changed. Either way the agent cannot tell from the command's own result.

Activity

  1. unbraind commented on Sep 27, 2026

    @unbraind
    OwnerAuthor

    Tracked in pm-aao1hy. The item is open under linked-test result tracking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions