Skip to content

Items can record that verification ran, but not what it measured — quality numbers end up in unqueryable prose #802

Closed

Description

@unbraind

What I hit

I rolled a coverage gate across 18 packages that each track their work with pm. Every one of them produced a measured baseline — line/branch/function percentages — that is the whole point of the work item. There is nowhere structured to put it.

pm test --run persists a real result, which is good:

"test_runs": [
  { "run_id": "test-local-…", "kind": "test", "status": "passed",
    "started_at": "…", "finished_at": "…", "recorded_at": "…",
    "passed": 1, "failed": 0, "skipped": 0 }
]

That answers did verification run and pass. It does not answer what did it measure. Coverage, p95 latency, bundle size, error budget, duplication percentage — none of them fit. passed/failed/skipped counts are the only numbers an item can hold.

So all 18 baselines went into --body prose:

"Measured baseline for this package is 63.21 line, 76.46 branch and 77.40 function coverage."

Why that hurts specifically here

pm list can filter --has-tests / --no-tests and nothing else in this space. So across a fleet that tracks everything in pm, I cannot ask:

  • which packages are below 80% line coverage
  • did this item's number go up or down since the last run
  • what is the fleet median, and which three are worst

Every one of those is a grep over prose today, which is exactly the state pm otherwise exists to replace. For a tool whose value proposition to agents is that the tracker is the context, a measured number is a strange thing to be unable to store.

Verified before filing

On 2026.7.28, fresh workspace, test-result-tracking set to on:

  • pm update --help — ~90 flags, none accept a numeric measurement. --field exists but is explicitly "Set a field declared by an active extension registerItemFields registration", so a package needs to ship an extension before it can record a number about itself.
  • pm test --add accepts command, path, scope, timeout_seconds, pm_context_mode — no result value.
  • pm test --run then persists the test_runs shape above. I confirmed persistence via pm get --json rather than assuming it from the run output.
  • pm list --help — only --has-tests / --no-tests.

What would close it

A metric on the item, recordable at the same moment the test run is: name, numeric value, optional unit, optional threshold, timestamped and appended like test_runs so a series accumulates rather than overwrites. Then pm list --metric-below coverage.lines=80 and a diff between the last two entries both become one command instead of a grep.

Attaching it to the test_runs entry rather than to the item would be the tighter design — the number and the run that produced it stay bound together, which is what makes it trustworthy later.

Context

Filed from a fleet of 18 pm-tracked packages that all landed a coverage gate this week; the baselines now sit in prose in 18 separate items. Happy to supply those items as a concrete corpus if useful.

Activity

  1. coderabbitai commented on Jul 29, 2026

    @coderabbitai
    🔗 Related PRs

    #289 - Fix test-all background progress and linked adapter installs [merged]
    #491 - Harden init target safety and package install verification [merged]
    #545 - SDK contract integrity: workspace paths, dependency truth, validation, and MCP parity [merged]
    #712 - SDK context integrity: atomic Plans, batch similarity, sparse settings, and tombstones [merged]
    #795 - SDK output budgets, exact dependency accounting, and portable merge contracts [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. unbraind commented on Jul 29, 2026

    @unbraind
    OwnerAuthor

    Canonical PM owner: pm-ygerpy. It is open and unclaimed under the shipped test-result tracking lineage, with typed links to pm-i2pc, pm-16f4, the SDK-first epic, and ecosystem conformance. The CodeRabbit enrichment was reviewed and reacted to; its related-PR inventory is useful context, while this record preserves the issue-specific acceptance contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions