Skip to content

Tracking: graph evidence — prism task-shaped ops on the Shale PR card #17

Open

Description

@tabladrum

Umbrella for integrating prism's task-shaped operations into the Shale card. Supersedes the scope sketch in RFC #5 with a measured foundation: the ops now exist, are released (prism ≥ v0.19.4 / grove ≥ v0.17.5), and carry a controlled study with independent compiler oracles behind them.

Thesis

Shale today renders testimony (agent intent + session evidence). Prism's deterministic ops upgrade the card to proof: "the graph confirms 38/38 required sites are touched" instead of "the agent says it updated the callers."

Measured basis (paper §5.5, 162-run compile experiment on jackson-databind): applying each agent answer as edits and compiling, every answer that missed a change-site failed to compile and every complete answer compiled. A missed site on this card is a pre-merge build-break catch — deterministic, reproducible, zero LLM cost.

Why this positions Shale

  1. No API key. Every LLM review bot needs a paid key and ships code to a vendor. This runs a pinned engine inside the user's CI: one workflow file, no secrets, code never leaves the runner.
  2. Deterministic where competitors are probabilistic. Same PR → same card, bit-for-bit (engine builds are hash-verified deterministic as of grove v0.17.x). An LLM reviewer's comment is an opinion; "3 sites missing, completeness: closed" is a gateable fact.
  3. The Codecov loop. The card on public PRs is the distribution channel — every adopting repo advertises to every contributor. No sales motion required.
  4. Citable evidence. The card's "why trust this" links to published benchmarks (https://provasign.dev/benchmarks/) and the study. No review bot has that.

Delivery order (each independently shippable)

Phase 1 = #8 + #9 + #10 on Go/Java. That alone is a launchable story: "agents write the code; Shale proves the change-set is complete."

Ground rules (carried from the study, non-negotiable)

  • Relay discipline: engine output renders as-is; no LLM between engine and card (measured: re-processing drops real sites and adds spurious ones).
  • Honesty surfaces: completeness field, caveats, and bucket distinctions (missing vs unverifiable vs ambiguous) always render; project-local is never a green checkmark; TS renders advisory until its measured precision supports more.
  • No nagging: zero triggers → zero graph sections. The card earns attention by being right and silent otherwise.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions