Tracking: reliable CI review outcomes and an opt-in shared review gate

[Qiyuanqiii]

Problem Statement

OCR needs a consistent way for CI to distinguish a successfully executed review from a review result that satisfies a repository's merge policy. A successful process exit, complete review coverage, acceptable findings, and successful feedback delivery are separate facts.

For example:

• A selected file times out while other files produce useful findings. Issue ocr timed out and failed for some files, but status code=0 #1027 demonstrates that an operationally incomplete review can still exit 0.
• A configured budget stops a review early. Keeping the existing successful process status may be appropriate for interactive use, but it cannot establish complete coverage for a required CI check.
• A complete review reports a high-severity finding. A team may want that finding to block merging, while another team may want advisory feedback only.
• Review execution succeeds, but publication fails, or the saved artifact describes a different base/head snapshot.

Without an explicit shared contract, each integration has to infer its own meaning of "review passed." That can produce inconsistent decisions and make incomplete or undelivered reviews look successful.

This issue proposes a bounded first version: reliable operational exit status, a shared opt-in evaluator over saved results, and opt-in integration with GitHub Actions and the GitLab CI example. Three open PRs already propose the implementation. This issue brings their scope, policy choices, and remaining acceptance work together; it does not imply that the approach has already been accepted or released.

Proposed Solution

Responsibilities of the three PRs

PR	Responsibility	Compatibility and dependency
#1030 - fix(cmd): return non-zero for partial operational failures	Return non-zero for partial operational failures, including timeouts and provider errors, while preserving completed findings and diagnostics. Apply the same execution policy to SARIF metadata. This is the targeted correction for the behavior reported in #1027.	Changes default execution status for operationally incomplete reviews. Budget-only partial reviews retain their historical successful status. Independent of the new gate.
#1457 - feat: add opt-in gate for saved review results	Add offline ocr gate evaluation with pass, fail, and inconclusive decisions. Require complete, consistent selected-item coverage and valid recorded delivery evidence. Optionally enforce severity thresholds and exact expected base/head object IDs.	Existing ocr review behavior is unchanged. Only invoking the gate opts into its policy. It makes no model, Git, or hosting calls and does not require #1030.
#1461 - feat(ci): add opt-in review gate to GitHub Action and GitLab CI	Connect both integrations to the same evaluator; freeze reviewed revisions, preserve execution/publication failures, publish useful results, retain diagnostics, and enforce the final outcome.	Disabled by default. Depends on #1457 and an OCR build containing gate. When enabled, bypass checkpoint range narrowing and advancement so the full reviewed range remains relevant to the decision.

These are deliberately separate review units: runtime correctness, reusable policy evaluation, and platform integration. #1461 currently includes the prerequisite changes in its aggregate diff and should be updated after #1457 lands.

Proposed contract

The shared evaluator emits a separate ocr.gate/v1 decision with ordered checks and reason codes. Only pass exits 0; both fail and inconclusive exit non-zero.

Evidence, assuming other applicable checks pass	Proposed gate decision
Every selected item completed or was reused under the existing resume contract; evidence is consistent	pass
A finding meets or exceeds an explicitly configured severity threshold	fail
Selected items failed, were waived, or remain incomplete, including budget stops	inconclusive
Zero selected items; missing or contradictory evidence; unsupported manifest version	inconclusive
Recorded code_comment failure, or a mismatch against supplied expected revisions	inconclusive
Unknown or missing finding severity when severity enforcement is enabled	inconclusive

A confirmed severity violation takes precedence over an inconclusive check, while the decision retains all evaluated checks. Severity and revision matching are optional in the standalone command; coverage and recorded delivery checks are mandatory once it is invoked. The CI integrations supply the frozen expected revisions.

CI must preserve three independent outcomes: review execution, publication, and gate evaluation. A passing artifact must not erase a process or publication failure. The proposed adapters attempt publication and evaluation before final enforcement and retain review/gate artifacts for diagnosis. Severity is evaluated against the original findings, before publication routing or deduplication can hide them from the decision.

The offline gate can inspect recorded delivery evidence, but it cannot independently attest that a hosting API accepted this run's feedback. The adapters therefore enforce publication separately, including confirmed final-summary publication and no remaining failed inline submissions.

Policy Choices and Tradeoffs

1. Default correctness versus opt-in merge policy. Operational review failures should be visible through the process status, as proposed in fix(cmd): return non-zero for partial operational failures #1030. Blocking on findings or on budget-limited coverage is a separate policy choice. The adapters default the gate to off, preserving advisory workflows and compatibility with older CLI builds.

2. Incomplete evidence versus false blocking. Treating incomplete, legacy, waived, or empty-selection results as inconclusive avoids silently approving an unproven review. It can also block legitimate workflows. Manifest v1 cannot distinguish an empty change from a change whose files were all excluded, so accepting zero selected items would require additional evidence or an explicit policy extension.

3. Severity enforcement versus model variability. Severity is useful for a configurable threshold, but model findings can be noisy or wrong. The threshold remains optional, and a gate pass is not a certification that the code has no defects. Maintainers should decide whether the proposed severity policy is suitable for a required check in their workflow.

4. Full-range coverage versus model cost. A checkpoint that reviews only the newest push can omit a blocking finding from an earlier push. The proposed gated integrations review the full merge-base-to-head range, which is safer but may repeat work and increase token cost. Cross-push reuse, as discussed in Review gate loops re-review the whole range every push: no cross-push incremental reuse despite per-item fingerprints already existing #854, could reduce that cost if it preserves coverage, finding validity, and revision provenance; it is a separate optimization.

5. Strict publication versus transient hosting failures. The proposed policy blocks when an inline submission remains failed, even if a fallback summary was published. This is conservative and may be stricter than some teams need. Accepting an alternative delivery route would require evidence that the relevant feedback was delivered, rather than treating any summary URL as success. In particular, an old sticky-comment URL does not prove the current update succeeded.

6. Snapshot identity versus live-head freshness. Exact base/head matching proves that the result describes the expected immutable inputs. It does not prove that a PR/MR is still at that head when review finishes or merging occurs. Live-head checks, branch protection, and merge-time freshness are separate concerns; this first version should not overstate that guarantee.

7. Evidence scope and trust. Coverage applies to the selected set, not automatically to every file in the diff. The artifact, workflow policy, and expected revisions must be trusted; the evaluator does not authenticate JSON or validate the repository's exclusion policy. A signed-artifact system, generalized policy language, and waiver framework are outside this proposal.

Feedback is particularly useful on the compatibility boundary, the zero-selection/waiver behavior, and whether the initial publication policy should remain this strict.

Alternatives Considered

• Use only ocr review exit codes. Necessary for operational correctness, but insufficient for optional severity policy, budget-only partial results, and independently recorded publication failures.
• Implement separate policies inside each adapter. Avoids a new command, but duplicates interpretation of the manifest and makes platform behavior easier to diverge. A shared evaluator also supports local inspection and other CI systems without another model call.
• Enable blocking by default. Simpler configuration, but changes existing advisory and budget-limited workflows and would require a compatible CLI release everywhere immediately.
• Create a larger remote service or policy framework first. Could eventually support more controls, but adds deployment and maintenance requirements before the basic saved-result contract has been agreed.

Validation and Acceptance

As of 2026-09-20, all three PRs are open, and all currently reported checks on their current heads are successful.

The PRs document Go regression/race tests, gate-package coverage, 55 GitHub Action contract tests, and 156 GitLab Python tests. Additional offline adapter scenarios use the real gate executable with review/publication test doubles.

The following is an ordinary Markdown tracking checklist, not a parent/sub-issue hierarchy:

• Propose the operational-status correction in fix(cmd): return non-zero for partial operational failures #1030, including partial-output preservation and SARIF consistency.
• Propose the shared evaluator and regression coverage in feat: add opt-in gate for saved review results #1457.
• Propose both CI adapters, compatibility checks, publication enforcement, documentation, and contract tests in feat(ci): add opt-in review gate to GitHub Action and GitLab CI #1461.
• Agree on the first-version contract and the policy tradeoffs above with maintainers.
• Complete review and merge the operational-status correction independently; merge the shared evaluator before its dependent integration and update the dependent branch accordingly.
• Validate gate-enabled hosted workflows with actual model/publication access: a passing review, incomplete review, blocking finding, and publication failure, retaining diagnostics in each failure case.
• Make a CLI build containing gate available through the intended release path and verify the documented version/configuration requirements for both integrations.

This tracking issue should be completed when the agreed first-version acceptance work is done. Related future work need not expand that boundary: live-head/merge-time freshness, selection and exclusion evidence, explicit waivers, safe cross-push reuse, and additional CI platforms can be discussed separately.

Affected Area

CLI / Commands; Output / Formatting; GitHub Action; GitLab CI example; Documentation.

Related Work

• ocr timed out and failed for some files, but status code=0 #1027: the existing operational partial-failure report addressed by fix(cmd): return non-zero for partial operational failures #1030.
• fix(cmd): return non-zero for partial operational failures #1030, feat: add opt-in gate for saved review results #1457, feat(ci): add opt-in review gate to GitHub Action and GitLab CI #1461: the three implementation PRs tracked here.
• Review gate loops re-review the whole range every push: no cross-push incremental reuse despite per-item fingerprints already existing #854: related cross-push reuse/cost discussion, with a different implementation scope.

AI/LLM Disclosure

Codex (GPT-6) was used to prepare this issue, inspect the related PRs, and verify their current GitHub status.
The related implementation work also used Codex; repository-required model reviews documented in the PRs used the open-code-review CLI with deepseek-v4-flash.
I proposed and organized the relevant ideas, conducted an initial review of my code, and take ownership of my output

[Qiyuanqiii]

I forgot to add the relevant md docs; I’ll complete them at noon.

[Qiyuanqiii]

All documents have been fully completed.