Skip to content

fix(cli): report verification failures in structured output - #1835

Merged
mldangelo-oai merged 6 commits into
openai:mainfrom
dipeshbabu:fix/verify-fix-auth-errors
Oct 23, 2026
Merged

mldangelo-oai merged 6 commits into
openai:mainfrom
dipeshbabu:fix/verify-fix-auth-errors

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1817. Failed verify-fix --json calls previously returned empty stdout, and a Codex exit without diagnostics provided no explanation. Failed verification now returns a structured error with exit code 2 while preserving stderr diagnostics.

Changes

  • Reuse the existing scan failure formatter for JSON/JSONL and full-output error envelopes.
  • Capture and forward diagnostics using the patch command's existing progress-stop pattern; provide an exit-status explanation when Codex supplies no diagnostic.
  • Preserve completed verification results and interruption exits 130/143. Add regression coverage and document the failure contract. No commands, flags, or authentication settings change.

Testing

Approved head: 1383e2cdcd7c4be93f3aca7f0b0319b82374ccd7. The final integration with main 53da0ecc80df21d69a7a264539f1a2352764fe75 was tested with Codex/SDK 0.164.0; merged main b97013c17bbd277d38268afd6775214577d1bb70 has the identical source tree.

  • Reproduced the original failure in a real terminal: the old CLI printed the authentication error inside its progress screen and then erased it. The fixed CLI leaves the diagnostic visible after the screen closes.
  • Real rejected synthetic ChatGPT credentials produced VERIFICATION_FAILED and exit 2 in JSON mode. Terminal visibility, JSON diagnostics, and sign-in advice passed again with Codex 0.164. Human and full-output error contracts also passed in the earlier 0.163 run.
  • Real before/after verification returned still_vulnerable/exit 1 and readable FIXED/exit 0. Independent HTTP checks confirmed the security boundary and legitimate behavior; source files remained unchanged.
  • Real Standard and bounded two-round Deep scans completed on both runtimes. Discovery workers, reducers, sealed-artifact validation, output consistency, and unchanged-source checks passed. The 0.164 runs used a small compatibility fixture; the earlier 0.163 runs covered the larger application fixture.
  • Full local SDK run on the 0.163 integration: 6,956 passed, 83 skipped, zero failures (seed 2887100861). The final 0.164 integration passed all 118 focused verification/diagnostic/hosted tests, types/generated-file checks, formatting, builds, portable plugin checks, and git diff --check.
  • Full source CI on the approved head passed: 48 successful jobs, two skipped. All required merge checks passed.

Risk and rollout

Failed JSON/JSONL calls now emit status: "failed", code: "VERIFICATION_FAILED", and message instead of empty stdout. Their full-output form uses ok: false; other formats retain stderr-only failures. Automation should inspect the exit code and failure fields. Completed verdicts, interruption behavior, and worker configuration remain unchanged.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the bug Something isn't working label Oct 20, 2026

@mldangelo-oai mldangelo-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reproduced the missing JSON result with real rejected ChatGPT authentication in isolated test state. This now returns a useful VERIFICATION_FAILED error and exit 2, while keeping the original diagnostic and sign-in guidance visible. Normal text output and the full-output envelope also behave correctly.

I checked real verification before and after repairing a synthetic application, ran Standard and bounded Deep scans with both fixes applied, and reran the focused CLI/hosted tests against current main. The verdicts, interruption behavior, and successful output contracts are preserved. Reusing the existing failure formatter and progress pattern keeps this small and consistent with the other commands. This is ready to merge.

@mldangelo-oai mldangelo-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real-terminal check confirms the original issue: the old CLI printed the authentication error inside the progress screen and then erased it; this fix leaves it visible and still returns structured JSON errors.

I rechecked the integration with current main and Codex 0.164, including before/after verification, interactive Markdown and JSON, and real Standard and bounded Deep scans. Those checks passed, as did the 118 focused regressions and the refreshed head’s full source CI. The fix stays small and follows the existing error-handling pattern. Approving this refreshed head.

@mldangelo-oai
mldangelo-oai merged commit b97013c into openai:main Oct 23, 2026
56 checks passed
@github-actions github-actions Bot mentioned this pull request Oct 23, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: verify-fix silently exits when ChatGPT authentication is invalid

3 participants