Skip to content

ci: the OpenID conformance suite checks auth, locally and on every OIDC change - #519

Draft
PiquelChips wants to merge 1 commit into
mainfrom
ci/conformance-suite
Draft

PiquelChips wants to merge 1 commit into
mainfrom
ci/conformance-suite

Conversation

@PiquelChips

@PiquelChips PiquelChips commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Nothing outside our own tests checked auth against the OpenID specifications. Now just conformance runs the OpenID Foundation's conformance suite against auth, and CI runs it on every change to auth's OAuth and OIDC code, on main and nightly.

  • One recipe, any target. just conformance builds auth and runs it with its own PostgreSQL and Mailpit behind Caddy's HTTPS; just conformance <url> runs against a preview, through its CLI. The suite runs from its prebuilt image, pinned to release-v5.3.4.
  • Passkey sign-in. The suite's browser can't do WebAuthn, so tests/conformance/driver.ts plays it in Chromium with a virtual passkey: it signs an account up, signs in, consents, confirms logout and uploads the screenshots REVIEW tests ask for. Results, screenshots and logs land in target/conformance, which CI uploads.
  • expected.json only shrinks. Every result must equal its entry, or be PASSED without one; an unexpected PASSED fails too. Each of the 44 entries links the issue that fixes it.
  • All four plans run, back-channel logout included. Caddy replaces the suite's nginx so auth can trust the suite's certificate, and a new test-only AUTH_LOOPBACK_LOGOUT_CA lets back-channel logout tokens reach it on localhost. A preview can't reach the suite on your machine, so it skips that plan.

What the run finds on main

Most of the Basic and logout plans fail at their first authorization: auth requires PKCE from every client and the suite sends none. That's #514, fixed by #521. With PKCE optional for confidential clients, what remains is:

The auth fixes (#521, #522, #523) no longer wait on this PR. Whichever merges second updates expected.json.

Closes #51

Note

Written by Claude Opus 5.5 in Claude Code (via T3 Code)

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@PiquelChips PiquelChips self-assigned this Nov 12, 2026
@PiquelChips
PiquelChips added this pull request to stack #525 November 12, 2026 20:18
@PiquelChips PiquelChips added the ci-cd CI-CD label Nov 12, 2026
@PiquelChips PiquelChips added this to the v0.1 milestone Nov 12, 2026
@github-project-automation github-project-automation Bot moved this to Needs Triage in origin-src.dev Nov 12, 2026
@PiquelChips PiquelChips moved this from Needs Triage to In Progress in origin-src.dev Nov 12, 2026
@PiquelChips PiquelChips added schema-change Any changes to the database schema standards Open standards compliance (OIDC, OAuth, RFCs) and removed schema-change Any changes to the database schema labels Nov 12, 2026
@PiquelChips
PiquelChips removed this pull request from stack #525 November 13, 2026 22:30
@PiquelChips
PiquelChips marked this pull request as draft November 13, 2026 22:31
@PiquelChips
PiquelChips added this pull request to stack #529 November 13, 2026 22:31
`just conformance [AUTH_URL]` runs the OpenID Foundation's conformance suite,
pinned to release-v5.3.4, against a local auth it starts or a preview. A
Playwright driver plays the browser with a virtual passkey. Every result is
compared with tests/conformance/expected.json, which records each current
deviation with the issue that fixes it. CI runs it on OIDC changes, on main
and nightly.

AUTH_LOOPBACK_LOGOUT_CA, for tests only, lets back-channel logout tokens
reach the suite on localhost.

Closes #51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd CI-CD standards Open standards compliance (OIDC, OAuth, RFCs)

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

ci: run the OpenID conformance suite

1 participant