Skip to content

ci: fail when the published OpenAPI spec is stale - #43841

Merged
villebro merged 3 commits into
apache:masterfrom
shivamgoel:ci/openapi-spec-drift
Sep 4, 2026
Merged

villebro merged 3 commits into
apache:masterfrom
shivamgoel:ci/openapi-spec-drift

Conversation

@shivamgoel

Copy link
Copy Markdown
Contributor

SUMMARY

docs/static/resources/openapi.json is generated from the running app but committed by hand, so it drifts silently. The last genuine regeneration before #43788 was #33378, sixteen months earlier. This regenerates the spec on every PR and fails on a dirty diff.

Two generation inputs are pinned deliberately, because getting either wrong was a real failure while preparing #43788. The job installs requirements/base.txt: apispec 6.10.0 renders marshmallow 4's unknown=RAISE as "additionalProperties": false where the pinned 6.6.1 does not, so an unpinned guard would fail every PR. And it passes no config file, so generation reflects the routes a default deployment registers — a config with feature flags enabled documents paths that 404 for everyone who hasn't enabled them.

No paths: filter, matching enforce-single-migration-head: a required check that never runs on a PR blocks it forever. The job is ~10s.

TESTING INSTRUCTIONS

Passes on unmodified master — the committed spec reproduces byte-identically on the pinned requirements. To see it fail, add a field to any marshmallow schema without regenerating:

SUPERSET__SQLALCHEMY_DATABASE_URI='sqlite:///:memory:' \
  FLASK_APP='superset.app:create_app()' superset update-api-docs
git diff --stat -- docs/static/resources/openapi.json

Adding one schema renumbers apispec's auto-generated suffixes (User1, Subject1), so diffs are routinely larger than the change.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
  • Introduces new feature or API
  • Removes existing feature or API

@bito-code-review

bito-code-review Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Bito Automatic Review Skipped - Files Excluded

Bito didn't auto-review this change because all changed files are in the exclusion list for automatic reviews. No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the excluded files settings here, or contact your Bito workspace admin at evan@preset.io.

@github-actions github-actions Bot added the github_actions Pull requests that update GitHub Actions code label Sep 4, 2026
@shivamgoel
shivamgoel force-pushed the ci/openapi-spec-drift branch from c8fa99d to 1fd71eb Compare September 4, 2026 01:38
@netlify

netlify Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for superset-docs-preview ready!

Name Link
🔨 Latest commit 9cd960a
🔍 Latest deploy log https://app.netlify.com/projects/superset-docs-preview/deploys/6a9afbf3be13e600085d8756
😎 Deploy Preview https://deploy-preview-43841--superset-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

# paths that would 404 for everyone who has not enabled them.
SUPERSET__SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
FLASK_APP: "superset.app:create_app()"
run: superset update-api-docs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new required check is already failing because regenerating the spec changes openapi.json (110 additions, 46 deletions), so the PR cannot merge as written. Should the regenerated artifact be included with this guard?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@bito-code-review

Copy link
Copy Markdown
Contributor

Yes, the regenerated artifact should be included in the PR. The current workflow is designed to enforce that the checked-in openapi.json matches the output generated by the project's pinned dependencies and configuration. If the spec changes during the check, it indicates that the committed file is stale or was generated in an environment that does not match the project's requirements. To resolve the failure, you should regenerate the spec locally using the exact environment specified in the workflow steps and commit the updated openapi.json to the PR.

.github/workflows/check-openapi-spec-drift.yml

- name: Assert the published spec is up to date
        run: |
          if ! git diff --quiet -- docs/static/resources/openapi.json; then
            echo "::error::docs/static/resources/openapi.json is stale."
            echo "Regenerate it on the pinned requirements, with no config file:"
            echo "  SUPERSET__SQLALCHEMY_DATABASE_URI='sqlite:///:memory:' \"
            echo "    FLASK_APP='superset.app:create_app()' superset update-api-docs"
            git diff --stat -- docs/static/resources/openapi.json
            exit 1
          fi

@pull-request-size pull-request-size Bot added size/L and removed size/M labels Sep 4, 2026
@github-actions github-actions Bot added the doc Namespace | Anything related to documentation label Sep 4, 2026
# paths that would 404 for everyone who has not enabled them.
SUPERSET__SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
FLASK_APP: "superset.app:create_app()"
run: superset update-api-docs

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a registration regression leaves the app with no v1 BaseApi, superset update-api-docs only prints API version not found and exits successfully without rewriting the file. The subsequent diff check then passes against the old committed spec, so this guard misses precisely that broken-generation state. Could this step fail when no spec was generated (or assert that the output was produced) before checking the diff?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in the update-api-docs CLI rather than the workflow, so every caller fails not just this guard.

The spec is generated from the running app but committed by hand, so it drifts
silently — the last genuine regeneration was apache#33378, sixteen months before the
one in apache#43788. Regenerate on every PR and fail on a dirty diff.

The job installs requirements/base.txt and passes no config file, because both
inputs change the output: apispec 6.10.0 renders marshmallow 4's unknown=RAISE
as "additionalProperties": false where the pinned 6.6.1 does not, and a config
with feature flags enabled documents paths that 404 by default.
The command printed "API version not found" to stderr and exited zero, leaving
the committed spec untouched. Any caller diffing the result — including the
drift guard added here — read that as an up-to-date spec.
@shivamgoel
shivamgoel force-pushed the ci/openapi-spec-drift branch from 197e9a6 to 9cd960a Compare September 4, 2026 17:12
@villebro villebro self-assigned this Sep 4, 2026
@codecov

codecov Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 16.66667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.46%. Comparing base (d9b201d) to head (9cd960a).
⚠️ Report is 10 commits behind head on master.

Files with missing lines Patch % Lines
superset/cli/update.py 16.66% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #43841   +/-   ##
=======================================
  Coverage   79.46%   79.46%           
=======================================
  Files        2895     2895           
  Lines      168167   168167           
  Branches    38995    38995           
=======================================
+ Hits       133627   133630    +3     
+ Misses      32041    32036    -5     
- Partials     2499     2501    +2     
Flag Coverage Δ
hive 37.77% <0.00%> (ø)
mysql 57.48% <0.00%> (ø)
postgres 57.52% <0.00%> (ø)
presto 39.66% <0.00%> (ø)
python 83.88% <16.66%> (+<0.01%) ⬆️
sqlite 57.21% <0.00%> (ø)
unit 74.45% <16.66%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@villebro villebro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, this is awesome! Waiting for CI to finish

@villebro
villebro merged commit c182266 into apache:master Sep 4, 2026
74 checks passed
@bito-code-review

Copy link
Copy Markdown
Contributor

Bito Automatic Review Skipped – PR Already Merged

Bito scheduled an automatic review for this pull request, but the review was skipped because this PR was merged before the review could be run.
No action is needed if you didn't intend to review it. To get a review, you can type /review in a comment and save it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc Namespace | Anything related to documentation github_actions Pull requests that update GitHub Actions code size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants