Skip to content

Add CodSpeed continuous benchmarking - #88

Merged
shenxianpeng merged 1 commit into
mainfrom
codspeed-wizard-1786537639327
Aug 12, 2026
Merged

shenxianpeng merged 1 commit into
mainfrom
codspeed-wizard-1786537639327

Conversation

@codspeed

@codspeed codspeed Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds continuous performance measurement to MkDocs NG with CodSpeed. Every pull request now runs a benchmark suite in CI and reports how the change affects build and rendering performance.

What was added

Benchmark suite (benchmarks/) — 23 benchmarks built on pytest-codspeed, run in CPU simulation mode. The corpus is a generated MkDocs project (31 Markdown pages across 5 nested sections, plus static assets) produced from a fixed pseudo-random seed, so the workload is identical on every run and results only move when MkDocs itself changes.

File Covers
test_build.py Full mkdocs build (with and without use_directory_urls), Markdown population of all pages, theme template rendering
test_structure.py get_files, get_navigation, File creation, file lookups, page rendering (regular and large page), get_toc
test_config.py load_config with full validation, YAML parsing, loader construction
test_search.py Search index entry creation, JSON generation, the HTML content parser
test_utils.py Front matter parsing, title extraction, relative/normalized URLs, nest_paths, tag stripping
corpus.py / conftest.py Deterministic corpus generation and shared fixtures

CI workflow (.github/workflows/benchmarks.yml) — runs on pull requests, pushes to main and workflow_dispatch (used by CodSpeed for backtests). It uses OpenID Connect for authentication, so no token secret is needed. Actions are pinned by commit SHA to match the convention used by the existing workflows.

Developer ergonomics — a bench Hatch environment (hatch run bench:run) so the benchmarks can be run locally like the other checks, a note in CONTRIBUTING.md, and the CodSpeed badge in README.md.

Verification

The suite was run locally under CodSpeed's CPU simulation instrument: 23 benchmarks passed and reported, including the full-site build (~1.5 s simulated), page population (~0.9 s) and theme rendering (~0.3 s). Benchmarks also pass as plain tests (pytest benchmarks, ~3 s) so they stay cheap to run outside CI. ruff check, ruff format, isort, codespell and markdownlint were run on the touched files.

Next steps

  • Merge this PR so the first run on main establishes the performance baseline; subsequent PRs will be compared against it.
  • Once a baseline exists, consider enabling performance checks to fail CI on significant regressions.
  • Extend the suite when new performance-sensitive code paths land (for example plugin event dispatch or mkdocs serve incremental rebuilds).

Add a pytest-codspeed benchmark suite covering the build pipeline, page
rendering, file collection, navigation, configuration loading, the search
index and the shared utilities, plus a GitHub Actions workflow that runs
them on every pull request.
@codspeed

codspeed Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Congrats! CodSpeed is installed 🎉

🆕 23 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Open in CodSpeed

@codspeed
codspeed Bot marked this pull request as ready for review August 12, 2026 12:48
@codspeed
codspeed Bot requested a review from shenxianpeng as a code owner August 12, 2026 12:48
@shenxianpeng shenxianpeng added the enhancement New feature or request label Aug 12, 2026
@shenxianpeng
shenxianpeng merged commit bb9d39d into main Aug 12, 2026
23 checks passed
@shenxianpeng
shenxianpeng deleted the codspeed-wizard-1786537639327 branch August 12, 2026 12:50
shenxianpeng added a commit that referenced this pull request Aug 12, 2026
Extend the benchmark suite merged in #88 with the scenarios needed to
track the build-performance work for upstream issue mkdocs/mkdocs#3695:

* full builds of the deterministic corpus at several sizes (10/50/200
  pages by default, overridable via MKDOCS_BENCH_SIZES for local
  scaling studies) - a local run at 100/400/1600 pages measured
  7.8/9.3/22.1 ms per page, i.e. clearly superlinear growth
* a dirty rebuild with no modified files, approximating the
  'mkdocs serve --dirty' inner loop
* a single page rendered against the largest site's navigation,
  isolating the O(N^2) sitewide template-rendering term

corpus.write_site() gains optional sections/pages_per_section
parameters (defaults unchanged, existing benchmarks unaffected).
shenxianpeng added a commit that referenced this pull request Aug 12, 2026
* Add scaling and serve-loop benchmarks to the CodSpeed suite

Extend the benchmark suite merged in #88 with the scenarios needed to
track the build-performance work for upstream issue mkdocs/mkdocs#3695:

* full builds of the deterministic corpus at several sizes (10/50/200
  pages by default, overridable via MKDOCS_BENCH_SIZES for local
  scaling studies) - a local run at 100/400/1600 pages measured
  7.8/9.3/22.1 ms per page, i.e. clearly superlinear growth
* a dirty rebuild with no modified files, approximating the
  'mkdocs serve --dirty' inner loop
* a single page rendered against the largest site's navigation,
  isolating the O(N^2) sitewide template-rendering term

corpus.write_site() gains optional sections/pages_per_section
parameters (defaults unchanged, existing benchmarks unaffected).

* Update release notes for CodSpeed benchmarking suite (#88, #89)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants