Skip to content

fix(ci): build dist before running benchmarks - #343

Merged
naorpeled merged 1 commit into
jeremydaly:mainfrom
naorpeled:fix/benchmarks-build-dist
Aug 8, 2026
Merged

fix(ci): build dist before running benchmarks#343
naorpeled merged 1 commit into
jeremydaly:mainfrom
naorpeled:fix/benchmarks-build-dist

Conversation

@naorpeled

Copy link
Copy Markdown
Collaborator

The Benchmarks workflow crashed on the v1.5.0 release (run 31267507913):

Error: Cannot find module '/home/runner/work/lambda-api/lambda-api/dist/cjs/index.js'.
Please verify that the package.json has a valid "main" entry
    at Object.<anonymous> (benchmarks/frameworks/lambda-api.js:10:19)

Root cause

benchmarks/frameworks/lambda-api.js loads the library from the working tree with require('../../'), which resolves through the root package's main. #326 changed main from index.js to ./dist/cjs/index.js, so the adapter now requires a build — but the workflow only ran npm ci inside benchmarks/, never installing or building the root package.

The benchmark suite (#327) merged before the dual build (#326), and benchmarks only run on release: published, so v1.5.0 was the first time the two met. The baseline framework has no lambda-api dependency and completed fine, which is why the log shows 10 green scenarios before the crash.

Fixes

1. Build dist/ before benchmarking. Adds a root npm ci --ignore-scripts + npm run build ahead of the benchmark install, matching how release.yml sequences the same steps.

2. Honor LAMBDA_API_VERSION in the adapter. Surfaced while verifying the first fix. run.js:65 already read the env var, but only for the run metadata that renders the caption — the per-framework table rows use each adapter's own version field, and lambda-api's read package.json directly. Since that ships a 0.0.0-development placeholder until publish time, every table row would have rendered as:

| lambda-api `0.0.0-development` | 298,049 | ...

while the caption directly above it correctly said lambda-api v1.5.0. That would have been committed straight into the README by the workflow's final step. The adapter now uses the same LAMBDA_API_VERSION || pkg.version precedence as run.js.

Verification

Reproduced locally by deleting dist/ — same MODULE_NOT_FOUND at the same adapter line; loads cleanly after npm run build.

Ran the exact workflow command end-to-end (node run.js --md results/RESULTS.md --json results/raw.json --update-readme with LAMBDA_API_VERSION=1.5.0). All 6 frameworks × 10 scenarios complete, and both the caption and every table row now read 1.5.0. Generated results were reverted out of this branch — the committed numbers should come from the runner, not my laptop.

The subsequent Format README step pins prettier@2, which matches the repo's ^2.3.2 devDependency, so no version skew there.

Note

The v1.5.0 README still has stale benchmark numbers, since the workflow never reached its commit step. Once this merges, re-running the workflow via workflow_dispatch with version 1.5.0 will refresh it.

The benchmark adapter loads lambda-api from the working tree via
require('../../'), which resolves through the root package's "main".
jeremydaly#326 changed that from index.js to ./dist/cjs/index.js, so the adapter
now needs a build that the workflow never performed — it only ran
npm ci inside benchmarks/. The job crashed on the first lambda-api
scenario with MODULE_NOT_FOUND.

Also honor LAMBDA_API_VERSION in the adapter. run.js already used it
for the caption, but the per-framework table rows read package.json
directly, so every row rendered as 0.0.0-development.
@naorpeled
naorpeled merged commit b3b8e1f into jeremydaly:main Aug 8, 2026
3 checks passed
@naorpeled

Copy link
Copy Markdown
Collaborator Author

Added a third fix (6bcb402): the final step now opens a PR instead of pushing to the default branch.

  • Commits the refreshed README to a per-run branch (benchmarks/refresh-<version>-<run_id>) so concurrent or repeated runs never clobber each other, then opens a PR against the default branch.
  • PR body links back to the run and embeds the full results table in a <details> block (~2.9KB, well under GitHub's 65536 body limit).
  • Renamed the workflow_dispatch input commitopen_pr since the semantics changed, and added pull-requests: write to the job permissions.
  • Dropped [skip ci] from the commit message — it was there to avoid re-triggering CI on a direct push, which no longer applies.

One thing to be aware of before merging: PRs opened with the default GITHUB_TOKEN do not trigger pull_request workflows — this is a deliberate GitHub restriction to prevent recursive runs. So the benchmark PR will show no status checks. If the default branch has required checks, it will not be mergeable without an admin override.

If that's a problem, the usual fix is to create the PR with a PAT or GitHub App token instead of github.token. I left it on GITHUB_TOKEN since that needs no new secret and the change is README-only — happy to switch if you'd rather the PR run CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant