feat: single apify-cli bundle + native Windows ARM64 (supersedes #1057) - #1169
feat: single apify-cli bundle + native Windows ARM64 (supersedes #1057)#1169vladfrangu wants to merge 14 commits into
Conversation
The CLI previously shipped three full bundles (apify, actor, apify-cli), dropping the same ~70MB binary three times into the install directory. Now we build one apify-cli bundle; apify and actor are tiny wrapper scripts that invoke it with APIFY_CLI_ENTRYPOINT set, which the entrypoint uses to pick the command set. - build-cli-bundles: build only apify-cli; publish apify-*/actor-* copies as backwards-compatible backups for the old upgrade flow - entrypoints: new apify-cli entrypoint + resolveEntrypoint() - install.sh / install.ps1 / dev-test-install.sh: drop the apify-cli binary and generate apify/actor (.cmd on Windows) wrapper scripts - upgrade: download the single apify-cli bundle; write it and the wrappers atomically (temp + rename) to avoid ETXTBSY on a running binary (Linux) - bundleMigration: on first run after upgrading from a legacy 3-bundle install, copy the running bundle to apify-cli and replace apify/actor with wrappers (rename-not-delete on Windows for the running exe) - useCLIVersionAssets: match only apify-cli-* assets, ignoring backups - add dev-test-install-legacy.sh to reproduce the old layout for testing
Bun now ships native Windows ARM64 builds, so drop the hack that compiled an x64 bundle and relabelled it as arm64. - build-cli-bundles: build a native bun-windows-arm64 target (full list and the win32 branch); remove the pwsh SystemType detection + APIFY_BUNDLE_ARCH override and the per-target metadata reset - install.ps1: detect the arch from PROCESSOR_ARCHITECTURE; only x64 has baseline (non-AVX2) builds, so ARM64 never downloads a -baseline bundle - install.sh: map MINGW64 ARM64 to the windows-arm64 target - useCLIVersionAssets: Windows ARM64 has a native (non-baseline) bundle, so stop requiring -baseline assets there (otherwise upgrade finds no asset) - CI (check/pre_release/release): build on windows-11-arm with native bun via setup-bun (drop the manual install workaround) and a bun-target-arch matrix Unlike #1057, asset names keep the arm64 suffix (matching the build output, useCLIMetadata's process.arch, and the asset matcher) rather than renaming to aarch64, which in #1057 mismatches the produced -arm64 artifacts.
Per Bun's supported-targets matrix (https://bun.com/docs/bundler/executables#supported-targets), baseline/modern (SIMD) variants only exist for x64, and @types/bun's CompileTarget union only accepts the SIMD-then-libc order (`bun-linux-x64-baseline-musl`). - reorder `bun-linux-x64-musl-baseline` -> `bun-linux-x64-baseline-musl` so it matches the type (no cast needed); bun accepts either order at build time - drop `bun-linux-arm64-musl-baseline`: arm64 has no baseline variant, so it was a meaningless duplicate of the arm64 musl build and nothing ever requested it - parse the target's trailing modifiers order-independently and still emit the asset suffix in the canonical `-musl-baseline` order the install/upgrade matchers expect, so every published asset name is unchanged
Baseline (non-AVX2) builds are an x64-only concept, so bun-darwin-arm64-baseline and bun-linux-arm64-baseline just duplicated their non-baseline arm64 builds. The install/upgrade flow never requests them (baseline is only ever appended for x64), and GitHub download counts confirm no real usage: across all releases the arm64 baseline assets sit at the scraper-noise floor (~205 total, max 3/release) versus darwin-arm64 at 1557 (max 260) and linux-arm64 at 262 (max 15).
There was a problem hiding this comment.
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Cuz it doesn't exist anyways Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Make the bundle the single source of truth for the apify/actor wrapper scripts, and broaden Windows shell coverage (inspired by npm's cmd-shim, which we don't need: it locates a node interpreter for script bins, whereas we exec a native binary with an env var). - install command gains a hidden --shims-only flag and now (re)writes the wrapper scripts via writeEntrypointShims(); install.sh/install.ps1/upgrade.ps1 invoke the bundle instead of hand-rolling the shim text, so it lives in one place (TS) rather than being duplicated across sh + ps1 + ts - writeEntrypointShims now writes, on Windows, a .cmd (with setlocal so APIFY_CLI_ENTRYPOINT no longer leaks into the caller's shell), a native .ps1, and an extensionless POSIX sh shim so the command also resolves from Git Bash / MSYS2 / Cygwin (previously only .cmd existed -> bare `apify` failed there) - migration reuses writeEntrypointShims and isolates the Windows legacy-.exe cleanup as its own step - install/upgrade scripts skip the automatic version check when only creating shims (local, offline operation; we just downloaded the requested version)
l2ysho
left a comment
There was a problem hiding this comment.
LGTM. Just question if we want to ship this as a part of 1.6.2. or we rather wait? cc @patrikbraborec
|
Thanks, looks cool. I would wait, and not ship it as part of 1.6.2. |
|
1.6.2 out so we can merge this cc @vladfrangu |
The install.sh, install.ps1 and upgrade.ps1 changes are moved to a stacked follow-up PR for isolated review. They depend on this PR's bundle/build, the `install --shims-only` flag, and the upgrade.ts<->upgrade.ps1 URL contract, so they are reviewed on top of this branch rather than independently.
|
Update: #1219 now targets |
The merge with master brought in the OS-keyring build logic (#1197), whose `keyringSubpackage(os, arch, Boolean(musl))` call referenced the positional `musl` variable that the single-bundle refactor had replaced with the order-independent `isMusl`. That left build-cli-bundles.ts referencing an undefined name, breaking `pnpm run build-bundles`. Pass `isMusl` instead. Verified by running build-bundles: all targets compile and the keyring native subpackage resolves per target.
The --no-git-checks workaround (from #1221) was only needed because the manual Windows ARM bun install left a stray install.ps1 in the checkout. This branch installs bun via native setup-bun, so the working tree stays clean and the check can guard the version bump again.
| import { processVersionCheck, resolveEntrypoint, runCLI } from './_shared.js'; | ||
|
|
||
| // A single bundle now powers both the `apify` and `actor` CLIs. The wrapper scripts created during | ||
| // install set `APIFY_CLI_ENTRYPOINT` to pick which command set to expose (see `resolveEntrypoint`). |
There was a problem hiding this comment.
Nit: we should probably mention the var in docs/vars.md.
There was a problem hiding this comment.
No, this is an internal env var. Users shouldn't mess with it unless they know what they're doing
There was a problem hiding this comment.
Nit: having apify.ts and actor.ts "serving" apify, apify-cli and actor scripts, name of this one seems a bit missleading. Maybe bundle.ts would be more fitting as it indicates this is the main entrypoint for the bundle (and is dynamic).
There was a problem hiding this comment.
Renamed to bundle.ts in 7ab7739 — the published asset prefix stays apify-cli (the build script now sets it explicitly instead of deriving it from the entrypoint file name).
| } | ||
|
|
||
| const [_version, assetOs, assetArch, assetBaselineOrMusl, assetBaseline] = asset.name | ||
| .slice('apify-cli-'.length) |
There was a problem hiding this comment.
The requiresBaseline branch is a little too permissive, specifically its first clause:
return assetBaseline === 'baseline' || assetBaselineOrMusl === 'baseline';
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one
Since the split is positional, baseline only lands in assetBaseline (slot 4) when musl occupies slot 3, so assetBaseline === 'baseline' is effectively "this asset is -musl-baseline". And we only reach this branch when requiresMusl is false, so that clause can only ever let a wrong-libc asset through. Alpine machines get -musl-baseline from the requiresMusl branch above, so nothing needs it.
Effect: a glibc machine without AVX2 matches both -baseline and -musl-baseline, both download to the same apify-cli path, and the musl one wins alphabetically, which then can't start on glibc.
Pre-existing and effectively never hit, so not a blocker.
Fix: the build script in this PR already parses these suffixes into booleans, and mirroring it makes the match exact (it also tightens requiresMusl, which currently accepts both -musl and -musl-baseline) and collapses all four ifs into one expression:
const [_version, assetOs, assetArch, ...modifiers] = asset.name
.slice('apify-cli-'.length)
.replace(versionWithoutV, 'version')
.replace('.exe', '')
.split('-');
return (
assetOs === metadata.platform &&
assetArch === metadata.arch &&
modifiers.includes('musl') === requiresMusl &&
modifiers.includes('baseline') === requiresBaseline
);
There was a problem hiding this comment.
Fixed in bc69583 with exactly the suggested shape — the matcher now splits the trailing modifiers and requires an exact boolean match on both musl and baseline, mirroring the build script. Verified with a truth-table over all 9 published asset shapes: every (platform, arch, musl, baseline) combination now matches exactly one asset, including the glibc-no-AVX2 case.
DaveHanns
left a comment
There was a problem hiding this comment.
Great job 🚀. Sorry it took so long to review; I have used it to learn the mechanics of CLI bundling in depth.
Mirror the build script's modifier parsing when filtering release assets: split the trailing modifiers and require both musl and baseline to match exactly. The previous positional check let a glibc machine without AVX2 match -musl-baseline too (both downloading to the same path, with the musl one winning alphabetically and failing to start on glibc), and let musl machines match -musl-baseline when only -musl was wanted. Addresses review feedback on #1169.
The entrypoint serves both the apify and actor CLIs (picked via APIFY_CLI_ENTRYPOINT), so naming it after just one of the published names was misleading next to the former apify.ts/actor.ts. The published asset prefix stays apify-cli - the build script now sets it explicitly instead of deriving it from the entrypoint file name. Addresses review feedback on #1169.
What & why
Reworks how the CLI bundles are built, installed, and upgraded, and adds native Windows ARM64 support.
Previously the bundle install shipped three full ~70 MB binaries (
apify,actor,apify-cli) — the same binary dropped three times into the install dir. This builds oneapify-clibundle;apifyandactorbecome tiny wrapper scripts that invoke it withAPIFY_CLI_ENTRYPOINTset, which the entrypoint uses to pick the command set.Note
This supersedes #1057 (native Windows ARM bundles) — those changes are included here, adapted on top of the single-bundle work. See the "Windows ARM64" section for one deliberate deviation.
Single-bundle layout
build-cli-bundles.ts): build only theapify-clientrypoint; additionally publishapify-*/actor-*copies as backwards-compatible backups so installs using the old two-bundle upgrade flow can still pull the new bundle during the transition.apify-cli.tsentrypoint +resolveEntrypoint()(readsAPIFY_CLI_ENTRYPOINT, falls back to the executable's basename).install.sh/install.ps1): drop theapify-clibinary and generateapify/actor(.cmdon Windows) wrapper scripts instead of three binaries.bundleMigration.ts): on the first run after upgrading from a legacy 3-bundle install, copy the running bundle toapify-cliand replaceapify/actorwith wrapper scripts. Writes are atomic (temp + rename) so we never truncate a running executable — an in-place overwrite fails withETXTBSYon Linux (verified in a container; this matters because theactorCLI runs in Linux Actor images). On Windows the running.exeis renamed (can't be deleted) and cleaned up on a later run.upgrade.ts): download the singleapify-clibundle, write it + the wrappers atomically.Windows ARM64 (supersedes #1057)
Bun now ships native Windows ARM64 builds, so the old hack (compile an x64 bundle on the ARM runner and relabel it
arm64) is gone.bun-windows-arm64target; drop thepwshSystemType detection +APIFY_BUNDLE_ARCHoverride.install.ps1: detect arch fromPROCESSOR_ARCHITECTURE; ARM64 never downloads a-baselinebundle (baseline is x64/AVX2-only).useCLIVersionAssets: Windows ARM64 now matches the native (non-baseline) asset — a fix chore: native windows arm bundles #1057 lacks; without it the upgrade command finds zero assets on Windows ARM64.check/pre_release/release): build onwindows-11-armwith nativesetup-bun(dropped the manual install workaround) via abun-target-archmatrix var.Deviation from #1057: asset names keep the
arm64suffix (matching the produced artifacts,useCLIMetadata'sprocess.arch, and the asset matcher) rather than renaming toaarch64, which in #1057 mismatches the-arm64artifacts the build actually emits.Bundle target cleanup
as nevercasts on the bun compile targets. Per Bun's supported-targets matrix,@types/bun'sCompileTargetonly accepts the SIMD-then-libc order, sobun-linux-x64-musl-baseline→bun-linux-x64-baseline-musl(bun accepts either order at build time). The target parser now reads the trailing modifiers order-independently and still emits the canonical-musl-baselineasset suffix, so every published asset name is unchanged.bun-{darwin,linux}-arm64-baseline(and the arm64 musl-baseline): baseline is x64-only, so these just duplicated the non-baseline arm64 builds. They're unreachable via the install/upgrade path, and download counts confirm no real usage (~205 total / max 3 per release — scraper-noise floor — vsdarwin-arm64at 1557 / max 260).Testing
pnpm run build,oxlint,oxfmt, and the local test suite all pass.apify-cli+ wrapper scripts, idempotent,actorruns in actor mode.ETXTBSYon in-place overwrite vs success with atomic rename on Linux.bun-windows-arm64and the reorderedbun-linux-x64-baseline-musltargets and appends.exefor Windows.Added
dev-test-install-legacy.shto reproduce the old 3-binary layout locally for testing the migration.closes #1221