Skip to content

feat: single apify-cli bundle + native Windows ARM64 (supersedes #1057) - #1169

Open
vladfrangu wants to merge 14 commits into
masterfrom
feat/single-cli-bundle
Open

feat: single apify-cli bundle + native Windows ARM64 (supersedes #1057)#1169
vladfrangu wants to merge 14 commits into
masterfrom
feat/single-cli-bundle

Conversation

@vladfrangu

@vladfrangu vladfrangu commented May 29, 2026

Copy link
Copy Markdown
Member

What & why

Reworks how the CLI bundles are built, installed, and upgraded, and adds native Windows ARM64 support.

Previously the bundle install shipped three full ~70 MB binaries (apify, actor, apify-cli) — the same binary dropped three times into the install dir. This builds one apify-cli bundle; apify and actor become tiny wrapper scripts that invoke it with APIFY_CLI_ENTRYPOINT set, which the entrypoint uses to pick the command set.

Note

This supersedes #1057 (native Windows ARM bundles) — those changes are included here, adapted on top of the single-bundle work. See the "Windows ARM64" section for one deliberate deviation.

Single-bundle layout

  • Build (build-cli-bundles.ts): build only the apify-cli entrypoint; additionally publish apify-*/actor-* copies as backwards-compatible backups so installs using the old two-bundle upgrade flow can still pull the new bundle during the transition.
  • Entrypoint: new single apify-cli.ts entrypoint + resolveEntrypoint() (reads APIFY_CLI_ENTRYPOINT, falls back to the executable's basename).
  • Install (install.sh / install.ps1): drop the apify-cli binary and generate apify/actor (.cmd on Windows) wrapper scripts instead of three binaries.
  • Self-migration (bundleMigration.ts): on the first run after upgrading from a legacy 3-bundle install, copy the running bundle to apify-cli and replace apify/actor with wrapper scripts. Writes are atomic (temp + rename) so we never truncate a running executable — an in-place overwrite fails with ETXTBSY on Linux (verified in a container; this matters because the actor CLI runs in Linux Actor images). On Windows the running .exe is renamed (can't be deleted) and cleaned up on a later run.
  • Upgrade (upgrade.ts): download the single apify-cli bundle, write it + the wrappers atomically.

Windows ARM64 (supersedes #1057)

Bun now ships native Windows ARM64 builds, so the old hack (compile an x64 bundle on the ARM runner and relabel it arm64) is gone.

  • Build a native bun-windows-arm64 target; drop the pwsh SystemType detection + APIFY_BUNDLE_ARCH override.
  • install.ps1: detect arch from PROCESSOR_ARCHITECTURE; ARM64 never downloads a -baseline bundle (baseline is x64/AVX2-only).
  • useCLIVersionAssets: Windows ARM64 now matches the native (non-baseline) asset — a fix chore: native windows arm bundles #1057 lacks; without it the upgrade command finds zero assets on Windows ARM64.
  • CI (check/pre_release/release): build on windows-11-arm with native setup-bun (dropped the manual install workaround) via a bun-target-arch matrix var.

Deviation from #1057: asset names keep the arm64 suffix (matching the produced artifacts, useCLIMetadata's process.arch, and the asset matcher) rather than renaming to aarch64, which in #1057 mismatches the -arm64 artifacts the build actually emits.

Bundle target cleanup

  • Removed the as never casts on the bun compile targets. Per Bun's supported-targets matrix, @types/bun's CompileTarget only accepts the SIMD-then-libc order, so bun-linux-x64-musl-baselinebun-linux-x64-baseline-musl (bun accepts either order at build time). The target parser now reads the trailing modifiers order-independently and still emits the canonical -musl-baseline asset suffix, so every published asset name is unchanged.
  • Stopped building bun-{darwin,linux}-arm64-baseline (and the arm64 musl-baseline): baseline is x64-only, so these just duplicated the non-baseline arm64 builds. They're unreachable via the install/upgrade path, and download counts confirm no real usage (~205 total / max 3 per release — scraper-noise floor — vs darwin-arm64 at 1557 / max 260).

Testing

  • pnpm run build, oxlint, oxfmt, and the local test suite all pass.
  • Verified the full legacy→migrated flow end-to-end on Linux (arm64 container): 3 binaries → single apify-cli + wrapper scripts, idempotent, actor runs in actor mode.
  • Confirmed ETXTBSY on in-place overwrite vs success with atomic rename on Linux.
  • Confirmed Bun accepts bun-windows-arm64 and the reordered bun-linux-x64-baseline-musl targets and appends .exe for Windows.

Added dev-test-install-legacy.sh to reproduce the old 3-binary layout locally for testing the migration.

closes #1221

The CLI previously shipped three full bundles (apify, actor, apify-cli),
dropping the same ~70MB binary three times into the install directory.
Now we build one apify-cli bundle; apify and actor are tiny wrapper
scripts that invoke it with APIFY_CLI_ENTRYPOINT set, which the entrypoint
uses to pick the command set.

- build-cli-bundles: build only apify-cli; publish apify-*/actor-* copies
  as backwards-compatible backups for the old upgrade flow
- entrypoints: new apify-cli entrypoint + resolveEntrypoint()
- install.sh / install.ps1 / dev-test-install.sh: drop the apify-cli binary
  and generate apify/actor (.cmd on Windows) wrapper scripts
- upgrade: download the single apify-cli bundle; write it and the wrappers
  atomically (temp + rename) to avoid ETXTBSY on a running binary (Linux)
- bundleMigration: on first run after upgrading from a legacy 3-bundle
  install, copy the running bundle to apify-cli and replace apify/actor
  with wrappers (rename-not-delete on Windows for the running exe)
- useCLIVersionAssets: match only apify-cli-* assets, ignoring backups
- add dev-test-install-legacy.sh to reproduce the old layout for testing
Bun now ships native Windows ARM64 builds, so drop the hack that compiled
an x64 bundle and relabelled it as arm64.

- build-cli-bundles: build a native bun-windows-arm64 target (full list and
  the win32 branch); remove the pwsh SystemType detection + APIFY_BUNDLE_ARCH
  override and the per-target metadata reset
- install.ps1: detect the arch from PROCESSOR_ARCHITECTURE; only x64 has
  baseline (non-AVX2) builds, so ARM64 never downloads a -baseline bundle
- install.sh: map MINGW64 ARM64 to the windows-arm64 target
- useCLIVersionAssets: Windows ARM64 has a native (non-baseline) bundle, so
  stop requiring -baseline assets there (otherwise upgrade finds no asset)
- CI (check/pre_release/release): build on windows-11-arm with native bun via
  setup-bun (drop the manual install workaround) and a bun-target-arch matrix

Unlike #1057, asset names keep the arm64 suffix (matching the build output,
useCLIMetadata's process.arch, and the asset matcher) rather than renaming to
aarch64, which in #1057 mismatches the produced -arm64 artifacts.
Per Bun's supported-targets matrix (https://bun.com/docs/bundler/executables#supported-targets),
baseline/modern (SIMD) variants only exist for x64, and @types/bun's CompileTarget
union only accepts the SIMD-then-libc order (`bun-linux-x64-baseline-musl`).

- reorder `bun-linux-x64-musl-baseline` -> `bun-linux-x64-baseline-musl` so it
  matches the type (no cast needed); bun accepts either order at build time
- drop `bun-linux-arm64-musl-baseline`: arm64 has no baseline variant, so it was a
  meaningless duplicate of the arm64 musl build and nothing ever requested it
- parse the target's trailing modifiers order-independently and still emit the
  asset suffix in the canonical `-musl-baseline` order the install/upgrade
  matchers expect, so every published asset name is unchanged
Baseline (non-AVX2) builds are an x64-only concept, so bun-darwin-arm64-baseline
and bun-linux-arm64-baseline just duplicated their non-baseline arm64 builds. The
install/upgrade flow never requests them (baseline is only ever appended for x64),
and GitHub download counts confirm no real usage: across all releases the arm64
baseline assets sit at the scraper-noise floor (~205 total, max 3/release) versus
darwin-arm64 at 1557 (max 260) and linux-arm64 at 262 (max 15).
@github-actions github-actions Bot added this to the 141st sprint - Tooling team milestone May 29, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label May 29, 2026
@vladfrangu
vladfrangu requested review from Copilot and l2ysho and removed request for Copilot May 29, 2026 14:30
@vladfrangu vladfrangu added the adhoc Ad-hoc unplanned task added during the sprint. label May 29, 2026
@vladfrangu
vladfrangu requested a review from Copilot May 29, 2026 14:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/check.yaml Outdated
Comment thread .github/workflows/pre_release.yaml Outdated
Comment thread .github/workflows/release.yaml Outdated
vladfrangu and others added 2 commits May 30, 2026 23:15
Cuz it doesn't exist anyways

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Make the bundle the single source of truth for the apify/actor wrapper scripts,
and broaden Windows shell coverage (inspired by npm's cmd-shim, which we don't
need: it locates a node interpreter for script bins, whereas we exec a native
binary with an env var).

- install command gains a hidden --shims-only flag and now (re)writes the
  wrapper scripts via writeEntrypointShims(); install.sh/install.ps1/upgrade.ps1
  invoke the bundle instead of hand-rolling the shim text, so it lives in one
  place (TS) rather than being duplicated across sh + ps1 + ts
- writeEntrypointShims now writes, on Windows, a .cmd (with setlocal so
  APIFY_CLI_ENTRYPOINT no longer leaks into the caller's shell), a native .ps1,
  and an extensionless POSIX sh shim so the command also resolves from Git Bash
  / MSYS2 / Cygwin (previously only .cmd existed -> bare `apify` failed there)
- migration reuses writeEntrypointShims and isolates the Windows legacy-.exe
  cleanup as its own step
- install/upgrade scripts skip the automatic version check when only creating
  shims (local, offline operation; we just downloaded the requested version)

@l2ysho l2ysho left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just question if we want to ship this as a part of 1.6.2. or we rather wait? cc @patrikbraborec

@patrikbraborec

Copy link
Copy Markdown
Contributor

Thanks, looks cool. I would wait, and not ship it as part of 1.6.2.

@l2ysho

l2ysho commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

1.6.2 out so we can merge this cc @vladfrangu

The install.sh, install.ps1 and upgrade.ps1 changes are moved to a stacked
follow-up PR for isolated review. They depend on this PR's bundle/build, the
`install --shims-only` flag, and the upgrade.ts<->upgrade.ps1 URL contract, so
they are reviewed on top of this branch rather than independently.
@vladfrangu

Copy link
Copy Markdown
Member Author

The public install/upgrade scripts (install.sh, install.ps1, upgrade.ps1) have been split out into #1219, stacked on this branch for isolated review. This PR no longer touches those three files; merge this one first, then #1219.

@vladfrangu

Copy link
Copy Markdown
Member Author

Update: #1219 now targets master directly (no longer stacked on this branch), per the plan to merge it once a release containing this PR's apify-cli bundle is out. This PR still excludes those three install scripts.

The merge with master brought in the OS-keyring build logic (#1197), whose
`keyringSubpackage(os, arch, Boolean(musl))` call referenced the positional
`musl` variable that the single-bundle refactor had replaced with the
order-independent `isMusl`. That left build-cli-bundles.ts referencing an
undefined name, breaking `pnpm run build-bundles`. Pass `isMusl` instead.

Verified by running build-bundles: all targets compile and the keyring native
subpackage resolves per target.
l2ysho added 2 commits July 1, 2026 17:18
The --no-git-checks workaround (from #1221) was only needed because the
manual Windows ARM bun install left a stray install.ps1 in the checkout.
This branch installs bun via native setup-bun, so the working tree stays
clean and the check can guard the version bump again.
Comment thread src/entrypoints/bundle.ts
import { processVersionCheck, resolveEntrypoint, runCLI } from './_shared.js';

// A single bundle now powers both the `apify` and `actor` CLIs. The wrapper scripts created during
// install set `APIFY_CLI_ENTRYPOINT` to pick which command set to expose (see `resolveEntrypoint`).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we should probably mention the var in docs/vars.md.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is an internal env var. Users shouldn't mess with it unless they know what they're doing

Comment thread src/entrypoints/bundle.ts

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: having apify.ts and actor.ts "serving" apify, apify-cli and actor scripts, name of this one seems a bit missleading. Maybe bundle.ts would be more fitting as it indicates this is the main entrypoint for the bundle (and is dynamic).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to bundle.ts in 7ab7739 — the published asset prefix stays apify-cli (the build script now sets it explicitly instead of deriving it from the entrypoint file name).

}

const [_version, assetOs, assetArch, assetBaselineOrMusl, assetBaseline] = asset.name
.slice('apify-cli-'.length)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The requiresBaseline branch is a little too permissive, specifically its first clause:

return assetBaseline === 'baseline' || assetBaselineOrMusl === 'baseline';
//     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one

Since the split is positional, baseline only lands in assetBaseline (slot 4) when musl occupies slot 3, so assetBaseline === 'baseline' is effectively "this asset is -musl-baseline". And we only reach this branch when requiresMusl is false, so that clause can only ever let a wrong-libc asset through. Alpine machines get -musl-baseline from the requiresMusl branch above, so nothing needs it.

Effect: a glibc machine without AVX2 matches both -baseline and -musl-baseline, both download to the same apify-cli path, and the musl one wins alphabetically, which then can't start on glibc.

Pre-existing and effectively never hit, so not a blocker.

Fix: the build script in this PR already parses these suffixes into booleans, and mirroring it makes the match exact (it also tightens requiresMusl, which currently accepts both -musl and -musl-baseline) and collapses all four ifs into one expression:

const [_version, assetOs, assetArch, ...modifiers] = asset.name
    .slice('apify-cli-'.length)
    .replace(versionWithoutV, 'version')
    .replace('.exe', '')
    .split('-');

return (
    assetOs === metadata.platform &&
    assetArch === metadata.arch &&
    modifiers.includes('musl') === requiresMusl &&
    modifiers.includes('baseline') === requiresBaseline
);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bc69583 with exactly the suggested shape — the matcher now splits the trailing modifiers and requires an exact boolean match on both musl and baseline, mirroring the build script. Verified with a truth-table over all 9 published asset shapes: every (platform, arch, musl, baseline) combination now matches exactly one asset, including the glibc-no-AVX2 case.

@DaveHanns DaveHanns left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job 🚀. Sorry it took so long to review; I have used it to learn the mechanics of CLI bundling in depth.

Mirror the build script's modifier parsing when filtering release assets: split
the trailing modifiers and require both musl and baseline to match exactly. The
previous positional check let a glibc machine without AVX2 match -musl-baseline
too (both downloading to the same path, with the musl one winning alphabetically
and failing to start on glibc), and let musl machines match -musl-baseline when
only -musl was wanted.

Addresses review feedback on #1169.
The entrypoint serves both the apify and actor CLIs (picked via
APIFY_CLI_ENTRYPOINT), so naming it after just one of the published names was
misleading next to the former apify.ts/actor.ts. The published asset prefix
stays apify-cli - the build script now sets it explicitly instead of deriving
it from the entrypoint file name.

Addresses review feedback on #1169.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pnpm version fails on Windows ARM bundle build — stray install.ps1 dirties working tree

6 participants