Summary
scripts/publish_spaces.sh rewrites releases/SHA256SUMS from scratch on every publish, so the bucket only ever holds checksums for the most recent build. The versioned tarballs it describes are immutable and stay served indefinitely — but their checksums do not. The moment a new nightly lands, every previously published artifact becomes unverifiable through the canonical manifest while remaining perfectly downloadable.
That makes it impossible to pin a WFL version and verify it, which is the combination anyone deploying WFL actually needs.
Where it happens
: > "$WORK/SHA256SUMS" # fresh file every run
...
( cd "$(dirname "$TARBALL")" && sha256sum "$(basename "$TARBALL")" ) >> "$WORK/SHA256SUMS"
...
put "$WORK/SHA256SUMS" "releases/SHA256SUMS" text/plain "$ROLLING" # overwrites
The header comment is explicit that this is intentional — releases/SHA256SUMS checksums for this publish. The issue is that the file's name and location say "manifest for the releases directory", and that is how consumers reasonably read it.
Observed impact
Downstream CI (a GitLab project deploying a WFL service) pinned 26.7.57 / 2d74737 and verified downloads against releases/SHA256SUMS. It was green for two days. When 26.7.59 published at 2026-07-30 09:22 UTC, three consecutive pipelines went red with:
FAILED: sha256 00432aac1517d80526223ae318d261e4ba4e92f7ab5528ba964277e4213ea042 is not in the published SHA256SUMS
published entries:
c2b799f2... wfl-26.7.59-linux-x86_64-579eb80.tar.gz
ecf8b6d9... wfl-26.7.59.msi
ecf97d19... vscode-wfl-26.7.59.vsix
Current bucket state confirms both halves — the artifact survives, the attestation does not:
$ curl -sI .../releases/wfl-26.7.57-linux-x86_64-2d74737.tar.gz
HTTP/2 200
content-length: 11792317
last-modified: Wed, 29 Jul 2026 10:50:53 GMT
$ curl -s .../releases/SHA256SUMS
c2b799f2... wfl-26.7.59-linux-x86_64-579eb80.tar.gz
ecf8b6d9... wfl-26.7.59.msi
ecf97d19... vscode-wfl-26.7.59.vsix
Nothing was wrong with the pinned build. It just stopped being provable.
The failure mode is also confusing in a specific way: it looks like an integrity violation. The error a consumer sees is "this checksum is not published", which reads as a tampered or corrupted artifact, when in fact the download was fine and the manifest simply moved on.
Why the obvious workarounds are unsatisfying
- Always track
latest — gives up reproducibility, and the project already documents that nightlies genuinely break (26.7.52 shipped typechecker errors, 26.7.53 shipped a startup hang). Pinning exists for good reason.
- Skip verification for pinned versions — trades a broken build for an unverified binary.
- Snapshot
SHA256SUMS at pin time — this works, and is what we did (below), but it means the canonical manifest is only usable during a window whose length nobody controls.
Suggested fix
Either would resolve it; the first is smaller and fully backward compatible.
1. Publish a per-release checksum file alongside each immutable artifact.
releases/wfl-<version>-linux-x86_64-<sha>.tar.gz.sha256
releases/wfl-<version>.msi.sha256
Immutable, cache alongside the artifact, never rewritten. releases/SHA256SUMS keeps its current meaning for anyone tracking latest.
2. Make SHA256SUMS cumulative — read the existing object, merge, re-upload. Fixes existing consumers with no client change, but it makes a rolling object grow without bound and introduces a read-modify-write on a file two concurrent publishes could race on.
If neither is wanted, it would help to rename the object to SHA256SUMS.latest (or document it prominently in the install docs). The current name invites exactly the assumption that broke here.
What we did downstream
Recorded the expected hash in our own repo and verify against that, falling back to SHA256SUMS only for the rolling latest canary where there is nothing stable to pin. That is arguably the correct trust model regardless — SHA256SUMS is mutable and served from the same host as the artifact, so it demonstrates transport integrity rather than authenticity. But it does mean the checksum has to be captured during the window in which that build is still the newest one, which is a sharp edge worth removing for everyone else.
Happy to open a PR for option 1 if that is the direction you'd prefer.
Bucket: wfl.nyc3.cdn.digitaloceanspaces.com · script added in #656 · publisher github-actions/blacksmith
Summary
scripts/publish_spaces.shrewritesreleases/SHA256SUMSfrom scratch on every publish, so the bucket only ever holds checksums for the most recent build. The versioned tarballs it describes are immutable and stay served indefinitely — but their checksums do not. The moment a new nightly lands, every previously published artifact becomes unverifiable through the canonical manifest while remaining perfectly downloadable.That makes it impossible to pin a WFL version and verify it, which is the combination anyone deploying WFL actually needs.
Where it happens
The header comment is explicit that this is intentional —
releases/SHA256SUMS checksums for this publish. The issue is that the file's name and location say "manifest for the releases directory", and that is how consumers reasonably read it.Observed impact
Downstream CI (a GitLab project deploying a WFL service) pinned
26.7.57 / 2d74737and verified downloads againstreleases/SHA256SUMS. It was green for two days. When 26.7.59 published at 2026-07-30 09:22 UTC, three consecutive pipelines went red with:Current bucket state confirms both halves — the artifact survives, the attestation does not:
Nothing was wrong with the pinned build. It just stopped being provable.
The failure mode is also confusing in a specific way: it looks like an integrity violation. The error a consumer sees is "this checksum is not published", which reads as a tampered or corrupted artifact, when in fact the download was fine and the manifest simply moved on.
Why the obvious workarounds are unsatisfying
latest— gives up reproducibility, and the project already documents that nightlies genuinely break (26.7.52 shipped typechecker errors, 26.7.53 shipped a startup hang). Pinning exists for good reason.SHA256SUMSat pin time — this works, and is what we did (below), but it means the canonical manifest is only usable during a window whose length nobody controls.Suggested fix
Either would resolve it; the first is smaller and fully backward compatible.
1. Publish a per-release checksum file alongside each immutable artifact.
Immutable, cache alongside the artifact, never rewritten.
releases/SHA256SUMSkeeps its current meaning for anyone trackinglatest.2. Make
SHA256SUMScumulative — read the existing object, merge, re-upload. Fixes existing consumers with no client change, but it makes a rolling object grow without bound and introduces a read-modify-write on a file two concurrent publishes could race on.If neither is wanted, it would help to rename the object to
SHA256SUMS.latest(or document it prominently in the install docs). The current name invites exactly the assumption that broke here.What we did downstream
Recorded the expected hash in our own repo and verify against that, falling back to
SHA256SUMSonly for the rollinglatestcanary where there is nothing stable to pin. That is arguably the correct trust model regardless —SHA256SUMSis mutable and served from the same host as the artifact, so it demonstrates transport integrity rather than authenticity. But it does mean the checksum has to be captured during the window in which that build is still the newest one, which is a sharp edge worth removing for everyone else.Happy to open a PR for option 1 if that is the direction you'd prefer.
Bucket:
wfl.nyc3.cdn.digitaloceanspaces.com· script added in #656 · publishergithub-actions/blacksmith