Skip to content

Read zarr driver secrets from outside the checkout - #1120

Merged
neuromechanist merged 3 commits into
devfrom
fix/zarr-secrets-outside-checkout
Aug 22, 2026
Merged

neuromechanist merged 3 commits into
devfrom
fix/zarr-secrets-outside-checkout

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

hallu-zarr.sh sourced .zarr-secrets.env from beside itself, with a comment stating the token should live "neither in crontab nor in any repo." #1109 moved the script into the nemar-cli checkout, so "beside this script" is now a git working tree and the default contradicts its own stated intent. The file is one git add -A away from being committed, and this repo has already had an unintended file committed twice (deno.lock, dropped in 745059e).

Found while staging the Hallu cutover, not by review.

What changed

  • Default is ${STATE_DIR}/.zarr-secrets.env, outside the clone alongside the queue DB and venv, which is machine-local state git never touches.
  • ZARR_SECRETS_FILE overrides, exclusively: a path that does not exist is now fatal rather than falling through to a different file.
  • An absent token now warns loudly instead of silently disabling every callback.
  • The beside-the-script path stays as a fallback for the deployment shape that keeps script and secrets together outside a checkout.
  • .gitignore covers .zarr-secrets.env as a backstop.
  • The secrets load moved below log()/err() so a missing token can reach the log file.

REQUIRED manual deploy step

The fallback does not carry the token through the cutover. It resolves against wherever the script currently is, so once the script lives in the clone it points at …/nemar-cli/scripts/zarr/.zarr-secrets.env, never at /data/projects/yahya/nemar/.zarr-secrets.env where the live token actually sits. The first revision of this PR claimed otherwise in both the comment and the commit message; review caught it and the comment now states the opposite explicitly.

The token must be copied to ${STATE_DIR}/.zarr-secrets.env before the cron is pointed at the new checkout. Already done on Hallu (mode 0600, byte-identical to the live file), so this PR does not strand the deployment. Without it the failure is silent in the worst way: conversions succeed and upload to S3 while D1's zarr_status stops advancing, visible only as a dashboard that quietly stopped moving. That is what the new warning exists to catch.

Test plan

All six precedence cases exercised on the conversion node:

  • both files present, STATE_DIR wins
  • legacy path only, falls back
  • neither present, empty token, exit 0, and a loud warning naming both paths searched
  • ZARR_SECRETS_FILE at an existing file beats both
  • ZARR_SECRETS_FILE at a missing file exits 1 without falling back
  • ZARR_SECRETS_FILE= empty string treated as unset, normal precedence
  • shellcheck --severity=warning clean, bash -n clean
  • a .zarr-secrets.env in scripts/zarr/ is ignored by git

Part of the epic #1108 deployment; no conversion behavior changes.

hallu-zarr.sh sourced .zarr-secrets.env from beside itself. Since #1109
moved the script into the nemar-cli checkout, that path resolves inside a
git working tree, contradicting the comment's own stated intent that the
token live in no repo.

Default moves to STATE_DIR, alongside the queue DB and venv. The legacy
path stays as a fallback so the existing deployment does not lose its
token mid-cutover and silently skip every callback. ZARR_SECRETS_FILE
overrides both. .gitignore covers the legacy path as a backstop.

Verified all four precedence cases on the conversion node.
@github-actions

github-actions Bot commented Aug 22, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1120

Version: 0.9.9-PR1120.db2dedf
Commit: db2dedf


🤖 Updated on each push.

Review of #1120 found the fallback claim was false. The beside-the-script
path resolves against wherever the script currently is, so after the move
into the clone it points inside the checkout, never at the old hand-copy
directory holding the live token. It carries nothing across the cutover;
relocating the token to STATE_DIR is a manual step. Comment corrected to
say so rather than promise a safety property that does not exist.

Two silent failures closed:

- An absent token made both callback POSTs no-op, so conversion kept
  uploading while D1 zarr_status stopped advancing. Now warns; still
  non-fatal, since the viewer reads index.json from S3.
- ZARR_SECRETS_FILE pointing at a missing path fell through to the legacy
  file, sourcing a different token than the operator named. Now fatal.

Secrets load moves below log()/err() so a missing token can reach the log.

Six precedence cases verified on the conversion node.
Re-review of #1120 found two problems with the first fix.

The secrets load ran at top level, so --stats and --requeue became
failable on a ZARR_SECRETS_FILE misconfiguration neither path uses. That
contradicts the deliberate design keeping --requeue ahead of the flock
and out of setup(): an operator reaching for it during an incident must
not be blocked by unrelated config. It is now a function called after
both early exits, just before the conversion loop.

The missing-token warning also named the two default search paths
unconditionally, so an explicit ZARR_SECRETS_FILE that existed but set
nothing produced a message describing a search that never happened. That
is the same defect as this PR's original comment, relocated into the
runtime diagnostic an operator reads during an outage. It now reports
which file was actually sourced.

Verified on the node: --requeue succeeds with a bad ZARR_SECRETS_FILE,
and all four warning branches name the right path.
@neuromechanist
neuromechanist merged commit c2a9b7f into dev Aug 22, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the fix/zarr-secrets-outside-checkout branch August 22, 2026 16:44

This branch was previously deployed

1 inactive deployment
npm — db2dedfb Deployed Aug 22, 2026 by neuromechanist via publish #2034
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant