Repository navigation
Read zarr driver secrets from outside the checkout - #1120
Merged
Merged
Conversation
hallu-zarr.sh sourced .zarr-secrets.env from beside itself. Since #1109 moved the script into the nemar-cli checkout, that path resolves inside a git working tree, contradicting the comment's own stated intent that the token live in no repo. Default moves to STATE_DIR, alongside the queue DB and venv. The legacy path stays as a fallback so the existing deployment does not lose its token mid-cutover and silently skip every callback. ZARR_SECRETS_FILE overrides both. .gitignore covers the legacy path as a backstop. Verified all four precedence cases on the conversion node.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1120Version: 🤖 Updated on each push. |
Review of #1120 found the fallback claim was false. The beside-the-script path resolves against wherever the script currently is, so after the move into the clone it points inside the checkout, never at the old hand-copy directory holding the live token. It carries nothing across the cutover; relocating the token to STATE_DIR is a manual step. Comment corrected to say so rather than promise a safety property that does not exist. Two silent failures closed: - An absent token made both callback POSTs no-op, so conversion kept uploading while D1 zarr_status stopped advancing. Now warns; still non-fatal, since the viewer reads index.json from S3. - ZARR_SECRETS_FILE pointing at a missing path fell through to the legacy file, sourcing a different token than the operator named. Now fatal. Secrets load moves below log()/err() so a missing token can reach the log. Six precedence cases verified on the conversion node.
Re-review of #1120 found two problems with the first fix. The secrets load ran at top level, so --stats and --requeue became failable on a ZARR_SECRETS_FILE misconfiguration neither path uses. That contradicts the deliberate design keeping --requeue ahead of the flock and out of setup(): an operator reaching for it during an incident must not be blocked by unrelated config. It is now a function called after both early exits, just before the conversion loop. The missing-token warning also named the two default search paths unconditionally, so an explicit ZARR_SECRETS_FILE that existed but set nothing produced a message describing a search that never happened. That is the same defect as this PR's original comment, relocated into the runtime diagnostic an operator reads during an outage. It now reports which file was actually sourced. Verified on the node: --requeue succeeds with a bad ZARR_SECRETS_FILE, and all four warning branches name the right path.
6 of 13 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hallu-zarr.shsourced.zarr-secrets.envfrom beside itself, with a comment stating the token should live "neither in crontab nor in any repo." #1109 moved the script into the nemar-cli checkout, so "beside this script" is now a git working tree and the default contradicts its own stated intent. The file is onegit add -Aaway from being committed, and this repo has already had an unintended file committed twice (deno.lock, dropped in 745059e).Found while staging the Hallu cutover, not by review.
What changed
${STATE_DIR}/.zarr-secrets.env, outside the clone alongside the queue DB and venv, which is machine-local state git never touches.ZARR_SECRETS_FILEoverrides, exclusively: a path that does not exist is now fatal rather than falling through to a different file..gitignorecovers.zarr-secrets.envas a backstop.log()/err()so a missing token can reach the log file.REQUIRED manual deploy step
The fallback does not carry the token through the cutover. It resolves against wherever the script currently is, so once the script lives in the clone it points at
…/nemar-cli/scripts/zarr/.zarr-secrets.env, never at/data/projects/yahya/nemar/.zarr-secrets.envwhere the live token actually sits. The first revision of this PR claimed otherwise in both the comment and the commit message; review caught it and the comment now states the opposite explicitly.The token must be copied to
${STATE_DIR}/.zarr-secrets.envbefore the cron is pointed at the new checkout. Already done on Hallu (mode 0600, byte-identical to the live file), so this PR does not strand the deployment. Without it the failure is silent in the worst way: conversions succeed and upload to S3 while D1'szarr_statusstops advancing, visible only as a dashboard that quietly stopped moving. That is what the new warning exists to catch.Test plan
All six precedence cases exercised on the conversion node:
ZARR_SECRETS_FILEat an existing file beats bothZARR_SECRETS_FILEat a missing file exits 1 without falling backZARR_SECRETS_FILE=empty string treated as unset, normal precedenceshellcheck --severity=warningclean,bash -nclean.zarr-secrets.envinscripts/zarr/is ignored by gitPart of the epic #1108 deployment; no conversion behavior changes.