Skip to content

Harden the Hallu self-deploy: one merged commit is code execution on the host #1379

Open

Description

@neuromechanist

Found while evaluating harm for epic #1336 phase 1 (#1339). Not part of that epic: reclassifying
the documentation raises the cost of finding this, where this issue fixes the substance.

The path

scripts/zarr/hallu-zarr.sh's setup() runs git fetch && git reset --hard origin/$ZARR_DRIVER_REF
(default main) against the public nemarOrg/nemar-cli, every run, hourly at :30. Since #1109
that clone supplies the shell script as well as the Python driver, so the reset redeploys the script
that performs the reset.

Consequence: anything that lands on main executes on the SDSC Hallu host within the hour, with
whatever that account can reach, including the AWS profile used to write the production S3 zarr
prefix.

How bad, honestly

Not drive-by exploitable. The keep-main and keep-dev rulesets are active, so this needs a merged
commit rather than a PR, which means a compromised or coerced maintainer account, or a review miss on
a change to scripts/zarr/. It is a supply-chain path, not an open door.

But it is the highest-value single mechanism in the system: it converts one merged commit into code
execution on a host with production credentials, and it is fully documented publicly. The self-deploy
property was a deliberate and good fix for a real problem (#1129: a node sat two deploys behind for
two days because a long drain never re-ran setup()). This issue is not about undoing it.

Options

  1. Pin to a signed tag rather than a branch ref. Deploy becomes an explicit act, and an
    unreviewed commit on main no longer ships. Costs the automatic-freshness property Long drains freeze the deployed Zarr driver: setup() never re-runs #1129 fixed,
    unless tagging is folded into the release flow.
  2. Verify commit signatures before reset --hard, refusing an unsigned or unknown-signer head.
    Keeps branch-following, adds a real gate. Needs a key policy and a documented recovery when the
    check fails at 03:30.
  3. Allowlist the paths the reset may change. Cheapest, weakest: it narrows blast radius without
    removing the mechanism.
  4. Split the deploy from the drain. The script re-checks origin/$DRIVER_REF between datasets
    and stops when it moves (Long drains freeze the deployed Zarr driver: setup() never re-runs #1129). A deliberate deploy step could reuse that.

Preference: 1 or 2, and folding the tag into the existing release flow so freshness survives.

Constraint

Hallu is unavailable until 2026-09-14. Nothing here can be verified on the host before then, and
verification is the whole job: the equivalent lesson from phase 0 was that a gate can look present
locally and protect nothing in production. Do not land a change to setup() without a run on the
node.

Related: ADR 0029 (the converter runs on the cron host, not Actions), #1109, #1129.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendBackend/API related

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions