Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .context/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ Architecture Decision Records (ADRs) capture significant decisions that shape th
- `0000-template.md` is the template; copy it to start a new ADR. Do not edit `0000-template.md` itself.
- Number sequentially. The next ADR after `0007-...` is `0008-...`.
- Status flows `proposed` -> `accepted` -> (later) `superseded by ADR-NNNN`. Never delete an ADR; supersede it.
- An `Amendment YYYY-MM-DD:` block inside an accepted ADR is for a CLARIFICATION that leaves the
verdict standing -- naming a mechanism the decision always implied, correcting a stale file path,
recording what the decision turned out to cost. Anything that changes what was decided is a new
ADR that supersedes this one, not an amendment: an amended verdict is invisible to anyone who
read the ADR before, and the whole point of the `Status` line is that the verdict is findable.
- Keep each ADR short. If it grows past two screens, you are probably writing a design doc, not a decision.

## When to write an ADR
Expand Down
29 changes: 23 additions & 6 deletions .context/systems-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,8 +280,13 @@ and `api-test.nemar.org/datasets/xx099905`'s

**Credentials, resolved by `--test` only when the variable is otherwise unset**
(an explicit env var still wins,
and `--test` refuses six prod values outright, normalized and case-insensitively —
see the guard rails in `hallu-zarr.sh`):
and `--test` refuses six prod values outright — see the guard rails in `hallu-zarr.sh`).
Each of the six is normalized for its own kind of value, and only two are case-folded:
`API_BASE` and `TEST_API_URL` go through `_is_prod_api_host`,
which lowercases and strips scheme, path, query, port and a trailing DNS dot;
`STATE_DIR` and `WORK_DIR` are path-normalized (repeated and trailing `/` collapsed,
`realpath` when the directory already exists) but NOT case-folded;
`S3_BUCKET` and `AWS_PROFILE` are plain exact string equality:

| Variable | `--test` default | Notes |
|---|---|---|
Expand All @@ -291,7 +296,7 @@ see the guard rails in `hallu-zarr.sh`):
| `ZARR_AWS_PROFILE` | `nemar-zarr-dev` | IAM user `nemar-hallu-zarr-dev`, `s3:Get/Put/Delete` on `nemar-dev/*/zarr/*` + `GetObject` on `nemar-dev/*/objects/*` + `ListBucket`; cannot write to the prod bucket — per the IAM policy as provisioned 2026-09-02 (inline policy `zarr-rw-dev` on user `nemar-hallu-zarr-dev`, no IaC in this repo) |
| `ZARR_STATE_DIR` | `${ZARR_BASE:-/mnt/local}/zarr-state-test` | |
| `ZARR_WORK_DIR` | `${ZARR_BASE:-/mnt/local}/zarr-scratch-test` | |
| `ZARR_DRIVER_REF` | `dev` | tracks the same unreleased branch the rest of staging tracks |
| `ZARR_DRIVER_REF` | `dev` | tracks the same unreleased branch the rest of staging tracks — but the installed cron line below overrides it with the epic branch while epic #1181 is open |
| `ZARR_JOBS` | `4` | deliberately low — a test instance shares Hallu's cores with the prod backfill and must not contend with it |

**Ops sanity check, no side effects:**
Expand Down Expand Up @@ -327,16 +332,28 @@ not the hand-placed bootstrap copy,
so `setup()`'s self-deploy keeps it current
— identical reasoning to §3.3's driver/script split.

**Cron (documented here; not installed by this phase — see the PR for what remains for ops):**
**Cron — INSTALLED. The line on Hallu, verbatim:**

```
15 3 * * * /mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh --test >> /mnt/local/zarr-state-test/.nm-zarr-cron.log 2>&1
15 3 * * * mkdir -p /mnt/local/zarr-state-test && ZARR_DRIVER_REF=feature/issue-1181-epic-zarr-serving /mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh --test >> /mnt/local/zarr-state-test/.nm-zarr-cron.log 2>&1
```

Nightly and off the prod cron's `:30` hourly tick
Nightly at 03:15 UTC and off the prod cron's `:30` hourly tick
— deliberately infrequent, since the test catalog is small
and the point is observability, not throughput.

Two parts of that line are not decoration:

- `mkdir -p` runs first so the redirect target's directory exists on a node where
the state dir has been wiped; `>>` would otherwise fail before the script ran,
and cron's only trace of it is mail nobody reads.
- **`ZARR_DRIVER_REF` is set EXPLICITLY, overriding `--test`'s `dev` default**,
because staging's job while epic #1181 is open is to prove the epic branch
before `dev` has it. **Flip it to `dev` (or drop the override) once the epic
merges** — left pointing at the epic branch it would pin staging to a ref that
stops moving, and `setup()`'s self-deploy would keep resetting the clone to a
stale commit while looking perfectly healthy.

**What is NOT shared with prod:**
state directory, lock, queue db, venv, driver clone,
`ZARR_JOBS` (capped to protect prod's cores), S3 bucket, IAM profile, API base,
Expand Down
16 changes: 13 additions & 3 deletions .context/validated_workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -1115,7 +1115,7 @@ curl -s -o /dev/null -w '%{http_code}\n' https://api.nemar.org/datasets/xx099905
6. **After the first real run, invoke the CLONE's copy, not the bootstrap copy.**
Step 1 above hand-places the script at `/mnt/local/zarr-state-test/hallu-zarr.sh`
only to get `setup()` running for the first time;
every run after that (cron included, once installed)
every run after that (the installed nightly cron included)
should invoke `/mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh` instead,
the same self-deploying clone copy the prod cron already uses
(see systems-inventory.md §3.3's "Deploying a converter change")
Expand Down Expand Up @@ -1181,7 +1181,11 @@ Merging this into the epic branch is enough to exercise the whole bump on the

- Production tracks `main` (`ZARR_DRIVER_REF=main`), so it does not run this code
until the release.
- The staging instance tracks the epic branch nightly and its queue holds only the
- The staging instance tracks the epic branch nightly — the installed crontab line
is `15 3 * * * mkdir -p /mnt/local/zarr-state-test && ZARR_DRIVER_REF=feature/issue-1181-epic-zarr-serving /mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh --test >> /mnt/local/zarr-state-test/.nm-zarr-cron.log 2>&1`,
which overrides `--test`'s own `dev` default and must be flipped back to `dev`
once the epic merges (systems-inventory.md §3.4 holds the same line and says why)
— and its queue holds only the
7 curated `xx0999NN` exemplars. That is under `ENGINE_REQUEUE_LIMIT` (25), so
`reconcile` requeues them WITHOUT an ack and the fleet re-converts through
engine 3 on its own -- eeg / ieeg / emg / meg / multi-modal / HED coverage,
Expand Down Expand Up @@ -1244,7 +1248,13 @@ curl -s https://zarr-test.nemar.org/xx099905/zarr/index.json |
# It re-queues nothing yet -- the guard below holds it, because ~800 rows is
# far over ENGINE_REQUEUE_LIMIT.

# 3. Preview the cost before paying it. Read-only, no network, no writes.
# 3. Preview the cost before paying it. No network, and it changes no job's
# STATE -- it only counts what a bump would re-queue. Not literally
# read-only: zarr_queue.py's main() connects for every subcommand, and
# connect runs migrate_schema, whose additive ALTER and engine_version seed
# are idempotent and re-run on every connect. On a node that has never run
# the current queue schema the preview writes those, exactly as the next
# reconcile would. Nothing is enqueued, parked, or reset.
ssh hallu '/mnt/local/zarr-state/nemar-cli/scripts/zarr/hallu-zarr.sh --preview-engine-bump'
# Expect: engine-preview: current=3 ... stale=<~800>, i.e. the whole catalogue.

Expand Down
72 changes: 62 additions & 10 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ jobs:
tests: ${{ steps.filter.outputs.tests }}
email: ${{ steps.filter.outputs.email }}
zarr: ${{ steps.filter.outputs.zarr }}
shared: ${{ steps.filter.outputs.shared }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
Expand All @@ -48,6 +49,14 @@ jobs:
- 'backend/**'
tests:
- 'test/**'
# shared/ is imported by BOTH src/ and backend/ but matched by
# neither filter, so a PR touching only shared/** -- the wire
# contract, the facet vocabulary, the two published Zarr JSON
# Schemas -- ran zero test jobs and merged green having been
# compiled by nothing. Wired into lint, unit-pure and the
# backend-facing jobs below.
shared:
- 'shared/**'
# Email + passwordless auth surface (#569 / #572 / #595).
# Gates the passwordless-auth-test job below — that suite
# triggers real Resend API calls and burns the daily quota,
Expand Down Expand Up @@ -161,11 +170,18 @@ jobs:
- name: Hallu zarr --test config tests
run: cd scripts/zarr && uv run --with pytest pytest test_hallu_zarr_config.py -v

# Tier-2 duration backfill (scripts/patch_duration.py), moved here from
# nemarDatasets/.github with the converter it imports (#1109). Unlike the
# zarr helper tests above this one needs REAL dependencies: it generates an
# EDF with pyedflib and reads it back through biosigIO, and self-skips
# without them -- so the install is what makes this job worth running.
# The FULL converter suite, plus the tier-2 duration backfill
# (scripts/patch_duration.py) it was originally created for (#1109).
#
# The job name is kept for the required-check configuration; its scope is
# deliberately wider than the name. `zarr-python-test` above is the fast
# pure-Python tier and installs only jsonschema + pyarrow, so five test
# classes -- 43 tests covering real-recording conversion, events.parquet
# publication end to end, convert_one, and the sample-index rate-ratio
# cases -- raised SkipTest on EVERY CI run and were reported as
# "skipped=9" in a green job. This job already installs the real stack
# (biosigIO + pyedflib) for the duration tests, so running the converter
# suite here is what actually executes them.
patch-duration-test:
runs-on: ubuntu-latest
# The pool-recovery tests drive real subprocesses; if the executor ever
Expand All @@ -185,8 +201,10 @@ jobs:
- name: Setup uv
uses: astral-sh/setup-uv@v5

- name: Install biosigIO + pyedflib
run: uv pip install --system -r scripts/zarr/requirements.txt pyedflib
- name: Install biosigIO + pyedflib + the schema/parquet validators
run: |
uv pip install --system -r scripts/zarr/requirements.txt pyedflib \
"jsonschema>=4.21" "pyarrow>=15.0" pytest

# test_patch_duration.py wraps every case in
# `unittest.skipUnless(HAVE_PYEDFLIB and HAVE_BIOSIGIO)`, and unittest exits
Expand All @@ -205,13 +223,47 @@ jobs:
- name: Tier-2 duration tests
run: python scripts/test_patch_duration.py

# The converter suite with the dependencies actually installed, under
# pytest so the skip COUNT is machine-readable (unittest's own runner
# reports it only in prose).
- name: Converter suite (full, real dependencies)
run: |
cd scripts/zarr
pytest -q -rs --junit-xml=/tmp/zarr-tests.xml \
test_generate_zarr.py test_zarr_queue.py \
test_purge_non_raw_stores.py test_hallu_zarr_config.py

# A skip is invisible in a green job, and that is exactly how 43 tests
# went unrun for the life of this epic. TWO are expected here and no
# more: the real-data download test (opted out of, or unreachable) and
# the bash-3.2 guard-rail test (this runner has bash 5). Anything above
# that is a dependency that did not install -- the failure mode the
# threshold exists to make loud.
- name: Fail if more than two tests skipped
run: |
python - <<'PY'
import sys, xml.etree.ElementTree as ET
root = ET.parse("/tmp/zarr-tests.xml").getroot()
suite = root if root.tag == "testsuite" else root.find("testsuite")
skipped = int(suite.get("skipped", "0"))
tests = int(suite.get("tests", "0"))
print(f"{tests} tests, {skipped} skipped")
if skipped > 2:
print(
"::error::%d tests skipped (expected at most 2: the real-data "
"download test and the bash-3.2 test). A dependency probably "
"did not install -- the suite passed without asserting." % skipped
)
sys.exit(1)
PY

# Type check and lint - always runs on code changes
lint:
runs-on: ubuntu-latest
needs: changes
# Matches unit-pure's trigger so that, as a required check, lint never
# "skips" on a backend-only PR (a skipped required check blocks merges).
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true' || needs.changes.outputs.zarr == 'true'
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true' || needs.changes.outputs.zarr == 'true' || needs.changes.outputs.shared == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand Down Expand Up @@ -239,7 +291,7 @@ jobs:
unit-pure:
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true' || needs.changes.outputs.zarr == 'true'
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true' || needs.changes.outputs.zarr == 'true' || needs.changes.outputs.shared == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand Down Expand Up @@ -297,7 +349,7 @@ jobs:
integration-dev:
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true'
if: needs.changes.outputs.cli == 'true' || needs.changes.outputs.tests == 'true' || needs.changes.outputs.backend == 'true' || needs.changes.outputs.shared == 'true'
env:
TEST_API_URL: https://nemar-api-dev.sccn-org.workers.dev
TEST_ADMIN_API_KEY: ${{ secrets.TEST_ADMIN_API_KEY }}
Expand Down
8 changes: 5 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,9 +405,11 @@ Environments and pre-release checks: [`.context/release-safety-playbook.md`](.co
probing.
**Events are a third document, `<id>/zarr/events.parquet`** (#1060): one row per
(event, channel group) across every store, with `sample_index` computed by the
converter — `round(onset_s * rate)` against the group's SERVING rate, which is
the only place the resampling relation is known exactly (`resample_poly` is
zero-phase, so there is no delay to subtract). It exists only when the index
converter as `math.floor(onset_s * rate + 0.5)` against the group's SERVING
rate — ties round UP, and it is deliberately not Python's `round()`, which
ties to even and so disagrees on every exact half-sample onset. The serving
rate is the only place the resampling relation is known exactly
(`resample_poly` is zero-phase, so there is no delay to subtract). It exists only when the index
names it (`events_parquet` / `events_row_count`); publishing it is best-effort
like the manifest, and the per-store `n_events` / `trial_types` come from the
same parse as its rows.
Expand Down
23 changes: 15 additions & 8 deletions backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -759,11 +759,17 @@ export default {
// Daily (prod "0 3 * * *", dev/staging "0 4 * * *"):
// Catalog sync runs via GitHub Action, not Worker cron.
//
// ALLOWLIST, epic #923 Phase 7. The dev/staging worker's D1 is a partial
// PRODUCTION MIRROR (real nm* dataset rows, real user email addresses) and
// the dev worker holds a real RESEND_API_KEY, so a daily job that selects
// rows by generic predicates acts on real production records. Only jobs
// proven safe against that mirror run outside production.
// ALLOWLIST, epic #923 Phase 7. The dev/staging D1's dataset catalog is no
// longer a production mirror -- it was purged to the curated fixtures (the
// `xx0999NN` exemplars plus the private E2E dataset) and must stay that way.
// The reason for this allowlist survives that purge: the `users` table was
// NOT purged (it still holds ~609 real email addresses), the dev worker
// holds a real RESEND_API_KEY, and the nemarDatasets GitHub org is SHARED
// between prod and dev because the org name is hardcoded rather than
// environment-scoped. So a daily job that selects rows by generic predicates
// can still email real people or destroy a real repo. Only jobs proven safe
// against that run outside production. See AGENTS.md, "Dev D1 shares
// production users and the GitHub org".
//
// A NEW DAILY JOB IS PRODUCTION-ONLY BY DEFAULT. Before adding one to the
// non-prod set below, confirm it cannot (a) email a real user, (b) dispatch
Expand All @@ -784,9 +790,10 @@ export default {
if (prodOnlyJobs) {
// #736 Phase 3: backstop re-dispatch of still-failed archive generations
// whose webhook retry chain broke (e.g. a lost archive-ready callback).
// PROD-ONLY: the candidate query has no dataset-id prefix filter, so on a
// mirror D1 it would repository_dispatch real Actions runs against the
// shared nemarDatasets org for real datasets.
// PROD-ONLY: the candidate query has no dataset-id prefix filter, so
// outside production it would repository_dispatch real Actions runs
// against the SHARED nemarDatasets org for whatever rows the dev D1 holds
// -- the fixtures there (nm099999, the exemplars) are real repos in it.
ctx.waitUntil(archiveRetrySweep(env));
// #1130: heal published versions whose S3 manifest never landed (the
// version-DOI callback swallows manifest failures by design, so without
Expand Down
Loading
Loading