Skip to content

Phase C2: nemar-python runtime image #3

Closed

Description

@neuromechanist

Goal

The nemar-python runtime image exists as code and as a CI product: Python 3.12 with eegprep, MNE-Python, NumPy, SciPy, pamica (CPU) and the signalJourney validator, built from a committed lock file, pushed to GHCR, turned into a SIF from its digest, signed with cosign keyless, and smoke-tested both as a container image and as a SIF under Apptainer. Every consumer refers to it by digest.

Routing

Scope

In: images/nemar-python/ (CPU flavor only), the dependency declaration and lock, the build and signing workflow, the digest record, the smoke test. Out: the CUDA flavor, the Fortran amica15 binary and the ICLabel weights check (#5); recipes; registering anything with Tapis.

Design references

Design page section 7 (runtimes); first-party design v1 F.2 (base, pinning, GHCR, cosign, SIF via oras://) and v2 B.6 (two flavors from one lock).

Changes

  • Root pyproject.toml: dependency group runtime (and extra cpu) pinning exact versions: eegprep[mne,pybids,picard,torch]==0.5.0 (do not use the eeglab extra: it installs an Octave bridge, which the no-MATLAB rule forbids), mne==1.13.2, pamica==0.4.0, mne-icalabel only if Phase C4: nemar-ica recipe #5 needs it, NumPy and SciPy as resolved, torch from the CPU index via [tool.uv.sources] with extra = "cpu". One root uv.lock serves image, recipes and tests. Versions were current on PyPI on 2026-11-03; re-check.
  • signalJourney validator: it is not on PyPI and resolves its schema relative to its source tree, so a plain install cannot find the schema. Add it as signaljourney-validator @ git+https://github.com/signalJourney/signalJourney@<commit> (commit 169c3261d044 on 2026-11-03, or newer), and vendor that commit's schema/ directory into the Python package nemar_process/_vendor/signalJourney/schema/ with a SOURCE file naming the commit. Provide nemar_process.sj.validator() returning Validator(schema_dir=<vendored dir>). File an upstream issue on signalJourney/signalJourney to publish the validator to PyPI with the schema packaged; do not edit that repository from here.
  • images/nemar-python/Dockerfile: python:3.12-slim pinned by digest, uv copied from a version-pinned image, uv sync --frozen --no-dev --group runtime --extra cpu into /opt/venv, OPENBLAS_NUM_THREADS=1, OMP_NUM_THREADS=1, PYTHONHASHSEED=0 and a non-root user, labels with source commit and lock hash. Phase C3: nemar-floor recipe #4 installs the recipes package; this phase installs a placeholder.
  • images/nemar-python/smoke.py: imports every pinned package and prints versions as JSON; runs one tiny eegprep step (load the smoke fixture, pop_eegfiltnew, clean_flatlines) and validates a hand-written signalJourney record with nemar_process.sj.validator().
  • tests/fixtures/smoke/ plus scripts/make_smoke_fixture.py: a deterministic synthetic 8-channel, 20-second .set (seeded NumPy), documented as synthetic and as a smoke input only; scientific fixtures are Phase C3: nemar-floor recipe #4's.
  • .github/workflows/image-nemar-python.yml: on pull request build without pushing; on tag image-nemar-python-v* or manual dispatch: docker/build-push-action to ghcr.io/nemarorg/nemar-python:<ver>-cpu, capture the digest output, cosign sign keyless on ghcr.io/nemarorg/nemar-python@sha256:<digest> (id-token: write), apptainer build the SIF from that digest reference, apptainer push it to oras://ghcr.io/nemarorg/nemar-python-sif:sha256-<hex>, cosign-sign the SIF artifact digest, run the smoke test in both forms, and attach nemar-python-<ver>-cpu-digests.json (image digest, SIF digest, lock sha256, eegprep, pamica, torch versions, run URL) to the release.
  • docs/images.md: how consumers reference a digest, how to verify (cosign verify --certificate-identity-regexp ... --certificate-oidc-issuer https://token.actions.githubusercontent.com), and that SIF compatibility with the cluster's Apptainer variant is unverified until the operator tests it.

Tests (write these first)

  1. uv lock --check passes and the lock contains no package named in a no-matlab denylist (oct2py, matlab, matlabengine). Mutate: add the eeglab extra.
  2. smoke.py in the lock's venv prints versions equal to the pins. Mutate: change one pin.
  3. The vendored schema validates a signalJourney example (copied to tests/fixtures/sj/) and rejects a record missing pipelineInfo. Mutate: empty schema_dir.
  4. CI job: build the image on the pull request, run smoke.py inside it, then build a SIF from the local image and run it under Apptainer with --containall --cleanenv --no-home. Mutate: remove scipy from the lock; the job must fail.
  5. actionlint passes and the Dockerfile base is a digest. Mutate: use a tag.

Acceptance criteria

  • The pull-request workflow builds and smoke-tests the image and the SIF.
  • A manual dispatch pushes, signs, builds the SIF from the digest, signs it, and attaches the digest file; cosign verify succeeds in the same run.
  • No MATLAB, Octave or eeglab extra in the image or the lock.
  • The upstream signalJourney packaging issue is filed and linked.

Gates

uv lock --check && uv run ruff check . && uv run ruff format --check . && uv run ty check && uv run pytest tests/images --cov and, on a machine with Docker and Apptainer, docker build -f images/nemar-python/Dockerfile . then docker run --rm <image> python /opt/smoke.py.

Rules that bite here

  • Digest everywhere: a tag moves, a digest does not; the Dockerfile base, the uv image and every consumer reference are digests.
  • No MATLAB path of any kind, including a transitive one: check the lock.
  • GHCR names are lowercase (nemarorg), and signing needs id-token: write in the job only.

Escalate to the owner if

  • GHCR publishing is not enabled for the repository.
  • The resolver cannot satisfy pamica's torch floor with the CPU index (record the exact error).

When merged

Record the first digest in docs/images.md; tell #6 and Epic A the image reference format; add the signalJourney and eegprep upstream issue links to docs/upstream.md.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in #7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).

    The SIF this phase builds is named by the image digest: <64 lowercase hex characters of the OCI digest>.sif, placed under the directory the Tapis application sets as NEMAR_SIF_ROOT. The launcher refuses a specification whose image.sif is not that name for image.digest.

  2. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #8 (squash). Rounds 1 and 2 converged; CI green on the final head (container and SIF smoke test, images, both launcher jobs). The publish, sign and release chain has NOT been run: it runs only from a tag or a dispatch on a commit of main, after the epic merges. Owner items before the first publish are in docs/images.md (GHCR publishing enabled, both packages linked to the repository with write access, a tag ruleset on image-nemar-python-v* and protection on main or an environment with reviewers). Upstream issues: signalJourney/signalJourney#6 and sccn/eegprep#533.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions