Skip to content

Phase C1: Repository scaffold and the nemar-launch launcher #2

Closed

Description

@neuromechanist

Goal

nemar-process gets its scaffold, CI and the launcher. After this phase a zip built from the repository, run the way Tapis runs a ZIP-runtime application, fetches a code bundle, stages datasets, runs the entry under Apptainer isolated from the host, and leaves provenance, a result manifest and one fixed failure code in a directory the container never saw, then uploads the outputs and those files through NEMAR's outputs routes.

Routing

Scope

In: layout (launcher/, images/, recipes/, deploy/tapis/, observer/, tests/, docs/, scripts/); AGENTS.md, CLAUDE.md (@AGENTS.md); uv project, ruff, ty, pytest, pre-commit hook, CI; the launcher; a deterministic zip builder. Out: images (#3), recipes (#4, #5), Tapis JSON and observer (#6).

Design references

Design page section 7 ("The launcher on Expanse"); implementation notes sections 2 and 5; ADR 0095 and shared/contract/data-plane.ts in nemar-cli: entries are {path, size, checksum?, checksum_algorithm?, bytes_url, url}; a 502 with error "Published file is missing from storage" is lost content, any other 502 a transport fault to retry.

Changes

  • Root pyproject.toml (hatchling, requires-python >=3.12), uv.lock, ruff.toml (launcher/ and observer/ use target-version = "py39"), .pre-commit-config.yaml (ruff-pre-commit pinned by rev, --fix --unsafe-fixes, staged files; the rule_templates reference file does not exist on this machine).
  • AGENTS.md: public-repo hygiene, no MATLAB, no mocks, uv/ruff/ty, upstream-first for eegprep gaps.
  • launcher/tapisjob_app.sh: picks the first of python3.12 to python3.9, then python3, that is at least 3.9 (override NEMAR_HOST_PYTHON); runs python -I -m nemar_launch; writes ${_tapisExecSystemOutputDir}/tapisjob.exitcode and exits with that number.
  • launcher/nemar_launch/ (stdlib only): spec.py, bundle.py, stage.py, run.py, provenance.py, upload.py, codes.py, __main__.py; launcher/schema/launch-spec.v1.json.
  • Launch spec nemar.launch-spec.v1 (the document NEMAR's Worker hands the launcher; distinct from the user-facing job specification of Phase A1: Compute contract and ADR 0118 nemar-cli#2006) is one JSON Schema shared with nemar-cli: Phase A1: Compute contract and ADR 0118 nemar-cli#2006 publishes it as shared/contract/compute-launch.schema.json. Whichever of Phase A1: Compute contract and ADR 0118 nemar-cli#2006 and this phase merges first writes it; the other copies it byte for byte. A drift test here compares the two when a nemar-cli checkout is beside it (CI sparse-checks-out nemar-cli dev), as the website does for nemar-cli's account copy. The launcher reads it from env NEMAR_LAUNCH_SPEC (JSON, at most 64 KiB, no unknown keys): job_id, runtime (python or image; else runtime_error, detail unsupported_runtime), entry or command, args, bundle {sha256, size} (optional with command; no URL), datasets [{id, version, include?}], api_base, image {uri, digest, sif}, env allowlist, resources {gpus}, limits {max_files, max_total_bytes}, job_token (required; used with api_base and the X-Compute-Job-Token header; the schema's $defs hold the bundle, outputs and complete shapes).
  • Output directory ${_tapisExecSystemOutputDir}: work/ (the only writable mount, /work, outputs in work/outputs/) and launcher/ (result.json, provenance.json, manifest.json, launcher.log). Code and datasets stage in node-local scratch (SLURM_TMPDIR, else TMPDIR), never in the archived directory.
  • Bundle (bundle.py): POST <api_base>/compute/jobs/<job_id>/bundle with X-Compute-Job-Token: <job_token> returns a 10-minute presigned GET of bundle.tar.gz, one archive of the whole folder; download it to scratch with the same retry and size and sha256 checks as dataset files, then extract with stdlib tarfile (Python 3.9) after refusing any member that is absolute, contains .., is a link, or is a device, FIFO or other non-regular, non-directory entry. A refusal or digest mismatch is code_fetch_failed.
  • Container: apptainer exec, else singularity exec, with --containall --cleanenv --no-home --pwd /work; binds work:/work, work/tmp:/tmp, code :/code:ro, each dataset :/data/<id>:ro; environment only via --env from the allowlist plus NEMAR_OUT=/work/outputs; --nv only when resources.gpus > 0.
  • Staging: bind NEMAR_LOCAL_DATASETS_ROOT when selected files match the manifest by size and checksum (SHA256E sha256, git blob SHA-1), recording mode: bind; otherwise download each bytes_url into scratch with 5 attempts, backoff with jitter, Range resume, size and checksum checks, 4 concurrent files, the spec caps (default 38,000 files).
  • After exit: remove symlinks under work/ (recorded), hash the tree into manifest.json; provenance.json holds spec digest, bundle digest, image uri and digest, runtime version, staging records, phase timings, Slurm job id, entry exit code. URLs are logged as host and path only.
  • Upload (upload.py, after the manifest is written): POST <api_base>/compute/jobs/<job_id>/outputs with the same header, sending the output manifest (path, size, sha256) and the sizes of the four launcher files; PUT each file to its presigned URL (4 concurrent, 3 attempts with backoff, one re-request after a 403, no redirect followed), outputs first and result.json last; then POST <api_base>/compute/jobs/<job_id>/outputs/complete. An API refusal (a cap or quota code) still uploads the launcher files, then fails. A failed upload is result_upload_failed, exit 16, unless an earlier code stands (the first failure wins). The token and presigned URLs go to no log or provenance file: hosts and paths only.
  • Exit numbers (in docs/launcher.md): ok 0, code_fetch_failed 10, dataset_fetch_failed 11, entry_not_found 12, runtime_error 13, out_of_memory 14, time_limit 15, result_upload_failed 16. Exit 137 is out_of_memory; SIGTERM at or after SLURM_JOB_END_TIME minus 120 s is time_limit, any other SIGTERM runtime_error (detail terminated).
  • scripts/build_launcher_zip.py (deterministic, tapisjob_app.sh at the root, --check); tests/standin/dataplane.py (stdlib server over tests/fixtures/dataset/, fault injection, request log).
  • .github/workflows/ci.yml: lint, ty, launcher tests on 3.9 (uv run --no-project --python 3.9 --with pytest pytest tests/launcher) and 3.12, pinned Apptainer, a small public image pinned by digest.

Tests (write these first)

  1. Spec: unknown key, missing upload, runtime: matlab, oversize, traversal globs refused. Mutate: loosen additionalProperties.
  2. The bundle route is called with the token header and its answer is downloaded; an archive digest mismatch or a route refusal is code_fetch_failed; a 503 then success retries. Mutate: skip the compare; attempts to 1.
  3. A tar.gz with a ../x, absolute, symlink, hardlink or device member is refused; a clean one extracts under Python 3.9. Mutate: drop the check.
  4. Missing entry is entry_not_found.
  5. Download resumes after a truncated body, rejects a wrong checksum, names the lost-content path, enforces caps. Mutate: drop the size check.
  6. Bind: a matching local copy makes the stand-in see only the manifest request; one wrong byte falls back to download. Mutate: size-only compare.
  7. Real Apptainer: no /launcher, no host env var, no $HOME, /code and /data unwritable, /work writable. Mutate: drop --containall.
  8. Exit 3 is runtime_error with entry_exit_code 3; SIGKILL is out_of_memory; SIGTERM past the end time is time_limit.
  9. A forged /work/provenance.json leaves launcher/provenance.json intact; a symlink in work/ is removed and recorded; manifest hashes match.
  10. tapisjob.exitcode holds the mapped number for every code. Mutate: change one.
  11. A presigned query string and the job token appear in no log or provenance file; two zip builds are byte-identical.
  12. Upload against tests/standin/callbacks.py (stdlib: the bundle, outputs and complete routes and an S3-style PUT receiver, request log, failure modes): files arrive with the token header and complete is called once, last; a 403 is re-requested once; a cap refusal still uploads the launcher files and exits 16; a PUT 5xx past the retries exits 16 and an earlier runtime_error stays. Mutate: call complete early; drop the retry.

Acceptance criteria

  • Tests pass in CI, container tests on real Apptainer.
  • Ruff, ty clean; launcher/ coverage at least 85%.
  • docs/launcher.md documents spec, layout, codes and environment; uv run pre-commit install is documented.

Gates

uv sync --frozen && uv run ruff check . && uv run ruff format --check . && uv run ty check && uv run pytest --cov=launcher --cov-fail-under=85 && uv run python scripts/build_launcher_zip.py --check

Rules that bite here

  • Nothing the container writes is trusted; provenance and manifest are written from outside, into an unbound directory, and --cleanenv leaves only the allowlist.
  • Tapis archives tapisjob.env by default and the spec carries the job token and no URL; the launcher never echoes the spec or a presigned address (Phase C5: Tapis definitions and the observer #6 turns launch files off).

Escalate to the owner if

  • Apptainer cannot run on hosted runners, or the node's host Python is older than 3.9.
  • A compute node cannot reach NEMAR's API or the R2 S3 endpoint over HTTPS (measured available in October 2026; still a question for SDSC).

When merged

Tell Epic A the spec schema and launcher/ file names are fixed; list the unverified items (host Python, local dataset root layout, presigned multipart upload to R2 with the per-file cap at 5 GB until verified, outbound HTTPS from nodes) in docs/unverified.md.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the nemar-cli phase A1 review (nemarOrg/nemar-cli#2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • Provenance key: launcher/provenance.json holds launch_spec_digest, the SHA-256 hex of the raw bytes of the NEMAR_LAUNCH_SPEC environment variable, hashed BEFORE parsing (os.environb[b"NEMAR_LAUNCH_SPEC"]). It is not called spec_digest; that name belongs to the job row in nemar-cli. The launch specification is ASCII-only JSON (every code unit at or above U+007F is written as \uXXXX by NEMAR), so the bytes survive Tapis, the shell and the locale.
    • Add a test that checks the golden vector published with nemar-cli's shared/contract/compute-launch.ts tests (a fixed specification with a non-ASCII character, a double quote and a space in args, its exact encoded text and its SHA-256), against the launcher's hashing.
    • Test 1: the key upload no longer exists in the specification; use job_token as the missing required key. Add: an extra key in the bundle response or the outputs response (and in one uploads item) is ACCEPTED and ignored, because a running launcher must survive the Worker adding a key; every other object in the specification stays closed.
    • The schema file is shared/contract/compute-launch.schema.json from nemar-cli after PR #2030 merges into the epic branch; copy it byte for byte from the merged commit, never from an open branch.
  2. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #7 (squash d17078b). Round 1 and round 2 reviews converged; CI green on fc8786a (launcher on Python 3.9 and 3.12, lint, nemar-cli-latest pin check). The launcher is re-pinned to the merged A1b contract (nemar-cli 7ba7d2f1, schema sha256 6707bb26d653...). Still unverified until a real Tapis tenant exists: listed in docs/unverified.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions