Repository navigation
Description
Goal
nemar-process gets its scaffold, CI and the launcher. After this phase a zip built from the repository, run the way Tapis runs a ZIP-runtime application, fetches a code bundle, stages datasets, runs the entry under Apptainer isolated from the host, and leaves provenance, a result manifest and one fixed failure code in a directory the container never saw, then uploads the outputs and those files through NEMAR's outputs routes.
Routing
- Readiness: ready
- Implementer: Sonnet
- Opus checkpoint: review before merge (the
execarguments and bind table, zip-slip and symlink handling, thatlauncher/is unreachable from inside) - Depends on: none ; Unblocks: Phase C2: nemar-python runtime image #3, Phase C3: nemar-floor recipe #4, Phase C5: Tapis definitions and the observer #6
- Repository and base branch: nemarOrg/nemar-process, epic branch
feature/issue-#1-epic-process
Scope
In: layout (launcher/, images/, recipes/, deploy/tapis/, observer/, tests/, docs/, scripts/); AGENTS.md, CLAUDE.md (@AGENTS.md); uv project, ruff, ty, pytest, pre-commit hook, CI; the launcher; a deterministic zip builder. Out: images (#3), recipes (#4, #5), Tapis JSON and observer (#6).
Design references
Design page section 7 ("The launcher on Expanse"); implementation notes sections 2 and 5; ADR 0095 and shared/contract/data-plane.ts in nemar-cli: entries are {path, size, checksum?, checksum_algorithm?, bytes_url, url}; a 502 with error "Published file is missing from storage" is lost content, any other 502 a transport fault to retry.
Changes
- Root
pyproject.toml(hatchling,requires-python >=3.12),uv.lock,ruff.toml(launcher/andobserver/usetarget-version = "py39"),.pre-commit-config.yaml(ruff-pre-commit pinned by rev,--fix --unsafe-fixes, staged files; therule_templatesreference file does not exist on this machine). AGENTS.md: public-repo hygiene, no MATLAB, no mocks, uv/ruff/ty, upstream-first for eegprep gaps.launcher/tapisjob_app.sh: picks the first ofpython3.12topython3.9, thenpython3, that is at least 3.9 (overrideNEMAR_HOST_PYTHON); runspython -I -m nemar_launch; writes${_tapisExecSystemOutputDir}/tapisjob.exitcodeand exits with that number.launcher/nemar_launch/(stdlib only):spec.py,bundle.py,stage.py,run.py,provenance.py,upload.py,codes.py,__main__.py;launcher/schema/launch-spec.v1.json.- Launch spec
nemar.launch-spec.v1(the document NEMAR's Worker hands the launcher; distinct from the user-facing job specification of Phase A1: Compute contract and ADR 0118 nemar-cli#2006) is one JSON Schema shared with nemar-cli: Phase A1: Compute contract and ADR 0118 nemar-cli#2006 publishes it asshared/contract/compute-launch.schema.json. Whichever of Phase A1: Compute contract and ADR 0118 nemar-cli#2006 and this phase merges first writes it; the other copies it byte for byte. A drift test here compares the two when a nemar-cli checkout is beside it (CI sparse-checks-out nemar-clidev), as the website does for nemar-cli's account copy. The launcher reads it from envNEMAR_LAUNCH_SPEC(JSON, at most 64 KiB, no unknown keys):job_id,runtime(pythonorimage; elseruntime_error, detailunsupported_runtime),entryorcommand,args,bundle {sha256, size}(optional withcommand; no URL),datasets [{id, version, include?}],api_base,image {uri, digest, sif},envallowlist,resources {gpus},limits {max_files, max_total_bytes},job_token(required; used withapi_baseand theX-Compute-Job-Tokenheader; the schema's$defshold the bundle, outputs and complete shapes). - Output directory
${_tapisExecSystemOutputDir}:work/(the only writable mount,/work, outputs inwork/outputs/) andlauncher/(result.json,provenance.json,manifest.json,launcher.log). Code and datasets stage in node-local scratch (SLURM_TMPDIR, elseTMPDIR), never in the archived directory. - Bundle (
bundle.py):POST <api_base>/compute/jobs/<job_id>/bundlewithX-Compute-Job-Token: <job_token>returns a 10-minute presigned GET ofbundle.tar.gz, one archive of the whole folder; download it to scratch with the same retry and size and sha256 checks as dataset files, then extract with stdlibtarfile(Python 3.9) after refusing any member that is absolute, contains.., is a link, or is a device, FIFO or other non-regular, non-directory entry. A refusal or digest mismatch iscode_fetch_failed. - Container:
apptainer exec, elsesingularity exec, with--containall --cleanenv --no-home --pwd /work; bindswork:/work,work/tmp:/tmp, code:/code:ro, each dataset:/data/<id>:ro; environment only via--envfrom the allowlist plusNEMAR_OUT=/work/outputs;--nvonly whenresources.gpus > 0. - Staging: bind
NEMAR_LOCAL_DATASETS_ROOTwhen selected files match the manifest by size and checksum (SHA256Esha256,gitblob SHA-1), recordingmode: bind; otherwise download eachbytes_urlinto scratch with 5 attempts, backoff with jitter,Rangeresume, size and checksum checks, 4 concurrent files, the spec caps (default 38,000 files). - After exit: remove symlinks under
work/(recorded), hash the tree intomanifest.json;provenance.jsonholds spec digest, bundle digest, image uri and digest, runtime version, staging records, phase timings, Slurm job id, entry exit code. URLs are logged as host and path only. - Upload (
upload.py, after the manifest is written):POST <api_base>/compute/jobs/<job_id>/outputswith the same header, sending the output manifest (path,size,sha256) and the sizes of the four launcher files; PUT each file to its presigned URL (4 concurrent, 3 attempts with backoff, one re-request after a 403, no redirect followed), outputs first andresult.jsonlast; thenPOST <api_base>/compute/jobs/<job_id>/outputs/complete. An API refusal (a cap or quota code) still uploads the launcher files, then fails. A failed upload isresult_upload_failed, exit 16, unless an earlier code stands (the first failure wins). The token and presigned URLs go to no log or provenance file: hosts and paths only. - Exit numbers (in
docs/launcher.md): ok 0,code_fetch_failed10,dataset_fetch_failed11,entry_not_found12,runtime_error13,out_of_memory14,time_limit15,result_upload_failed16. Exit 137 isout_of_memory; SIGTERM at or afterSLURM_JOB_END_TIMEminus 120 s istime_limit, any other SIGTERMruntime_error(detailterminated). scripts/build_launcher_zip.py(deterministic,tapisjob_app.shat the root,--check);tests/standin/dataplane.py(stdlib server overtests/fixtures/dataset/, fault injection, request log)..github/workflows/ci.yml: lint, ty, launcher tests on 3.9 (uv run --no-project --python 3.9 --with pytest pytest tests/launcher) and 3.12, pinned Apptainer, a small public image pinned by digest.
Tests (write these first)
- Spec: unknown key, missing
upload,runtime: matlab, oversize, traversal globs refused. Mutate: loosenadditionalProperties. - The bundle route is called with the token header and its answer is downloaded; an archive digest mismatch or a route refusal is
code_fetch_failed; a 503 then success retries. Mutate: skip the compare; attempts to 1. - A
tar.gzwith a../x, absolute, symlink, hardlink or device member is refused; a clean one extracts under Python 3.9. Mutate: drop the check. - Missing entry is
entry_not_found. - Download resumes after a truncated body, rejects a wrong checksum, names the lost-content path, enforces caps. Mutate: drop the size check.
- Bind: a matching local copy makes the stand-in see only the manifest request; one wrong byte falls back to download. Mutate: size-only compare.
- Real Apptainer: no
/launcher, no host env var, no$HOME,/codeand/dataunwritable,/workwritable. Mutate: drop--containall. - Exit 3 is
runtime_errorwithentry_exit_code 3; SIGKILL isout_of_memory; SIGTERM past the end time istime_limit. - A forged
/work/provenance.jsonleaveslauncher/provenance.jsonintact; a symlink inwork/is removed and recorded; manifest hashes match. tapisjob.exitcodeholds the mapped number for every code. Mutate: change one.- A presigned query string and the job token appear in no log or provenance file; two zip builds are byte-identical.
- Upload against
tests/standin/callbacks.py(stdlib: the bundle, outputs and complete routes and an S3-style PUT receiver, request log, failure modes): files arrive with the token header andcompleteis called once, last; a 403 is re-requested once; a cap refusal still uploads the launcher files and exits 16; a PUT 5xx past the retries exits 16 and an earlierruntime_errorstays. Mutate: callcompleteearly; drop the retry.
Acceptance criteria
- Tests pass in CI, container tests on real Apptainer.
- Ruff, ty clean;
launcher/coverage at least 85%. -
docs/launcher.mddocuments spec, layout, codes and environment;uv run pre-commit installis documented.
Gates
uv sync --frozen && uv run ruff check . && uv run ruff format --check . && uv run ty check && uv run pytest --cov=launcher --cov-fail-under=85 && uv run python scripts/build_launcher_zip.py --check
Rules that bite here
- Nothing the container writes is trusted; provenance and manifest are written from outside, into an unbound directory, and
--cleanenvleaves only the allowlist. - Tapis archives
tapisjob.envby default and the spec carries the job token and no URL; the launcher never echoes the spec or a presigned address (Phase C5: Tapis definitions and the observer #6 turns launch files off).
Escalate to the owner if
- Apptainer cannot run on hosted runners, or the node's host Python is older than 3.9.
- A compute node cannot reach NEMAR's API or the R2 S3 endpoint over HTTPS (measured available in October 2026; still a question for SDSC).
When merged
Tell Epic A the spec schema and launcher/ file names are fixed; list the unverified items (host Python, local dataset root layout, presigned multipart upload to R2 with the per-file cap at 5 GB until verified, outbound HTTPS from nodes) in docs/unverified.md.
Activity
neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the nemar-cli phase A1 review (nemarOrg/nemar-cli#2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.
launcher/provenance.jsonholdslaunch_spec_digest, the SHA-256 hex of the raw bytes of theNEMAR_LAUNCH_SPECenvironment variable, hashed BEFORE parsing (os.environb[b"NEMAR_LAUNCH_SPEC"]). It is not calledspec_digest; that name belongs to the job row in nemar-cli. The launch specification is ASCII-only JSON (every code unit at or above U+007F is written as \uXXXX by NEMAR), so the bytes survive Tapis, the shell and the locale.shared/contract/compute-launch.tstests (a fixed specification with a non-ASCII character, a double quote and a space inargs, its exact encoded text and its SHA-256), against the launcher's hashing.uploadno longer exists in the specification; usejob_tokenas the missing required key. Add: an extra key in the bundle response or the outputs response (and in oneuploadsitem) is ACCEPTED and ignored, because a running launcher must survive the Worker adding a key; every other object in the specification stays closed.shared/contract/compute-launch.schema.jsonfrom nemar-cli after PR #2030 merges into the epic branch; copy it byte for byte from the merged commit, never from an open branch.neuromechanist commentedon Nov 4, 2026
Merged into the epic branch by #7 (squash d17078b). Round 1 and round 2 reviews converged; CI green on fc8786a (launcher on Python 3.9 and 3.12, lint, nemar-cli-latest pin check). The launcher is re-pinned to the merged A1b contract (nemar-cli 7ba7d2f1, schema sha256 6707bb26d653...). Still unverified until a real Tapis tenant exists: listed in docs/unverified.md.