Repository navigation
Description
Goal
The Tapis side of the repository exists as code: the nemar-launch application (ZIP runtime, BATCH) and the nemar-observer application (ZIP runtime, FORK), system templates with placeholders only, JSON Schemas derived from Tapis's own OpenAPI specifications that CI validates them against, a dry-run-by-default registration script, and the observer that summarises the Slurm queue and posts it to NEMAR.
Routing
- Readiness: ready (the unverified Tapis and SDSC items below wait on the operator; the code takes them as configuration)
- Implementer: Sonnet
- Opus checkpoint: review before merge (archive settings, FIXED variables, no credential or host detail, token handling)
- Depends on: Phase C1: Repository scaffold and the nemar-launch launcher #2 (zip builder, env names, output layout) ; Unblocks: Epic A's Tapis client and observer route integration
- Repository and base branch: nemarOrg/nemar-process, epic branch
feature/issue-#1-epic-process
Scope
In: deploy/tapis/, scripts/derive_tapis_schemas.py, scripts/register.py, observer/, CI validation, docs/operator.md. Out: registering against a real tenant, system credentials, the NEMAR receiving route (Epic A), interactive-session applications.
Design references
Design page sections 7 and 8, section 12 questions 2, 6, 9, 11, 18; implementation notes section 5. Tapis ZIP runtime: containerImage is the archive URL or an absolute path, tapisjob_app.sh is the entry, tapisjob.exitcode is documented for FORK jobs only, and tapisjob.sh and tapisjob.env are archived unless includeLaunchFiles is false.
Changes
deploy/tapis/openapi/AppsAPI.yamlandSystemsAPI.yamlvendored fromtapis-project/openapi-appsat10ec91aa31c1andopenapi-systemsat929af4b8855d(BSD-3 text kept asLICENSE-tapis, commits current on 2026-11-03),SOURCErecords both.scripts/derive_tapis_schemas.py: extractsReqPostAppandReqPostSystemwith their$refclosure intodeploy/tapis/schemas/app.schema.jsonandsystem.schema.json, converting OpenAPI 3.0 to JSON Schema and settingadditionalProperties: falseon every object (the specifications leave objects open; a typo must fail). CI re-derives and fails on a diff.deploy/tapis/apps/nemar-launch.json:id nemar-launch,versionequal to the launcher version,runtime ZIP,containerImagethe release-asset URL ofnemar-launch-<version>.zip(public),jobType BATCH,strictFileInputs trueand nofileInputs,archiveMode ALWAYS,archiveOnAppError true,archiveFilter {includeLaunchFiles: false, excludes: ["tapisjob.env", "tapisjob.sh"]},NEMAR_LAUNCH_SPEC(the name Phase C1: Repository scaffold and the nemar-launch launcher #2 reads; it carries the per-job upload token) asREQUIRED,FIXEDvariables for anything the submitter must not change, andnotes.launcher_sha256equal to the reproducible zip's digest (CI rebuilds and compares). Execution system, queue and resources are set by NEMAR at submit time; there are no archive system fields, so Tapis archives to its default on the execution system with launch files excluded.deploy/tapis/apps/nemar-observer.json:runtime ZIP,jobType FORK, the observer zip URL, the same archive filter, variablesNEMAR_OBSERVER_URLandNEMAR_OBSERVER_TOKENasREQUIRED.deploy/tapis/systems/exec.json.tmpl(SSH execution system, Slurm, logical queuesshared,compute,gpu,gpu-shared,debug,gpu-debug,preemptwith limits from the public Expanse user guide, listed as unverified; staticeffectiveUserIdplaceholder). There is no S3 archive system template: results reach NEMAR through the launcher's upload and Tapis holds no storage credential. Placeholders such as${EXEC_HOST},${COMMUNITY_ACCOUNT},${EXEC_ROOT_DIR}only; no credential field.scripts/register.py:string.Templaterendering from the environment or--vars, unresolved placeholder is an error; dry run by default (method, path, body hash);--applydoesGETthenPOSTonly when absent and never changes an existing app version; token fromTAPIS_TOKEN, base fromTAPIS_BASE_URL, never from arguments.observer/observe.py(stdlib, Python 3.9 floor): modesquickanddaily. Per partitionsinfo -h -o "%P|%a|%D|%T",squeue -h -o "%P|%T|%D|%C|%r";squeue --start -h -j <ids>for the Slurm ids inNEMAR_OBSERVE_JOBIDS(at most 500, from TapisremoteJobId);expanse-clientbalance as captured lines, parsed only when the format is recognised, otherwise{"status": "unparsed", "raw_lines": [...]}and never invented;dailyaddssacct -X -P -S <day> -E <day+1> -A <account>(account fromNEMAR_SLURM_ACCOUNT) with job id, partition, state, elapsed, allocated TRES, submit, start, end only, no user names. Postsnemar.observation.v1JSON toNEMAR_OBSERVER_URLwithAuthorization: Bearer $NEMAR_OBSERVER_TOKEN, never follows a redirect, caps the body at 1 MiB, never prints the token, retries 3 times..github/workflows/ci.ymladditions: derive-and-diff, validate (each app JSON and each rendered template with the example values indeploy/tapis/example.env), launcher zip digest check, observer tests.
Tests (write these first)
- The apps validate against the derived schemas; adding a stray key or removing a required key fails. Mutate: drop
additionalProperties: false. - Re-deriving gives no diff against the committed schemas; a hand-edited schema fails CI. Mutate: edit one.
- Rendering fails on an unresolved placeholder or a value with a newline.
register.pydry run sends no request (a local stand-in records zero);--applycreates once, then is a no-op; an existing app is never overwritten. Mutate: treat 409 as success.- A grep test: no hostname, account, allocation id or home path pattern in
deploy/orobserver/, and noarchiveSystemId,archiveSystemDiror S3 system anywhere underdeploy/. Mutate: add one. - Observer parsers read fixture text in the Slurm manual's formats (stated in a header comment: not captured from a cluster) via stand-in executables on
PATH; unknown balance output isunparsed. Mutate: swap two columns. - Observer posts to a local stand-in with the bearer token, refuses a redirect, and the token appears in no output. Mutate: print the headers.
Acceptance criteria
- Apps and rendered systems validate in CI; schemas reproducible.
-
docs/operator.mdanddocs/unverified.mdwritten. - No credential, host, account or tenant name in the tree.
Gates
uv run python scripts/derive_tapis_schemas.py --check && uv run ruff check . && uv run ruff format --check . && uv run ty check && uv run pytest tests/deploy tests/observer --cov=observer --cov-fail-under=85
Rules that bite here
includeLaunchFilesdefaults to true and archivestapisjob.env, which holds the launch spec (with the upload token) and the observer token: both apps turn it off.- Credentials come only from the environment;
register.pynever logs them. - Strictness is added by the derivation, never by hand edits.
Escalate to the owner if
- An operator test contradicts the Tapis documentation.
- Compute nodes cannot make outbound HTTPS requests to NEMAR's API or the R2 S3 endpoint (measured available in October 2026; still a question for SDSC).
When merged
Unverified, waiting on the operator: outbound HTTPS from nodes to the API and the R2 S3 endpoint (question for SDSC); the default archive on the execution system with launch files excluded; BATCH reading of tapisjob.exitcode; strictFileInputs with FIXED variables; maximum environment value length; FORK running Slurm commands on a login node (question 6); expanse-client output format; queue limits; host Python version; SIF compatibility with the cluster's Apptainer; local dataset root layout.
Activity
neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in #7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).
Application and system definitions must, per the Opus checkpoint of the launcher:
archiveIncludeLaunchFilesto false (Tapis archivestapisjob.env, which holds the per-job token);${JobUUID}) separate from the execution directory, and putjobWorkingDiron scratch under a 0700 root, never$HOME(quota);NEMAR_SIF_ROOTas a FIXED variable and a node-localTMPDIR;CUDA_VISIBLE_DEVICESthrough for GPU jobs;--no-mount bind-pathseffect, that the PID namespace exists, Slurm's grace period between SIGTERM and SIGKILL (KillWait), how Tapis handlestapisjob.exitcode, and whether the site allows--net --network none(the container shares the host network today; a question for the site operator).neuromechanist commentedon Nov 5, 2026
Merged into the epic branch by #9. Round 1 (Opus checkpoint) and round 2 (Sonnet delta) converged; targeted suites pass (587 deploy and observer tests; launcher gate 1247 pass, coverage 92.43%); every guard mutated. NOTHING IS REGISTERED: registration with any tenant is BLOCKED until the launch-specification wire encoding is made shell-inert in the shared contract, the launcher and the client together (nemarOrg/nemar-cli#2043), and the owner decides the release-asset procedure (immutable releases, tag rulesets, a reviewed release workflow), whether the entry scripts delete the launch files after start, and --no-mount hostfs. Nothing here was verified against a real tenant: docs/unverified.md lists what the definitions assume.