Skip to content

Phase C5: Tapis definitions and the observer #6

Closed

Description

@neuromechanist

Goal

The Tapis side of the repository exists as code: the nemar-launch application (ZIP runtime, BATCH) and the nemar-observer application (ZIP runtime, FORK), system templates with placeholders only, JSON Schemas derived from Tapis's own OpenAPI specifications that CI validates them against, a dry-run-by-default registration script, and the observer that summarises the Slurm queue and posts it to NEMAR.

Routing

  • Readiness: ready (the unverified Tapis and SDSC items below wait on the operator; the code takes them as configuration)
  • Implementer: Sonnet
  • Opus checkpoint: review before merge (archive settings, FIXED variables, no credential or host detail, token handling)
  • Depends on: Phase C1: Repository scaffold and the nemar-launch launcher #2 (zip builder, env names, output layout) ; Unblocks: Epic A's Tapis client and observer route integration
  • Repository and base branch: nemarOrg/nemar-process, epic branch feature/issue-#1-epic-process

Scope

In: deploy/tapis/, scripts/derive_tapis_schemas.py, scripts/register.py, observer/, CI validation, docs/operator.md. Out: registering against a real tenant, system credentials, the NEMAR receiving route (Epic A), interactive-session applications.

Design references

Design page sections 7 and 8, section 12 questions 2, 6, 9, 11, 18; implementation notes section 5. Tapis ZIP runtime: containerImage is the archive URL or an absolute path, tapisjob_app.sh is the entry, tapisjob.exitcode is documented for FORK jobs only, and tapisjob.sh and tapisjob.env are archived unless includeLaunchFiles is false.

Changes

  • deploy/tapis/openapi/AppsAPI.yaml and SystemsAPI.yaml vendored from tapis-project/openapi-apps at 10ec91aa31c1 and openapi-systems at 929af4b8855d (BSD-3 text kept as LICENSE-tapis, commits current on 2026-11-03), SOURCE records both.
  • scripts/derive_tapis_schemas.py: extracts ReqPostApp and ReqPostSystem with their $ref closure into deploy/tapis/schemas/app.schema.json and system.schema.json, converting OpenAPI 3.0 to JSON Schema and setting additionalProperties: false on every object (the specifications leave objects open; a typo must fail). CI re-derives and fails on a diff.
  • deploy/tapis/apps/nemar-launch.json: id nemar-launch, version equal to the launcher version, runtime ZIP, containerImage the release-asset URL of nemar-launch-<version>.zip (public), jobType BATCH, strictFileInputs true and no fileInputs, archiveMode ALWAYS, archiveOnAppError true, archiveFilter {includeLaunchFiles: false, excludes: ["tapisjob.env", "tapisjob.sh"]}, NEMAR_LAUNCH_SPEC (the name Phase C1: Repository scaffold and the nemar-launch launcher #2 reads; it carries the per-job upload token) as REQUIRED, FIXED variables for anything the submitter must not change, and notes.launcher_sha256 equal to the reproducible zip's digest (CI rebuilds and compares). Execution system, queue and resources are set by NEMAR at submit time; there are no archive system fields, so Tapis archives to its default on the execution system with launch files excluded.
  • deploy/tapis/apps/nemar-observer.json: runtime ZIP, jobType FORK, the observer zip URL, the same archive filter, variables NEMAR_OBSERVER_URL and NEMAR_OBSERVER_TOKEN as REQUIRED.
  • deploy/tapis/systems/exec.json.tmpl (SSH execution system, Slurm, logical queues shared, compute, gpu, gpu-shared, debug, gpu-debug, preempt with limits from the public Expanse user guide, listed as unverified; static effectiveUserId placeholder). There is no S3 archive system template: results reach NEMAR through the launcher's upload and Tapis holds no storage credential. Placeholders such as ${EXEC_HOST}, ${COMMUNITY_ACCOUNT}, ${EXEC_ROOT_DIR} only; no credential field.
  • scripts/register.py: string.Template rendering from the environment or --vars, unresolved placeholder is an error; dry run by default (method, path, body hash); --apply does GET then POST only when absent and never changes an existing app version; token from TAPIS_TOKEN, base from TAPIS_BASE_URL, never from arguments.
  • observer/observe.py (stdlib, Python 3.9 floor): modes quick and daily. Per partition sinfo -h -o "%P|%a|%D|%T", squeue -h -o "%P|%T|%D|%C|%r"; squeue --start -h -j <ids> for the Slurm ids in NEMAR_OBSERVE_JOBIDS (at most 500, from Tapis remoteJobId); expanse-client balance as captured lines, parsed only when the format is recognised, otherwise {"status": "unparsed", "raw_lines": [...]} and never invented; daily adds sacct -X -P -S <day> -E <day+1> -A <account> (account from NEMAR_SLURM_ACCOUNT) with job id, partition, state, elapsed, allocated TRES, submit, start, end only, no user names. Posts nemar.observation.v1 JSON to NEMAR_OBSERVER_URL with Authorization: Bearer $NEMAR_OBSERVER_TOKEN, never follows a redirect, caps the body at 1 MiB, never prints the token, retries 3 times.
  • .github/workflows/ci.yml additions: derive-and-diff, validate (each app JSON and each rendered template with the example values in deploy/tapis/example.env), launcher zip digest check, observer tests.

Tests (write these first)

  1. The apps validate against the derived schemas; adding a stray key or removing a required key fails. Mutate: drop additionalProperties: false.
  2. Re-deriving gives no diff against the committed schemas; a hand-edited schema fails CI. Mutate: edit one.
  3. Rendering fails on an unresolved placeholder or a value with a newline.
  4. register.py dry run sends no request (a local stand-in records zero); --apply creates once, then is a no-op; an existing app is never overwritten. Mutate: treat 409 as success.
  5. A grep test: no hostname, account, allocation id or home path pattern in deploy/ or observer/, and no archiveSystemId, archiveSystemDir or S3 system anywhere under deploy/. Mutate: add one.
  6. Observer parsers read fixture text in the Slurm manual's formats (stated in a header comment: not captured from a cluster) via stand-in executables on PATH; unknown balance output is unparsed. Mutate: swap two columns.
  7. Observer posts to a local stand-in with the bearer token, refuses a redirect, and the token appears in no output. Mutate: print the headers.

Acceptance criteria

  • Apps and rendered systems validate in CI; schemas reproducible.
  • docs/operator.md and docs/unverified.md written.
  • No credential, host, account or tenant name in the tree.

Gates

uv run python scripts/derive_tapis_schemas.py --check && uv run ruff check . && uv run ruff format --check . && uv run ty check && uv run pytest tests/deploy tests/observer --cov=observer --cov-fail-under=85

Rules that bite here

  • includeLaunchFiles defaults to true and archives tapisjob.env, which holds the launch spec (with the upload token) and the observer token: both apps turn it off.
  • Credentials come only from the environment; register.py never logs them.
  • Strictness is added by the derivation, never by hand edits.

Escalate to the owner if

  • An operator test contradicts the Tapis documentation.
  • Compute nodes cannot make outbound HTTPS requests to NEMAR's API or the R2 S3 endpoint (measured available in October 2026; still a question for SDSC).

When merged

Unverified, waiting on the operator: outbound HTTPS from nodes to the API and the R2 S3 endpoint (question for SDSC); the default archive on the execution system with launch files excluded; BATCH reading of tapisjob.exitcode; strictFileInputs with FIXED variables; maximum environment value length; FORK running Slurm commands on a login node (question 6); expanse-client output format; queue limits; host Python version; SIF compatibility with the cluster's Apptainer; local dataset root layout.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in #7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).

    Application and system definitions must, per the Opus checkpoint of the launcher:

    • set archiveIncludeLaunchFiles to false (Tapis archives tapisjob.env, which holds the per-job token);
    • use a per-job output directory (${JobUUID}) separate from the execution directory, and put jobWorkingDir on scratch under a 0700 root, never $HOME (quota);
    • give end users no Tapis Files access to the execution system;
    • set NEMAR_SIF_ROOT as a FIXED variable and a node-local TMPDIR;
    • pass the validated CUDA_VISIBLE_DEVICES through for GPU jobs;
    • document, as unverified until measured on the node: --no-mount bind-paths effect, that the PID namespace exists, Slurm's grace period between SIGTERM and SIGKILL (KillWait), how Tapis handles tapisjob.exitcode, and whether the site allows --net --network none (the container shares the host network today; a question for the site operator).
  2. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #9. Round 1 (Opus checkpoint) and round 2 (Sonnet delta) converged; targeted suites pass (587 deploy and observer tests; launcher gate 1247 pass, coverage 92.43%); every guard mutated. NOTHING IS REGISTERED: registration with any tenant is BLOCKED until the launch-specification wire encoding is made shell-inert in the shared contract, the launcher and the client together (nemarOrg/nemar-cli#2043), and the owner decides the release-asset procedure (immutable releases, tag rulesets, a reviewed release workflow), whether the entry scripts delete the launch files after start, and --no-mount hostfs. Nothing here was verified against a real tenant: docs/unverified.md lists what the definitions assume.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions