Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f90e5fc
docs: add ADR 0118, the compute gateway
neuromechanist Nov 4, 2026
57e211a
feat: add compute state machine and words
neuromechanist Nov 4, 2026
08a19ba
feat: add compute contract and launch schema
neuromechanist Nov 4, 2026
5f45cf3
feat: add inline job header parser
neuromechanist Nov 4, 2026
574e9a2
docs: note where compute words now live
neuromechanist Nov 4, 2026
71f98cc
fix: refuse a trailing newline in launch schema
neuromechanist Nov 4, 2026
67e2705
test: exempt compute documents from strict guard
neuromechanist Nov 4, 2026
8dc1351
fix: drop over_ceiling and pin state words
neuromechanist Nov 4, 2026
c5414af
fix: refuse an unknown header kind
neuromechanist Nov 4, 2026
c8205e4
fix: tighten rates, GPU rule and default pins
neuromechanist Nov 4, 2026
727f35d
fix: name the launch digest, write ASCII JSON
neuromechanist Nov 4, 2026
cd6caf1
fix: open the launcher's response schemas
neuromechanist Nov 4, 2026
ede0a98
fix: a valid job always fits the launch spec
neuromechanist Nov 4, 2026
9bcc7d5
fix: tighten image uri and sif, move schema id
neuromechanist Nov 4, 2026
da2816a
fix: pin every bound from one table, strict Ajv
neuromechanist Nov 4, 2026
87c98cd
docs: spell out ADR 0118 terms, add the digests
neuromechanist Nov 4, 2026
1ab8bac
test: leave each launcher file out in turn
neuromechanist Nov 4, 2026
681e7a2
test: type the launch budget cases
neuromechanist Nov 4, 2026
301721a
Refuse lone surrogates in a job spec
neuromechanist Nov 4, 2026
9fdc8c5
Guard catches every way to import a strict schema
neuromechanist Nov 4, 2026
f30de76
Classify schemas held in arrays and unions
neuromechanist Nov 4, 2026
33e80df
Pin every launchSpecFits worst-case field
neuromechanist Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .context/compute-program/implementation-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,22 @@ From Tapis v3 documentation and OpenAPI specifications, and the SDSC Expanse use
- **Auth.** Tokens in `X-Tapis-Token`; no client-credentials grant; `POST /v3/tokens` mints user tokens for a service credential (the Drupal `tapis_auth` pattern); which credential NEMAR gets is a question for the operator.
- **Expanse SU.** 1 SU = 1 core-hour (2 GB per core) or 1 GPU-hour (92 GB per GPU); charged on reservation with a 1 SU minimum; whole `compute` node 128 per hour, whole `gpu` node 4 per hour; `preempt` 0.8 and not refunded; 48-hour limit except preempt (7 days); per-account concurrency caps (for example 24 running GPUs in `gpu-shared`).
- **ACCESS attribution.** `POST https://allocations-api.access-ci.org/acdb/gateway/v2/job_attributes` with `jobid`, `submittime`, `xsederesourcename`, `gatewayuser`; `debug` for tests; the key is shown once.

## 6. Words and contracts shipped in phase A1 (ADR 0118)

Import these; never declare a second copy of a word, a number or a state.
A migration that has to spell a word in a `CHECK` copies it from here and a test compares the two.

- **Words and states.** `shared/compute-state.ts`, import-free: the ten job states and the only edges between them (forward and self-edges, `unreported` reachable from every live state), `stateFromTapisStatus` (an unknown status is null, never guessed), the launcher and platform failure codes with `isPlatformFailure(code, detail)` and `failureCodeFromTapisCondition`, the refusal codes with `COMPUTE_REFUSAL_STATUS` (the one table of HTTP statuses), and the ledger kinds, sources, kinds, job flags and cancel reasons.
Every word is snake_case and matches `COMPUTE_WORD_RE`.
- **Money, specification and policy.** `shared/contract/compute.ts` (zod; re-exported from `shared/contract/index.ts`): `JobSpecSchema`, `PARTITIONS`, `RATE_TABLE_V1`, `estimateSuMilli` (the hold, from a specification), `suMilliFor` (a charge, from Tapis's resources and seconds), `parseSuToMilli`, `formatSu`, `ComputePolicySchema` with the frozen `DEFAULT_COMPUTE_POLICY`, and the shapes of the launcher's own files.
Wire keys of a specification are snake_case (`memory_mb`); keys of the policy are camelCase.
A partition with a ceiling in the policy is offered to people; one without is for NEMAR's pipelines only.
- **Launcher.** `shared/contract/compute-launch.ts` and `compute-launch.schema.json`: the launch specification `nemar.launch-spec.v1` (`encodeLaunchSpec` writes it within 64 KiB), the bundle, outputs and outputs-response shapes, the results-bucket keys (`resultsOutputKey`, `resultsLauncherKey`) and, in the schema's `$defs`, `result`, `manifest` and `provenance`.
The schema is shared byte for byte with `nemarOrg/nemar-process`; a test here holds it to the zod schema, and whichever repository merges first writes it.
In the records bucket the launcher's `manifest.json` is kept as `result-manifest.json` (`RECORDS_RESULT_MANIFEST_FILE`; phase A3 builds the key); on the node and in the results bucket it is `manifest.json`, and there is no `failure.json`.
The two responses the launcher reads are open (`.passthrough()`, and no `additionalProperties: false` in the schema); every request and every document is closed.
`encodeLaunchSpec` writes ASCII-only JSON and checks the 64 KiB cap after escaping, and `launchSpecFits(job)` tells the Worker whether a job fits before it takes a hold; a valid job specification always fits.
Its schema also refuses a lone surrogate in the entry, a command item, an argument or an include pattern (`computeIsWellFormed`), because no process argument can carry one.
Two digests: the job row's `spec_digest` (SHA-256 of the exact bytes of `records/<id>/spec.json`, phase A3) and the launcher's `launch_spec_digest` (SHA-256 of the exact raw bytes of the `NEMAR_LAUNCH_SPEC` value, hashed before parsing, recorded in `provenance.json` and stored on the job row by phase A5).
- **Header.** `shared/compute-header.ts`, import-free: `parseJobHeader` for a `# /// nemar` or `% /// nemar` block or a whole `nemar-job.toml` (a small TOML subset, every refusal a fixed code with its line), and `headerToSpecFields`, which maps `memory`, `time` and `id@version` onto specification fields.
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ so a first load is 18.9 MB.
Adding scipy, the next thing a preprocessing step needs, brings it to 35.2 MB, scipy alone being 16.3 MB.
The verdict stands; the cost it accepted is smaller than it said.

**Amendment 2026-11-03:** Stage 3, HPC submission, is specified by [ADR 0118](0118-the-compute-gateway-records-every-tenant-job-and-keeps-the-ledger-in-expanse-service-units.md): who runs a job, who holds its record and how time is counted.
The verdict stands.

Supersedes ADR 0025.
The parts of 0025 that still hold are restated here so this document is the single current statement;
0025 is kept for its history.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# ADR 0118: The compute gateway records every tenant job and keeps the ledger in Expanse service units

**Status:** accepted
**Date:** 2026-11-03
**Owner:** Seyed Yahya Shirazi

Epic #2005, issue #2006.
This Architecture Decision Record (ADR) refines stage 3 of [ADR 0049](0049-compute-runs-in-the-browser-osa-owns-the-runtime-only-hpc-is-gated.md) and keeps [ADR 0040](0040-approval-is-the-single-writer-of-upload-access.md) and [ADR 0042](0042-upload-access-is-requested-once-by-the-person-who-wants-it.md) intact: compute access is its own grant, never `service_access`, and approval is a person's decision.

## Context

ADR 0049 left stage 3, high-performance computing (HPC) submission, as "a NEMAR gateway" without saying who runs a job, who holds its record or how time is counted.
NEMAR now runs jobs on the San Diego Supercomputer Center's (SDSC) Expanse through a Tapis tenant, from three front doors (the command-line interface (CLI), the website, a partner site) and for its own pipelines.
Tapis has no allocations and no idempotency key, signs no webhook, lets a job's owner edit its tags, and a partner site can mint a Tapis token for any username, so the Tapis record alone cannot say who ran what or what it cost.

## Decision

1. **A community account runs every job**, so people need no account of their own in the National Science Foundation's Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, or on Expanse.
2. **NEMAR is the record of every tenant job, whichever front door submitted it**: who ran it, exactly what was submitted, what it cost, what happened and the logs.
3. **The ledger is append-only integer thousandths of an Expanse service unit (SU)**, never floating point.
One SU is one core-hour with up to 2 GB per core, or one graphics processing unit (GPU) hour.
A hold precedes every submit, as one conditional insert that checks the person's pool and the platform budget; a refused hold writes nothing.
4. **A charge settles twice.**
Provisionally from the Tapis record when the job ends, then finally from Expanse's own accounting, as an adjustment; a difference within the owner's policy tolerance is routine and a larger one is reported.
A failure NEMAR caused is refunded; a user-caused one, including a result upload refused for size or quota, is not.
5. **An unreserved partner job is cancelled and charged to the partner's pool, never to a person**, because a Tapis `owner` is not proof of who ran a job.
6. **Webhooks are hints.**
NEMAR reads the job back from Tapis before it acts, polls as a backstop, and tags every job `nemar_job:<id>` so a retried submit finds the job instead of submitting twice.
7. **Retention is a policy row, not a constant**: records 7 years, logs 1 year, results 30 days and never fewer than 14.
Storage is two private Cloudflare R2 buckets, records (locked by prefix) and results (expiring), outside the Amazon Web Services (AWS) account that holds the Open Data buckets.
Only the Worker reaches them, through bindings, and no storage credential reaches Tapis, a node or the launcher: uploads go to addresses the Worker signs for one file of one job.
8. **`COMPUTE_ENABLED` is a dark switch**: exactly `1` and absent from the wrangler configuration. The switch and the test that pins it arrive in phase A4 (issue #2009); every compute cron runs in production only.
9. **NEMAR reports ACCESS gateway attributes for every tenant job**, once, under a per-account pseudonym that is not the sign-in subject.
10. **Every compute word is declared once**, in `shared/compute-state.ts` (states, failure, refusal, ledger and cancel words), `shared/contract/compute.ts` (the job specification, partition and rate tables, owner policy and launcher file shapes) and `shared/contract/compute-launch.ts` (the launch specification and the launcher's routes, with their JSON Schema, which `nemarOrg/nemar-process` copies byte for byte).
11. **Two digests, named apart.** `spec_digest` is the job row's: the SHA-256 of the exact bytes of `records/<id>/spec.json`. `launch_spec_digest` is the launcher's: the SHA-256 of the exact raw bytes of the `NEMAR_LAUNCH_SPEC` value, which is ASCII-only JSON, hashed before parsing.
12. **Responses are open, requests and documents are closed.** The two responses a launcher reads ignore unknown keys, so one that runs for 48 hours survives the Worker adding a key; the job specification, owner policy, launch specification and the launcher's own files refuse them.

## Consequences

- Later phases import these modules instead of declaring their own words, numbers or states, and a test fails when one is added or dropped.
- NEMAR carries a ledger, a record store and retention duties it did not have; the periods are owner policy, so counsel's answer changes a row, not a deploy.
- The rate table (2 GB per core, 92 GB per GPU, preempt at 0.8 with the 1 SU minimum applied after that factor) and the `debug` ceilings are unconfirmed with SDSC and stay marked so until the first charge is compared with Expanse.
- Access, approval and the partner contract are not decided here; they follow ADR 0040 and ADR 0042 in Epics B and D.

## Alternatives considered

- **Per-person Expanse accounts.** Cannot be asked of this audience, and ACCESS does not need them for a community account.
- **The Tapis record as the system of record.** It has no allocation and no stable owner, and a partner can write both owner and tags.
- **Fractional SU in floating point.** A ledger that cannot be summed exactly cannot be reconciled with Expanse's.
1 change: 1 addition & 0 deletions .context/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ invisible one — which is why it is checked rather than trusted.
- [ADR 0115](0115-a-caller-may-ask-for-the-public-face-of-accounts-it-names.md) - A caller may ask for the public face of accounts it names (accepted, maintainer, 2026-10-28): `NemarApiRpc` gains `describeAccounts`, which takes the visitor's own credential, resolved by `resolvePrincipal` and refused as it refuses, and 1 to 100 distinct user ids (the ids are the subject of the read and never its authority), and answers one entry per id in the request's order: `active` (username, given and family name, and the ORCID iD only when it is verified and canonical, `orcid_verified` 1 and `identity_conflict` 0 under ADR 0043), `inactive` (a pending account) or `gone` (revoked, deleted or never there), the last two carrying nothing but the id and the state, and never the email, role, status or any other column; a bad request is `invalid_request`, judged before the credential, maintenance mode is mirrored as a read, faults throw and nothing logged names an account or a credential, except the line `parseRole` already logs for the caller's own unreadable role (ADR 0115, Consequences); `active` is an active status (ADR 0040), not a promise that `resolvePrincipal` would resolve the account, and an id named twice refuses the whole request; in force through this ADR, it clarifies ADR 0078 rule 6 (a caller may keep a user id as a key and must not cache or persist what the method answers) and amends the sentence of rule 7 that names the one use of a key (`resolvePrincipal` and `describeAccounts`)
- [ADR 0116](0116-a-phrase-search-returns-the-datasets-judged-to-answer-it.md) - A phrase search returns the datasets judged to answer it, not every dataset that shares a word (proposed, maintainer, 2026-10-29): a query of two or more content words draws at most 30 candidates from each of the lexical, semantic and name/authors/tasks legs, has `@cf/baai/bge-reranker-base` score the pool, and keeps a dataset the cross-encoder endorses (0.10 and 20% of the best), or both legs found and it does not strongly reject (0.005), or whose name, authors or tasks hold every informative word; a word in over 25% of public datasets does not drive a match; the vector and the cross-encoder read one text (abstract, keywords, modalities, tasks) so the vectors are rebuilt; a word no dataset contains is replaced by a catalog word one typo away and the response says so (`corrected_query`); a modality named in the query is set aside from matching and its datasets are listed first (`modality_boost`), by whole token; filters are applied after the verdict and never change it; every failure serves the fused list and `reranked` marks the cut list; `count` of a phrase is the number judged relevant, a single word is not cut
- [ADR 0117](0117-an-upstream-re-pulls-own-pull-request-is-waived-from-the-review-by-nemars-record.md) - An upstream re-pull's own pull request is waived from the review by NEMAR's record (proposed, maintainer, 2026-10-29, issue #1998; partially supersedes ADR 0107 and ADR 0111 decision 7): the pull request of an open `upstream_repulls` record, from and into the dataset's own repository, from the record's release branch, at the head NEMAR's tools recorded (which they record only after the CLI's enrichment-only check) gets one `skipped` `NEMAR PR Review` check titled "Waived: upstream re-pull" and no row, dispatch or model call, decided by one pure rule (`shared/repull-review-waiver.ts`) before the exemption, a block or any restart, never from a branch name or the pull request's text; a head not yet recorded is reviewed, and once recorded the record's transition and every later publish of that head's review publish the waiver, with the review's findings as recommendations ("Recommended issues, none filed automatically"); `skipped` is now the conclusion of `exempt` and `waived` only; the queue verdict `waived` is derived and never stored; the merge gates, the Worker's machine clean included, accept it without `--force`, and the screen, BIDS validation, `version-check`, `clean` and the merge cap are unchanged; the Worker's own review of the App's re-pull pull request is retired; filing the recommendations as issues is a later step, not built
- [ADR 0118](0118-the-compute-gateway-records-every-tenant-job-and-keeps-the-ledger-in-expanse-service-units.md) - The compute gateway records every tenant job and keeps the ledger in Expanse service units (accepted, maintainer, 2026-11-03, epic #2005, issue #2006; refines ADR 0049 stage 3 and keeps ADR 0040 and ADR 0042 intact): a community account runs every job and NEMAR is the record of every tenant job, whichever front door submitted it; the ledger is append-only integer thousandths of an Expanse service unit (SU) with a hold before every submit, a refused hold writing nothing; a charge is settled provisionally from the Tapis record and finally from Expanse's own accounting within a policy tolerance, a failure NEMAR caused is refunded and a user-caused one is not; an unreserved partner job is cancelled and charged to the partner pool, never the person; webhooks are hints and NEMAR reads the job back; retention (7 years, 1 year, 30 days) is a policy row, and storage is two private Cloudflare R2 buckets with no storage credential outside the Worker; `COMPUTE_ENABLED` is a dark switch, with its test in phase A4; NEMAR reports gateway attributes for every tenant job to the National Science Foundation's Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program; the job row's `spec_digest` and the launcher's `launch_spec_digest` are two digests; the two responses a launcher reads are open and every request and document is closed; every compute word is declared once in `shared/compute-state.ts`, `shared/contract/compute.ts` and `shared/contract/compute-launch.ts`

## Backfill note (2026-07-31)

Expand Down
11 changes: 10 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,16 @@ dataset's own repository, from the record's release branch, at the head NEMAR's
`skipped` check titled "Waived: upstream re-pull" with no review row, dispatch or model call, decided before
the administrator exemption or a block; the queue verdict is `waived` and the merge needs no `--force`, the
Worker's own review of the App's re-pull pull request is retired, and what the rule cannot tell, such as a
human-written `.nemar/` file on the branch, is listed in the ADR).
human-written `.nemar/` file on the branch, is listed in the ADR),
0118 (the compute gateway: a community account runs every job and NEMAR is the record of every tenant job,
whichever front door submitted it; the ledger is append-only integer thousandths of an Expanse service unit with a
hold before every submit, settled provisionally from the Tapis record and finally from Expanse's accounting; an
unreserved partner job is cancelled and charged to the partner pool, never the person; webhooks are hints and NEMAR
reads the job back; retention is a policy row and storage is two private R2 buckets with no storage credential outside
the Worker; `COMPUTE_ENABLED` is a dark switch, with its test in phase A4; the job row's `spec_digest` and the
launcher's `launch_spec_digest` are two digests; the two responses a launcher reads are open and every request and
document is closed; every compute word is declared once in `shared/compute-state.ts`, `shared/contract/compute.ts` and
`shared/contract/compute-launch.ts`).

**Account copy and the profile-gap matrix are declared once, in
[`shared/contract/account-copy.ts`](shared/contract/account-copy.ts) and
Expand Down
Loading
Loading