Skip to content

Epic: Third-party compute: access, user API, nemar compute, partner contract (#2021) - #2116

Open
neuromechanist wants to merge 18 commits into
devfrom
feature/issue-2021-epic-compute-users
Open

neuromechanist wants to merge 18 commits into
devfrom
feature/issue-2021-epic-compute-users

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Epic #2021: third-party compute. Merge as a merge commit (no squash, no admin bypass), after the compute epic (#2063, merged).

What lands

Everything stays dark: COMPUTE_ENABLED and the partner switch are absent from backend/wrangler-sccn.toml and pinned by tests; every compute cron is production-only.

Gates run on this head

  • bun run typecheck, bun run lint, bun run migrations:d1-check (real D1, catalog matches), ADR index test: pass.
  • Backend: OIDC token allowlist, compute route inventory, compute switches, email-code fence, compute end-to-end: 68 pass, 0 fail.
  • Both epic end-to-end suites in one run (test/compute-user-e2e-cli.test.ts, test/compute-partner-e2e.integration.test.ts): 23 pass, 0 fail.
  • Epic-level Opus review of the whole branch: findings fixed in Epic review fixes 1: recovery, cancels, excess #2099 and Reword compute page lede without backticks #2100.

Known red check: website parity

website parity (two tests in test/account-copy-parity.test.ts) fails until the website carries the account copy this PR adds.
The test lists 190 absent website keys: the compute_admin.* and compute_page.* groups (website epic nemarOrg/website#493), 23 compute_access.* keys and 5 gap.* keys.
Running test/account-copy-parity.test.ts with NEMAR_WEBSITE_ACCOUNT_COPY pointed at website PR #511's src/lib/account-copy.ts gives 10 pass, 0 fail, so the exception closes when #511 reaches staging.
This is the one-time parity exception agreed for this release order: this epic, then the website epic into staging. No other check is expected to fail.

Account-facing compute nudges wait for the terms (owner decision)

The owner ruled that third-party compute must not confuse anyone before it is open, so three things that would have changed what every existing account sees are now gated on COMPUTE_TERMS_PUBLISHED (no new flag):

  • The affiliation gap, and compute_access among any gap's blocks, in profile_gaps (GET /users/me, GET /auth/me): one rule, computeProfileGaps. While the terms are unpublished profile_gaps reads exactly as it did before this epic. A compute request's own refusal still lists what it needs.
  • The nemar auth signup institution prompt (an explicit --affiliation is still saved).
  • The nemar auth profile "Compute access" row.

They appear when the owner flips the constant. The copy keys are unchanged, so the website needs no change.

Before merging into dev

  • Re-run bunx cfman wrangler --account sccn d1 migrations list nemar-db-dev --remote --env dev -c backend/wrangler-sccn.toml and confirm 0131, 0132 and 0134 are still unapplied. Migration 0134 was edited in place on this branch; if it shows applied, a new migration is needed for the two columns.
  • There is no 0133: no migration was needed for that number, so the gap is not a missing file.

Not part of this PR

D5 (#2026): the compute terms and privacy changes stay a held draft in nemarOrg/docs (#105) until counsel approves them. The Tapis tenant, buckets, secrets, service account and pool funding are owner actions; nothing here runs against a real tenant.

* Add compute access columns and state machine

* Add compute gap block, copy and account fields

* Add compute_requests email category and mails

* Add compute access request route

* Add compute access decision routes

* Cascade compute access, guard test-account sweep

* Add nemar compute request-access and access admin

* Pin compute access scan and /auth/me fields

* Record compute access in ADR 0118 and 0119

* Declare the notified stamp on the admin account view

* Offer affiliation in profile and signup

* Record review decisions in ADR 0118

* Fix compute access review findings

* Fix tests broken by compute access changes
* Add compute user contract and queue estimates

* Add compute read routes and credential rule

* Add compute read indexes

* Add nemar compute read commands

* Record the compute read routes in ADR 0118 and 0119

* Test the stale refresh guards

* Debounce the live job refresh per job

* Page jobs by id, not by row number

* Cache queue estimates for 30 seconds

* Test grant, client and maintenance paths

* Word the frozen note by access state

* Document stats bases and review decisions

* Make the job list index partial so Epic A plans hold
* Add submit and bundle contracts and words

* Share dataset gate; guard access in the hold

* Release abandoned uploads on the tick

* Add person job routes and submit tests

* Test finalize, release, logs, results, cancel

* Let the job's replay judge the request hash

* Pin the order for paused and full accounts

* Document the submit amendment and notes

* Tidy submit tests; address expiry test

* Review fixes: upload window, CORS, races, signed length

* Update the ADR for the review round

* Format review-round files

* Bind the sweep's terminal write to the new term

* Make the new terminal term null-safe

* Keep the submit responses open; requests in own file

* Point the notes at the request file
* Add person job-submission API client calls

* Add job header and flag resolution

* Add deterministic bundle packer

* Add credential scan for bundles

* Add --wait polling for compute jobs

* Add bundle upload with retries

* Add local job preparation for compute submit

* Add compute submit, logs, results and cancel commands

* Test result downloads and log display

* Test compute submit on this machine

* Test one set of bundle caps in CLI and Worker

* Test compute commands against the real routes

* Document the compute submit CLI half

* Harden results download writes and links

* Harden the bundle upload address and redirects

* Make the redirect test see the landing page

* Report unreadable files as local refusals

* Scan UTF-16 text and more credential file names

* Read UTF-16 without TextDecoder and test each view

* Say no job data left the machine; note rerun cost

* Bound the wait: ceiling, 429, hung reads, second Ctrl-C

* Make a runaway wait fail fast in tests

* Discriminate the packer's failures by reason

* Format an unrelated test file

* Test --force through the real routes

* Record the hardened wait and download rules

* Unset TEST_API_URL by deleting it in a test
* Add migration 0134 for the partner contract

* Add the partner contract files and words

* Take client caps and key claims in createJob

* Keep unverified partner jobs out of the lifecycle

* Add the partner routes and verification

* Add the hold reaper and the sweep's deferral

* Add the move-charge operation and command

* Test the move of an unreserved job's charge

* Pin the reserved-job term of the move

* Test the partner credential matrix

* Test the partner reservation

* Fix the broken-policy case of the reservation test

* Test the access guard of the hold itself

* Test the partner report and its verification

* Pin the price check and the unique bind

* Pin the unique-bind backstop

* Test the partner release

* Pin the release statement's own term

* Test the hold reaper and the sweep's deferral

* Pin the partner routes in the inventories

* Add the partner end-to-end test

* Record the partner contract in the ADRs

* Lint the partner tests and drop the smoke test

* Keep a cancel when a binding is withdrawn

* Fix the withdrawn-binding test

* Add the partner policy block and review words

Adds policy.partner (visibility confirmation, execution system), the words tapis_job_unreadable, partner_not_enabled and unverifiable, the hold's state, the Tapis id the table keeps, and the columns partner_expires_at and partner_files_digest. Contract and schema tests pass; behavior follows in the next commits.

* Verify partner reports against what NEMAR reads

A job NEMAR may not read binds nothing and one unreadable for two hours is refused as unverifiable; scheduler options and the execution system are checked; a request claims its key only after its cheap checks; a refused job owes no ACCESS report and no log copy; a same-key request that lost a race is answered with the winner; the contract is closed until the owner confirms. Tests: unverifiable, holds, report, release, e2e (mutation-checked).

* Reap and shelter by stored expiry and policy

The reaper uses the stored expiry and named indexes, never releases on an empty search until the owner confirms, counts only the reservation owner's jobs toward ambiguity and prunes old idempotency keys. The sweep shelters only a sole, matching, owner-agreeing job and removes it before its slice, with a backlog-aware window. Tests: hold-reaper, migration (plans pinned).

* Cap a partner job's final charge at its hold

The observer charges the person at most the hold; the excess is an overrun of the partner pool, flagged and audited once. Test: compute-partner-final (mutation-checked).

* Hide an unverified partner job's logs

The logs route answers logs_not_available for a partner row that is not verified. Test: compute-partner-logs.

* Pin and compare a partner's declared files

The store's SHA-256 is compared with the declared one where it is returned, a copy refused because a concurrent report wrote the destination counts as copied, and the manifest comment is corrected. Test: compute-partner-files.

* Require proof of owner to move a charge

Moving a charge to an account the job was not attributed to needs --confirm-unproven-owner; the audit row and the answer say whether the owner was proven; success is judged by changes > 0. Tests: move-charge, ledger-budget, CLI.

* Record the partner review round in the ADRs

ADR 0118 items 17 to 24, the ADR 0119 words and implementation-notes section 16. Documentation only.

* Read parameterSet in both wire forms

The Jobs specification types parameterSet as a JSON string; the accessor parses it under a size bound and still reads the object form. The Tapis stand-in now serves the string by default. Tests: compute-partner-scheduler (mutation-checked); tapis-client and partner suites pass.

* Accept owner-listed scheduler options

policy.partner.allowedSchedulerArgs lists exact option strings accepted besides at most one --gpus=N within the hold; GPU options cannot be listed. Test: compute-partner-allowed-args (mutation-checked).

* Refuse unverifiable jobs only on sustained failure

A bound job is refused as unverifiable after three failed reads on separate ticks spanning two hours, counted on the row; a refusal that can be read later and matches is verified again instead of cancelled. A report binds pending only for an unavailable Tapis. A cancel Tapis definitively refuses ends the row, and an unread job is charged the whole hold to the partner. Tests: compute-partner-unverifiable, migration (mutation-checked).

* Count the shelter and ambiguity the same way

The sweep's tag search keeps only jobs the reservation's Tapis user owns that are not over and not accounted, and the reaper ignores an ended first attempt when a live retry exists. Tests: compute-hold-reaper (mutation-checked).

* Guard the partner switch and unfreeze the stale

A policy write that drops the partner confirmation while partner reservations are open is refused unless --force-partner-off, which says how many are frozen. While it is off, a reservation a day past expiry whose person ended or cancelled is released without a search, and a repeat of a hold's key still replays. Tests: compute-partner-switch, admin-compute-cli (mutation-checked).

* Check the sweep's shelter lazily

The shelter is evaluated while the run's slice fills and stops reading when it is full, instead of reading up to 100 candidates before every slice. Test: compute-hold-reaper (mutation-checked).

* Cap a partner's provisional charge at its hold

The provisional settlement charges the person at most the hold and puts the rest on the partner through one shared helper, which the final settlement uses too; the flag word and audit row need a difference beyond the owner's tolerance, and the final stamp reads the person's pool only. A job with no Slurm id and no final is charged to the person by its provisional settlement alone, never above the hold. Tests: compute-partner-final (mutation-checked).

* Document the second partner review round

Migration 0134 header (seven columns, all flag words, the 7-day prune), ADR 0118 items 25 to 33, implementation notes, and the staging note that the development Worker's partner policy stays unconfirmed. Documentation and comments only.

* Rename the partner e2e to the integration tier

Its header claims integration but the workflow routes by file name or marker, and the file holds no marker; the .integration.test.ts suffix is the convention for such suites. Tested: ci-tier-routing and the renamed suite pass.

* Pin moveComputeCharge in the export surface

The api/compute module gained moveComputeCharge in phase D4; it is added to the module pin and the post-split additions. Tested: api-export-surface and ci-tier-routing pass.
* Add compute page copy keys

* Read balance sentences from ACCOUNT_COPY

* Do not promise an allocation or a human review

* Read the queue note in the CLI help

* Soften the compute access lede

* Soften the remaining allocation promises
* Add compute_admin copy keys

* Reword compute_admin copy and add error keys
* Keep a later cancel through a refusal

A refusal no longer writes cancel_requested_at or cancel_reason (partner_refused_at is its clock), so a person's, an administrator's or an account-end cancel recorded after it survives a recovery or a re-binding; recovery and verification re-check the person's standing in their own statements, the request path sends no blind cancel and a blind cancel is stamped before it is sent (partner_cancel_sent_at) which forbids a recovery, a report that triggers a recovery is answered verified, and the failed-read counters reset at every binding. Tests: compute-partner-recovery (mutation-checked).

* Write a partner's excess from the final figure

The provisional settlement only caps the person at the hold for a job with a Slurm id; the observer's final figure writes the partner's excess, and a job with no Slurm id keeps the provisional excess. The excess entry is gated on the person's charge being exactly the hold. Tests: compute-partner-final (mutation-checked).

* Refuse every GPU-like option in the owner's list

allowedSchedulerArgs refuses any gpu, gres or tres token in any case, -G, and abbreviations of --gpus, --gres and --tres, with a test of near neighbors that stay allowed. The count in partner_reservations_open names only what the switch freezes (unbound reservations). Tests: compute-partner-allowed-args, compute-partner-switch (mutation-checked).

* Say what a suspension leaves for partners

The suspend and revoke consequences and results now say an unreported partner reservation is left to expire, and the mismatch flag meaning covers partner refusals and capped excess. Changed keys: compute_admin.dialog.consequence.suspend, .revoke; compute_admin.result.suspended, .revoked; compute_admin.flag.meaning.mismatch. Parity with the website copy is a follow-up (already red on this branch).

* Record the epic-level review in the ADR

ADR 0118 items 34 to 40 (a refusal records no cancel intent, recovery conditions, the counters, who writes the excess) and the implementation notes. Documentation only.

* Refuse to verify a job the person cancelled

VERIFIED_SQL needs no cancel recorded; verifyBoundJob refuses a matching job with a cancel recorded (or an ended person) as hold_ended, and the caller cancels it at Tapis so it stops and the partner pays. Tests: compute-partner-recovery, including the person-cancel and suspend/re-grant cases and races on the verifying statement's own cancel and standing terms (mutation-checked).

* Stamp every cancel of an unverifiable row

The readable cancel path stamps partner_cancel_sent_at before it is sent, as the blind one does, and sends nothing if a recovery committed first. Test: compute-partner-recovery (mutation-checked).

* Refuse a provisional excess once the job has a Slurm id

The provisional excess statement requires no Slurm id in the statement itself, so a commit built from a stale record cannot write an excess the final settlement could never correct. Test: compute-partner-final with a concurrent commit injected after the cap is decided (mutation-checked).

* Make a silent provisional cap visible

A partner job with a Slurm id capped at its hold is marked (flag_overrun, partner_flag capped_provisional) with one audit row and no ledger entry; the final settlement clears or replaces the word; the status view counts capped jobs whose final is a week overdue (partner_capped_no_final). Tests: compute-partner-final, compute-status-route (mutation-checked).

* Guard a policy change that loosens bound jobs

A write that drops the partner block, or changes its execution system or listed scheduler options, while bound pending or unverifiable jobs wait is refused with the bound count named apart from the open count, unless forced. Tests: compute-partner-switch (mutation-checked).

* Judge scheduler option names, not values

schedulerOptionNamesGpus looks at the option name only, so values such as --account=progress are allowed while gpu, gres, tres names, -G and abbreviations stay refused; the schema's doc comment is back above the schema; the policy set help and ADR 0118 items 41 to 44 say which scheduler options an owner must never list. Tests: compute-partner-allowed-args (mutation-checked).

* Tell the person a refused job is being cancelled

cancel_requested reads true in the person's and the administrator's job view while a refused partner job (partner_refused_at set) has not ended, as it did before a refusal stopped recording a cancel intent; stale comments about a refusal's cancel intent are corrected; the vacuous adoption-reset test is replaced by one that isolates BIND_ADOPT_SQL's reset. Tests: compute-partner-recovery (mutation-checked).
* Document the compute job environment in help

* Keep third-party compute apart from NEMAR's own pipelines in help; test no first-party flags

* Read the bundle back without Python: own ustar reader plus system tar
…c-compute-users

# Conflicts:
#	.context/decisions/0119-nemar-is-an-openid-connect-provider-and-orcid-stays-the-upstream-identity.md
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR2116

Version: 0.10.33-PR2116.82ddd0a
Commit: 82ddd0a


🤖 Updated on each push.

computeProfileGaps hides the affiliation gap and the compute_access block while COMPUTE_TERMS_PUBLISHED is false; an explicit compute_access request still lists them. Tested: matrix and route tests.
nemar auth signup no longer asks for an institution, and nemar auth profile omits the Compute access row, until the compute terms are published. Tested: unit and CLI tests.
@neuromechanist

Copy link
Copy Markdown
Contributor Author

Merge notes for whoever merges dev into this branch after the tenant ceiling (#2126) and the pipeline caps (#2122) land on dev. A trial merge of #2126 into this branch conflicts in 8 files, including both the guards block and the refusal-wording tail of createJob in backend/src/services/compute-jobs.ts.

  • Keep the pipeline guard, the tenant guard and the guardRefusal ordering in the guards block. The person and partner cases of the tenant suite catch a dropped guard.
  • The refusal tail on this branch ends with return refusal("over_concurrency"). If that tail is kept and only the tenant guard is added, every tenant refusal is worded over_concurrency. Use guardRefusal's result instead.
  • The access guard here needs two words (suspended or no_compute_access), but a NamedGuard holds one fixed word. It has to keep its own read before the others.
  • platform_busy must be added to this branch's closed partner table and sentences (COMPUTE_PARTNER_ERROR_STATUS and COMPUTE_PARTNER_SENTENCES, where "a word outside the list is a bug in the Worker"), to COMPUTE_SUBMIT_SENTENCES for people, and as a step of the reserve order in compute-partner.ts (after the client caps).
  • The resulting order of refusals is: access, suspended, balance, budget, person cap, client caps, tenant. That matches "tenant last".

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Merge notes from the portable compute epic (#2145, branch feature/portable-compute-epic), for whichever of the two lands second.
The merge checklist:

  • resolveRun(db, id, wanted, except, recipe) keeps that parameter order, with both sides' callers.
  • CANCEL_PENDING_FAILED_SQL keeps AND executor = 'tapis' at the top level, outside the partner OR.
  • The JOB_INSERT_SQL pin is re-baselined from this pull request's text.
  • This pull request's ABANDONED_UPLOADS_SQL, and any new selection of created AND tapis_uuid IS NULL, gain executor = 'tapis'.
  • Then bun test backend/test/compute-runner-jobs.test.ts backend/test/compute-runner-queue.test.ts and the source-search tests pass.

This epic (portable compute, #2145) keeps migrations 0140 to 0149 (owner, Q5); a migration of yours that lands after it takes a number above 0149.

This branch was previously deployed

1 inactive deployment
npm — 82ddd0a0 Deployed Nov 10, 2026 by neuromechanist via publish #4382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant