Repository navigation
Description
Goal
NEMAR can prove that every job in the tenant is accounted for, finalize charges against Expanse's own records, report on any job, person or quarter, and keep records only as long as policy says.
This phase also owns the epic's end-to-end proof: an admin runs a pipeline for a fixture dataset and the job is held, submitted, advanced by webhook and cron, settled, finalized by an observer post, recorded and readable with nemar admin compute record, and revoking a test account mid-run records cancellation.
Routing
- Readiness: ready
- Implementer: Sonnet
- Opus checkpoint: none
- Depends on: Phase A5: Job lifecycle, webhooks, settlement and results #2010, Phase A6: First-party pipeline dispatch #2011, Phase A8: Revocation cascade and ACCESS gateway attribution #2013 ; Unblocks: Epic D
- Repository and base branch: nemarOrg/nemar-cli, epic branch
feature/issue-#2005-epic-compute-core
Scope
In: migration 0118_compute_observer.sql; the tenant sweep; observer ingest and final settlement; admin reporting commands and routes; legal holds; the retention sweep; the end-to-end test.
Out: dispatching the observer as a Tapis FORK application every 15 minutes (needs SDSC's answer, design section 12 question 6; this phase mints the run token and ingests posts), the partner contract and client pools beyond the one tenant partner pool used here (Epic D), admin mail about mismatches (the flag and jobs list --flag are the report), the website.
Design references
Design sections 5 (final settle, tolerance, overrun), 6 (jobs that skipped the reservation), 8 (observer), 9 (records, retention, reporting, access). Templates: routes/callbacks/repull-state.ts (unsigned sender, bounded body, uniform refusals); sweepUnreportedDispatches (watchdog); the daily prodOnlyJobs block in scheduled().
Changes
- Migration
0118_compute_observer.sql:compute_observer_runs(id, nonce, created_at, expires_at, status IN ('issued','ingested','expired'));compute_queue_snapshots(run_id, partition, pending, running, captured_at);compute_sacct_rows(slurm_job_id PRIMARY KEY, run_id, partition, state, elapsed_seconds, alloc_nodes, alloc_cpus, alloc_gpus, alloc_mem_mb, submit_time, start_time, end_time, charged_milli NULL, matched_job_id NULL, ingested_at);compute_legal_holds(id, scope IN ('job','account'), ref, reason fixed word, placed_by, placed_at, released_at NULL). No foreign keys, no PII. - Observer:
POST /admin/compute/observer-runs(admin, audited) mints a one-hour token (domain tagcompute-observerover the run id and nonce);POST /compute/observerinroutes/compute.ts, headerX-Compute-Observer-Token, no session, bounded body (1 MiB, at most 5,000 rows), uniform refusals, one ingest per run. It stores snapshots and rows, then for each row matching a terminal job with a provisional settle and no final one calls Phase A2: Allocation ledger and owner policy #2007'ssettleFinalwithcharged_milliwhen sent, else recomputed bysuMilliForfrom the allocation fields. A difference withinsettle.toleranceMilliortolerancePercentis routine; beyond it setsflag_mismatchand still settles to Expanse's figure. Rows matching no job are storedmatched_job_idNULL and surface injobs list. Pin the route inbackend/test/compute-route-inventory.test.ts. - Tenant sweep
sweepTenantJobs(env, "live" | "week"): lists the tenant withlistType=ALL_JOBSthrough Phase A4: Tapis client and stand-in #2009's client (bounded pages, filter in code, so the unverified search syntax is not relied on), and for each Tapis job not bound bytapis_uuidfirst reads its tags; anemar_job:<id>tag binds an existing row (the submit race) and is not cancelled. Otherwise: cancel at Tapis, insert an index row (sourcepartner,spec_digestthe sha256 of the Tapis fields read,cancel_reasonunreserved, orunattributedwhen the owner maps to no active account, flags set), and write achargeentry (no hold) to the partner pooltenantwith the budget poolpeople, never to a person. Jobs owned byPIPELINES_TAPIS_OWNERwith a row are left alone.liveruns on the 30-minute tick,week(last seven days) in theprodOnlyJobsdaily block, both exact production only and only whencomputeEnabled; entries incron-sweep-wiring.test.ts. - Reporting, routes in
routes/admin/compute.tsand commands insrc/commands/admin-compute-jobs.ts:nemar admin compute jobs list [--flag <name>] [--state] [--account <id>] [--limit] [--after];record <nemar id | tapis uuid | slurm id>(index row, ledger entries, provenance, manifest summary; with--identity --reason <text>or--spec, owner only, through Phase A3: Job records and the results and records buckets #2008'sreadRecordObject);report --account <id> --from --to [--csv];access-report --quarter <yyyy>Q<n> [--csv](distinctgateway_userand job counts by partition and source for jobs submitted in the quarter, unattributed excluded);holdandrelease-holdonjob:<id>oraccount:<id>(owner only, audited), throughPOST /compute/holdsandDELETE /compute/holds/:idunder/admin. - Retention
runComputeRetentionSweep(env)(exact production, daily block, entry in the wiring test): for terminal jobs with no job or account legal hold, deleterecords/<id>/afterretention_untilandlogs/<id>/onceretention.logsDayshave passed sinceended_at, through the binding. A legal hold is a D1 flag the sweep honours (R2 has no versioning and no per-object hold, and the bucket lock covers only the retention period). The sweep never changes a lock; a lifecycle rule cannot delete a locked object, so this sweep is what deletes records, and any refusal on arecords/orlogs/key is countedlocked, whatever its text, and retried the next day. Results expire by the bucket's lifecycle rule, so the sweep only reconciles: a results object still present a day afterretention.resultsDaysis deleted and countedlate, and a hold never keeps a result. Then reduce the index row to anonymous counts (nullaccount_id,tapis_owner,gateway_user,idempotency_key). One audit row of counts and one log line, nothing per job. - End-to-end test
backend/test/compute-e2e.test.tsand a CLI leg that spawns the real CLI against aBun.serveof the real routes (templatetest/admin-repull-auto-cli.test.ts). "Local backend" means that in-process Worker withENVIRONMENTdevelopmentandCOMPUTE_ENABLED1(the configuration staging runs), real D1 and the Tapis, S3-API (over Miniflare R2) and ACCESS stand-ins; jobs advance by webhook andjobs advance, and the production-only crons are exercised by their own tests with a production-flagged env. Include a short comment saying so.
Tests (write these first)
- End to end, in order: fund the pipelines pool,
pipeline runa fixture EEG dataset, hold present, submit seen by the stand-in, the launcher stand-in declares, uploads and completes its outputs through the outputs routes, a webhook hint then a cron tick advance it tofinished, provisional charge, an observer post finalizes within tolerance,record <slurm id>shows the chain, ACCESS stand-in received one attribute. Then a second job, revoke its account mid-run, and assertaccount_ended, a frozen pool and a Tapis cancel after the sweep. - Observer: a wrong, expired and reused token get identical refusals; an in-tolerance row settles silently, an out-of-tolerance row sets
flag_mismatchand still settles; unmatched rows are listed. Mutation: widen the tolerance test to always true; the mismatch case must fail. - Tenant sweep: an untagged job is cancelled, indexed
unreservedand charged to thetenantpool and not to any account pool; a tagged submit-race job is bound, not cancelled; a pipeline-owned job with a row is untouched; a run when Tapis is unreachable changes nothing. - Retention: honours job and account legal holds, counts
locked(a bucket handle whosedeletethrows for locked keys, the one boundary fake, since Miniflare does not enforce locks), never touches a live job; a released hold lets the next run delete; a results object past its date is deleted and countedlate. - Commands:
jobs list --flag unreserved,recordby all three identifiers,--identityrefuses an admin and needs a reason,access-reportcounts a person once however many jobs.
Acceptance criteria
- The end-to-end test passes and is the proof recorded in the epic PR.
- No unreserved job is ever charged to an account pool.
- Identity and bundle reads are owner-only, reasoned and audited; admin views show no identity.
- New crons are production-only, wired and self-fenced.
Gates
bun test backend/test/compute-e2e.test.ts backend/test/compute-observer.test.ts backend/test/compute-sweep.test.ts backend/test/compute-retention.test.ts backend/test/cron-sweep-wiring.test.ts test/admin-compute-jobs-cli.test.ts test/admin-route-inventory.unit.test.tsbun run migrations:d1-check,bun run typecheck,bunx biome check src shared backend/src, then the fullbun test
Rules that bite here
- Nothing the job or the observer posts is trusted beyond its bounds; the observer is an unsigned sender with a per-run token.
- Never charge a person for a job they did not reserve (ADR 0118); an admin may move a charge later, by an
adjustentry. - A bucket lock is not bypassed by code; deletion at period end is by count, logged, never per job.
- Admin views never expose identity; every owner read states a reason.
Escalate to the owner if
- Expanse's accounting cannot be exported and the final settle needs another source.
- The tenant listing returns only the credential's own jobs (the design's open question 2); the fallback is that NEMAR is the only submitter.
When merged
Update design sections 5, 6, 8 and 9 with the shipped names, record the local end-to-end commands and results in the epic pull request, and update the owner runbook for the observer token.
Activity
neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the phase A1 review (PR #2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.
spec_digestis the SHA-256 of aspec.jsonwritten from the Tapis fields NEMAR read, andlaunch_spec_digeststays NULL. The end-to-end stand-in's launcher hashes the value it received from the Tapis stand-in.nemar admin compute recordoutput is passthrough (unknown keys accepted); it never uses a strict compute schema from shared/contract/compute.ts.neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the review and Opus checkpoint of phase A2 (the allocation ledger, #2032). This comment is part of the phase brief; where it differs from the body above, this comment wins.
Retention and anonymisation of the ledger, which this phase owns: the ledger and policy tables are append-only (triggers abort UPDATE and DELETE), and design sections 5 and 9 say records are deleted at the end of the retention period and a deleted account's rows are reduced to anonymous usage counts. Plan the path in this phase with its own migration: allow ONLY an anonymising update (the account id in
compute_pools.holder_refand the actor incompute_ledger.actor_user_idreplaced by a fixed anonymous token) under a narrowly scoped exception, never any change to an amount, a kind, a job or a hold, and never a delete of a row that still has a live hold; state in the plan how the totals invariant (pool totals equal the sums of ledger deltas) stays true. Record the decision in the ADR 0118 amendment and the design page. The A2 migration header says retention is this phase's job.neuromechanist commentedon Nov 5, 2026
Driver amendment, 2026-11-04, from the Opus checkpoints of phases A5 (#2041) and A8 (#2046). This comment is part of the phase brief; where it differs from the body above, this comment wins.
failedstate by their fixed word (outcome_unknown,no_gateway_user,no_submit_time,unconfigured, and the retry-exhausted words) and the tenant sweep must surface them; add an owner-only re-queue command that requires an explicit acknowledgement that the report is confirmed NOT recorded at ACCESS (an unknown outcome is never resent automatically: a report may be lost, never doubled).UNSETTLED_TERMINAL_SQLandUNCOPIED_TERMINAL_SQLorder by age with no rotation (a copy that fails on every attempt holds a slot for three days; ten or more such rows delay newer copies);ORPHAN_ROWS_SQLordering is static for terminal rows;SET_ORPHAN_SQLrecords one orphan per row; the 30-minute tick equals the default reservation wait; terminal rows whose Tapis record is permanently unreadable are retried for ever;CANCEL_ROWS_SQLtakes the oldest ten (ten or more stuck cancels block newer ones, and the revocation cascade adds many intents at once); the staging Worker never cancels at Tapis (the cancel sweep is production-only).frozenfrom the access state), because the revocation cascade and Epic D's suspension both freeze through the same bit.failedwithno_gateway_user: the partner contract (Epic D phase D4) must supply a pseudonym for account-less partner jobs (for example per client and Tapis owner).source = 'partner': the partner contract's); a partner reservation of an ended account is Epic D's to handle.neuromechanist commentedon Nov 6, 2026
Merged into the compute epic (PR 2055, e6ca96f).