Repository navigation
Description
Goal
NEMAR has an index row for every compute job, two private Cloudflare R2 buckets (Worker bindings, one scoped token for presigning) for what a job submitted and what it produced, a service that writes the per-job record, and one internal function that creates a job and its hold in a single batch.
Nothing submits to Tapis yet.
The record is what an auditor asks for later, so its shape and privacy boundary are fixed here.
Routing
- Readiness: ready
- Implementer: Sonnet
- Opus checkpoint: none
- Depends on: Phase A1: Compute contract and ADR 0118 #2006, Phase A2: Allocation ledger and owner policy #2007 ; Unblocks: Phase A5: Job lifecycle, webhooks, settlement and results #2010, Phase A6: First-party pipeline dispatch #2011
- Repository and base branch: nemarOrg/nemar-cli, epic branch
feature/issue-#2005-epic-compute-core. Do not run in parallel with Phase A4: Tapis client and stand-in #2009: both edittypes/bindings.tsandwrangler-sccn.toml.
Scope
In: migration 0115_compute_jobs.sql; R2 bindings, the S3-API signer and a test S3-API stand-in over Miniflare R2; backend/src/services/compute-records.ts and compute-jobs.ts.
Out: any Tapis call (#2009, #2010), routes and CLI for reading records (#2012), the results bucket's writer (the outputs routes, #2010), compute access (Epic D).
Design references
Design sections 5 (hold), 7 (bundle upload, results), 9 (what is recorded, separation, access). ADR 0118. backend/src/services/s3.ts, testS3EndpointOverride in services/environment.ts, the NEUROBAGEL binding in wrangler-sccn.toml and types/bindings.ts, backend/test/helpers/neurobagel-harness.ts (real Miniflare R2), backend/src/db/audit-log.ts.
Changes
- Migration
0115_compute_jobs.sql(style of0112):compute_jobswithid TEXT PRIMARY KEY(job_plus 24 lowercase hex, length CHECK),account_id INTEGER NULL(NULL means no NEMAR account),tapis_owner TEXT NULL,gateway_user TEXT NULL(copied fromcompute_gateway_userswhen the job is created; NULL for a job with no account),idempotency_key TEXT NULL,source,client_id NULL,kind(from Phase A1: Compute contract and ADR 0118 #2006's words),statedefaultcreated,failure_code NULL,cancel_reason NULL,cancel_requested_at NULL,tapis_uuid TEXT NULLunique,tapis_status NULL,slurm_job_id NULL,app_id,app_version,image_digest NULL,spec_digest NOT NULL,partition,pool,nodes,cores_per_node,memory_mb,gpus,minutes,created_at,submitted_at,remote_submitted_at,started_at,ended_at,tapis_synced_at,records_written_at,hold_ledger_id,held_milli,charged_milli,settled_provisional_at,settled_final_at,flag_unreserved,flag_unattributed,flag_mismatch,flag_overrun,retention_until NULL,legal_hold DEFAULT 0,webhook_nonce NULL. No foreign keys, no PII columns, fixed-word CHECKsNOT GLOB '*[^a-z0-9_]*', a CHECK thatstate = 'failed'if and only iffailure_codeis set, partial unique index on(account_id, idempotency_key)where the key is set, a partial index over live states, an index on(account_id, created_at). A test asserts the CHECK word lists equal the arrays inshared/compute-state.ts. - Bindings:
[[r2_buckets]]and[[env.dev.r2_buckets]]entriesCOMPUTE_RECORDSandCOMPUTE_RESULTS(copy theNEUROBAGELpair; the four buckets already exist:nemar-compute-recordsandnemar-compute-resultsfor production,nemar-compute-records-devandnemar-compute-results-devfor dev, and these exact names go inbucket_name), optionalR2Bucketintypes/bindings.ts. The S3 API needs bucket names, which a binding hides, soCOMPUTE_RECORDS_BUCKET_NAMEandCOMPUTE_RESULTS_BUCKET_NAMEgo in[vars]and[env.dev.vars]both, with the same four names. Worker secretsCOMPUTE_R2_ACCESS_KEY_ID,COMPUTE_R2_SECRET_ACCESS_KEYandCOMPUTE_R2_S3_ENDPOINT(names only in the toml comment block, the account endpoint is not written in this public repository; the owner keeps them in the Infisical folder/backend/compute, indevandprod, and sets them as Worker secrets, never an agent): one R2 API token per environment, Object Read & Write on its two buckets (R2 has no write-only or prefix scoping). Lock and lifecycle rules are bucket configuration, never code. backend/src/services/compute-s3.ts:computeS3(env)returns an aws4fetch client (regionauto) with endpoint and bucket names, ornullwhen anything is missing;presignPut,presignGetandcopyObject(CopyObject,x-amz-copy-source) sign path-style against it, becausegeneratePresignedPutUrlsandgeneratePresignedGetUrlhardcode the AWS host; expiry above 7 days is refused. The endpoint isglobalThis.NEMAR_COMPUTE_S3_URLwhen set, elseCOMPUTE_R2_S3_ENDPOINT(testS3EndpointOverrideis dead in production, so a production-flagged test could not reach a stand-in otherwise; a deployed Worker cannot set a global).- Same migration:
compute_gateway_users(account_id INTEGER PRIMARY KEY, gateway_user TEXT NOT NULL UNIQUE, created_at), withgateway_usergw_plus 20 lowercase base32 characters (CHECK on shape), created withINSERT OR IGNOREin the create-job batch on an account's first job, never rotated and never reused; this is the pseudonymous gateway user Phase A8: Revocation cascade and ACCESS gateway attribution #2013 reports to ACCESS, and the reason it is a stored random value rather than a key derivation is that a derived value would change if the key rotated. compute-records.ts, through the bindings. Records bucket keys:records/<id>/identity.json,spec.json,submitted/bundle.tar.gz(orsubmitted/<path>per file for a partner),bundle-manifest.json,result-manifest.json,provenance.json,result.json;logs/<id>/tapisjob.out,launcher.log;incoming/<id>/bundle.tar.gz(orincoming/<id>/<path>per file for a partner) for uploads in flight. Results bucket keys (Phase A5: Job lifecycle, webhooks, settlement and results #2010 writes them):jobs/<id>/outputs/<path>,jobs/<id>/launcher/....writeIdentitySnapshot(reads theusersrow; writes ORCID iD, name, email, country, affiliation,account_id,sub: null, capture time; nothing else);writeSpec;presignIncomingUploads(env, jobId, files)(one-hour PUTs; one filebundle.tar.gzfor a bundle job, the declared files for a partner; caps frombundleLimits, traversal refused asassertValidS3Keydoes);finalizeBundle(the declared manifest of the files inside the archive, validated for caps and traversal, with the archive's sha256 and size; binding HEAD ofincoming/<id>/bundle.tar.gz, a size mismatch deletes nothing and refuses; then, hold committed,CopyObjectintorecords/<id>/submitted/bundle.tar.gz, HEAD of the copy,records/<id>/bundle-manifest.json,records_written_at) andfinalizeSubmittedFiles(the per-file form for partner uploads);presignBundleRead(env, jobId), a 10-minute presigned GET of that job's ownrecords/<id>/submitted/bundle.tar.gzand no other key, called by Phase A5: Job lifecycle, webhooks, settlement and results #2010's bundle route;readRecordObject(env, db, { actor, jobId, name, reason }), which requires role owner and a reason of 8 to 200 characters and writes thecompute.record.readaudit row BEFORE returning, refusing if that write fails (bytes through the binding, or a 5-minute presigned GET for onesubmitted/file or log, the only presigned read of a record besidespresignBundleRead). Presigned PUTs signx-amz-checksum-sha256when aws4fetch accepts it (comment whether R2 enforces it); else the bound is HEAD size only.compute-jobs.ts:createJob(env, input)validates the spec against policy ceilings andpartition_not_offeredfor non-pipeline sources, computes the hold withestimateSuMilli, then runs ONE batch: Phase A2: Allocation ledger and owner policy #2007's hold INSERT withextraGuards(live-job limit frommaxLiveJobsPerPerson) followed by a job INSERT whose WHERE requires that hold row to exist, so a refused hold leaves no job row. Records are written after commit, never before (a bucket lock would make an orphan record for a refused job permanent);records_written_atis set when they have landed (for an upload job, byfinalizeBundle) and Phase A5: Job lifecycle, webhooks, settlement and results #2010 refuses to submit without it. A replayed(account_id, idempotency_key)returns the existing job. Setsgateway_userfromENCRYPTION_KEYby HKDF with the labelnemar/compute-gateway-user/v1(copyservices/log-pseudonym.ts; the literalnemar-pipelinesfor the service account) andwebhook_noncefromcrypto.randomUUID().- Test stand-in
backend/test/helpers/compute-s3-standin.ts: aBun.servespeaking the path-style S3 subset the signer uses (PUT, GET, HEAD, DELETE,CopyObject, listing) over the same Miniflare R2 buckets the binding paths use (mf.getR2Bucket, asneurobagel-harness.ts), so a presigned PUT and a binding read share one store; it checksX-Amz-Expiresat most 604800, logs requests, and has failure modes. Header note: Miniflare does not enforce bucket lock.
Tests (write these first)
backend/test/compute-jobs-migration.test.ts(bun:sqlite and Miniflare D1): each CHECK refuses its bad row; no column is named for a name, email or country; state words equal Phase A1: Compute contract and ADR 0118 #2006's arrays.backend/test/compute-jobs.test.ts: a refused hold (balance, platform budget, live-job limit) leaves zero job rows, zero ledger rows and no object in either bucket; an accepted one leaves one of each linked byhold_ledger_id; a replayed key returns the same job. Mutation: remove the EXISTS term from the job INSERT; a refused hold must then be seen leaving a job, so the test fails.backend/test/compute-records.test.tson Miniflare R2: the identity snapshot holds exactly the listed fields;readRecordObjectrefuses an admin and a short reason, writes the audit row first (a failing read still leaves it), refuses when the audit insert fails, and gives a 5-minute presigned GET for asubmitted/file;presignBundleReadsigns only that job'sbundle.tar.gzfor 10 minutes (mutation: take the key from the caller).- Same file:
finalizeBundlerefuses a manifest with../, a double-encoded traversal, too many files or an oversized total, and refuses an archive size mismatch; on a match the log shows oneCopyObjectand the copy reads back through the binding withbundle-manifest.jsonbeside it;finalizeSubmittedFilescopies each partner file once. Mutation: copy before the job commits; test 2's refused hold must then leave an object inrecords/. backend/test/compute-buckets-config.test.ts: the toml has both bindings in[[r2_buckets]]and[[env.dev.r2_buckets]]and both*_BUCKET_NAMEvars in both vars blocks set to the four exact bucket names, and no secret name with a value; a production env ignoresS3_ENDPOINT_URLbut honours the global; an expiry above 7 days is refused.
Acceptance criteria
- A job and its hold are created or refused together; refusals leave no trace.
- No identity field exists in D1 job rows; identity lives only in the records bucket.
- Reading identity or a bundle is owner-only, reasoned and audited first.
- Nothing is copied into
records/before the hold commits. -
bun run migrations:d1-checkpasses.
Gates
bun test backend/test/compute-jobs-migration.test.ts backend/test/compute-jobs.test.ts backend/test/compute-records.test.ts backend/test/compute-buckets-config.test.tsbun run migrations:d1-check,bun run typecheck,bunx biome check src shared backend/src,bun test
Rules that bite here
- The repository is public: bucket names are generic, and the R2 account endpoint is a secret, never in the toml.
- Records are immutable once written, and a locked object cannot be overwritten or deleted: never write one for a job that was refused.
- D1 job rows carry the account id and fixed words only; logs carry job ids and fixed words, never a path a person chose.
- Dev and production use different buckets and different R2 tokens.
Escalate to the owner if
aws4fetchcannot sign the checksum header and the owner may prefer a different integrity bound.- Bucket lock or lifecycle rules would be needed in code rather than bucket configuration, or R2 refuses a presigned PUT, a
CopyObjector an expiry the design assumes.
When merged
Update design section 9 with the key layout and the real column list, and add the owner's bucket, lock, lifecycle and R2 token steps to the owner actions.
Activity
neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the phase A1 review (PR #2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.
spec_digestis the SHA-256 hex of the exact bytes ofrecords/<id>/spec.json: the job specification for CLI, web and pipeline jobs, the hold body for a partner job (Phase D4: Partner reservation and report contract #2025). Compute the text once before the INSERT and write the same text after the commit. It records what was asked for, never the launch specification, which carriesjob_token.launch_spec_digest, added by phase A5's migration 0116 (Phase A5: Job lifecycle, webhooks, settlement and results #2010). Do not add it here.failure_code,cancel_reasonandfailure_detailby shape (NOT GLOB '*[^a-z0-9_]*', keeping the 50-character GLOB limit in mind), not with an IN list, so every word a later epic adds does not force a rebuild of the table.manifest.jsonasresult-manifest.json; the node and the results bucket usemanifest.json. There is nofailure.json.neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (nemarOrg/nemar-process#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in nemarOrg/nemar-process#7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).
Routes and records, for the outputs and bundle routes:
maxFilesreach 100,000 outputs while Phase A5: Job lifecycle, webhooks, settlement and results #2010 caps bodies at 1 MiB (roughly 8,000 entries). Reconcile in the plan: derive the cap from the policy'smaxFiles, or boundmaxFilesto what fits. The launcher validates its request against the 1 MiB cap it knows and fails an over-cap job with the refusal detailtoo_many_outputs.COMPUTE_PATH_RE(the same predicate as the JSON Schema) for output paths and do NOT reuseassertValidS3Keyfor output keys: it refuses any..inside a name (a..b.csv) and throws on a stray%(100%.txt), which would fail the upload with a code that refunds the job. Build the key by encoding, and test both names.too_many_outputs,results_too_large,results_quota_exceeded) so the job is charged, never with a word thatisPlatformFailurerefunds.failure_detail, and say howdeclared.jsonand the claimed bytes change.complete.completemay be seen twice (the launcher retries it on a 409results_incompleteand on an ambiguous timeout): it must be idempotent.x-amz-checksum-sha256) on output PUTs is deferred: it must be signed with the URL and verified on real R2 first; decide in epic A, not now.neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the review and Opus checkpoint of phase A2 (the allocation ledger, #2032). This comment is part of the phase brief; where it differs from the body above, this comment wins.
changes() > 0or an INSERT ... SELECT that checks the hold row): a job-count guard that reads rows must run inside the same statement as the hold, or two concurrent submissions race the guard.job:<id>:hold) rather than letting each caller choose one.neuromechanist commentedon Nov 4, 2026
Merged into the epic branch by #2035 (squash
c2bb5279), 2026-11-04, after a two-reviewer Sonnet panel (no Opus checkpoint for this phase); every accepted finding fixed (triage on the pull request). Gates: 466 targeted tests, typecheck and the D1 migration check pass (186 objects); the full CI run is the epic gate.What later phases get, as built: migration 0115 (
compute_jobs,compute_gateway_users, with every primary key NOT NULL and the pseudonym table protected against REPLACE),compute-jobs.ts(createJob: the hold and the job row in one batch, the live-job limit inside the hold statement,spec_digestover the exact recorded bytes, a partner hold body taken as sent),compute-records.ts(records written once, a create-only conditional write, finalize for a bundle and for a partner's per-file upload capped at 500),compute-s3.ts(aws4fetch presigning against R2), the two bucket bindings and their toml pins; ADR 0118 amendment items 1 to 14.Open: real R2 behaviour is unverified (signed checksum header on a presigned PUT,
CopyObject, conditional create, bucket lock); the outputs and complete routes and the job token's validity belong to phase A5; the partner contract (500 files) belongs to phase D. Still owed from this phase's "When merged" step: design page section 9, by a docs pull request.