Skip to content

Phase A3: Job records and the results and records buckets #2008

Closed

Description

@neuromechanist

Goal

NEMAR has an index row for every compute job, two private Cloudflare R2 buckets (Worker bindings, one scoped token for presigning) for what a job submitted and what it produced, a service that writes the per-job record, and one internal function that creates a job and its hold in a single batch.
Nothing submits to Tapis yet.
The record is what an auditor asks for later, so its shape and privacy boundary are fixed here.

Routing

Scope

In: migration 0115_compute_jobs.sql; R2 bindings, the S3-API signer and a test S3-API stand-in over Miniflare R2; backend/src/services/compute-records.ts and compute-jobs.ts.
Out: any Tapis call (#2009, #2010), routes and CLI for reading records (#2012), the results bucket's writer (the outputs routes, #2010), compute access (Epic D).

Design references

Design sections 5 (hold), 7 (bundle upload, results), 9 (what is recorded, separation, access). ADR 0118. backend/src/services/s3.ts, testS3EndpointOverride in services/environment.ts, the NEUROBAGEL binding in wrangler-sccn.toml and types/bindings.ts, backend/test/helpers/neurobagel-harness.ts (real Miniflare R2), backend/src/db/audit-log.ts.

Changes

  • Migration 0115_compute_jobs.sql (style of 0112): compute_jobs with id TEXT PRIMARY KEY (job_ plus 24 lowercase hex, length CHECK), account_id INTEGER NULL (NULL means no NEMAR account), tapis_owner TEXT NULL, gateway_user TEXT NULL (copied from compute_gateway_users when the job is created; NULL for a job with no account), idempotency_key TEXT NULL, source, client_id NULL, kind (from Phase A1: Compute contract and ADR 0118 #2006's words), state default created, failure_code NULL, cancel_reason NULL, cancel_requested_at NULL, tapis_uuid TEXT NULL unique, tapis_status NULL, slurm_job_id NULL, app_id, app_version, image_digest NULL, spec_digest NOT NULL, partition, pool, nodes, cores_per_node, memory_mb, gpus, minutes, created_at, submitted_at, remote_submitted_at, started_at, ended_at, tapis_synced_at, records_written_at, hold_ledger_id, held_milli, charged_milli, settled_provisional_at, settled_final_at, flag_unreserved, flag_unattributed, flag_mismatch, flag_overrun, retention_until NULL, legal_hold DEFAULT 0, webhook_nonce NULL. No foreign keys, no PII columns, fixed-word CHECKs NOT GLOB '*[^a-z0-9_]*', a CHECK that state = 'failed' if and only if failure_code is set, partial unique index on (account_id, idempotency_key) where the key is set, a partial index over live states, an index on (account_id, created_at). A test asserts the CHECK word lists equal the arrays in shared/compute-state.ts.
  • Bindings: [[r2_buckets]] and [[env.dev.r2_buckets]] entries COMPUTE_RECORDS and COMPUTE_RESULTS (copy the NEUROBAGEL pair; the four buckets already exist: nemar-compute-records and nemar-compute-results for production, nemar-compute-records-dev and nemar-compute-results-dev for dev, and these exact names go in bucket_name), optional R2Bucket in types/bindings.ts. The S3 API needs bucket names, which a binding hides, so COMPUTE_RECORDS_BUCKET_NAME and COMPUTE_RESULTS_BUCKET_NAME go in [vars] and [env.dev.vars] both, with the same four names. Worker secrets COMPUTE_R2_ACCESS_KEY_ID, COMPUTE_R2_SECRET_ACCESS_KEY and COMPUTE_R2_S3_ENDPOINT (names only in the toml comment block, the account endpoint is not written in this public repository; the owner keeps them in the Infisical folder /backend/compute, in dev and prod, and sets them as Worker secrets, never an agent): one R2 API token per environment, Object Read & Write on its two buckets (R2 has no write-only or prefix scoping). Lock and lifecycle rules are bucket configuration, never code.
  • backend/src/services/compute-s3.ts: computeS3(env) returns an aws4fetch client (region auto) with endpoint and bucket names, or null when anything is missing; presignPut, presignGet and copyObject (CopyObject, x-amz-copy-source) sign path-style against it, because generatePresignedPutUrls and generatePresignedGetUrl hardcode the AWS host; expiry above 7 days is refused. The endpoint is globalThis.NEMAR_COMPUTE_S3_URL when set, else COMPUTE_R2_S3_ENDPOINT (testS3EndpointOverride is dead in production, so a production-flagged test could not reach a stand-in otherwise; a deployed Worker cannot set a global).
  • Same migration: compute_gateway_users(account_id INTEGER PRIMARY KEY, gateway_user TEXT NOT NULL UNIQUE, created_at), with gateway_user gw_ plus 20 lowercase base32 characters (CHECK on shape), created with INSERT OR IGNORE in the create-job batch on an account's first job, never rotated and never reused; this is the pseudonymous gateway user Phase A8: Revocation cascade and ACCESS gateway attribution #2013 reports to ACCESS, and the reason it is a stored random value rather than a key derivation is that a derived value would change if the key rotated.
  • compute-records.ts, through the bindings. Records bucket keys: records/<id>/identity.json, spec.json, submitted/bundle.tar.gz (or submitted/<path> per file for a partner), bundle-manifest.json, result-manifest.json, provenance.json, result.json; logs/<id>/tapisjob.out, launcher.log; incoming/<id>/bundle.tar.gz (or incoming/<id>/<path> per file for a partner) for uploads in flight. Results bucket keys (Phase A5: Job lifecycle, webhooks, settlement and results #2010 writes them): jobs/<id>/outputs/<path>, jobs/<id>/launcher/.... writeIdentitySnapshot (reads the users row; writes ORCID iD, name, email, country, affiliation, account_id, sub: null, capture time; nothing else); writeSpec; presignIncomingUploads(env, jobId, files) (one-hour PUTs; one file bundle.tar.gz for a bundle job, the declared files for a partner; caps from bundleLimits, traversal refused as assertValidS3Key does); finalizeBundle (the declared manifest of the files inside the archive, validated for caps and traversal, with the archive's sha256 and size; binding HEAD of incoming/<id>/bundle.tar.gz, a size mismatch deletes nothing and refuses; then, hold committed, CopyObject into records/<id>/submitted/bundle.tar.gz, HEAD of the copy, records/<id>/bundle-manifest.json, records_written_at) and finalizeSubmittedFiles (the per-file form for partner uploads); presignBundleRead(env, jobId), a 10-minute presigned GET of that job's own records/<id>/submitted/bundle.tar.gz and no other key, called by Phase A5: Job lifecycle, webhooks, settlement and results #2010's bundle route; readRecordObject(env, db, { actor, jobId, name, reason }), which requires role owner and a reason of 8 to 200 characters and writes the compute.record.read audit row BEFORE returning, refusing if that write fails (bytes through the binding, or a 5-minute presigned GET for one submitted/ file or log, the only presigned read of a record besides presignBundleRead). Presigned PUTs sign x-amz-checksum-sha256 when aws4fetch accepts it (comment whether R2 enforces it); else the bound is HEAD size only.
  • compute-jobs.ts: createJob(env, input) validates the spec against policy ceilings and partition_not_offered for non-pipeline sources, computes the hold with estimateSuMilli, then runs ONE batch: Phase A2: Allocation ledger and owner policy #2007's hold INSERT with extraGuards (live-job limit from maxLiveJobsPerPerson) followed by a job INSERT whose WHERE requires that hold row to exist, so a refused hold leaves no job row. Records are written after commit, never before (a bucket lock would make an orphan record for a refused job permanent); records_written_at is set when they have landed (for an upload job, by finalizeBundle) and Phase A5: Job lifecycle, webhooks, settlement and results #2010 refuses to submit without it. A replayed (account_id, idempotency_key) returns the existing job. Sets gateway_user from ENCRYPTION_KEY by HKDF with the label nemar/compute-gateway-user/v1 (copy services/log-pseudonym.ts; the literal nemar-pipelines for the service account) and webhook_nonce from crypto.randomUUID().
  • Test stand-in backend/test/helpers/compute-s3-standin.ts: a Bun.serve speaking the path-style S3 subset the signer uses (PUT, GET, HEAD, DELETE, CopyObject, listing) over the same Miniflare R2 buckets the binding paths use (mf.getR2Bucket, as neurobagel-harness.ts), so a presigned PUT and a binding read share one store; it checks X-Amz-Expires at most 604800, logs requests, and has failure modes. Header note: Miniflare does not enforce bucket lock.

Tests (write these first)

  1. backend/test/compute-jobs-migration.test.ts (bun:sqlite and Miniflare D1): each CHECK refuses its bad row; no column is named for a name, email or country; state words equal Phase A1: Compute contract and ADR 0118 #2006's arrays.
  2. backend/test/compute-jobs.test.ts: a refused hold (balance, platform budget, live-job limit) leaves zero job rows, zero ledger rows and no object in either bucket; an accepted one leaves one of each linked by hold_ledger_id; a replayed key returns the same job. Mutation: remove the EXISTS term from the job INSERT; a refused hold must then be seen leaving a job, so the test fails.
  3. backend/test/compute-records.test.ts on Miniflare R2: the identity snapshot holds exactly the listed fields; readRecordObject refuses an admin and a short reason, writes the audit row first (a failing read still leaves it), refuses when the audit insert fails, and gives a 5-minute presigned GET for a submitted/ file; presignBundleRead signs only that job's bundle.tar.gz for 10 minutes (mutation: take the key from the caller).
  4. Same file: finalizeBundle refuses a manifest with ../, a double-encoded traversal, too many files or an oversized total, and refuses an archive size mismatch; on a match the log shows one CopyObject and the copy reads back through the binding with bundle-manifest.json beside it; finalizeSubmittedFiles copies each partner file once. Mutation: copy before the job commits; test 2's refused hold must then leave an object in records/.
  5. backend/test/compute-buckets-config.test.ts: the toml has both bindings in [[r2_buckets]] and [[env.dev.r2_buckets]] and both *_BUCKET_NAME vars in both vars blocks set to the four exact bucket names, and no secret name with a value; a production env ignores S3_ENDPOINT_URL but honours the global; an expiry above 7 days is refused.

Acceptance criteria

  • A job and its hold are created or refused together; refusals leave no trace.
  • No identity field exists in D1 job rows; identity lives only in the records bucket.
  • Reading identity or a bundle is owner-only, reasoned and audited first.
  • Nothing is copied into records/ before the hold commits.
  • bun run migrations:d1-check passes.

Gates

  • bun test backend/test/compute-jobs-migration.test.ts backend/test/compute-jobs.test.ts backend/test/compute-records.test.ts backend/test/compute-buckets-config.test.ts
  • bun run migrations:d1-check, bun run typecheck, bunx biome check src shared backend/src, bun test

Rules that bite here

  • The repository is public: bucket names are generic, and the R2 account endpoint is a secret, never in the toml.
  • Records are immutable once written, and a locked object cannot be overwritten or deleted: never write one for a job that was refused.
  • D1 job rows carry the account id and fixed words only; logs carry job ids and fixed words, never a path a person chose.
  • Dev and production use different buckets and different R2 tokens.

Escalate to the owner if

  • aws4fetch cannot sign the checksum header and the owner may prefer a different integrity bound.
  • Bucket lock or lifecycle rules would be needed in code rather than bucket configuration, or R2 refuses a presigned PUT, a CopyObject or an expiry the design assumes.

When merged

Update design section 9 with the key layout and the real column list, and add the owner's bucket, lock, lifecycle and R2 token steps to the owner actions.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the phase A1 review (PR #2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • spec_digest is the SHA-256 hex of the exact bytes of records/<id>/spec.json: the job specification for CLI, web and pipeline jobs, the hold body for a partner job (Phase D4: Partner reservation and report contract #2025). Compute the text once before the INSERT and write the same text after the commit. It records what was asked for, never the launch specification, which carries job_token.
    • The launcher's own digest is a different column, launch_spec_digest, added by phase A5's migration 0116 (Phase A5: Job lifecycle, webhooks, settlement and results #2010). Do not add it here.
    • Check failure_code, cancel_reason and failure_detail by shape (NOT GLOB '*[^a-z0-9_]*', keeping the 50-character GLOB limit in mind), not with an IN list, so every word a later epic adds does not force a rebuild of the table.
    • The records bucket keeps the launcher's manifest.json as result-manifest.json; the node and the results bucket use manifest.json. There is no failure.json.
  2. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (nemarOrg/nemar-process#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in nemarOrg/nemar-process#7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).

    Routes and records, for the outputs and bundle routes:

    • The outputs request body cap and the contract disagree: the owner policy lets maxFiles reach 100,000 outputs while Phase A5: Job lifecycle, webhooks, settlement and results #2010 caps bodies at 1 MiB (roughly 8,000 entries). Reconcile in the plan: derive the cap from the policy's maxFiles, or bound maxFiles to what fits. The launcher validates its request against the 1 MiB cap it knows and fails an over-cap job with the refusal detail too_many_outputs.
    • One path predicate on both sides: use the contract's COMPUTE_PATH_RE (the same predicate as the JSON Schema) for output paths and do NOT reuse assertValidS3Key for output keys: it refuses any .. inside a name (a..b.csv) and throws on a stray % (100%.txt), which would fail the upload with a code that refunds the job. Build the key by encoding, and test both names.
    • A refusal caused by the submitter's output (count, size, quota, bad name) answers with the matching refusal word (too_many_outputs, results_too_large, results_quota_exceeded) so the job is charged, never with a word that isPlatformFailure refunds.
    • A second outputs request that names no outputs is the launcher's follow-up after a refusal (it uploads only the four launcher files): accept it, never clear failure_detail, and say how declared.json and the claimed bytes change.
    • Upload addresses live one hour and the launcher re-requests an expired address (bounded: 2 per object, 50 per run): re-issue after expiry, never after complete.
    • complete may be seen twice (the launcher retries it on a 409 results_incomplete and on an ambiguous timeout): it must be idempotent.
    • The per-object checksum header (x-amz-checksum-sha256) on output PUTs is deferred: it must be signed with the URL and verified on real R2 first; decide in epic A, not now.
    • The job token stops being valid when the job is terminal (the extra day is for the webhook token only).
  3. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of phase A2 (the allocation ledger, #2032). This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • Put the hold and the job row in ONE batch, with the job insert gated on the hold having happened (changes() > 0 or an INSERT ... SELECT that checks the hold row): a job-count guard that reads rows must run inside the same statement as the hold, or two concurrent submissions race the guard.
    • A job id gets exactly one hold for ever (a partial unique index enforces it): a resubmission needs a new job id. A replayed hold key returns the original hold even after that hold was closed.
    • Derive the hold's idempotency key from the job (for example job:<id>:hold) rather than letting each caller choose one.
  4. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #2035 (squash c2bb5279), 2026-11-04, after a two-reviewer Sonnet panel (no Opus checkpoint for this phase); every accepted finding fixed (triage on the pull request). Gates: 466 targeted tests, typecheck and the D1 migration check pass (186 objects); the full CI run is the epic gate.

    What later phases get, as built: migration 0115 (compute_jobs, compute_gateway_users, with every primary key NOT NULL and the pseudonym table protected against REPLACE), compute-jobs.ts (createJob: the hold and the job row in one batch, the live-job limit inside the hold statement, spec_digest over the exact recorded bytes, a partner hold body taken as sent), compute-records.ts (records written once, a create-only conditional write, finalize for a bundle and for a partner's per-file upload capped at 500), compute-s3.ts (aws4fetch presigning against R2), the two bucket bindings and their toml pins; ADR 0118 amendment items 1 to 14.

    Open: real R2 behaviour is unverified (signed checksum header on a presigned PUT, CopyObject, conditional create, bucket lock); the outputs and complete routes and the job token's validity belong to phase A5; the partner contract (500 files) belongs to phase D. Still owed from this phase's "When merged" step: design page section 9, by a docs pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementhpc

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions