Repository navigation
Description
Goal
Build the core of the compute gateway in nemarOrg/nemar-cli: an owner policy and an append-only ledger in integer thousandths of an Expanse service unit (SU), a per-job index and record store, a typed Tapis client, the submit, webhook, cron and settlement lifecycle, revocation cascades, ACCESS gateway attribution, reconcile and reporting, and dispatch of NEMAR's own processing pipelines.
First-party pipelines come first: when this epic is merged and the owner has a tenant, the owner's own pipeline jobs can run end to end.
Why
NEMAR's data-quality work waits for it, and every later compute surface (the OpenID Connect provider's first client, the partner site, nemar compute, the website) sits on the same ledger and the same job record.
The design makes NEMAR the system of record for every job in the tenant, so the record, the accounting and the reconcile are built once, here, and tested against stand-ins before the tenant exists.
Phases
- Phase A1: Compute contract and ADR 0118 #2006 Contract: ADR 0118, state machine, job specification, rate and policy contract, header parser
- Phase A2: Allocation ledger and owner policy #2007 Ledger and policy: migration 0114, atomic hold, owner and admin routes,
nemar admin compute policy|pool - Phase A3: Job records and the results and records buckets #2008 Jobs and records: migration 0115, private R2 buckets with bindings and signer, record service, create-job with its hold
- Phase A4: Tapis client and stand-in #2009 Tapis client: two credential strategies, stand-in,
computeEnabled, status view - Phase A5: Job lifecycle, webhooks, settlement and results #2010 Lifecycle: submit, webhook, cron, results, cancel (Opus checkpoint)
- Phase A8: Revocation cascade and ACCESS gateway attribution #2013 Revocation cascade and ACCESS gateway attribution
- Phase A6: First-party pipeline dispatch #2011 Pipeline dispatch: stamps, recipe registry, eligibility,
pipeline run|status, daily dispatcher - Phase A7: Tenant reconcile, observer and reporting #2012 Reconcile and reporting: tenant sweep, observer ingest, admin reports, retention, the end-to-end proof
Order: A1, A2, A3, A4, A5, A8, A6, A7.
A3 and A4 both edit types/bindings.ts and wrangler-sccn.toml, so run them one after the other; nothing else in the list has a safe parallel pair.
A2 and A5 carry an Opus checkpoint before merge.
Where it lives
- Integration branch:
dev. Epic branch:feature/issue-#2005-epic-compute-core. Phase branches are cut from the epic branch and squash-merge into it. - Batch: 1, first priority. Epic B (sign-in) and the
nemar-processepic run beside it. - Reserved migrations: 0114 to 0119. Planned use: 0114 ledger and policy, 0115 jobs and gateway pseudonyms, 0116 lifecycle (result verification and output-upload columns), 0117 pipeline stamps, 0118 observer and legal holds, 0119 ACCESS attribution reports (Phase A8: Revocation cascade and ACCESS gateway attribution #2013).
- Reserved ADR: 0118 (the compute gateway), written in Phase A1: Compute contract and ADR 0118 #2006.
- Design page:
admin/design/compute-gateway(nemarOrg/docs,src/content/docs/admin/design/compute-gateway.md). - Implementation notes:
.context/compute-program/implementation-notes.mdon nemar-clidev.
Decisions the phase specs settled (so nobody reopens them)
- Pool totals are written only by an AFTER INSERT trigger on the ledger; a hold is one INSERT whose WHERE holds the whole check.
- Compute words (states, failure codes, refusal codes) are snake_case and live once in
shared/compute-state.ts. computeEnabledis exactly"1"in exactlyproductionordevelopment(staging must be able to test against the dev tenant); every compute cron stays production-only, and in development jobs advance by webhook, read-through refresh andnemar admin compute jobs advance. The S3-API stand-in is reached through aglobalThisoverride read at call time, ahead ofCOMPUTE_R2_S3_ENDPOINT.compute_jobs.account_idmay be NULL (no NEMAR account). The ACCESS gateway user is a per-account random pseudonym incompute_gateway_users(created on the account's first job, never rotated, never reused), copied onto each job row; it is deliberately not the OIDC subject, so ACCESS's unique-user counts do not change when Epic B lands and do not depend on a key that can rotate.- Webhook token is a domain-tagged HMAC over the job id and a per-job nonce; the nonce is cleared a day after the job ends, which is the expiry.
- Revocation reaches
finalizeRevocationandtombstoneUserCascade(the owner soft delete runs the latter); role demotion and key revocation do not end compute jobs. - Unreserved jobs are charged to the partner pool
tenant, never to a person. - Compute storage is two private Cloudflare R2 buckets, apart from the AWS Open Data account:
COMPUTE_RECORDS(bucket lock by prefix, set by the owner) andCOMPUTE_RESULTS(lifecycle expiry, no lock), reached through Worker bindings and presigned through R2's S3 API with one scoped token only the Worker holds. Binding paths are tested on real Miniflare R2. - No storage credential reaches Tapis or the node. After the container exits the launcher calls
POST /compute/jobs/:id/outputswith the per-job tokenjob_token(a domain-tagged HMAC like the webhook token, carried in the launch specification) in headerX-Compute-Job-Token, uploads to the presigned URLs returned, then callsoutputs/complete; it fetches its code bundle the same way, fromPOST /compute/jobs/:id/bundle, a 10-minute presigned GET of that job's own bundle and nothing else. A failed upload is the seventh launcher failure code,result_upload_failed, exit number 16.
Cross-repository contract
The launcher contract is shared with nemarOrg/nemar-process phase nemarOrg/nemar-process#2 (one JSON Schema, a drift test there), so no extra issue is needed; both gain the job_token field and the seventh failure code together.
Definition of done
- All seven phases merged into the epic branch, each with its gates and review recorded.
- The end-to-end test
backend/test/compute-e2e.test.tspasses: with the in-process Worker (real routes, real D1, production-flagged environment) and the Tapis, S3-API (over Miniflare R2) and ACCESS stand-ins, an admin runsnemar admin compute pipeline runfor a fixture EEG dataset; the job is held, submitted, its outputs uploaded through the outputs routes by a launcher stand-in, advanced by a webhook hint and a cron tick, settled provisionally, finalized by an observer post, recorded, and shown bynemar admin compute record <id>; and revoking a test account mid-run records cancellation, freezes its pools and cancels at Tapis after the retried sweep. - The commands and results of that run are pasted in the epic pull request.
-
bun run typecheck,bun run migrations:d1-checkand the fullbun testare green on the epic branch, and the full CI run is green. - No tenant name, allocation, account name, host or spend figure appears anywhere in the diff.
- Both dark switches are absent from
backend/wrangler-sccn.tomland pinned by test; every new cron is production-only. - The epic pull request into
devstays a draft until the items above hold; then one integration-scoped review, then the owner's approval, then a merge commit (no squash, no admin bypass).
Not in this epic
The OpenID Connect provider (Epic B), compute access, the partner contract and nemar compute (Epic D), website pages, the launcher and runtime images (nemar-process), mail about mismatches, and the 15-minute observer dispatch.
Owner actions that gate real use
The owner creates the two R2 buckets in each environment (production and dev names differ) with their bucket lock and lifecycle rules (records/ 7 years, logs/ 1 year, incoming/ deleted after 2 days; results deleted after 30 days, incomplete multipart uploads aborted after 1 day) and one R2 API token per environment (Object Read & Write on its two buckets) set as the three COMPUTE_R2_* Worker secrets, sets the Tapis and ACCESS secrets, creates the nemar-pipelines service account, funds the pipelines pool, and turns on COMPUTE_ENABLED then COMPUTE_PIPELINES_DISPATCH_ENABLED; none of that is part of a phase.
Activity
docs: add compute program roadmap and playbook (#2027)
Merge pull request #2063 from nemarOrg/feature/issue-2005-epic-comput…
neuromechanist commentedon Nov 10, 2026
The compute gateway core is merged into dev (#2063, merge commit 3af1638). Everything stays dark: the compute switches are absent from the production configuration. Remaining work is the Tapis tenant integration, tracked in the owner items, not this epic.