Repository navigation
Description
Goal
An administrator can run a NEMAR processing recipe on a public EEG dataset version through the compute gateway, and a production-only daily dispatcher can do the same for the catalog, one stamp per dataset version, recipe, recipe version and engine so the queue knows what is current.
Nothing dispatches by itself until the owner turns the switch on.
Routing
- Readiness: ready
- Implementer: Sonnet
- Opus checkpoint: none
- Depends on: Phase A3: Job records and the results and records buckets #2008, Phase A5: Job lifecycle, webhooks, settlement and results #2010 ; Unblocks: Phase A7: Tenant reconcile, observer and reporting #2012
- Repository and base branch: nemarOrg/nemar-cli, epic branch
feature/issue-#2005-epic-compute-core
Scope
In: migration 0117_compute_pipeline_stamps.sql; shared/compute-recipes.ts; the eligibility predicate; the dispatch service, admin routes and CLI; the daily dispatcher.
Out: the recipes' own code, container images and Tapis application definitions (nemarOrg/nemar-process, a separate epic), publication of products and the held tier (first-party processing design), creating the service account (an owner runbook step below, not code), user-facing nemar compute run (Epic D).
Design references
Design sections 5 (pools and ceilings, large partitions are pipelines only) and 10 (identity, recipes, registry, dispatch, cost). First-party processing design in origin/docs/first-party-processing-design (recipes nemar-floor and nemar-ica, eligibility, 20-dataset pilot). Templates: the term list in backend/src/services/neurobagel-eligibility.ts (one predicate, SQL and TypeScript side by side, driven by a test through both); runRepullSchedulerCron in services/upstream-repull-scheduler.ts; autoRepullEnabled; ADR 0065 (anonymity), ADR 0033 (stamp what converted a dataset).
Changes
- Owner runbook, no code: create the account
nemar-pipelines(sign in once with a mailbox the owner controls), thennemar admin kind nemar-pipelines serviceandnemar admin keys create nemar-pipelines <name>. The platformpipelinespools from Phase A2: Allocation ledger and owner policy #2007 are what the owner funds withnemar admin compute pool grant --platform pipelines; the account itself holds no pool. - Migration
0117_compute_pipeline_stamps.sql:compute_pipeline_stamps(id, dataset_id, version, recipe, recipe_version, engine, attempt, state IN ('dispatched','succeeded','failed','refused'), job_id TEXT NULL, refusal_code NULL fixed word, created_at, updated_at), no foreign keys, and a partial unique index on(dataset_id, version, recipe, recipe_version, engine)wherestate IN ('dispatched','succeeded')so a failed or refused attempt does not block a retry and history stays. AddDELETE FROM compute_pipeline_stamps WHERE dataset_id = ?todeleteDatasetCascadeinservices/deletion.ts; the job rows stay. shared/compute-recipes.ts(import-free): a registry entry per recipe withname,version,appId,appVersion,engine(for exampleeegprep@<version>), partition defaults,autoDispatch, and an eligibility tag. Entries:nemar-floor(CPU,shared,autoDispatch: true) andnemar-ica(GPU,gpu-shared, one GPU,autoDispatch: falseuntil the pilot). The numbers are initial and the app id and version follow whatevernemar-processpublishes; a test pins only that every entry is valid underJobSpecSchemaand inside the partition table.services/compute-pipeline-eligibility.ts: named terms, SQL and TypeScript together: active, public,anonymous = 0(NULL is unknown and unknown is not false),first_published_atset, not withdrawn, DOI not tombstoned, has adataset_versionsrow, modality tokeneeg(use the whole-token clause inservices/dataset-filters.ts), realnmoronid outside the reserved fixture band. Decided from the D1 row on every dispatch, never from the request.services/compute-pipeline.ts:dispatchPipelineJob(env, { recipe, datasetId, version?, actor })resolves the newestdataset_versionsrow when no version is given, re-checks eligibility, resolves thenemar-pipelinesservice account (elsepipelines_account_missing), claims the stamp first (a live or succeeded tuple islive_attempt_exists), builds aJobSpecwhose datasets point at the data plane (<API_BASE_URL>/data/<id>/<version>/manifest.json, at most 38,000 entries), calls Phase A3: Job records and the results and records buckets #2008'screateJobwith sourcepipeline, pool = the platformpipelinespool, Tapis ownerPIPELINES_TAPIS_OWNER, idempotency key from the tuple and attempt, links the stamp to the job, then calls Phase A5: Job lifecycle, webhooks, settlement and results #2010'ssubmitComputeJob. A refused hold marks the stamprefusedwith the word.syncPipelineStamps(env)movesdispatchedstamps tosucceededorfailedfrom their job's terminal state, fails a stamp that still has no job after 30 minutes (a crash between claim and create), and runs before every dispatch and status read.pipelineDryRun: the eligible datasets, the estimated SU per job and in total, and what blocks each ineligible one; it reads only and works in every environment.- Admin routes in
routes/admin/compute.ts:POST /compute/pipeline/run(bodyrecipe, thendatasetwith optionalversion, oreligiblewithlimit, plusdry_run) andGET /compute/pipeline/status(filtersrecipe,dataset,state, paged); a non-dry run refusescompute_disabledunlesscomputeEnabled. Auditcompute.pipeline.runwith counts. - CLI
nemar admin compute pipeline run <recipe> --dataset <id>[@version] | --eligible --limit <n> [--dry-run]andpipeline status, insrc/commands/admin-compute-pipeline.ts, every command with--json. - Daily dispatcher
runComputePipelineDispatchCron(env)wired into theprodOnlyJobsblock ofscheduled(): does nothing unlesscomputePipelinesDispatchEnabled; dispatchesautoDispatchrecipes only, at mostpolicy.pipelines.dailyJobCapminus today's pipeline jobs, and never pastpolicy.pipelines.maxLiveJobsper pool; a refused hold stops the run for that pool. Add it tocron-sweep-wiring.test.tsasprod-onlywith its own self-fence.
Tests (write these first)
backend/test/compute-pipeline-eligibility.test.ts: the SQL and the TypeScript form agree over a generated table of rows; an anonymous, private, withdrawn, non-EEG, versionless and fixture-band row each fail exactly their term. Mutation: changeanonymous = 0toIS NOT 1; the NULL row must flip and fail.backend/test/compute-pipeline.test.tsagainst real D1 and the stand-ins: a run holds, creates, stamps and submits; a second run of the same tuple islive_attempt_exists; a failed job allows a new attempt; a refused hold leaves arefusedstamp and no job; the version defaults to the newest.- Race: two concurrent runs of one tuple produce one stamp and one job.
- Dispatcher: off without both switches, honours the daily cap and the live cap, skips non-
autoDispatchrecipes; production-only by wiring and by self-fence. backend/test/compute-pipeline-routes.test.tsandtest/admin-compute-pipeline-cli.test.ts(templatetest/admin-repull-auto-cli.test.ts): dry run writes nothing in a non-production environment; a member is refused;--eligible --limit 2dispatches two.- Dataset delete removes stamps and keeps jobs.
Acceptance criteria
- No recipe runs on an anonymous, private, withdrawn or non-EEG dataset.
- A stamp tuple is unique while live or succeeded, and the dispatcher is dark without its switch.
- The Worker, not the request, supplies the repository-like facts: dataset row, version row, account and pool.
- Wiring, route-inventory and cron tests pass.
Gates
bun test backend/test/compute-pipeline-eligibility.test.ts backend/test/compute-pipeline.test.ts backend/test/cron-sweep-wiring.test.ts backend/test/compute-switches-flag.test.ts backend/test/compute-pipeline-routes.test.ts test/admin-compute-pipeline-cli.test.ts test/admin-route-inventory.unit.test.tsbun run migrations:d1-check,bun run typecheck,bunx biome check src shared backend/src,bun test
Rules that bite here
- ADR 0065: anonymity is decided from the D1 row; never re-add a blind or a filter at a call site.
- The dev worker shares the
nemarDatasetsorg and real users: nothing here may touch GitHub or mail, and the dispatcher is production-only. - Dark switches are absent from the toml and pinned by the test from Phase A4: Tapis client and stand-in #2009.
- Datasets deleted must not leave stamps; jobs and ledger rows are retained by design.
Escalate to the owner if
- The recipe defaults need real figures before the pilot, or the 20-dataset pilot should run under a different cap.
- The service account cannot be created without a new route.
When merged
Update design section 10 with the stamp tuple and the dispatcher's caps, and write the owner runbook for the service account and the pilot.
Activity
neuromechanist commentedon Nov 4, 2026
Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (nemarOrg/nemar-process#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in nemarOrg/nemar-process#7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).
Dispatch builds the launch specification through
dataTagForVersion(A1b) for every dataset version and refuses an eligible-dataset record whose version has no data-plane tag; the recipe's image is referenced by the digest-named SIF.neuromechanist commentedon Nov 5, 2026
Driver amendment, 2026-11-04, from the Opus checkpoint of phase A5 (#2041). This comment is part of the phase brief; where it differs from the body above, this comment wins.
createJob: the dispatch records the recipe name together with the resolved runtime, image (the digest-pinned runtime reference) and entry or command, sospec.json,spec_digestandlaunch_spec_digestall describe what actually ran. The A5 launch-specification builder is a pure function of the recorded spec and keeps refusing a recipe-only spec (it has no entry or command to launch).JobSpecSchemaalready allows a recipe alongside an entry or command.COMPUTE_RUNTIME_IMAGE: ONE digest-pinned reference (theuriof the image workflow'slaunch_image), from which the builder derivesdigestandsifwith the shared contract functions. It is set per environment by the owner after the first publish, outside the toml;/admin/compute/statusreports whether it is set and its digest. A recipe's runtime resolves to this image unless the recipe names another reviewed digest.neuromechanist commentedon Nov 5, 2026
Driver amendment, 2026-11-04, from the C3 phase plan (nemarOrg/nemar-process#4, planner comments). This comment is part of the phase brief; where it differs from the body above, this comment wins. Everything here is a plan, not yet merged code: re-check it against nemar-process's
docs/recipes.mdwhen phase C3 merges.envallowlist is closed to six names (OMP_NUM_THREADS,MKL_NUM_THREADS,OPENBLAS_NUM_THREADS,NUMEXPR_NUM_THREADS,PYTHONHASHSEED,MPLBACKEND), so the recipe cannot read an image digest fromNEMAR_IMAGE_DIGEST. The dispatcher passes it in the command's ARGUMENTS:nemar-recipe run nemar-floor --dataset-dir <dir> --out <dir> --image-digest sha256:<hex> [--recording <path> ...](the arguments are free: up to 64 of 1,024 characters).desc-nemarfloor_recordings.tsv, the complete per-recording table (failed and skipped recordings live only there; the dataset record is capped at 256 KB and embeds a truncated table beyond about 1,500 recordings). The dispatcher and the result reader must expect it underoutputs/product/.nemar-recipe run: 0 with counts in the dataset record unless NOTHING succeeded (then 4); a recording that fails is recorded and the others continue; a.fifrecording isskippedwith reasonunsupported_format. The registry reads counts from the dataset record.nemar-floor1.0.0 as the plan fixes it (owner decision O1 pending, reversible before release); the stamp you record for a recipe job must come from the recipe's own record, not from a constant in the Worker.createJob(earlier amendment) records the recipe name together with the resolved runtime image reference (COMPUTE_RUNTIME_IMAGE), the recipe's argument list (including--image-digest) and the recordings the job selects; the recipe is NEVER run with an image other than the one the digest names.neuromechanist commentedon Nov 5, 2026
Amendment from phase C3 (nemarOrg/nemar-process#4, draft PR nemarOrg/nemar-process#11), 2026-11-05. Three facts that phase A6 must carry:
--image-digest. The launch specification's environment allowlist stays closed to its six names, so A6 passes the digest as an argument and does not add a variable.desc-nemarfloor_recordings.tsv, which holds the complete per-recording table. A6's job-output expectations must allow it. The dataset record holds a capped summary and states when rows are left out.A fourth point, a recording whose channels are all flat, is still open in the C3 review. This amendment will be updated when it is decided.
neuromechanist commentedon Nov 5, 2026
Amendment from the C3 review (nemarOrg/nemar-process#4, draft PR nemarOrg/nemar-process#11).
The nemar-floor recipe takes its image digest as an argument,
--image-digest. The recipe cannot check that value, so it is the caller's claim.Phase A6 must compare the digest it passes with the digest recorded in the launcher's provenance.json before it accepts a result, and fail the job on a mismatch. This adds to the earlier amendment in this issue (the argument, the extra recordings table and the exit codes); it does not replace it.
neuromechanist commentedon Nov 6, 2026
Merged into the compute epic (PR 2054).