Skip to content

Phase A6: First-party pipeline dispatch #2011

Closed

Description

@neuromechanist

Goal

An administrator can run a NEMAR processing recipe on a public EEG dataset version through the compute gateway, and a production-only daily dispatcher can do the same for the catalog, one stamp per dataset version, recipe, recipe version and engine so the queue knows what is current.
Nothing dispatches by itself until the owner turns the switch on.

Routing

Scope

In: migration 0117_compute_pipeline_stamps.sql; shared/compute-recipes.ts; the eligibility predicate; the dispatch service, admin routes and CLI; the daily dispatcher.
Out: the recipes' own code, container images and Tapis application definitions (nemarOrg/nemar-process, a separate epic), publication of products and the held tier (first-party processing design), creating the service account (an owner runbook step below, not code), user-facing nemar compute run (Epic D).

Design references

Design sections 5 (pools and ceilings, large partitions are pipelines only) and 10 (identity, recipes, registry, dispatch, cost). First-party processing design in origin/docs/first-party-processing-design (recipes nemar-floor and nemar-ica, eligibility, 20-dataset pilot). Templates: the term list in backend/src/services/neurobagel-eligibility.ts (one predicate, SQL and TypeScript side by side, driven by a test through both); runRepullSchedulerCron in services/upstream-repull-scheduler.ts; autoRepullEnabled; ADR 0065 (anonymity), ADR 0033 (stamp what converted a dataset).

Changes

  • Owner runbook, no code: create the account nemar-pipelines (sign in once with a mailbox the owner controls), then nemar admin kind nemar-pipelines service and nemar admin keys create nemar-pipelines <name>. The platform pipelines pools from Phase A2: Allocation ledger and owner policy #2007 are what the owner funds with nemar admin compute pool grant --platform pipelines; the account itself holds no pool.
  • Migration 0117_compute_pipeline_stamps.sql: compute_pipeline_stamps(id, dataset_id, version, recipe, recipe_version, engine, attempt, state IN ('dispatched','succeeded','failed','refused'), job_id TEXT NULL, refusal_code NULL fixed word, created_at, updated_at), no foreign keys, and a partial unique index on (dataset_id, version, recipe, recipe_version, engine) where state IN ('dispatched','succeeded') so a failed or refused attempt does not block a retry and history stays. Add DELETE FROM compute_pipeline_stamps WHERE dataset_id = ? to deleteDatasetCascade in services/deletion.ts; the job rows stay.
  • shared/compute-recipes.ts (import-free): a registry entry per recipe with name, version, appId, appVersion, engine (for example eegprep@<version>), partition defaults, autoDispatch, and an eligibility tag. Entries: nemar-floor (CPU, shared, autoDispatch: true) and nemar-ica (GPU, gpu-shared, one GPU, autoDispatch: false until the pilot). The numbers are initial and the app id and version follow whatever nemar-process publishes; a test pins only that every entry is valid under JobSpecSchema and inside the partition table.
  • services/compute-pipeline-eligibility.ts: named terms, SQL and TypeScript together: active, public, anonymous = 0 (NULL is unknown and unknown is not false), first_published_at set, not withdrawn, DOI not tombstoned, has a dataset_versions row, modality token eeg (use the whole-token clause in services/dataset-filters.ts), real nm or on id outside the reserved fixture band. Decided from the D1 row on every dispatch, never from the request.
  • services/compute-pipeline.ts: dispatchPipelineJob(env, { recipe, datasetId, version?, actor }) resolves the newest dataset_versions row when no version is given, re-checks eligibility, resolves the nemar-pipelines service account (else pipelines_account_missing), claims the stamp first (a live or succeeded tuple is live_attempt_exists), builds a JobSpec whose datasets point at the data plane (<API_BASE_URL>/data/<id>/<version>/manifest.json, at most 38,000 entries), calls Phase A3: Job records and the results and records buckets #2008's createJob with source pipeline, pool = the platform pipelines pool, Tapis owner PIPELINES_TAPIS_OWNER, idempotency key from the tuple and attempt, links the stamp to the job, then calls Phase A5: Job lifecycle, webhooks, settlement and results #2010's submitComputeJob. A refused hold marks the stamp refused with the word. syncPipelineStamps(env) moves dispatched stamps to succeeded or failed from their job's terminal state, fails a stamp that still has no job after 30 minutes (a crash between claim and create), and runs before every dispatch and status read. pipelineDryRun: the eligible datasets, the estimated SU per job and in total, and what blocks each ineligible one; it reads only and works in every environment.
  • Admin routes in routes/admin/compute.ts: POST /compute/pipeline/run (body recipe, then dataset with optional version, or eligible with limit, plus dry_run) and GET /compute/pipeline/status (filters recipe, dataset, state, paged); a non-dry run refuses compute_disabled unless computeEnabled. Audit compute.pipeline.run with counts.
  • CLI nemar admin compute pipeline run <recipe> --dataset <id>[@version] | --eligible --limit <n> [--dry-run] and pipeline status, in src/commands/admin-compute-pipeline.ts, every command with --json.
  • Daily dispatcher runComputePipelineDispatchCron(env) wired into the prodOnlyJobs block of scheduled(): does nothing unless computePipelinesDispatchEnabled; dispatches autoDispatch recipes only, at most policy.pipelines.dailyJobCap minus today's pipeline jobs, and never past policy.pipelines.maxLiveJobs per pool; a refused hold stops the run for that pool. Add it to cron-sweep-wiring.test.ts as prod-only with its own self-fence.

Tests (write these first)

  1. backend/test/compute-pipeline-eligibility.test.ts: the SQL and the TypeScript form agree over a generated table of rows; an anonymous, private, withdrawn, non-EEG, versionless and fixture-band row each fail exactly their term. Mutation: change anonymous = 0 to IS NOT 1; the NULL row must flip and fail.
  2. backend/test/compute-pipeline.test.ts against real D1 and the stand-ins: a run holds, creates, stamps and submits; a second run of the same tuple is live_attempt_exists; a failed job allows a new attempt; a refused hold leaves a refused stamp and no job; the version defaults to the newest.
  3. Race: two concurrent runs of one tuple produce one stamp and one job.
  4. Dispatcher: off without both switches, honours the daily cap and the live cap, skips non-autoDispatch recipes; production-only by wiring and by self-fence.
  5. backend/test/compute-pipeline-routes.test.ts and test/admin-compute-pipeline-cli.test.ts (template test/admin-repull-auto-cli.test.ts): dry run writes nothing in a non-production environment; a member is refused; --eligible --limit 2 dispatches two.
  6. Dataset delete removes stamps and keeps jobs.

Acceptance criteria

  • No recipe runs on an anonymous, private, withdrawn or non-EEG dataset.
  • A stamp tuple is unique while live or succeeded, and the dispatcher is dark without its switch.
  • The Worker, not the request, supplies the repository-like facts: dataset row, version row, account and pool.
  • Wiring, route-inventory and cron tests pass.

Gates

  • bun test backend/test/compute-pipeline-eligibility.test.ts backend/test/compute-pipeline.test.ts backend/test/cron-sweep-wiring.test.ts backend/test/compute-switches-flag.test.ts backend/test/compute-pipeline-routes.test.ts test/admin-compute-pipeline-cli.test.ts test/admin-route-inventory.unit.test.ts
  • bun run migrations:d1-check, bun run typecheck, bunx biome check src shared backend/src, bun test

Rules that bite here

  • ADR 0065: anonymity is decided from the D1 row; never re-add a blind or a filter at a call site.
  • The dev worker shares the nemarDatasets org and real users: nothing here may touch GitHub or mail, and the dispatcher is production-only.
  • Dark switches are absent from the toml and pinned by the test from Phase A4: Tapis client and stand-in #2009.
  • Datasets deleted must not leave stamps; jobs and ledger rows are retained by design.

Escalate to the owner if

  • The recipe defaults need real figures before the pilot, or the 20-dataset pilot should run under a different cap.
  • The service account cannot be created without a new route.

When merged

Update design section 10 with the stamp tuple and the dispatcher's caps, and write the owner runbook for the service account and the pilot.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of the launcher (nemarOrg/nemar-process#7). This comment is part of the phase brief; where it differs from the body above, this comment wins. The launcher-side facts it relies on are in nemarOrg/nemar-process#7 and in the nemar-cli follow-up phase A1b (launch-specification and launcher-detail contract).

    Dispatch builds the launch specification through dataTagForVersion (A1b) for every dataset version and refuses an eligible-dataset record whose version has no data-plane tag; the recipe's image is referenced by the digest-named SIF.

  2. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the Opus checkpoint of phase A5 (#2041). This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • A job created from a recipe is resolved BEFORE createJob: the dispatch records the recipe name together with the resolved runtime, image (the digest-pinned runtime reference) and entry or command, so spec.json, spec_digest and launch_spec_digest all describe what actually ran. The A5 launch-specification builder is a pure function of the recorded spec and keeps refusing a recipe-only spec (it has no entry or command to launch). JobSpecSchema already allows a recipe alongside an entry or command.
    • Until this phase exists, no create path may take a hold for a recipe-only spec: refuse it before the hold.
    • NEMAR's runtime image comes from the owner-set Worker variable COMPUTE_RUNTIME_IMAGE: ONE digest-pinned reference (the uri of the image workflow's launch_image), from which the builder derives digest and sif with the shared contract functions. It is set per environment by the owner after the first publish, outside the toml; /admin/compute/status reports whether it is set and its digest. A recipe's runtime resolves to this image unless the recipe names another reviewed digest.
  3. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the C3 phase plan (nemarOrg/nemar-process#4, planner comments). This comment is part of the phase brief; where it differs from the body above, this comment wins. Everything here is a plan, not yet merged code: re-check it against nemar-process's docs/recipes.md when phase C3 merges.

    • The launch specification's env allowlist is closed to six names (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS, PYTHONHASHSEED, MPLBACKEND), so the recipe cannot read an image digest from NEMAR_IMAGE_DIGEST. The dispatcher passes it in the command's ARGUMENTS: nemar-recipe run nemar-floor --dataset-dir <dir> --out <dir> --image-digest sha256:<hex> [--recording <path> ...] (the arguments are free: up to 64 of 1,024 characters).
    • The recipe writes one extra file besides the per-recording records and the dataset record: desc-nemarfloor_recordings.tsv, the complete per-recording table (failed and skipped recordings live only there; the dataset record is capped at 256 KB and embeds a truncated table beyond about 1,500 recordings). The dispatcher and the result reader must expect it under outputs/product/.
    • Exit codes of nemar-recipe run: 0 with counts in the dataset record unless NOTHING succeeded (then 4); a recording that fails is recorded and the others continue; a .fif recording is skipped with reason unsupported_format. The registry reads counts from the dataset record.
    • The recipe's step order differs from the first-party design's (flatline before drift): the recipe version is nemar-floor 1.0.0 as the plan fixes it (owner decision O1 pending, reversible before release); the stamp you record for a recipe job must come from the recipe's own record, not from a constant in the Worker.
    • Recipe resolution before createJob (earlier amendment) records the recipe name together with the resolved runtime image reference (COMPUTE_RUNTIME_IMAGE), the recipe's argument list (including --image-digest) and the recordings the job selects; the recipe is NEVER run with an image other than the one the digest names.
  4. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Amendment from phase C3 (nemarOrg/nemar-process#4, draft PR nemarOrg/nemar-process#11), 2026-11-05. Three facts that phase A6 must carry:

    • The nemar-floor recipe takes its image digest as an argument, --image-digest. The launch specification's environment allowlist stays closed to its six names, so A6 passes the digest as an argument and does not add a variable.
    • The recipe writes one extra file next to its output, desc-nemarfloor_recordings.tsv, which holds the complete per-recording table. A6's job-output expectations must allow it. The dataset record holds a capped summary and states when rows are left out.
    • Exit codes: 0 with counts when at least one recording succeeded; exit 4 only when nothing succeeded.

    A fourth point, a recording whose channels are all flat, is still open in the C3 review. This amendment will be updated when it is decided.

  5. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Amendment from the C3 review (nemarOrg/nemar-process#4, draft PR nemarOrg/nemar-process#11).

    The nemar-floor recipe takes its image digest as an argument, --image-digest. The recipe cannot check that value, so it is the caller's claim.

    Phase A6 must compare the digest it passes with the digest recorded in the launcher's provenance.json before it accepts a result, and fail the job on a mismatch. This adds to the earlier amendment in this issue (the argument, the extra recordings table and the exit codes); it does not replace it.

  6. neuromechanist commented on Nov 6, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the compute epic (PR 2054).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementhpc

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions