Skip to content

Phase A1: Compute contract and ADR 0118 - #2030

Merged
neuromechanist merged 22 commits into
feature/issue-2005-epic-compute-corefrom
feature/issue-2006-phase1-compute-contract-and-adr-0118
Nov 4, 2026
Merged

neuromechanist merged 22 commits into
feature/issue-2005-epic-compute-corefrom
feature/issue-2006-phase1-compute-contract-and-adr-0118

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Phase A1 of the compute gateway core (epic #2005, issue #2006): the written contract before any table or route exists. Nothing here touches D1, the Worker, the CLI or the network. This body is current as of the round 2 review fixes (head 33e80dfc).

What changed

  • ADR 0118 (accepted, 2026-11-03) with its index line, its entry in AGENTS.md's load-bearing list, and an Amendment line on ADR 0049 pointing its stage 3 at it. ADR 0040, 0042 and 0049 are refined, not changed. Round 1: terms spelled out on first use, the three homes named the same in the ADR, index and AGENTS.md, COMPUTE_ENABLED and its test moved to phase A4, and two new decisions (two digests; responses open, requests and documents closed).
  • shared/compute-state.ts (import-free): the ten job states and the only edges between them, stateFromTapisStatus, the launcher and platform failure codes with isPlatformFailure and failureCodeFromTapisCondition (mapped from the whole published Tapis condition list), the refusal codes with COMPUTE_REFUSAL_STATUS, and the ledger, source, kind, flag and cancel words. over_ceiling is gone: the per-job limit refusal is over_job_limit.
  • shared/contract/compute.ts (zod, re-exported from shared/contract/index.ts): JobSpecSchema, PARTITIONS, RATE_TABLE_V1, estimateSuMilli, suMilliFor, parseSuToMilli, formatSu, ComputePolicySchema with the frozen DEFAULT_COMPUTE_POLICY, the launcher constants (including RECORDS_RESULT_MANIFEST_FILE, COMPUTE_PATH_SEGMENT, COMPUTE_MAX_OUTPUT_FILES) and the shapes of result.json, manifest.json and provenance.json.
  • shared/contract/compute-launch.ts and shared/contract/compute-launch.schema.json: the launch specification nemar.launch-spec.v1, the bundle, outputs and outputs-response shapes, the results-bucket keys, encodeLaunchSpec (ASCII-only JSON, the 64 KiB cap checked after escaping) and launchSpecFits. The schema is meant to be copied byte for byte by nemarOrg/nemar-process.
  • shared/compute-header.ts (import-free): parseJobHeader for # /// nemar, % /// nemar and nemar-job.toml (an unknown kind throws RangeError), and headerToSpecFields.
  • test/contract-schemas.test.ts: the strict-schema guard exempts shared/contract/compute.ts and compute-launch.ts by name; a classification table checks every exported zod schema of the two files that is, or holds, an object (an object, an array of them, a union) by behaviour (nine strict, two lenient; an unlisted one fails, and so does a kind of schema the walk does not know), and an import guard keeps strict names out of every other file under shared/contract, however they are taken (static, through ./index.js, with a .ts extension, beside a default, or by a dynamic import() or require()).
  • Pure unit tests for all of it, and section 6 of .context/compute-program/implementation-notes.md saying where the words now live.

Round 1 review fixes

  • Responses are open, everything else closed: BundleResponseSchema, OutputsResponseSchema and its uploads items are .passthrough(), and the schema has no additionalProperties: false on bundle_response, outputs_response or its upload items (a running launcher must survive the Worker adding a key); requests and documents stay closed.
  • Two digests: the provenance key is now launch_spec_digest, the SHA-256 of the exact raw bytes of the NEMAR_LAUNCH_SPEC value, hashed before parsing. spec_digest stays the job row's digest of records/<id>/spec.json (a comment, not a launcher key).
  • The launch specification is ASCII-only JSON (every UTF-16 unit from U+007F up is a lowercase \uXXXX escape) and the cap is checked after escaping.
  • Preempt: the 1000 minimum applies after the 800/1000 factor, one exact round-up; stated the same way in the code and the ADR. A rate with no driver (a per-core partition with neither cores nor memory, a per-GPU one with neither GPUs nor memory) throws RangeError instead of pricing the minimum. A GPU partition needs gpus of at least 1 and a CPU partition 0.
  • Decision on item 7 (a valid job always fits the launch specification): I bounded the serialized size inside JobSpecSchema rather than tightening per-field caps, because a non-ASCII character costs six bytes once escaped and no per-field cap that is generous for ASCII is also safe for any mix. The entry, command, arguments, include patterns and environment together may take 49,152 bytes (LAUNCH_VARIABLE_BYTES_MAX, counted by launchVariableBytes); everything else is under 4.3 KiB at its own worst case, which leaves about 12 KiB of slack. launchSpecFits(job) lets the Worker refuse with invalid_spec before taking a hold. The per-field caps are unchanged, so a job with 64 arguments of 1,024 plain characters is now refused. Round 2 adds the other half of "a valid job always launches": every string that reaches a process must be well-formed UTF-16.
  • image.uri follows the job's own digest-pinned reference grammar and image.sif is a relative path of A-Za-z0-9_.+- segments with no leading dash, no . or .. segment and no leading or trailing slash. The contract states that the launcher must run the container with a clean environment.
  • Every numeric and length bound of the launch schema comes from one table of literals, judged by Ajv (built strict: true, strictRequired: false) and by zod on both sides of its edge (70 rows); a second table does the same for the job specification and the owner policy (97 rows). To let strict Ajv compile, the four contains subschemas gained type: object; I also removed the minLength and minItems keywords (and the matching zod .min(1)) that a pattern or the contains clauses already imply.
  • The schema $id is now the file's GitHub address, the form deploy/neurobagel/index.schema.json uses; the API serves only the Zarr schemas.

Round 2 review fixes

  • Lone surrogates: JobSpecSchema refuses a lone surrogate in the entry, a command item, an argument and an include pattern, each issue naming the field by its path (entry, command.1, args.1, datasets.0.include.1) with the message "a lone surrogate is not text a process argument can carry", never the value. Such a string is valid JavaScript and encodes as \ud800, but no process argument can carry it, so the job would have failed only after its hold existed. An environment value needs no check (printable ASCII only), and a test pins that. The check is computeIsWellFormed, a loop over the UTF-16 units: this tree builds for ES2022 and String.prototype.isWellFormed is newer, so a test compares the loop with isWellFormed instead (all 2,048 surrogate units alone, a stride of every high and low pair, and a seeded fuzz of 20,000 strings). The shared JSON Schema is unchanged (it still lets a lone-surrogate escape through); the Worker's refusal is the guard, so a valid job specification still always fits the launch specification, and now is also well-formed.
  • Import guard: the scan is a pure function of a file's path and text (strictComputeTakes) that judges a specifier by the file it resolves to, so ./compute, ./compute.js, ./compute.ts and ../contract/compute.js are one file. It reports a strict name or a namespace taken from either strict file or from ./index.js (also ./index, ./index.ts, . and ..), a default beside named or namespace imports, and any dynamic import() or require() of them (a dynamic specifier that is not a plain string is reported as unresolvable). It still allows index.ts's own export *, import type { JobSpec } and non-strict names. 68 probes pin each shape, flagged and allowed, with the same function scanning the real files.
  • launchSpecFits worst case: three job shapes (an entry with an environment and include patterns, an image command with a GPU, and a recipe with neither) each have the exact size of the worst-case document with no arguments pinned as a literal (2,119, 2,012 and 2,016 bytes, cross-checked in Python), and launchSpecFits is held to that document to the byte: with arguments that bring it to 65,536 bytes it fits, at 65,537 it does not. A second test makes each Worker field of the worst case the largest the schema allows (one past it is refused). Shrinking or dropping any one field of either document now fails a test, including a bundle size of 1.
  • Classification: the walk reaches through arrays, unions, discriminated unions, records, maps, sets, tuples, intersections, optional, nullable, default, catch, readonly, branded, promise, lazy and pipe wrappers, and a kind it does not know throws, so an export the test cannot classify fails it. A schema that is not itself an object (an array of them, a union) must say where its unknown key goes, since a key added to an array proves nothing about its elements. Probes built in the test cover each kind, a self-referential schema and an unknown kind.
  • Mutation: the new code was mutated line by line (about 150 mutants across the four items, each bound and branch of the well-formedness loop, the three field checks, every part of the specifier resolution, statement and dynamic-call scanning, every field of launchSpecFits and the worst-case document, and every case of the schema walk). Each survivor of a first pass became a probe or a simplification, and all are now killed except two equivalent ones: lowering the start of the low-surrogate range by one unit (the high branch has already taken that unit) and a bundle size of 9007199254740990 for 9007199254740991 (the same number of digits, so the same bytes).

Gates

  • bun test test/compute-state.unit.test.ts test/compute-contract.unit.test.ts test/compute-header.unit.test.ts test/adr-index.unit.test.ts test/contract-schemas.test.ts: 552 pass, 0 fail, 70725 expect() calls, across 5 files.
  • bun run typecheck (tsc --noEmit for the CLI and the backend): exit 0, no output.
  • bunx biome check shared test/compute-state.unit.test.ts test/compute-contract.unit.test.ts test/compute-header.unit.test.ts test/contract-schemas.test.ts: Checked 91 files. No fixes applied.
  • Second consumer: Python jsonschema (Draft 2020-12) over the shared schema with 58 documents and checks, plus the golden vector (the SHA-256 of the raw bytes, then the parse): 0 disagreements at round 1. The schema file has not changed since (same SHA-256 and size below), so it was not run again.
  • Mutation (round 1): every earlier mutation list was re-run against the final code, and every number in the schema, compute-launch.ts and compute.ts was bumped by one in each direction (185, 55 and 174 mutants): all killed, 0 survivors, including each of the reviewer's survivors. About 860 single-line mutants were killed in all. The mutants still alive are equivalent: the K memory unit (it has no multiplier, so it is refused anyway) and the exemption filter in the import guard (neither exempt file imports a strict name from the other), plus the control mutation that imports a lenient name, which the guard allows by design.
  • The whole bun test is for the driver to run on the final head. Note that a bun test was running in this worktree while I edited in round 1 (started about 01:21 by someone else), so any result from before this head is stale.
  • test/contract-schemas.test.ts has one TypeScript error that is not from this branch (line 494, an older user-status test; the file is outside the repo's typecheck include).

Golden vector for nemar-process

One fixed launch specification (a non-ASCII character, an astral character, a double quote and a space in args), the exact text encodeLaunchSpec writes for it and the SHA-256 of that text, all pinned as literals in test/compute-contract.unit.test.ts. The launcher must report the digest for these bytes and parse them back to the same specification.

Specification: { job_id: "job_0123456789abcdef01234567", runtime: "python", entry: "run.py", args: ["--label", 'café "q" \u{1F600}'], bundle: { sha256: "a" x 64, size: 1234 }, datasets: [{ id: "nm000104", version: "1.0.2" }], api_base: "https://api.example.test", image: { uri: "docker://ghcr.io/example/runtime@sha256:" + "b" x 64, digest: "sha256:" + "b" x 64, sif: "runtime.sif" }, env: { OMP_NUM_THREADS: "4" }, resources: { gpus: 0 }, limits: { max_files: 38000, max_total_bytes: 1073741824 }, job_token: "t" x 43 }

Encoded text (708 bytes, ASCII):

{"job_id":"job_0123456789abcdef01234567","runtime":"python","entry":"run.py","args":["--label","café \"q\" 😀"],"bundle":{"sha256":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","size":1234},"datasets":[{"id":"nm000104","version":"1.0.2"}],"api_base":"https://api.example.test","image":{"uri":"docker://ghcr.io/example/runtime@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","digest":"sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sif":"runtime.sif"},"env":{"OMP_NUM_THREADS":"4"},"resources":{"gpus":0},"limits":{"max_files":38000,"max_total_bytes":1073741824},"job_token":"ttttttttttttttttttttttttttttttttttttttttttt"}

SHA-256 of those bytes: 68ef4653b8745eebf16427cd87a01e6d672b8062f8bfef819fcd1d745f0e346a

SHA-256 of shared/contract/compute-launch.schema.json: 0fb99d9b020fca1848c8008b0c84e996a3a9369aabea327796723658bfea6789 (18,959 bytes).

Decisions the issue left open

These are the owner's to confirm; each is one value in one place and marked in the code.

  • An existing test, test/contract-schemas.test.ts ("no schema file under shared/contract calls .strict()", epic Epic: Account tiers: verified base access, one-time upload approval, identity hardening, real-name attribution #1250), contradicts the issue's strict policy and launch specification. I kept the issue's strictness and exempted compute.ts and compute-launch.ts by name, and the new classification test and import guard keep the line between closed requests and documents and open responses where it is. Settled by the round 1 advisor; recorded here for the owner.
  • JOB_ENV_ALLOWLIST was not named. First list: OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS, PYTHONHASHSEED, MPLBACKEND, each value printable ASCII of at most 256 characters. Widening it changes the shared JSON Schema.
  • COMPUTE_REFUSAL_STATUS values were not given: 400 invalid_spec; 403 no_compute_access, suspended; 404 job_not_found; 410 results_expired; 413 bundle_too_large, bundle_too_many_files, results_too_large, too_many_outputs; 422 over_job_limit, partition_not_offered, runtime_unavailable, registry_not_allowed, dataset_not_eligible, recipe_unknown; 503 compute_disabled, pipelines_account_missing; 409 the rest.
  • Policy defaults the issue names but gives no number for: maxLiveJobsPerPerson 4; pipelines { dailyJobCap: 50, maxLiveJobs: { cpu: 8, gpu: 4 } }; settle { toleranceMilli: 1000, tolerancePercent: 5 }; renewal off with no interval; no clients; no registries; bundleLimits 100 MiB, 2,000 files and 25 MiB a file (the numbers in Phase D3: User job submission and nemar compute submit #2024). A partition with a ceiling in the policy is offered to people, one without is for pipelines only. Client caps are maxOpenReservations and maxReservedMilliPerDay. The bounds on the retention fields (records 1 to 30 years, logs 1 to 3650 days, results 14 to 365 days) are mine.
  • Ceilings: debug is 2 nodes of 128 cores for 30 minutes, gpu-debug 1 node of 4 GPUs for 30 minutes, a GPU node has 40 cores. All are unconfirmed and marked so beside the code. The 5 GB result file cap is decimal, 5,000,000,000, under R2's 5 GiB single-PUT limit.
  • Header grammar: a table header is unsupported_value; dotted and quoted keys are bad_line; a stray #x line in a block is bad_line; headerToSpecFields reuses unsupported_value for a bad memory, time or dataset; memory is binary (16G is 16384 MB); time takes h and m only; a dataset may be an id alone (the CLI resolves the version) and an id is two lowercase letters and six digits. The 256 KiB cap applies to the whole input, so a caller passes a bounded prefix.
  • Job specification: wire keys are snake_case (memory_mb; output_dir fixed at outputs); runtime is python, matlab or image (MATLAB is a valid specification that a route refuses); entry and command are exclusive; an entry needs the bundle; an image job needs a digest-pinned image; a recipe job needs none of them.
  • Launch specification: image { uri, digest, sif } is always present (a Python job runs NEMAR's runtime image); args, datasets, env, resources and limits are required so the Worker writes an explicit document; api_base may be http so stand-ins work; the provenance image.uri is still the looser ^[^\s]{1,512}. I added result, manifest and provenance to the schema's $defs and left out a complete shape because Phase A5: Job lifecycle, webhooks, settlement and results #2010 gives that route no body.
  • The ADR also spells out CLI, AWS and ADR on first use, beyond the four terms the review named.

What I could not do

  • The design page (admin/design/compute-gateway, sections 5 and 7) is not updated with the constants as shipped: the docs checkout is read-only for this phase, so that "When merged" step stays with the driver.
  • Nothing was recorded from a real tenant: the rate table, the debug ceilings and the preempt basis are the design's and stay marked unconfirmed (design section 12, question 5).
  • I did not run the whole bun test in rounds 1 or 2, as asked.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR2030

Version: 0.10.28-PR2030.33e80df
Commit: 33e80df


🤖 Updated on each push.

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Review triage, driver, 2026-11-04. Round 1 was a three-reviewer panel (contract and types, tests and errors, comments and simplification) plus an Opus advisor on the open specification questions; round 2 was one delta reviewer on the fix commits. All accepted findings are fixed on this branch; the advisor's rulings are recorded in the amendment comment on #2006.

Rejected, with the reason:

  • Cross-field invariants in the policy (per-job bytes at most per-person bytes at most the total): the owner sets these values and they are a policy choice, not a contract rule.
  • Splitting shared/contract/compute.ts: the issue prescribes one file.
  • The ADR date (2026-11-03): prescribed by the issue.
  • The zod and Ajv difference in counting non-BMP characters for maxLength: zod is the stricter side, which is the safe direction for what the Worker writes.
  • A settle-tolerance combination helper and tying runtime to entry or command: the design page does not specify either, so they belong to phases A2 and A5 and to the launcher, not to this contract.
  • The rate table as a policy row: the issue keeps RATE_TABLE_V1 a code constant, and the design wording is an open owner question that does not block A2.

Deferred to the later phases it concerns, by amendment comments on their issues: the refusal words of D3, the placement of declared.json (A5), the Tapis owner question (A4, A5).

@neuromechanist
neuromechanist merged commit c5816b7 into feature/issue-2005-epic-compute-core Nov 4, 2026
19 checks passed

This branch was previously deployed

1 inactive deployment
npm — 33e80dfc Deployed Nov 4, 2026 by neuromechanist via publish #4139
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant