Repository navigation
Phase A1: Compute contract and ADR 0118 - #2030
Merged
neuromechanist merged 22 commits intoNov 4, 2026
Merged
neuromechanist merged 22 commits into
neuromechanist merged 22 commits into
Conversation
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR2030Version: 🤖 Updated on each push. |
This was referenced Nov 4, 2026
Contributor
Author
|
Review triage, driver, 2026-11-04. Round 1 was a three-reviewer panel (contract and types, tests and errors, comments and simplification) plus an Opus advisor on the open specification questions; round 2 was one delta reviewer on the fix commits. All accepted findings are fixed on this branch; the advisor's rulings are recorded in the amendment comment on #2006. Rejected, with the reason:
Deferred to the later phases it concerns, by amendment comments on their issues: the refusal words of D3, the placement of |
neuromechanist
merged commit Nov 4, 2026
c5816b7
into
feature/issue-2005-epic-compute-core
19 checks passed
8 of 9 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase A1 of the compute gateway core (epic #2005, issue #2006): the written contract before any table or route exists. Nothing here touches D1, the Worker, the CLI or the network. This body is current as of the round 2 review fixes (head
33e80dfc).What changed
COMPUTE_ENABLEDand its test moved to phase A4, and two new decisions (two digests; responses open, requests and documents closed).shared/compute-state.ts(import-free): the ten job states and the only edges between them,stateFromTapisStatus, the launcher and platform failure codes withisPlatformFailureandfailureCodeFromTapisCondition(mapped from the whole published Tapis condition list), the refusal codes withCOMPUTE_REFUSAL_STATUS, and the ledger, source, kind, flag and cancel words.over_ceilingis gone: the per-job limit refusal isover_job_limit.shared/contract/compute.ts(zod, re-exported fromshared/contract/index.ts):JobSpecSchema,PARTITIONS,RATE_TABLE_V1,estimateSuMilli,suMilliFor,parseSuToMilli,formatSu,ComputePolicySchemawith the frozenDEFAULT_COMPUTE_POLICY, the launcher constants (includingRECORDS_RESULT_MANIFEST_FILE,COMPUTE_PATH_SEGMENT,COMPUTE_MAX_OUTPUT_FILES) and the shapes ofresult.json,manifest.jsonandprovenance.json.shared/contract/compute-launch.tsandshared/contract/compute-launch.schema.json: the launch specificationnemar.launch-spec.v1, the bundle, outputs and outputs-response shapes, the results-bucket keys,encodeLaunchSpec(ASCII-only JSON, the 64 KiB cap checked after escaping) andlaunchSpecFits. The schema is meant to be copied byte for byte bynemarOrg/nemar-process.shared/compute-header.ts(import-free):parseJobHeaderfor# /// nemar,% /// nemarandnemar-job.toml(an unknown kind throws RangeError), andheaderToSpecFields.test/contract-schemas.test.ts: the strict-schema guard exemptsshared/contract/compute.tsandcompute-launch.tsby name; a classification table checks every exported zod schema of the two files that is, or holds, an object (an object, an array of them, a union) by behaviour (nine strict, two lenient; an unlisted one fails, and so does a kind of schema the walk does not know), and an import guard keeps strict names out of every other file undershared/contract, however they are taken (static, through./index.js, with a.tsextension, beside a default, or by a dynamicimport()orrequire())..context/compute-program/implementation-notes.mdsaying where the words now live.Round 1 review fixes
BundleResponseSchema,OutputsResponseSchemaand itsuploadsitems are.passthrough(), and the schema has noadditionalProperties: falseonbundle_response,outputs_responseor its upload items (a running launcher must survive the Worker adding a key); requests and documents stay closed.launch_spec_digest, the SHA-256 of the exact raw bytes of theNEMAR_LAUNCH_SPECvalue, hashed before parsing.spec_digeststays the job row's digest ofrecords/<id>/spec.json(a comment, not a launcher key).\uXXXXescape) and the cap is checked after escaping.gpusof at least 1 and a CPU partition 0.JobSpecSchemarather than tightening per-field caps, because a non-ASCII character costs six bytes once escaped and no per-field cap that is generous for ASCII is also safe for any mix. The entry, command, arguments, include patterns and environment together may take 49,152 bytes (LAUNCH_VARIABLE_BYTES_MAX, counted bylaunchVariableBytes); everything else is under 4.3 KiB at its own worst case, which leaves about 12 KiB of slack.launchSpecFits(job)lets the Worker refuse withinvalid_specbefore taking a hold. The per-field caps are unchanged, so a job with 64 arguments of 1,024 plain characters is now refused. Round 2 adds the other half of "a valid job always launches": every string that reaches a process must be well-formed UTF-16.image.urifollows the job's own digest-pinned reference grammar andimage.sifis a relative path ofA-Za-z0-9_.+-segments with no leading dash, no.or..segment and no leading or trailing slash. The contract states that the launcher must run the container with a clean environment.strict: true, strictRequired: false) and by zod on both sides of its edge (70 rows); a second table does the same for the job specification and the owner policy (97 rows). To let strict Ajv compile, the fourcontainssubschemas gainedtype: object; I also removed theminLengthandminItemskeywords (and the matching zod.min(1)) that a pattern or thecontainsclauses already imply.$idis now the file's GitHub address, the formdeploy/neurobagel/index.schema.jsonuses; the API serves only the Zarr schemas.Round 2 review fixes
JobSpecSchemarefuses a lone surrogate in the entry, a command item, an argument and an include pattern, each issue naming the field by its path (entry,command.1,args.1,datasets.0.include.1) with the message "a lone surrogate is not text a process argument can carry", never the value. Such a string is valid JavaScript and encodes as\ud800, but no process argument can carry it, so the job would have failed only after its hold existed. An environment value needs no check (printable ASCII only), and a test pins that. The check iscomputeIsWellFormed, a loop over the UTF-16 units: this tree builds for ES2022 andString.prototype.isWellFormedis newer, so a test compares the loop withisWellFormedinstead (all 2,048 surrogate units alone, a stride of every high and low pair, and a seeded fuzz of 20,000 strings). The shared JSON Schema is unchanged (it still lets a lone-surrogate escape through); the Worker's refusal is the guard, so a valid job specification still always fits the launch specification, and now is also well-formed.strictComputeTakes) that judges a specifier by the file it resolves to, so./compute,./compute.js,./compute.tsand../contract/compute.jsare one file. It reports a strict name or a namespace taken from either strict file or from./index.js(also./index,./index.ts,.and..), a default beside named or namespace imports, and any dynamicimport()orrequire()of them (a dynamic specifier that is not a plain string is reported as unresolvable). It still allowsindex.ts's ownexport *,import type { JobSpec }and non-strict names. 68 probes pin each shape, flagged and allowed, with the same function scanning the real files.launchSpecFitsworst case: three job shapes (an entry with an environment and include patterns, an image command with a GPU, and a recipe with neither) each have the exact size of the worst-case document with no arguments pinned as a literal (2,119, 2,012 and 2,016 bytes, cross-checked in Python), andlaunchSpecFitsis held to that document to the byte: with arguments that bring it to 65,536 bytes it fits, at 65,537 it does not. A second test makes each Worker field of the worst case the largest the schema allows (one past it is refused). Shrinking or dropping any one field of either document now fails a test, including a bundlesizeof 1.launchSpecFitsand the worst-case document, and every case of the schema walk). Each survivor of a first pass became a probe or a simplification, and all are now killed except two equivalent ones: lowering the start of the low-surrogate range by one unit (the high branch has already taken that unit) and a bundle size of 9007199254740990 for 9007199254740991 (the same number of digits, so the same bytes).Gates
bun test test/compute-state.unit.test.ts test/compute-contract.unit.test.ts test/compute-header.unit.test.ts test/adr-index.unit.test.ts test/contract-schemas.test.ts: 552 pass, 0 fail, 70725 expect() calls, across 5 files.bun run typecheck(tsc --noEmitfor the CLI and the backend): exit 0, no output.bunx biome check shared test/compute-state.unit.test.ts test/compute-contract.unit.test.ts test/compute-header.unit.test.ts test/contract-schemas.test.ts:Checked 91 files. No fixes applied.jsonschema(Draft 2020-12) over the shared schema with 58 documents and checks, plus the golden vector (the SHA-256 of the raw bytes, then the parse): 0 disagreements at round 1. The schema file has not changed since (same SHA-256 and size below), so it was not run again.compute-launch.tsandcompute.tswas bumped by one in each direction (185, 55 and 174 mutants): all killed, 0 survivors, including each of the reviewer's survivors. About 860 single-line mutants were killed in all. The mutants still alive are equivalent: theKmemory unit (it has no multiplier, so it is refused anyway) and the exemption filter in the import guard (neither exempt file imports a strict name from the other), plus the control mutation that imports a lenient name, which the guard allows by design.bun testis for the driver to run on the final head. Note that abun testwas running in this worktree while I edited in round 1 (started about 01:21 by someone else), so any result from before this head is stale.test/contract-schemas.test.tshas one TypeScript error that is not from this branch (line 494, an older user-status test; the file is outside the repo's typecheck include).Golden vector for nemar-process
One fixed launch specification (a non-ASCII character, an astral character, a double quote and a space in
args), the exact textencodeLaunchSpecwrites for it and the SHA-256 of that text, all pinned as literals intest/compute-contract.unit.test.ts. The launcher must report the digest for these bytes and parse them back to the same specification.Specification:
{ job_id: "job_0123456789abcdef01234567", runtime: "python", entry: "run.py", args: ["--label", 'café "q" \u{1F600}'], bundle: { sha256: "a" x 64, size: 1234 }, datasets: [{ id: "nm000104", version: "1.0.2" }], api_base: "https://api.example.test", image: { uri: "docker://ghcr.io/example/runtime@sha256:" + "b" x 64, digest: "sha256:" + "b" x 64, sif: "runtime.sif" }, env: { OMP_NUM_THREADS: "4" }, resources: { gpus: 0 }, limits: { max_files: 38000, max_total_bytes: 1073741824 }, job_token: "t" x 43 }Encoded text (708 bytes, ASCII):
SHA-256 of those bytes:
68ef4653b8745eebf16427cd87a01e6d672b8062f8bfef819fcd1d745f0e346aSHA-256 of
shared/contract/compute-launch.schema.json:0fb99d9b020fca1848c8008b0c84e996a3a9369aabea327796723658bfea6789(18,959 bytes).Decisions the issue left open
These are the owner's to confirm; each is one value in one place and marked in the code.
test/contract-schemas.test.ts("no schema file under shared/contract calls .strict()", epic Epic: Account tiers: verified base access, one-time upload approval, identity hardening, real-name attribution #1250), contradicts the issue's strict policy and launch specification. I kept the issue's strictness and exemptedcompute.tsandcompute-launch.tsby name, and the new classification test and import guard keep the line between closed requests and documents and open responses where it is. Settled by the round 1 advisor; recorded here for the owner.JOB_ENV_ALLOWLISTwas not named. First list:OMP_NUM_THREADS,MKL_NUM_THREADS,OPENBLAS_NUM_THREADS,NUMEXPR_NUM_THREADS,PYTHONHASHSEED,MPLBACKEND, each value printable ASCII of at most 256 characters. Widening it changes the shared JSON Schema.COMPUTE_REFUSAL_STATUSvalues were not given: 400invalid_spec; 403no_compute_access,suspended; 404job_not_found; 410results_expired; 413bundle_too_large,bundle_too_many_files,results_too_large,too_many_outputs; 422over_job_limit,partition_not_offered,runtime_unavailable,registry_not_allowed,dataset_not_eligible,recipe_unknown; 503compute_disabled,pipelines_account_missing; 409 the rest.maxLiveJobsPerPerson4;pipelines{ dailyJobCap: 50, maxLiveJobs: { cpu: 8, gpu: 4 } };settle{ toleranceMilli: 1000, tolerancePercent: 5 };renewaloff with no interval; no clients; no registries;bundleLimits100 MiB, 2,000 files and 25 MiB a file (the numbers in Phase D3: User job submission and nemar compute submit #2024). A partition with a ceiling in the policy is offered to people, one without is for pipelines only. Client caps aremaxOpenReservationsandmaxReservedMilliPerDay. The bounds on the retention fields (records 1 to 30 years, logs 1 to 3650 days, results 14 to 365 days) are mine.debugis 2 nodes of 128 cores for 30 minutes,gpu-debug1 node of 4 GPUs for 30 minutes, a GPU node has 40 cores. All are unconfirmed and marked so beside the code. The 5 GB result file cap is decimal, 5,000,000,000, under R2's 5 GiB single-PUT limit.unsupported_value; dotted and quoted keys arebad_line; a stray#xline in a block isbad_line;headerToSpecFieldsreusesunsupported_valuefor a bad memory, time or dataset; memory is binary (16Gis 16384 MB); time takeshandmonly; a dataset may be an id alone (the CLI resolves the version) and an id is two lowercase letters and six digits. The 256 KiB cap applies to the whole input, so a caller passes a bounded prefix.memory_mb;output_dirfixed atoutputs);runtimeispython,matlaborimage(MATLAB is a valid specification that a route refuses);entryandcommandare exclusive; anentryneeds thebundle; animagejob needs a digest-pinnedimage; a recipe job needs none of them.image { uri, digest, sif }is always present (a Python job runs NEMAR's runtime image);args,datasets,env,resourcesandlimitsare required so the Worker writes an explicit document;api_basemay behttpso stand-ins work; the provenanceimage.uriis still the looser^[^\s]{1,512}. I addedresult,manifestandprovenanceto the schema's$defsand left out acompleteshape because Phase A5: Job lifecycle, webhooks, settlement and results #2010 gives that route no body.What I could not do
admin/design/compute-gateway, sections 5 and 7) is not updated with the constants as shipped: the docs checkout is read-only for this phase, so that "When merged" step stays with the driver.debugceilings and the preempt basis are the design's and stay marked unconfirmed (design section 12, question 5).bun testin rounds 1 or 2, as asked.