Skip to content

Phase A2: Allocation ledger and owner policy #2007

Closed

Description

@neuromechanist

Goal

NEMAR can say how many SU a holder may spend and refuse a spend that would exceed it, atomically.
After this phase there is a versioned owner policy, per-holder pools (accounts, the platform's two budgets, partner clients), an append-only ledger whose totals can never disagree with it, a service that holds, releases, settles, adjusts and refunds, and owner and admin routes and CLI commands to set policy and grant SU.

Routing

Scope

In: migration 0114_compute_ledger.sql; backend/src/services/compute-ledger.ts and compute-policy.ts; admin routes; nemar admin compute policy|pool.
Out: the job table and the job-count guards on a hold (#2008), settle callers (#2010, #2012), automatic renewal and approval-time default grants (Epic D; the policy stores renewal and defaultGrantMilli and nothing here reads them), mail.

Design references

Design section 5 (pools, ledger, life of a charge), ADR 0118, ADR 0034 (no new datasets columns), DISPATCH_OPEN_CLAIM_SQL in backend/src/services/upstream-repull-dispatch.ts (one INSERT-SELECT with the check in its WHERE, meta.changes as the winner test), 0112_upstream_repull_automatic.sql (migration style).

Changes

  • Migration backend/src/db/migrations/0114_compute_ledger.sql, header comment per column, CHECKs that are TRUE or FALSE, no foreign keys, GLOBs of 50 characters or fewer: compute_policy(version INTEGER PRIMARY KEY AUTOINCREMENT, policy_json TEXT NOT NULL CHECK json_valid, set_by INTEGER NOT NULL, note TEXT, created_at), append-only by BEFORE UPDATE and DELETE triggers raising ABORT; the current policy is the highest version, and no row means DEFAULT_COMPUTE_POLICY. compute_pools(id, holder_kind IN ('account','platform','partner'), holder_ref TEXT, pool IN ('cpu','gpu'), granted_milli, held_milli, charged_milli, frozen, created_at, updated_at, UNIQUE(holder_kind, holder_ref, pool)) with each total at least 0; the migration seeds the four platform rows (pipelines and people, each cpu and gpu). compute_ledger(id, kind IN the eight, pool_id, budget_pool_id NULL, granted_delta_milli, held_delta_milli, charged_delta_milli, job_id TEXT NULL, hold_ref INTEGER NULL, reason TEXT NULL fixed word, actor_user_id NULL, idempotency_key TEXT NOT NULL UNIQUE, created_at) append-only by triggers, with CHECKs fixing which deltas each kind may carry (grant: granted+; hold: held+; release: held-; charge: held at most 0, charged at least 0; refund: charged-; adjust: exactly one of granted or charged nonzero; overrun: charged+; expire: granted-), a partial unique index on hold_ref for kinds release and charge (a hold closes once), and budget_pool_id allowed only for kinds that spend. Decision: an AFTER INSERT trigger on compute_ledger adds the three deltas to pool_id's row and the held and charged deltas to budget_pool_id's row. The ledger is therefore the only writer of totals, "only if the insert happened" holds by construction, and changes() is not needed. (If D1 refuses the trigger, stop and escalate; do not fall back to application-side updates.)
  • compute-ledger.ts (export every SQL as *_SQL): HOLD_INSERT_SQL, holdStatements(db, { poolId, budgetPoolId, amountMilli, jobId, idempotencyKey, extraGuards }) returning the one INSERT whose WHERE checks the pool not frozen, pool available at least the amount, budget pool available at least the amount, the key unused, and any caller guard; placeHold (runs it, reads which guard failed on zero changes and returns insufficient_balance, suspended or platform_budget_exhausted; a replayed key returns the original); releaseHold; settleProvisional(db, { jobId, chargeMilli }) (one charge entry closing the hold, charge capped at the hold, an overrun entry for any excess in the same batch); settleFinal (an adjust of the charged delta to the final figure, once per job); refundJob; grantPool; adjustPool; expireUnused; availableMilli(row) (granted minus held minus charged, may be negative); ensurePool(holderKind, holderRef, pool) (an idempotent insert; Phase A7: Tenant reconcile, observer and reporting #2012 uses it for the tenant partner pool).
  • compute-policy.ts: getPolicy(db) (validated, default on no row) and putPolicy(db, ownerId, input, note), which validates with the strict zod schema and inserts the version and an audit row in one batch.
  • backend/src/routes/admin/compute.ts exporting registerComputeAdminRoutes(admin), registered in routes/admin/index.ts: GET and PUT /compute/policy (owner, audit compute.policy.set); GET /compute/pools, GET /compute/pools/by-id/:accountId (admin); POST /compute/pools/by-id/:accountId/grant and /adjust (admin, Idempotency-Key header required, reason is a fixed word, audit compute.pool.grant and compute.pool.adjust); POST /compute/pools/platform/:key/grant and /adjust (owner). Errors are { error, code }; an invalid body names fields and never echoes values.
  • CLI: src/lib/api/compute.ts, src/commands/admin-compute.ts (computeCommand with policy show|set --file <json> [--note] and pool show|grant|adjust, amounts in SU through parseSuToMilli, one fresh Idempotency-Key per invocation), registered in src/commands/admin.ts beside mailCommand; add the module to MODULE_EXPORTS and POST_SPLIT_ADDITIONS in test/api-export-surface.unit.test.ts; pin the routes in test/admin-route-inventory.unit.test.ts.

Tests (write these first)

  1. backend/test/compute-ledger-migration.test.ts on bun:sqlite and Miniflare D1 (template: upstream-repull-dispatches-migration.test.ts): every CHECK refuses its bad row; update and delete of a ledger or policy row abort; run bun run migrations:d1-check.
  2. backend/test/compute-ledger-race.test.ts: on Miniflare D1, two concurrent placeHold for the last SU of one pool; exactly one wins, the other writes no ledger row and returns insufficient_balance. Mutation: delete the availability term from the WHERE; the test must fail.
  3. Same file: platform budget exhausted while the person has balance returns platform_budget_exhausted; a frozen pool returns suspended; a replayed idempotency key returns the first hold and moves no total.
  4. backend/test/compute-ledger.test.ts: after a seeded random sequence of every operation, each pool's three totals equal the sums of its ledger deltas (the trigger invariant); settle then final adjust ends at the final figure; excess use writes an overrun and negative available; a hold cannot be settled twice. Mutation: drop one delta from the trigger.
  5. backend/test/compute-policy-routes.test.ts and compute-pool-routes.test.ts through app.fetch: a member and an admin are refused on policy; owner PUT of an invalid body is 400 without echoing values; each write leaves an audit row; a repeated grant key grants once.
  6. test/admin-compute-cli.test.ts (template test/admin-repull-auto-cli.test.ts): policy set then show round trips; pool grant 12.5 moves 12500; a member's key exits 1.

Acceptance criteria

  • Totals equal ledger sums after the random-sequence test; the race test proves one winner.
  • No code path updates compute_pools totals directly (grep and a test).
  • bun run migrations:d1-check passes sequentially.
  • Routes and CLI module are pinned in the inventory and export-surface tests.

Gates

  • bun test backend/test/compute-ledger-migration.test.ts backend/test/compute-ledger-race.test.ts backend/test/compute-ledger.test.ts backend/test/compute-policy-routes.test.ts backend/test/compute-pool-routes.test.ts test/admin-compute-cli.test.ts test/admin-route-inventory.unit.test.ts test/api-export-surface.unit.test.ts
  • bun run migrations:d1-check
  • bun run typecheck
  • bunx biome check src shared backend/src
  • bun test

Rules that bite here

  • A refused hold writes nothing; the rate-limit counters are never a quota.
  • Integer thousandths; an overrun may make available negative but never a stored total.
  • The ledger is append-only and jobs or entries never cascade from the account row.
  • compute_policy is owner-only; D1 rejects SQL bun:sqlite accepts (GLOB length, compound SELECT terms).

Escalate to the owner if

  • D1 will not run the triggers.
  • Admin-only pool grants look too permissive once the platform budget is considered; the owner may want owner-only grants.

When merged

Update design section 5 with the table and entry names and the trigger decision, and note the new owner commands in the runbook list.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the phase A1 review (PR #2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • Response side is lenient. The CLI parses GET /compute/policy with a passthrough schema (unknown keys accepted at the top level and nested); ComputePolicySchema (strict) is only for checking policy set input and for putPolicy. Add a test where policy show prints a response that carries unknown keys at the top level and nested.
    • getPolicy stays strict on stored rows and fails closed (503) on a row it cannot parse. It never falls back to DEFAULT_COMPUTE_POLICY.
    • New policy keys are added as optional, so older stored rows keep parsing.
    • Reason: a released CLI must keep working when the Worker adds a policy key; the strict schemas of shared/contract/compute.ts are exempt from the old-CLI guard on the condition that no response is parsed with them.
  2. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #2032 (squash 941ba788), 2026-11-04, after a three-reviewer panel, the Opus checkpoint and its delta; every accepted finding fixed (triage on the pull request). Gates: 973 targeted tests across the ledger, contract and CLI suites, typecheck and the D1 migration check pass (178 objects); the full CI run is the epic gate.

    What later phases get, as built: compute_policy, compute_pools and compute_ledger (migration 0114) with thirteen triggers so the table itself refuses what the hold statement refuses, compute-ledger.ts (hold, release, settle, final, refund, grant, adjust, expire; the hold key is derived from the job; a replay of a closed hold is refused hold_closed with closedBy; a final after a refund is refused refunded; the final settlement always records its key), compute-policy.ts, the admin routes and nemar admin compute policy|pool.

    Open for the repository owner, needed before phase A5: whether a refund or a charge correction credits the people budget back (what the code does today, with account charge corrections owner-only as an interim rule) or only moves the person's pool. The schema does not decide it. Still owed from this phase's "When merged" step: the design page (section 5) and the runbook list of owner commands, by a docs pull request.

  3. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    A2 follow-up merged into the epic branch by #2044: the people budget now records consumption only (the owner's decision of 2026-11-04, option A). The refund path is deleted; a charge correction that lowers a pool's charge can never name the budget (the service refuses, the route names the budget only on a raise, and the statement itself carries the refusal); only a job's final settlement and the owner's own platform correction can lower the budget. Optional hardening left for the owner: a table trigger refusing a budget-naming lowering and kind refund (migration 0114 is unreleased).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementhpc

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions