Skip to content

Phase A7: Tenant reconcile, observer and reporting #2012

Closed

Description

@neuromechanist

Goal

NEMAR can prove that every job in the tenant is accounted for, finalize charges against Expanse's own records, report on any job, person or quarter, and keep records only as long as policy says.
This phase also owns the epic's end-to-end proof: an admin runs a pipeline for a fixture dataset and the job is held, submitted, advanced by webhook and cron, settled, finalized by an observer post, recorded and readable with nemar admin compute record, and revoking a test account mid-run records cancellation.

Routing

Scope

In: migration 0118_compute_observer.sql; the tenant sweep; observer ingest and final settlement; admin reporting commands and routes; legal holds; the retention sweep; the end-to-end test.
Out: dispatching the observer as a Tapis FORK application every 15 minutes (needs SDSC's answer, design section 12 question 6; this phase mints the run token and ingests posts), the partner contract and client pools beyond the one tenant partner pool used here (Epic D), admin mail about mismatches (the flag and jobs list --flag are the report), the website.

Design references

Design sections 5 (final settle, tolerance, overrun), 6 (jobs that skipped the reservation), 8 (observer), 9 (records, retention, reporting, access). Templates: routes/callbacks/repull-state.ts (unsigned sender, bounded body, uniform refusals); sweepUnreportedDispatches (watchdog); the daily prodOnlyJobs block in scheduled().

Changes

  • Migration 0118_compute_observer.sql: compute_observer_runs(id, nonce, created_at, expires_at, status IN ('issued','ingested','expired')); compute_queue_snapshots(run_id, partition, pending, running, captured_at); compute_sacct_rows(slurm_job_id PRIMARY KEY, run_id, partition, state, elapsed_seconds, alloc_nodes, alloc_cpus, alloc_gpus, alloc_mem_mb, submit_time, start_time, end_time, charged_milli NULL, matched_job_id NULL, ingested_at); compute_legal_holds(id, scope IN ('job','account'), ref, reason fixed word, placed_by, placed_at, released_at NULL). No foreign keys, no PII.
  • Observer: POST /admin/compute/observer-runs (admin, audited) mints a one-hour token (domain tag compute-observer over the run id and nonce); POST /compute/observer in routes/compute.ts, header X-Compute-Observer-Token, no session, bounded body (1 MiB, at most 5,000 rows), uniform refusals, one ingest per run. It stores snapshots and rows, then for each row matching a terminal job with a provisional settle and no final one calls Phase A2: Allocation ledger and owner policy #2007's settleFinal with charged_milli when sent, else recomputed by suMilliFor from the allocation fields. A difference within settle.toleranceMilli or tolerancePercent is routine; beyond it sets flag_mismatch and still settles to Expanse's figure. Rows matching no job are stored matched_job_id NULL and surface in jobs list. Pin the route in backend/test/compute-route-inventory.test.ts.
  • Tenant sweep sweepTenantJobs(env, "live" | "week"): lists the tenant with listType=ALL_JOBS through Phase A4: Tapis client and stand-in #2009's client (bounded pages, filter in code, so the unverified search syntax is not relied on), and for each Tapis job not bound by tapis_uuid first reads its tags; a nemar_job:<id> tag binds an existing row (the submit race) and is not cancelled. Otherwise: cancel at Tapis, insert an index row (source partner, spec_digest the sha256 of the Tapis fields read, cancel_reason unreserved, or unattributed when the owner maps to no active account, flags set), and write a charge entry (no hold) to the partner pool tenant with the budget pool people, never to a person. Jobs owned by PIPELINES_TAPIS_OWNER with a row are left alone. live runs on the 30-minute tick, week (last seven days) in the prodOnlyJobs daily block, both exact production only and only when computeEnabled; entries in cron-sweep-wiring.test.ts.
  • Reporting, routes in routes/admin/compute.ts and commands in src/commands/admin-compute-jobs.ts: nemar admin compute jobs list [--flag <name>] [--state] [--account <id>] [--limit] [--after]; record <nemar id | tapis uuid | slurm id> (index row, ledger entries, provenance, manifest summary; with --identity --reason <text> or --spec, owner only, through Phase A3: Job records and the results and records buckets #2008's readRecordObject); report --account <id> --from --to [--csv]; access-report --quarter <yyyy>Q<n> [--csv] (distinct gateway_user and job counts by partition and source for jobs submitted in the quarter, unattributed excluded); hold and release-hold on job:<id> or account:<id> (owner only, audited), through POST /compute/holds and DELETE /compute/holds/:id under /admin.
  • Retention runComputeRetentionSweep(env) (exact production, daily block, entry in the wiring test): for terminal jobs with no job or account legal hold, delete records/<id>/ after retention_until and logs/<id>/ once retention.logsDays have passed since ended_at, through the binding. A legal hold is a D1 flag the sweep honours (R2 has no versioning and no per-object hold, and the bucket lock covers only the retention period). The sweep never changes a lock; a lifecycle rule cannot delete a locked object, so this sweep is what deletes records, and any refusal on a records/ or logs/ key is counted locked, whatever its text, and retried the next day. Results expire by the bucket's lifecycle rule, so the sweep only reconciles: a results object still present a day after retention.resultsDays is deleted and counted late, and a hold never keeps a result. Then reduce the index row to anonymous counts (null account_id, tapis_owner, gateway_user, idempotency_key). One audit row of counts and one log line, nothing per job.
  • End-to-end test backend/test/compute-e2e.test.ts and a CLI leg that spawns the real CLI against a Bun.serve of the real routes (template test/admin-repull-auto-cli.test.ts). "Local backend" means that in-process Worker with ENVIRONMENT development and COMPUTE_ENABLED 1 (the configuration staging runs), real D1 and the Tapis, S3-API (over Miniflare R2) and ACCESS stand-ins; jobs advance by webhook and jobs advance, and the production-only crons are exercised by their own tests with a production-flagged env. Include a short comment saying so.

Tests (write these first)

  1. End to end, in order: fund the pipelines pool, pipeline run a fixture EEG dataset, hold present, submit seen by the stand-in, the launcher stand-in declares, uploads and completes its outputs through the outputs routes, a webhook hint then a cron tick advance it to finished, provisional charge, an observer post finalizes within tolerance, record <slurm id> shows the chain, ACCESS stand-in received one attribute. Then a second job, revoke its account mid-run, and assert account_ended, a frozen pool and a Tapis cancel after the sweep.
  2. Observer: a wrong, expired and reused token get identical refusals; an in-tolerance row settles silently, an out-of-tolerance row sets flag_mismatch and still settles; unmatched rows are listed. Mutation: widen the tolerance test to always true; the mismatch case must fail.
  3. Tenant sweep: an untagged job is cancelled, indexed unreserved and charged to the tenant pool and not to any account pool; a tagged submit-race job is bound, not cancelled; a pipeline-owned job with a row is untouched; a run when Tapis is unreachable changes nothing.
  4. Retention: honours job and account legal holds, counts locked (a bucket handle whose delete throws for locked keys, the one boundary fake, since Miniflare does not enforce locks), never touches a live job; a released hold lets the next run delete; a results object past its date is deleted and counted late.
  5. Commands: jobs list --flag unreserved, record by all three identifiers, --identity refuses an admin and needs a reason, access-report counts a person once however many jobs.

Acceptance criteria

  • The end-to-end test passes and is the proof recorded in the epic PR.
  • No unreserved job is ever charged to an account pool.
  • Identity and bundle reads are owner-only, reasoned and audited; admin views show no identity.
  • New crons are production-only, wired and self-fenced.

Gates

  • bun test backend/test/compute-e2e.test.ts backend/test/compute-observer.test.ts backend/test/compute-sweep.test.ts backend/test/compute-retention.test.ts backend/test/cron-sweep-wiring.test.ts test/admin-compute-jobs-cli.test.ts test/admin-route-inventory.unit.test.ts
  • bun run migrations:d1-check, bun run typecheck, bunx biome check src shared backend/src, then the full bun test

Rules that bite here

  • Nothing the job or the observer posts is trusted beyond its bounds; the observer is an unsigned sender with a per-run token.
  • Never charge a person for a job they did not reserve (ADR 0118); an admin may move a charge later, by an adjust entry.
  • A bucket lock is not bypassed by code; deletion at period end is by count, logged, never per job.
  • Admin views never expose identity; every owner read states a reason.

Escalate to the owner if

  • Expanse's accounting cannot be exported and the final settle needs another source.
  • The tenant listing returns only the credential's own jobs (the design's open question 2); the fallback is that NEMAR is the only submitter.

When merged

Update design sections 5, 6, 8 and 9 with the shipped names, record the local end-to-end commands and results in the epic pull request, and update the owner runbook for the observer token.

Activity

  1. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the phase A1 review (PR #2030) and the Opus advisor. This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • For an unreserved (partner) row, spec_digest is the SHA-256 of a spec.json written from the Tapis fields NEMAR read, and launch_spec_digest stays NULL. The end-to-end stand-in's launcher hashes the value it received from the Tapis stand-in.
    • The CLI's parse of nemar admin compute record output is passthrough (unknown keys accepted); it never uses a strict compute schema from shared/contract/compute.ts.
  2. neuromechanist commented on Nov 4, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the review and Opus checkpoint of phase A2 (the allocation ledger, #2032). This comment is part of the phase brief; where it differs from the body above, this comment wins.

    Retention and anonymisation of the ledger, which this phase owns: the ledger and policy tables are append-only (triggers abort UPDATE and DELETE), and design sections 5 and 9 say records are deleted at the end of the retention period and a deleted account's rows are reduced to anonymous usage counts. Plan the path in this phase with its own migration: allow ONLY an anonymising update (the account id in compute_pools.holder_ref and the actor in compute_ledger.actor_user_id replaced by a fixed anonymous token) under a narrowly scoped exception, never any change to an amount, a kind, a job or a hold, and never a delete of a row that still has a live hold; state in the plan how the totals invariant (pool totals equal the sums of ledger deltas) stays true. Record the decision in the ADR 0118 amendment and the design page. The A2 migration header says retention is this phase's job.

  3. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Driver amendment, 2026-11-04, from the Opus checkpoints of phases A5 (#2041) and A8 (#2046). This comment is part of the phase brief; where it differs from the body above, this comment wins.

    • Reporting must list ACCESS report rows in the failed state by their fixed word (outcome_unknown, no_gateway_user, no_submit_time, unconfigured, and the retry-exhausted words) and the tenant sweep must surface them; add an owner-only re-queue command that requires an explicit acknowledgement that the report is confirmed NOT recorded at ACCESS (an unknown outcome is never resent automatically: a report may be lost, never doubled).
    • Open points carried from A5 for the tenant reconcile: UNSETTLED_TERMINAL_SQL and UNCOPIED_TERMINAL_SQL order by age with no rotation (a copy that fails on every attempt holds a slot for three days; ten or more such rows delay newer copies); ORPHAN_ROWS_SQL ordering is static for terminal rows; SET_ORPHAN_SQL records one orphan per row; the 30-minute tick equals the default reservation wait; terminal rows whose Tapis record is permanently unreadable are retried for ever; CANCEL_ROWS_SQL takes the oldest ten (ten or more stuck cancels block newer ones, and the revocation cascade adds many intents at once); the staging Worker never cancels at Tapis (the cancel sweep is production-only).
    • A frozen pool is never unfrozen by this epic: Epic D's compute-access grant must unfreeze in the same batch as the grant and record the freeze cause (or derive frozen from the access state), because the revocation cascade and Epic D's suspension both freeze through the same bit.
    • A partner job with no account has no gateway pseudonym and ends its ACCESS report failed with no_gateway_user: the partner contract (Epic D phase D4) must supply a pseudonym for account-less partner jobs (for example per client and Tapis owner).
    • The revocation cascade leaves a partner reservation alone (source = 'partner': the partner contract's); a partner reservation of an ended account is Epic D's to handle.
  4. neuromechanist commented on Nov 6, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the compute epic (PR 2055, e6ca96f).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementhpc

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions