Skip to content

Phase A8: Revocation cascade and ACCESS gateway attribution #2013

Closed

Description

@neuromechanist

Goal

Ending an account ends its compute, and ACCESS learns the end user of every job.
After this phase, revoking or deleting an account records the cancellation of every live job and frees unsubmitted holds in the same database batch as the other credential cascades, a retried sweep cancels at Tapis, and NEMAR reports each Slurm job's pseudonymous gateway user to ACCESS exactly once.

Routing

Scope

In: migration 0119_compute_access_reports.sql, computeReleaseStatements, its two caller sites, the ACCESS reporter and its cron.
Out: suspension of compute and its grant changes (Epic D), OIDC grant revocation (Epic B), the tenant-wide sweep (#2012).

Design references

Design sections 7 (cancellation and revocation) and 9 (attribution to ACCESS). AGENTS.md "Revocation cascades, always". Templates: mailReleaseStatements in services/ticket-assignment.ts; the 30-minute tick in scheduled().

Changes

  • Revocation: computeReleaseStatements(userId) in services/compute-release.ts (template mailReleaseStatements): for jobs not yet at Tapis, cancel and release the hold in SQL; for jobs at Tapis, set cancel_reason = 'account_ended' and cancel_requested_at; set frozen = 1 on the account's pools. Every statement must end in a WHERE clause, because tombstoneUserCascade appends a guard to each. Add it to the batch of finalizeRevocation (routes/admin/users.ts) and to tombstoneUserCascade (db/user-tombstone.ts), which the owner soft delete and the test-account sweep both run, so there are two caller sites, not three. Role demotion and key revocation deliberately do not end compute jobs; say why in the docblock. Update the caller-count sentences in services/docs-auth.ts and ticket-assignment.ts and extend backend/test/admin-credential-cascade.test.ts.
  • Migration 0119_compute_access_reports.sql: compute_access_reports(job_id PRIMARY KEY, state IN ('pending','sent','failed'), attempts, next_attempt_at, last_error_code fixed word NULL, sent_at NULL, created_at). Fill Phase A5: Job lifecycle, webhooks, settlement and results #2010's hook: the batch in which a job's record first shows a Slurm id inserts a pending row (INSERT OR IGNORE, so a replayed read never adds a second).
  • services/access-attributes.ts: POST { jobid, submittime, xsederesourcename, gatewayuser } to <base>/job_attributes (base from globalThis.NEMAR_ACCESS_ATTRIBUTES_URL, else the ACCESS address) with debug set outside production; key from the secret ACCESS_GATEWAY_ATTRIBUTES_KEY, resource name from ACCESS_RESOURCE_NAME, both optional (a row stays pending with unconfigured); exponential retry, failed after a bound. The key header name and body spelling are unverified: isolate them in two constants and say so. The gateway user is the job row's gateway_user (Phase A3: Job records and the results and records buckets #2008), and nemar-pipelines for pipeline jobs.
  • runAccessReportCron(env) sends due rows; wire it into the 30-minute tick's production-only block like Phase A5: Job lifecycle, webhooks, settlement and results #2010's crons, self-fenced with isExactProduction, with a tick-prod-only entry in cron-sweep-wiring.test.ts.

Tests (write these first)

  1. Cascade: revoking and tombstoning an account mid-run records account_ended, frees unsubmitted holds, freezes pools, and the retried sweep cancels at Tapis; mutation: remove the statement from either caller.
  2. ACCESS reporter against a stand-in: body fields, debug outside production, retry then failed, never sent twice for one job.
  3. A replayed status read after the Slurm id first appears inserts no second report row. Mutation: drop OR IGNORE.

Acceptance criteria

  • Revocation and deletion record cancellation of every live job in the same batch as the other cascades, through both caller sites.
  • No Tapis or ACCESS call runs inside a database batch.
  • Every Slurm job is reported once, with the pseudonymous gateway user, and never with a name or email.
  • Cascade and wiring tests are updated and pass.

Gates

  • bun test backend/test/admin-credential-cascade.test.ts backend/test/compute-lifecycle.test.ts backend/test/cron-sweep-wiring.test.ts backend/test/cron-dev-safety.test.ts
  • bun run migrations:d1-check, bun run typecheck, bunx biome check src shared backend/src, bun test

Rules that bite here

  • Revocation cascades, always: the batch, the counts and a retried sweep for the network call, never a Tapis call inside the batch.
  • Every release statement ends in a WHERE clause, because tombstoneUserCascade appends a guard to each.
  • The ACCESS key is a Worker secret the owner sets; never put it on a command line or in a job.
  • New crons are production-only.

Escalate to the owner if

  • Tapis cannot cancel a job the credential did not submit.
  • The ACCESS interface refuses the pseudonymous gateway user, or needs fields the design does not list.

When merged

Update the revocation paragraph of design section 7 and the attribution paragraph of section 9 as shipped, and list the reporter cron in the systems inventory runbook.

Activity

  1. neuromechanist commented on Nov 5, 2026

    @neuromechanist
    ContributorAuthor

    Merged into the epic branch by #2046 (squash cf34441). Round 1 (Opus checkpoint), round 2 (Sonnet delta) and a rebase after the A2 follow-up converged. Revocation and deletion record the cancellation of every live NEMAR job in the same batch as the other cascades (both caller sites), partner reservations are left to the partner contract, the sweep cancels at Tapis, and every Slurm job is reported to ACCESS at most once with the pseudonymous gateway user (a report may be lost, never doubled). Owner items: the secret is ACCESS_GATEWAY_ATTRIBUTES_KEY plus ACCESS_RESOURCE_NAME (both owner-set Worker secrets, pinned absent from the toml); the ACCESS interface constants (header name, body spelling, time format, debug field, duplicate behaviour) are UNVERIFIED and must be confirmed when the key is issued. Carried to other phases: A7 (list failed report rows, owner re-queue with acknowledgement, the A5 and A8 open points), Epic D (freeze cause, pseudonym for account-less partner jobs, the partner-reservation items).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or enhancementhpc

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions