Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .context/phase5-cross-repo-owner-deploys.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Phase 5 cross-repo owner deploys (epic #923)

These changes live in the **`nemarDatasets/.github`** repo (the central-workflow repo) and in
prod infra, NOT in nemar-cli. Phase 5 in nemar-cli only adds the dispatch payload fields and the
config-gated forwarder; the workflows that CONSUME them must be updated by the owner. All are
backward-compatible: each new field is optional and the workflows default to the current prod
behavior when it is absent, so they can be deployed before or after the nemar-cli merge.

## 1. Central workflows read `s3_bucket` (required for the dev-bucket split)

nemar-cli now sends `s3_bucket` in the `client_payload` of `generate-manifest`, `generate-archive`,
and `generate-zarr` (`backend/src/services/github/dispatch.ts`), sourced from the worker's
`S3_BUCKET` (prod `nemar`, dev `nemar-dev`). The corresponding workflows in `nemarDatasets/.github`
currently hardcode `s3://nemar`. Update each to:

```yaml
env:
S3_BUCKET: ${{ github.event.client_payload.s3_bucket || 'nemar' }}
```

and replace every hardcoded `s3://nemar/...` / `--bucket nemar` with `s3://${S3_BUCKET}/...`. Affected
workflows: `run-generate-manifest.yml` (required — without it, dev manifests land in the prod
bucket), `run-generate-archive.yml`, `run-generate-zarr.yml`.

## 2. Workflows read `callback_base_url` (staging callbacks hit the dev worker)

`generate-archive` / `generate-zarr` now also receive `callback_base_url` (the worker's
`API_BASE_URL`; prod `https://api.nemar.org`, dev `https://api-test.nemar.org`). Build their POST-back
URLs from it instead of a hardcoded host:

```yaml
CALLBACK_BASE="${{ github.event.client_payload.callback_base_url || 'https://api.nemar.org' }}"
# ...POST "$CALLBACK_BASE/webhooks/archive-ready" (or /webhooks/zarr-ready)
```

(`generate-manifest` already receives a fully-built `callback_url` from nemar-cli, so it needs no
`callback_base_url` change.)

## 3. `emit_manifest.py` — parameterize `DATA_NEMAR_BASE` (Phase 4 comment-review follow-up)

`nemarDatasets/.github` `scripts/emit_manifest.py` hardcodes `DATA_NEMAR_BASE = "https://data.nemar.org"`
in the git-committed raw manifest's `bytes_url`. Phase 4 parameterized the *served* `bytes_url`
(`DATA_BASE_URL` -> `data-test.nemar.org` on staging), so for staging exemplars the two now disagree.
Make `emit_manifest.py` read an override with the prod default:

```python
DATA_NEMAR_BASE = os.environ.get("DATA_NEMAR_BASE", "https://data.nemar.org")
```

and have `run-generate-manifest.yml` pass `DATA_NEMAR_BASE` from a new
`client_payload.data_base_url` (add it to `triggerManifestGeneration` alongside `s3_bucket` when this
is deployed) so the raw and served `bytes_url` match on staging. Prod leaves it unset -> byte-identical.

## 4. `onboard-openneuro.yml` — parameterize the hardcoded webhook host (low urgency)

`onboard-openneuro.yml` has five hardcoded `https://api.nemar.org/webhooks/import-state` literals.
Replace with `${{ github.event.client_payload.callback_base_url || 'https://api.nemar.org' }}` so a
dev-worker-dispatched onboarding calls back to the dev worker. Low urgency (staging exemplars use the
clone tool, not OpenNeuro onboarding), but worth folding into the same pass.

## Deploy order

Deploy 1 + 2 before pointing the dev worker at the exemplar fleet (else dev manifests/archives/zarr
write to the prod bucket / call back to the prod worker). 3 + 4 can follow. None affect prod: absent
fields -> prod defaults.
43 changes: 43 additions & 0 deletions backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ import { reconcileReservedVersionDois } from "./services/doi-reconcile";
import {
getAdminEmailsForCategory,
resolveEmailConfig,
sendExemplarInvariantAlertEmail,
sendStalenessAdminReviewEmail,
sendStalenessWarningEmail,
} from "./services/email";
Expand Down Expand Up @@ -268,6 +269,48 @@ async function scheduledCleanup(env: Bindings): Promise<void> {
}> = [];
const MAX_DELETIONS_PER_RUN = 10;

// Prod-invariant guard (epic #923): the Phase 4 visibility SQL carve-outs admit
// is_exemplar=1 rows with NO runtime env check, relying on the invariant that
// production D1 never has such a row (the creation endpoint 403s in prod). If
// one ever appears here, a bug bypassed that gate and the row is silently
// public across catalog/search/data-index — surface it loudly rather than
// letting it hide.
if (env.ENVIRONMENT === "production") {
try {
const exemplarLeak = await db
.prepare("SELECT COUNT(*) as n FROM datasets WHERE is_exemplar = 1")
.first<{ n: number }>();
const leakCount = exemplarLeak?.n ?? 0;
if (leakCount > 0) {
console.error(
`[cleanup] INVARIANT VIOLATION: ${leakCount} is_exemplar=1 row(s) exist in PRODUCTION. These are staging-only and are now silently public via the exemplar visibility carve-outs. Investigate the exemplar creation gate immediately.`,
);
// Active escalation: a silent public data exposure must page a human, not
// sit in Worker Logs until someone happens to read them.
if (env.RESEND_API_KEY) {
try {
const emailCfg = resolveEmailConfig(env);
const adminEmails = await getAdminEmailsForCategory(db, "publication_request");
if (adminEmails.length > 0) {
await sendExemplarInvariantAlertEmail(
adminEmails,
leakCount,
env.RESEND_API_KEY,
emailCfg.fromEmail,
emailCfg.replyTo,
emailCfg.isDev,
);
}
} catch (emailErr) {
console.error("[cleanup] failed to send exemplar-invariant alert email:", emailErr);
}
}
}
} catch (err) {
console.error("[cleanup] exemplar-invariant check failed:", err);
}
}

/** Delete each dataset in `rows`, pushing outcomes into `results`. */
async function deleteRows(rows: Array<{ dataset_id: string }>): Promise<void> {
for (const row of rows) {
Expand Down
2 changes: 1 addition & 1 deletion backend/src/routes/admin/datasets-lifecycle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1569,7 +1569,7 @@ export function registerDatasetLifecycleRoutes(admin: AdminRouter): void {
callbackToken,
callbackUrl,
pat,
{ skipCanary: skip_canary ?? false, skipCallback },
{ skipCanary: skip_canary ?? false, skipCallback, s3Bucket: c.env.S3_BUCKET },
);
return c.json({ dispatched: true, dataset_id, version });
} catch (err) {
Expand Down
259 changes: 259 additions & 0 deletions backend/src/routes/admin/exemplar.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,259 @@
/**
* Admin routes: staging exemplar datasets (epic #923, Phase 5).
*
* A curated fleet of xx-prefixed "exemplar" copies of real datasets that pass
* through the full publish / DOI / reindex pipeline on the staging environment
* (test.nemar.org). These endpoints are the FIRST writers of `is_exemplar=1`;
* they 403 in production so a prod D1 row can never carry the flag — the Phase 4
* visibility carve-outs rely on that invariant (see services/exemplar.ts).
* Modeled on routes/admin/imports.ts (repo-create -> D1-insert -> rollback).
*/

import { zValidator } from "@hono/zod-validator";
import { z } from "zod";
import { auditLogStatement } from "../../db/audit-log";
import type { DataCiteEnrichment } from "../../services/datacite";
import {
buildOrcidEnrichment,
createConceptDoi as dispatchCreateConceptDoi,
} from "../../services/doi";
import { isNonProductionEnv } from "../../services/environment";
import { type GitHubRepo, createRepository, deleteRepository } from "../../services/github";
import { getDatasetsToken } from "../../services/github-auth";
import { extractRepoName, readRepoMetadata } from "../../services/repo-metadata";
import type { AdminRouter } from "./shared";

/** Exemplar id band xx099900-xx099999 (canonical 8-char id: xx + 6 digits). */
export const EXEMPLAR_ID_RE = /^xx0999\d{2}$/;
/** Source dataset the exemplar is cloned from (an nm/on dataset). */
const SOURCE_ID_RE = /^(nm|on)\d{6}$/;

export function registerExemplarRoutes(admin: AdminRouter): void {
const createExemplarSchema = z.object({
dataset_id: z
.string()
.regex(EXEMPLAR_ID_RE, "Exemplar id must be in the xx099900-xx099999 band"),
source_id: z
.string()
.regex(SOURCE_ID_RE, "source_id must be an nm/on dataset id (e.g. nm000132)"),
name: z.string().min(1).max(200).optional(),
description: z.string().optional(),
});

/**
* POST /admin/datasets/exemplar
*
* Create a staging exemplar: a private nemarDatasets/<xx> repo + a D1 row with
* is_exemplar=1. The CLI clone tool then copies data into the dev bucket and
* publishes. 403 in production (staging-only fleet).
*/
admin.post("/datasets/exemplar", zValidator("json", createExemplarSchema), async (c) => {
// is_exemplar=1 must never exist in a production D1 (Phase 4 gate invariant).
if (!isNonProductionEnv(c.env)) {
return c.json(
{
error: "Exemplar creation is disabled in production",
message:
"Exemplar (xx099900-xx099999) datasets are a staging-only fleet; create them on the dev/test worker.",
},
403,
);
}

const { dataset_id, source_id, name, description } = c.req.valid("json");
const db = c.env.DB;
const adminUser = c.get("user");
const displayName = name || `[TEST COPY] ${source_id}`;

const existing = await db
.prepare("SELECT dataset_id FROM datasets WHERE dataset_id = ?")
.bind(dataset_id)
.first<{ dataset_id: string }>();
if (existing) {
return c.json({ error: `Dataset ${dataset_id} already exists` }, 409);
}

const pat = await getDatasetsToken(c.env);
let githubRepo: GitHubRepo;
try {
githubRepo = await createRepository(
dataset_id,
`${displayName} - NEMAR staging exemplar (cloned from ${source_id})`,
true,
pat,
);
} catch (error) {
const msg = error instanceof Error ? error.message : String(error);
if (msg.includes("already exists")) {
return c.json({ error: `GitHub repo nemarDatasets/${dataset_id} already exists` }, 409);
}
console.error("Failed to create GitHub repo for exemplar:", error);
return c.json({ error: `Failed to create GitHub repository: ${msg}` }, 500);
}

try {
await db
.prepare(
`INSERT INTO datasets (dataset_id, name, description, owner_user_id, github_repo, is_sandbox, is_exemplar, visibility, source, source_id, last_activity_at)
VALUES (?, ?, ?, ?, ?, 1, 1, 'private', 'nemar-exemplar', ?, datetime('now'))`,
)
.bind(
dataset_id,
displayName,
description || null,
adminUser.id,
githubRepo.full_name,
source_id,
)
.run();
} catch (error) {
console.error("Failed to insert exemplar record:", error);
const dbMsg = error instanceof Error ? error.message : String(error);
try {
await deleteRepository(dataset_id, pat);
} catch (cleanupErr) {
console.error("Failed to clean up GitHub repo after D1 failure:", cleanupErr);
return c.json(
{
error: `Failed to create exemplar record: ${dbMsg}. GitHub repo nemarDatasets/${dataset_id} was created but could not be cleaned up. Manual deletion required.`,
},
500,
);
}
return c.json({ error: `Failed to create exemplar record: ${dbMsg}` }, 500);
}

try {
await auditLogStatement(db, {
userId: adminUser.id,
action: "exemplar_created",
details: JSON.stringify({ dataset_id, source_id, name: displayName }),
}).run();
} catch (err) {
console.error(`Failed to write audit log for exemplar ${dataset_id}:`, err);
}

return c.json(
{
dataset_id,
name: displayName,
github_repo: githubRepo.full_name,
source: "nemar-exemplar",
source_id,
},
201,
);
});

/**
* POST /admin/datasets/:id/exemplar/remint-dois
*
* Idempotent EZID re-mint of an exemplar's concept DOI (sandbox shoulder).
* EZID identifiers are deterministic and the client tolerates "already
* exists", so re-running recovers from a reset EZID sandbox or a drifted
* _target. Version DOIs re-mint through the normal publish/version-doi flow.
* 403 in production.
*/
admin.post("/datasets/:id/exemplar/remint-dois", async (c) => {
if (!isNonProductionEnv(c.env)) {
return c.json({ error: "Exemplar DOI re-mint is disabled in production" }, 403);
}
const datasetId = c.req.param("id");
const db = c.env.DB;
if (!EXEMPLAR_ID_RE.test(datasetId)) {
return c.json({ error: "Not an exemplar id (xx099900-xx099999)" }, 400);
}

const dataset = await db
.prepare(
`SELECT d.dataset_id, d.name, d.description, d.github_repo, d.concept_doi, d.is_exemplar,
u.username as owner_username, u.orcid as owner_orcid
FROM datasets d JOIN users u ON d.owner_user_id = u.id WHERE d.dataset_id = ?`,
)
.bind(datasetId)
.first<{
dataset_id: string;
name: string;
description: string | null;
github_repo: string | null;
concept_doi: string | null;
is_exemplar: number | null;
owner_username: string;
owner_orcid: string | null;
}>();
if (!dataset) {
return c.json({ error: "Dataset not found" }, 404);
}
if (dataset.is_exemplar !== 1) {
return c.json({ error: "Not an exemplar dataset" }, 400);
}

// Assemble metadata like the concept-DOI route, then re-mint (sandbox).
let bidsDescription: Record<string, unknown> | undefined;
let repoEnrichment: DataCiteEnrichment | undefined;
if (dataset.github_repo) {
const repoName = extractRepoName(dataset.github_repo);
if (repoName) {
const baseEnrichment = buildOrcidEnrichment(
undefined,
dataset.owner_username,
dataset.owner_orcid || undefined,
);
const repoMeta = await readRepoMetadata(
repoName,
await getDatasetsToken(c.env),
baseEnrichment,
dataset.name,
);
bidsDescription = repoMeta.bidsDescription;
repoEnrichment = repoMeta.enrichment;
}
}

try {
const result = await dispatchCreateConceptDoi(
{
provider: "ezid",
datasetId,
datasetName: dataset.name,
datasetDescription: dataset.description,
githubRepo: dataset.github_repo,
bidsDescription,
enrichment: repoEnrichment,
uploaderOrcid: dataset.owner_orcid || undefined,
uploaderName: dataset.owner_username,
sandbox: true,
},
{
EZID_USERNAME: c.env.EZID_USERNAME,
EZID_PASSWORD: c.env.EZID_PASSWORD,
EZID_SANDBOX_USERNAME: c.env.EZID_SANDBOX_USERNAME,
EZID_SANDBOX_PASSWORD: c.env.EZID_SANDBOX_PASSWORD,
ZENODO_API_KEY: c.env.ZENODO_API_KEY,
ZENODO_SANDBOX_API_KEY: c.env.ZENODO_SANDBOX_API_KEY,
FRONTEND_URL: c.env.FRONTEND_URL,
DATASET_LANDING_BASE_URL: c.env.DATASET_LANDING_BASE_URL,
},
);
await db
.prepare(
"UPDATE datasets SET concept_doi = ?, ezid_identifier = ?, ezid_status = ?, doi_provider = 'ezid', updated_at = datetime('now') WHERE dataset_id = ?",
)
.bind(result.doi, result.providerRecordId, result.status, datasetId)
.run();
return c.json(
{
dataset_id: datasetId,
concept_doi: result.doi,
status: result.status,
warnings: result.warnings,
},
200,
);
} catch (error) {
const msg = error instanceof Error ? error.message : String(error);
console.error(`Failed to re-mint exemplar DOIs for ${datasetId}:`, error);
return c.json({ error: `Failed to re-mint DOIs: ${msg}` }, 500);
}
});
}
Loading
Loading