Skip to content

Capture the real import failure (#1309 Half A) - #1351

Merged
neuromechanist merged 1 commit into
devfrom
feature/issue-1309-half-a-capture
Sep 10, 2026
Merged

neuromechanist merged 1 commit into
devfrom
feature/issue-1309-half-a-capture

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

The cross-repo half of #1309 that epic #1306 had to leave unlanded, plus the test that keeps it joined
to the half already shipped. Companion PR: nemarDatasets/.github#115 (byte-identical file).

Half B shipped a classifier over import_jobs.last_error. Half A is what makes anything worth
classifying arrive there. Until now the only failure message reaching the database came from the
workflow's report job:

terminal: prepare=failure copy=failure finalize=failure

Every machine-filed tracking issue reads exactly that. The classifier has been correct and idle.

What changed

Each phase job (prepare, copy, finalize) reports its own failure with the CLI's failure line.
That placement is forced, not preferred: jobs don't share a filesystem, so report cannot read the
tee'd log, and matrix job outputs overwrite each other across datasets, so a per-dataset error cannot
travel that way either. copy and finalize gain the tee prepare already had.

The roll-up stays as the backstop and cannot clobber the specific message, because
lastErrorAssignmentSql refuses to overwrite a specific error with a generic one — ADR 0051's rule
doing exactly the job it was written for.
Half A needed a guarantee Half B had already built.

The extraction, and why each choice is what it is

  • Reads the tee'd CLI output, never the Actions log. A bare 403 matches apt's Packages [403 kB]
    on every runner, and the upstream marker appears in every Actions log because the reporting step's
    own source is echoed — the likeliest origin of the misattributed upstream-403 labels on the open
    issues.
  • A six-line window from the last failure marker, not one line. ora().fail() interpolates
    subprocess stderr and git prints what the classifier needs on continuation lines
    (remote: error: GH013:, hint: … diverging commits). One line would have dropped two of the four
    real causes — I checked, rather than assumed.
  • grep -nF plus a sed window, not an awk bracket class. ubuntu-latest's awk is mawk, which
    is byte-oriented, so a multibyte character inside [...] matches its individual bytes. This one I
    wrote wrong first and caught before it shipped.
  • jq -nc for every payload. The hand-rolled JSON it replaces only carried fixed strings, so it
    worked until the message stopped being fixed; the failure mode is a silent 400 on the one callback
    carrying the diagnosis.
  • An empty extraction becomes a named fallback, never an empty error_message — the classifier
    reads a roll-up or a null as unknown on purpose, and an empty string would look like a measured
    "no reason".

Test plan

backend/test/import-capture-workflow.test.ts, 12 cases, in the required pure tier.

Verified end to end rather than by inspection: the workflow's own shell pipeline was run over logs
in the shape the CLI writes, and each extracted string fed to the real classifyImportFailure:

captured message cause
Failed to clone: fatal: Authentication failed for … auth_invalid
Failed to enable s3-PUBLIC remote: … annex-uuid file indicates it is used by a different special remote annex_uuid_conflict
Failed to push: remote: error: GH013: Repository rule violations found … branch_protection
Failed to push: hint: … diverging commits git_divergence
the no-line fallback unknown → needs-triage
the terminal: roll-up unknown → needs-triage

Those first four are the causes behind the 2026-07/09 outage, so the taxonomy is being re-derived from
the messages that will now actually be sent.

The file-invariant half of the test asserts the tee, set -o pipefail, a per-stage reporter, the jq
payloads, and the read-the-tee'd-log rule. Verified toothy by removing a tee and a jq and watching
two tests fail. That matters because this file is deployed by hand-copying, so a well-meaning edit here
is the likeliest way for capture to regress silently.

Also: the two copies had already drifted

.github/dataset-workflows/onboard-openneuro.yml and the live org-repo file were byte-identical when I
started and had drifted by exactly nemarDatasets/.github#107 (--yes on the three login lines) by the
time I looked — that PR was open and is now merged. This syncs it. The drift window between "merged
there" and "copied here" is the standing hazard; the invariant test narrows what a drift can break, but
it cannot detect one, since the live file isn't checked out here.

Half A of #1309, the cross-repo half the epic left unlanded, plus the test
that keeps it joined to Half B.

Until now the only failure message reaching the database came from the `report`
job: `terminal: prepare=failure copy=failure finalize=failure`. Every
machine-filed tracking issue said exactly that, so there was nothing to
classify and a human read Actions logs and applied labels by hand. Half B
shipped a classifier over text that was never being sent.

Each phase job now reports its own failure, carrying the CLI's failure line. It
runs in the failing job rather than in `report` because jobs do not share a
filesystem -- `report` cannot read the tee'd log -- and because matrix job
outputs overwrite each other across datasets, so a per-dataset error cannot
travel that way. `copy` and `finalize` gain the tee that `prepare` already had.

The `report` roll-up stays as the backstop and cannot clobber the specific
message: `lastErrorAssignmentSql` refuses to overwrite a specific error with a
generic one, which is ADR 0051's rule doing exactly the job it was written for.

Extraction details that are decisions, not incidentals:

- It reads the tee'd CLI output, never the Actions log. A bare `403` matches
  apt's "Packages [403 kB]" on every runner, and the upstream marker appears in
  every Actions log because the reporting step's own source is echoed -- the
  likeliest origin of the misattributed `upstream-403` labels on the existing
  issues.
- A WINDOW of six lines from the last failure marker, not one line.
  `ora().fail()` interpolates subprocess stderr and git prints what the
  classifier needs on continuation lines: `remote: error: GH013:`,
  `hint: ... diverging commits`. One line would have dropped two of the four
  real causes.
- `grep -nF` plus a `sed` window rather than an awk bracket class:
  ubuntu-latest's `awk` is mawk, which is byte-oriented, so a multibyte
  character inside [...] matches its individual bytes.
- `jq -nc` builds every payload. The hand-rolled JSON this replaces only ever
  carried fixed strings; an error line with a quote or a backslash would have
  produced a malformed body and a silent 400 on the one callback carrying the
  diagnosis.
- An empty extraction becomes a named fallback, never an empty `error_message`:
  the classifier reads a roll-up or a null as `unknown` on purpose, and an empty
  string would look like a measured "no reason".

Verified end to end by running the workflow's own pipeline over logs in the
shape the CLI writes, then feeding each result to the real classifier: the
expired PAT, the annex-uuid collision, the branch-protection ruleset and the
git divergence -- the four causes behind the outage -- classify correctly, and
both the fallback and the roll-up stay `unknown`/`needs-triage`.

`backend/test/import-capture-workflow.test.ts` pins both halves. The wording of
a message is the only thing joining two repositories, so neither repo's own
tests could catch them drifting; it also asserts the tee, the per-stage
reporter, the jq payloads and the read-the-log-not-the-Actions-log rule, since
this file is deployed by hand-copying and an edit here is the likeliest
regression. Verified toothy by removing a tee and a jq.

Also syncs this copy with `nemarDatasets/.github` PR #107 (`--yes` on the three
login lines), which was merged there first: the copies were byte-identical
before it and had already drifted by exactly that change.

This branch was previously deployed

1 inactive deployment
npm — 37d8ed21 Deployed Sep 10, 2026 by neuromechanist via publish #2466
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant