Repository navigation
Capture the real import failure (#1309 Half A) - #1351
Merged
Merged
Conversation
Half A of #1309, the cross-repo half the epic left unlanded, plus the test that keeps it joined to Half B. Until now the only failure message reaching the database came from the `report` job: `terminal: prepare=failure copy=failure finalize=failure`. Every machine-filed tracking issue said exactly that, so there was nothing to classify and a human read Actions logs and applied labels by hand. Half B shipped a classifier over text that was never being sent. Each phase job now reports its own failure, carrying the CLI's failure line. It runs in the failing job rather than in `report` because jobs do not share a filesystem -- `report` cannot read the tee'd log -- and because matrix job outputs overwrite each other across datasets, so a per-dataset error cannot travel that way. `copy` and `finalize` gain the tee that `prepare` already had. The `report` roll-up stays as the backstop and cannot clobber the specific message: `lastErrorAssignmentSql` refuses to overwrite a specific error with a generic one, which is ADR 0051's rule doing exactly the job it was written for. Extraction details that are decisions, not incidentals: - It reads the tee'd CLI output, never the Actions log. A bare `403` matches apt's "Packages [403 kB]" on every runner, and the upstream marker appears in every Actions log because the reporting step's own source is echoed -- the likeliest origin of the misattributed `upstream-403` labels on the existing issues. - A WINDOW of six lines from the last failure marker, not one line. `ora().fail()` interpolates subprocess stderr and git prints what the classifier needs on continuation lines: `remote: error: GH013:`, `hint: ... diverging commits`. One line would have dropped two of the four real causes. - `grep -nF` plus a `sed` window rather than an awk bracket class: ubuntu-latest's `awk` is mawk, which is byte-oriented, so a multibyte character inside [...] matches its individual bytes. - `jq -nc` builds every payload. The hand-rolled JSON this replaces only ever carried fixed strings; an error line with a quote or a backslash would have produced a malformed body and a silent 400 on the one callback carrying the diagnosis. - An empty extraction becomes a named fallback, never an empty `error_message`: the classifier reads a roll-up or a null as `unknown` on purpose, and an empty string would look like a measured "no reason". Verified end to end by running the workflow's own pipeline over logs in the shape the CLI writes, then feeding each result to the real classifier: the expired PAT, the annex-uuid collision, the branch-protection ruleset and the git divergence -- the four causes behind the outage -- classify correctly, and both the fallback and the roll-up stay `unknown`/`needs-triage`. `backend/test/import-capture-workflow.test.ts` pins both halves. The wording of a message is the only thing joining two repositories, so neither repo's own tests could catch them drifting; it also asserts the tee, the per-stage reporter, the jq payloads and the read-the-log-not-the-Actions-log rule, since this file is deployed by hand-copying and an edit here is the likeliest regression. Verified toothy by removing a tee and a jq. Also syncs this copy with `nemarDatasets/.github` PR #107 (`--yes` on the three login lines), which was merged there first: the copies were byte-identical before it and had already drifted by exactly that change.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The cross-repo half of #1309 that epic #1306 had to leave unlanded, plus the test that keeps it joined
to the half already shipped. Companion PR: nemarDatasets/.github#115 (byte-identical file).
Half B shipped a classifier over
import_jobs.last_error. Half A is what makes anything worthclassifying arrive there. Until now the only failure message reaching the database came from the
workflow's
reportjob:Every machine-filed tracking issue reads exactly that. The classifier has been correct and idle.
What changed
Each phase job (
prepare,copy,finalize) reports its own failure with the CLI's failure line.That placement is forced, not preferred: jobs don't share a filesystem, so
reportcannot read thetee'd log, and matrix job outputs overwrite each other across datasets, so a per-dataset error cannot
travel that way either.
copyandfinalizegain the teepreparealready had.The roll-up stays as the backstop and cannot clobber the specific message, because
lastErrorAssignmentSqlrefuses to overwrite a specific error with a generic one — ADR 0051's ruledoing exactly the job it was written for. Half A needed a guarantee Half B had already built.
The extraction, and why each choice is what it is
403matches apt'sPackages [403 kB]on every runner, and the upstream marker appears in every Actions log because the reporting step's
own source is echoed — the likeliest origin of the misattributed
upstream-403labels on the openissues.
ora().fail()interpolatessubprocess stderr and git prints what the classifier needs on continuation lines
(
remote: error: GH013:,hint: … diverging commits). One line would have dropped two of the fourreal causes — I checked, rather than assumed.
grep -nFplus asedwindow, not an awk bracket class.ubuntu-latest'sawkis mawk, whichis byte-oriented, so a multibyte character inside
[...]matches its individual bytes. This one Iwrote wrong first and caught before it shipped.
jq -ncfor every payload. The hand-rolled JSON it replaces only carried fixed strings, so itworked until the message stopped being fixed; the failure mode is a silent 400 on the one callback
carrying the diagnosis.
error_message— the classifierreads a roll-up or a null as
unknownon purpose, and an empty string would look like a measured"no reason".
Test plan
backend/test/import-capture-workflow.test.ts, 12 cases, in the required pure tier.Verified end to end rather than by inspection: the workflow's own shell pipeline was run over logs
in the shape the CLI writes, and each extracted string fed to the real
classifyImportFailure:Failed to clone: fatal: Authentication failed for …auth_invalidFailed to enable s3-PUBLIC remote: … annex-uuid file indicates it is used by a different special remoteannex_uuid_conflictFailed to push: remote: error: GH013: Repository rule violations found …branch_protectionFailed to push: hint: … diverging commitsgit_divergenceunknown→needs-triageterminal:roll-upunknown→needs-triageThose first four are the causes behind the 2026-07/09 outage, so the taxonomy is being re-derived from
the messages that will now actually be sent.
The file-invariant half of the test asserts the tee,
set -o pipefail, a per-stage reporter, the jqpayloads, and the read-the-tee'd-log rule. Verified toothy by removing a tee and a
jqand watchingtwo tests fail. That matters because this file is deployed by hand-copying, so a well-meaning edit here
is the likeliest way for capture to regress silently.
Also: the two copies had already drifted
.github/dataset-workflows/onboard-openneuro.ymland the live org-repo file were byte-identical when Istarted and had drifted by exactly nemarDatasets/.github#107 (
--yeson the three login lines) by thetime I looked — that PR was open and is now merged. This syncs it. The drift window between "merged
there" and "copied here" is the standing hazard; the invariant test narrows what a drift can break, but
it cannot detect one, since the live file isn't checked out here.