Skip to content

Manifest generator drops small regular git files at dataset root (README, CHANGES, dataset_description.json) #509

Closed

Description

@neuromechanist

Symptom

For on* datasets (and any dataset whose root BIDS metadata is small), data.nemar.org/<id>/<v>/manifest.json returns ZERO root files even though those files exist in the git tree at the tagged commit.

Example: `on005262` v1.0.0.

GitHub tree at refs/tags/v1.0.0 (verified via gh api repos/nemarDatasets/on005262/git/trees/c4ddc459...):

blob  100644          427  .gitattributes
blob  100644           66  CHANGES
blob  100644          432  README.md
blob  100644          461  dataset_description.json

data.nemar.org/on005262/v1.0.0/manifest.json contains 559 entries — all under sub-*/ or .nemar/. None of the root files appear.

For comparison, `nm000104` v2.0.0 has 5 root files in its manifest (README.md is 9763 bytes there).

Root cause

backend/src/services/manifest.ts lines 85-150 (generateManifest).

Walkthrough for README.md (432 bytes):

  1. Line 100-107 — file lands in pointerCandidates because 20 < 432 < 500 and path doesn't start with .git.
  2. Line 115-133 — pointer-resolution loop fetches the blob, calls parseAnnexPointer(content). Returns null because it's a real README, not an annex pointer. Nothing is added to files.
  3. Line 136-141 — regularFiles filter excludes anything in pointerCandidates. So README.md is excluded from this loop too.

Result: README.md is silently dropped from the manifest. Same path for CHANGES (66B), dataset_description.json (461B).

nm000104 doesn't hit this because its root files are larger:

  • README.md = 9763 bytes (> 500, skips the pointer-candidate filter, lands cleanly in regularFiles)
  • dataset_description.json = 1353 bytes (same)
  • participants.tsv = 2291 bytes (same)

So the bug only surfaces for datasets with small BIDS docs at root — common for OpenNeuro mirrors.

Secondary bug

!entry.path.startsWith(".git") on line 138 also catches .gitattributes because of the substring match. The intent was clearly .git/ (with trailing slash) since .github/ is excluded separately. As written, .gitattributes is excluded too.

Fix sketch

After the pointer-resolution loop, retain candidates whose parseAnnexPointer returned null and add them to regularFiles. Two-line change:

// Inside the inner loop, track the misses:
const nonAnnexCandidates: Array<typeof entry> = [];
for (const { entry, key } of results) {
  if (key) {
    files[entry.path] = { key, size: ..., checksum: ... };
  } else {
    nonAnnexCandidates.push(entry);  // <-- new
  }
}

// Then in regularFiles filter, also include the misses:
const regularFiles = [
  ...nonAnnexCandidates,
  ...entries.filter(
    (entry) =>
      !entry.path.startsWith(".git/") &&        // <-- trailing slash
      entry.path !== ".gitattributes" &&        // explicit allow (or just keep this in)
      !entry.path.startsWith(".github/") &&
      !pointerCandidates.some((p) => p.path === entry.path),
  ),
];

Or cleaner: do a single pass over entries, decide per-entry whether it's annex or git-tree based on the content check result, and skip nothing.

Impact

Affects every on* dataset's detail page on the new nemar.org (nemarOrg/website Phase 2 just shipped). The website now falls back to fetching the README from the dataset's GitHub repo when it isn't in the manifest, but that loses the per-version snapshot guarantee — fixing this server-side restores it.

Also affects rclone / aria2c users syncing via data.nemar.org who won't see README/CHANGES/dataset_description.json in the file list.

Verification

After the fix, curl -s https://data.nemar.org/on005262/v1.0.0/manifest.json | jq '[.[] | select(.path | contains("/") | not)]' should return 4 entries (.gitattributes, CHANGES, README.md, dataset_description.json).

Related: #449 (the data.nemar.org epic), nemarOrg/website#1 (detail page).

Activity

  1. added 2 commits that reference this issue on May 15, 2026
    a982a76
    99f21e2
  2. neuromechanist commented on May 15, 2026

    @neuromechanist
    ContributorAuthor

    Fixed in #510 (squash-merged to dev as 99f21e2). Will reach main on the next dev→main release. Verified end-to-end:

    • Small regular root files (README.md, CHANGES, dataset_description.json, .gitattributes) now flow through the regular-files path even when they fall in the 20-500B annex-pointer candidate band.
    • .gitattributes no longer accidentally excluded by the .git substring match.
    • 4-case regression test in backend/test/manifest-small-root-files.test.ts (all green in CI).

    Step toward #448 (OpenNeuro mirror DOI mapping), not a complete fix for that epic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions