Skip to content

run-bids-validation: annex-pointer filter misses unlocked pointer files (GNU find -size -1k matches only empty files) #128

Open

Description

@bruAristimunha

Found while publishing iEEG datasets with annexed NIfTI files.

What happens

run-bids-validation fails with NIFTI_HEADER_UNREADABLE (and INVALID_GZIP for annexed .tsv.gz) on files that are git-annex pointers, although the workflow is meant to skip those. Every summary ends with "(skipped 0 annex-pointer error(s))":

  • nm000372: 1 error, sub-01/anat/sub-01_T1w.nii.gz, run 37598060353
  • nm000378: 36 errors, one per sub-*_T1w.nii, run 37595917824
  • nm000254 (already published): 1259 errors (432 NIFTI_HEADER_UNREADABLE, 827 INVALID_GZIP), run 36644096331

Cause

Step "Collect git-annex pointer files" in .github/workflows/run-bids-validation.yml (line 165 on main):

find . -type f -not -path './.git/*' -size -1k -exec grep -l '^/annex/objects/' {} +

GNU find rounds a file's size up to whole units before comparing, so -size -1k means "0 KiB after rounding up", i.e. only empty files. The pointers on main are unlocked pointer files (mode 100644, ~106 bytes; nemar-cli pushes the adjusted/unlocked branch), so they are never collected, and the symlink branch of the step does not match them either.

Reproduction (GNU findutils 4.7.0):

printf '/annex/objects/SHA256E-s1--x.nii.gz\n' > p
find . -type f -size -1k      # no output
find . -type f -size -1024c   # ./p

Proposed fix

Replace -size -1k with -size -1024c (or -size -2k). The existing location filter should then skip these errors as intended.

Changing the datasets instead does not look right: nemar-cli's annex policy annexes NIfTI on purpose and strips inherited annex.largefiles attributes.

It does not appear to block publication (nm000254 is published), but it puts a red BIDS check on the affected datasets.

Related: nemarOrg/nemar-cli#267 (where the filter was introduced).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions