Skip to content

S3 contract: anonymous top-level ls on s3://nemar and s3://nmdatasets (bucket root only, never subfolders) #1071

Closed

Description

@neuromechanist

Contract

Both of these must work, unauthenticated:

aws s3 ls --no-sign-request s3://nemar/
aws s3 ls --no-sign-request s3://nmdatasets/

returning the top-level prefixes (dataset IDs) and top-level objects only. Listing INSIDE any prefix stays denied — aws s3 ls --no-sign-request s3://nemar/on002718/ and any --recursive listing must keep failing. Object-level access control is untouched: this changes nothing about who can GET what; it only eases discovery at the bucket root.

Today both return AccessDenied (verified 2026-08-10). The Zarr index/catalog work (#1059, #1061, #1062) assumes "anonymous ListBucket is denied" as a blanket fact; after this change the accurate statement is "denied except the bare top level", which is friendlier to agents/HPC users who start from a bucket URI, while index.json / zarr-catalog.json remain the entry points for anything deeper.

How: ListBucket conditions encode "top level only" exactly

s3:prefix and s3:delimiter are policy condition keys on ListBucket. aws s3 ls s3://nemar/ issues ListObjectsV2 with empty prefix and delimiter=/; a subfolder ls sends prefix=<id>/; --recursive sends no delimiter. So this Allow matches exactly the contract and nothing else:

{
  "Sid": "AnonymousTopLevelListOnly",
  "Effect": "Allow",
  "Principal": "*",
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::nemar",
  "Condition": {
    "StringEquals": { "s3:prefix": "", "s3:delimiter": "/" }
  }
}

Same statement with arn:aws:s3:::nmdatasets on that bucket.

Caveats to check while applying

  1. Read the current bucket policy first (aws s3api get-bucket-policy --bucket nemar). If an existing explicit Deny statement covers s3:ListBucket for anonymous principals (the private-dataset NotResource deny-list pattern from STS per-session ListBucket over-scoping (cross-dataset key enumeration) #941 may), the new Allow is overridden; the Deny then needs a matching StringNotEquals prefix/delimiter carve-out, which is a more delicate edit. Save the old policy JSON before any change.
  2. Top-level listing reveals every prefix, including private/unpublished dataset IDs and infrastructure prefixes (staging/, catalog objects). CommonPrefixes cannot be filtered per-prefix by policy. Accepted trade-off per the contract (IDs are sequential and low-sensitivity; object access is unchanged) — recording it here so it is a decision, not an oversight.
  3. Staging first: apply to s3://nemar-dev, verify all four behaviors (root ls works unsigned; subfolder ls denied; recursive denied; object GET behavior unchanged for a public and a private dataset), then prod nemar, then nmdatasets.
  4. nmdatasets (us-east-2) responds 403 today; confirm which account owns it and what it holds before applying.

Acceptance

Activity

  1. neuromechanist commented on Aug 11, 2026

    @neuromechanist
    ContributorAuthor

    Applied and verified on all three buckets (2026-08-10, backups of the prior policies saved before each change):

    • Added the AnonymousTopLevelListOnly statement (s3:ListBucket allowed for Principal: * only when s3:prefix = "" and s3:delimiter = "/") to nemar-dev, then nemar, then nmdatasets. No existing statements were modified; there were no explicit Denies to carve out.
    • aws s3 ls --no-sign-request s3://nemar/ lists 786 top-level entries; subfolder ls and --recursive remain AccessDenied.
    • Object access unchanged: anonymous GET of a public object returns 200, private-dataset GET (nm000116) returns 403, staging GET returns 403. (A GET of a nonexistent key reads as 403 for anonymous callers, which is expected S3 behavior, not a regression.)
    • nmdatasets: top-level ls works; the pre-existing NeurIPS25/emg subfolder listing grants still work; other subfolders (NeurIPS26) denied.
    • nemar-dev staging parity confirmed first with the same four checks.

    Noting on #1059/#1061/#1062 that "anonymous ListBucket is denied" is now "denied except the bare top level".

  2. neuromechanist commented on Aug 11, 2026

    @neuromechanist
    ContributorAuthor

    Pre-change policy backups, for rollback reference:

    nemar-dev:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "PublicReadExceptPrivate",
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:GetObject",
                "NotResource": "arn:aws:s3:::nemar-dev/staging/*"
            }
        ]
    }

    nemar:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "PublicReadExceptPrivate",
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:GetObject",
                "NotResource": [
                    "arn:aws:s3:::nemar/staging/*",
                    "arn:aws:s3:::nemar/nm000116/*",
                    "arn:aws:s3:::nemar/nm000117/*",
                    "arn:aws:s3:::nemar/nm000164/*",
                    "arn:aws:s3:::nemar/nm000226/*",
                    "arn:aws:s3:::nemar/nm000278/*",
                    "arn:aws:s3:::nemar/nm000279/*",
                    "arn:aws:s3:::nemar/on003645/*",
                    "arn:aws:s3:::nemar/on004148/*",
                    "arn:aws:s3:::nemar/on004212/*",
                    "arn:aws:s3:::nemar/on005279/*",
                    "arn:aws:s3:::nemar/on005516/*",
                    "arn:aws:s3:::nemar/on007987/*",
                    "arn:aws:s3:::nemar/on008014/*",
                    "arn:aws:s3:::nemar/on008017/*",
                    "arn:aws:s3:::nemar/on008065/*",
                    "arn:aws:s3:::nemar/on008092/*",
                    "arn:aws:s3:::nemar/on008099/*",
                    "arn:aws:s3:::nemar/on008115/*",
                    "arn:aws:s3:::nemar/xx000002/*"
                ]
            }
        ]
    }

    nmdatasets:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::nmdatasets/NeurIPS25/*"
            },
            {
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::nmdatasets/emg/*"
            },
            {
                "Effect": "Allow",
                "Principal": "*",
                "Action": "s3:ListBucket",
                "Resource": "arn:aws:s3:::nmdatasets",
                "Condition": {
                    "StringLike": {
                        "s3:prefix": [
                            "NeurIPS25/*",
                            "NeurIPS25",
                            "emg",
                            "emg/*"
                        ]
                    }
                }
            }
        ]
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions