Repository navigation
Description
Contract
Both of these must work, unauthenticated:
aws s3 ls --no-sign-request s3://nemar/
aws s3 ls --no-sign-request s3://nmdatasets/returning the top-level prefixes (dataset IDs) and top-level objects only. Listing INSIDE any prefix stays denied — aws s3 ls --no-sign-request s3://nemar/on002718/ and any --recursive listing must keep failing. Object-level access control is untouched: this changes nothing about who can GET what; it only eases discovery at the bucket root.
Today both return AccessDenied (verified 2026-08-10). The Zarr index/catalog work (#1059, #1061, #1062) assumes "anonymous ListBucket is denied" as a blanket fact; after this change the accurate statement is "denied except the bare top level", which is friendlier to agents/HPC users who start from a bucket URI, while index.json / zarr-catalog.json remain the entry points for anything deeper.
How: ListBucket conditions encode "top level only" exactly
s3:prefix and s3:delimiter are policy condition keys on ListBucket. aws s3 ls s3://nemar/ issues ListObjectsV2 with empty prefix and delimiter=/; a subfolder ls sends prefix=<id>/; --recursive sends no delimiter. So this Allow matches exactly the contract and nothing else:
{
"Sid": "AnonymousTopLevelListOnly",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::nemar",
"Condition": {
"StringEquals": { "s3:prefix": "", "s3:delimiter": "/" }
}
}Same statement with arn:aws:s3:::nmdatasets on that bucket.
Caveats to check while applying
- Read the current bucket policy first (
aws s3api get-bucket-policy --bucket nemar). If an existing explicitDenystatement coverss3:ListBucketfor anonymous principals (the private-datasetNotResourcedeny-list pattern from STS per-session ListBucket over-scoping (cross-dataset key enumeration) #941 may), the new Allow is overridden; the Deny then needs a matchingStringNotEqualsprefix/delimiter carve-out, which is a more delicate edit. Save the old policy JSON before any change. - Top-level listing reveals every prefix, including private/unpublished dataset IDs and infrastructure prefixes (
staging/, catalog objects). CommonPrefixes cannot be filtered per-prefix by policy. Accepted trade-off per the contract (IDs are sequential and low-sensitivity; object access is unchanged) — recording it here so it is a decision, not an oversight. - Staging first: apply to
s3://nemar-dev, verify all four behaviors (root ls works unsigned; subfolder ls denied; recursive denied; object GET behavior unchanged for a public and a private dataset), then prodnemar, thennmdatasets. nmdatasets(us-east-2) responds 403 today; confirm which account owns it and what it holds before applying.
Acceptance
-
aws s3 ls --no-sign-request s3://nemar/lists top-level prefixes -
aws s3 ls --no-sign-request s3://nemar/<id>/still AccessDenied -
aws s3 ls --no-sign-request s3://nemar/ --recursivestill AccessDenied - Public-dataset object GET (unsigned) and private-dataset GET (denied) behavior unchanged
- Same three list behaviors on
s3://nmdatasets/ -
nemar-devcarries the same policy (staging parity) - Zarr index format v3: declare the data plane, the source tree, and a real schema #1059/Make zarr.nemar.org the contract: proxy the index, redirect the bytes #1061/Expose Zarr state in the public catalog, and publish a top-level zarr catalog #1062 wording updated where they claim ListBucket is fully denied
Activity
neuromechanist commentedon Aug 11, 2026
Applied and verified on all three buckets (2026-08-10, backups of the prior policies saved before each change):
AnonymousTopLevelListOnlystatement (s3:ListBucketallowed forPrincipal: *only whens3:prefix = ""ands3:delimiter = "/") tonemar-dev, thennemar, thennmdatasets. No existing statements were modified; there were no explicit Denies to carve out.aws s3 ls --no-sign-request s3://nemar/lists 786 top-level entries; subfolder ls and--recursiveremain AccessDenied.nmdatasets: top-level ls works; the pre-existing NeurIPS25/emg subfolder listing grants still work; other subfolders (NeurIPS26) denied.nemar-devstaging parity confirmed first with the same four checks.Noting on #1059/#1061/#1062 that "anonymous ListBucket is denied" is now "denied except the bare top level".
neuromechanist commentedon Aug 11, 2026
Pre-change policy backups, for rollback reference:
nemar-dev:{ "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadExceptPrivate", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "NotResource": "arn:aws:s3:::nemar-dev/staging/*" } ] }nemar:{ "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadExceptPrivate", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "NotResource": [ "arn:aws:s3:::nemar/staging/*", "arn:aws:s3:::nemar/nm000116/*", "arn:aws:s3:::nemar/nm000117/*", "arn:aws:s3:::nemar/nm000164/*", "arn:aws:s3:::nemar/nm000226/*", "arn:aws:s3:::nemar/nm000278/*", "arn:aws:s3:::nemar/nm000279/*", "arn:aws:s3:::nemar/on003645/*", "arn:aws:s3:::nemar/on004148/*", "arn:aws:s3:::nemar/on004212/*", "arn:aws:s3:::nemar/on005279/*", "arn:aws:s3:::nemar/on005516/*", "arn:aws:s3:::nemar/on007987/*", "arn:aws:s3:::nemar/on008014/*", "arn:aws:s3:::nemar/on008017/*", "arn:aws:s3:::nemar/on008065/*", "arn:aws:s3:::nemar/on008092/*", "arn:aws:s3:::nemar/on008099/*", "arn:aws:s3:::nemar/on008115/*", "arn:aws:s3:::nemar/xx000002/*" ] } ] }nmdatasets:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::nmdatasets/NeurIPS25/*" }, { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::nmdatasets/emg/*" }, { "Effect": "Allow", "Principal": "*", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::nmdatasets", "Condition": { "StringLike": { "s3:prefix": [ "NeurIPS25/*", "NeurIPS25", "emg", "emg/*" ] } } } ] }