jclouds s3 endpoint seemingly not supporting <bucketname>.s3.provider.com endpoints #759

Activity

1. gaul commented on Jan 11, 2025

   

   gaul

   on Jan 11, 2025

   Owner

   More actions

   What does your s3proxy.conf look like? Do you use the s3 provider (correct) or the aws-s3 provider (incorrect)?
2. Jayd603 commented on Jan 11, 2025

   

   Jayd603

   on Jan 11, 2025

   Last edited by gaul

   Author

   More actions

   the s3 provider (not AWS). pretty basic config using the docker container:

   docker run -d --restart=unless-stopped --net=host --name=s3-jamesd -e S3PROXY_AUTHORIZATION=aws-v2-or-v4 -e S3PROXY_IDENTITY=jamesd -e S3PROXY_CREDENTIAL='xxx' -e S3PROXY_ENDPOINT="http://127.0.0.1:32335" -e S3PROXY_CORS_ALLOW_ALL="true" -e S3PROXY_IGNORE_UNKNOWN_HEADERS="true" -e JCLOUDS_PROVIDER="s3" -e S3PROXY_V4_MAX_NON_CHUNKED_REQ_SIZE="310000000" -e JCLOUDS_ENDPOINT="https://bucketname.nyc3.digitaloceanspaces.com" -e JCLOUDS_IDENTITY="xxx" -e JCLOUDS_CREDENTIAL="xxx" andrewgaul/s3proxy:latest
   
   when using the bucketname in the endpoint:
   jd@jd-desktop-1:~/Apps$ s3cmd ls s3://
   ERROR: S3 error: 403 (AccessDenied): Forbidden
   jd@jd-desktop-1:~/Apps$ s3cmd ls s3://dyni-jamesd/
   ERROR: Access to bucket 'dyni-jamesd' was denied
   ERROR: S3 error: 403 (AccessDenied): Forbidden
   
   when not using it:
   jd@jd-desktop-1:~/Apps$ s3cmd ls s3://
   ERROR: S3 error: 403 (AccessDenied): Forbidden
   jd@jd-desktop-1:~/Apps$ s3cmd ls s3://dyni-jamesd/
                             DIR  s3://dyni-jamesd/Backups/
   jd@jd-desktop-1:~/Apps$
   
3. gaul commented on Jan 12, 2025

   

   gaul

   on Jan 12, 2025

   Last edited by gaul

   Owner

   More actions

   Could you try setting jclouds.s3.virtual-host-buckets=true in s3proxy.conf?
4. Jayd603 commented on Jan 12, 2025

   

   Jayd603

   on Jan 12, 2025

   Author

   More actions

   I set that and it is listed in the process list in the docker container -Djclouds.s3.virtual-host-buckets=true and I am seeing the same behavior as before.
5. gaul commented on Aug 2, 2026

   

   gaul

   on Aug 2, 2026

   Last edited by gaul

   Owner

   More actions

   Sorry for the long delay here. I traced this properly and there are two independent things going on, neither of which is fixed by the endpoint.

   The endpoint must name the service, never a bucket. S3Proxy addresses the bucket itself, so whatever endpoint it is given gets the bucket appended. Running master against a backend that logs what it receives, a client request for s3://dyni-jamesd/ arrives as:

   GET /dyni-jamesd?list-type=2&max-keys=1000
   

   With JCLOUDS_ENDPOINT=https://dyni-jamesd.nyc3.digitaloceanspaces.com, DigitalOcean therefore sees https://dyni-jamesd.nyc3.digitaloceanspaces.com/dyni-jamesd?list-type=2 — the bucket named twice. That is structural rather than a missing setting, so a bucket-scoped endpoint cannot work.

   My suggestion of jclouds.s3.virtual-host-buckets=true was wrong and could not have helped: it moves the bucket into the hostname, which would have produced dyni-jamesd.dyni-jamesd.nyc3.digitaloceanspaces.com. That property no longer exists in the upcoming 4.0.0, which replaces the jclouds providers with the AWS SDK and always uses path-style addressing for non-AWS endpoints.

   The 403 from s3cmd ls s3:// is your credential, not the endpoint. Your output shows it failing with both endpoints, while s3cmd ls s3://dyni-jamesd/ succeeds with the regional one — so the bucket-scoped key is allowed to list that bucket but not to list the account's buckets. S3Proxy passes the backend's error through untouched; I confirmed that a backend answering 403 to GET / produces exactly the 403 you saw, while the bucket listing returns 200 in the same run.

   So the configuration you already found is the correct one: keep JCLOUDS_ENDPOINT=https://nyc3.digitaloceanspaces.com and address the bucket by name.

   On being "in the correct root directory": S3 has no way to express this. Objects cannot live at the root of an endpoint, and ls s3:// is ListBuckets by definition, so the closest equivalent is having that call return just the bucket you can use.