Repository navigation
Regression: 403 Auth Failure for S3-hosted (Cloudfront) Private Repos #3303
Description
-
I am on the latest Poetry version.
-
I have searched the issues of this repo and believe that this is not a duplicate. (There are similar tickets, notably Auth failure when installing from multiple private repos #3291 but they aren't specifically targeting a repo hosted on S3/Cloudfront)
-
If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption). -
OS version and name:
python:3.7-alpine(Docker) -
Poetry version: 1.1.4
-
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/dvf/a117ebdb0358cb388c5145cfdd39b46e
Issue
- We noticed a regression from Poetry 1.0.3 to 1.1.4.
- We're using a private PyPI repository hosted on S3 protected by Cloudfront using Basic Auth (we use s3pypi)
When we tried to install a new private dependency, or remove a non-private dependency (with common sub dependencies with a private dependency) we saw this error:
➜ project git:(master) poetry remove django-structlog
Updating dependencies
Resolving dependencies... (0.2s)
RepositoryError
403 Client Error: Forbidden for url: https://XXXXXXXXXXXXXXXX/pyjwt/
at ~/.poetry/lib/poetry/repositories/legacy_repository.py:393 in _get
389│ if response.status_code == 404:
390│ return
391│ response.raise_for_status()
392│ except requests.HTTPError as e:
→ 393│ raise RepositoryError(e)
394│
395│ if response.status_code in (401, 403):
396│ self._log(
397│ "Authorization error accessing {url}".format(url=url), level="warn"
Hypothesis of why this is happening
By default, Cloudfront returns a 403 for non-existing files. This is a good security practice as it prevents leakage of information to unauthenticated parties. However, (from the above code) Poetry only falls back to public PyPI if it encounters a 404 when requesting a package.
Quick fix
The quick fix for this is to have your private repo (in our case Cloudfront) return a default error response code of 404.
Here are docs on how to do that: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages-response-code.html
Recommendations
Poetry shouldn't fall back to the public repo on a 401/403/404 without first checking if it can successfully auth to the repo. And failing that, it should throw an appropriate error.
Thanks to @proxyroot and @vagelim for helping debug and fix.
Activity
sschrijver commentedon Nov 2, 2020
This is also an issue (but maybe slightly different, I receive a "Forbidden" error) with a private PyPI registry that stores packages on S3 (without cloudfront). It returns a s3 URL (with temp auth token) with the message that the request is forbidden, whilst copying the URL and pasting it in a new browser initiates the correct download.
Downgrading to
1.0.10solved this error, this error happens from version1.1.xabn commentedon Nov 2, 2020
This block
poetry/poetry/repositories/legacy_repository.py
Lines 392 to 396 in 6ddd58f
Should move to before
poetry/poetry/repositories/legacy_repository.py
Line 388 in 6ddd58f
abn commentedon Nov 2, 2020
@dvf thank you for the great issue report! 🎉
Regarding the fallback behaviour, I am reluctant to suggest we change at that at the moment since I think there are cases where authentication/authorsation is not available and a public package can still be added. Happy to discuss that and change behaviour in another issue. For this one, lets treat it as a bug and resolve it as such.
abn commentedon Nov 2, 2020
@sschrijver think this was reported elsewhere, the issue here is that we use basic authentication as configured along with those requests. The existence of an "Authorization" header the "authorised" url ends up failing on the server side. IIRC, this was an issue with PyPI cloud since it deviates from the defined/expected behaviour for an index. See #3041 (comment) for that bit. I have not looked into the comments after that one yet. Additionally, note that there is the issue that using "authorised" urls will have other side-effects when the token used expires as these are used in the lock file.
cereblanco commentedon Nov 3, 2020
hello! if no one started working on this yet, I can create the fix. 💭
jeremysprofile commentedon Nov 10, 2020
This also affects Sonatype Nexus PyPI repositories when using poetry 1.1.4, and downgrading to 1.0.10 fixes the issue there as well.
repositories: fix 401, 403 handling
github-actions commentedon Mar 2, 2024
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.