Missing credentials when attempting to install dependency from private Gitlab project #5955

Activity

1. cbuffett

   added

   kind/bugSomething isn't working as expectedSomething isn't working as expected

   status/triageThis issue needs to be triagedThis issue needs to be triaged

   on Jul 5, 2022
2. jaklan commented on Jul 25, 2022

   

   jaklan

   on Jul 25, 2022

   Last edited by jaklan

   More actions

   Bump this one, #5567 doesn't really solve the issue - as mentioned above, it works only when using poetry, which makes a package unusable if you want to specify it as a pip dependency.
3. neersighted commented on Jul 25, 2022

   

   neersighted

   on Jul 25, 2022

   Member

   More actions

   My post on the linked PR (please don't start discussion there where it is hard to find and tangential):

   Generally I would avoid bundling private Git deps, and if you have to SSH keys would be the preferred way to solve this. I'm really not sure that a 'controlled' option to leak credentials into the lockfile (and thus built artifacts) is a good idea as it's incredibly likely to bite more users than it helps.

   Also, the 'correct' way to solve this would be uploading your packages to a private registry -- private Git deps is not a great option for this workflow. I do think that just because something is possible as a workflow, doesn't mean it's necessarily a good one. Maybe beefing up our docs to suggest against using Git deps for private packages would be helpful.

   Regardless, let's take discussion back to #5955 as this is not a great forum/place for a feature request.
4. jaklan commented on Jul 25, 2022

   

   jaklan

   on Jul 25, 2022

   Last edited by jaklan

   More actions

   @neersighted

   • SSH is of course a safer option, but if I don't miss any options - it would be pretty problematic to apply here, because in GitLab you can either:

     • add SSH keys of CI runners, servers to your personal account - not a real solution for a project setup
     • add them as deploy keys for a given project - you would need to add your keys in all dependencies' repos, which means you have to each time collaborate with Maintainers of all the dependencies, quite unmaintainable

   • that option is actually widely available anyway - it works with pip, it works with git, it even works with poetry when you add the entry manually. Ofc it's not the safest solution, there is even a 5-year issue in pip repo related to that:
     Please add support for credentials via environment variables pypa/pip#4789
     but for some reason it's still not resolved... I wouldn't say blocking people from specifying the dependency together with the token would help in any way - it just generates an additional problem without any obvious alternative, so we could end up with people adding the problematic dependencies with pip install after running poetry install - which makes the whole setup even worse.

   • I fully agree a private registry is the best solution here, but as I mentioned - usually you don't have a control of it for such dependencies, you can of course try to reach to proper people and ask for it, but it doesn't guarantee the success (especially within the expected timeline).

   In general, imho that comment from @abn: #2062 (comment) is pretty reasonable here - so don't recommend that, maybe even provide some warning, but anyway - simply don't block people from doing that. It was also nicely summed up by @xinbinhuang here: #2062 (comment).
5. neersighted commented on Jul 25, 2022

   

   neersighted

   on Jul 25, 2022

   Last edited by neersighted

   Member

   More actions

   The direction we're going is pretty firmly against leaking credentials in the lockfile/distfiles: #3995 is likely going in to 1.3.

   I'm unclear on what the workflow is, but it sounds like you may be part of a large organization where you can generate deploy tokens or use user credentials to access repos, but you cannot have them add SSH deploy keys or upload to a registry?

   In that case I'd suggest use of gitcredentials, which pip should support during install.

   I am still not sure of any use case where credentials in the lockfile/distfiles are the only solution, but please let me know if there's a situation where it's not usable. I firmly believe secrets should be kept in your CI server/deployment tooling and not in your repo/artifacts where they can be leaked easily.
6. jaklan commented on Jul 25, 2022

   

   jaklan

   on Jul 25, 2022

   Last edited by jaklan

   More actions

   @neersighted

   • but even Add a warning if credentials are written in lock file #3995 is only about warnings, but it's not blocking - and I agree a warning in that scenario would be beneficial
   • about the workflow:
     • deploy keys - you would need to add them each time in each related repo when someone wants to enable installing such dependencies via SSH in their CI / server, it simply doesn't make sense (ofc you can just share a common SSH key among projects, but then it's nothing more than overcomplicated deploy token, which is even more pointless
       🙃)
     • deploy tokens - you can just generate them "once" (let's skip token rolling for simplification) with only read_repository scope enabled, so it can be utilised by all the people in the company then. Of course there's a potential risk of a leak - if your self-hosted instance is accessible from the Internet, someone outside the company would be able to pull the affected repository having the token. If it's hidden behind VPN - it could be acceptable though
     • packages - yes, we talk about the scenario when you cannot enforce on people to do proper packaging
   • about gitcredentials - that's actually interesting, wasn't aware of that one. Seems that sth like the below:
     https://stackoverflow.com/a/43022442/6534668
     could be a workaround, but from the other side - the most convenient solution would be to utilise a common deploy token anyway, because AFAIK project access tokens allow to pull packages from other internal registries, but doesn't allow to clone other internal repositories (which I find personally quite strange, but maybe I miss something). Of course we don't have the token in git anymore, which is definitely beneficial, but if we want to keep it shareable - it's pretty easy to leak it anyway, just in different ways.

   My point is - even if we don't allow to provide tokens via poetry add, there are many other security gotchas, we wouldn't save the world anyway. Instead of that, we can actually complicate the life for people who need that functionality for whatever reason. And putting all the security deliberations aside - the whole issue is simply about not failing when running poetry add to follow the behaviour of pip and git. Only that, because as mentioned - it simply works when added manually to pyproject.toml.

   Personally - I really believe the warning message, maybe even with some note like consider usage of gitcredentials instead, would be a sufficient compromise. And the end of the day - we're in the Python world, so it's easier to ask for forgiveness than permission + we are all consenting adults 😉

   👍React with 👍1Reacted by snaeil
7. cbuffett commented on Jul 25, 2022

   

   cbuffett

   on Jul 25, 2022

   Author

   More actions

   @neersighted You're correct in that I work for a large organization where it can be difficult to get packages uploaded to a private registry (additional CI logic/maintenance). Using deploy keys may be a simpler option, but would require working with each dependent project to accept said key as alluded to by @jaklan. I will look into the gitcredentials route as an alternative in the meantime.
8. neersighted commented on Jul 26, 2022

   

   neersighted

   on Jul 26, 2022

   Last edited by neersighted

   Member

   More actions

   I do agree that the ask -- letting Poetry validate URLs with your credentials in them is simple. However, there is a reason we haven't fixed this long-standing bug and instead have added more alternatives -- you may be comfortable embedding secrets into your repository/history/build artifacts, but the fact that these files are sensitive (when with other projects/ecosystems they are not) violates the element of least surprise.

   An explicit gitcredentials or other alternatives (env vars for poetry install, etc etc) require the programmer to explicitly make the secrets available to the CI/deployment environment, and separates the handling of secrets from the handling of code.

   I think there is room to improve the docs here (e.g. adding a FAQ entry about how to handle private Git deps if you cannot use the preferred alternative of a registry), as well as food for thought for the PR warning on credentials in the lockfile (eventually, we want to make it an error) -- but I don't think making it easier to embed credentials (or encouraging its use) as it currently stands does anything to improve the situation.

   It may make your specific use case easier, but I am very worried that wider adoption of such a workflow is a footgun for the whole ecosystem.
9. jaklan commented on Jul 26, 2022

   

   jaklan

   on Jul 26, 2022

   More actions

   @neersighted

   but I am very worried that wider adoption of such a workflow is a footgun for the whole ecosystem

   Do you really think simply non-failing poetry add command (without any additional mentions in docs about it) would have any real impact on the adoption of it? If we could achieve it with git for years, if we could achieve it with pip for years - and we still can - if there are dozens of SO posts and issues mentioning such commands... the source of adoption is simply in another place. People don't try to do it with poetry because they found out a bug and decided to make it useful, but because they already know that option from the "pre-poetry" world.

   I think there is room to improve the docs here (e.g. adding a FAQ entry about how to handle private Git deps if you cannot use the preferred alternative of a registry)

   Fully agree, I would say even more - it's just crucial here. If you really want to change people mindset, extensive docs would be much more beneficial than just blocking people of doing sth without any obvious alternative (how much people would think about gitcredentials by themselves instead of falling back to pip install?). What's also important - such docs cannot be only poetry-oriented - #5567 can be mentioned, but it needs a clear statement it wouldn't work if someone tries to install your package as a dependency with pip.

   Having that, we would be able to utilise it in a warning / error message when trying to run poetry add with a token. Now, when it just fails with some generic error message, which makes you googling it and looking at multiple issues in GH - we just make it very confusing. So my point is - such docs should have a really high priority then.

   How I personally would imagine the roadmap here:

   1. Fix the bug in poetry add (don't overthink it yet - under-the-hood, with a manual entry in pyproject.toml, it works anyway, so just treat it as a bug, not an accidental security mechanism)
   2. Create extensive docs about the recommended solutions
   3. Provide warning messages starting from e.g. 1.3, pointing to the above docs

   and then, if you really decide to make it an error one day:

   4. Add to the warning information it will become an error in e.g. 1.4
   5. Give people some time to understand the change and get familiar with the alternatives
   6. Release 1.4 and make it an actual error

   Imho such a workflow allows you to achieve the final goal, but it also takes into account users' perspective (and emotions) and make the "mindset changing" simply easier to introduce, without unnecessary irritation or even frustration.
10. cbuffett commented on Aug 4, 2022

    

    cbuffett

    on Aug 4, 2022

    Author

    More actions

    I've confirmed that using gitcredentials (by running additional git commands in my Dockerfile to build the gitcredentials file) works and allows the downstream pip/poetry calls to access the private repo. So this is a workaround, albeit a not particularly graceful one. The deploy token is stored in the Dockerfile (this could be moved to a CI variable, though local non-CI builds would still need this provided from somewhere), though I suppose this is no different than having it stored in pyproject.toml and/or poetry.lock. At least with our use of multistage builds, the credentials do not persist into the final image.
11. jaklan commented on Aug 4, 2022

    

    jaklan

    on Aug 4, 2022

    More actions

    @cbuffett have you tried with envars?
    https://stackoverflow.com/a/43022442/6534668
12. cbuffett commented on Feb 6, 2025

    

    cbuffett

    on Feb 6, 2025

    Author

    More actions

    @jaklan I haven't, but the solution I have implemented for my CI pipeline works well enough. The bigger issue plaguing me is when it comes to updating the poetry package itself. While poetry build and poetry install work, commands like poetry lock and poetry add still result in the Invalid git url error, even with the poetry credentials configured for the downstream dependency, i.e., the one that's installed as part of the direct dependency.

    To build upon my previous example:

    [tool.poetry]
    name = "<package_name>"
    version = "0.0.0"
    description = "test"
    authors = ["Chris Buffett <email@domain.com>"]
    license = "Proprietary"
    
    [tool.poetry.dependencies]
    python = "^3.7"
    requests = "^2.25.1"
    dask = "^2021.4.1"
    fsspec = "^2021.4.0"
    partd = "^1.2.0"
    
     # This depends on another package using git+https://<gitlab_deploy_token>:<gitlab_token_password>@<private_gitlab_repo>/<private_package_2>.git
    <private_package_name> = {git = https://<private_gitlab_repo>/<private_package>.git", rev = "master"}
    
    [tool.poetry.extras]
    dask = ["dataframe"]
    
    [tool.poetry.dev-dependencies]
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    I've configured poetry credentials for https://<private_gitlab_repo>/<private_package_2>.git, but poetry add returns

    poetry lock
    Updating dependencies
    Resolving dependencies... (13.7s)
    
    Invalid git url "git+https://<gitlab_deploy_token>:<gitlab_token_password>@<private_gitlab_repo>/<private_package_2>.git"