Skip to content

Extras of a dependency's dependency are not installed #1609

New issue
New issue
Closed
#2887
python-poetry/poetry-core
#78
Closed
Extras of a dependency's dependency are not installed#1609
#2887
python-poetry/poetry-core
#78
Assignees
sdispater
Labels
area/solverRelated to the dependency resolverkind/bugSomething isn't working as expected
Milestone
1.1

Description

@hukkin
hukkin
opened on Nov 20, 2019
  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

Poetry does not install extras of a dependency's dependency, if the dependency's dependency is also referenced without the extra.

Here's the pyproject.toml. The generated poetry.lock lockfile is here.

Pyproject.toml requires PyOTA package, which requires requests[security]. The security extra is not installed though. This can be observed by the fact the the lockfile has no pyOpenSSL entry, which is a part of the security extra of requests.

Note that we remove requests = "*" from pyproject.toml, then the security extra will be installed as expected. It seems the issue only occurs when the package that needs extras is referenced elsewhere with no mention of the extras.

Activity

  1. hukkin
    added
    kind/bugSomething isn't working as expected
    on Nov 20, 2019
  2. sdispater
    added
    area/solverRelated to the dependency resolver
    on Nov 21, 2019
  3. sdispater
    self-assigned this
    on Nov 21, 2019
  4. sdispater
    modified the milestones: 1.0, Future on Nov 21, 2019

  5. sdispater commented on Nov 21, 2019

    @sdispater
    sdispater
    on Nov 21, 2019
    Member

    This is a tricky one.

    I can definitely reproduce and I know why that happens but actually fixing this is not trivial. Unfortunately, it's unlikely that it will be fixed before the 1.0.0 release because it will need a tweak in the resolution logic.

    For the time being, a workaround is to declare your dependency on requests like the following:

    requests = {"version": "*", extras = ["security"]}

  6. hukkin commented on Nov 22, 2019

    @hukkin
    hukkin
    on Nov 22, 2019
    Author

    Thanks @sdispater for having a look.

    There is a case where not even the workaround works. I'm not sure how related this is, maybe I should file another bug report for this...?

    Anyways I'll try to briefly explain it here: Let's imagine a world where the PyOTA package we required has not been released on PyPI, but only on a git server. Then our pyproject.toml (with the workaround) will look something like:

    requests = { version = "*", extras = [ "security" ] }
PyOTA = { git = "https://github.com/iotaledger/iota.py", tag = "2.1.0" }

    Running poetry install will then create a perfect poetry.lock that includes the requests[security] extra. However, the next time we run poetry update what happens is, that the security extra gets removed (copy-pasting command line output here):

    $ poetry update
Updating dependencies
Resolving dependencies... (3.8s)

Writing lock file


Package operations: 0 installs, 1 update, 4 removals

  - Updating PyOTA (0.0.0 3057a1b -> 2.1.0 2.1.0)
  - Removing cffi (1.13.2)
  - Removing cryptography (2.8)
  - Removing pycparser (2.19)
  - Removing pyopenssl (19.1.0)

    Following poetry update calls will not bring the security extra back.

    The obvious workaround for this is to require the packages of the security extra explicitly, but this gets very awkward if the extra contains many packages.

    P.S. Thanks for poetry! Despite these issues it's still an amazing tool.

  7. sdispater commented on Nov 22, 2019

    @sdispater
    sdispater
    on Nov 22, 2019
    Member

    The only way we can solve the extras issues currently is by always resolving the extra dependencies even though they were not opted in. That would fix the issue but it might introduce another which is that resolution conflict might be raised by extra dependencies even though they were never opted in.

    I don't see a perfect solution for this at the moment. The Python ecosystem is so unique and complex (some might say convoluted) that coming up with a perfect dependency resolver is almost impossible and will sometimes require the help of the end user by hinting it on how it should proceed.

  8. absassi commented on Jan 8, 2020

    @absassi
    absassi
    on Jan 8, 2020

    @sdispater I haven't looked at how the code works, but do you really need to resolve all extras to fix this? It seems like you can keep a list of required extras when resolving a dependency, and if that dependency is required again by some other path you don't skip it or resolve it again from scratch, but just check if there are new extras and only resolve them. You may need to keep a list of required extras for each package to append to whenever you resolve an extra, and you may need some work on how you resolve a dependency in order to be able to do more work on it afterwards (i.e. resolve more extras), but seems doable.

    Another (maybe simpler) alternative, is to consider requests[security] as a new virtual dependency, which has the same versions available as requests, but installs no package and depends on the concatenated lists of requirements for requests and for security extra. In other words, when you find requests = {version = "x", extras = ["security"]} you understand that as "requests[security]" = {version = "x"} and know that to find the metadata of requests[security], you look at the requests metadata, but take the combined requirements from requests and from its security extra instead. The current resolver should be able to resolve this correctly and without any false-positive conflicts. You can pre-create all these virtual extra packages as soon as you parse the metadata of requests if you wish, but only really require these virtual dependencies when you see them referenced somewhere.

  9. daniellehanks commented on Apr 20, 2020

    @daniellehanks
    daniellehanks
    on Apr 20, 2020

    Having this problem when installing a google lib that does not require the grpcio-gcp extra on google-api-core (e.g. google-cloud-bigtable), then installing one that does (e.g. google-cloud-spanner). The grpcio-gcp extra does not get added unless the adds are done with the one requiring grpcio-gcp first.

  10. sdispater
    modified the milestones: Future, 1.1 on Aug 11, 2020
  11. sdispater
    closed this as completedin poetry-core#78on Sep 17, 2020

  12. mvoitko commented on Oct 17, 2020

    @mvoitko
    mvoitko
    on Oct 17, 2020

    Still relevant to me.
    Poetry version 1.1.2

    poetry install "google-cloud-bigquery[pandas]"

    google-cloud-bigquery is installed. pandas is not.

  13. abn commented on Oct 17, 2020

    @abn
    abn
    on Oct 17, 2020
    Member

    @mvoitko can you please try 1.1.3? There is also an issue with nested extras that will be resolved later with 1.1.4.

  14. mvoitko commented on Oct 23, 2020

    @mvoitko
    mvoitko
    on Oct 23, 2020

    @abn updating helped

  15. vlad0337187 commented on Apr 15, 2022

    @vlad0337187
    vlad0337187
    on Apr 15, 2022
    Contributor

    updating to 1.1.3 didn't help me.

    I still see extras of dependency in pyproject.toml of main app, it it wasn't installed

  16. github-actions commented on Mar 2, 2024

    @github-actions
    github-actionsBot
    on Mar 2, 2024 – with GitHub Actions

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  17. github-actions
    locked as resolved and limited conversation to collaborators on Mar 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/solverRelated to the dependency resolverkind/bugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions