Skip to content

Resolution fails for boto3/botocore in multiple python versions #3367

New issue
New issue
Closed
#4695
Closed
Resolution fails for boto3/botocore in multiple python versions#3367
#4695
Labels
area/solverRelated to the dependency resolverkind/bugSomething isn't working as expected

Description

@mcouthon
mcouthon
opened on Nov 15, 2020
  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: macOS Catalina 10.15.7
  • Poetry version: Poetry version 1.1.4
  • Link of a Gist with the contents of your pyproject.toml file: link

Issue

We need very specific boto3/botocore versions for compatibility with legacy code. In our non-poetry projects we set the following in the requirements.txt:

botocore==1.12.192;python_version<"3.0"
botocore==1.17.63;python_version>"3.0"
boto3==1.9.192;python_version<"3.0"
boto3==1.14.63;python_version>"3.0"

And installation works great in both python 2.7 and python 3.x.

I've attempted to create a comparable pyproject.toml:

[tool.poetry.dependencies]
python = "^2.7 || ^3.6"

botocore = [
    {version = "1.12.192", python = "^2"},
    {version = "1.17.63", python = "^3"}
]
boto3 = [
    {version = "1.9.192", python = "^2"},
    {version = "1.14.63", python = "^3"}
]

But poetry lock results in an error (on both python versions). Here's the error in a python 3.8 env:

   1: conflict: boto3 (1.14.63) depends on botocore (>=1.17.63,<1.18.0)
   1: ! not botocore (>=1.17.63,<1.18.0) is satisfied by botocore (1.12.192)
   1: ! which is caused by "test depends on botocore (1.12.192)"
   1: ! thus: boto3 is forbidden
   1: ! boto3 (1.14.63) is satisfied by boto3 (1.14.63)
   1: ! which is caused by "test depends on boto3 (1.14.63)"
   1: ! thus: version solving failed

And while the resolution starts with:

   0: Retrying dependency resolution with the following overrides ({Package('test', '0.1.0'): {'botocore': <Dependency botocore (1.12.192)>, 'boto3': <Dependency boto3 (1.9.192)>}}).

It at some point switches to:

   0: Retrying dependency resolution with the following overrides ({Package('test', '0.1.0'): {'botocore': <Dependency botocore (1.12.192)>, 'boto3': <Dependency boto3 (1.14.63)>}}).

Which of course causes the error (as these two are indeed incompatible versions).

Full output of poetry lock -vvv can be seen here.

Activity

  1. mcouthon
    added
    kind/bugSomething isn't working as expected
    status/triageThis issue needs to be triaged
    on Nov 15, 2020
  2. finswimmer
    added
    area/solverRelated to the dependency resolver
    on Nov 18, 2020

  3. tchar commented on Aug 10, 2021

    @tchar
    tchar
    on Aug 10, 2021

    Not sure if I should create a new issue, but this looks similar to mine.

    I have a similar problem with dev dependencies.

    [tool.poetry.dependencies]
python = ">=2.7"
six = [
    # Tox needs this version
    {version = "^1.14.0", python = ">=2.7 !=3.0.* !=3.1.* !=3.2.*"},
     # Python 3.0 through 3.2 need this version
    {version = "==1.11.0", python = ">=3.0.* <=3.2.*"}
]

[tool.poetry.dev-dependencies]
# Tox should not be installed on python versions where 
# six==1.11.0 is installed (3.0, 3.1, 3.2) plus (3.3, 3.4) which are not supported by tox.
tox = { version = "^3.24.1", python = ">=2.7 !=3.0.* !=3.1.* !=3.2.* !=3.3.* !=3.4.*" }

    Output

      SolverProblemError

  Because no versions of tox match >3.24.1,<4.0.0
   and tox (3.24.1) depends on six (>=1.14.0), tox (>=3.24.1,<4.0.0) requires six (>=1.14.0).
  So, because yourmodule depends on both six (==1.11.0) and tox (^3.24.1), version solving failed.

    I forced uploaded to PyPI and I can normally install my package through pip (it just mentions that the dev dependency is not getting installed, which end users don't really care unless they want to test)

    Also I am telling poetry not to install tox on a superset of python versions that I specified for six==1.11.0, so this should't even be a conflict.

  4. nugend commented on Sep 2, 2021

    @nugend
    nugend
    on Sep 2, 2021

    Anyone have any ideas on how to work around this issue?

  5. mcouthon commented on Sep 2, 2021

    @mcouthon
    mcouthon
    on Sep 2, 2021
    ContributorAuthor

    We just gave up, and created a python2 incompatible version of the package 🤷

  6. radoering commented on Nov 1, 2021

    @radoering
    radoering
    on Nov 1, 2021
    Member

    I think, I found a solution for this issue. With the changes in PR #4695, I was able to lock the boto3/botocore example. The solution relies on poetry recognizing if the intersection of markers is empty. Unfortunately, there is also some room for improvement. See #4694 for details. Without this improvement to poetry-core, some unnecessary combinations of overrides are solved in dependency resolution, resulting in an addtional version of botocore in the lock file. I am not 100 % sure, but I think this additional entry in the lock file does no damage, because it not used for any python version.

  7. GtTmy commented on Dec 23, 2021

    @GtTmy
    GtTmy
    on Dec 23, 2021

    I have a similar case. poetry (1.1.12) cannot solve the following dependencies.
    And I confirmed that #4695 makes it possible.

    [tool.poetry.dependencies]
python = "^2.7 || ^3.7"

[tool.poetry.dev-dependencies]
pytest = [
    {version="^4.6.11", python="^2.7"},
    {version="^6.2.0", python="^3.7"}
]
pytest-mock = [
    {version="^1.13.0", python="^2.7"},
    {version="^3.6.1", python="^3.7"}
]

    With poetry 1.1.12

    https://gist.github.com/GtTmy/22bfd306a53ed7538eededd31f443cca

    With #4695

    $ python -m poetry update
Updating dependencies
Resolving dependencies... (14.0s)

Writing lock file

Package operations: 9 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.6)
  • Installing attrs (21.2.0)
  • Installing iniconfig (1.1.1)
  • Installing packaging (21.3)
  • Installing pluggy (1.0.0)
  • Installing py (1.11.0)
  • Installing toml (0.10.2)
  • Installing pytest (6.2.5)
  • Installing pytest-mock (3.6.1)

  8. GrimHacker commented on Mar 14, 2022

    @GrimHacker
    GrimHacker
    on Mar 14, 2022

    I believe i'm seeing this as well.
    And I confirmed that #4695 makes it possible.

    ├── grimlib
│   └── __init__.py
├── poetry.lock
└── pyproject.toml


    [tool.poetry]
name = "grimlib"
version = "1.0.0" 
description = "poetry test"
authors = ["GrimHacker"]

[tool.poetry.dependencies]
python = "^2.7 || ^3.7"

# For py3.10 we need to use pytest >=6.2.5 (https://github.com/pytest-dev/pytest/issues/8539)
# For pytest >= 6.1.0 we need to use pytest-rerunfailures >=9.1.1 (https://github.com/pytest-dev/pytest-rerunfailures/issues/128)
[tool.poetry.dev-dependencies]
pytest = [
    { version = "<=4.6.11", python = "~2.7" },  # dropped py2 in 5.0
    { version = "<8", python = ">=3.7,<3.10" },
    { version = ">=6.2.5,<8", python = "^3.10" }  # >=6.2.5 https://github.com/pytest-dev/pytest/issues/8539
]
pytest-rerunfailures = [
    { version = "<9.0", python = "~2.7" },  # dropped py2 in 9.0
    { version = "<11", python = ">=3.7,<3.10" },
    { version = ">=9.1.1,<11", python = "^3.10"}  # For pytest >= 6.1.0 we need to use pytest-rerunfailures >=9.1.1 (https://github.com/pytest-dev/pytest-rerunfailures/issues/128)
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

    With poetry 1.1.13

    $ poetry lock -vvv
...
  SolverProblemError

  Because no versions of pytest-rerunfailures match >9.1.1,<10.0 || >10.0,<10.1 || >10.1,<10.2 || >10.2,<11
   and pytest-rerunfailures (9.1.1) depends on pytest (>=5.0), pytest-rerunfailures (>=9.1.1,<10.0 || >10.0,<10.1 || >10.1,<10.2 || >10.2,<11) requires pytest (>=5.0).
  And because pytest-rerunfailures (10.0) depends on pytest (>=5.3), pytest-rerunfailures (>=9.1.1,<10.1 || >10.1,<10.2 || >10.2,<11) requires pytest (>=5.0).
  And because pytest-rerunfailures (10.1) depends on pytest (>=5.3)
   and pytest-rerunfailures (10.2) depends on pytest (>=5.3), pytest-rerunfailures (>=9.1.1,<11) requires pytest (>=5.0).
  So, because grimlib depends on both pytest (<=4.6.11) and pytest-rerunfailures (>=9.1.1,<11), version solving failed.

    With #4695

    $ poetry lock -vvv
...
   0: Resolved with overrides: ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}, Package('pytest', '4.6.11'): {'more-itertools': <Dependency more-itertools (>=4.0.0,<6.0.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}, Package('pytest', '4.6.11'): {'more-itertools': <Dependency more-itertools (>=4.0.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<8)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<8)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (>=6.2.5,<8)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (>=6.2.5,<8)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (>=9.1.1,<11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<empty>)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<empty>)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<empty>)>}})
  9. KaoruNishikawa
    added a commit that references this issue on Mar 28, 2022

    #506 Use Pipenv

    d0566aa
  10. abn
    closed this as completedin #4695on Apr 26, 2022
  11. mkniewallner
    removed
    status/triageThis issue needs to be triaged
    on Jun 11, 2022

  12. github-actions commented on Mar 1, 2024

    @github-actions
    github-actionsBot
    on Mar 1, 2024 – with GitHub Actions

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  13. github-actions
    locked as resolved and limited conversation to collaborators on Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/solverRelated to the dependency resolverkind/bugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions