pathological version solving? (django-distill + boto3)

Poetry version: 1.7.1
Python version: 3.11
OS version and name: macOS 14.1.1
pyproject.toml:

[tool.poetry]
name = "poetrytemp"

[tool.poetry.dependencies]
python = "^3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I can reproduce this in an empty poetry project.

attempt to poetry add -vv 'django-distill[amazon]'

output is like:

Resolving dependencies...
   1: fact: poetrytemp is 0.1.0
   1: derived: poetrytemp
   1: fact: poetrytemp depends on django-distill (^3.1.3)
   1: selecting poetrytemp (0.1.0)
   1: derived: django-distill[amazon] (>=3.1.3,<4.0.0)
   1: fact: django-distill (3.1.3) depends on django-distill (3.1.3)
   1: fact: django-distill (3.1.3) depends on django (*)
   1: fact: django-distill (3.1.3) depends on requests (*)
   1: fact: django-distill (3.1.3) depends on boto3 (*)
   1: selecting django-distill[amazon] (3.1.3)
   1: derived: boto3
   1: derived: requests
   1: derived: django
   1: derived: django-distill (==3.1.3)
   1: fact: django-distill (3.1.3) depends on django (*)
   1: fact: django-distill (3.1.3) depends on requests (*)
   1: selecting django-distill (3.1.3)
   1: fact: django (5.0) depends on asgiref (>=3.7.0)
   1: fact: django (5.0) depends on sqlparse (>=0.3.1)
   1: fact: django (5.0) depends on tzdata (*)
   1: selecting django (5.0)
   1: derived: tzdata
   1: derived: sqlparse (>=0.3.1)
   1: derived: asgiref (>=3.7.0)
   1: fact: requests (2.31.0) depends on charset-normalizer (>=2,<4)
   1: fact: requests (2.31.0) depends on idna (>=2.5,<4)
   1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3)
   1: fact: requests (2.31.0) depends on certifi (>=2017.4.17)
   1: selecting requests (2.31.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<3)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2,<4)
   1: selecting urllib3 (2.1.0)
   1: selecting certifi (2023.11.17)
   1: selecting charset-normalizer (3.3.2)
   1: selecting idna (3.6)
   1: selecting sqlparse (0.4.4)
   1: selecting asgiref (3.7.2)
   1: fact: boto3 (1.34.7) depends on botocore (>=1.34.7,<1.35.0)
   1: fact: boto3 (1.34.7) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: boto3 (1.34.7) depends on s3transfer (>=0.10.0,<0.11.0)
   1: selecting boto3 (1.34.7)
   1: derived: s3transfer (>=0.10.0,<0.11.0)
   1: derived: jmespath (>=0.7.1,<2.0.0)
   1: derived: botocore (>=1.34.7,<1.35.0)
   1: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0)
   1: selecting s3transfer (0.10.0)
   1: fact: botocore (1.34.7) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: botocore (1.34.7) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: derived: not botocore (==1.34.7)
   1: fact: no versions of botocore match >1.34.7,<1.35.0
   1: conflict: no versions of botocore match >1.34.7,<1.35.0
   1: derived: not botocore (>1.34.7,<1.35.0)
   1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0)
   1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0"
   1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: derived: not botocore (>=1.34.7,<1.35.0)
   1: derived: not boto3 (==1.34.7)
   2: selecting certifi (2023.11.17)
   2: selecting charset-normalizer (3.3.2)
   2: selecting idna (3.6)
   2: selecting sqlparse (0.4.4)
   2: selecting asgiref (3.7.2)
   2: fact: boto3 (1.34.6) depends on botocore (>=1.34.6,<1.35.0)
   2: fact: boto3 (1.34.6) depends on jmespath (>=0.7.1,<2.0.0)
   2: fact: boto3 (1.34.6) depends on s3transfer (>=0.10.0,<0.11.0)
   2: selecting boto3 (1.34.6)
   2: derived: s3transfer (>=0.10.0,<0.11.0)
   2: derived: jmespath (>=0.7.1,<2.0.0)
   2: derived: botocore (>=1.34.6,<1.35.0)
   2: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0)
   2: selecting s3transfer (0.10.0)
   2: fact: botocore (1.34.6) depends on jmespath (>=0.7.1,<2.0.0)
   2: fact: botocore (1.34.6) depends on python-dateutil (>=2.1,<3.0.0)
   2: fact: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1)
   2: derived: not botocore (==1.34.6)
   2: fact: no versions of botocore match >1.34.6,<1.34.7
   2: conflict: no versions of botocore match >1.34.6,<1.34.7
   2: derived: not botocore (>1.34.6,<1.34.7)
   2: conflict: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1)
   2: ! botocore (==1.34.6) is partially satisfied by not botocore (>1.34.6,<1.34.7)
   2: ! which is caused by "no versions of botocore match >1.34.6,<1.34.7"
   2: ! thus: botocore (>=1.34.6,<1.34.7) requires urllib3 (>=1.25.4,<2.1)

this continues forever in a fruitless search, downloading the wheel for every minor version of botocore

the problem seems to be something to do with urllib3

   1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3)

   1: derived: urllib3 (>=1.21.1,<3)

   1: selecting urllib3 (2.1.0)

   1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)

   1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)
   1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0)
   1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0"
   1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)
   1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)

it's not clear from the output why https://pypi.org/project/urllib3/1.26.18/ couldn't be used to satisfy >=1.25.4,<2.1

Ok but then the odd thing is if I do this:

poetry add boto3
poetry add 'django-distill[amazon]'

then it works fine!

If we look at the setup.py for django-distill: https://github.com/meeb/django-distill/blob/v3.1.3/setup.py#L30
it specifies:

    extras_require = {
        'amazon': ['boto3'],

So no version constraint on boto3.

It doesn't make much sense that dependency resolution fails one way but not the other?

boto3 and urllib3 are known to be an unhappy pairing, you will find lots of similar issues if you search. Current view is that there is nothing much to be done about it, sorry.

please close.

@anentropic

Thanks for opening this issue @anentropic ! I ran into a similar pattern recently and found that the following the workaround you mentioned helped too.

@dimbleby - thanks too for your highlighting the related elements and that there may be a pattern at hand! If it already exists, could you link to the relevant issue, PR, or discussion where boto3 and urllib3 incompatibilities are tracked? I'd mostly suggest this to be sure others who run into the same problem can understand the context and make sure a resolution eventually is addressed. Even if this isn't directly a Poetry issue to solve, it does exhibit itself through the use of the tool, so it could be helpful to make sure others are informed (especially as this can result in a lot of time through attempted dependency resolution).

@d33bs

@d33bs

you will find lots of similar issues if you search

please do that yourself if you want to find the similar issues, feel free to add links

@anentropic

Thanks for the workaround, @anentropic!

As @d33bs writes, this issue exhibits itself through Poetry, so some warning or other user facing information in these situations would be great!

ps. for future searches, we hit this when installing fiftyone

similar issues:

most of these: https://github.com/python-poetry/poetry/issues?q=is%3Aissue+boto3+is%3Aclosed
Poetry cannot correctly select dependencies #5896 ... this comment on there appears to say that the version solving will eventually succeed if you can wait long enough for it to exhaust literally all boto3 versions before it tries reducing the urllib3 version.

My first thought is that some sort of bisecting strategy might help it converge on the working solution quicker.

I guess the issue is Poetry feels that just because boto3 1.34.0 and 1.33.0 both require urllib3 (>=1.25.4,<2.1) it can't trust that somewhere in the middle say boto3 1.33.8 might randomly allow urllib3 2.1.0, and then it has found the absolute latest version of boto3 and urllib3 that satisfy the constraints, which was only possible by exhaustively searching every boto3 version.

So there are two fixes users could take right now:

anything which pins boto3 to a specific version, e.g. explicitly install boto3 first, thus causing the urllib3 <2.1 constraint to get enforced immediately
or presumably explicitly pinning urllib3 <2.1 would have the same effect

FWIW pdm seems to cope with this fine:

$ pdm add 'django-distill[amazon]'
Adding packages to default dependencies: django-distill
🔒 Lock successful
Changes are written to pyproject.toml.
Synchronizing working set with resolved packages: 15 to add, 0 to update, 0 to remove

  ✔ Install asgiref 3.7.2 successful
  ✔ Install idna 3.6 successful
  ✔ Install jmespath 1.0.1 successful
  ✔ Install python-dateutil 2.8.2 successful
  ✔ Install requests 2.31.0 successful
  ✔ Install charset-normalizer 3.3.2 successful
  ✔ Install s3transfer 0.10.0 successful
  ✔ Install six 1.16.0 successful
  ✔ Install sqlparse 0.4.4 successful
  ✔ Install urllib3 2.0.7 successful
  ✔ Install django-distill 3.1.3 successful
  ✔ Install django 5.0.1 successful
  ✔ Install boto3 1.34.35 successful
  ✔ Install certifi 2024.2.2 successful
  ✔ Install botocore 1.34.35 successful

🎉 All complete!

all resolved in a reasonable amount of time.

So I wonder what they're doing differently.

So I wonder what they're doing differently.

it's just good or bad luck whether your solver happens first to explore a path that fixes boto3 first (when it is very easy to find a satisfactory urllib3) or a path that fixes urllib3 first (when it is very hard to find a satisfactory boto3)

pdm has its own pathological cases!

the most useful thing you can do right now, for future-you and the rest of the ecosystem, is to go and offer merge requests to django-distill or fiftyone or whoever, putting a (recent) lower bound on their boto3 dependency.

Then no installer is exposed to having to backtrack through the thousands of versions of boto3 that amazon release

I've run into this issue with Celery 5.4.0. I don't need boto3 or the related sqs package that Celery directly has as an optional dependency, so the workaround I want is to simply exclude the sqs extra. I thought that having extras set in pyproject.toml would mean that those and only those extras would be pulled in, but despite setting it with exactly the extras I do want for Celery (and Kombu) it continues to insist on downloading every single release of boto3.

It usually takes about 950 seconds before it starts downloading anything. I'm a little afraid that setting Celery to 5.4.0 will make all poetry lock runs in the future take 20+ minutes because of a package I don't even want and am trying to exclude.

Edit: Apparently the pytest-celery package (from my dev group) can also pull in sqs and that was why Poetry seemed to be ignoring the extras I gave for Celery itself. I didn't find that sooner because boto3 was being installed for the first time and so poetry show --tree didn't list these packages and what was pulling them in. Overall an incredibly frustrating experience, but at least fixing that has it working in the usual 5 seconds or so.

$ time poetry add --lock celery=5.4.0 pytest-celery
Using version ^1.1.1 for pytest-celery

Updating dependencies
Resolving dependencies... (3.7s)

Writing lock file

real    0m5.155s
user    0m2.697s
sys     0m0.148s

hard to guess what happened to you, but neither celery nor pytest-celery is problematic

Probably a result of changing several dependency specs all at once, and one hack that I remember made sense at the time but was probably the real culprit.

[tool.poetry.dependencies]
-celery = {version="^5.3.6", extras=["redis", "tblib", "auth"]}
+celery = {version="5.4.0", extras=["redis", "tblib", "auth", "gevent"]}

 [tool.poetry.group.dev.dependencies]
-pytest = "^4.6"
# this is the hack: re-declaring celery in the dev group to add the pytest extra.
-celery = {version="*", extras=["pytest"]}
+pytest = "*"
# What I should have done in the first place was just declare pytest-celery directly:
+pytest-celery = {version="*", extras=["rabbitmq", "redis"]}
# With this, it's working like normal even with celery pinned to 5.4.0, the change that triggered this whole thing.

When I ran poetry update celery or poetry lock, it would take 20 minutes to get to the point where it started downloading every botocore release. I never let it run to completion, and the furthest it got after several attempts was botocore 1.15.44 (that particular run was 1098.3s).

It seems that having celery[pytest] was equivalent to having pytest-celery[all], which then pulled in sqs and thus boto3.

edit:

hrm, I thought the diff would make it pretty clear, but not really... The problematic config was:

[tool.poetry.dependencies]
celery = {version="5.4.0", extras=["redis", "tblib", "auth", "gevent"]}

[tool.poetry.group.dev.dependencies]
pytest = "*"
celery = {version="*", extras=["pytest"]}

when previously celery was set to 5.3.6 and pytest-celery matched to that automatically. I believe sqs is a new extra for both Celery and pytest-celery as of 5.4, but I could be wrong.

still locks in about five seconds for me.

Yes there are cases that cause a lot of backtracking. Quite possibly you hit one, though you are unable to say what it was.

But there still is nothing much to be done about it.

As I said back in February: the most useful thing you can do right now, for future-you and the rest of the ecosystem, is to go and offer merge requests to django-distill or fiftyone or whoever, putting a (recent) lower bound on their boto3 dependency.

Then no installer is exposed to having to backtrack through the thousands of versions of boto3 that amazon release

Be that is it may, I probably could have resolved the case if I at least knew why botocore was being pulled in. I'd never even heard of botocore before yesterday, and poetry show was (correctly, I guess) saying there was nothing to show because botocore (and sqs and boto3) wasn't actually installed. It took most of the afternoon to figure out pytest-celery was the culprit. A simple message would have saved me a lot of time and frustration:

Updating dependencies
Resolving dependencies... 
pytest-celery[sqs] depends on botocore, which needs to be downloaded to continue dependency resolution.
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/ab/e5/4bf433f4fe4bb193f65c9d8be0b72d31c0ac04a564b787cc0f0929644325/botocore-1.17.26-py2.py

@MajorDallas

Love the idea of a message which @MajorDallas suggested! Perhaps this isn't in scope for Poetry to fix, but developers often may have to dig deeply to find where the error may be coming from with little notification at the moment from Poetry by default. Maybe a timed notification for long-running procedures related to one dependency resolution could help! For example, if the resolution process is taking an excessive amount of time (i.e. longer than an hour or whatever is reasonable) an additional message appears during the process to notify the user about which dependency is causing the jam.

Verbose logging output already includes such information

That is good to know, and I'm a little embarrassed it didn't occur to me try it 😅

Still, I agree with d33bs: if it can be detected that things are taking longer than normal (say, more than 10 minutes or more than 30 backtracking steps), a message about what is being is resolved (without needing a v or three) would be nice. That makes it a simple decision for the user whether to wait it out or revise the declared dependencies.

Navigation Menu

Repository navigation

pathological version solving? (django-distill + boto3) #8823

Issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pathological version solving? (django-distill + boto3) #8823

Description

Issue

Activity

dimbleby commented on Dec 26, 2023

d33bs commented on Feb 1, 2024

dimbleby commented on Feb 4, 2024

Northo commented on Feb 6, 2024

anentropic commented on Feb 6, 2024

dimbleby commented on Feb 6, 2024

dimbleby commented on Feb 6, 2024

MajorDallas commented on Aug 20, 2024

dimbleby commented on Aug 20, 2024

MajorDallas commented on Aug 20, 2024

dimbleby commented on Aug 20, 2024

MajorDallas commented on Aug 21, 2024

d33bs commented on Aug 21, 2024

dimbleby commented on Aug 21, 2024

MajorDallas commented on Aug 22, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions