Repository navigation
Support for build variants/profiles #2613
Description
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have searched the documentation and believe that my question is not covered.
Feature Request
For some Python packages, there are different wheel variants with identical package names, which are distinguished only based on PEP-440 local version labels (the label in 1.2.3+label). E.g. for PyTorch or mxnet, this is used to deliver different variants that use CPU or one of several CUDA versions. This is independent of system or hardware platform and Python version - PyTorch have a wheel for almost every combination of these.
For example, for PyTorch 1.5.1 with Python 3.8 on Linux there are versions pytorch-1.5.1 (for CUDA 10.2), pytorch-1.5.1+cu101 (for CUDA 10.1), pytorch-1.5.1+cu92 (for CUDA 9.2) and pytorch-1.5.1+cpu (for CPU).
As far as I know, the only way Poetry supports for switching between these variants on a given platform is to modify the pyproject.toml file, which obviously is not so optimal. Markers are not a solution here because it is not possible switch between different variants on the same machine. Neither are optional dependencies and extras, because the package names are identical for the different variants.
In the Java world, Maven offers build profiles, which allow for specifying different sets of dependencies, build plugins etc.. Maven profiles can be activated and deactivated on a per-build basis (or using triggers similar to markers, but that is not interesting for this use case) and independently of each other.
Maven profiles are a bit complex, but I think picking and modifying some of its ideas for this specific use case could be useful for Poetry.
I'd suggest the following syntax in pyproject.toml:
[tool.poetry]
name = "myproject"
version = "1.2.3"
...
[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...
[[tool.poetry.profile]]
name = "cuda-10.1-backend"
# Suggested syntax for specifying different local versions for different profiles
localversion = "cuda101"
[tool.poetry.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
{url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
{url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]
[[tool.poetry.profile]]
name = "cpu-backend"
localversion = "cpu"
[tool.poetry.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
{url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
{url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]
There could be one lock file per profile/combination of profiles (which I think would be easier to understand and maintain, and probably also to implement) or one global lockfile which combines several profiles/profile combinations.
The profiles could be activated using a CLI switch --profile/-p for poetry update/lock/install/add/etc.;
E.g. poetry lock -p cpu-backend would only lock the versions for the CPU variant, and poetry install -p cpu-backend would install the global packages and those defined in the cpu-backend lockfile.
Another variant would be profile groups where one of several alternatives must be active, and one of them may be active by default:
[tool.poetry]
name = "myproject"
version = "1.2.3"
...
[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...
[[tool.poetry.profilegroup]]
name = "backend"
# Either cuda-10.1 or cpu must be activated during lock/install/update, cpu is active by default, but remains deactivated if cuda-10.1 is activated
[[tool.poetry.profilegroup.profile]]
name = "cuda-10.1"
localversion = "cuda101"
[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
{url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
{url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]
[[tool.poetry.profilegroup.profile]]
name = "cpu"
default = true
localversion = "cpu"
[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
{url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
{url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]
For the profile group variant the syntax could be "-p backend=cpu" etc.
An alternative solutions might be allowing to specify constraints under extras, but I don't think that's the point of extras.
Activity
tarasivashchuk commentedon Nov 1, 2020
Anything on this? I always have to create a separate script to install PyTorch because of this.
sinoroc commentedon Nov 1, 2020
I am not familiar with that topic, but if I understood right these discussions might be related:
[My impression is that it is not something that poetry can/should tackle on its own. I feel like there is some background work that should be done in the larger Python packaging community beforehand: standardizations, and so on.]
jhrmnn commentedon Feb 19, 2021
A very trivial kind of "support" would be to (optionally?) not reinstall a package by
installwhen the installed version differs from the locked one only in the build label. Then one could at least once "overwrite" a package manually, and then Poetry would leave it be. Instead with Poetry 1.1.4, this happens:kikohs commentedon Mar 15, 2021
After spending a couple of hours on this issue, I found a "solution" by combining Poetry and pip just for PyTorch. You don't need to specify the wheel URLs directly and thus remain cross-platform.
I'm using Poe The Poet, a nice task runner for Poetry that allows to run any arbitrary command.
You can run:
and then:
Hope it helps.
sinoroc commentedon Mar 16, 2021
Reminder that you should always call the executable modules
python -m pip,pythonX.Y -m pip, etc. instead of the (unreliable) script wrapperspip,pipX.Y, etc.https://snarky.ca/why-you-should-use-python-m-pip/
kikohs commentedon Mar 17, 2021
@sinoroc I'm not sure it's really needed here as everything runs into Poetry virtual env but one can never be too careful! Thanks for sharing. I edited the comment.
sinoroc commentedon Mar 17, 2021
Absolutely, it should not be needed. But I find it is a good habit to take. Especially since in this case it is in a file, so you type the long form just once. The shorter script wrappers are good for interactive command line sessions (less characters to type).
jhrmnn commentedon May 21, 2021
Relevant discussion on Python.org:
https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125
vedosis commentedon Jun 2, 2021
At least within
pyproject.tomlI can get just torch to install via:However, the version comes across as
1.8.1+cpu. So trying to installtorchvisionin the same way fails because:Any concerns with
versionto be specified and for that value to override what's discovered for dependency resolution?Alternatively, dropping
+labelduring version resolution would work for me.I'm happy to do the implementation if you don't have spare cycles to do it.
kaustubhharapanahalli commentedon Sep 15, 2021
Hello, any update on this issue? I am facing the same issue.
Looks like the issue is because of considering the +cpu or +cu111 in the version as the URL is being provided.
According to PEP 440, poetry should be ignoring these local versions: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
danmackinlay commentedon Sep 19, 2021
@kaustubhharapanahalli Support for neural net packages is one of many unresolved issues waiting in the ticket queue for poetry at the moment. If we want any of them to progress we might need to investigate volunteering as poetry maintainers.
kaustubhharapanahalli commentedon Sep 21, 2021
@danmackinlay I see, thanks for the update.
danmackinlay commentedon Oct 6, 2021
@kaustubhharapanahalli regarding the pull-request backlog, see #4595
kaustubhharapanahalli commentedon Oct 6, 2021
@danmackinlay looks like the resolving of this issue might take some time if my understanding is right 😅
Actually there is a fix that was developed and PR was raised (#4221). So not really sure how long it might take as well. But I do support the governance ticket :)
danmackinlay commentedon Oct 27, 2021
NB the hack from March to use pip to install a desired package over the poetry version still works. It has various side effects that are undesirable (e.g. it must be run each time poetry "resolves" dependencies), so I find it easier to ditch poetry entirely in favor of a virtualenv in this case, but if there are some other features of poetry that you need this might be of interest.
mastern2k3 commentedon May 30, 2022
this used to work on preview release, but now it stopped working
is anybody else experiencing that?
392781 commentedon Jan 19, 2023
Why doesn't poetry support something akin to
--extra-index-urlinpip? I think this would solve the problem (at least in the case of PyTorch).Right now, Poetry just defaults to installing the nvidia dependencies for PyTorch regardless if your hardware requires them.