Repository navigation
pip.parse does not accept lock files with duplicated dependencies #1615
Description
🐞 bug report
Affected Rule
pip.parse
Is this a regression?
Yes, this behavior is only present when using bzlmod
Description
When using pip.parse, if a dependency is repeated in your requirements lock, you'll get an error. This is handled just fine by pip and is necessary for tools like poetry to export this way because it's possible for the same dependency to have different constraints in different parts of the dependency tree.
Ref: python-poetry/poetry#5688
🔬 Minimal Reproduction
Repo with repro here: https://github.com/AndrewGuenther/rules_python_1615_repro
The main branch uses bzlmod and workspace uses WORKSPACE. The workspace branch builds fine, but main does not. Same requirements on both branches, the only difference is bzlmod.
🔥 Exception or Error
If you try and build the example above, you'll get an error to the following effect:
ERROR: Traceback (most recent call last):
File "/home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl", line 298, column 30, in _pip_impl
_create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides)
File "/home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl", line 129, column 20, in _create_whl_repos
whl_library(
Error in repository_rule: A repo named pip_38_pyjwt is already generated by this module extension at /home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl:129:20
ERROR: error evaluating module extension pip in @@rules_python~0.27.1//python/extensions:pip.bzl
INFO: Elapsed time: 1.071s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
FAILED:
Fetching module extension pip in @@rules_python~0.27.1//python/extensions:pip.bzl; starting
🌍 Your Environment
Operating System:
Ubuntu
Output of bazel version:
6.4.0
Rules_python version:
0.27.1
Anything else relevant?
Activity
AndrewGuenther commentedon Dec 15, 2023
So this appears to be a bzlmod exclusive behavior? We can see this issue with bzlmod starting from 0.15.0, but if we use
pip_parsein ourWORKSPACEpointing to the same version it works just fine. Are these implementations different between bzlmod and non-bzlmod?AndrewGuenther commentedon Dec 15, 2023
Alright, I've now created a repo which shows this is exclusively an issue with
bzlmod: https://github.com/AndrewGuenther/rules_python_1615_reproThe
mainbranch uses bzlmod and theworkspacebranch is the equivalent.mainfails to build, butworkspacebuilds no problem.rickeylev commentedon Dec 15, 2023
That behavior differs between bzlmod and workspace here is a bit strange. The two implementation re-use a lot of the lower level pieces.
The thing that sticks out to me is both use the same function to parse the requirements. That parsing ignores the
[extras]part of a requirements line, e.g.foo ...andfoo[bar] ...gets returned as[foo, foo]in the parsed result. So, something in the workspace code must be doing...something...to prevent a duplicate call towhl_library(name="foo"), but I don't see where that is happening.rickeylev commentedon Dec 15, 2023
Uhhh, huh.
So the workspace code path isn't de-duplicating the names. It is calling
whl_library(name="pip_pyjwt")twice, but the repository rule implementation is only invoked once. I have no idea how this isn't an error. I don't see anymaybewrappers being used.In this case, the last one wins. The last one is the one whose requirement line sorts last (there's an earlier sorted() call on the list of requirement lines).
In practice, I think this works out OK? Because the requirements file expands out extras into their actual packages? I'm not too familiar with how extras work. If so, then an earlier line being ignored doesn't matter.
It should be easy to mimic this behavior in bzlmod. We just have to dedupe the names, giving preference to the last one.
AndrewGuenther commentedon Dec 15, 2023
@rickeylev I'm happy to submit a PR if you can point me in a direction. Unless this is one of those "it'd take the same time to explain as it would to implement" sort of things.
[-]pip_parse does not accept lock files with duplicated dependencies[/-][+]pip.parse does not accept lock files with duplicated dependencies[/+]fix(bzlmod pip.parse): allow requirements with extras
rickeylev commentedon Dec 15, 2023
I've got a fix, so now its just about constructing a test case.
Is there a trick to generating a requirements file with both
pyjwtandpyjwt[crypto]entries? When I do it myself, it just haspyjwt[crypto].fix(bzlmod pip.parse): allow requirements with extras
fix(bzlmod pip.parse): allow requirements with extras
fix(bzlmod pip.parse): allow requirements with extras
AndrewGuenther commentedon Dec 16, 2023
It's a quirk with
poetryand it'sexportfunction. If you have apyproject.tomllike this:and run
poetry exportyou'll get two instances ofpyjwtin the requirements output.AndrewGuenther commentedon Dec 16, 2023
The relevant poetry behavior is discussed a bit more in this issue: python-poetry/poetry#5688
It's not so much an issue with extras as it is with constraints in general.
fix(bzlmod pip.parse): allow requirements with extras
fix(bzlmod pip.parse): allow requirements with duplicate package lines (