Repository navigation
Regression in namespace package metadata on poetry-core 1.1.0rc1 #6198
Description
-
I am on the latest Poetry version.n/a - I have searched the issues of this repo and believe that this is not a duplicate.
-
If an exception occurs when executing a command, I executed it again in debug mode (n/a-vvvoption).
- OS version and name: MacOS 12.5.1
- Poetry version: poetry-core 1.1.0rc1
- Link of a Gist with the contents of your pyproject.toml file: see below
Issue
An addition in poetry-core between 1.1.0a7 and 1.1.0b1 introduced a bug in package metadata name for namespaced packages:
>>> from importlib import metadata
>>> metadata('mynamespace.foomodule')
importlib.metadata.PackageNotFoundError: mynamespace.foomoduleThe package is instead registered as mynamespace-foomodule. This is a particular problem since importlib.metadata is the canonical way to single-source the __version__
This bug occurs on Python 3.8 and 3.9 (but not 3.10 insterestingly enough). The change that introduced the bug: python-poetry/poetry-core#328. Ever since this change, . is normalized to - in package names, which is incorrect for namespace packages.
To reproduce
pyproject.toml
[tool.poetry]
name = "mynamespace.foomodule"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [
{ include = "mynamespace"},
]
[tool.poetry.dependencies]
python = "^3.8"
[build-system]
requires = ["poetry-core==1.1.0rc1"]
build-backend = "poetry.core.masonry.api"files
mynamespace/
├── __init__.py # contains: __path__ = __import__('pkgutil').extend_path(__path__, __name__)
└── foomodule
└── __init__.py # empty
Then, run:
# make sure you're on python 3.8 or 3.9
pip install .
# this raises an error, but shouldn't
python -c "from importlib import metadata; metadata.version('mynamespace.foomodule')"What to do?
I don't mind submitting a fix in poetry-core, but I'm of course missing a lot of context on why the change was made in the first place.
Activity
[-]Regression in namespace package metadata poetry-core 1.1.0rc1[/-][+]Regression in namespace package metadata on poetry-core 1.1.0rc1[/+]dimbleby commentedon Aug 19, 2022
Would be interested in a link to the python 3.10 MR / discussion around the changed behaviour, if you can dig it out.
My insta-take is that poetry is not wrong here: would be interesting to see whether that upstream discussion is framed as "some callers do it wrong but we can help" or as "oops, we should work with canonicalized package names". I'm hoping for the latter.
Meanwhile, can you use canonicalized package names all along? The examples at https://packaging.python.org/en/latest/guides/packaging-namespace-packages/ use
mynamespace-subpackage-a(notmynamespace.subpackage.a), suggesting that this was always the preferred approach.ariebovenberg commentedon Aug 20, 2022
@dimbleby thanks for the info, digging into the background I now understand the reasoning behind canonicalization, and I can understand heading in that direction.
Good find in the namespace packaging docs! Using
name = "mynamespace-foomodule"inpyproject.tomlbypasses the entire issue.PeterJCLaw commentedon Oct 11, 2022
@dimbleby given the frequency that this issue has appeared and that fact that it ends up being breaking (or very nearly so) for a number of use cases, would you consider adding a warning from
poetry(ideally with a link to an explanation and how to fix) when it does this canonicalisation? Alternatively I would encourage thatpoetryactually just reject these names rather than silently change them in ways that aren't always compatible with previous versions of the same package.It feels like either of those would ensure that the issue is visible when the offending package is built, rather than only once it has already been published and users are trying to use it.
A major version bump for this arguably breaking change might have been worth considering too (though I realise there's not much can be done about that now).
Aside: In my case we actually had issues uploading it to PyPI which seemed to stem from the source distribution keeping the dot, while the wheel didn't (though we weren't able to reproduce that failure mode); then confusion when PyPI had seemed to change the name of the project unexpectedly. In the end we pinned an older version of
poetryand yanked the version of the package with the adjusted name in order to avoid breakage for now. I realise that's not a permanent solution of course.poetryalso seemed to be using underscores rather than dashes in the artefact names, which seems to disagree with canonicalisation to dashes and further confused us.dimbleby commentedon Oct 11, 2022
Doubtful. If you want to offer an MR then I expect it would get proper consideration, but I've no interest in doing this myself.
If you want to be able to upload packages to pypi with dots in their names, please go encourage pypi/warehouse#10030. If and when that is resolved at pypi, it will be straightforward for poetry to follow.
PeterJCLaw commentedon Oct 12, 2022
For clarity: part of what causes confusion for me on this is that PyPI demonstrably does accept distribution files with dots in (e.g:
sr.robot3) when the package name also contains dots.I (now) realise there are PEPs which suggest that package names shouldn't contain dots (though the discussion on the practicalities of that and where those PEPs apply seems far from resolved), however that doesn't change the reality that dots are being used in the wild and this change was breaking for a number of projects.
The fact that PyPI doesn't accept doted package names to have underscored file names seems like a separate issue to me.
I'll have a look at putting together some changes; does poetry have a warnings framework I should be looking to use? From a quick look I can see there's use of the standard library
loggingmodule which seems to line up with the printed output (plus some colourisation I've not found), is that the way to go here?neersighted commentedon Oct 12, 2022
The issue is that PyPI accepts dotted names, but not in a consistent way. We can't programmatically normalize a name to be valid for PyPI and allow dots/capitalization, and we don't want to leak PyPI's idiosyncratic validation into Poetry code. See #1202 for more of this.
Certainly, you're welcome to add a warning when names do get normalized and may surprise the user when they go to reference them -- that being said, I don't think Poetry should force people to use normalized names in their pyproject.toml -- it's just that the artifacts we produce need to be normalized as this causes less problems than not normalizing as we did before (there were a lot of ugly bugs around this).
Clarify dots canonicalisation change is breaking
Clarify dots canonicalisation change is breaking
github-actions commentedon Mar 1, 2024
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.