Package dependencies not installed when private PyPi repository present

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: MacOS Big Sur 11.2.3
Poetry version: 1.1.6

Issue

I encountered an issue with dependencies when using a private repository. I have a pyproject.toml like

[tool.poetry]
name = "my-awesome-package"
version = "0.1.0"
description = "My package."
authors = ["Me"]
packages = [
    { include = "sub_package_1" },
    { include = "sub_package_2" }
]
include = ["path_to_files/**/*.gz"]

[[tool.poetry.source]]
name = 'private_pypi'
url = 'https://mypypi.dom/pypi/simple'

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
build-backend = "poetry.masonry.api"
requires = ["poetry>=1.1.2"]

And I'd like to add a new package, explicitly poetry add rasterio.
When the private repository (which has nothing to do with the desired package) is present in pyproject.toml, only
the package itself gets installed without its dependencies

poetry add rasterio                                                                                                                                                                                                                                                
Using version ^1.2.3 for rasterio

Updating dependencies
Resolving dependencies... (10.2s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing rasterio (1.2.3)

When I remove the private repository part

[[tool.poetry.source]]
name = 'private_pypi'
url = 'https://mypypi.dom/pypi/simple'

and try to add the package, all of its dependencies get installed correctly

poetry add rasterio
Using version ^1.2.3 for rasterio

Updating dependencies
Resolving dependencies... (3.5s)

Writing lock file

Package operations: 6 installs, 0 updates, 0 removals

  • Installing click (7.1.2)
  • Installing affine (2.3.0)
  • Installing click-plugins (1.1.1)
  • Installing cligj (0.7.1)
  • Installing snuggs (1.4.7)
  • Installing rasterio (1.2.3)

It seems that the private repository is somehow interfering with the resolution of the dependencies.
I have tried to recreate both the poetry.lock file and the virtual environment from scratch with no change in this behaviour.

Initially, I have discovered this issue through a transitive dependency (a dependency on the private pypi repository, which I was trying to install, depends on rasterio) when I tried to poetry update, the dependencies affine, click-plugins , snuggs and cligj got uninstalled, so the issue seemed to be related to #3224 and #3684, but the issue can be reproduced with this minimalistic setup.

I have specified the missing dependencies explicitly in the pyproject.toml file for the time being. This however is a suboptimal solution as it bloats pyproject.toml.

Thank you for your help!

See #4439 sounds like it could be your problem

I cannot reproduce this on 1.2.x -- if you are still running into this, please comment back here with a model project that demonstrates the issue/reproduces it for you.

Reproduced with Poetry 1.2.2 / macOS 12.6.1 (21G217) with exactly this package rasterio.

The pyproject.toml:

[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["theirix <theirix@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
rasterio = "1.2.10"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[[tool.poetry.source]]
name = "REDACTED"
url = "REDACTED"

Commands:

% export REPOSITORY_USERNAME=REDACTED
% export REPOSITORY_PASSWORD=REDACTED
% poetry new foobar
% cd foobar
% # add a private PyPi repository (GitLab-based if it matters) to the toml
% poetry env use /usr/local/opt/python@3.9/bin/python3.9
Using virtualenv: /Users/irix/.virtualenvs/foobar-w9rLp08K-py3.9
% poetry add rasterio@1.2.10

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing rasterio (1.2.10)

If I don't use a private repository, all dependencies are installed as expected:

Package operations: 10 installs, 1 update, 0 removals

  • Installing click (8.1.3)
  • Installing numpy (1.23.5)
  • Installing pyparsing (3.0.9)
  • Installing affine (2.3.1)
  • Installing attrs (22.1.0)
  • Installing certifi (2022.9.24)
  • Installing click-plugins (1.1.1)
  • Installing cligj (0.7.2)
  • Updating setuptools (65.5.1 -> 65.6.3)
  • Installing snuggs (1.4.7)
  • Installing rasterio (1.2.10)

@neersighted

@neersighted newest 1.3.1 still has this issue. I think this bug should be reopened.

I'm still unable to reproduce this against devpi. Can you please provide poetry add in verbose mode, as well as the precise command you're using to add a custom source? How are you communicating the credentials to Poetry? They should be added using poetry config.

I decided to reread the documentation about secondary package sources. My scenario was about publishing private packages to the private repository. It is not the same as using the private repository as a full PyPI mirror.

So in my example, I hadn't marked the repository as secondary (no secondary = true in my snippet above). So poetry expects to find all packages in a private repository, even those that weren't published to the secondary repository. After marking it as a secondary, poetry successfully installs a rasterio package with all its dependencies.

Hello,

We are facing the same issue, and I'm not sure anyone posted the -vvv logs before, so I will add it from our side.

I'm using poetry to install tensorflow 2.11.0 from devpi, however, it is resolving the dependencies differently than if the sources was PyPI directly. I can also provide a full reproduce case with docker-compose, as we use docker for both devpi and poetry.

Here is the basic output below when trying to install tensorflow from PyPI vs. devpi. It seems poetry is installing the tensorflow (intel / aws), causing it to detect circular dependencies, and eventually failing silently.

Using the following versions:
Python 3.10.9
Poetry 1.3.2 or 1.4.0
devpi-server 6.8.0

Worth noting:

We use a custom / private index to host internal packages + bases fallback to pypi mirrored index.
Installing using pip works fine.

Source (PyPI): 82 packages found for tensorflow *
Using version ^2.11.0 for tensorflow

Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on tensorflow (^2.11.0)
   1: selecting test (0.1.0)
   1: derived: tensorflow (>=2.11.0,<3.0.0)
Source (PyPI): 1 packages found for tensorflow >=2.11.0,<3.0.0
   1: fact: tensorflow (2.11.0) depends on absl-py (>=1.0.0)
   1: fact: tensorflow (2.11.0) depends on astunparse (>=1.6.0)
   1: fact: tensorflow (2.11.0) depends on flatbuffers (>=2.0)
   1: fact: tensorflow (2.11.0) depends on gast (>=0.2.1,<=0.4.0)
   1: fact: tensorflow (2.11.0) depends on google-pasta (>=0.1.1)
   1: fact: tensorflow (2.11.0) depends on h5py (>=2.9.0)
   1: fact: tensorflow (2.11.0) depends on libclang (>=13.0.0)
   1: fact: tensorflow (2.11.0) depends on numpy (>=1.20)
   1: fact: tensorflow (2.11.0) depends on opt-einsum (>=2.3.2)
   1: fact: tensorflow (2.11.0) depends on packaging (*)
   1: fact: tensorflow (2.11.0) depends on protobuf (>=3.9.2,<3.20)
   1: fact: tensorflow (2.11.0) depends on setuptools (*)
   1: fact: tensorflow (2.11.0) depends on six (>=1.12.0)
   1: fact: tensorflow (2.11.0) depends on termcolor (>=1.1.0)
   1: fact: tensorflow (2.11.0) depends on typing-extensions (>=3.6.6)
   1: fact: tensorflow (2.11.0) depends on wrapt (>=1.11.0)
   1: fact: tensorflow (2.11.0) depends on grpcio (>=1.24.3,<2.0)
   1: fact: tensorflow (2.11.0) depends on tensorboard (>=2.11,<2.12)
   1: fact: tensorflow (2.11.0) depends on tensorflow-estimator (>=2.11.0,<2.12)
   1: fact: tensorflow (2.11.0) depends on keras (>=2.11.0,<2.12)
   1: fact: tensorflow (2.11.0) depends on tensorflow-io-gcs-filesystem (>=0.23.1)
   1: selecting tensorflow (2.11.0)
   1: derived: tensorflow-io-gcs-filesystem (>=0.23.1)
   1: derived: keras (>=2.11.0,<2.12)
   1: derived: tensorflow-estimator (>=2.11.0,<2.12)
   1: derived: tensorboard (>=2.11,<2.12)
   1: derived: grpcio (>=1.24.3,<2.0)
   1: derived: wrapt (>=1.11.0)
   1: derived: typing-extensions (>=3.6.6)
   1: derived: termcolor (>=1.1.0)
   1: derived: six (>=1.12.0)
   1: derived: setuptools
   1: derived: protobuf (>=3.9.2,<3.20)
   1: derived: packaging
   1: derived: opt-einsum (>=2.3.2)
   1: derived: numpy (>=1.20)
   1: derived: libclang (>=13.0.0)
   1: derived: h5py (>=2.9.0)
   1: derived: google-pasta (>=0.1.1)
   1: derived: gast (>=0.2.1,<=0.4.0)
   1: derived: flatbuffers (>=2.0)
   1: derived: astunparse (>=1.6.0)
   1: derived: absl-py (>=1.0.0)

Source (devpi): 82 packages found for tensorflow *
Using version ^2.11.0 for tensorflow

Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on tensorflow (^2.11.0)
   1: selecting test (0.1.0)
   1: derived: tensorflow (>=2.11.0,<3.0.0)
Source (devpi): 1 packages found for tensorflow >=2.11.0,<3.0.0
   1: fact: tensorflow (2.11.0) depends on tensorflow-cpu-aws (2.11.0)
   1: fact: tensorflow (2.11.0) depends on tensorflow-intel (2.11.0)
   1: selecting tensorflow (2.11.0)
   1: derived: tensorflow-intel (==2.11.0)
   1: derived: tensorflow-cpu-aws (==2.11.0)

Problem

I tried to inspect it further today, and noticed that the package definition is somehow different based on the source (PyPI vs devpi). Notice how the requires for tensorflow is empty from PyPI but has two false positives for devpi. Also, it seems the python markers and other constraints are completely missing.

~~Just a hunch, I noticed poetry treats "legacy" source the same way it treats local cache. Perhaps this is bypassing full dependency resolution?~~

Root Cause

The issue is basically this:

poetry/src/poetry/repositories/http_repository.py

Lines 112 to 120 in f27308f

    
           # We ought just to be able to look at any of the available wheels to read 
        
           # metadata, they all should give the same answer. 
        
           # 
        
           # In practice this hasn't always been true. 
        
           # 
        
           # Most of the code in here is to deal with cases such as isort 4.3.4 which 
        
           # published separate python3 and python2 wheels with quite different 
        
           # dependencies.  We try to detect such cases and combine the data from the 
        
           # two wheels into what ought to have been published in the first place...

I stopped the debugger here:

poetry/src/poetry/repositories/http_repository.py

Lines 208 to 210 in f27308f

    
           if platform_specific_wheels: 
        
               first_wheel = platform_specific_wheels[0] 
        
               return self._get_info_from_wheel(first_wheel)

And by getting the metadata from the linux x86 wheel, the requires_dist showed correctly. I think the issue is two fold:

Wheels might have different metadata
Even if the metadata pulled is wrong, poetry seemed to be trying to install all requires_dist regardles of markers.

<poetry.inspection.info.PackageInfo object at 0x7f32163889a0>
special variables:
function variables:
cache_version: None
files: []
name: 'tensorflow'
requires_dist: ['absl-py (>=1.0.0)', 'astunparse (>=1.6.0)', 'flatbuffers (>=2.0)', 'gast (<=0.4.0,>=0.2.1)', 'google-pasta (>=0.1.1)', 'grpcio (<2.0,>=1.24.3)', 'h5py (>=2.9.0)', 'keras (<2.12,>=2.11.0)', 'libclang (>=13.0.0)', 'numpy (>=1.20)', 'opt-einsum (>=2.3.2)', 'packaging', 'protobuf (<3.20,>=3.9.2)', 'setuptools', ...]
requires_python: '>=3.7'
summary: 'TensorFlow is an open source machine learning framework for everyone.'
version: '2.11.0'
yanked: False
_cache_version: None
_find_dist_info: <function PackageInfo._find_dist_info at 0x7f3217285160>
_from_distribution: <bound method PackageInfo._from_distribution of <class 'poetry.inspection.info.PackageInfo'>>
_from_sdist_file: <bound method PackageInfo._from_sdist_file of <class 'poetry.inspection.info.PackageInfo'>>
_get_poetry_package: <function PackageInfo._get_poetry_package at 0x7f3217285310>
_source_reference: None
_source_type: 'file'
_source_url: '/tmp/tmp9ojxm7d8/tensorflow-2.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl'

Debug: Package Info Parsing from Wheel vs. JSON (Legacy vs. PyPI)

This shows the wheel metadata extracted from Legacy index. I wanted to use it as a test case, but I couldn't find the best place in the unit tests to test the LegacyRepository resolving this metadata.

<pkginfo.wheel.Wheel object at 0x7f4fe62dd6d0>
special variables:
function variables:
author: 'Google Inc.'
author_email: 'packages@tensorflow.org'
classifiers: ['Development Status :...ion/Stable', 'Environment :: GPU :...DA :: 11.2', 'Intended Audience ::...Developers', 'Intended Audience :: Education', 'Intended Audience ::...e/Research', 'License :: OSI Appro...re License', 'Programming Language...ython :: 3', 'Programming Language... 3 :: Only', 'Programming Language...on :: 3.10', 'Programming Language...hon :: 3.7', 'Programming Language...hon :: 3.8', 'Programming Language...hon :: 3.9', 'Topic :: Scientific/...ngineering', 'Topic :: Scientific/...telligence', ...]
description: None
description_content_type: 'text/markdown'
download_url: 'https://github.com/tensorflow/tensorflow/tags'
dynamic: ()
filename: '/tmp/tmprx643vvq/tensorflow-2.11.0-cp39-cp39-win_amd64.whl'
home_page: 'https://www.tensorflow.org/'
keywords: 'tensorflow tensor machine learning'
license: 'Apache 2.0'
maintainer: None
maintainer_email: None
metadata_version: '2.1'
name: 'tensorflow'
obsoletes: ()
obsoletes_dist: ()
platforms: ['UNKNOWN']
project_urls: ()
provides: ()
provides_dist: ()
provides_extras: ()
requires: ()
requires_dist: ['tensorflow-cpu-aws (..."aarch64")', 'tensorflow-intel (==... "Windows"']
requires_external: ()
requires_python: '>=3.7'
summary: 'TensorFlow is an open source machine learning framework for everyone.'
supported_platforms: ()
version: '2.11.0'
_getHeaderAttrs: <bound method Distribution._getHeaderAttrs of <pkginfo.wheel.Wheel object at 0x7f4fe62dd6d0>>

On the other hand, this is the package info from PyPI.

info
{'author': 'Google Inc.', 'author_email': 'packages@tensorflow.org', 'bugtrack_url': None, 'classifiers': ['Development Status :...ion/Stable', 'Environment :: GPU :...DA :: 11.2', 'Intended Audience ::...Developers', 'Intended Audience :: Education', 'Intended Audience ::...e/Research', 'License :: OSI Appro...re License', 'Programming Language...ython :: 3', 'Programming Language... 3 :: Only', 'Programming Language...on :: 3.10', ...], 'description': '[![Python](https://i...CENSE).\n\n\n', 'description_content_type': 'text/markdown', 'docs_url': None, 'download_url': 'https://github.com/t...rflow/tags', 'downloads': {'last_day': -1, 'last_month': -1, 'last_week': -1}, 'home_page': 'https://www.tensorflow.org/', 'keywords': 'tensorflow tensor ma...e learning', 'license': 'Apache 2.0', 'maintainer': '', 'maintainer_email': '', ...}
special variables:
function variables:
'author': 'Google Inc.'
'author_email': 'packages@tensorflow.org'
'bugtrack_url': None
'classifiers': ['Development Status :...ion/Stable', 'Environment :: GPU :...DA :: 11.2', 'Intended Audience ::...Developers', 'Intended Audience :: Education', 'Intended Audience ::...e/Research', 'License :: OSI Appro...re License', 'Programming Language...ython :: 3', 'Programming Language... 3 :: Only', 'Programming Language...on :: 3.10', 'Programming Language...hon :: 3.7', 'Programming Language...hon :: 3.8', 'Programming Language...hon :: 3.9', 'Topic :: Scientific/...ngineering', 'Topic :: Scientific/...telligence', ...]
'description': "[![Python](https://img.shields.io/pypi/pyversions/tensorflow.svg?style=plastic)](https://badge.fury.io/py/tensorflow)\n[![PyPI](https://badge.fury.io/py/tensorflow.svg)](https://badge.fury.io/py/tensorflow)\n\nTensorFlow is an open source software library for high performance numerical\ncomputation. Its flexible architecture allows easy deployment of computation\nacross a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters\nof servers to mobile and edge devices.\n\nOriginally developed by researchers and engineers from the Google Brain team\nwithin Google's AI organization, it comes with strong support for machine\nlearning and deep learning and the flexible numerical computation core is used\nacross many other scientific domains. TensorFlow is licensed under [Apache\n2.0](https://github.com/tensorflow/tensorflow/blob/master/LICENSE).\n\n\n"
'description_content_type': 'text/markdown'
'docs_url': None
'download_url': 'https://github.com/tensorflow/tensorflow/tags'
'downloads': {'last_day': -1, 'last_month': -1, 'last_week': -1}
'home_page': 'https://www.tensorflow.org/'
'keywords': 'tensorflow tensor machine learning'
'license': 'Apache 2.0'
'maintainer': ''
'maintainer_email': ''
'name': 'tensorflow'
'package_url': 'https://pypi.org/project/tensorflow/'
'platform': None
'project_url': 'https://pypi.org/project/tensorflow/'
'project_urls': {'Download': 'https://github.com/t...rflow/tags', 'Homepage': 'https://www.tensorflow.org/'}
'release_url': 'https://pypi.org/project/tensorflow/2.11.0/'
'requires_dist': ['absl-py (>=1.0.0)', 'astunparse (>=1.6.0)', 'flatbuffers (>=2.0)', 'gast (<=0.4.0,>=0.2.1)', 'google-pasta (>=0.1.1)', 'h5py (>=2.9.0)', 'libclang (>=13.0.0)', 'numpy (>=1.20)', 'opt-einsum (>=2.3.2)', 'packaging', 'protobuf (<3.20,>=3.9.2)', 'setuptools', 'six (>=1.12.0)', 'termcolor (>=1.1.0)', ...]
'requires_python': '>=3.7'
'summary': 'TensorFlow is an open source machine learning framework for everyone.'
'version': '2.11.0'
'yanked': False
'yanked_reason': None
len(): 26

Require dist are completely different!

Debug: Package object

Here is a cleaned up diff to see the difference between Package objects PyPI vs Legacy + the stack trace for it:
https://editor.mergely.com/zDeiGtTI/

__init__ (...internal\poetry\src\poetry\packages\dependency_package.py:13)
append (...internal\poetry\src\poetry\packages\package_collection.py:33)
__init__ (...internal\poetry\src\poetry\packages\package_collection.py:27)
search_for (...internal\poetry\src\poetry\puzzle\provider.py:345)
_search_for (...internal\poetry\src\poetry\mixology\version_solver.py:60)
_get_min (...internal\poetry\src\poetry\mixology\version_solver.py:396)
_choose_package_version (...internal\poetry\src\poetry\mixology\version_solver.py:409)
solve (...internal\poetry\src\poetry\mixology\version_solver.py:112)
resolve_version (...internal\poetry\src\poetry\mixology\__init__.py:18)
_solve (...internal\poetry\src\poetry\puzzle\solver.py:157)
_solve_in_compatibility_mode (...internal\poetry\src\poetry\puzzle\solver.py:135)
_solve (...internal\poetry\src\poetry\puzzle\solver.py:161)
_solve_in_compatibility_mode (...internal\poetry\src\poetry\puzzle\solver.py:135)
_solve (...internal\poetry\src\poetry\puzzle\solver.py:161)
solve (...internal\poetry\src\poetry\puzzle\solver.py:74)
_do_install (...internal\poetry\src\poetry\installation\installer.py:263)
run (...internal\poetry\src\poetry\installation\installer.py:116)
handle (...internal\poetry\src\poetry\console\commands\add.py:262)
execute (...pypoetry\virtualenvs\poetry-3zbKfmgm-py3.8\lib\python3.8\site-packages\cleo\commands\command.py:62)
run (...pypoetry\virtualenvs\poetry-3zbKfmgm-py3.8\lib\python3.8\site-packages\cleo\commands\base_command.py:119)

Hacky Solution

Here is a hacky solution for the impatient. Only tested on our linux docker image pipeline.

Use at your own risk!

The patch script:

"""
Script to patch poetry to fix fetching certain wheels from legacy repository.
See https://github.com/python-poetry/poetry/issues/4000
"""

# filter wheels func to be appended at the end of the file
FILTER_WHEELS_FUNC = '''
def filter_wheels(wheels):
    """Filter out wheels that are not compatible with the current platform."""
    import sys
    from poetry.utils.env import SystemEnv

    env = SystemEnv(Path(sys.executable))
    markers = env.get_marker_env()

    filtered_wheels = wheels

    platform_wheels = [w for w in filtered_wheels if markers.get("sys_platform", "") in w]
    if platform_wheels:
        filtered_wheels = platform_wheels

    machine_wheels = [w for w in filtered_wheels if markers.get("platform_machine", "") in w]
    if machine_wheels:
        filtered_wheels = machine_wheels

    
    return filtered_wheels
'''

# The function call to filter the wheels
FILTER_WHEELS_CALL = "first_wheel = filter_wheels(platform_specific_wheels)[0]"

# The code to be replaced by the filter_wheels_call
FILTER_WHEELS_CALL_TO_REPLACE="first_wheel = platform_specific_wheels[0]"


def main():
    import sys
    from pathlib import Path

    # get the path to the poetry package
    poetry_path = Path(sys.argv[1]).resolve()

    # search for the target file to be patched
    filename = "http_repository.py"
    target_file = next(poetry_path.glob(f"**/{filename}"))

    # patch it
    with target_file.open("r") as f:
        content = f.read()

    # check if the file is already patched
    if FILTER_WHEELS_CALL in content or FILTER_WHEELS_CALL in content:
        print(f"File {target_file} already patched")
        return
    
    # append the filter_wheels function
    content += "\n\n"
    content += FILTER_WHEELS_FUNC

    # replace the function call
    content = content.replace(FILTER_WHEELS_CALL_TO_REPLACE, FILTER_WHEELS_CALL)

    # write the patched file
    with target_file.open("w") as f:
        f.write(content)
    
    print(f"Successfully patched {target_file}")


if __name__ == "__main__":
    main()

In the Dockerfile:

# patch poetry
# see: https://github.com/python-poetry/poetry/issues/4000
COPY --chown=${USER_ID}:${USER_ID} ./resources/patch_poetry.py /tmp/patch_poetry.py
RUN python /tmp/patch_poetry.py "/home/${USER}/.local/share/pypoetry/venv/lib" \
  && rm /tmp/patch_poetry.py

the assumption that all distributions of a particular version of a package share metadata is unlikely to be dropped from poetry: cross-platform builds bring enough complication without trying to add another dimension (not to mention the performance implications of having to download every distribution separately in case it has different metadata).

this assumption about same-metadata-on-all-distributions is poetry's rather than a spec requirement (though some of us think that it ought to be!) so tensorflow is not "wrong" here: however it is incompatible with the approach that poetry takes.

in tensorflow/tensorflow#58770 I see that tensorflow moved in the direction of using the same metadata everywhere, I think as an explicit concession to poetry, but that work looks at best incomplete.

well, perhaps that shows a willingness that you can use. ie probably your best bet is to make a merge request at tensorflow to complete the job that was started there: to make it true that the same metadata is expressed in all tensorflow distributions, using markers to spell out which bits are relevant on which platforms.

@dimbleby

@dimbleby could we perhaps appease to the “performance implication” part, and suggest the following:

“Downloading the first wheel, which may very well be incompatible, would downgrade performance when the actual wheel needed will probably be downloaded anyway later down the line. Picking the correct wheel from this stage is an optimization, which also happens to have a nice side-effect of fixing legacy repos”.

😄?

it's not an optimization, though, because wheels downloaded at this point aren't cached. So downloading the platform-compatible one at this point wouldn't save a later download.

we discussed this at #6547 (comment) and took the position that it was better to encourage packagers to publish consistent metadata. (Also there was a meaningful performance hit)

I'm not 100% against "prefer a platform-compatible wheel": but cross-platform builds are important to poetry and poetry users, and unless metadata is consistent then it can't achieve that.

I'd definitely recommend that one of you who cares about this at least try submitting an MR to tensorflow!

@dimbleby

EDIT

Just saw the comment on TF issue. My whole assumption was based on the fact that the wheel metadata mistmatch is not part of the standard, but if so, then for sure the best way forward is to fix TF and any other misbehaving packages.

(In the meantime, this plugin might help: https://github.com/Mazyod/poetry-legacy-index)

@dimbleby TF is a downstream dependency for us, and could take 6 months or more to be resolved on our side. Sure, would be good for the long run, but as app developers (sorry for the cynicism) but we care mostly about our stuff working, rather than the ecosystem as a whole.

Regardless though, appreciate your time and thoughts on the subject!

Happened to hit the same thing today with pydantic. I can easily reproduce it with a blank project using the below pyproject.toml. Removing the comments to enable a custom tool.poetry.source makes poetry install ignore all pydantic dependencies (see logs further down).

[tool.poetry]
name = "pydantic-test"
version = "0.1.0"
description = ""
authors = ["Foo <foo@email.com>"]
license = "Proprietary"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
pydantic = "^2.7.0"


#[[tool.poetry.source]]
#name = "r-pypi"
#url = "https://your-own-mirror.company.com/api/pypi/r-pypi-pypi-org/simple"
#priority = "primary"



[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Using PyPi.

Using version ^2.7.0 for pydantic

Updating dependencies
Resolving dependencies... (0.2s)

Package operations: 4 installs, 0 updates, 0 removals

  - Installing typing-extensions (4.11.0)
  - Installing annotated-types (0.6.0)
  - Installing pydantic-core (2.18.1)
  - Installing pydantic (2.7.0)

Writing lock file

Using private repository.

Using version ^2.7.0 for pydantic

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  - Installing pydantic (2.7.0)

Writing lock file

Environment:

poetry 1.8.2.
python 3.12.3.
macOS 14.4.1 (Sonoma).

not the same thing

I think this one should be closed, it seems to have boiled down to "poetry assumes that all distributions of a package provide the same metadata and goes wrong when that is not true" - which is indeed the case, but I think there is very little chance that poetry will change anything on that front.

(packages that don't follow this pattern should be considered not poetry-compatible, and if their maintainers care about that then they should provide consistent metadata.)

Not sure about it "not being the same thing" (definitely the same symptoms) — but managed to fix it by following this comment, which solved it for me.

Essentially:

poetry self lock
poetry self install
rm poetry.lock
poetry install

Seems like a bug of sorts to me.

still not the same thing, but a duplicate of a dozen others (including the one that you found), please stop polluting this unrelated issue

Navigation Menu

Repository navigation

Package dependencies not installed when private PyPi repository present #4000

Issue

Problem

Root Cause

Debug: Package Info Parsing from Wheel vs. JSON (Legacy vs. PyPI)

Debug: Package object

Hacky Solution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# We ought just to be able to look at any of the available wheels to read
	# metadata, they all should give the same answer.
	#
	# In practice this hasn't always been true.
	#
	# Most of the code in here is to deal with cases such as isort 4.3.4 which
	# published separate python3 and python2 wheels with quite different
	# dependencies. We try to detect such cases and combine the data from the
	# two wheels into what ought to have been published in the first place...

	if platform_specific_wheels:
	first_wheel = platform_specific_wheels[0]
	return self._get_info_from_wheel(first_wheel)

Package dependencies not installed when private PyPi repository present #4000

Description

Issue

Activity

Korijn commented on Aug 25, 2021

neersighted commented on Oct 5, 2022

theirix commented on Dec 7, 2022

theirix commented on Dec 21, 2022

neersighted commented on Dec 21, 2022

theirix commented on Jan 19, 2023

mbbyn commented on Mar 5, 2023

mbbyn commented on Mar 6, 2023

Problem

Root Cause

Debug: Package Info Parsing from Wheel vs. JSON (Legacy vs. PyPI)

Debug: Package object

mbbyn commented on Mar 6, 2023

Hacky Solution

dimbleby commented on Mar 6, 2023

Mazyod commented on Mar 6, 2023

dimbleby commented on Mar 6, 2023

mbbyn commented on Mar 7, 2023

nfantone commented on Apr 22, 2024

dimbleby commented on Apr 22, 2024

nfantone commented on Apr 22, 2024

dimbleby commented on Apr 22, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions