Provide index to pip for installation of sdist build dependencies

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Windows 10, Version 1909
Poetry version: 1.1.3
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/ajacquah/4b6cab23073fb21c03eaa436b911f89d

Issue

Pip as invoked by poetry installs build depdencies from pypi.org instead of custom repository. And since I am running this behind a firewall with no proxy settings this fails. My understanding was that poetry would use default repository for all package actions. Can you please take a look?

To reproduce:

Create new project. $poetry new --src test-pkg
Add a private default source. This source is a mirror of pypi.org + internal packages.
From limited connectivity environment, try to add black dev dependency: poetry add black -D

Result stack trace:

Command C:\Source\test-pkg\.venv\Scripts\pip.exe install --no-deps file:///C:/Users/ajacquah/AppData/Local/pypoetry/Cache/artifacts/8b/b9/b7/d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727/black-20.8b1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\ajacquah\appdata\local\pypoetry\cache\artifacts\8b\b9\b7\d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727\black-20.8b1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Source\rails-test\.venv\Scripts\python.exe' 'C:\Source\rails-test\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
         cwd: None
    Complete output (7 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CB670>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBAF0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBC40>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBEE0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4A3BB0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
    ERROR: No matching distribution found for setuptools>=41.0
    ----------------------------------------
  ERROR: Command errored out with exit status 1: 'C:\Source\test-pkg\.venv\Scripts\python.exe' 'C:\Source\test-pkg\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.


  at c:\python38\lib\site-packages\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):

Thanks for your help!

@ajacquah

@ajacquah can you provide the output of the following please?

poetry add black -D -vvv

# if black is already in the pyproject.toml you can do this too
poetry lock -vvv

@ajacquah

@ajacquah the issue here is that the isolated build environment required to build balck from sdist is not able to be setup as it cannot download setuptools.

Poetry when building wheels from sdist for dependencies (not the project itself) will handover build to pip via pip install /path/to/sdist. This will in turn trigger a PEP 517 isolated build within pip. During this process, it needs to get the dependencies required for the build. This step is handled by pip and not by poetry in this case.

As for why the sdist is being used, you will have to check if proxy has the wheels available. If not, that could explain why an sdist is downloaded and used.

Eventually we would want to consume this part of the pipe too, and eventually setup the build environment ourselves. Once that happens this should, and a lot of other annoyances, go away.

At initial glance, I reckon your proposal is reasonable and should not impact other aspects as only assist and path cases require the isolated build environment.

Can you maybe verify that doing so manually works in your environment?

It seems this is already fixed in 1.1.4 release.

Faced a similar problem, but this problem is not present on version 1.0.10.

@abn

hello @abn and maintainers. Any further thoughts on this? I've seen a few suggestion for work-arounds on related issues but it would be great to see this working for the configured default package index.

I haven't found an effective workaround; plus I'm having the opposite problem: it's failing to find pypi.org dependencies in my custom secondary repository. I'm not sure if I should open a separate bug for it.

now that I think about it I need to go and look if we make the source for a given package available to the PEP 517 build backend fetching process

IIRC, we will use the sources from pyproject.toml of the root project. Relevant lines of code:

poetry/src/poetry/installation/chef.py

Line 116 in c8945eb

env = IsolatedEnv(venv, self._config)

poetry/src/poetry/installation/chef.py

Line 75 in c8945eb

pool = Factory.create_pool(self._config)

Hi guys
Can you communicate on an expected release date for 1.4
I have the same issue in my new company where I can't update the global pip config (/etc/pip.conf yeah don't laugh)
Need to consider my options quickly about this please

just for information even setting PIP_INDEX_URL as root doesn't fix the issue for non-root user on 1.3.2

same issue in 1.3.2, run install command from EnvCommandError log

root@1c15287fd90a:/develop# '/develop/.venv/bin/python'  '-m'  'pip'  'install'  '--use-pep517'  '--disable-pip-version-check'  '--isolated'  '--no-input'  '--prefix'  '/develop/.venv'  '--no-deps'  '/root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz'
Processing /root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [86 lines of output]
      Collecting cython
        WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0xffffbbeca350>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/be/f9/198c7b9d125b29bcedaf923f5f2c29bd4b4534d4a8dd2ee5780244337af4/Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl
        WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0xffffbbeca650>, 'Connection to files.pythonhosted.org timed out. (connect timeout=15)')': /packages/be/f9/198c7b9d125b29bcedaf923f5f2c29bd4b4534d4a8dd2ee5780244337af4/Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl
        Downloading Cython-0.29.33-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB)
                                                    0.0/1.8 MB ? eta -:--:--
      ERROR: Exception:
      Traceback (most recent call last):
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
          yield
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
          data = self._fp_read(amt) if not fp_closed else b""
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
          return self._fp.read(amt) if amt is not None else self._fp.read()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
          data = self.__fp.read(amt)
        File "/usr/local/lib/python3.10/http/client.py", line 465, in read
          s = self.fp.read(amt)
        File "/usr/local/lib/python3.10/socket.py", line 705, in readinto
          return self._sock.recv_into(b)
        File "/usr/local/lib/python3.10/ssl.py", line 1274, in recv_into
          return self.read(nbytes, buffer)
        File "/usr/local/lib/python3.10/ssl.py", line 1130, in read
          return self._sslobj.read(len, buffer)
      TimeoutError: The read operation timed out

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
          status = run_func(*args)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 415, in run
          requirement_set = resolver.resolve(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
          result = self._result = resolver.resolve(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
          state = resolution.resolve(requirements, max_rounds=max_rounds)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
          self._add_to_criteria(self.state.criteria, r, parent=None)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
          if not criterion.candidates:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
          return bool(self._sequence)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
          return any(self)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
          return (c for c in iterator if id(c) not in self._incompatible_ids)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
          candidate = func()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
          self._link_candidate_cache[link] = LinkCandidate(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
          super().__init__(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
          self.dist = self._prepare()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
          dist = self._prepare_distribution()
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
          return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
          return self._prepare_linked_requirement(req, parallel_builds)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement
          local_file = unpack_url(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url
          file = get_http_url(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url
          from_path, content_type = download(link, temp_dir.path)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/network/download.py", line 147, in __call__
          for chunk in chunks:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
          for chunk in iterable:
        File "/develop/.venv/lib/python3.10/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
          for chunk in response.raw.stream(
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream
          data = self.read(amt=amt, decode_content=decode_content)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
          with self._error_catcher():
        File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
          self.gen.throw(typ, value, traceback)
        File "/develop/.venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
          raise ReadTimeoutError(self._pool, None, "Read timed out.")
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

      [notice] A new release of pip is available: 23.0 -> 23.0.1
      [notice] To update, run: python -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

but, i run command again with -i, success

root@1c15287fd90a:/develop# '/develop/.venv/bin/python'  '-m'  'pip'  'install'  '--use-pep517'  '--disable-pip-version-check'  '--isolated'  '--no-input'  '--prefix'  '/develop/.venv'  '--no-deps'  '/root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz' -i https://pypi.tuna.tsinghua.edu.cn/simple
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing /root/.cache/pypoetry/artifacts/45/a5/fb/6083820494451489aab8af4f89f9a7ee683bac6b5213cd7f9e3119671b/tinyaes-1.0.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: tinyaes
  Building wheel for tinyaes (pyproject.toml) ... done
  Created wheel for tinyaes: filename=tinyaes-1.0.4-cp310-cp310-linux_aarch64.whl size=69242 sha256=441eed9d11bee252d488affb3e555e1d42ff066b3cd6541c730ef156353b1f63
  Stored in directory: /root/.cache/pip/wheels/61/f2/1f/f107e37fb4cfde950ba8696a16ab4bb6425f37e673e872940e
Successfully built tinyaes
Installing collected packages: tinyaes
Successfully installed tinyaes-1.0.4

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said

pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/

As root or with non root user?

poetry 1.4 is out, I guess this should be closed

@qkponton

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/
As root or with non root user?

root.this will change default pip index for every user no matter whether is it root. @qkponton

1.4 release solved the issue for me

@qkponton

I have the same issue but solved by change global pip config /etc/pip.conf as neersighted said
pip config set global.index-url <index-url>
cp ~/.config/pip/pip.conf /etc/
As root or with non root user?
root.this will change default pip index for every user no matter whether is it root. @qkponton

@YAGregor yeah the issue was with a non root user :)

I can confirm that this issue appears to be resolved for our case. Poetry appears to now be respecting the Artifactory instance we've defined in our pyproject.toml file.

For anyone that needs it, this is how we configured Artifactory in our pyproject.toml file.

[[tool.poetry.source]]
name = "artifactory"
url = "https://<your-artifactory-instance>/artifactory/api/pypi/<your-repository>/simple"
default = true

Interestingly, it appears that our internal certificates are not an issue, even without defining our PEM file in the config. I'm running on Windows so not sure if Poetry is using the certs in the Windows cert store or just ignoring SSL validation all together?

Closing, as the issue seems to be fixed. If you encounter this in new versions of Poetry, please open new issue.

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Navigation Menu

Repository navigation

Provide index to pip for installation of sdist build dependencies #3249

Issue

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Provide index to pip for installation of sdist build dependencies #3249

Description

Issue

Activity

abn commented on Oct 19, 2020

abn commented on Oct 20, 2020

abn commented on Oct 20, 2020

qiuwei commented on Oct 27, 2020

Woklex commented on Dec 31, 2020

aja-intel commented on Mar 29, 2021

Liz4v commented on Aug 10, 2021

23 remaining items

radoering commented on Feb 22, 2023

qkponton commented on Feb 23, 2023

qkponton commented on Feb 24, 2023

idhyt commented on Feb 27, 2023

YAGregor commented on Feb 27, 2023

qkponton commented on Mar 1, 2023

dimbleby commented on Mar 1, 2023

YAGregor commented on Mar 2, 2023

qkponton commented on Mar 2, 2023

qkponton commented on Mar 2, 2023

michaelishri commented on Mar 2, 2023

Secrus commented on Apr 18, 2023

github-actions commented on Feb 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions