Skip to content

Poetry not pulling git submodules when installing scipy from source #4629

New issue
New issue
Closed
#5428
Closed
Poetry not pulling git submodules when installing scipy from source#4629
#5428
Labels
kind/bugSomething isn't working as expected

Description

@JPFrancoia
JPFrancoia
opened on Oct 12, 2021

  • I am on the latest Poetry version.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

  • OS version and name: Docker

  • Poetry version: 1.1.11

Issue

This isn't about installing scipy, I have no problem installing this dependency when it's the latest version from Pypi (i.e: when the wheel is already built).

When installing scipy from source, I'm running into a problem that seems to be because poetry didn't do a git submodule update --init when cloning the source.

I will provide minimal Docker files to reproduce the issue.

This is the baseline, where I install a branch of scipy from source. This works:

FROM python:3.8

RUN apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev

COPY . /app

WORKDIR app/

RUN pip install poetry

RUN pip install git+https://github.com/mckib2/scipy@boost-ibeta-derivative-overflow

However, when I try this instead:

FROM python:3.8

RUN apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev

COPY . /app

WORKDIR app/

RUN pip install poetry

RUN poetry config virtualenvs.create false \
  && pip install wheel \
  && poetry install -vv -n --no-dev

I end up with the following error:

Traceback (most recent call last):
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
#9 785.8           main()
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
#9 785.8           json_out['return_val'] = hook(**hook_input['kwargs'])
#9 785.8         File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 151, in prepare_metadata_for_build_wheel
#9 785.8           return hook(metadata_directory, config_settings)
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 166, in prepare_metadata_for_build_wheel
#9 785.8           self.run_setup()
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
#9 785.8           super(_BuildMetaLegacyBackend,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
#9 785.8           exec(compile(code, __file__, 'exec'), locals())
#9 785.8         File "setup.py", line 622, in <module>
#9 785.8           setup_package()
#9 785.8         File "setup.py", line 618, in setup_package
#9 785.8           setup(**metadata)
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/core.py", line 137, in setup
#9 785.8           config = configuration()
#9 785.8         File "setup.py", line 528, in configuration
#9 785.8           config.add_subpackage('scipy')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/setup.py", line 20, in configuration
#9 785.8           config.add_subpackage('sparse')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/setup.py", line 14, in configuration
#9 785.8           config.add_subpackage('linalg')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/linalg/setup.py", line 14, in configuration
#9 785.8           config.add_subpackage('_propack')
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
#9 785.8           config_list = self.get_subpackage(subpackage_name, subpackage_path,
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
#9 785.8           config = self._get_configuration_from_setup_py(
#9 785.8         File "/tmp/pip-build-env-esb62brk/overlay/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
#9 785.8           config = setup_module.configuration(*args)
#9 785.8         File "scipy/sparse/linalg/_propack/setup.py", line 36, in configuration
#9 785.8           check_propack_submodule()
#9 785.8         File "scipy/sparse/linalg/_propack/setup.py", line 13, in check_propack_submodule
#9 785.8           raise RuntimeError("Missing the `PROPACK` submodule! Run "
#9 785.8       RuntimeError: Missing the `PROPACK` submodule! Run `git submodule update --init` to fix this.
#9 785.8       ----------------------------------------
#9 785.8   WARNING: Discarding file:///usr/local/src/scipy. Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpk70ej4xc Check the logs for full command output.
#9 785.8   ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpk70ej4xc Check the logs for full command output.
#9 785.8   WARNING: You are using pip version 21.2.4; however, version 21.3 is available.
#9 785.8   You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#9 785.8
#9 785.8
#9 785.8   at /usr/local/lib/python3.8/site-packages/poetry/utils/env.py:1183 in _run
#9 785.9       1179│                 output = subprocess.check_output(
#9 785.9       1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
#9 785.9       1181│                 )
#9 785.9       1182│         except CalledProcessError as e:
#9 785.9     → 1183│             raise EnvCommandError(e, input=input_)
#9 785.9       1184│
#9 785.9       1185│         return decode(output)
#9 785.9       1186│
#9 785.9       1187│     def execute(self, bin, *args, **kwargs):
#9 785.9
------
executor failed running [/bin/sh -c poetry config virtualenvs.create false   && pip install wheel   && poetry install -vv -n --no-dev]: exit code: 1

And here is the pyproject.toml that I used:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = ">=3.8,<3.10"
scipy = { git = "https://github.com/mckib2/scipy.git", branch = "boost-ibeta-derivative-overflow" }
# scipy = { git = "ssh://git@github.com/mckib2/scipy.git", branch = "boost-ibeta-derivative-overflow" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["setuptools", "poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Do you have any idea what could be causing this issue?

Activity

  1. JPFrancoia
    added
    kind/bugSomething isn't working as expected
    status/triageThis issue needs to be triaged
    on Oct 12, 2021

  2. abn commented on Apr 8, 2022

    @abn
    abn
    on Apr 8, 2022
    Member

    Can you please try the fix at #5428.

    Using pipx

    pipx install --suffix=@5428 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/5428/head'

    Using a container (podman | docker)

    podman run --rm -i --entrypoint bash docker.io/python:3.8 <<EOF
set -xe
apt-get update \
    && apt-get install -y make gcc g++ curl libpq-dev python3-dev libpng-dev libxft-dev gfortran python3-dev libopenblas-dev liblapack-dev
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/5428/head
poetry new foobar
pushd foobar
poetry add git+https://github.com/mckib2/scipy#boost-ibeta-derivative-overflow
EOF
    Created package foobar in foobar
/foobar /
Creating virtualenv foobar-lWDpn5M1-py3.8 in /root/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (14.6s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing numpy (1.22.3)
  • Installing scipy (1.8.0.dev0+0.1d1ccf6 1d1ccf6)
  3. abn
    closed this as completedin #5428on May 2, 2022
  4. mkniewallner
    removed
    status/triageThis issue needs to be triaged
    on Jun 11, 2022

  5. github-actions commented on Mar 1, 2024

    @github-actions
    github-actionsBot
    on Mar 1, 2024 – with GitHub Actions

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  6. github-actions
    locked as resolved and limited conversation to collaborators on Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions