Skip to content

Regression in 1.2.x when installing fasttext - can't compile due to not finding pybind and/or pip #6113

New issue
New issue
Closed as not planned
Closed as not planned
Regression in 1.2.x when installing fasttext - can't compile due to not finding pybind and/or pip#6113
Labels
status/external-issueIssue is caused by external project (platform, dep, etc)

Description

@cfculhane
cfculhane
opened on Aug 4, 2022

Issue

I've been enjoying the speed and other improvements to the 1.2.x betas, however one package is not installing, and it installs fine on 1.1.14 - fasttext.

Below is the log when installing in 1.2.0b3 with poetry install -vvv

(poetrybug-py3.8)  ~/repos/poetrybug      poetry install -vvv                                                                                                                              
Loading configuration file /home/chris/.config/pypoetry/config.toml
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 3 skipped

  • Installing fasttext (0.9.2): Pending...
  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2): Failed

  Stack trace:

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1454 in _run
      1452│                 )
      1453│             else:
    → 1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )

  1  ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:415 in check_output
       413│         kwargs['input'] = empty
       414│ 
    →  415│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       416│                **kwargs).stdout
       417│ 

  CalledProcessError

  Command '['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:49 in pip_install
       47│ 
       48│     try:
    →  49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
       51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1417 in run_pip
      1415│         pip = self.get_pip_command(embedded=True)
      1416│         cmd = pip + list(args)
    → 1417│         return self._run(cmd, **kwargs)
      1418│ 
      1419│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1707 in _run
      1705│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1706│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1707│         return super()._run(cmd, **kwargs)
      1708│ 
      1709│     def get_temp_environ(

  EnvCommandError

  Command ['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python: No module named pip
        Traceback (most recent call last):
          File "<string>", line 38, in __init__
        ModuleNotFoundError: No module named 'pybind11'
        
        During handling of the above exception, another exception occurred:
        
        Traceback (most recent call last):
          File "/tmp/tmpoxq8b58i", line 363, in <module>
            main()
          File "/tmp/tmpoxq8b58i", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/tmp/tmpoxq8b58i", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
            return self._get_build_requires(
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
            self.run_setup()
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 281, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup
            exec(code, locals())
          File "<string>", line 72, in <module>
          File "<string>", line 41, in __init__
        RuntimeError: pybind11 install failed.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1458 in _run
      1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )
      1457│         except CalledProcessError as e:
    → 1458│             raise EnvCommandError(e, input=input_)
      1459│ 
      1460│         return decode(output)
      1461│ 
      1462│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  4  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:437 in _execute_install
      435│ 
      436│     def _execute_install(self, operation: Install | Update) -> int:
    → 437│         status_code = self._install(operation)
      438│ 
      439│         self._save_url_reference(operation)

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:480 in _install
      478│         )
      479│         self._write(operation, message)
    → 480│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      481│ 
      482│     def _update(self, operation: Install | Update) -> int:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:119 in pip_install
      117│     ) -> int:
      118│         try:
    → 119│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      120│         except EnvCommandError as e:
      121│             output = decode(e.e.output)

  PoetryException

  Failed to install /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed

And the log when the same is run on 1.1.14:

 ~/repos/poetrybug2      poetry add fasttext -vvv                                                                                                                                          
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug2-KxUlQ7WG-py3.8
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 33 packages found for fasttext *
Using version ^0.9.2 for fasttext

Updating dependencies
Resolving dependencies...
   1: fact: poetrybug2 is 0.1.0
   1: derived: poetrybug2
   1: fact: poetrybug2 depends on fasttext (^0.9.2)
   1: selecting poetrybug2 (0.1.0)
   1: derived: fasttext (>=0.9.2,<0.10.0)
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 1 packages found for fasttext >=0.9.2,<0.10.0
   1: fact: fasttext (0.9.2) depends on pybind11 (>=2.2)
   1: fact: fasttext (0.9.2) depends on numpy (*)
   1: selecting fasttext (0.9.2)
   1: derived: numpy
   1: derived: pybind11 (>=2.2)
PyPI: No release information found for numpy-0.9.6, skipping
PyPI: No release information found for numpy-0.9.8, skipping
PyPI: No release information found for numpy-1.0.3, skipping
PyPI: No release information found for numpy-1.0.4, skipping
PyPI: No release information found for numpy-1.0b1, skipping
PyPI: No release information found for numpy-1.0b4, skipping
PyPI: No release information found for numpy-1.0b5, skipping
PyPI: No release information found for numpy-1.0rc1, skipping
PyPI: No release information found for numpy-1.0rc2, skipping
PyPI: No release information found for numpy-1.0rc3, skipping
PyPI: No release information found for numpy-1.1.1, skipping
PyPI: No release information found for numpy-1.2.0, skipping
PyPI: No release information found for numpy-1.2.1, skipping
PyPI: No release information found for numpy-1.4.0, skipping
PyPI: 87 packages found for numpy *
PyPI: No release information found for pybind11-1.6, skipping
PyPI: 22 packages found for pybind11 >=2.2
   1: selecting pybind11 (2.10.0)
PyPI: Getting info for numpy (1.23.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.23.1.tar.gz
   1: selecting numpy (1.23.1)
   1: Version solving took 5.889 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 3 installs, 0 updates, 0 removals

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Installing...
  • Installing numpy (1.23.1)
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Installing...
  • Installing pybind11 (2.10.0)
  • Installing fasttext (0.9.2): Pending...
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2)

Tried a few things:

  • If I start a fresh project with 1.1.14 and add/install fasttext -> no error
  • If I start a fresh project with 1.2.0b3 and add/install fasttext -> fails
  • If I start with 1.1.14, then update to 1.2.0b3, and add/install fasttextt -> fails
  • If I start with 1.2.0n3, then downgrade to 1.1.14, and add/install fasttext -> fails

Seems to be something with the temp env created for the build? Not quite sure, and it installs totally fine with pip.

Activity

  1. cfculhane
    added
    kind/bugSomething isn't working as expected
    status/triageThis issue needs to be triaged
    on Aug 4, 2022

  2. dimbleby commented on Aug 4, 2022

    @dimbleby
    dimbleby
    on Aug 4, 2022
    Contributor

    seems to have been broken at #3835, which prevents the fasttext install process from finding the already-installed (and needed) pybind11

  3. dimbleby commented on Aug 4, 2022

    @dimbleby
    dimbleby
    on Aug 4, 2022
    Contributor

    However fasttext is probably Doing It Wrong by not declaring pybind11 as a build-time requirement.

    Probably you can and should fix this over there by pointing them at https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#build-system-requirement and / or submitting an MR.

  4. dimbleby commented on Aug 4, 2022

    @dimbleby
    dimbleby
    on Aug 4, 2022
    Contributor

    I see that facebookresearch/fastText#1223 has been open for a while so I expect that you'll have to submit an MR if you want this fixed, rather than hoping that they take it on themselves.

  5. dimbleby commented on Aug 5, 2022

    @dimbleby
    dimbleby
    on Aug 5, 2022
    Contributor

    I just noticed facebookresearch/fastText#1270, not clear whether this is blocked on no-one being interested or on the contributor signing the CLA.

    Anyway poetry is not at fault here, this can be closed.

  6. mkniewallner
    added
    status/external-issueIssue is caused by external project (platform, dep, etc)
    and removed
    kind/bugSomething isn't working as expected
    status/triageThis issue needs to be triaged
    on Aug 5, 2022
  7. mkniewallner
    closed this as not plannedon Aug 5, 2022

  8. cfculhane commented on Aug 8, 2022

    @cfculhane
    cfculhane
    on Aug 8, 2022
    Author

    Thanks for looking into this - I will investigate a fix and submit a MR, apologies for taking the blame to poetry!

  9. juhoinkinen commented on Sep 2, 2022

    @juhoinkinen
    juhoinkinen
    on Sep 2, 2022

    While waiting the issue to be fixed on fastText side, Poetry can install fastText using the prebuilt wheels available from fasttext-wheel PyPI project.

  10. victorphamdeveloper commented on Oct 28, 2022

    @victorphamdeveloper
    victorphamdeveloper
    on Oct 28, 2022

    @juhoinkinen I tried to add fasttext-wheel but poetry keeps complaining it cannot find any versions matching 0.9.2 for fasttext-wheel

  11. thesofakillers commented on Jan 16, 2023

    @thesofakillers
    thesofakillers
    on Jan 16, 2023

    wait so how did this issue get resolved? Poetry still fails when running poetry add fasttext

  12. Secrus commented on Jan 16, 2023

    @Secrus
    Secrus
    on Jan 16, 2023
    Member

    wait so how did this issue get resolved? Poetry still fails when running poetry add fasttext

    Yes, but Poetry isn't at fault here, fasttext doesn't provide Poetry with enough info on what built dependencies they need. See PRs mentioned above.

  13. thesofakillers commented on Jan 16, 2023

    @thesofakillers
    thesofakillers
    on Jan 16, 2023

    Thanks @Secrus ! For anyone interested I think the clearest PR to follow is facebookresearch/fastText#1292. Seems like fasttext is pretty dead.

    as mentioned on there, for anyone looking for a quick workaround:

    poetry add git+https://github.com/cfculhane/fastText

    or what i did (ugly)

    pip install fasttext
    poetry add fasttext

  14. thesofakillers
    added a commit that references this issue on Jan 16, 2023

    update wechsel to 0.0.4

    f2010fb
  15. facebook-github-bot
    added a commit that references this issue on Nov 27, 2023

    Add pyproject.toml to comply with PEP-518 (#1292)

    de458dd
  16. vstakhov
    added a commit that references this issue on Feb 19, 2024

    Add pyproject.toml to comply with PEP-518 (facebookresearch#1292)

    42c560f

  17. github-actions commented on Feb 29, 2024

    @github-actions
    github-actionsBot
    on Feb 29, 2024 – with GitHub Actions

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  18. github-actions
    locked as resolved and limited conversation to collaborators on Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status/external-issueIssue is caused by external project (platform, dep, etc)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions