"expected string or bytes-like object" on poetry install

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: based on official docker image python:3.8-slim-buster (see below for Dockerfile)
Poetry version: 1.1.4
virtualenv version: 20.4.2
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/wichert/ba05b0505c24ee7c71c5d7deb2ac2cfe

Issue

When I run poetry install in a private GitHub runner on an GHES installation I started getting an error (see further below for full output):

  TypeError

  expected string or bytes-like object

  at /usr/local/lib/python3.8/site-packages/poetry/core/utils/helpers.py:24 in canonicalize_name

Debug output

Creating virtualenv amp-portal in /__w/backend/backend/amp-portal/.venv
Using virtualenv: /__w/backend/backend/amp-portal/.venv

  Stack trace:

  11  /usr/local/lib/python3.8/site-packages/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  10  /usr/local/lib/python3.8/site-packages/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

   9  /usr/local/lib/python3.8/site-packages/clikit/api/command/command.py:163 in _do_handle
       161│         if self._dispatcher and self._dispatcher.has_listeners(PRE_HANDLE):
       162│             event = PreHandleEvent(args, io, self)
     → 163│             self._dispatcher.dispatch(PRE_HANDLE, event)
       164│ 
       165│             if event.is_handled():

   8  /usr/local/lib/python3.8/site-packages/clikit/api/event/event_dispatcher.py:22 in dispatch
        20│ 
        21│         if listeners:
     →  22│             self._do_dispatch(listeners, event_name, event)
        23│ 
        24│         return event

   7  /usr/local/lib/python3.8/site-packages/clikit/api/event/event_dispatcher.py:89 in _do_dispatch
        87│                 break
        88│ 
     →  89│             listener(event, event_name, self)
        90│ 
        91│     def _sort_listeners(self, event_name):  # type: (str) -> None

   6  /usr/local/lib/python3.8/site-packages/poetry/console/config/application_config.py:141 in set_installer
       139│ 
       140│         poetry = command.poetry
     → 141│         installer = Installer(
       142│             event.io,
       143│             command.env,

   5  /usr/local/lib/python3.8/site-packages/poetry/installation/installer.py:65 in __init__
        63│         self._installer = self._get_installer()
        64│         if installed is None:
     →  65│             installed = self._get_installed()
        66│ 
        67│         self._installed_repository = installed

   4  /usr/local/lib/python3.8/site-packages/poetry/installation/installer.py:561 in _get_installed
       559│ 
       560│     def _get_installed(self):  # type: () -> InstalledRepository
     → 561│         return InstalledRepository.load(self._env)
       562│ 

   3  /usr/local/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:118 in load
       116│                 path = Path(str(distribution._path))
       117│                 version = distribution.metadata["version"]
     → 118│                 package = Package(name, version, version)
       119│                 package.description = distribution.metadata.get("summary", "")
       120│ 

   2  /usr/local/lib/python3.8/site-packages/poetry/core/packages/package.py:51 in __init__
        49│         Creates a new in memory package.
        50│         """
     →  51│         super(Package, self).__init__(
        52│             name,
        53│             source_type=source_type,

   1  /usr/local/lib/python3.8/site-packages/poetry/core/packages/specification.py:19 in __init__
        17│     ):  # type: (str, Optional[str], Optional[str], Optional[str], Optional[str], Optional[List[str]]) -> None
        18│         self._pretty_name = name
     →  19│         self._name = canonicalize_name(name)
        20│         self._source_type = source_type
        21│         self._source_url = source_url

  TypeError

  expected string or bytes-like object

  at /usr/local/lib/python3.8/site-packages/poetry/core/utils/helpers.py:24 in canonicalize_name
       20│ _canonicalize_regex = re.compile("[-_]+")
       21│ 
       22│ 
       23│ def canonicalize_name(name):  # type: (str) -> str
    →  24│     return _canonicalize_regex.sub("-", name).lower()
       25│ 
       26│ 
       27│ def module_name(name):  # type: (str) -> str
       28│     return canonicalize_name(name).replace(".", "_").replace("-", "_")

Dockerfile for base docker image

ARG DEBIAN_VERSION=buster
ARG PYTHON_VERSION=3.8
FROM python:${PYTHON_VERSION}-slim-${DEBIAN_VERSION}

ENV \
    DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=true \
    POETRY_NO_INTERACTION=1

RUN set -x \
    && apt-get update \
    && apt-get install -yq --no-install-recommends libpq-dev build-essential sudo \
    && rm -rf /var/lib/apt/lists/*

RUN pip install poetry==1.1.4

There are more similar reports:

We should try to narrow it down, maybe there is a dependency somewhere with packaging issues.

Some extra information:

we use a GHES installation with multiple runners. On some runners it fails every time, on others it always succeeds.
this is not dependent on the local .venv directory: that is newly created on every run
it is not dependent on what is in the users home directory: that is part of the docker container and is identical on all runners.

So there is something very specific to a runner that is independent from the docker image that triggers this error.

I see. This is good info. This changes my perspective on this issue.

Perhaps this could be a broken cache entry somewhere?

I'm adding @netcaptors here; he is a member of the team that manages the github runners and has more insight into their configuration than I have.

@stephsamson

I'm out of my depth on this. I'll ping @stephsamson and @finswimmer maybe they'll be able to give more insight or redirect to other maintainers.

But you seem to say it's rather new behaviour, although there was no new poetry release in a while, right?

This particularly error is new for us today as far as I can tell. I expect that there is something in the environment that poetry needs to deal with that changed and triggered this error.

Could it be something new in the container image?
https://github.com/docker-library/python/commits/2d63cd17c78a868e35479463b8470c0eb3a30096/3.8/buster/slim/Dockerfile
Latest changes seem to be about new pip versions.

Doubtful, we had not done a rebuild of our docker image at the moment this error started occurring as far as I know.

As magically as this error started appearing, it seems to have disappeared today. As far as I know the only change that happened is that chmod -R github-runner:github-runner /home/github-runner/runner/_work/ was run on the runner instance that was showing the error. The best theory I have is that there was a directory or file somewhere that was not readable by the user, which triggered this error. Perhaps a .dist-info directory that was not readable?

I saw this issue locally today on my mac. I used poetry 1.1.4 and it worked for me until today, for some reason, poetry install doesn't work and reported the same error in this issue. Commands like poetry show stops working too with the same error.

I tried revert everything to a specific git commit I confirmed working recently (delete the entire folder and fetch from git again, so it doesn't seem to me a permission issue), but the same error was reported.

I got the same problem. I dropped into PDB and tried to track it down.

name is None and it comes from here:

https://github.com/python-poetry/poetry/blob/1.1.4/poetry/repositories/installed_repository.py#L115

Then if I run this in PDB:
pp [x.metadata["Name"] for x in metadata.distributions(path=[env.sys_path[4]])]

I get for my specific .venv/project I get something like:

[ '...',
 'dbus-python',
 None,
 'pluggy',
 'flake8',
 '...']

Note the None.

Now let's find the culprit:
pp [x.files if x.metadata["name"] is None else "SKIP" for x in metadata.distributions(path=[env.sys_path[4]])]

[ '...',
 'SKIP',
 [PackagePath('more_itertools-8.7.0.dist-info/INSTALLER'),
  PackagePath('more_itertools-8.7.0.dist-info/LICENSE'),
  PackagePath('more_itertools-8.7.0.dist-info/METADATA'),
  PackagePath('more_itertools-8.7.0.dist-info/RECORD'),
  PackagePath('more_itertools-8.7.0.dist-info/REQUESTED'),
  PackagePath('more_itertools-8.7.0.dist-info/WHEEL'),
  PackagePath('more_itertools-8.7.0.dist-info/direct_url.json'),
  PackagePath('more_itertools-8.7.0.dist-info/top_level.txt'),
  PackagePath('more_itertools/__init__.py'),
  PackagePath('more_itertools/__init__.pyi'),
  PackagePath('more_itertools/__pycache__/__init__.cpython-39.pyc'),
  PackagePath('more_itertools/__pycache__/more.cpython-39.pyc'),
  PackagePath('more_itertools/__pycache__/recipes.cpython-39.pyc'),
  PackagePath('more_itertools/more.py'),
  PackagePath('more_itertools/more.pyi'),
  PackagePath('more_itertools/py.typed'),
  PackagePath('more_itertools/recipes.py'),
  PackagePath('more_itertools/recipes.pyi')],
 'SKIP',
 '...']

For some reason PathDistribution object representing more_itertools package has empty metadata and version fields.

Interestingly enough it's also the only dependency of my project dependencies which upgraded when I last ran poetry update. Not sure if that's related, or they broke something.

Hope this helps.

EDIT:
I actually checked the METADATA file in more_itertools-8.7.0.dist-info directory and it's empty.

Extra info:
pip version in venv: 21.0.1
pip version outside: 20.3.3
poetry version: 1.1.4

EDIT 2:
I bet it has something to do with pip version as mentioned before.

EDIT 3:
I ended up having to clear ~/.cache/pypoetry seems like the whole artifact was corrupted. Without that even deleting and reinstalling .venv would crash on poetry install.

The TypeError "expected string or bytes-like object" appears to obscure various kinds of issues people posted here. It'd be great if the actual underlying root causes can be surfaced instead.

In my case, I ran into the issue in the following combination:

create and use a conda env
use poetry install to install package A from source.
use poetry install to install package B from source.

TypeError is encountered in installing package B.

I later realize doing poetry install on multiple packages in a single conda environment is probably problematic, if not outright unsupported. If that's indeed the case, it'd be great if a more explicit error is raised instead.

Still getting this error on Poetry 1.1.13. I have no .egg-info folders anywhere. Deleting my virtual environment is not an option. I have several stubs there that need to stay in place and re-installing my large project every single time this happens is not reasonable.

Still getting this error on Poetry 1.1.13. I have no .egg-info folders anywhere. Deleting my virtual environment is not an option. I have several stubs there that need to stay in place and re-installing my large project every single time this happens is not reasonable.

The fix is in Poetry 1.2b2 -- Poetry 1.1.x is not seeing releases as we work up to a full release of 1.2.

Still getting this error on Poetry 1.1.13. I have no .egg-info folders anywhere. Deleting my virtual environment is not an option. I have several stubs there that need to stay in place and re-installing my large project every single time this happens is not reasonable.

The fix is in Poetry 1.2b2 -- Poetry 1.1.x is not seeing releases as we work up to a full release of 1.2.

Yup. Discovered that after commenting about your poor documentation.

For me, version 1.2.0b2 found multiple kiwisolver .dist-info folders in my site-packages. One for an older version and one for the latest. I had no .egg-info folders anywhere, so I think this was the culprit.

Still getting this error on Poetry 1.1.13. I have no .egg-info folders anywhere. Deleting my virtual environment is not an option. I have several stubs there that need to stay in place and re-installing my large project every single time this happens is not reasonable.

The fix is in Poetry 1.2b2 -- Poetry 1.1.x is not seeing releases as we work up to a full release of 1.2.

Yup. Discovered that after commenting about your poor documentation.

Poetry is free software developed by volunteers without any expectation of warranty. I am sorry that you had to struggle to find the relevant documentation, and I will make a note of improving it before release. Note that we are still in beta for 1.2 and things are therefore still in flux. Please refrain from using an unnecessarily hostile or confrontational tone.

For me, version 1.2.0b2 found multiple kiwisolver .dist-info folders in my site-packages. One for an older version and one for the latest. I had no .egg-info folders anywhere, so I think this was the culprit.

I'm glad you were able to sort the issues out. Please test 1.2 in your environment and open issues for any regressions or deficiencies (like documentation) you may find.

Apologies, my frustration came out and that was unprofessional. I appreciate your help. Thank you for responding so quickly.

I tried using 1.2.0rc1 and it fails a different way so I can't tell if it fixes the name problem. It's in a similar area of code and has the same kind of feel, so I figured I'd at least mention it here since I really don't have time to open a whole new ticket:

  • Updating certifi (2021.10.8 /Users/kentpitman/Library/Caches/pypoetry/artifacts/61/99/d0/9b77a70db9834d3d8281903159311224b92e6ac1cdab98dca7f6367740/certifi-2021.10.8-py2.py3-none-any.whl -> 2022.6.15): Failed

  AssertionError

  

  at fsenv/lib/python3.7/site-packages/poetry/utils/env.py:343 in find_distribution_files_with_name
       339│     ) -> Iterable[Path]:
       340│         for distribution in self.distributions(
       341│             name=distribution_name, writable_only=writable_only
       342│         ):
    →  343│             assert distribution.files is not None
       344│             for file in distribution.files:
       345│                 if file.name == name:
       346│                     yield Path(
       347│                         distribution.locate_file(file),  # type: ignore[no-untyped-call]

That assertion does behave helpfully because it stops everything dead in its tracks. Is it really that fatal? I would think it would be less disruptive to merge lines 343 and 344 as

for file in distribution.files or []:  # sometimes distribution.files is None

possibly adding a log.warning if you want someone to know about the problem? Putting assert in production code seems iffy. I thought that was only for testing.

if you care then raise a new ticket and say how to reproduce, no point in making updates in unrelated and already-closed issues

Definitely do open a new issue if you can reproduce this consistently in a container/VM.

With regard to assert in production code, it's there for typing purposes and is very helpful for catching this kind of bug. It should be impossible for a None to sneak in there according to the intention of the authors of that section of code, and yet it happened -- this is why typing (and typing asserts) are very helpful for writing robust Python code.

I'll try to open a ticket if I get time, but as you probably know, it takes time to do a good ticket that isn't overly complicated. It helps to know that you think this is not a known problem, since that makes the investment in time seem more justified. Thanks.

Still experience this :(

Still experience this :(

This is a closed issue that could refer to a wild assortment of different root causes -- if you are running into this, please narrow it down to a minimal reproduction and open an issue so we can try to understand how to solve it.

Navigation Menu

Repository navigation

"expected string or bytes-like object" on poetry install #3628

Issue

Debug output

Dockerfile for base docker image

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

"expected string or bytes-like object" on poetry install #3628

Description

Issue

Debug output

Dockerfile for base docker image

Activity

sinoroc commented on Feb 1, 2021

sinoroc commented on Feb 1, 2021

wichert commented on Feb 1, 2021

sinoroc commented on Feb 1, 2021

wichert commented on Feb 1, 2021

wichert commented on Feb 1, 2021

sinoroc commented on Feb 1, 2021

wichert commented on Feb 1, 2021

sinoroc commented on Feb 1, 2021

wichert commented on Feb 1, 2021

wichert commented on Feb 2, 2021

niyue commented on Feb 15, 2021

MaikuMori commented on Feb 17, 2021

38 remaining items

orionlee commented on May 20, 2022

adam-grant-hendry commented on Jun 15, 2022

neersighted commented on Jun 15, 2022

adam-grant-hendry commented on Jun 15, 2022

neersighted commented on Jun 15, 2022

adam-grant-hendry commented on Jun 15, 2022

netsettler commented on Aug 23, 2022

dimbleby commented on Aug 23, 2022

neersighted commented on Aug 23, 2022

netsettler commented on Aug 24, 2022

aalok-sathe commented on Sep 2, 2022

neersighted commented on Sep 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions