Skip to content

Handle invalidation of corrupt cache files #2674

Closed

Description

@abn
Poetry
Version: 1.1.0a3
Python:  3.8.3

Virtualenv
Python:         3.8.3
Implementation: CPython
Path:           /path/to/project/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /usr

Issue

In cases where the a download fails, invalid or incomplete content downloaded, there seems to be a scenario where the cache file exists but was not cleaned up. Causing any following attempts to fail with an unpack error inside the pip subprocess. The below output is from an issue that happened when the underlying network changed during initial install and the process was killed. Then an install command was issued again, causing the issue.

  EnvCommandError

  Command ['/path/to/project/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', 'file:///home/user/.cache/pypoetry/artifacts/c8/9d/8a/0ac7b9700197ce07d35f1f4db4c2afa79c83149e7ad6fafd4e82aa1143/traitlets-4.3.3-py2.py3-none-any.whl'] errored with the following return code 2, and output: 
  Processing /home/user/.cache/pypoetry/artifacts/c8/9d/8a/0ac7b9700197ce07d35f1f4db4c2afa79c83149e7ad6fafd4e82aa1143/traitlets-4.3.3-py2.py3-none-any.whl
  ERROR: Exception:
  Traceback (most recent call last):
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
      status = self.run(options, args)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/commands/install.py", line 401, in run
      resolver.resolve(requirement_set)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 202, in resolve
      self._resolve_one(requirement_set, req)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 368, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 315, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 191, in prepare_linked_requirement
      unpack_url(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/download.py", line 465, in unpack_url
      unpack_file_url(link, location, download_dir, hashes=hashes)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/download.py", line 429, in unpack_file_url
      unpack_file(from_path, location, content_type)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 249, in unpack_file
      unzip_file(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 114, in unzip_file
      zip = zipfile.ZipFile(zipfp, allowZip64=True)
    File "/usr/lib64/python3.8/zipfile.py", line 1268, in __init__
      self._RealGetContents()
    File "/usr/lib64/python3.8/zipfile.py", line 1335, in _RealGetContents
      raise BadZipFile("File is not a zip file")
  zipfile.BadZipFile: File is not a zip file
  WARNING: You are using pip version 19.3.1; however, version 20.1.1 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env.py:937 in _run
       933│                 output = subprocess.check_output(
       934│                     cmd, stderr=subprocess.STDOUT, **kwargs
       935│                 )
       936│         except CalledProcessError as e:
    →  937│             raise EnvCommandError(e, input=input_)
       938│ 
       939│         return decode(output)
       940│ 
       941│     def execute(self, bin, *args, **kwargs):

Activity

  1. added
    kind/bugSomething isn't working as expected
    status/triageThis issue needs to be triaged
    on Jul 16, 2020
  2. bratao commented on Jul 25, 2020

    @bratao

    Hi @abn , maybe #2388 is related to this bug?
    #2388 is a very important scenario ( using a private pypi) that is failing in multiple machines to me.

  3. abn commented on Jul 25, 2020

    @abn
    MemberAuthor

    Could be part of the reason. However, I suspect that one is more to do with not being able to download the artifact (eg: html error page being downloaded). Fixing this will simply invalidate it and download again, which might again return the bad content.

  4. Eugeny commented on Sep 7, 2020

    @Eugeny

    Also, clearing the cache doesn't remove these corrupted downloads

  5. noritada commented on Oct 18, 2020

    @noritada

    I encountered the same issue and found this issue report.
    In my case, interruption of package installation by signal creates a broken cache file, and subsequent installation trials all failed.

    I worked around by manually downloading the tarball and putting it to the path shown in the error message.

  6. earonesty commented on Dec 20, 2021

    @earonesty

    disabling the experimental new installer "fixes" this, but really just need to download to tmp, validate, and rename later instead of polluting the cache : poetry config experimental.new-installer false

  7. earonesty commented on Dec 20, 2021

    @earonesty

    using this fork, also fixes this for me: #3301

  8. neersighted commented on Oct 11, 2022

    @neersighted
    Member

    Solved in cc3f994 -- we now don't make a file available in the cache until the download is complete.

  9. github-actions commented on Mar 1, 2024

    @github-actions

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  10. locked as resolved and limited conversation to collaborators on Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working as expectedstatus/triageThis issue needs to be triaged

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions