Encoding failure is not caught and propagated

@neersighted

Poetry version: 1.2.1 and 1.3.0.dev01
Python version: 3.9.13
OS version and name: Windows 11 Enterprise (build 22000.856)
pyproject.toml:

[tool.poetry]
name = "test-poetry-master"
version = "0.1.0"
description = ""
authors = ["kristang"]
readme = "README.md"

[[tool.poetry.source]]
name = "private-pypi"
url = "https://private/pypi/private-pypi/simple/"
secondary = true

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^7.1.1"
mypy = "^0.982"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I am on the latest stable Poetry version, installed using a recommended method.
I have searched the issues of this repo and believe that this is not a duplicate.
I have consulted the FAQ and blog for any relevant entries or release notes.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Additionally

Discussed the issue with @neersighted as well as @finswimmer and @radoering on the Poetry Discord.

Issue

When adding a secondary, private source and attempting to authenticate via basic-http, poetry silently ignores a UnicodeEncodingError originating from a failed latin1 encoding of a str.

Instead, poetry moves on to the public pypi.org feed and informs the user that the searched package is not available - even if it is available on pypi.org

The error stems from loading in a user/password from Windows Certificate Manager which appears to be corrupted and cannot be encoded with 'latin1'

The source of the encoding error is poetry invoking _basic_auth_str from auth.py in the requests package:
https://github.com/psf/requests/blob/7104ad4b135daab0ed19d8e41bd469874702342b/requests/auth.py#L56-L60

The originating call in poetry is request in utils.authenticator.

poetry/src/poetry/utils/authenticator.py

Lines 213 to 216 in 0ffe91c

    
           if credential.username is not None or credential.password is not None: 
        
               request = requests.auth.HTTPBasicAuth( 
        
                   credential.username or "", credential.password or "" 
        
               )(request)

solve() has a fairly wide try-catch with a base Exception

poetry/src/poetry/mixology/version_solver.py

Lines 110 to 123 in 0ffe91c

    
           try: 
        
               next: str | None = self._root.name 
        
               while next is not None: 
        
                   self._propagate(next) 
        
                   next = self._choose_package_version() 
        
               return self._result() 
        
           except Exception: 
        
               raise 
        
           finally: 
        
               self._log( 
        
                   f"Version solving took {time.time() - start:.3f} seconds.\n" 
        
                   f"Tried {self._solution.attempted_solutions} solutions." 
        
               )

Reproducing

As this involves a broken str that fails to encode reproducibility is a bit hard. I will try and create tests and create a PR if my skills are sufficient enough to create something useful.

Running pipenv lock -vvv yields:

  Stack trace:

  4  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
      155│ 
      156│         try:
    → 157│             result = resolve_version(self._package, self._provider)
      158│ 
      159│             packages = result.packages

  3  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:113 in solve
      111│             next: str | None = self._root.name
      112│             while next is not None:
    → 113│                 self._propagate(next)
      114│                 next = self._choose_package_version()
      115│ 

  1  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:152 in _propagate
      150│                     # where that incompatibility will allow us to derive new assignments
      151│                     # that avoid the conflict.
    → 152│                     root_cause = self._resolve_conflict(incompatibility)
      153│ 
      154│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:351 in _resolve_conflict
      347│             )
      348│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      349│             self._log(f"! thus: {incompatibility}")
      350│ 
    → 351│         raise SolveFailure(incompatibility)
      352│ 
      353│     def _choose_package_version(self) -> str | None:
      354│         """
      355│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

   9  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

   8  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   7  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   5  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  SolverProblemError

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:163 in _solve
      159│             packages = result.packages
      160│         except OverrideNeeded as e:
      161│             return self._solve_in_compatibility_mode(e.overrides)
      162│         except SolveFailure as e:
    → 163│             raise SolverProblemError(e)
      164│ 
      165│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      166│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      167│

No-where in the stacktrace is a user informed that the underlying issue is an encoding failure of credentials.

Pseduo fix

Wrapping requests.auth.HTTPBasicAuth in a try-except can catch the UnicodeEncodeError and propagate it to the user:

if credential.username is not None or credential.password is not None:
    try:
        request = requests.auth.HTTPBasicAuth(
            credential.username or "", credential.password or ""
        )(request)
    except UnicodeEncodeError as e:
        raise PoetryException(e)

The new stacktrace now contains the correct UnicodeEncodeError:

  Stack trace:

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:216 in request
      214│         if credential.username is not None or credential.password is not None:
      215│             try:
    → 216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)

  1  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:95 in __call__
       93│ 
       94│     def __call__(self, r):
    →  95│         r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
       96│         return r
       97│ 

  UnicodeEncodeError

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:60 in _basic_auth_str
       56│     if isinstance(username, str):
       57│         username = username.encode("latin1")
       58│ 
       59│     if isinstance(password, str):
    →  60│         password = password.encode("latin1")
       61│ 
       62│     authstr = "Basic " + to_native_string(
       63│         b64encode(b":".join((username, password))).strip()
       64│     )

The following error occurred when trying to handle this error:


  Stack trace:

  24  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  23  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  22  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  21  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

  20  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  19  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  18  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  17  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

  16  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

  15  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

  14  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  13  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
       155│ 
       156│         try:
     → 157│             result = resolve_version(self._package, self._provider)
       158│ 
       159│             packages = result.packages

  12  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  11  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:114 in solve
       112│             while next is not None:
       113│                 self._propagate(next)
     → 114│                 next = self._choose_package_version()
       115│ 
       116│             return self._result()

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:413 in _choose_package_version
       411│             dependency = unsatisfied[0]
       412│         else:
     → 413│             dependency = min(*unsatisfied, key=_get_min)
       414│ 
       415│         locked = self._provider.get_locked(dependency)

   9  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:399 in _get_min
       397│ 
       398│             try:
     → 399│                 num_packages = len(self._dependency_cache.search_for(dependency))
       400│             except ValueError:
       401│                 num_packages = 0

   8  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61│         if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63│         else:
        64│             packages = [

   7  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\provider.py:316 in search_for
        314│             return PackageCollection(dependency, packages)
        315│ 
     →  316│         packages = self._pool.find_packages(dependency)
        317│ 
        318│         packages.sort(

   6  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\pool.py:136 in find_packages
       134│         packages: list[Package] = []
       135│         for repo in self.repositories:
     → 136│             packages += repo.find_packages(dependency)
       137│         return packages
       138│ 

   5  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\repository.py:42 in find_packages
        40│         ignored_pre_release_packages = []
        41│ 
     →  42│         for package in self._find_packages(dependency.name, constraint):
        43│             if package.yanked and not isinstance(constraint, Version):
        44│                 # PEP 592: yanked files are always ignored, unless they are the only

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:96 in _find_packages
        94│             versions = self._cache.store("matches").get(key)
        95│         else:
     →  96│             page = self._get_page(f"/{name}/")
        97│             if page is None:
        98│                 self._log(

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:150 in _get_page
       148│     def _get_page(self, endpoint: str) -> SimpleRepositoryPage | None:
       149│         try:
     → 150│             response = self._get_response(endpoint)
       151│         except UnicodeEncodeError as e:
       152│             raise e

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\http.py:272 in _get_response
       270│         url = self._url + endpoint
       271│         try:
     → 272│             response: requests.Response = self.session.get(
       273│                 url, raise_for_status=False, timeout=REQUESTS_TIMEOUT
       274│             )

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:274 in get
       272│ 
       273│     def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 274│         return self.request("get", url, **kwargs)
       275│ 
       276│     def post(self, url: str, **kwargs: Any) -> requests.Response:

  PoetryException

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:220 in request
      216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)
      219│             except UnicodeEncodeError as e:
    → 220│                 raise PoetryException(e)
      221│ 
      222│         session = self.get_session(url=url)
      223│         prepared_request = session.prepare_request(request)
      224│

solve() has a fairly wide try-catch with a base Exception

but since it re-raises that exception that's not masking anything, surely? Seems plausible that there's a too-broad-except somewhere, but not that one?

@dimbleby

@dimbleby: Yes, it might be somewhere else. That was just some extra info.

extra, but I think not relevant!

poetry/src/poetry/mixology/version_solver.py

Line 400 in 0ffe91c

except ValueError:

looks like a more plausible candidate to me, or possibly one of the other places where poetry catches ValueError - that would capture a unicode error.

so far as I can see that try-except can simply be removed, which should allow your error to propagate. If you fancy submitting an MR, I reckon that's the way to go

I'm not that experienced in best-practice in terms of changes like this, but it seems a bit dangerous to just remove a try-except like that - but if you feel confident in doing so I have created PR #6786.

search_for should not raise ValueError to indicate that a package cou…

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Navigation Menu

Repository navigation

Encoding failure is not caught and propagated #6784

Additionally

Issue

Reproducing

Pseduo fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if credential.username is not None or credential.password is not None:
	request = requests.auth.HTTPBasicAuth(
	credential.username or "", credential.password or ""
	)(request)

	try:
	next: str \| None = self._root.name
	while next is not None:
	self._propagate(next)
	next = self._choose_package_version()

	return self._result()
	except Exception:
	raise
	finally:
	self._log(
	f"Version solving took {time.time() - start:.3f} seconds.\n"
	f"Tried {self._solution.attempted_solutions} solutions."
	)

Encoding failure is not caught and propagated #6784

Description

Additionally

Issue

Reproducing

Pseduo fix

Activity

dimbleby commented on Oct 12, 2022

kristang commented on Oct 12, 2022

dimbleby commented on Oct 12, 2022

dimbleby commented on Oct 12, 2022

kristang commented on Oct 12, 2022

github-actions commented on Mar 1, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions