Skip to content

IncompleteRead while watching #972

Closed
Closed
IncompleteRead while watching#972
Labels
kind/bugCategorizes issue or PR as related to a bug.

Description

@max-rocket-internet
max-rocket-internet
opened on Sep 26, 2019

What happened (please include outputs or screenshots):

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 603, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 668, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 607, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "controller.py", line 21, in <module>
    for event in stream:
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 134, in stream
    for line in iter_resp_lines(resp):
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 47, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 696, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 380, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

What you expected to happen:

The watcher to always recover and continue watching for events.

How to reproduce it (as minimally and precisely as possible):

Tricky. I've only seen it once and it was random. My application just watches a CRD for changes.

Environment:

  • Kubernetes version (kubectl version): v1.12.10-eks
  • Python version (python --version) 3.7.0
  • Python client version (pip list | grep kubernetes) 9.0.0

Duplicate issues:

Identical issues that were close by the bot: #728 #540

Activity

  1. max-rocket-internet
    added
    kind/bugCategorizes issue or PR as related to a bug.
    on Sep 26, 2019

  2. micw523 commented on Oct 1, 2019

    @micw523
    micw523
    on Oct 1, 2019
    Contributor

    There should be a PR in progress: kubernetes-client/python-base#133

  3. jinchihe commented on Dec 19, 2019

    @jinchihe
    jinchihe
    on Dec 19, 2019

    Hey @micw523 How to work around this? Seems the PR still be not merged. Thanks.

  4. micw523 commented on Dec 19, 2019

    @micw523
    micw523
    on Dec 19, 2019
    Contributor

    Hi @jinchihe , @roycaihw might have a better idea about this.

  5. jinchihe commented on Dec 20, 2019

    @jinchihe
    jinchihe
    on Dec 20, 2019

    @roycaihw is there way to work around the issue? except shorten waiting time. Thanks a lot!

  6. jinchihe commented on Dec 30, 2019

    @jinchihe
    jinchihe
    on Dec 30, 2019

    I worked around this via @retrying.retry. :-) but still I think we need the enhancement, thanks.

  7. max-rocket-internet commented on Jan 2, 2020

    @max-rocket-internet
    max-rocket-internet
    on Jan 2, 2020
    Author

    I worked around this via @retrying.retry

    How does that work?

  8. jinchihe commented on Jan 3, 2020

    @jinchihe
    jinchihe
    on Jan 3, 2020

    @max-rocket-internet I added the @retrying.retry(wait_fixed=1000, stop_max_attempt_number=20) for my watching function :-)

  9. fejta-bot commented on Apr 2, 2020

    @fejta-bot
    fejta-bot
    on Apr 2, 2020

    Issues go stale after 90d of inactivity.
    Mark the issue as fresh with /remove-lifecycle stale.
    Stale issues rot after an additional 30d of inactivity and eventually close.

    If this issue is safe to close now please do so with /close.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /lifecycle stale

  10. k8s-ci-robot
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Apr 2, 2020
  11. airshipbot
    added a commit that references this issue on Apr 20, 2020

    Reestablish watch and retry wait for some errors

    6b2c724

  12. fejta-bot commented on May 2, 2020

    @fejta-bot
    fejta-bot
    on May 2, 2020

    Stale issues rot after 30d of inactivity.
    Mark the issue as fresh with /remove-lifecycle rotten.
    Rotten issues close after an additional 30d of inactivity.

    If this issue is safe to close now please do so with /close.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /lifecycle rotten

  13. k8s-ci-robot
    added
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on May 2, 2020

  14. fejta-bot commented on Jun 1, 2020

    @fejta-bot
    fejta-bot
    on Jun 1, 2020

    Rotten issues close after 30d of inactivity.
    Reopen the issue with /reopen.
    Mark the issue as fresh with /remove-lifecycle rotten.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /close

  15. k8s-ci-robot
    closed this as completedon Jun 1, 2020

  16. k8s-ci-robot commented on Jun 1, 2020

    @k8s-ci-robot
    k8s-ci-robot
    on Jun 1, 2020
    Contributor

    @fejta-bot: Closing this issue.

    Details

    In response to this:

    Rotten issues close after 30d of inactivity.
    Reopen the issue with /reopen.
    Mark the issue as fresh with /remove-lifecycle rotten.

    Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
    /close

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

  17. pitwegner commented on Mar 21, 2021

    @pitwegner
    pitwegner
    on Mar 21, 2021

    /reopen
    /remove-lifecycle rotten
    This issue is still present

  18. k8s-ci-robot commented on Mar 21, 2021

    @k8s-ci-robot
    k8s-ci-robot
    on Mar 21, 2021
    Contributor

    @pitwegner: You can't reopen an issue/PR unless you authored it or you are a collaborator.

    Details

    In response to this:

    /reopen
    /remove-lifecycle rotten
    This issue is still present

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

  19. k8s-ci-robot
    removed
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    on Mar 21, 2021

  20. Guanyan1996 commented on Aug 31, 2021

    @Guanyan1996
    Guanyan1996
    on Aug 31, 2021

    This issue is still present

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions