This repository was archived by the owner on Mar 13, 2022. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 13, 2022. It is now read-only.
Watch stream should handle HTTP error before unmarshaling event #57
Description
I could be mistaken but looking at the infinite loop for the watch stream doesn't handle the case when you receive an event that is expired i.e. a HTTP status code of 410.
while True:
resp = func(*args, **kwargs)
try:
for line in iter_resp_lines(resp):
yield self.unmarshal_event(line, return_type)
if self._stop:
break
finally:
kwargs['resource_version'] = self.resource_version
resp.close()
resp.release_conn()Looking at the code it seems that if the event is expired then resp should return something along the lines of
{'raw_object': {u'status': u'Failure', u'kind': u'Status', u'code': 410, u'apiVersion': u'v1', u'reason': u'Gone', u'message': u'too old resource version: 2428 (88826)', u'metadata': {}}, u'object': {'api_version': 'v1',
'kind': 'Status',
And unmarshall_event should fail to deserialize the object and break. And self.resource_version should just be stuck on the resource_version of the event that was expired.
Am I missing something here?
Activity
[-]Does the Watch stream handles expired Events[/-][+]Does the Watch stream handle expired Events?[/+]Sturgelose commentedon May 16, 2018
We are having exactly the issue the you described: our watch gets a 401, it disconnects and reconnects automatically ending in an undesired status.
Our case is:
resource_version = Nonein its body (as you can see in the snippet in this issue: K8s Watch: How to deal with 401 Status Errors python#403). Reason of the error is that we are doing many calls to the API.watchhandles the event and restarts the connection. However, it savesresource_versionvalue to the object, making it having a None value.resource_versionin the request has a value of None and this is not allowed in K8s API.Possible solutions:
unmarshall_eventshould validate the status code before saving the state to the object, so an invalid state is not stored in the watch object.Stack trace:
Note that the only place where resource_version is managed inside of the
watchlogic is inwatch.py. In the rest/http logic, this field is not touched.Which should be the option to fix this? I'd be willing to give a PR to fix the issue.
roycaihw commentedon Jun 13, 2018
The current watch client code is dynamic and assumes that API server always responds with correct API object to deserialize. When API server responds with
410in case where resource version is too old (e.g. kubernetes-client/python#484 (comment)) or401in @Sturgelose's case, the errors are not raised/handled but get "deserialized".I would suggest that we should check if the response is error code before deserializing, and reason about the proper behavior for different error code.
[-]Does the Watch stream handle expired Events?[/-][+]Watch stream should handle HTTP error before unmarshaling event[/+]rocky4570 commentedon Sep 24, 2018
Yeah we have the same issue with timeout from AWS ELB, resulting in "Invalid value for
spec, must not beNone".Was going to put in PR for unmarshal_event to raise TimeoutError so as to avoid too many changes affecting existing use cases:
Line 82 or kubernetes/watch/watch.py
if js.get('type') == 'Error':
if js.get('object', {}).get('reason') == 'Expired':
raise TimeoutError(js{'object']['message'])
so that any logging is a little more clearer than potentially looking like its an issue elsewhere depending on what is being watched.
Handle error events in watch
Handle error events in watch
Handle error events in watch
Handle error events in watch
mitar commentedon Feb 11, 2019
From API concepts:
It would be great if Python code could correctly support this automatically.
mitar commentedon Feb 11, 2019
Based on #102 I made an improved version:
resource_versionfrom the latest event.This is almost what API concepts describe with one important difference: API concepts and go library provide a function called "list and watch" which ties listing and watching together. This package (to my knowledge) does not provide that. Because of that we have to throw an exception if the latest event is too old. Otherwise we could just retry the whole "list and watch".
The problem with current approach (in my branch) is that if code was not able to resume automatically (because last event was too old) then just restarting watch with
resource_versionset toNoneis not good enough. It could happen that you just missed some event. In some cases this might be OK (if you do not need to capture all changes). But to my understanding, to not miss any event you should then do "list and watch" to recover. And this currently has to be done manually on the outside. We might want to provide such utility function covering all (edge) cases.zw0610 commentedon Apr 10, 2019
Is there any reason not adding
exceptclause in the while loop withinstreammethod, as:mitar commentedon Apr 10, 2019
I do not think so. The only reason why exception is thrown there is because we access wrong fields in the
dataobject. Also, what you made as a snippet above eats all and every exception, this is definitely not something you want to do.3 remaining items
mitar commentedon Sep 27, 2019
/remove-lifecycle stale
fejta-bot commentedon Dec 26, 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
mitar commentedon Dec 26, 2019
/remove-lifecycle stale
fejta-bot commentedon Mar 25, 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
mitar commentedon Mar 25, 2020
/remove-lifecycle stale
fejta-bot commentedon Jun 23, 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
mitar commentedon Jun 23, 2020
/remove-lifecycle stale