Unmarshalling fails in log stream (read_namespaced_pod_log) obtained via watch

What happened (please include outputs or screenshots):

I'm using the following code to monitor log outputs of a pod:

from kubernetes import client, config, watch

config.load_kube_config()
k_api = client.CoreV1Api()

w = watch.Watch()
for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
    print(msg)

This works in general. However if the process running in the pod prints a line that consists of a plain integer (e.g. echo 42), the message unmarshalling crashes:

TypeError                                 Traceback (most recent call last)
/path/to/script.py in <module>()
      1 w = watch.Watch()
----> 2 for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
      3     print(msg)

/home/myuser/.virtualenvs/ml/local/lib/python2.7/site-packages/kubernetes/watch/watch.pyc in stream(self, func, *args, **kwargs)
    143             try:
    144                 for line in iter_resp_lines(resp):
--> 145                     yield self.unmarshal_event(line, return_type)
    146                     if self._stop:
    147                         break

/home/myuser/.virtualenvs/ml/local/lib/python2.7/site-packages/kubernetes/watch/watch.pyc in unmarshal_event(self, data, return_type)
     91         except ValueError:
     92             return data
---> 93         js['raw_object'] = js['object']
     94         if return_type:
     95             obj = SimpleNamespace(data=json.dumps(js['raw_object']))

TypeError: 'int' object has no attribute '__getitem__'

What you expected to happen:

It should not crash depending on the line content.

How to reproduce it (as minimally and precisely as possible):

Use the code above to fetch logs.
echo 42 in a pod.

Anything else we need to know?:

The problem only occurs when accessing logs as a stream. Fetching the entire logs with a plain read_namespaced_pod_log works even if lines contain pure integeres.
Possibly related to Failed to get a log stream when log line is json type #982 ?

Environment:

Kubernetes version (kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

OS: Ubuntu 18.04
Python version: Python 2.7.12
Python client version: 10.0.1

/assign

I think I have found another problem with reading log message streams: empty lines.

If I do something like echo -e "\n" in a POD, the code above errors with:

  File "myscript.py", line X, in some_func
    for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 144, in stream
    for line in iter_resp_lines(resp):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 48, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/urllib3/response.py", line 731, in read_chunked
    "Response is not chunked. "
urllib3.exceptions.ResponseNotChunked: Response is not chunked. Header 'transfer-encoding: chunked' is missing.

Should I open a separate issue, or does it make sense to fix that in one go?

Add tests of unmarshaling literals to watch.py

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

The issue still exists and is pretty big showstopper in using the streaming API.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

/remove-lifecycle rotten

@bluenote10

@bluenote10 I do not think bot recognized your command because of backticks.

@jiahuif

/unassign

@jiahuif has a reproduce and WIP fix in kubernetes-client/python-base#174

Fixed Watch.unmarshal_event when data is not a JSON-serialized object

Navigation Menu

Repository navigation

Unmarshalling fails in log stream (read_namespaced_pod_log) obtained via watch #983

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unmarshalling fails in log stream (read_namespaced_pod_log) obtained via watch #983

Description

Activity

roycaihw commented on Oct 15, 2019

bluenote10 commented on Oct 16, 2019

fejta-bot commented on Jan 14, 2020

bluenote10 commented on Jan 14, 2020

fejta-bot commented on Feb 13, 2020

mitar commented on Feb 13, 2020

mitar commented on Feb 13, 2020

roycaihw commented on Feb 13, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions