Unsupported Media Type (HTTP 415) when patching custom resources

Actual behaviour

v10.0.0 was released this night. Since then, custom resource patching is broken:

HTTP 415 "Unsupported Media Type" when patching the custom resources.

Traceback (most recent call last):
  File "kopf-134.py", line 15, in <module>
    rsp = api.patch_namespaced_custom_object(**request_kwargs)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 1951, in patch_namespaced_custom_object
    (data) = self.patch_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, body, **kwargs)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 2057, in patch_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 393, in request
    body=body)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/rest.py", line 286, in PATCH
    body=body)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (415)
Reason: Unsupported Media Type
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 04 Jul 2019 11:53:38 GMT', 'Content-Length': '263'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json","reason":"UnsupportedMediaType","code":415}

When tracing step by step, the actual Content-Type used is application/json-patch+json, which should match the supported types from the error message.

There is no way to control Content-Type from the user side, i.e. when calling the patch-method.

Expected behaviour

Patching works smoothly.

It all works fine if installed as

pip install 'kubernetes<10.0.0'

Steps to reproduce:

minikube start --kubernetes-version=v1.15.0
kubectl apply -f https://raw.githubusercontent.com/zalando-incubator/kopf/master/examples/crd.yaml
kubectl apply -f https://raw.githubusercontent.com/zalando-incubator/kopf/master/examples/obj.yaml

Any other CRD should show the same behaviour (probably); this one is just for a quick example.

pip install kubernetes==10.0.0

Then a Python script:

import kubernetes

kubernetes.config.load_kube_config()  # developer's config files

request_kwargs = {
    'group': 'zalando.org',
    'version': 'v1',
    'plural': 'kopfexamples',
    'name': 'kopf-example-1',
    'namespace': 'default',
    'body': {},
}

api = kubernetes.client.CustomObjectsApi()
rsp = api.patch_namespaced_custom_object(**request_kwargs)
print(rsp)

Any body content cause the same error, event the empty body — so, I guess, it is not about the patch-content itself.

Versions

Kubernetes: 1.15.0
kubernetes==10.0.0
Python 3.7

Notes

Might be related:

Restrict kubernetes client library to version 9.x.x

Found the root cause:

With 9.0.0, the type was application/merge-patch+json.
With 10.0.0, it is application/json-patch+json.

The select_header_content_type() method choses the first content type from a list of available. The content is not checked:

            return content_types[0]

The CustomObjectsApi feeds it with these 2 content-types:

        header_params['Content-Type'] = self.api_client.\
            select_header_content_type(['application/json-patch+json', 'application/merge-patch+json'])

Previously (in 9.0.0), it was one:

        header_params['Content-Type'] = self.api_client.\
            select_header_content_type(['application/merge-patch+json'])

This happened because of 9c8bd4a.

And so, the default content type has suddenly changed since 9.0 to 10.0.0.

The Kubernetes API doesn't like it, because application/json-patch+json is a list, not dict (see Wikipedia):

[
    { "op": "add", "path": "/myPath", "value": ["myValue"] }
]

However, the content data sent by the users all over the world was not changed, and most likely remains a dict as per application/merge-patch+json content-type (since it worked before).

PS: With body changed from {} to [], the example from the issue description works.

/assign

Thanks for the detailed bug report. I see the select_header_content_type is not intelligent as I would expected.

kube-apiserver supports both JSON patch and JSON merge patch as content types for patching custom resources. The problem is select_header_content_type unconditionally puts the first type from the list into the HTTP header. So it used to be people using the python client can only use JSON patch as body (which was bad), but upgrading the python client suddenly changes the HTTP header to use JSON merge patch, which breaks existing users. What's more, JSON patch is no longer an option given how select_header_content_type works.

I think we should do two things:

For short-term, we should restore the old behavior (setting JSON patch as content type) with a patch release v10.0.1. This can be achieved by excluding feat: custom objects can be merged by json-patch gen#119 and doing a release
For long-term, we should give users the option to specify the content-type. It can be defaulted as JSON patch, but currently it's really hard for users to hijack the HTTP request and change the header. I think removing select_header_content_type is part of migrate from swagger-codegen to openapi-generator gen#93. Ref [WIP] feat: client generated by openapi-generator #738 (comment)

hotfix kubernetes-client#866: switch the order of content-type so JSO…

fixed in 10.0.1

/close

@roycaihw

@roycaihw: Closing this issue.

Details

In response to this:

fixed in 10.0.1

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

which version is this fix in?

I still get the same problem when patching profiles.
kubectl patch profile foo --patch ....

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3"
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb",

Any idea why I'm still seeing it?

Fix custom objects API to preserve backward compatibility

For long-term, we should give users the option to specify the content-type. It can be defaulted as JSON patch, but currently it's really hard for users to hijack the HTTP request and change the header. I think removing select_header_content_type is part of kubernetes-client/gen#93. Ref #738 (comment)

This would be great!

Until this is implemented and for everybody coming here in search for a way to apply a JSON patch to a custom resource with content_type="application/json-patch+json", the dynamic client supports this option already. See for example here.

@roycaihw

Reading this issue, I still don't understand what needs done to actually use this ;)

I'm calling the following code with version 19.15.0 of the python bindings.

        body = [{"op": "remove", "path": "/spec/remoteWrite"}]
        response = api.patch_namespaced_custom_object(
            group="monitoring.coreos.com",                                                                                                                  version="v1",
            name="kube-prometheus-stack-prometheus",
            namespace="monitoring",                                                                                                                         plural="prometheuses",
            body=body,

However, I receive the following error:

Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '95e7d775-0b04-4674-bcba-f35bf7a9cac8', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'aa6ddeef-d18c-429c-a54e-d2be86b818b6', 'Date': 'Wed, 17 Nov 2021 01:33:09 GMT', 'Content-Length': '812'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":" \"\" is invalid: patch: Invalid value: \"[{\\\"op\\\":\\\"remove\\\",\\\"path\\\":\\\"/spec/remoteWrite\\\"}]\": couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }","reason":"Invalid","details":{"causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"[{\\\"op\\\":\\\"remove\\\",\\\"path\\\":\\\"/spec/remoteWrite\\\"}]\": couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }","field":"patch"}]},"code":422}

/cc @roycaihw @nolar

For future readers, the body is a JSON patch which is described here. In particular, it took me a while to figure out how to add an annotation with a / in it:

        body = [
            {
                "op": "add",
                # '~1' is an escaped '/'
                # See https://jsonpatch.com/#json-pointer for escaping rules
                "path": "/metadata/annotations/myproject.io~1archive-name",
                "value": archive.metadata.name,
            }
        ]

Fix custom objects API to preserve backward compatibility

Navigation Menu

Repository navigation

Unsupported Media Type (HTTP 415) when patching custom resources #866

Actual behaviour

Expected behaviour

Steps to reproduce:

Versions

Notes

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unsupported Media Type (HTTP 415) when patching custom resources #866

Description

Actual behaviour

Expected behaviour

Steps to reproduce:

Versions

Notes

Activity

nolar commented on Jul 4, 2019

roycaihw commented on Jul 9, 2019

roycaihw commented on Jul 17, 2019

roycaihw commented on Aug 13, 2019

k8s-ci-robot commented on Aug 13, 2019

abdulsalama commented on Apr 8, 2020

5 remaining items

ableuler commented on Jun 1, 2021

retr0h commented on Nov 17, 2021

jmgilman commented on Mar 2, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions