Python Logging API always defaults to Custom logs

[laturner42]

When using the Python Stackdriver Logging API, there is not a well-defined way to specify where you want your logs to land. All of them will go into Custom Logs. The way around this is to use a Logging Handler, but this is not well written anywhere except for this one particular blog post:

https://medium.com/google-cloud/cloud-logging-though-a-python-log-handler-a3fbeaf14704#.39a97qpib

The reason it cannot be done through the standard API is because this line

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/logging/google/cloud/logging/logger.py#L127

overwrites whatever resource you would rather write to. No parameter I could find (or the Google engineer I asked) would set the resource in order to change the destination of the logs. The closest we could find is here:

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/logging/google/cloud/logging/connection.py#L131

but this is set by the previous line stated above. Changing this resource manually is a pretty important feature to us that we are surprised is not already there.

[tseaver]

We would need to add a MonitoredResource type (including mapping the list API method for them), and then allow passing an instance of it through to the logger, either as an optional constructor paramater or as a keyword parameter for log_text and log_struct.

@waprin Any thoughts?

[waprin]

yes @HiroLord, you are definitely right, you should be able to specify a monitored resource.

As a stopgap to at least get the resources types right on GAE/GKE I have #2668 which uses the fluentd handlers. However you are right that in general the client library should allow you to specify a monitored resource of your choice.

I played around with it a bit and found that besides the resource type, I also needed to set the right labels in order for the logs to show up in places where I expect. But there are different labels for different monitored resource types. So I'm not totally sure about the scope of the change necessary.

Out of curiosity, what types of resources are you trying to write to?

[waprin]

Also @HiroLord just a small correction - the logging handler is totally orthogonal to this issue. Handlers are just a way of mapping Python logging calls to API calls, but we have logging handlers in this repo as well, though at the moment they only write to custom logs.

The real reason you can accomplish what you want with that blog post is it's using https://github.com/google/google-api-python-client, which is a lower level auto-generated library, and exposes all the possible ways to write to the API. This library tries to be a more convenient and idiomatic, but in this case it missed the use case of writing to specific resources, which we can fix. Once it's fixed, you can use the handlers or just write to the API directly.

[laturner42]

We are trying to write logs from our compute engine instances using this API. Our end goal is to pipe std_out from different programs into this Python logger to get all of our logs in the same location on stack driver for each instance.

[laturner42]

@waprin Ahh, that makes sense. I was curious how we were able to work around it with handlers, but the lower level API makes sense. Thanks!

[waprin]

@HiroLord you might want to consider installing the fluentd agent and just writing your logs to a file it can pickup from...admittedly the documentation on doing that could also use some work:

https://cloud.google.com/logging/docs/agent/installation

@tseaver the short version of what I'm saying is I agree with you suggested as a way to enable people to accomplish this, though I'm not totally sure we need the MonitoredResource objects and the list method, since I think it will be very uncommon for people to programmatically pick the resource type, so just letting people pass in the resource type as a string might be sufficient.

We might want to consider ways to make getting the right labels/log names together a bit more usable.

[waprin]

@tseaver @dhermes circling back on this, I think this does need to be added the logger surface because it's pretty basic functionality to the api .

I think @tseaver's proposed plan to add the MonitoredResource type does make sense.

However, the code already exists in google.cloud.monitoring.resource.py and google.cloud.monitoring.label.py. And if you look at the Logging API and the Monitoring API protos, you will see that they do share the MonitoredResource as common object.

So I think (but I'm not sure) that google.cloud.monitoring.resource.py should be in some sort of common package. But it doesn't feel like it belongs in core, and a new package feels like overkill. But I think a new package like common makes the most sense.

So @dhermes @tseaver WDYT about moving resource.py to a common package?

@karan since we want to write to the correct monitored resource this is blocking the shutdown handler (although we could temporarily use the HTTP or Gax client direclty)

[lukesneeringer]

But it doesn't feel like it belongs in core, and a new package feels like overkill. But I think a new package like common makes the most sense.

I am confused. A new package feels like overkill, and is most sensible? (Apologies if I am not understanding, those two sentences just seem contradictory to me.)

I would be fine with just putting it in core. I am nervous about the idea of having core and common as I cannot think of any defining rule of what belongs in which of them.

[waprin]

Thanks for chiming in. Yes I was thinking out loud and was a bit unclear. In my head core involves code relating to the client but common would be shared API data objects. But I agree it's a blurry distinction so moving to core sounds good to me.

[daiyueweng]

Hi, I see that in _make_entry_resource of logger.py, there is a resource dictionary that has a key -
resource which specifies resource type, so maybe I can pass in/define whatever resource I like to write to in there?

Or, instead, I have to use write_entries in _gax.py and __http.py?

[waprin]

@daiyueweng I might be missing something, but even if you use _make_entry_resource there's no method that accepts the object that returns. All of the log_* methods call _make_entry_resource themselves. Plus it's not ideal to use a private method.

I think I explained it slightly poorly in the previous issue. You should be able to use the public method client.logging_api, which will pick the underlying transports between _gax.py and _httpy.py the same way it does for the other methods, so you shouldn't have to choose between them. You can look at those files to get an idea of their methods signatures though, write_entries being the one you probably want.

[daiyueweng]

@waprin So I cloned the google.cloud.logging library into a directory/package(called stackdriver_logging) of my project and changed all the references to google.cloud.logging to stackdriver_logging.google.cloud.logging. Slightly modified the client.py and logger.py in order to use Monitored Resource as follows (moved to a gist by @dhermes, with diffs provided as well).

When I unit tested the code, I got the following error when I tried to use project resource type,

Traceback (most recent call last):
  File "/home/user_name/.local/lib/python3.5/site-packages/google/protobuf/json_format.py", line 441, in _ConvertFieldValuePair
    message_descriptor.full_name, name))
google.protobuf.json_format.ParseError: Message type "google.api.MonitoredResource" has no field named "p".