Make Key.toUrlSafe() consistent with the URL-safe key displayed in Google Cloud Datastore Console

[sai-pullabhotla]

The Key.toUrlSafe() returns a URL encoded (similar to URLEncoder.encode) version of the key like the one below:

partition_id+%7B%0A++project_id%3A+%22my-project%22%0A++namespace_id%3A+%22MyNamespace%22%0A%7D%0Apath+%7B%0A++kind%3A+%22MyEntity%22%0A++name%3A+%22MyKey%22%0A%7D%0A

The Google Cloud Datastore Console displays a Base64 encoded version of the key, which looks something like -

agVoZWxsb3IPCxIHQWNjb3VudBiZiwIM

The Console allows filtering on key by URL-safe key, which works fine if I use the URL-safe value displayed by the GCD Console, but does not work when using the value returned from Key.toUrlSafe().

The Console Documentation states that -

URL-safe key

This is a base64-encoded, serialized version of your entity key. The encoding method is available in any Cloud Datastore client library

Example: agVoZWxsb3IPCxIHQWNjb3VudBiZiwIM

So, the client libraries are expected to match with the Base64 version? Nonetheless, it would be nice to have the client libraries and Console in sync to help with crosschecking/debugging purposes.

[mziccard]

@sai-pullabhotla This is interesting. @aozarov any reason our toUrlSafe is different from the one used in the cloud console? Do you think it's possible this changed over time and we failed to keep up-to-date?

[aozarov]

I think that at the time that we provided the toUrlSafe it only worked for this library (there was no "standard"). I am glad to see that now there is and we should change the code to adopt it.

[mziccard]

Then @sai-pullabhotla thanks for bringing this up. I'll work on a fix soon.

[mziccard]

@eddavisson @pcostell can you guys help me understand how the base64 URL-safe key shown in the Cloud Console is computed?

I expected it to match the following code:

BaseEncoding.base64Url().omitPadding().encode(toPb().toByteArray());

But that is not the case.

[pcostell]

The base64 URL-safe key shown in the console only works with the App Engine client libraries. If you think it would be useful for it to provide a URL-safe key usable in the Google Cloud client libraries, please send feedback in the Cloud Console so it can get triaged by the right team.

[mziccard]

@pcostell Can't we generate this base64 URL-safe in Google Cloud? I think it would make sense for us to generate that value as it is used in AE libraries as well as in the Cloud console.

[JoshuaFox]

Parsing and generating URL-Safe keys in the format of the Google App Engine API is essential for us to be able to use the Google Cloud API.

[lesv]

@JoshuaFox Are you asking for compatibility within the library, or between GAE Standard API's / Cloud Console and this library?

Also, how would it affect you if we changed the format generated by this library to be compatible with GAE Standard / Cloud Console?

[JoshuaFox]

I'm not sure what "compatibility within the library" means, but of course I'd like all kinds of compatibility. Specifically, we have URL-Safe keys generated in the GAE API which need to be parsed into Key objects in Google Cloud API; and the Google Cloud API code has to be able to generate keys in the GAE URL-Safe string format.

The Cloud Console is less important.

In any case, we have since Jan 12 written code that digs into this format and does the above conversions, so for us it is less urgent; though still, this compatibility seems like an obvious requirement.