Bad dependency for `google-cloud-logging` vs `google-cloud-datastore`

[gregw]

For https://github.com/GoogleCloudPlatform/jetty-runtime we are trying to use both google-cloud-logging and google-cloud-datastore, but there appears to be incompatible dependencies.

With a simple pom like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <name>Google Cloud Example</name>
  <groupId>org.acme</groupId>
  <version>1.0.0</version>
  <artifactId>example</artifactId>
  <packaging>jar</packaging>

  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-logging</artifactId>
      <version>1.0.1</version>
    </dependency>
  </dependencies>
</project>

we get the following dependency:tree:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ example ---
[INFO] org.acme:example:jar:1.0.0
[INFO] \- com.google.cloud:google-cloud-logging:jar:1.0.1:compile
[INFO]    +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26:compile
[INFO]    +- com.google.cloud:google-cloud-core:jar:1.0.1:compile
[INFO]    |  +- com.google.guava:guava:jar:19.0:compile
[INFO]    |  +- joda-time:joda-time:jar:2.9.2:compile
[INFO]    |  +- org.json:json:jar:20160810:compile
[INFO]    |  +- com.google.api:api-common:jar:1.0.0:compile
[INFO]    |  +- com.google.api:gax:jar:1.0.0:compile
[INFO]    |  +- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
[INFO]    |  |  \- com.google.code.gson:gson:jar:2.7:compile
[INFO]    |  +- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
[INFO]    |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.9:compile
[INFO]    +- com.google.cloud:google-cloud-core-grpc:jar:1.0.1:compile
[INFO]    |  +- com.google.protobuf:protobuf-java:jar:3.2.0:compile
[INFO]    |  \- io.grpc:grpc-protobuf:jar:1.2.0:compile
[INFO]    |     \- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile
[INFO]    +- com.google.api:gax-grpc:jar:0.16.0:compile
[INFO]    |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.4.0:compile
[INFO]    |  |  +- com.google.http-client:google-http-client:jar:1.19.0:compile
[INFO]    |  |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO]    |  |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO]    |  |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    |  |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO]    |  |  \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile
[INFO]    |  |     \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
[INFO]    |  +- com.google.auto.value:auto-value:jar:1.2:compile
[INFO]    |  +- org.threeten:threetenbp:jar:1.3.3:compile
[INFO]    |  \- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO]    +- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.1.9:compile
[INFO]    +- io.grpc:grpc-netty:jar:1.2.0:compile
[INFO]    |  +- io.grpc:grpc-core:jar:1.2.0:compile (version selected from constraint [1.2.0,1.2.0])
[INFO]    |  |  +- com.google.errorprone:error_prone_annotations:jar:2.0.11:compile
[INFO]    |  |  +- io.grpc:grpc-context:jar:1.2.0:compile
[INFO]    |  |  \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
[INFO]    |  +- io.netty:netty-codec-http2:jar:4.1.8.Final:compile (version selected from constraint [4.1.8.Final,4.1.8.Final])
[INFO]    |  |  +- io.netty:netty-codec-http:jar:4.1.8.Final:compile
[INFO]    |  |  |  \- io.netty:netty-codec:jar:4.1.8.Final:compile
[INFO]    |  |  \- io.netty:netty-handler:jar:4.1.8.Final:compile
[INFO]    |  |     \- io.netty:netty-buffer:jar:4.1.8.Final:compile
[INFO]    |  |        \- io.netty:netty-common:jar:4.1.8.Final:compile
[INFO]    |  \- io.netty:netty-handler-proxy:jar:4.1.8.Final:compile
[INFO]    |     +- io.netty:netty-transport:jar:4.1.8.Final:compile
[INFO]    |     |  \- io.netty:netty-resolver:jar:4.1.8.Final:compile
[INFO]    |     \- io.netty:netty-codec-socks:jar:4.1.8.Final:compile
[INFO]    +- io.grpc:grpc-stub:jar:1.2.0:compile
[INFO]    \- io.grpc:grpc-auth:jar:1.2.0:compile
[INFO]       \- com.google.auth:google-auth-library-credentials:jar:0.4.0:compile

Note that 0.4.0 version of com.google.auth:google-auth-library-oauth2-http:jar

The google-cloud-datastore version depends instead on 0.6.1 as you can see in the following dependency:tree:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ example ---
[INFO] org.acme:example:jar:1.0.0
[INFO] \- com.google.cloud:google-cloud-datastore:jar:1.0.1:compile
[INFO]    +- com.google.cloud:google-cloud-core:jar:1.0.1:compile
[INFO]    |  +- com.google.guava:guava:jar:19.0:compile
[INFO]    |  +- joda-time:joda-time:jar:2.9.2:compile
[INFO]    |  +- org.json:json:jar:20160810:compile
[INFO]    |  +- com.google.api:api-common:jar:1.0.0:compile
[INFO]    |  |  +- com.google.auto.value:auto-value:jar:1.1:compile
[INFO]    |  |  \- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO]    |  +- com.google.api:gax:jar:1.0.0:compile
[INFO]    |  |  \- org.threeten:threetenbp:jar:1.3.3:compile
[INFO]    |  +- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
[INFO]    |  |  \- com.google.code.gson:gson:jar:2.7:compile
[INFO]    |  +- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
[INFO]    |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.9:compile
[INFO]    +- com.google.cloud:google-cloud-core-http:jar:1.0.1:compile
[INFO]    |  +- com.google.auth:google-auth-library-credentials:jar:0.6.1:compile
[INFO]    |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.6.1:compile
[INFO]    |  |  \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile
[INFO]    |  |     \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
[INFO]    |  +- com.google.http-client:google-http-client:jar:1.21.0:compile
[INFO]    |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO]    |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO]    |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO]    |  +- com.google.oauth-client:google-oauth-client:jar:1.21.0:compile
[INFO]    |  +- com.google.api-client:google-api-client:jar:1.21.0:compile
[INFO]    |  +- com.google.http-client:google-http-client-appengine:jar:1.21.0:compile
[INFO]    |  \- com.google.http-client:google-http-client-jackson:jar:1.21.0:compile
[INFO]    |     \- org.codehaus.jackson:jackson-core-asl:jar:1.9.11:compile
[INFO]    +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
[INFO]    |  \- com.google.protobuf:protobuf-java:jar:3.0.0:compile
[INFO]    +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.3.0:compile
[INFO]    |  \- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
[INFO]    \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.9:compile
[INFO]       +- io.grpc:grpc-stub:jar:1.2.0:compile
[INFO]       |  \- io.grpc:grpc-core:jar:1.2.0:compile
[INFO]       |     +- com.google.errorprone:error_prone_annotations:jar:2.0.11:compile
[INFO]       |     +- io.grpc:grpc-context:jar:1.2.0:compile
[INFO]       |     \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
[INFO]       \- io.grpc:grpc-protobuf:jar:1.2.0:compile
[INFO]          \- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile

If a pom lists both datastore and logging dependencies, for some reason the 0.4.0 version is picked, which fails to work with datastore (@janbartel can you comment with the exception you are getting).

[gregw]

Strangely enough, if I checkout from source with the v0.17.2 tag, then a dependency:tree in google-cloud-logging reports the correct 0.6.1 version? So perhaps something went strange in the release process?

[janbartel]

Here's the relevant portion of the stack trace we get when we try to use gcloud datastore with gcloud logging:

java.lang.NoClassDefFoundError: com/google/auth/http/HttpTransportFactory
	at com.google.cloud.http.HttpTransportOptions.<init>(HttpTransportOptions.java:127)
	at com.google.cloud.http.HttpTransportOptions.<init>(HttpTransportOptions.java:42)
	at com.google.cloud.http.HttpTransportOptions$Builder.build(HttpTransportOptions.java:88)
	at com.google.cloud.datastore.DatastoreOptions.getDefaultHttpTransportOptions(DatastoreOptions.java:142)
	at com.google.cloud.datastore.DatastoreOptions$DatastoreDefaults.getDefaultTransportOptions(DatastoreOptions.java:137)
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:254)
	at com.google.cloud.datastore.DatastoreOptions.<init>(DatastoreOptions.java:102)
	at com.google.cloud.datastore.DatastoreOptions.<init>(DatastoreOptions.java:35)
	at com.google.cloud.datastore.DatastoreOptions$Builder.build(DatastoreOptions.java:89)
	at com.google.cloud.datastore.DatastoreOptions.getDefaultInstance(DatastoreOptions.java:158)
	at org.eclipse.jetty.gcloud.session.GCloudSessionDataStore.doStart(GCloudSessionDataStore.java:451)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
	at org.eclipse.jetty.server.session.AbstractSessionCache.doStart(AbstractSessionCache.java:225)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
	at org.eclipse.jetty.server.session.SessionHandler.doStart(SessionHandler.java:503)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:844)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1501)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1463)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)

[janbartel]

Note that excluding the older 0.4.0 com.google.auth:google-auth-library-oauth2-http:jar in favour of the newer 0.6.1 version reveals that the dependency 'io.grpc:grpc-auth:jar:1.2.0also has a dependency clash, this time forcom.google.auth:google-auth-library-credentials:jar:0.4.0`.

Here's the exception:

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/auth/ServiceAccountSigner
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at com.google.cloud.ServiceOptions.defaultCredentials(ServiceOptions.java:269)
	at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:244)
	at com.google.cloud.logging.LoggingOptions.<init>(LoggingOptions.java:102)
	at com.google.cloud.logging.LoggingOptions$Builder.build(LoggingOptions.java:96)
	at com.google.cloud.logging.LoggingOptions.getDefaultInstance(LoggingOptions.java:55)
	at com.google.cloud.logging.LoggingHandler.<init>(LoggingHandler.java:174)
	at com.google.cloud.logging.LoggingHandler.<init>(LoggingHandler.java:155)
	at com.google.cloud.logging.LoggingHandler.<init>(LoggingHandler.java:124)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.logging.LogManager$5.run(LogManager.java:966)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:958)
	at java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:1578)
	at java.util.logging.LogManager.access$1500(LogManager.java:145)
	at java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:1667)
	at java.util.logging.Logger.getHandlers(Logger.java:1777)
	at java.util.logging.Logger.log(Logger.java:735)

[meltsufin]

@janbartel @gregw There is an effort to tackle the dependency conflict issues across the board in the client libs. This is a useful discovery for that effort, but hopefully we're able to resolve it soon. @garrettjonesgoogle PTAL

[garrettjonesgoogle]

@shinfan , I am marking this P0 because datastore and logging can't be used together currently.

[shinfan]

I will take a look.

[shinfan]

I have excluded both google-auth-library-credentials and google-auth-library-oauth2-http in my pom and my sample application executes without exception.

This is the dependency section my pom:

<dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-logging</artifactId>
      <version>1.0.1</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-oauth2-http</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-credentials</artifactId>
        </exclusion>
      </exclusions> 
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-datastore</artifactId>
      <version>1.0.1</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-oauth2-http</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-credentials</artifactId>
        </exclusion>
      </exclusions> 
    </dependency>
    <dependency>
      <groupId>com.google.auth</groupId>
      <artifactId>google-auth-library-oauth2-http</artifactId>
      <version>0.6.1</version>
     </dependency>
     <dependency>
      <groupId>com.google.auth</groupId>
      <artifactId>google-auth-library-credentials</artifactId>
      <version>0.6.1</version>
    </dependency>
  </dependencies>

You also can view and download the entire sample application here

[ofiesh]

@shinfan fyi pubsub has the same problem as logging in that grpc-auth 1.2 uses om.google.auth:google-auth-library-credentials:jar:0.4.0:compile

I'm sure a fix in the parent will fix both, but be aware of it.

https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-pubsub/pom.xml#L55
<dependency> <groupId>io.grpc</groupId> <artifactId>grpc-auth</artifactId> </dependency>

[hienle-hps]

Similar issue in google-cloud-resourcemanager. It uses v1beta1-rev10 of the client-library and refers to Cloudresourcemanager which is now camel case in all the latest client libraries. Gradle tends to resolve the newer client libraries, so this happens at runtime:

java.lang.NoClassDefFoundError: com/google/api/services/cloudresourcemanager/Cloudresourcemanager$Builder