CloudStorage: ApiProxy$RPCFailedException: The remote RPC to the application server failed for the call urlfetch.Fetch().

[jbaldassari]

In a Java 8 flexible environment I'm trying to retrieve a file from cloud storage like this:

    private static byte[] retrieveFile(final String fileName) {
        final Storage cloudStorage = StorageOptions.getDefaultInstance().getService();
        final String bucketName = GcpProjects.getDefaultProjectId() + "-data";
        try {
            final Blob blob = cloudStorage.get(BlobId.of(bucketName, fileName));
            if (!blob.exists()) {
                throw new RuntimeException(fileName + " not found in bucket " + bucketName);
            }
            return blob.getContent();
        } catch (StorageException e) {
            throw new RuntimeException("Error obtaining " + fileName + " from bucket " + bucketName, e);
        }
    }

This worked fine for me in the J8 standard environment, but in the flexible environment the Storage.get(BlobId) call fails with:

Caused by: com.google.cloud.storage.StorageException: com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC to the application server failed for the call urlfetch.Fetch().
	at com.google.cloud.storage.StorageException.translateAndThrow(StorageException.java:71) ~[google-cloud-storage-1.3.1.jar:1.3.1]
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:196) ~[google-cloud-storage-1.3.1.jar:1.3.1]
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:202) ~[google-cloud-storage-1.3.1.jar:1.3.1]
Caused by: com.google.apphosting.api.ApiProxy$RPCFailedException: The remote RPC to the application server failed for the call urlfetch.Fetch().
	at com.google.apphosting.vmruntime.VmApiProxyDelegate.runSyncCall(VmApiProxyDelegate.java:175) ~[appengine-managed-runtime-1.9.40.sync-SNAPSHOT.jar:201703172103]
	at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeApiCall(VmApiProxyDelegate.java:155) ~[appengine-managed-runtime-1.9.40.sync-SNAPSHOT.jar:201703172103]
	at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeSyncCallWithTimeout(VmApiProxyDelegate.java:143) ~[appengine-managed-runtime-1.9.40.sync-SNAPSHOT.jar:201703172103]
	at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeSyncCall(VmApiProxyDelegate.java:132) ~[appengine-managed-runtime-1.9.40.sync-SNAPSHOT.jar:201703172103]
	at com.google.apphosting.vmruntime.VmApiProxyDelegate.makeSyncCall(VmApiProxyDelegate.java:75) ~[appengine-managed-runtime-1.9.40.sync-SNAPSHOT.jar:201703172103]
	at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:118) ~[appengine-api-1.0-sdk-1.9.40.jar:na]
	at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:40) ~[appengine-api-1.0-sdk-1.9.40.jar:na]
	at com.google.api.client.extensions.appengine.http.UrlFetchRequest.execute(UrlFetchRequest.java:74) ~[google-http-client-appengine-1.22.0.jar:1.22.0]
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) ~[google-http-client-1.22.0.jar:1.22.0]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[google-api-client-1.22.0.jar:1.22.0]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[google-api-client-1.22.0.jar:1.22.0]
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[google-api-client-1.22.0.jar:1.22.0]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:337) ~[google-cloud-storage-1.3.1.jar:1.3.1]
	at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:191) ~[google-cloud-storage-1.3.1.jar:1.3.1]
	at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:188) ~[google-cloud-storage-1.3.1.jar:1.3.1]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94) ~[gax-1.5.0.jar:na]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54) ~[google-cloud-core-1.3.1.jar:1.3.1]
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:188) ~[google-cloud-storage-1.3.1.jar:1.3.1]

It doesn't say why the urlfetch call failed, unfortunately. This was built from a Gradle project with the following dependencies:

dependencies {
  providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
  providedCompile 'com.google.appengine:appengine:+'
  testCompile 'org.eclipse.jetty:jetty-server:9.+'
  testCompile 'org.eclipse.jetty:jetty-servlet:9.+'
  compile group: 'com.google.guava', name: 'guava', version: '19.+'
  compile group: 'com.google.inject', name: 'guice', version: '4.+'
  compile(group: 'com.google.api-client', name: 'google-api-client', version: '1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
  compile(group: 'com.google.api-client', name: 'google-api-client-appengine', version: '1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
  compile 'com.google.auth:google-auth-library-credentials:0.+'
  compile 'com.google.cloud:google-cloud-core:1.+'
  compile('com.google.apis:google-api-services-cloudkms:v1-rev15-1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }
  compile 'com.google.cloud:google-cloud-storage:1.+'
  compile ('com.google.cloud:google-cloud-datastore:1.+')
  compile group: 'com.google.cloud', name: 'google-cloud-pubsub', version: '0.20+'
}

[neozwu]

It shouldn't call urlfetch. We have logic in our code to dispatch urlfetch only for GAE J7, but normal http transport for GAE J8. I think this logic must have been broken for GAE flex.

[jbaldassari]

I think I figured this out. The following dependency seemed to be required when I was using the J8 standard environment:

  compile(group: 'com.google.api-client', name: 'google-api-client-appengine', version: '1.+') {
     exclude(group: 'com.google.guava', module: 'guava-jdk5')
  }

After removing it, I'm no longer seeing that issue in the flexible environment.

It's pretty confusing that there are certain magic dependencies that need to be there in some cases but not in others. com.google.appengine:appengine-api-1.0-sdk is another one for the standard environment. Everything will compile fine without it, but it won't run in AE correctly. Anyway I just wanted to point out that it could be documented better. I'll close this issue because removing google-api-client-appengine fixed the problem. Sorry for the distraction.