Cloud Tasks Deadline Error, suggested fix w/ GAX dependency conflicts with KMS

[finci-square]

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The request deadline is 2019-06-18T20:59:52.360292137-07:00. The deadline cannot be more than 30s in the future.
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The request deadline is 2019-06-18T20:59:52.360292137-07:00. The deadline cannot be more than 30s in the future.

Using Cloud Tasks and Cloud KMS API in the same maven project.

<dependencyManagement>
		...
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>libraries-bom</artifactId>
        <version>2.8.0</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>
</dependencyManagement>
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-kms</artifactId>
</dependency>
<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-tasks</artifactId>
</dependency>

In issue #5555, it was suggested to import a specific version of GAX

<dependency>
    <groupId>com.google.api</groupId>
    <artifactId>gax</artifactId>
    <version>1.47.0</version>
</dependency>

However after specifying 1.47.0 GAX, code calling KeyManagementServiceClient.create() is failing with:

Caused by: java.lang.AbstractMethodError: Receiver class com.google.api.gax.grpc.InstantiatingGrpcChannelProvider does not define or inherit an implementation of the resolved method abstract needsCredentials()Z of interface com.google.api.gax.rpc.TransportChannelProvider.
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:157) ~[gax-1.47.0.jar:1.47.0]
	at com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.create(GrpcKeyManagementServiceStub.java:320) ~[google-cloud-kms-1.14.0.jar:1.14.0]
	at com.google.cloud.kms.v1.stub.KeyManagementServiceStubSettings.createStub(KeyManagementServiceStubSettings.java:292) ~[google-cloud-kms-1.14.0.jar:1.14.0]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.<init>(KeyManagementServiceClient.java:155) ~[google-cloud-kms-1.14.0.jar:na]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:136) ~[google-cloud-kms-1.14.0.jar:na]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:127) ~[google-cloud-kms-1.14.0.jar:na]

Any thoughts on how to resolve? thanks!!

[elharo]

Try upgrading libraries-bom to 3.0.0

[finci-square]

@elharo unfortunately no change

[finci-square]

@elharo to clarify, I've removed the extra specification of the GAX dependency as that prevents my app from starting (I use KMS at runtime to decrypt required runtime secrets) and upgraded the libraries-bom to 3.0.0 but the tasks API continues to return

io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The request deadline is 2019-12-02T11:56:57.52236040075-08:00. The deadline cannot be more than 30s in the future.

[finci-square]

ref googleapis/google-cloud-go#1577

[elharo]

That deadline is in the past. I wonder if that's relevant. Not sure I can suggest more here without having al the code to look at.

[finci-square]

Is there anything I should look for? I don't set or configure any timeout properties in my code.

here's code where im attempting to queue a task

public SyncJob queueTaskFor(SyncJob syncJob) throws IOException {
    if (environment.isLocal() && !environment.shouldTestTaskQueueLocally()) {
      this.logger.info(String.format("[syncjob=%s] Not queueing app engine task, running locally",
          syncJob.getId()));
      return syncJob;
    }
    try (CloudTasksClient tasksClient = CloudTasksClient.create()) {
      tasksClient.createTask(buildQueueName(), buildTask(syncJob.getId()));
    }
    syncJob.setQueuedAt(Instant.now());
    return this.syncJobRepositoryService.save(syncJob);
  }

  private QueueName buildQueueName() {
    return QueueName.newBuilder()
        .setProject(environment.getProjectId())
        .setQueue(TasksService.BATCH_WORKER_QUEUE_NAME)
        .setLocation(TasksService.APP_ENGINE_LOCATION)
        .build();
  }

  private Task buildTask(String syncDefinitionId) {
    return Task.newBuilder()
        .setAppEngineHttpRequest(
            AppEngineHttpRequest.newBuilder()
                .setBody(ByteString.copyFrom(syncDefinitionId, Charset.defaultCharset()))
                .setAppEngineRouting(
                    AppEngineRouting.newBuilder()
                        .setService(BATCH_WORKER_SERVICE_NAME)
                        .build())
                .setRelativeUri(BATCH_WORKER_URI)
                .setHttpMethod(com.google.cloud.tasks.v2.HttpMethod.POST)
                .build()
        ).build();
  }

[finci-square]

wait- at the time of posting, 2019-12-02T11:56:57.52236040075-08:00 was not in the past?

It was 10 minutes in the future..

Example of what i just ran, time is now about 11:58:

2019-12-02T12:07:55.51012918975-08:00

[finci-square]

is there a way to set this deadline from the SDK in the meantime? I'm not seeing it though.

And, an instance with manual scaling should support a request deadline of 24 hours, correct?

[elharo]

Have you tried gax 1.50.1? If you bring everything up to the latest version, it should work. At the very least there shouldn't be any AbstractMethodErrors. We shouldn't have introduced a new method in the TransportChannelProvider interface. That was our mistake. However I think this is all resolved at head.

[finci-square]

@elharo this doesnt appear to be true.

I've brought everything up:

<dependencyManagement>
<dependencies>
<dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>libraries-bom</artifactId>
                <version>3.0.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-kms</artifactId>
            <version>1.35.0</version>
        </dependency>

<!--        Required to Clear KMS Runtime Error-->
        <dependency>
            <groupId>com.google.api.grpc</groupId>
            <artifactId>proto-google-cloud-kms-v1</artifactId>
            <version>0.82.0</version>
        </dependency>

<!--        Required for Cloud Tasks Support-->
        <dependency>
            <groupId>com.google.api</groupId>
            <artifactId>gax</artifactId>
            <version>1.50.1</version>
        </dependency>

note this log message

Caused by: java.lang.AbstractMethodError: Receiver class com.google.api.gax.grpc.InstantiatingGrpcChannelProvider does not define or inherit an implementation of the resolved method abstract needsCredentials()Z of interface com.google.api.gax.rpc.TransportChannelProvider.
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:152) ~[gax-1.50.1.jar:1.50.1]
	at com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.create(GrpcKeyManagementServiceStub.java:320) ~[google-cloud-kms-1.14.0.jar:1.14.0]
	at com.google.cloud.kms.v1.stub.KeyManagementServiceStubSettings.createStub(KeyManagementServiceStubSettings.java:292) ~[google-cloud-kms-1.14.0.jar:1.14.0]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.<init>(KeyManagementServiceClient.java:155) ~[google-cloud-kms-1.14.0.jar:na]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:136) ~[google-cloud-kms-1.14.0.jar:na]
	at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:127) ~[google-cloud-kms-1.14.0.jar:na]

at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:152) ~[gax-1.50.1.jar:1.50.1]

w/ gax 1.50.1 i still see this

[elharo]

OK, I can probably reproduce that though it would still help to have a full project, not just a pom.xml.

[finci-square]

Ok I'll produce a minimal project demonstrating the problem

On Tue, Dec 3, 2019 at 2:42 PM Elliotte Rusty Harold < ***@***.***> wrote: OK, I can probably reproduce that though it would still help to have a full project, not just a pom.xml. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6963?email_source=notifications&email_token=ANQHD2XAA4EZK77I2TFTFH3QW3OHBA5CNFSM4JT7FAD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF3CKEA#issuecomment-561390864>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANQHD2RVGGB2VAYFM2FPWT3QW3OHBANCNFSM4JT7FADQ> .

-- Jordan Finci | Solutions Engineer | finci@squareup.com | (301) 785-6038

[elharo]

You might need an explicit dependency on gax-grpc:

            <dependency>
            <groupId>com.google.api</groupId>
            <artifactId>gax-grpc</artifactId>
            <version>1.50.1</version>
        </dependency>