BigQuery's QueryResultsOption.maxWaitTime Not Respected

[dclements]

I may be missing something very simple, but the QueryResultsOption.maxWaitTime does not appear to be followed.

Version: 0.26.0-beta.

Code sample below:

bigquery.query(
            queryRequest,
            JobId.of(),
            QueryOption.of(QueryResultsOption.pageSize(1000L)),
            QueryOption.of(
              QueryResultsOption.maxWaitTime(
                1000L
              )
            )
          )

This is based on the example from: https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery

I expected this to block for 1 s and then return, regardless of whether it has completed, and then return (possibly with response.jobCompleted returning false).

When I run a query that lasts 15 seconds, it will block for all 15 seconds, however, the full time of the query (until jobCompleted is true). No matter what I set the value of maxWaitTime to, it never seems to return before jobCompleted returns true.

[vam-google]

There was a recent change in BigQuery API which caused that.

Now, BigQuery.query() is a synchronous operation, which is essentially a wrapper around creating a QueryJob and waiting for its completion (notice how now query() method accepts QueryJobConfiguration as an input).

The reason behind that change was to avoid using the query API at all, since it has various limitations. Now the client library never calls that API.

But I agree the result of the change is now confusing and there should be additional work done to polish this new implementation.

But currently there are 2 options you can provide, which have similar names and can be confused with each other:
QueryOption.of(QueryResultsOption.maxWaitTime(1000L))
and
QueryOption.of(RetryOption.totalTimeout(Duration.ofMillis(1000L)))

So to achieve what you want you need the second option. The first option still exists to allow configuring parameters for the getQueryResults call, which is used to do the actual polling (waiting) for the query job completion.

Also note that now if RetryOption.totalTimeout() expires an exception will be thrown (instead of returning a result), with e.getCause() instanceof PollException == true

For your case, you most probably want to create a QueryJob manually, start it, and then wait for its completion. Please check ITJobSnippets.testWaitForWithOptions() for an example.

[vam-google]

Some enhancements/fixes are needed for BigQuery:

1. Better documentation for BigQuery.query() method: explicitly specify that it is a synchronous call now and what happens if polling fails
2. Rename QueryResultsOptions.maxWaitTime() or at least add better documentation about what that parameter regulates
3. Better handle PollException (probably throw it directly instead of wrapped in BigQueryException, provide more details in PollException about failed polling task)

[dclements]

Awesome, that makes sense and thank you for the clarification. To add to the enhancements, some additional points in the documentation that seem out of date (or at least confusing):

The javadoc for maxWaitTime reads:

Returns an option that sets how long to wait for the query to complete, in milliseconds,
before returning. Default is 10 seconds. If the timeout passes before the job completes,
 {@link QueryResponse#jobCompleted()} will be {@code false}.

The javadoc for BigQuery#getQueryResults shows the old pattern, and seems to use query without any parameters that would indicate manual polling.

[yihanzhen]

Move this issue to feature request backlog and close it here.