Invalid Statistics type for Load job

[Syed-SnapLogic]

Hi,
I have following code snippet in my project.

Credentials c = new ServiceAccountCredentials();  /* hiding credentials here*/
 BigQueryOptions options = BigQueryOptions.newBuilder().setCredentials(c).setProjectId('id').build();
 BigQuery bq = options.getService();
 TableId id = TableId.of('project id', 'dataset id', 'table id');
 Table t = bq.getTable(id);
 t.load(FormatOptions.csv(), "gs://bucket/file1.csv");
 System.out.println(loadJob.getStatistics());

I was expecting the statistics to be of 'Load' type but its of 'Copy' type.

Can anyone tell me where am I doing wrong? Indeed, data is getting inserted into table but I am missing the output statistics (no of rows, no of bytes, etc) due to wrong type being returned.

[vam-google]

Hi @Syed-SnapLogic, I don't think you are doing anything wrong. I tried to reproduce your issue and I have exactly same result: insert a load job, but receive CopyStatistics.

What happens here is the following: on the moment when the job statistics object should be created, there is statistics type info missing (either it is not returned from the server or is lost somewhere in between), and when there is no statistics type present, the google-cloud-java bigquery client defaults to CopyStatistics (check JobStatistics.fromPb() method for details). When that code is executed the actual statisticsPb object has only 2 fields: creationTime, startTime (i.e. no type info at all).

I need to investigate more why that happens, but there is a good chance that it is either bug on the client or the server side.

[Syed-SnapLogic]

Hi @vam-google, I found the resolution. You can still keep your hunt if you wish.
If I say,

Job loadJob = table.load(...);
System.out.println(loadJob.getStatistics());

then certainly it would give copy statistics. However, doing

Job loadJob = table.load(...);
 loadJob = loadJob.waitFor();
System.out.println(loadJob.getStatistics());

is giving load statistics.

[vam-google]

Thanks for the update! It is good that it works for you now, but in that case you are dealing with an already completed job. In other words if the job is completed, it returns the proper statistics type, if it is in progress it returns a wrong one. This is still not good.