Skip to content

datastore emulator sometimes crashes, which causes test flakyness #3276

Description

@ensonic

We're using bazel to build our software. At the Ci we noticed test flakyness. When I run the tests localy like:

bazel test --runs_per_test=10000 //src/java/com/example/foo:FooTest

I get about 5 failures that look like this:

SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@74287ea3
com.google.cloud.datastore.DatastoreException: I/O error
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
	at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
	at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
	at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
	at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:73)
	at com.google.cloud.datastore.QueryResultsImpl.<init>(QueryResultsImpl.java:57)
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:167)
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:158)
	at com.cloudrobotics.map.MapService.deleteOccupancyGrid(MapService.java:519)
	at com.cloudrobotics.map.MapService.deleteLayer(MapService.java:252)
	at com.cloudrobotics.map.MapService.deleteLayer(MapService.java:272)
	at com.cloudrobotics.map.v1alpha1.MapServiceGrpc$MethodHandlers.invoke(MapServiceGrpc.java:1755)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:653)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.halfClosed(ServerImpl.java:664)
	at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.halfClose(InProcessTransport.java:621)
	at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:422)
	at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:47)
	at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:47)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:177)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:119)
	at com.cloudrobotics.map.v1alpha1.MapServiceGrpc$MapServiceBlockingStub.deleteLayer(MapServiceGrpc.java:1371)
	at com.cloudrobotics.map.MapServiceTest.testGetOccupancyGrid_afterDeleteLayer_throwsNotFound(MapServiceTest.java:472)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:89)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:112)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:144)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:82)
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:226)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:195)
	at com.google.datastore.v1.client.Datastore.runQuery(Datastore.java:119)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:180)
	... 57 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1022)
	at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1020)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1019)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
	at sun.net.www.protocol.http.HttpURLConnection.access$100(HttpURLConnection.java:91)
	at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1283)
	at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1281)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1280)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:183)
	... 59 more

Right now we're using a helper like this to mitigate it a bit
https://gist.github.com/ensonic/a1b64be415f537b59bcc951883b29596
but of course this won't help tests where datastore looses connection in the middle of the test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: datastoreIssues related to the Datastore API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions