PubSub: High volume of DNS failures to _grpc_config.pubsub.googleapis.com / _grpclb._tcp.pubsub.googleapis.com.

[rdsubhas]

We are seeing a high volume of DNS lookup failures to _grpc_config.pubsub.googleapis.com. and _grpclb._tcp.pubsub.googleapis.com. when using the pubsub library.

Environment details

1. PubSub
2. Linux, container-os (Google Kubernetes Engine)
3. Java version: 17
4. com.google.cloud:google-cloud-pubsub library version: 1.138.0

Steps to reproduce

1. Instantiate PubSub client
2. It makes failing NXDOMAIN DNS lookups to _grpc_config.pubsub.googleapis.com. and _grpclb._tcp.pubsub.googleapis.com.

Code example

String projectId = "your-project-id-here"; // I used a test project
String topicId = "your-topic-id-here"; // I used a test topic
TopicName topicName = TopicName.of(projectId, topicId);
Publisher publisher = Publisher.newBuilder(topicName).build();

String message = "Hello World!";
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(ByteString.copyFromUtf8(message)).build();

publisher.publish(pubsubMessage);

Run this by setting Log Level to FINEST:

# logging.properties
.level = FINEST
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
io.grpc.level=FINEST
com.google.cloud.pubsub.level = FINEST

# Run the example by setting this system variable
-Djava.util.logging.config.file=.../logging.properties

We can see in the output:

[2025-04-30 12:01:03] [FINER  ] About to query SRV records for _grpclb._tcp.pubsub.googleapis.com
[2025-04-30 12:01:04] [FINE   ] SRV lookup failure
[2025-04-30 12:01:04] [FINER  ] About to query TXT records for _grpc_config.pubsub.googleapis.com
[2025-04-30 12:01:04] [FINE   ] ServiceConfig resolution failure
[2025-04-30 12:01:04] [FINE   ] No TXT records found for pubsub.googleapis.com

This repeats tons of times through the course of the application's lifecycle. We see 22000 DNS failures a day on average.

What is expected

• The official google libraries are configured out of the box correctly
• or there is a way to turn off this behavior and not create DNS error noise

External references such as API reference guides

• https://github.com/grpc/proposal/blob/master/A5-grpclb-in-dns.md

[michaelpri10]

Hello! I am still looking into this issue, but from my testing, these DNS failures appear to be benign and do not affect the performance of the publisher client. Please let me know if this is not the case. To avoid the noise, you can increase the logging level of specific classes that are outputting the DNS failure. In particular adding:

io.grpc.grpclb.GrpclbNameResolver.level=INFO

to your logging configuration can reduce noise in the logs. You can also do the same with the io.grpc.internal.DnsNameResolver and io.grpc.internal.JndiResourceResolverFactory classes.

[rdsubhas]

Hi @michaelpri10 thanks for looking into it.

The primary reason we reported this is: It creates 22000 DNS lookup failures on our DNS servers. We could hide the logs on the application, but we can't hide the DNS lookup errors on our DNS servers.

Hence we reported this, because it's odd for official google libraries to be generating this volume of NXDOMAIN DNS errors out of the box. Thx...

[dustin-auby]

Hi All, i am experiencing this issue too, except once it starts the publishing does fail and no messages make it to pub sub. i get exactly the same error. but i can still ping the dns from the host it is only this lib that fails the dns request.

[michaelpri10]

@dustin-auby Have you made any changes to your Publisher client, such as using a custom channel provider? Even with these DNS failures, publishes should still work and we are not able to reproduce this issue causing publishes to fail.

[michaelpri10]

The Java gRPC library no longer requires the grpclb package as of v1.77.0, but per googleapis/sdk-platform-java#4012, v1.77.0 is not yet compatible with the shared libraries used for Google Cloud Java client libraries. Once this issue is resolved, we should no longer see these DNS lookup failures.

[michaelpri10]

Using the newest com.google.cloud:libraries-bom version 26.80.0 and the most current version of the Cloud Pub/Sub Java client library, I no longer see DNS lookup failures in the logs. If after upgrading you still see the failures, ensure that there is not a transient dependency on the io.grpc:grpc-grpclb package (e.g., by using mvn dependency:tree).