Publisher : Lifetime of the publisher object (Pubsub)

[hmigneron]

When using the pubsub java client, how long should a Publisher object live for ?

In a scenario where we get frequent HTTP requests that get translated into Pubsub messages, doing

try {
    publisher = Publisher.defaultBuilder(topic).build();

    //... do publish stuff
}
finally { 
    if (publisher != null) {
        publisher.shutdown();
    }
}

each time seems inefficient. I couldn't find any documentation on the expected lifetime of the object.

How often should the publisher be shutdown (either in terms of time or messages published) ?

[garrettjonesgoogle]

We recommend keeping it around for the lifetime of your process since it has non-trivial startup overhead.

[hmigneron]

OK thank you. Considering the process might live for a very long time in our case (multiple days), should we schedule period shutdowns to keep things safe or is that not even necessary ?

[garrettjonesgoogle]

You shouldn't have to schedule periodic shutdowns - we would consider it a bug if that was necessary.

[hmigneron]

Thank you very much for your answers !

[hienle-hps]

Sorry to resurrect a closed thread.

@garrettjonesgoogle Now that AppEngine Standard Java 8 is GA, what's the recommended approach for using PubSub during an AppEngine request and ensuring prompt publishing, is it along the lines of:

• one Publisher created in Servlet.init()
• using BatchSettings with low thresholds
  • or setEnabled(false)

Are special considerations needed for the Publisher's executor relative to the Request's Thread and limits?

[garrettjonesgoogle]

Assigning to @pongad for him to answer.

[pongad]

I'm quite rusty on AppEngine. @jabubake Please let me know if I got anything wrong.

Regarding threads, I believe threads still cannot be created "normally". So you need to call setExecutorProvider and create one that spawns threads using AppEngine's ThreadManager according to this doc.

If you set BatchSettings with 1 element count, you'll effectively disable batching. After calling publish you can call get() on the returned future to wait for the message to be published. I believe threads still cannot outlast the request.

Jisha, I believe we talked a while back about creating a doc or sample app for pubsub on GAE; has this happened?

@hienle-hps Your referring to setEnabled made me realize something rather unfortunate. The setEnabled option was added later to BatchingSettings and the current pubsub implementation doesn't know about it, so setEnabled(false) won't actually do anything.

@garrettjonesgoogle I think this should be fixed in BatchSettings itself. Perhaps we can make it so that a disabled batch will have capacity of 1 element, 1 byte, and 1ms timeout. The current surface, I think, is another bug waiting to bite people. WDYT?

[jabubake]

@hienle-hps
you are not required use the setExecutorProvider on App Engine J8
https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/appengine-java8/pubsub/src/main/java/com/example/appengine/pubsub/PubSubPublish.java

is a basic example.
For prompt publishing, setting your element count low or the delay threshold low (if element count is not guaranteed) would work.

For a long living publisher, either one that is instantiated within the servlet or as a WebListener as done in this sample : https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/appengine-java8/spanner/src/main/java/com/example/appengine/spanner/SpannerClient.java

would be useful.

Given you intend to block the request thread, as pongad@ pointed out,
worth looking into basic / manual scaling :
https://cloud.google.com/appengine/docs/standard/java/an-overview-of-app-engine#scaling_types_and_instance_classes

[hienle-hps]

Thanks for the examples and clarification!