fix(StateMediator): Double StopEvent timeout to 60s

[runkelcorey]

Summary of changes

Doubles sync_timeout for StopEvents in the hopes (gulp) of quashing API-AAR. This is a coarse knob to turn but I think it's premature to try other options. Since #1065, all incremental updates take less than 2s to process but the initial load of StopEvent can take upwards of 10s. It seems like the startup of the supervision tree imposes additional performance impacts on this loading process and, yesterday, dev experienced API-AAR once again.

My question for the reviewer is: is there a better way to only adjust the initial timeout? I don't want to enable this to silently take 50s on incremental loads. After investigating, I don't see a backwards compatible way to do this: timeout options are passed by a private function in S3Mediator that's used for initial as well as recurring updates. The equivalent function in Mediator is never modified and this would be a break with the existing design pattern (plus, redundant code for the CR crowding case).

[npatel-mbta]

Probably good as is.

Is there logging/other way to check effectiveness of the change?

[runkelcorey]

Is there logging/other way to check effectiveness of the change?

Not as much as I'd like. I'm still unsure why https://mbtace.sentry.io/issues/7395872189/events/a00525a16a1149e4bbed4ae1cee2226f/ recurs when we see log messages like init_table Elixir.State.StopEvent #PID<0.2898.0> took 7592.186ms. My theory is that the processes that handle the message are being paused by the supervision tree mid-execution because, on startup, the GenServer needs to refresh the entire table. The only real metric is the continued absence of that Sentry error