This repository was archived by the owner on Mar 7, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 7, 2024. It is now read-only.
BigQuery is very flakey on local / self-hosted #10
Description
Bigquery client takes > 30s for insert requests when run locally. Need to investigate why this is happening, since if we're facing this, other self-hosted clients would be too.
What's mystifying is that it works perfectly fine on cloud - there's never* a timeout.
While we don't lose data, thanks to the retry job queues, it would be nice to figure out why this is happening in the first place.
More context: See #9
Activity
Twixes commentedon Jun 14, 2021
Could you flesh this issue out a bit more please? I have no idea what the problem is without reading all of the linked issue (which is real ｌｏｎｇ)
neilkakkar commentedon Jun 15, 2021
Local / self-hosted:
vs Cloud:
Note the timing differences, too!
[-]BigQuery is very flakey on local[/-][+]BigQuery is very flakey on local / self-hosted[/+]andrewm4894 commentedon Jun 17, 2021
just adding an example from our posthog - lots of variance and some longish looking insert times and some failing it seems once they hit 30s
neilkakkar commentedon Jun 17, 2021
More data points: I tried the bigquery plugin while setting the
TASK_TIMEOUTto 60 seconds, and I still found similar issues (although a bit fewer, since it meant the requests taking between 30 to 60s were ok now).Also, it probably doesn't have much to do with payload size. (i.e. have noticed smaller payloads taking a lot longer, and then succeeding almost instantly on retry)
Potential places to look for solutions:
insertAllendpoint too frequently?