Argus/Colossus QueryNode result caching to improve latency

[zeeshanakram3]

Context

We are going with the proposal described in #4414 for assets request authentication by Argus/Colossus.

Look into consistent in-memory caching of the actor persona along with the valid permissions instead of querying the information from QN. Ways to avoid cache inconsistencies seem to be 1) keeping local caching instance subscribed to the changes of processor DB and updating the corresponding cache entry whenever processor DB record changes, 2) probably a more straightforward approach is to keep the cache record for a limited time and then re-fetch the record after this, there is a chance that cache would give outdated personae for a given member, but that's not a huge problem if TTL of cache record in small

[bedeho]

• Are we agreeing to focus on only doing authentication of whether data should be fetched by the given node or not, and to separate out questions about whether the given user has access rights to a token based system?
• I really like thee simplicity of 2) where you just routinely fetch all data objects you are supposed to serve, but one problem is how to deal with a new upload? if people try to access an upload that was just completed, then they will fail, because all distributors need to wait some minimum period of time before they re-fetch a list of what objects they should serve. One solution is to assume this is a global parameter value X, and that all apps and infra waits at least X since upload confirmation by liason before it tries to speak to a distributor for a data object that has been uploaded. This is not a great solution, but may be better than nothing. Another alternative is to introduce visibility into the new future lead endpoint into what all distributors have at all times.

[mnaamani]

This is a general problem for all and any query being made by both Argus and Colossus in-fact.

Research:
Research if there are any Apollo Client specific configuration options for caching queries.

Proposal options (caching/optimising queries)

• Nodes should cache results.
• Aggregate response for identical queries that are being concurrently made by incoming client requests.
• It should be possible to customise the caching parameters for each known query.
• In process vs External service? -> Internal for optimizing latency (common shared general npm package that can be used by both Argus/Colossus)

Some tooling that we found that could be useful:

• https://github.com/graphql/dataloader

Note: here we are omitting here any optimizations of the Queries themselves or DB schema, DB engine configurations.

[mnaamani]

Note: here we are omitting here any optimizations of the Queries themselves or DB schema, DB engine configurations.

On that front, perhaps a simple config tuning can be done, following lead by Orion team:

https://github.com/Joystream/orion/blob/master/db/postgres.conf

Howto use in docker-compose:
https://github.com/Joystream/orion/blob/8965f6676a8495896bcbf94eb2102e2977ad48d1/docker-compose.yml#L19

[mnaamani]

I tried increasing the shared_buffer memory to 2GB, it didn't help individual queries.

[mnaamani]

Some latency came form queries requiring local temp storage, increasing work_mem in postgres config helped eliminate the temp files removing any latency introduced by disk io latency.