[Task]: Reduce amount of data transferred between addons-server and Elasticsearch

[diox]

Description

We store all translations of every translated field in an array in ES. Those translations are not indexed, ES doesn't use them at all for finding documents, they are purely data we store so that we can return a full add-on object in search without needing to query MySQL.

For each search API request, we're transferring a lot of data between addons-server and Elasticsearch because of those translations: even when we know the language of the API request through the lang parameter, ES will still send to addons-server every translation of every field of every object in the response. The API code will then filter out what it needs to return to the client according to the lang parameter.

If we could tell ES to exclude the translations that are not needed for a particular request, we would transfer way less data, gaining in performance and saving some data transfer costs.

Acceptance Criteria

• Less data is transferred between ES and addons-server

┆Issue is synchronized with this Jira Task

[diox]

To do that, for each translated field we'd need to:

• Store the translations as an object with languages being keys, translations being values
• Store an additional key for the translation in the default language of the add-on, to have a fallback if the field isn't translated in the requested language
• At request time, if a lang is passed in the request: exclude all translation keys for languages of all translated fields other than:
  • The language of the request
  • The additional fallback translation
• In the API field, grab the translation in the lang of the request or use the default one.
• Possible extra optimization: don't store translations that are the same as the one in the default language, that's useless.

Note: excluding all translation keys for each field is going to be a large number of excludes... but it supports wildcards, so we could exclude all, then include only the ones we care about. That way we don't even need to list all languages anywhere.

To execute this plan in a backwards-compatible way, we should either:

• Add the new translations objects at indexing, and use them at search time when present, but keep adding the old field at indexing time for now. Then remove it in a future push, having done a full reindex in the meantime.
• Or, add the code to use the new objects at search time when present, push that, then in a future push, add the new objects at indexing time instead of the old one (but keep the backwards-compatibility at search time, since you still have old data laying around). Then fully reindex to enjoy the full benefits.

Note: while we're at it, we should also use params() with proper exclude/include rather than extra(). That is purely for convenience / making our code more modern, it wouldn't change anything to the request though.