Skip to content

[$500] [HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests #28172

Closed

Description

@muttmuure

https://expensify.slack.com/archives/C05LX9D6E07/p1695105894373289

Problem

When we're offline, we'll build up a ton of requests if we keep navigating. OpenReport is probably the biggest offender. That's bad because we'll process the whole queue once we're back online, and because of that requests that I actually need to execute when I'm online will have to wait until the whole queue is processed.

Let's say you have navigated to 5 reports while offline.

When you go online:

We'll start processing the queue.
Open report usually takes a couple seconds per request, so we'll wait 10s until you do all requests
Because we also queue the onyxUpdates for write requests, we'll need for the write operation for all of them to finish before we actually see anything in the report we just navigated

Solution

Only call OpenReport exactly once per report

Use a report metadata key to store if we have ever called the OpenReport for a specific report

Issue OwnerCurrent Issue Owner: @sakluger
Upwork Automation - Do Not Edit
  • Upwork Job URL: https://www.upwork.com/jobs/~01697b67b9aa3280da
  • Upwork Job ID: 1742376039165644800
  • Last Price Increase: 2024-01-03

Activity

  1. changed the title [-][HOLD on #reliable-updates] The whole queue of offline requests are processed when you come back online, delaying online requests[/-] [+]The whole queue of offline requests are processed when you come back online, delaying online requests[/+] on Oct 5, 2023
  2. roryabraham commented on Oct 19, 2023

    @roryabraham
    Contributor

    One idea for a solution to help with the scenario where there are many essentially duplicate requests in the queue would be to add an idempotencyKey to the request object. In the case of OpenReport, I think this could be some combination of the command name and reportID: OpenReport?reportID=1234. Then whenever we're adding a request to the sequentialQueue, we look for any requests with a matching idempotencyKey and only keep the newest one, replacing any older duplicate requests in the queue with the newest one.

    Using this technique we would de-dupe requests in the queue to help reduce the size of the queue when we come back online

    Before After
    image image

    Edit: You don't just "keep the newest instance of an idempotent request only, I think you'd want to replace the previous instance with the newest one, so that the order of operations is preserved

  3. roryabraham commented on Oct 19, 2023

    @roryabraham
    Contributor

    If we wanted to take it a step further, I have another idea building off an idea I've proposed before in a different context...

    Solution:

    Replace the sequential queue with a directed acyclic graph (DAG) of requests, which we can call the RequestGraph.

    • Each request is an object with an ID and an arbitrary set of data. Each request object represents a nodes in the graph.
    • Each request can have zero or more optional parentRequests. The presence of a parentRequest represents a directed edge in the graph.
    • API.write will add a node (and potentially an edge) to the graph. It will throw an error if the edge introduces a cycle in the graph. 🚫 ♻️
    • The RequestGraph, will continuously perform a parallel breadth-first search from every root node in the graph. i.e: whenever a request is added to the graph, find all root nodes in the graph, and for each:
      • Process the request, then process all child requests, then process all their children, etc...
      • If any parent request fails, then all its descendant requests are cancelled
      • Only once there are no more children originating from this root, delete the parent request and all descendants
    • Requests with a parentRequest can reference data from the parent request. Depending on the server response, a parent request can update its data and dependent children will be able to reference that updated data (eliminating the need for the HandleUnusedOptimisticID middleware).
    • Just like with the sequentialQueue, requests can have an idempotencyKey, and if a node with the matching idempotencyKey is found in the graph we replace the old request with the new one.

    This means that instead of processing write requests one-by-one in the sequential queue, we can process non-conflicting requests in parallel and only have to process requests that depend on other parent requests to run in sequence. This could massively improve the speed at which we process requests made offline, especially on a slower network.

    Before After
    image image

    It's worth nothing that this would be a pretty big change, so if we were going to roll this out I think the prudent thing to do would be to make the default behavior of the RequestGraph the same as the SequentialQueue. i.e: any time we add a request to the RequestGraph, we assume that its parentRequest is the last request added to the graph. Thus at first the graph ends up just being a linear queue as it is today.

    flowchart LR
        subgraph first requests
            A(Request A)
        end
    
        subgraph second requests
            B(Request B)
        end
    
        subgraph third requests
            C(Request C)
        end
    
        subgraph fourth requests
            D(Request D)
        end
    
        subgraph fifth requests
            E(Request E)
        end
    
        subgraph sixth requests
            F(Request F)
        end
    
        A --> B --> C --> D --> E --> F
        
    
    Loading

    Only when we're certain which parentRequest a new request is dependent upon, the developer will call API.write and provide the ID of a parentRequest. This will mean that request can happen as soon as that parent request is done, instead of having to wait for unrelated requests to finish.

    flowchart LR
        subgraph first requests
            A(Request A)
        end
    
        subgraph second requests
            B(Request B)
            D(Request D)
        end
    
        subgraph third requests
            C(Request C)
        end
    
        subgraph fourth requests
            E(Request E)
        end
    
        subgraph fifth requests
            F(Request F)
        end
    
        A --> B --> C
        A --> D
        C --> E --> F
        
    
    Loading

    If we know that a request is independent of any other requests in the graph, we can provide an empty array for parentRequests, and that would allow it to run immediately the next time the RequestGraph processes.

    flowchart LR
        subgraph first requests
            A(Request A)
            F(Request F)
        end
    
        subgraph second requests
            B(Request B)
            D(Request D)
        end
    
        subgraph third requests
            C(Request C)
        end
    
        subgraph fourth requests
            E(Request E)
        end
    
        A --> B --> C
        A --> D
        C --> E
        
    
    Loading
  4. WojtekBoman commented on Oct 23, 2023

    @WojtekBoman
    Contributor

    @muttmuure Can you assign me and @kosmydel to this issue?

  5. melvin-bot commented on Oct 23, 2023

    @melvin-bot

    📣 @WojtekBoman! 📣
    Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
    Please follow these steps:

    1. Make sure you've read and understood the contributing guidelines.
    2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
    3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
    4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
      Screen Shot 2022-11-16 at 4 42 54 PM
      Format:
    Contributor details
    Your Expensify account email: <REPLACE EMAIL HERE>
    Upwork Profile Link: <REPLACE LINK HERE>
    
  6. kosmydel commented on Oct 24, 2023

    @kosmydel
    Contributor

    Comment to enable the assignment :)

  7. WojtekBoman commented on Oct 24, 2023

    @WojtekBoman
    Contributor

    @roryabraham I have one question about the first proposed approach to solve this problem. Is the reportId parameter sufficient to make requests idempotent? I checked params of this request and besides the reportId it also accepts params: emailList, accountIDList etc. In the first approach if we send OpenReport request twice with the same reportID but with different additional parameters, we will get response only from the last sent request. Is this okay? I'd like to be sure that this how it should work.

  8. roryabraham commented on Oct 25, 2023

    @roryabraham
    Contributor

    Is the reportId parameter sufficient to make requests idempotent?

    It depends on the request. Each request may have it's own idempotencyKey, and it's up to us as developers to determine what data is appropriate to use for the idempotencyKey.

    In the first approach if we send OpenReport request twice with the same reportID but with different additional parameters, we will get response only from the last sent request. Is this okay?

    I think that is indeed ok for OpenReport. The only data that's written in OpenReport is when the user last read the report, so we only care about the last call.

  9. roryabraham commented on Oct 25, 2023

    @roryabraham
    Contributor

    if other params than reportID are included that means we are creating a new report

    @mountiny made a great point here that if we make a report optimistically OpenReport will have more params that we don't want to just throw away.

    So maybe what we want to do is – instead of just throwing away the earlier request and only keeping the later one, we should:

    • merge the params, optimisticData, successData, failureData of the later request into the earlier one
    • Keep the request in the same position in the queue that the earlier request had
  10. self-assigned this
    on Oct 25, 2023
  11. 50 remaining items

  12. melvin-bot commented on Jan 1, 2024

    @melvin-bot
  13. roryabraham commented on Jan 1, 2024

    @roryabraham
    Contributor

    @sakluger only action-item for you here is to issue a standard C+ review payment to @alitoshmatov for #32246. Thanks!

  14. added
    ExternalAdded to denote the issue can be worked on by a contributor
    on Jan 3, 2024
  15. melvin-bot commented on Jan 3, 2024

    @melvin-bot
  16. changed the title [-][HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/-] [+][$500] [HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/+] on Jan 3, 2024
  17. added
    Help WantedApply this label when an issue is open to proposals by contributors
    on Jan 3, 2024
  18. melvin-bot commented on Jan 3, 2024

    @melvin-bot

    Current assignee @alitoshmatov is eligible for the External assigner, not assigning anyone new.

  19. sakluger commented on Jan 3, 2024

    @sakluger
    Contributor

    @alitoshmatov I sent you an offer on Upwork, please let me know once you've accepted. Thanks!

  20. alitoshmatov commented on Jan 3, 2024

    @alitoshmatov
    Contributor

    @sakluger Accepted the offer

  21. sakluger commented on Jan 4, 2024

    @sakluger
    Contributor

    Completed payment, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionDailyKSv2ExternalAdded to denote the issue can be worked on by a contributorHelp WantedApply this label when an issue is open to proposals by contributorsNewFeatureSomething to build that is a new item.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions