You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[$500] [HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests #28172
When we're offline, we'll build up a ton of requests if we keep navigating. OpenReport is probably the biggest offender. That's bad because we'll process the whole queue once we're back online, and because of that requests that I actually need to execute when I'm online will have to wait until the whole queue is processed.
Let's say you have navigated to 5 reports while offline.
When you go online:
We'll start processing the queue.
Open report usually takes a couple seconds per request, so we'll wait 10s until you do all requests
Because we also queue the onyxUpdates for write requests, we'll need for the write operation for all of them to finish before we actually see anything in the report we just navigated
Solution
Only call OpenReport exactly once per report
Use a report metadata key to store if we have ever called the OpenReport for a specific report
Issue OwnerCurrent Issue Owner: @saklugerUpwork Automation - Do Not Edit
changed the title [-][HOLD on #reliable-updates] The whole queue of offline requests are processed when you come back online, delaying online requests[/-][+]The whole queue of offline requests are processed when you come back online, delaying online requests[/+]on Oct 5, 2023
One idea for a solution to help with the scenario where there are many essentially duplicate requests in the queue would be to add an idempotencyKey to the request object. In the case of OpenReport, I think this could be some combination of the command name and reportID: OpenReport?reportID=1234. Then whenever we're adding a request to the sequentialQueue, we look for any requests with a matching idempotencyKey and only keep the newest one, replacing any older duplicate requests in the queue with the newest one.
Using this technique we would de-dupe requests in the queue to help reduce the size of the queue when we come back online
Before
After
Edit: You don't just "keep the newest instance of an idempotent request only, I think you'd want to replace the previous instance with the newest one, so that the order of operations is preserved
If we wanted to take it a step further, I have another idea building off an idea I've proposed before in a different context...
Solution:
Replace the sequential queue with a directed acyclic graph (DAG) of requests, which we can call the RequestGraph.
Each request is an object with an ID and an arbitrary set of data. Each request object represents a nodes in the graph.
Each request can have zero or more optional parentRequests. The presence of a parentRequest represents a directed edge in the graph.
API.write will add a node (and potentially an edge) to the graph. It will throw an error if the edge introduces a cycle in the graph. 🚫 ♻️
The RequestGraph, will continuously perform a parallel breadth-first search from every root node in the graph. i.e: whenever a request is added to the graph, find all root nodes in the graph, and for each:
Process the request, then process all child requests, then process all their children, etc...
If any parent request fails, then all its descendant requests are cancelled
Only once there are no more children originating from this root, delete the parent request and all descendants
Requests with a parentRequest can reference data from the parent request. Depending on the server response, a parent request can update its data and dependent children will be able to reference that updated data (eliminating the need for the HandleUnusedOptimisticID middleware).
Just like with the sequentialQueue, requests can have an idempotencyKey, and if a node with the matching idempotencyKey is found in the graph we replace the old request with the new one.
This means that instead of processing write requests one-by-one in the sequential queue, we can process non-conflicting requests in parallel and only have to process requests that depend on other parent requests to run in sequence. This could massively improve the speed at which we process requests made offline, especially on a slower network.
Before
After
It's worth nothing that this would be a pretty big change, so if we were going to roll this out I think the prudent thing to do would be to make the default behavior of the RequestGraph the same as the SequentialQueue. i.e: any time we add a request to the RequestGraph, we assume that its parentRequest is the last request added to the graph. Thus at first the graph ends up just being a linear queue as it is today.
flowchart LR
subgraph first requests
A(Request A)
end
subgraph second requests
B(Request B)
end
subgraph third requests
C(Request C)
end
subgraph fourth requests
D(Request D)
end
subgraph fifth requests
E(Request E)
end
subgraph sixth requests
F(Request F)
end
A --> B --> C --> D --> E --> F
Loading
Only when we're certain which parentRequest a new request is dependent upon, the developer will call API.write and provide the ID of a parentRequest. This will mean that request can happen as soon as that parent request is done, instead of having to wait for unrelated requests to finish.
flowchart LR
subgraph first requests
A(Request A)
end
subgraph second requests
B(Request B)
D(Request D)
end
subgraph third requests
C(Request C)
end
subgraph fourth requests
E(Request E)
end
subgraph fifth requests
F(Request F)
end
A --> B --> C
A --> D
C --> E --> F
Loading
If we know that a request is independent of any other requests in the graph, we can provide an empty array for parentRequests, and that would allow it to run immediately the next time the RequestGraph processes.
flowchart LR
subgraph first requests
A(Request A)
F(Request F)
end
subgraph second requests
B(Request B)
D(Request D)
end
subgraph third requests
C(Request C)
end
subgraph fourth requests
E(Request E)
end
A --> B --> C
A --> D
C --> E
📣 @WojtekBoman! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:
Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
Format:
Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>
@roryabraham I have one question about the first proposed approach to solve this problem. Is the reportId parameter sufficient to make requests idempotent? I checked params of this request and besides the reportId it also accepts params: emailList, accountIDList etc. In the first approach if we send OpenReport request twice with the same reportID but with different additional parameters, we will get response only from the last sent request. Is this okay? I'd like to be sure that this how it should work.
Is the reportId parameter sufficient to make requests idempotent?
It depends on the request. Each request may have it's own idempotencyKey, and it's up to us as developers to determine what data is appropriate to use for the idempotencyKey.
In the first approach if we send OpenReport request twice with the same reportID but with different additional parameters, we will get response only from the last sent request. Is this okay?
I think that is indeed ok for OpenReport. The only data that's written in OpenReport is when the user last read the report, so we only care about the last call.
changed the title [-][HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/-][+][$500] [HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/+]on Jan 3, 2024
Activity
[-][HOLD on #reliable-updates] The whole queue of offline requests are processed when you come back online, delaying online requests[/-][+]The whole queue of offline requests are processed when you come back online, delaying online requests[/+]roryabraham commentedon Oct 19, 2023
One idea for a solution to help with the scenario where there are many essentially duplicate requests in the queue would be to add an
idempotencyKeyto the request object. In the case ofOpenReport, I think this could be some combination of the command name and reportID:OpenReport?reportID=1234. Then whenever we're adding a request to the sequentialQueue, we look for any requests with a matchingidempotencyKeyand only keep the newest one, replacing any older duplicate requests in the queue with the newest one.Using this technique we would de-dupe requests in the queue to help reduce the size of the queue when we come back online
Edit: You don't just "keep the newest instance of an idempotent request only, I think you'd want to replace the previous instance with the newest one, so that the order of operations is preserved
roryabraham commentedon Oct 19, 2023
If we wanted to take it a step further, I have another idea building off an idea I've proposed before in a different context...
Solution:
Replace the sequential queue with a directed acyclic graph (DAG) of requests, which we can call the
RequestGraph.parentRequests. The presence of aparentRequestrepresents a directed edge in the graph.API.writewill add a node (and potentially an edge) to the graph. It will throw an error if the edge introduces a cycle in the graph. 🚫 ♻️RequestGraph, will continuously perform a parallel breadth-first search from every root node in the graph. i.e: whenever a request is added to the graph, find all root nodes in the graph, and for each:parentRequestcan reference data from the parent request. Depending on the server response, a parent request can update its data and dependent children will be able to reference that updated data (eliminating the need for theHandleUnusedOptimisticIDmiddleware).idempotencyKey, and if a node with the matchingidempotencyKeyis found in the graph we replace the old request with the new one.This means that instead of processing write requests one-by-one in the sequential queue, we can process non-conflicting requests in parallel and only have to process requests that depend on other parent requests to run in sequence. This could massively improve the speed at which we process requests made offline, especially on a slower network.
It's worth nothing that this would be a pretty big change, so if we were going to roll this out I think the prudent thing to do would be to make the default behavior of the
RequestGraphthe same as theSequentialQueue. i.e: any time we add a request to theRequestGraph, we assume that itsparentRequestis the last request added to the graph. Thus at first the graph ends up just being a linear queue as it is today.flowchart LR subgraph first requests A(Request A) end subgraph second requests B(Request B) end subgraph third requests C(Request C) end subgraph fourth requests D(Request D) end subgraph fifth requests E(Request E) end subgraph sixth requests F(Request F) end A --> B --> C --> D --> E --> FOnly when we're certain which
parentRequesta new request is dependent upon, the developer will callAPI.writeand provide the ID of aparentRequest. This will mean that request can happen as soon as that parent request is done, instead of having to wait for unrelated requests to finish.flowchart LR subgraph first requests A(Request A) end subgraph second requests B(Request B) D(Request D) end subgraph third requests C(Request C) end subgraph fourth requests E(Request E) end subgraph fifth requests F(Request F) end A --> B --> C A --> D C --> E --> FIf we know that a request is independent of any other requests in the graph, we can provide an empty array for
parentRequests, and that would allow it to run immediately the next time theRequestGraphprocesses.flowchart LR subgraph first requests A(Request A) F(Request F) end subgraph second requests B(Request B) D(Request D) end subgraph third requests C(Request C) end subgraph fourth requests E(Request E) end A --> B --> C A --> D C --> EWojtekBoman commentedon Oct 23, 2023
@muttmuure Can you assign me and @kosmydel to this issue?
melvin-bot commentedon Oct 23, 2023
📣 @WojtekBoman! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:
Format:
kosmydel commentedon Oct 24, 2023
Comment to enable the assignment :)
WojtekBoman commentedon Oct 24, 2023
@roryabraham I have one question about the first proposed approach to solve this problem. Is the reportId parameter sufficient to make requests idempotent? I checked params of this request and besides the reportId it also accepts params: emailList, accountIDList etc. In the first approach if we send OpenReport request twice with the same reportID but with different additional parameters, we will get response only from the last sent request. Is this okay? I'd like to be sure that this how it should work.
roryabraham commentedon Oct 25, 2023
It depends on the request. Each request may have it's own
idempotencyKey, and it's up to us as developers to determine what data is appropriate to use for theidempotencyKey.I think that is indeed ok for
OpenReport. The only data that's written inOpenReportis when the user last read the report, so we only care about the last call.roryabraham commentedon Oct 25, 2023
@mountiny made a great point here that if we make a report optimistically
OpenReportwill have more params that we don't want to just throw away.So maybe what we want to do is – instead of just throwing away the earlier request and only keeping the later one, we should:
params,optimisticData,successData,failureDataof the later request into the earlier one50 remaining items
melvin-bot commentedon Jan 1, 2024
Triggered auto assignment to @sakluger (
NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details.roryabraham commentedon Jan 1, 2024
@sakluger only action-item for you here is to issue a standard C+ review payment to @alitoshmatov for #32246. Thanks!
melvin-bot commentedon Jan 3, 2024
Job added to Upwork: https://www.upwork.com/jobs/~01697b67b9aa3280da
[-][HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/-][+][$500] [HOLD for payment 2023-12-20] The whole queue of offline requests are processed when you come back online, delaying online requests[/+]melvin-bot commentedon Jan 3, 2024
Current assignee @alitoshmatov is eligible for the External assigner, not assigning anyone new.
sakluger commentedon Jan 3, 2024
@alitoshmatov I sent you an offer on Upwork, please let me know once you've accepted. Thanks!
alitoshmatov commentedon Jan 3, 2024
@sakluger Accepted the offer
sakluger commentedon Jan 4, 2024
Completed payment, thanks!