Repository navigation
SocketAsyncEngine.Unix perf experiments #14304
Description
I'm looking into ways that may improve SocketAsyncEngine.Unix performance.
Based on kestrel-linux-transport implementation, this is what I'm thinking of:
- Batch receives and sends on the epoll thread using Linux AIO. Note that this isn't the POSIX AIO library. The functions called here map directly to syscalls.
and
class SocketAsyncEventArgs
{
public bool RunContinuationsAsynchronously { get; set; } = true;
public bool PreferSynchronousCompletion { get; set; } = true;
}-
Setting
RunContinuationsAsynchronouslytofalseallowsSocketAsyncEngineto invoke callbacks directly from the epoll thread. This avoid context switching cost. -
Setting
PreferSynchronousCompletiontofalseallowsSocketAsyncEngineto not try a synchronous attempt and go async immediately. An implementation could set this totruefor sends, andfalsefor receives. Or for receives it could be changed based on whether the last receive filled up the buffer completely. Going async immediately saves an attempt syscall and allows batching.
The defaults match the current behavior.
io_uring is also an interesting option to explore. I'm not looking into that atm because the features we need are in 5.5 kernel which was released only a couple of days ago.
I'm going to do a poc in a separate repo for benchmarking. Anyone interested is free to review PRs, or fork and do some experimentation. When I have a working implementation, I'll need some help with benchmarking and see what is the change in perf.
cc @stephentoub @geoffkizer @davidfowl @halter73 @adamsitnik @benaadams @VSadov @damageboy @lpereira @dotnet/ncl
Activity
stephentoub commentedon Jan 31, 2020
Thanks, Tom.
Realistically, what does that mean in terms of when a) a typical developer/app would have the features available, and b) a motivated developer/app would have the features available? If io_uring made a substantial difference, seems like something we could do proactively and then have folks that really wanted the boost patch their configurations?
lpereira commentedon Jan 31, 2020
@tmds Have you seen the https://github.com/tkp1n/IoUring project? It's a transport similar to k-l-t, but using io_uring. It's being actively developed by a single person, but it's worth running a round of benchmarks with it in the Citrine machine just to have a feel of how much io_uring would make things better. I'll talk with @sebastienros today to arrange this.
lpereira commentedon Jan 31, 2020
I'm very positive it'll make a substantial difference. And, you're right, using
io_uringwill require some changes that users will need to know before they can enjoy the difference. Mainly, a new-ish kernel (5.5+, potentially 5.6+ when it releases), and some changes in the security limits (mainly the amount of pages a user process can lock).We can also substitute (well, add and detect if it's available at runtime) the usage of epoll in
pal_networking.cand useio_uringas an epoll substitute. This should significantly reduce the syscall chatter there. (Other uses ofpoll(), like synchmgr and pal_io could move toio_uring, too. We might consider having a single ring for all these file descriptors that are watched and have these implementations use that instead of callingpoll()/epoll()directly, too, hiding all of these platform-specific stuff in a single place. (With time, other things could be changed to useio_uring: there are plenty of other operations that it can perform, including accepting sockets, connecting to a remote host, opening files, all asynchronously.)wfurt commentedon Jan 31, 2020
cc: @scalablecory as he was experimenting with
io_uringas well.benaadams commentedon Jan 31, 2020
Don't know about other distros, but Ubuntu 20.04 will be the next LTS, and is due to be released on April 23, 2020, it will have the 5.5 or later kernel and is currently in their daily builds.
tmds commentedon Jan 31, 2020
Probably not everyone is changing to a new LTS release when it comes out, but many keep on using the one they are at.
I think the Linux AIO option is worth exploring because it will reduce syscalls significantly by batching on the event loop thread, just like io_uring, but doesn't require a recent kernel.
io_uring is worth exploring also.
Yes, I'm following repo (which takes some good bits from k-l-t). And also interested to see benchmark results.
The syscall chatter for epoll in pal_networking.c is actually surprisingly low because EPOLLET is used.
(In retrospect I should have done the same thing in k-l-t.)
So advantageous of io_uring need to come from running operations on it.
benaadams commentedon Jan 31, 2020
Yes, but the LTS release is the low bar where it becomes easier to adopt (other than the previous LTS moving out of support)
If LTS OS is out in April with io_uring and .NET 5 is out in Nov, I'd hope it would use io_uring; rather than waiting for the next .NET release a year later.
tmds commentedon Jan 31, 2020
Ah, yes, if you want it, it's available in an LTS release.
Are you assuming io_uring will perform significantly better than Linux AIO for batching?
tmds commentedon Jan 31, 2020
I'm assuming it will be slightly better, but maybe not even measurable.
We need to measure to know.
Any feedback on the proposed properties? These apply also for io_uring backend.
PreferSynchronousCompletionneeds to befalsefor maximum batching.lpereira commentedon Jan 31, 2020
To be honest with you, Linux AIO is a disaster, for all sorts of different reasons. I wouldn't even consider supporting it in the first place if there's an alternative like
io_uring.benaadams commentedon Jan 31, 2020
I'd hope it would perform better in general as it was meant to overcome some of the issues with AIO https://kernel.dk/io_uring.pdf; and could also be used for file IO etc.
However; as you say just because io_uring would become more generally available, AIO would have wider support and the linux transport does perform better than Net.Sockets. So if there is the resource to do both...
Windows doesn't queue completions to the ThreadPool (which is why Kestrel has the additional ThreadPool queue for Windows but not on Linux). Is this because multiple threads can listen to the completion, while on Linux only one thread can listen so it needs to be more available to listen, whereas on Windows another thread can pick up the next events if one gets blocked?
tmds commentedon Jan 31, 2020
epoll_wait can be called from multiple threads (events go in the argument). io_uring can be called only from one thread (events are written to completion queue).
tmds commentedon Jan 31, 2020
And because requests are in a shared memory request queue, all requests need to come from that thread too.
This is different from epoll, epoll_ctl can be called from any thread.
16 remaining items
stephentoub commentedon Feb 21, 2020
Sounds great, @tmds. Thanks for the update. Looking forward to results when you've got 'em.
tmds commentedon Mar 17, 2020
hi, though it has been quit here, we (@antonfirsov, @adamsitnik, @tkp1n and myself) have been working on this, I'd like to share some intermediate conclusions:
RunContinuationsAsynchronouslytofalseprovides a way to make Kestrel receive loop run on io thread, so receives will be batched and don't have to be tried synchronously.PreferSynchronousCompletion=false) for batching regressed performance.kevingosse commentedon Oct 3, 2020
I was reminded of this thread thanks to Stephen's talk at dotnetos yesterday 😄
I'm wondering why (seemingly) nobody has tried mimicking the Windows I/O threadpool on Linux (this was suggested by Ben Adams as well in this thread I believe).
Unless I'm missing a fundamental detail, epoll's behavior is very close to IOCP if used with the EPOLLEXCLUSIVE flag. We could have an I/O threadpool on Linux using the same growing heuristics as on Windows, waiting on epoll, and inlining the callback. This way we avoid the hop to the threadpool and we keep the ability to scale up when a continuation hijacks the thread to execute a blocking operation.
epoll can't limit the number of active threads like IOCP does, but this can probably be emulated somehow.
Would it make sense for me to spend time prototyping this, or am I missing something?
stephentoub commentedon Oct 3, 2020
Thanks for watching. Hope you enjoyed it.
You're of course welcome to, and it'd be interesting to see what results you get. Thanks for offering.
However, I'm a bit skeptical you'll see wins. In particular, even if you solve the scaling mechanism, you'd likely need to change the buffer size passed to epoll wait. Today, a single wait can receive up to 1024 events. You'd likely need to make that 1, or else a single wait could get multiple events to be processed, and inlining of the processing of the Nth event could result in stalling all of those received in the batch after that. That would mean you might find yourself making 1024 epoll wait calls where previously you were only making 1, and I'd expect (though could be wrong) that's going to dwarf costs associated with the shuffle to the thread pool.
Note that for .NET 5, we changed how that handoff to the thread pool works, making it much more efficient, to the point where it's now competitive with @tmds's transport that does maintain one thread per core, inlines all processing, etc.
tmds commentedon Oct 6, 2020
This was implemented in #35330. Instead of dispatching each event from epoll separately, a batch of event that come from the epoll are now dispatched to the threadpool together.
I'm going to close this because it is no longer tracking anything on-going from my side.