Background and motivation
Prototype: main...MihaZupan:http-connectionEviction
SocketsHttpHandler exposes logic for establishing new connections (namely the ConnectCallback), and some properties to control how long connections are kept around (PooledConnectionLifetime, PooledConnectionIdleTimeout).
Since DNS is only consulted when a connection is established, a common approach to ensure that the application responds to DNS changes is to configure the PooledConnectionLifetime, forcing the handler to periodically recycle connections.
But in the majority of cases, nothing actually changed, and you're forced to throw away perfectly good connections that have already been warmed up - doing all the handshakes, and possibly expanding TCP windows. This is clearly not ideal for latency.
There is also no good way to currently force connections to be shut down, short of tearing down the transport streams from under the client, or manually load balancing requests across multiple handlers.
I'm proposing we expose a callback that gives the user the option to decide whether connections should be evicted early.
The callback would not be guaranteed to run on every request. Instead, it would happen sometimes in the background. My current prototype works by periodically calling these callbacks from maintenance timers we already have.
If the user returned false in the callback, that would mark the connection for eviction, and from then on we wouldn't schedule new requests on that connection, but pending ones would be allowed to finish normally. Existing scavenging logic would eventually throw away any idle marked connections.
The callback may be invoked multiple times in parallel for a given handler instance, but only once at a time for a given connection.
To make it possible to associate custom state with a connection, we should expose the long ConnectionId, a unique identifier for a given connection within the current process. The concept is an existing one since we already expose connection IDs via public telemetry.
The RemoteEndPoint property would expose the same info we currently report in telemetry, which we extract from the socket if possible (hence nullable).
API Proposal
namespace System.Net.Http;
public partial sealed class SocketsHttpHandler
{
// Existing
// public Func<SocketsHttpConnectionContext, CancellationToken, ValueTask<Stream>>? ConnectCallback { get; set; }
// public Func<SocketsHttpPlaintextStreamFilterContext, CancellationToken, ValueTask<Stream>>? PlaintextStreamFilter { get; set; }
[Experimental]
public Func<SocketsHttpConnectionEvictionContext, CancellationToken, Task<bool>>? ShouldEvictConnection { get; set; }
}
[Experimental]
public sealed class SocketsHttpConnectionEvictionContext // New type
{
// No public ctor
public TimeSpan Age { get; }
public long ConnectionId { get; }
public Version HttpVersion { get; }
public DnsEndPoint DnsEndPoint { get; }
public IPEndPoint? RemoteEndPoint { get; }
}
public partial sealed class SocketsHttpConnectionContext // Existing type used for ConnectCallback
{
[Experimental]
public long ConnectionId { get; }
}
public partial sealed class SocketsHttpPlaintextStreamFilterContext // Existing type used for PlaintextStreamFilter
{
[Experimental]
public long ConnectionId { get; }
}API Usage
Evict connections when DNS changes
var handler = new SocketsHttpHandler
{
PooledConnectionLifetime = Timeout.InfiniteTimeSpan,
ShouldEvictConnection = async (context, ct) =>
{
if (context.RemoteEndPoint is IPEndPoint connectionIp)
{
var addresses = await _cache.GetOrCreateAsync(
context.DnsEndPoint.Host,
async ct => await Dns.GetHostAddressesAsync(context.DnsEndPoint.Host, connectionIp.AddressFamily, ct),
cancellationToken: ct);
return !addresses.Contains(connectionIp.Address);
}
// This was a custom stream created via ConnectCallback (unreachable with default settings), fall back to a lifetime limit
return context.Age > TimeSpan.FromMinutes(10);
}
};
Evict based on information from a custom transport stream used via ConnectCallback
ConcurrentDictionary<long, WriteCountingStream> connections = [];
new SocketsHttpHandler
{
ConnectCallback = async (context, ct) =>
{
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
try
{
socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
// ...
var networkStream = new NetworkStream(socket, ownsSocket: true);
var stream = new WriteCountingStream(networkStream, context.ConnectionId, connections);
connections.TryAdd(context.ConnectionId, stream);
return stream;
}
catch
{
socket.Dispose();
throw;
}
},
ShouldEvictConnection = async (context, ct) =>
{
if (context.Age.TotalSeconds > 5 && // Grace period
connections.TryGetValue(context.ConnectionId, out WriteCountingStream stream))
{
double averageBitrate = stream.TotalBytesWritten / context.Age.TotalSeconds * 8;
return averageBitrate is < 10 * 1024 or > 1024 * 1024 * 1024; // Only keep connections with bitrates between 10 kbit and 1 gbit.
}
return false;
}
};
Evict connections with many consecutive 502s
Requires that ConnectionId be exposed on the request/response message (follow-up proposal #130108), otherwise it takes lots of ceremony to obtain the data from EventSource telemetry.
HttpMessageHandler handler = new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(10), // Eviction decisions complement each other
ShouldEvictConnection = async (context, ct) =>
{
if (_cache.TryGetValue(context.ConnectionId, out StrongBox<int> failureCounter))
{
return failureCounter.Value > 10;
}
return false;
}
};
handler = new Http502ResponseCounterHandler(handler, _cache);sealed class Http502ResponseCounterHandler(HttpMessageHandler handler, IMemoryCache cache) : DelegatingHandler(handler)
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
if (request.ConnectionId is long connectionId)
{
var counter = cache.GetOrCreate(connectionId, _ => new StrongBox<int>());
if (response.StatusCode == HttpStatusCode.BadGateway)
{
Interlocked.Increment(ref counter.Value);
}
else if (counter.Value > 0)
{
Interlocked.Decrement(ref counter.Value);
}
}
return response;
}
}
Alternative Designs
- The
RemoteEndPoint is typed as IPEndPoint? right now, but we could consider making it EndPoint? instead in case we later expose APIs on the ConnectCallback context that allow users to populate the info even if the stream isn't a NetworkStream. That would also help the gap in telemetry where we can't currently report the info.
HttpVersion seems like a reasonable name to me, but we've also used NegotiatedHttpVersion on SocketsHttpPlaintextStreamFilterContext.- I left the details about when/how often we run the callback as an implementation detail right now with the assumption that you normally have some grace period, so there's no need to recheck everything 10 times a second and immediately react. The prototype currently uses
Min(5 seconds, existing cleanup timer interval). The callback implementor can always rate limit calls themselves, or we could make it configurable in the future.
Risks
We could be accidentally exposing too much internal info. We're adding [Experimental] for now to keep the option to tweak the API in the future.
Background and motivation
Prototype: main...MihaZupan:http-connectionEviction
SocketsHttpHandlerexposes logic for establishing new connections (namely theConnectCallback), and some properties to control how long connections are kept around (PooledConnectionLifetime,PooledConnectionIdleTimeout).Since DNS is only consulted when a connection is established, a common approach to ensure that the application responds to DNS changes is to configure the
PooledConnectionLifetime, forcing the handler to periodically recycle connections.But in the majority of cases, nothing actually changed, and you're forced to throw away perfectly good connections that have already been warmed up - doing all the handshakes, and possibly expanding TCP windows. This is clearly not ideal for latency.
There is also no good way to currently force connections to be shut down, short of tearing down the transport streams from under the client, or manually load balancing requests across multiple handlers.
I'm proposing we expose a callback that gives the user the option to decide whether connections should be evicted early.
The callback would not be guaranteed to run on every request. Instead, it would happen sometimes in the background. My current prototype works by periodically calling these callbacks from maintenance timers we already have.
If the user returned
falsein the callback, that would mark the connection for eviction, and from then on we wouldn't schedule new requests on that connection, but pending ones would be allowed to finish normally. Existing scavenging logic would eventually throw away any idle marked connections.The callback may be invoked multiple times in parallel for a given handler instance, but only once at a time for a given connection.
To make it possible to associate custom state with a connection, we should expose the
long ConnectionId, a unique identifier for a given connection within the current process. The concept is an existing one since we already expose connection IDs via public telemetry.The
RemoteEndPointproperty would expose the same info we currently report in telemetry, which we extract from the socket if possible (hence nullable).API Proposal
API Usage
Evict connections when DNS changes
Evict based on information from a custom transport stream used via ConnectCallback
Evict connections with many consecutive 502s
Requires that
ConnectionIdbe exposed on the request/response message (follow-up proposal #130108), otherwise it takes lots of ceremony to obtain the data from EventSource telemetry.Alternative Designs
RemoteEndPointis typed asIPEndPoint?right now, but we could consider making itEndPoint?instead in case we later expose APIs on theConnectCallbackcontext that allow users to populate the info even if the stream isn't aNetworkStream. That would also help the gap in telemetry where we can't currently report the info.HttpVersionseems like a reasonable name to me, but we've also usedNegotiatedHttpVersiononSocketsHttpPlaintextStreamFilterContext.Min(5 seconds, existing cleanup timer interval). The callback implementor can always rate limit calls themselves, or we could make it configurable in the future.Risks
We could be accidentally exposing too much internal info. We're adding
[Experimental]for now to keep the option to tweak the API in the future.