How to get response content without storing it in MemoryStream

[Racer396]

Hello!
Before v107 i use DownloadData with AdvancedResponseWriter, and as i remember, it doesn't store ResponseContent in memory and allow to write something like this:

               var request = new RestRequest(downloadUri, Method.GET);
                request.AdvancedResponseWriter = (stream, response) =>
                {
                    using var fStream = File.OpenWrite(filePath);
                    var buffer = new byte[1024*64];
                    long totalCount = 0;
                    while (totalCount != response.ContentLength)
                    {
                        var count = stream.Read(buffer, 0, buffer.Length);
                        totalCount += count;
                        fStream.Write(buffer, 0, count);
                    }
                };
                _client.DownloadData(request);

Now in v107.3, when i use DownloadDataAsync or just ExecuteAsync, it store all data in memory and only after that return me the MemoryStream.
How can i avoid storing ResponseContent in memory and write it in the file as is?
Because in my case, i need to downloading big files, which size can be a few gigabytes, and storing it in memory, on clients desktop PC - looks like a bad idea.

RestSharp/src/RestSharp/Response/ResponseHandling.cs

Line 38 in 17a3532

public static Task<Stream?> ReadResponse(this HttpResponseMessage response, CancellationToken cancellationToken) {

[alexeyzimarev]

Use DownloadStreamAsync https://restsharp.dev/usage.html#downloading-binary-data

[Racer396]

But the question was about DownloadStreamAsync .
Because when i use it, it store all data in memory, in MemoryStream, and only after that, it returns stream.
In first message, i ref a method from ResponseHandling, and if this method is used for DownloadAsyncStream, i dont now how it may work async as i want, without storing all in memory, because of - https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpcontent.readasstreamasync?view=net-6.0

The returned Task object will complete after all of the stream that represents content has been read.

It's returns stream only after all content was read.
Therefore, this means we have memory stream, that read all response in it, and only after that return to me this memory stream from DownloadStreamAsync method.

May be i do something wrong, and i would appreciate it if you could give an example of the correct usage.

[Racer396]

And intresting discussion about that in dotnet repo: dotnet/runtime#31316

[alexeyzimarev]

What the referenced issue states is not exactly right. If you look at StreamJsonAsync, it uses DownloadDataAsync and continuously reads the stream. The stream content is available as soon as it gets something from the server.

RestSharp/src/RestSharp/RestClientExtensions.cs

Lines 325 to 350 in cd4c434

	public static async IAsyncEnumerable<T> StreamJsonAsync<T>(
	this RestClient client,
	string resource,
	[EnumeratorCancellation] CancellationToken cancellationToken
	) {
	var request = new RestRequest(resource) { CompletionOption = HttpCompletionOption.ResponseHeadersRead };
	#if NETSTANDARD
	using var stream = await client.DownloadStreamAsync(request, cancellationToken).ConfigureAwait(false);
	#else
	await using var stream = await client.DownloadStreamAsync(request, cancellationToken).ConfigureAwait(false);
	#endif
	if (stream == null) yield break;
	var serializer = client.Serializers[DataFormat.Json].GetSerializer();
	using var reader = new StreamReader(stream);
	while (!reader.EndOfStream && !cancellationToken.IsCancellationRequested) {
	var line = await reader.ReadLineAsync().ConfigureAwait(false);
	if (string.IsNullOrWhiteSpace(line)) continue;
	var response = new RestResponse { Content = line };
	yield return serializer.Deserializer.Deserialize<T>(response)!;
	}
	}

The key here is this line:

RestSharp/src/RestSharp/RestClientExtensions.cs

Line 330 in cd4c434

var request = new RestRequest(resource) { CompletionOption = HttpCompletionOption.ResponseHeadersRead };

[Racer396]

CompletionOption = HttpCompletionOption.ResponseHeadersRead
Yes, that what i need. Thank you so much! :)

[kendallb]

I just ran into this also. Wouldn't it make more sense to always set that completion option for DownloadStreamAsync? Otherwise DownloadStreamAsync really is not much point, because it defeats the purpose of actually downloading into a stream?

[kendallb]

Also that would make DownloadDataAsync be more efficient, as currently it will end up downloading the data first into a memory stream, which is then streamed into a byte[] array to return the caller. If it was actually streaming the response content, then the byte[] array is built more efficiently (internally it's streamed into a memory stream 16K at a time which is then turned into the byte array). So if we are downloading say a 2MB file, there will be a 2MB memory stream created by the DownloadStreamAsync() function, then another 2MB memory stream that the data gets copied into, which then gets turned into the final byte array.

[kendallb]

Here is a PR

#1825