Ignore invalid response header in HttpClient request

In .net standard 2.0
When i use System.Net.Http.HttpClient to request url data,throw an exception like:

The url run in web browser work well,but its response headers like:

The reponse headers is not standardized,in .net core it will throw an exception,but in framework4.5 work well.

Any solution for this in .net core.

@stephentoub

The reponse headers is not standardized,in .net core it will throw an exception,but in .NET Framework 4.5 works well.

This looks like another case where we need to adjust SocketsHttpHandler behavior to ignore (with logging) malformed header lines and not reject the response entirely.

I've just tested PlatformHandler/WinHTTP, it does not allow this. Will check Framework next.

@davidsh

I can't replicate this in 4.7.2 either. It gives the following misleading error (CR is indeed followed by LF):

WebException: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

However, Chrome, Firefox, and Curl do allow it. I think it would be reasonable (and trivial) to allow it in SocketsHttpHandler. Thoughts, @davidsh @wfurt @stephentoub?

@davidsh

WebException: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

I'm curious if the parsing will work if you add the 'UseUnsafeHeaderParsing' config item. It affects .NET Framework HttpWebRequest stack and in theory will affect HttpClient. You need to put this into the app.config for .NET Framework application.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
    </startup>
  <system.net>
    <settings>
      <httpWebRequest useUnsafeHeaderParsing ="true"/>
    </settings>
  </system.net>
</configuration>

https://docs.microsoft.com/en-us/dotnet/api/system.net.configuration.httpwebrequestelement.useunsafeheaderparsing?view=netframework-4.8

However, Chrome, Firefox, and Curl do allow it. I think it would be reasonable (and trivial) to allow it in SocketsHttpHandler. Thoughts, @davidsh @wfurt @stephentoub?

However, if this isn't a regression in .NET Core 3.0 compared with .NET Core 2.x and if this doesn't easily repro in .NET Framework without extra config settings, then this wouldn't meet the bar for a .NET Core 3.0 change.

Also, see related issue about malformed CR-LF sequences: #25319

Works with useUnsafeHeaderParsing="true".

Does not work on 2.2. Moving to Future.

@scalablecory

Triage: We should create a new setting or something like that (likely not API). Bonus: Use same switch name as .NET Framework has.
It will match .NET Framework and also browsers behavior (which @scalablecory tried).

In this specific case we should either ignore the line with date without header name (likely what .NET Framework does) or use the text prior to column ':' as key and the rest as value.

Having a similar issue with a webservice where the response headers contain a header with a '=' instead of '-' in the header name.

Any possible workarounds in Core 2.X/3.0 until the useUnsafeHeaderParsing switch is implemented? Tried with both System.Net.WebRequest and System.Net.Http.HttpClient, and get the same error.

I'm trying to connect to a State Server for storing data (yes, I know it's better to use SQL Server or Redis cache, but I can't install anything on the server or create tables on the SQL Server, and I need to survive application restarts), but the state server responds with an invalid header, it doesn't send the HTTP1.1 part, but just '200 OK'. Because of that, HttpClient throws an exception:

'Received an invalid status line: '200 OK'.'

So I need a way to ignore those errors. In the meantime I use TcpClient to send and receive the data, but I rather use HttpClient.

@ChrisVanDijk

@ChrisVanDijk that is not invalid header, that is invalid response format of the first line.
This is not something covered by this issue.

Personally I don't think it is something we should be resilient to, unless we find out it is super-wide spread.
Do you know what the server is? Is there a chance to fix it instead?

@karelz

@karelz it't the good old ASP.NET State Service, according to the documentation it should respond with a standard response format, but it doesn't. Maybe that's a bug, or they did it on purpose.

I don't think this happens a lot, and I hope nobody connects to the state service outside of the old ASP.NET world. I shouldn't use it either, but for this project I don't have another option. I've fixed it by using TcpClient, which works fine for cases like this.

I have the same issue and unfortunately the server is out of my reach and cannot be fixed to return proper HTTP, so I had to go a bit lower on the network stack and send an HTTP message using TCP.

I got the original code from here but it was broken, so I had to fix it and refactored it a bit to make it cleaner and async. I didn't test the SSL code at all because I didn't need it.

I'll share the code here, but I don't claim that it's in any way production ready or safe:

//using System.Net.Security;
//using System.Net.Sockets;
//using System.Security.Authentication;
//using System.Security.Cryptography.X509Certificates;

public static class TcpClientForHttp
{
    public static Task<MemoryStream> SendHttpRequestAsync(
        Uri uri,
        HttpMethod httpMethod,
        Version httpVersion,
        CancellationToken cancellationToken)
    {
        string strHttpRequest = $"{httpMethod} {uri.PathAndQuery} HTTP/{httpVersion}\r\n";
        strHttpRequest += $"Host: {uri.Host}:{uri.Port}\r\n";
        // Any other HTTP headers can be added here ....
        strHttpRequest += "\r\n";

        return SendWebRequest(uri, strHttpRequest, cancellationToken);
    }

    private static async Task<MemoryStream> SendWebRequest(Uri uri, string request, CancellationToken cancellationToken)
    {
        bool isHttps = uri.Scheme == Uri.UriSchemeHttps;

        using var tcpClient = new TcpClient();
        await tcpClient.ConnectAsync(uri.Host, uri.Port, cancellationToken);
        await using NetworkStream ns = tcpClient.GetStream();

        var resultStream = new MemoryStream();

        if (isHttps)
        {
            await using var ssl = new SslStream(ns, false, ValidateServerCertificate, null);
            await ssl.AuthenticateAsClientAsync(new SslClientAuthenticationOptions
                {
                    TargetHost = uri.Host,
                    ClientCertificates = null,
                    EnabledSslProtocols = SslProtocols.None,
                    CertificateRevocationCheckMode = X509RevocationMode.NoCheck
                },
                cancellationToken);

            await using var sslWriter = new StreamWriter(ssl);
            await sslWriter.WriteAsync(request);
            await sslWriter.FlushAsync();

            await ssl.CopyToAsync(resultStream, cancellationToken);
        }
        else
        {
            // Normal HTTP
            await using var nsWriter = new StreamWriter(ns);
            await nsWriter.WriteAsync(request);
            await nsWriter.FlushAsync();

            await ns.CopyToAsync(resultStream, cancellationToken);
        }

        resultStream.Position = 0;
        return resultStream;
    }

    private static bool ValidateServerCertificate(
        object sender,
        X509Certificate? certificate,
        X509Chain? chain,
        SslPolicyErrors sslPolicyErrors)
    {
        return true; // Accept all certs
    }
}

Ironically, MS's own Graph API sometimes returns invalid HTTP. Instead of starting the response with a standard HTTP/1.1 blahblah string it just sends "0" (zero). And the error I get is received an invalid status line: '0'.

It'd be great if HttpClient was able to ignore/override those errors so I can look at the actual response.

P.S. This is under .NET 6

Ironically, MS's own Graph API sometimes returns invalid HTTP. Instead of starting the response with a standard HTTP/1.1 blahblah string it just sends "0" (zero). And the error I get is received an invalid status line: '0'.

It'd be great if HttpClient was able to ignore/override those errors so I can look at the actual response.

P.S. This is under .NET 6

I arrived here investigating the exact same issue in a .NET 7 app. 😣 We get the same HttpRequestException: Received an invalid status line: '0'. exception message

@tsulli

@tsulli yes, it looks like MS Graph API has introduced some "intelligent" WAF that blocks some requests and returns invalid HTTP-message, that HttpClient cannot parse.

P.S. (offtopic) Theresa, FWIW we were able to work this around by using request-batching https://learn.microsoft.com/en-us/graph/json-batching?context=graph%2Fapi%2F1.0&view=graph-rest-1.0

@alex-jitbit

@alex-jitbit We'll give that a shot, thanks for the tip!

@AnthonyMastrean

The problem is that if the response is structurally broken it is difficult to even start processing. That is irrelevant to who generates it and why.
The example @AnthonyMastrean provided is quire different IMHO e.g. some extra characters in header -> that is something that can be dealt with.

Ironically, MS's own Graph API sometimes returns invalid HTTP. Instead of starting the response with a standard HTTP/1.1 blahblah string it just sends "0" (zero). And the error I get is received an invalid status line: '0'.

It'd be great if HttpClient was able to ignore/override those errors so I can look at the actual response.

P.S. This is under .NET 6

Sorry for dropping in on this thread, but I've also encountered this issue. It happens sometimes when updating some emails using Graph. Did you, perhaps, find a workaround?

retries? It seems to me that the right move is the push and fix Graph API to produce valid HTTP.

@Vixan

@Vixan yes, found a workaorund, mentioned here: #29927 (comment)

I've spent a couple of days trying to figure out how to work around this issue by filtering out invalid headers before ParseHeaderNameValue() attempts to parse them, but haven't figured it out. Can someone please help finish this code or share some other workaround?

SocketsHttpHandler handler = new();

handler.PlaintextStreamFilter = (context, token) =>
{    
    // how do I remove invalid headers here?                        


    return ValueTask.FromResult(context.PlaintextStream);    
};

HttpClient client = new(handler);
HttpResponseMessage response = client.GetAsync("www.google.com").Result;

I've spent a couple of days trying to figure out how to work around this issue by filtering out invalid headers before ParseHeaderNameValue() attempts to parse them, but haven't figured it out. Can someone please help finish this code or share some other workaround?
SocketsHttpHandler handler = new();

handler.PlaintextStreamFilter = (context, token) =>
{    
    // how do I remove invalid headers here?                        


    return ValueTask.FromResult(context.PlaintextStream);    
};

HttpClient client = new(handler);
HttpResponseMessage response = client.GetAsync("www.google.com").Result;

Did you solve this problem?

Did you solve this problem?

Sorry, I eventually solved the problem and finished my program, but I don't remember how I did it now. I see my code uses the NuGet packages "Selenium.WebDriver.ChromeDriver" and "HtmlAgilityPack" for something. And I see I created a separate C# console app project I named "FileDownloader" dedicated to just downloading files, which my main project calls using Process.Start() and passing in the URL to download as an argument. I think I had to use that separate project to work around the issue with the invalid headers. That FileDownloader project calls this method just before using HttpClient to download the file, but I couldn't say now whether it was the solution or just something I tried and never removed:

public static bool SetAllowUnsafeHeaderParsing20()
        {
            //Get the assembly that contains the internal class
            Assembly aNetAssembly = Assembly.GetAssembly(typeof(System.Net.Configuration.SettingsSection));
            if (aNetAssembly != null)
            {
                //Use the assembly in order to get the internal type for the internal class
                Type aSettingsType = aNetAssembly.GetType("System.Net.Configuration.SettingsSectionInternal");
                if (aSettingsType != null)
                {
                    //Use the internal static property to get an instance of the internal settings class.
                    //If the static instance isn't created already, the property will create it for us.
                    object anInstance = aSettingsType.InvokeMember("Section", BindingFlags.Static | BindingFlags.GetProperty | BindingFlags.NonPublic, null, null, new object[] { });

                    if (anInstance != null)
                    {
                        //Locate the private bool field that tells the framework is unsafe header parsing should be allowed or not
                        FieldInfo aUseUnsafeHeaderParsing = aSettingsType.GetField("useUnsafeHeaderParsing", BindingFlags.NonPublic | BindingFlags.Instance);
                        if (aUseUnsafeHeaderParsing != null)
                        {
                            aUseUnsafeHeaderParsing.SetValue(anInstance, true);
                            return true;
                        }
                    }
                }
            }
            return false;
        }

Thank you very much for your answer, but useUnsafeHeaderParsing="true" is only applicable to net 4.8.1 and below, and I finally use Tcp to download.

useUnsafeHeaderParsing="true" is only applicable to net 4.8.1 and below

Thanks, that helped me remember my workaround better: I set up the FileDownloader project as .Net 4.8.1 specifically for that reason. The rest of my projects are .Net 7, and those just run the FileDownloader executable whenever they need to download a file that may get an invalid response header. If Microsoft fixes this problem in a newer .Net version, I can get rid of the FileDownloader project and just use HttpClient in the main project.

Hi, you need to wrap the stream with your own custom stream implementation. Something like:

SocketsHttpHandler handler = new();

handler.PlaintextStreamFilter = (context, token) =>
{    
    // how do I remove invalid headers here?                        

    var streamWrapper = new HttpStreamWithHeaderReplacement(context.PlaintextStream, <Replacement details config> );
    return ValueTask.FromResult(streamWrapper);    
};

HttpClient client = new(handler);
HttpResponseMessage response = client.GetAsync("www.google.com").Result;

I think MS needs to really do a working example of this. You essentially need to buffer the stream ahead to get the entire header, do the replacement / transformation and then exhaust the buffer as a response, then carry on forwarding stream requests to the underlying stream.

Here's a version we have been working on in development (caveat, it needs more testing / optimisation work but should get you in the right direction):

    internal class HttpStreamWithHeaderReplacement : Stream
    {
        private readonly Stream _originalStream;
        private readonly IList<IHttpStreamReplacement> _replacements;
        public bool ForceExit { get; private set; }
        public bool HeadersFound { get; private set; } = false;
        public int BufferOffset { get; private set; } = 0;

        public HttpStreamWithHeaderReplacementPool(Stream originalStream, IList<IHttpStreamReplacement> replacements)
        {
            _originalStream = originalStream;
            _replacements = replacements;
        }

        public override int Read(byte[] buffer, int offset, int count)
            => Read(buffer.AsSpan().Slice(offset, count));

        public override int Read(Span<byte> buffer)
        {
            while (!HeadersFound && !ForceExit)
            {
                EnsureBuffer(buffer.Length);
                var readBufferSpan = _internalMemoryBuffer.Memory.Span.Slice(_usedBufferLength, buffer.Length);

                var chunkLength = _originalStream.Read(readBufferSpan);
                if (chunkLength > 0)
                {
                    _usedBufferLength += chunkLength;
                    var usedBufferSpan = _internalMemoryBuffer.Memory.Span[.._usedBufferLength];

                    var s = Encoding.ASCII.GetString(usedBufferSpan);

                    var doubleCRLFMatch = Regex.Match(s, @"^\s*$", RegexOptions.Multiline); // Find double CR / Empty line
                    if (doubleCRLFMatch.Success)
                    {
                        // Found the header
                        var headerLength = doubleCRLFMatch.Index;
                        var headers = s[..headerLength];
                        var newHeaders = headers;
                        var hasReplacement = false;
                        foreach (var findReplace in _replacements)
                        {
                            hasReplacement |= findReplace.TryReplaceContent(newHeaders, out newHeaders);
                        }

                        if (hasReplacement)
                        {
                            var remainderBufferSpan = usedBufferSpan[headerLength..];
                            CombineAndReplaceBuffer(newHeaders, remainderBufferSpan);
                        }
                        HeadersFound = true;
                    }

                    // Completely arbitrary 20K buffer limit to stop reading forever (safety valve)
                    if (_usedBufferLength > 20000) 
                    {
                        ForceExit = true; // give up
                    }
                }
                else
                {
                    // EOF
                    break;
                }
            }

            // Check if buffer needs to be exhausted
            if (BufferOffset >= _usedBufferLength)
            {
                // Buffer is empty
                // - Just return from the normal stream
                return _originalStream.Read(buffer);
            }
            else
            {
                // Take a chunk off the buffer
                var actualCount = buffer.Length;
                if (BufferOffset + buffer.Length > _usedBufferLength)
                {
                    // Need to truncate the count as its shorter than the remaining internal buffer
                    actualCount = _usedBufferLength - BufferOffset;
                }
                var blockSpan = _internalMemoryBuffer!.Memory.Slice(BufferOffset, actualCount);
                blockSpan.Span.CopyTo(buffer);
                BufferOffset += actualCount;
                return actualCount;
            }
        }

        private int _usedBufferLength = 0;
        private IMemoryOwner<byte>? _internalMemoryBuffer = null;

        [MemberNotNull(nameof(_internalMemoryBuffer))]
        public void EnsureBuffer(int additionalBytes)
        {
            if (_internalMemoryBuffer == null)
            {
                _internalMemoryBuffer = MemoryPool<byte>.Shared.Rent(additionalBytes);
            }
            else if (_internalMemoryBuffer.Memory.Length < _usedBufferLength + additionalBytes)
            {
                var newBuffer = MemoryPool<byte>.Shared.Rent(_usedBufferLength + additionalBytes);
                _internalMemoryBuffer.Memory[.._usedBufferLength].CopyTo(newBuffer.Memory);

                ExchangeBuffer(newBuffer, _usedBufferLength);
            }
        }

        public void CombineAndReplaceBuffer(ReadOnlySpan<char> sourceHeader, ReadOnlySpan<byte> sourceRemainingBuffer)
        {
            var newBufferLength = sourceHeader.Length + sourceRemainingBuffer.Length;
            var replacedBuffer = MemoryPool<byte>.Shared.Rent(newBufferLength);

            // Copy the new Header Bytes into the new Buffer
            var newHeaderLength = Encoding.ASCII.GetBytes(sourceHeader, replacedBuffer.Memory.Span);

            if (sourceRemainingBuffer.Length > 0)
            {
                // Copy the new the remainder Bytes into the new Buffer
                var targetRemainderSpan = replacedBuffer.Memory.Span[newHeaderLength..];
                sourceRemainingBuffer.CopyTo(targetRemainderSpan);
            }

            ExchangeBuffer(replacedBuffer, newBufferLength);
        }

        public void ExchangeBuffer(IMemoryOwner<byte> newBuffer, int usedBufferLength)
        {
            _internalMemoryBuffer?.Dispose();
            _internalMemoryBuffer = newBuffer;
            _usedBufferLength = usedBufferLength;
        }

        protected override void Dispose(bool disposing)
        {
            _internalMemoryBuffer?.Dispose();
            _internalMemoryBuffer = null;
            _originalStream?.Dispose();

            base.Dispose(disposing);
        }
        
        public override bool CanRead => _originalStream.CanRead;

        public override bool CanSeek => false;

        public override bool CanWrite => _originalStream.CanWrite;

        public override long Length => _originalStream.Length;

        public override long Position { get => _originalStream.Position; set => throw new NotSupportedException(); }

        public override void Flush()
        {
            _originalStream.Flush();
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            throw new NotSupportedException();
        }

        public override void SetLength(long value)
        {
            throw new NotSupportedException();
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            _originalStream.Write(buffer, offset, count);
        }
    }

The IHttpStreamReplacement was declared like this, so you can implement your own version of this to do what you want:

    public interface IHttpStreamReplacement
    {
        bool TryReplaceContent(string content, out string result);
    }

Navigation Menu

Repository navigation

Ignore invalid response header in HttpClient request #29927

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ignore invalid response header in HttpClient request #29927

Description

Activity

davidsh commented on Jun 21, 2019

scalablecory commented on Jun 22, 2019

scalablecory commented on Jun 22, 2019

davidsh commented on Jun 22, 2019

scalablecory commented on Jun 27, 2019

karelz commented on Oct 8, 2019

FullstackSensei commented on Oct 23, 2019

ChrisVanDijk commented on Mar 2, 2020

karelz commented on Mar 2, 2020

ChrisVanDijk commented on Mar 3, 2020

14 remaining items

sherman89 commented on May 20, 2022

alex-jitbit commented on Feb 27, 2023

tsulli commented on Feb 28, 2023

alex-jitbit commented on Feb 28, 2023

tsulli commented on Feb 28, 2023

wfurt commented on Feb 28, 2023

Vixan commented on Mar 10, 2023

wfurt commented on Mar 10, 2023

alex-jitbit commented on Mar 10, 2023

Germs2004 commented on Mar 16, 2023

AmberHan commented on Jul 13, 2023

Germs2004 commented on Jul 14, 2023

AmberHan commented on Jul 14, 2023

Germs2004 commented on Jul 14, 2023

Simon-Gregory-LG commented on Jul 14, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions