Issue with SqlConnection.Open on netcore3 linux images

I am having an issue when trying to open a connection to our SQL Server 2016 SP1 CU7 server from latest (not sure if it happens with previous) netcore 3 preview 9. The code just hangs when calling Open (same result with OpenAsync) and doesn't continue. Tried with both Microsoft.Data.SqlClient and System.Data.SqlClient NuGet packages.

Same code works fine when executed in my windows laptop outside Docker. If I switch to netcore 2.2 it works fine inside docker (changing images as well).

I have created a small repo with a repro: https://github.com/adrian-lopez-softtek/NetCore3SqlOpenIssue/

Tried with both images (the commented ones in the Dockerfile) with same result.

Any help would be appreciated, as I'm creating a new project with version 3 and don't want to switch to 2.2 at this point, but if I'm not able to make it work I will probably have to.

Thanks!

I did one more test connecting to one a Azure SQL database, and with this one works fine. Our on premise server version is 13.0.4466.4 vs cloud 12.0.2000.8, in case this helps

Managed to get it working using preview 9 ubuntu images (3.0.100-preview9-disco & 3.0.0-preview9-disco). Not sure what happens with debian images, but they didn't work for me.

@adrian-lopez-softtek

-disco resolved this for us with SQL Server 2014 and Linux containers - You are a lifesaver @adrian-lopez-softtek !

Just tested this with RC1 and new debian docker images and the issue is still there

I am having the same issue: program connects just fine from my laptop and hangs on.

In a tcpdump I can see that there is a TDS7 pre-login request and response being sent, but the connection hangs before the credentials are checked. (Tested by sending bad credentials to the server with the same result as the correct credentials.)

I can confirm that the Ubuntu image does connect.

There are a few issues out there about hangs but this looks the freshest. We are seeing a deadlock between SNIMarsHandle and SNIMarsConnection. When we disable multiple active result sets on the connection string, the issue is gone.

snimarshandle handleack -- lock(this)
snimarsconn handlereceivingcomplete -- lock(this)

snimarsconn sendasync -- lock(this)
snimarshandle internalsendasync  -- lock (this)

@aidanjryan

Hi @aidanjryan

As we have more MARS related issues reported and users have reported regressions that occurred in System.Data.SqlClient. Could you test this case with System.Data.SqlClient 4.6.1 and confirm if that was a working version? We're trying to reach the root cause of the problems.

@cheenamalhotra

@cheenamalhotra we were using System.Data.SqlClient with the same connection string, same client code, with no issues until changing to .NET Core 3.0 preview 7. I assume that was System.Data.SqlClient 4.6.1.

@adrian-lopez-softtek

Hello - we ran in exactly the same issue with both SqlClient-Packages (System.Data + Microsoft.Data). When we used the original Microsoft baseimages for netcore 3.0 we had this issue. When we changed to Ubuntu 18.04 (3.0-bionic) everything worked fine - thanks to @adrian-lopez-softtek for this hint!

I am not sure if it's the same issue, in my case, the first time after I deployed a simple dotnet core 3.0 web api to linux image, sqlconnection open failed or takes around 100 seconds to just open the connection, then the database connection works well and very fast, after around 5 or 10 minutes, again the sqlconnection.open takes 100 seconds every time unless I redeploy the web api again.
I tried sqlserver 2008 and 2016, also I tried dotnet core 2.1 and 3.0, also tried Sysyem.Data.Sqlclient and Microsoft.Data.SqlClient, also tried -bionic and -disco images, all the same issue.
Appreciate if anyone has some hints.
Thanks.

@gfzhang8

@gfzhang8 Do you have MultipleActiveResultSets=True in your connection string? Disabling this resolved the hang for us.

@aidanjryan

@aidanjryan Thanks, but MultipleActiveResultSets=True is not in my connection string.

Additionally, I can reproduce the issue every time after redeploy, the first time fails or takes 100 second, then works well and connection open fast, after 5 or 10 minutes, fails or takes 100 second, which is the most strange thing to me, why the hang-time is exactly 100 seconds every time even when fails.
BTW, when connection open fail, the exception is something like 'cannot write to transport...', will attach the exact exception stack later.
And I confirmed the same code deployed on a Windows server has no issue to connect to the same database with same connection string.

SQL Server 2016 still supports older TLS versions and it is possible TLS 1.2 is disabled.

You can confirm which TLS version is in use by capturing network packets when you create connection with Wireshark and looking into pre-login handshake packets. You will find "ClientHello" and "ServerHello" packets that also contain information about TLS version in use.

e.g.

@cheenamalhotra

Hey @cheenamalhotra thanks for the links. Any plans on giving a proper error/exception when server doesn't support TLS1.2 ? Current behavior is a bit misleading and requires some digging to find out what is really happening.

Thanks again! 😄

@adrian-lopez-softtek

@adrian-lopez-softtek
Yes we are looking into providing proper error message if possible.

Looks like you have to add SQL Server certificate to trusted in order to utilize TLS1.2. Copy root certificate to /usr/local/share/ca-certificates in container and run update-ca-certificates

@epignosisx

Faced a similar issue in my work setup connecting to a SQL Server 2014 (12.0.6108.1). In my case the server had TLS1.2 enabled and working which I was able to verify through network monitor. But the problem was due to this Seclevel=2 setting in the Debian 10 openssl.cnf as mentioned by @epignosisx

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

I changed only the SECLEVEL=1 and was able to connect from the Debian images. Digging deeper into Seclevel it seems that level 2 is more strict on the cipher suites (https://www.openssl.org/docs/man1.1.0/man1/ciphers.html) Especially it leaves out SHA1 which is the signature algorithm used in the self signed certificates in SQL Server (until 2016) used for connection login data encryption (ref). Probably why @cheenamalhotra's tests with SQL Server 2016 & 2017 worked.

And as expected changing the certificate on my server to use a SHA256 did the trick. (Used this as ref)

@iinuwa

I tried our setup again and investigated different docker images:

with netcore 2.1 the sql client connects without problems to our server

with netcore 3.0:

mcr.microsoft.com/dotnet/core/runtime-deps:3.0
the sql client hangs in a call to OpenAsync and never returns

mcr.microsoft.com/dotnet/core/runtime-deps:3.0-buster-slim
the sql client hangs in a call to OpenAsync and never returns

mcr.microsoft.com/dotnet/core/runtime-deps:3.0-bionic
everything fine!

like @iinuwa posted before you can see in wireshark that in the bad case the server closes the socket connection after the TDS7 prelogin message, which contains the TLS "Client Hello"

I decoded the TLS payload of the good and bad prelogin message and there you can see that in the "bad" case the supported versions are TLS 1.3 and 1.2.
In the "good" case versions 1.0, 1.1, 1.2 and 1.3 are supported.

In Debian 10 TLS support of 1.0 and 1.1 was deactivated, so the behaviour of the used baseimages is ok. The working bionic image with Ubuntu 18.04 still supports these TLS versions.

I think that the sql server closes the connection cause it only supports TLS 1.1 and the request of the client signals that this version is not supported by the client.

So - i guess there are two problems now:
Seems that our sql server needs an update to support at least TLS 1.2 - have to contact our IT with this.

The hang in the OpenAsync of the SQL-Client seems like ab bug to me, cause the server closes the socket connection - expecting an exception in this case!

Hope this helps a little bit

This is the correct response, for the problem that we find with openasync using NetCore 3.1 (the last version until now - January 2020) we just need to pull using docker in this way FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-bionic

@cheenamalhotra

SQL Server 2016 still supports older TLS versions and it is possible TLS 1.2 is disabled.

You can confirm which TLS version is in use by capturing network packets when you create connection with Wireshark and looking into pre-login handshake packets. You will find "ClientHello" and "ServerHello" packets that also contain information about TLS version in use.

e.g.

@cheenamalhotra
This is my ClientHello when connecting from Linux image:

It seems like it's using TLS1.0?, but I do have
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2
in the openssl.conf file.

And when I connecting from Windows image, the ClientHello is TLS1.2:

Any idea?
Thanks a lot!

Looks like you have to add SQL Server certificate to trusted in order to utilize TLS1.2. Copy root certificate to /usr/local/share/ca-certificates in container and run update-ca-certificates

What's the certificate and where I can find it?
Thanks.

Same problem here after migration from 2.2 to 3.1 LTS
Database : SQL Server 2016 on Windows Server (TLS 1.2 enabled)
Asp.net core app (2.2) on debian : works.
Same Asp.net core app (3.1 LTS) on debian : doesn't work.
Tried both System.Data.SqlClient and Microsoft.Data.SqlClient.

Worked using the docker command :
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf

Please try to change Connection Timeout from 0 to 30, If you have given Connection Timeout = 0

Thanks

I have this problem.

Im using Azure Web App Container with Linux. trying connect to a SQL Azure Database.

The error is intermittent, but very frequent

2020-04-11T19:37:40.418839620Z Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) 2020-04-11T19:37:40.419144120Z ---> System.IO.IOException: The write operation failed, see inner exception. 2020-04-11T19:37:40.419164520Z ---> System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Connection reset by peer.) 2020-04-11T19:37:40.419611020Z ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. 2020-04-11T19:37:40.419622520Z ---> System.Net.Sockets.SocketException (104): Connection reset by peer 2020-04-11T19:37:40.419634920Z at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size) 2020-04-11T19:37:40.419639720Z --- End of inner exception stack trace --- 2020-04-11T19:37:40.420576920Z at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size) 2020-04-11T19:37:40.420588220Z at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async) 2020-04-11T19:37:40.421101420Z --- End of inner exception stack trace --- 2020-04-11T19:37:40.421112620Z at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) 2020-04-11T19:37:40.421125920Z at System.Threading.Tasks.Task.Wait() 2020-04-11T19:37:40.422007020Z at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count) 2020-04-11T19:37:40.422020520Z at System.Net.Security.SslStream.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer)
2020-04-11T19:37:40.422683520Z at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer) 2020-04-11T19:37:40.422695320Z --- End of inner exception stack trace --- 2020-04-11T19:37:40.422709020Z at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory1 buffer)
2020-04-11T19:37:40.423597421Z at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2020-04-11T19:37:40.423609421Z at Microsoft.Data.SqlClient.SNI.SNIPacket.WriteToStream(Stream stream)
2020-04-11T19:37:40.423622021Z at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
2020-04-11T19:37:40.424319421Z at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
2020-04-11T19:37:40.424331221Z at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
2020-04-11T19:37:40.424841321Z at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
2020-04-11T19:37:40.425237321Z at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
2020-04-11T19:37:40.425578921Z at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) 2020-04-11T19:37:40.425899721Z at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
2020-04-11T19:37:40.426233821Z at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) 2020-04-11T19:37:40.426245321Z at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
2020-04-11T19:37:40.426756421Z at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry) 2020-04-11T19:37:40.426780421Z at Microsoft.Data.SqlClient.SqlConnection.Open() 2020-04-11T19:37:40.426785721Z at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected) 2020-04-11T19:37:40.427572121Z at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected) 2020-04-11T19:37:40.427584421Z at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject) 2020-04-11T19:37:40.428292521Z at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.InitializeReader(DbContext _, Boolean result)
2020-04-11T19:37:40.428726021Z at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded)
2020-04-11T19:37:40.428749821Z at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.MoveNext() 2020-04-11T19:37:40.428755521Z ClientConnectionId:2389060f-e541-494a-a770-bc9501877192

Hi everyone, since #577 fixes the hang issue and will be released with Microsoft.Data.SqlClient v2.0.0, we will close the issue. This fix will also be backported to System.Data.SqlClient soon.

The recommended solution for anyone facing "End of Stream reached" exception in future is to verify target SQL Server supports TLS 1.2+ and server certificates are encrypted with SHA256+.

There are workarounds to switch back to lower TLS version if needed, as discussed above, but starting next release (v2.0), applications will also receive a warning as implemented in #591 if a lower insecure TLS version was negotiated with server, since these versions are not recommended for client applications. It includes raising warning for TLS v1.0 and TLS 1.1 protocols.

Good to know this is being fixed. I can confirm it's still an issue in aspnet:3.1.5-Buster and it's working fine in aspnet:3.1-Bionic. Thanks!

@BackTrak

I can confirm it's still an issue in aspnet:3.1.5-Buster

@BackTrak Have you tried with Microsoft.Data.SqlClient v2.0.0?

Navigation Menu

Repository navigation

Issue with SqlConnection.Open on netcore3 linux images #201

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with SqlConnection.Open on netcore3 linux images #201

Description

Activity

adrian-lopez-softtek commented on Sep 16, 2019

adrian-lopez-softtek commented on Sep 16, 2019

vik-singh commented on Sep 17, 2019

adrian-lopez-softtek commented on Sep 18, 2019

iinuwa commented on Sep 25, 2019

aidanjryan commented on Oct 3, 2019

cheenamalhotra commented on Oct 16, 2019

aidanjryan commented on Oct 17, 2019

clane2812 commented on Oct 30, 2019

gfzhang8 commented on Nov 26, 2019

aidanjryan commented on Nov 26, 2019

gfzhang8 commented on Nov 26, 2019

gfzhang8 commented on Nov 26, 2019

14 remaining items

cheenamalhotra commented on Dec 17, 2019

adrian-lopez-softtek commented on Dec 17, 2019

cheenamalhotra commented on Dec 17, 2019

IharYakimush commented on Dec 17, 2019

rvignesh89 commented on Jan 4, 2020

jepsilon commented on Jan 11, 2020

gfzhang8 commented on Jan 23, 2020

gfzhang8 commented on Jan 23, 2020

jaimemorais commented on Jan 23, 2020

nayanishdamania commented on Mar 5, 2020

penihel commented on Apr 12, 2020

cheenamalhotra commented on Jun 10, 2020

BackTrak commented on Jun 30, 2020

cheenamalhotra commented on Jun 30, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions