Repository navigation
Regression 2.1.13: AuthenticationException in Centos 7 using WCF #31110
Description
Describe the bug
I use the WCF client from .NET Core to access Windows WCF services from Linux.
Everything works correctly using .NET Core 2.1.3
Upgrading to 2.1.13 results in the exception below when calling the WCF client.
I also see the same problem with .NET Core 2.2 latest or 3.0 latest.
Expected behavior
I use the WCF client from .NET Core to access Windows WCF services from Linux.
Everything works correctly using .NET Core 2.1.3
I expected this to continue working in newer versions of .NET Core.
Additional context
I use kinit before making this call to ensure that I'm authenticated correctly.
Am I missing some dependencies, or could this be something else?
Linux Version
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
.NET Core Version
dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.802
Commit: 177d0b2525
Runtime Environment:
OS Name: centos
OS Version: 7
OS Platform: Linux
RID: centos.7-x64
Base Path: /usr/share/dotnet/sdk/2.1.802/
Host (useful for support):
Version: 2.1.13
Commit: 1a165a1588
.NET Core SDKs installed:
2.1.802 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.13 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.13 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.13 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Full Stack trace
System.ComponentModel.Win32Exception: GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database).
The above exception was the direct cause of the following exception:
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
at System.Net.Security.NegoState.StartSendAuthResetSignal(LazyAsyncResult lazyResult, Byte[] message, Exception exception)
at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.NegotiateStream.BeginAuthenticateAsClient(NetworkCredential credential, ChannelBinding binding, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.NegotiateStream.BeginAuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel, AsyncCallback asyncCallback, Object asyncState)
at System.Net.Security.NegotiateStream.<>c__DisplayClass27_0.<AuthenticateAsClientAsync>b__0(AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
at System.Net.Security.NegotiateStream.AuthenticateAsClientAsync(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgradeAsync(Stream stream, OutWrapper`1 remoteSecurity)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
System.ServiceModel.Security.SecurityNegotiationException: Authentication failed, see inner exception.
at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgradeAsync(Stream stream, OutWrapper`1 remoteSecurity)
at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgradeAsync(Stream stream)
at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgradeAsync(StreamUpgradeInitiator upgradeInitiator, OutWrapper`1 connectionWrapper, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreambleAsync(IConnection connection, ArraySegment`1 preamble, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnectionAsync(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpenAsync(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.OnOpenAsyncInternal(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.System.ServiceModel.IAsyncCommunicationObject.OpenAsync(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.OpenAsyncInternal(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.Runtime.TaskHelpers.CallActionAsync[TArg](Action`1 action, TArg argument)
at System.ServiceModel.Channels.CommunicationObject.OpenOtherAsync(ICommunicationObject other, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpenAsync(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.OnOpenAsyncInternal(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.System.ServiceModel.IAsyncCommunicationObject.OpenAsync(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.OpenAsyncInternal(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(MethodCall methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(MethodInfo targetMethod, Object[] args)
--- End of stack trace from previous location where exception was thrown ---
at System.Reflection.DispatchProxyGenerator.Invoke(Object[] args)
at generatedProxy_1.GetById(Int32 )
at MyApp.Client.Connections.BaseConnection`1.ExecuteOnSingleUseChannel[T](Func`2 activity)
Activity
Cronan commentedon Oct 8, 2019
Crossposted from here
mconnew commentedon Oct 8, 2019
Moved to corefx repo as this isn't a bug/change introduced in WCF code. @davidsh, are you aware of any changes in NegotiateStream between 2.1.3 and 2.1.13 which would have caused this?
@Cronan, if this works with 2.1.3 on the same host, then that should rule out any problems with kinit. I also wouldn't expect an error saying it couldn't find the server in the database if there was a problem with kinit as that implies it's communicating with the Kerberos server successfully.
Cronan commentedon Oct 8, 2019
mconnew commentedon Oct 8, 2019
@Cronan, it might be useful to find the exact version which caused the break. I wasn't going to suggest that as it can be a lot of work and it's possible that the appropriate devs might know which changes may have broken this, but if you have it working in docker, it might not be a lot of work. Especially if you do a binary search through the versions.
Something quicker that would likely be useful is to capture a Kerberos trace for when it's working and when it's not working. You can do this by setting the environment variable KRB5_TRACE before you run your code to capture Kerberos tracing. You can do one of the following:
Output log to stdout
Output log to file
davidsh commentedon Oct 8, 2019
Thanks for bringing this issue to my attention. Yes, there were changes between 2.1.3 and 2.1.13. We ported some fixes from .NET Core 3.0 into the latest servicing releases for 2.1 and 2.2. These were high priority Linux Kerberos related fixes.
See: dotnet/corefx#40109
I'll need to dig into your particular scenario to see why it appears to have been regressed by these changes.
davidsh commentedon Oct 8, 2019
Can you please provide a repro for this? Or describe in great detail the scenario, networking environment and especially how Kerberos is being used. What OS is the server? What OS is the client?
For the Linux client, please include the krb5.conf file on the machine and a KRB5_TRACE.LOG as described above. But, please don't post any confidential information to this GitHub issue. If necessary, you can mask off any information in the traces that is confidential.
Cronan commentedon Oct 8, 2019
mconnew commentedon Oct 8, 2019
@Cronan, I wouldn't try changing your Kerberos stack at this point, it will just add noise. Things were working with .NET Core 2.1.3 so unless we find a reason to believe there's an issue with the version of Kerberos you are running with, just leave everything which isn't .NET Core in the same state as when it's working with .NET Core 2.1.3.
mconnew commentedon Oct 8, 2019
@davidsh, He's provided details of the client in the initial bug description. He also explained the server is a Windows WCF service. Does the specific Windows version matter?
@Cronan, for your WCF client, are you explicitly specifying a service identity? You would have a reference to the one of the classes SpnEndpointIdentity, UpnEndpointIdentity or DnsEndpointIdentity. If you don't have a reference to any of these in your code, we default to using a targetName of host/hostname where hostname is extracted from the uri of the service you are connecting to. So depending on your the address you provided to WCF it might not be a FQDN.
davidsh commentedon Oct 8, 2019
No. But we need a repro. At the very least, snippets of WCF code for the server and client and how the service is configured.
Also, can you please get KRB5_TRACES for both the working scenario (2.1.3) and broken scenario (2.1.13)?
As a workaround can you try setting this environment variable on the Linux client? It will enable the 'legacy' HTTP stack (uses libcurl on Linux). I'm curious if it will help with this problem at least as a workaround for now:
export DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0mconnew commentedon Oct 8, 2019
@davidsh, this is not using HTTP, this is NegotiateStream.
davidsh commentedon Oct 8, 2019
Oh. Good point. Yes, setting that environment variable won't change anything.
Cronan commentedon Oct 9, 2019
Sorry, I actually meant Centos rpms, I was typing that on a phone ...
29 remaining items
karelz commentedon Jan 22, 2020
@Cronan it was part of our triage, so I assume @davidsh didn't feel like he has all the info to make it actionable yet.
There is lots of back and forth on this issue. What is IMO needed is clear repro environment setup that will allow us to reproduce it locally and dig deeper. Do you think you can help us identify the key environment setup knobs to reproduce it?
davidsh commentedon Feb 18, 2020
Actually, I do understand the root cause here. This issue is caused by a regression in behavior due to a PR fixing related Linux Kerberos issues as described above. See: #31110 (comment)
I know how to reproduce this issue.
The fix for this issue is complex because it will require some changes to the PAL layer to fix both this issue and preserve the other fixes previously made. I have been thinking about the proper fix. We can also test the fix in the 'runtime-libraries enterprise-linux' DevOps pipline that I previously created.
karelz commentedon Jun 18, 2020
@davidsh do you think we can address this realistically in 5.0?
davidsh commentedon Jun 19, 2020
It's a non-trivial fix. The fix has to be done in the Linux PAL layer. It has to be able to try two different forms of Kerberos principal names. I.e. try one form and if it doesn't match to an SPN in Kerberos, then try the alternate format.
Based on current priorities and staffing I don't see this as achievable for 5.0.
Cronan commentedon Jun 19, 2020
That’s disappointing but understandable. If someone can point me to the relevant place in the code to start looking I’m prepared to try to take this on.
rzikm commentedon May 28, 2025
Since CentOS 7 is past EoL, and there have been multiple changes in the affected area. Since there is no repro attached I am going to close this issue.
If the issue is still reproducible on newer .NET versions, feel free to open another issue with attached repro steps.