Clean Enlistment Test Failures

After installing Windows 10 on a Parallels VM (on a MacPro), installing VS 2015, cloning, then running build, and build-tests from a VS 2015 Developer Command Prompt I consistently get the following errors:

Verified fixed by dotnet/corefx#24052

C:\git\corefx\Tools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Globalization.Tests' please check C:\git\corefx\bin/tests/System.Globalization.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details! [C:\git\corefx\src\System.Globalization\tests\System.Globalization.Tests.csproj]

Verified fixed by dotnet/corefx#24016

C:\git\corefx\Tools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Console.Tests' please check C:\git\corefx\bin/tests/System.Console.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details! [C:\git\corefx\src\System.Console\tests\System.Console.Tests.csproj]

Verified fixed by dotnet/corefx#24147

C:\git\corefx\Tools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Net.NameResolution.Functional.Tests' please check C:\git\corefx\bin/tests/System.Net.NameResolution.Functional.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details!
[C:\git\corefx\src\System.Net.NameResolution\tests\FunctionalTests\System.Net.NameResolution.Functional.Tests.csproj]

Verified fixed by dotnet/corefx#24009 and dotnet/corefx#24097

C:\git\corefx\Tools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Drawing.Common.Tests' please check C:\git\corefx\bin/tests/System.Drawing.Common.Tests/netcoreapp-Windows_NT-Debug-x64/testResults
.xml for details! [C:\git\corefx\src\System.Drawing.Common\tests\System.Drawing.Common.Tests.csproj]

The console log.

The console log should include the error messages from the tests that failed. Is these any more information higher up in the log?

@mellinoe

@mellinoe Thanks for taking a look.

I posted a gist of the log here.

Here are some excerpts:

System.Globalization.Tests.NumberFormatInfoNumberDecimalDigits.NumberDecimalDigits_Get(format: NumberFormatInfo { CurrencyDecimalDigits = 2, CurrencyDecimalSeparator = ".", CurrencyGroupSeparator = ",", CurrencyGroupSizes = [3], CurrencyNegativePattern = 1, ... }, expectedWindows: 2, expectedIcu: 3) [FAIL]
Assert.Equal() Failure
Expected: 2
Actual: 3

System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_NoneIPAddress_Fail(address: 255.255.255.255) [FAIL]
Assert.Throws() Failure
Expected: typeof(System.Net.Sockets.SocketException)
Actual: (No exception was thrown)

@tarekgh

There are a few spurious System.Drawing failures in there -- I will try to submit a PR that fixes them.

@tarekgh Could you look at the Globalization failure and see if we know about it?

@danmosemsft I think he is hitting the issue you described here: dotnet/corefx#21518. Did we file anything about that, or did it get fixed at some point?

Out of curiosity, what version of Windows are you running on?

@kingces95

@kingces95 what Windows version you are running on?

Win 10 Enterprise, fully patched. Running inside parallels with a mac host.

@kingces95

@kingces95 if you can give me the exact version? in the command window type "systeminfo" and then give me the line like the following:

OS Version:                10.0.15063 N/A Build 15063

I am just wondering why this is failing on your machine and not failing in ours?

@tarekgh

@tarekgh The console tests are failing on some of my machines too. I think it has something to do with high-dpi and/or having physical monitor attached. If somebody wants to stop by my office, I will be happy to provide repro (I do not think it repros over TS).

@mellinoe

@mellinoe I don't see that info (about the console failure) but if it is that, I believe it was a bug in RS2. Not sure about RS1. No, I did not complete the PR dotnet/corefx#21518 - It's just setting console title to a huge string, so IMO it's not worth bothering to fix, the right thing is to adorn this test so it doesn't run on affected Windows versions.

It may even have been particular builds of RS2 - I know it reproed consistently on my home laptop, but never at work. Again, that assumes it was failing for that reason and not some other random console problem.

@jkotas

@jkotas the globalization test failure has nothing to do with the machine configuration in general. do you see the globalization test failing on your machine?

@danmosemsft is right. Windows had a regression in RS2 which they fixed on RS3. The regression caused the framework to fail when setting the console title with long text. I agree it is not worth adding work around in the framework for this one as the problem occurred on RS2 only and happen when setting the Window title to a long string which is not a common scenario at all.

Globalization tests are fine for me. The tests I see failing intermittently or on some machines are System.DirectoryServices.*, System.Net.*, System.Drawing and System.Console.

@kingces95

@kingces95 one more ask. could you run "intl.cpl" and then click on "Additional Settings" button and tell me the value of "No. of digits after decimal"? I assume you are running with en-US locale. thanks for your help.

@jkotas

@jkotas if you have RS2 then "WindowAndCursorProps.Title_Set_Windows(lengthOfTitle: 1024) [FAIL]" expected to fail

@tarekgh

perhaps @tarekgh you could slap a check for Windows version on that one so it doesn't come up again? I should have done this myself.

@karelz

Yes, the same code, when run against .NET Framework should throw SocketException, but it does not on a windows vm on top of parallels on mac

Ok, so the variable here is that the "hardware" is not a real machine but rather the Parallels hypervisor architecture on a Mac.

cc: @karelz You'll need a Mac person to troubleshoot this since this is a Mac specific problem causing different behaviors to leak thru into the Windows container.

@kingces95

@kingces95 It'd be great if we could add a mac/parallels run to the CI.

Naively I would expect Windows under Parallels on Mac to behave in similar way as in any other virtualization technology (with Windows/Linux host). Ideally it should behave the same way as on physical HW (I assume that's how Windows VM on Windows/Linux host behaves).
I wonder why that is not the case here - maybe Parallels/driver bug?

Given that we do not have unlimited Mac HW, adding new CI legs to run Windows in Mac/Parallels is very costly for us. It also adds complexity to the infrastructure - and we have much higher priority gaps/bugs in infrastructure, so this would be very low pri from that perspective.
The impact on CoreFX contributors seems to be fairly contained ("just" 4 tests difference, brought down to 2 now) compared to other problems/differences in our test bed affecting contributors.

Overall, it feels like being reactive to discovered differences seems the right thing to do from cost/benefit point of view at this moment.

If there is a way to detect we are running under Mac/Parallels virtualization, we should be able to disable/modify the tests in such environment. Is it possible to detect?
Otherwise we could just consider this kind of error in particular tests as "by design".

Parallels is injecting extra DNS mappings. They maybe unusual, but I would not consider them to be a bug.

You can replicate the parallels behavior on regular Windows installation by going to C:\Windows\System32\drivers\etc\hosts and adding the following line there:

       255.255.255.255  whatever

Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]); will happily print 255.255.255.255 once you do that. So this looks like a test sensitive to machine environment that should better be OuterLoop or something.

Also, adding the 255.255.255.255 entry is recommended as workaround for all sorts of problems. For example, https://technet.microsoft.com/en-us/library/security/ms16-077.aspx suggests to add 255.255.255.255 wpad as a workaround. So this problem is not specific to Parallels. Any machine with these sort of workarounds applied will have the same problem.

Also, adding the 255.255.255.255 entry is recommended as workaround for all sorts of problems. For example, https://technet.microsoft.com/en-us/library/security/ms16-077.aspx suggests to add 255.255.255.255 wpad as a workaround. So this problem is not specific to Parallels. Any machine with these sort of workarounds applied will have the same problem.

Interesting. We'll need to study this test and replace "255.255.255.255" with something else that is appropriate for this particular test.

@jkotas

Thanks @jkotas! We should probably split off the Networking test discussion and harden the test more against not-so-common environment settings.

We should probably split off the Networking test discussion and harden the test more against not-so-common environment settings.

Based on this discussion, it sounds like these particular tests are invalid. The test was originally created because it was assumed that doing a DNS lookup against IPAddress.None (which is the broadcast address, 255.255.255.255) would always be invalid. Then it was tweaked to ignore the test on MacOS.

public static IEnumerable<object[]> GetNoneAddresses()
{
    yield return new object[] { IPAddress.None };
}

[PlatformSpecific(~TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Fail(IPAddress address)
{
    string addressString = address.ToString();

    await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(address));
    await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(addressString));
}

[PlatformSpecific(TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Success(IPAddress address)
{
    IPHostEntry result = await Dns.GetHostEntryAsync(address);
    Assert.NotNull(result);
    Assert.NotNull(result.AddressList);
    Assert.Equal(1, result.AddressList.Length);
    Assert.Equal(address, result.AddressList[0]);
}

But in reality, since this IP address could have a DNS mapping on any platform via hard-coding it in the hosts file, this test should be removed. I will submit a PR to take care of it.

Remove invalid NameResolution tests (#24147)

@mellinoe

Woot! Everything is fixed!

@mellinoe I'd close this issue out except I'm not sure which change fixed the drawing issue so I'm not sure if it's actually fixed or just a sporadic error and I happened to get lucky this time.

@kingces95

@kingces95 Glad to hear it! Drawing was fixed by dotnet/corefx#24009 and dotnet/corefx#24097. The problem in both is that the tests were making assumptions about what kind of printer you had installed, basically, and would fail if something was different than expected.

Sync (#5)

Navigation Menu

Repository navigation

Clean Enlistment Test Failures #23532

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Clean Enlistment Test Failures #23532

Description

Activity

mellinoe commented on Sep 13, 2017

kingces95 commented on Sep 13, 2017

mellinoe commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

kingces95 commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

jkotas commented on Sep 13, 2017

danmoseley commented on Sep 13, 2017

danmoseley commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

jkotas commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

tarekgh commented on Sep 13, 2017

danmoseley commented on Sep 13, 2017

23 remaining items

davidsh commented on Sep 16, 2017

karelz commented on Sep 16, 2017

jkotas commented on Sep 16, 2017

jkotas commented on Sep 16, 2017

davidsh commented on Sep 16, 2017

karelz commented on Sep 18, 2017

davidsh commented on Sep 18, 2017

kingces95 commented on Sep 19, 2017

mellinoe commented on Sep 19, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions