MSTest 2.2.4 does not work with [DataTestMethod] [DynamicData] and a complex object

Description

Everything was working fine in 2.2.3. Then we updated to 2.2.4 and it does not work anymore.
So we have some classes with [DataTestMethod] and some dynamic data that we feed into it.
The tests were green in 2.2.3 and upgrading to 2.2.4 break them with this error:

C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug>dotnet test MsTest224DataTestBug.csproj
 Determining projects to restore...
 Restored C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\MsTest224DataTestBug.csproj (in 529 ms).
 MsTest224DataTestBug -> C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll
Test run for C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
An exception occurred while invoking executor 'executor://mstestadapter/v2': Type 'MsTest224DataTestBug.MyStringContainer' with data contract name 'MyStringContainer:http://schemas.datacontract.org/2004/07/MsTest224DataTestBug' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
Stack trace:
  at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
  at System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObject(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.XmlObjectSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(XmlDictionaryWriter writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(Stream stream, Object graph)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.DataSerializationHelper.Serialize(Object[] data)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ToTestCase()
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.SendTestCases(String source, IEnumerable`1 testElements, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext, IMessageLogger logger)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(String source, IMessageLogger logger, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)


Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Steps to reproduce

Here is a full sample reproducing the bug:
https://github.com/TheFireCookie/MsTest224DataTestBug
In this repo, there is a folder MsTest223DataTestBug in which the command 'dotnet test' works correctly, the other folder is MsTest224DataTestBug and in this one, it's exactly the same code except I've upgraded MSTest to 2.2.4 and there a 'dotnet test' fails with the error written earlier.

Expected behavior

No regression and we can still use [DynamicData] with complex objects

Actual behavior

We cannot use anymore [DynamicData] with complex objects, only simple types like string, int etc

Environment

Tested and reproduced on Windows 10 and CentOS 7.9.
It's running on .NET 5.0.

AB#1334620

Thank you for the report. This bug is caused because of a change in how we handle attributes derived from ITestDataSource.

I am working on a fix, and will release a preview Monday for this bug.

It would be nice if regression tests for these cases were added in test/UnitTests

sk6868

sk6868 we actually have tests, but apparently they are not complex enough.

Medeni Baykal (@Haplois)

Medeni Baykal (@Haplois) yes I meant adding complex type (currently there only exists DynamicData tests using string and int types) use case tests to the existing regression tests :)

Matthias Mertens (@TheFireCookie)

This is fixed, and a preview is available in our daily feed. If you are not sure about how to connect to our daily feed please check out this page.

Fixed packages:

Could you please test it and verify this is fixed the issue?

/cc: Matthias Mertens (@TheFireCookie), sk6868, DOliana, Dave Smits (@davesmits), Rik Crompton (@rikrak), Andrew (@andessi), Majid Iqbal (@Akinzekeel), chriswelles

Medeni Baykal (@Haplois)

Medeni Baykal (@Haplois) thanks for the quick update.

I just tested the preview and re-ran all of my tests. The ones that were bugged out before (using types such as double and char) are working now. However a new bug appears to have popped up.

The following kind of tests are failing for me now (they worked previously in 2.2.3, not sure about 2.2.4):

[DataTestMethod]
[DataRow("john.doe@example.com", "abc123", null)]
[DataRow("john.doe@example.com", "abc123", "/unit/test")]
public async Task Can_SignIn(string Email, string Password, string ReturnUrl) { ... }

This is the error reported in the test explorer for these tests:

  Message: 
    Value cannot be null. (Parameter 'typeName')

  Stack trace: 
    RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
    Type.GetType(String typeName)

Update: there is now also an issue with byte-based enums (int-based enums work)

Message: 
    Test method My_Test_Method threw exception: 
    System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'MyByteBasedEnum'.

  Stack trace: 
    RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
    RuntimeType.CheckValue(Object value, Binder binder, CultureInfo culture, BindingFlags invokeAttr)
    MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
    RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    MethodBase.Invoke(Object obj, Object[] parameters)
    ThreadOperations.ExecuteWithAbortSafety(Action action)

Matthias Mertens (@TheFireCookie)

This is hopefully fixed, and a preview is available in our NuGet feed.

Fixed packages:

Could you please test it and verify if this fixed the issue?

/cc: Matthias Mertens (@TheFireCookie), sk6868, DOliana, Dave Smits (@davesmits), Rik Crompton (@rikrak), Andrew (@andessi), Majid Iqbal (@Akinzekeel), chriswelles, Howard Richards (@conficient), xantiva

Majid Iqbal (@Akinzekeel)

So my test with the complex objects are working now.

I've also tested the method of Majid Iqbal (@Akinzekeel) and it seems to work fine too.

When can we expect a real release of MSTest with the fixed issues?
Have a nice day!

Création d'un projet de reproduction du problème (microsoft/testfx#844)

Hi,

I try the v2.2.5-preview-20210604-01, but my problem persists.

I create a sample here : https://github.com/ybru-isiware/MsTest224DataTestBug.

This sample work well in 2.2.3.

Sample content

In the sample, i create a User class and a UserService class.
The UserService have one methode named ParseUserDatas who take a string for extract user information.

In my test project, i create a DataTestMethod and use DynamicDataAttribute.

The test method take two parameters :

[DataTestMethod()]
[DynamicData(nameof(GetParseUserDatas), DynamicDataSourceType.Method)]
public void ParseUserDatasTest(string userDatas, User expectedUser)
{
   // ...
}

The datas look like that :

public static IEnumerable<object[]> GetParseUserDatas()
{
    yield return new object[] {
        "John;Doe",
        new User()
        {
            FirstName = "John",
            LastName = "Doe"
        }
    };
}

Error :

 ParseUserDatasTest (John;Doe,MsTest224DataTestBugLib.User)
   Source: UserServiceTests.cs line 17
   Duration: < 1 ms

  Message: 
    Element ':root' contains data of the 'http://schemas.datacontract.org/2004/07/MsTest224DataTestBugLib:User' data contract. The deserializer has no knowledge of any type that maps to this contract. Add the type corresponding to 'User' to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding it to the list of known types passed to DataContractSerializer.
  Stack Trace: 
    XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
    XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
    DataContractJsonSerializerImpl.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
    XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
    XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
    DataContractJsonSerializerImpl.ReadObject(XmlDictionaryReader reader)
    DataContractJsonSerializerImpl.ReadObject(Stream stream)

Have a nice day !

v2.2.5-preview-20210604-01 fixed almost all cases for me, but byte-enums are still failing.

Test method signature:

[DataTestMethod]
[DoNotParallelize]
[DataRow(ShipmentThresholdPolicy.AdvancePaymentRequired, false, false)]
[DataRow(ShipmentThresholdPolicy.PackBeforePayment, true, false)]
[DataRow(ShipmentThresholdPolicy.PackAndShipBeforePayment, true, true)]
public async Task Publish_Sets_PackBefore_Properties(ShipmentThresholdPolicy policy, bool canPackBeforePayment, bool canShipBeforePayment) { ... }

Enum definition:

public enum ShipmentThresholdPolicy : byte
{
    AdvancePaymentRequired,
    PackBeforePayment,
    PackAndShipBeforePayment
}

Error:
System.ArgumentException: Object of type 'System.Int32' cannot be converted to type '(redactedNamespace).ShipmentThresholdPolicy'.

When testing update to 2.2.4 I ran into an issue with string[] and int[] params in DataRow attributes causing test-discovery exception. Developer Community ticket someone posted here:
https://developercommunity.visualstudio.com/t/cant-set-arrayofstring-as-the-datarow-in-mstest/1435155

Also I had an issue with floating point value params (1.5, 1.5f, 1.5d) being casted to decimal value and causing test failures.

I've run our tests against v2.2.5-preview-20210604-01 posted above - both of the issues I ran into in 2.2.4 seem to be resolved.

Majid Iqbal (@Akinzekeel)

Majid Iqbal (@Akinzekeel)
v2.2.5-preview-20210604-01 fixed almost all cases for me, but byte-enums are still failing.

I tried to reproduce this, but my tests are passing. Could you clean your solution, make sure you're using the latest preview and try again?

ybru-isiware

Thank you ybru-isiware,

I was able to repro the issue. Working on it.

Medeni Baykal (@Haplois)

Medeni Baykal (@Haplois) I found out that the test works when I use the command line (dotnet test), however it keeps failing in Visual Studio no matter what. I've tried cleaning & rebuilding and also deleted the .vs folder, restarted Visual Studio, cleared the test results cache but the test still fails.

I'm using these versions for the test project:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.5-preview-20210604-02" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.5-preview-20210604-02" />

Do you have any idea what could be wrong?

Majid Iqbal (@Akinzekeel)

Majid Iqbal (@Akinzekeel), could you try cleaning and rebuilding the solution? You also might need to restart VS.

Fixed a bug in `ITestDataSource` data deserialization (#864)

Majid Iqbal (@Akinzekeel)

With #864, last failing case is fixed too. We also added some tests. Majid Iqbal (@Akinzekeel), can you verify?

Fixed packages:

Majid Iqbal (@Akinzekeel)

With #864, last failing case is fixed too. We also added some tests. Majid Iqbal (@Akinzekeel), can you verify?

Fixed packages:

MSTest.TestAdapter v2.2.5-preview-20210605-01

MSTest.TestFramework v2.2.5-preview-20210605-01

Medeni Baykal (@Haplois) I can confirm that this version fixed all of my problems. Thank you!

Majid Iqbal (@Akinzekeel)

Thank you for confirming Majid Iqbal (@Akinzekeel), closing the issue.

Medeni Baykal (@Haplois)

Medeni Baykal (@Haplois)

With #864, last failing case is fixed too. We also added some tests.

Is there a timeline on when 2.2.5 is planned to be released?

Squashed commit of the following:

Navigation Menu

Repository navigation

MSTest 2.2.4 does not work with [DataTestMethod] [DynamicData] and a complex object #844

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Sample content

Error :

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MSTest 2.2.4 does not work with [DataTestMethod] [DynamicData] and a complex object #844

Description

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Activity

Haplois commented on May 27, 2021

sk6868 commented on May 27, 2021

Haplois commented on May 27, 2021

sk6868 commented on May 27, 2021

Haplois commented on May 31, 2021

Akinzekeel commented on May 31, 2021

Haplois commented on Jun 4, 2021

TheFireCookie commented on Jun 4, 2021

ybru-isiware commented on Jun 4, 2021

Sample content

Error :

Akinzekeel commented on Jun 4, 2021

PRRB commented on Jun 4, 2021

Haplois commented on Jun 4, 2021

Haplois commented on Jun 4, 2021

Akinzekeel commented on Jun 4, 2021

Haplois commented on Jun 4, 2021

Haplois commented on Jun 5, 2021

Akinzekeel commented on Jun 5, 2021

Haplois commented on Jun 6, 2021

japj commented on Jun 16, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions