No unique GUIDs w/ [DataTestMethod]

[JDCain]

Description

Tests using [DataTestMethod] don't generate unique guids per test instance.

I noticed this as I was using [DataTestMethod] to run the same Selenium Test with different browsers. When running the dataCollector://microsoft/VideoRecorder/1.0 was recording every datatestmethod run within the same wmv file.

Steps to reproduce

Use any ITestLogger on the command line of vstest.console or a datacollector such as video datacollector://microsoft/VideoRecorder/1.0 in a runsettings file.

Using this repo:
https://github.com/JDCain/DynamicTestGuid
Command: vstest.console ClassLibrary1.dll /logger:ClassLibrary2 /settings:example.runsettings

Expected behavior

Starting test execution, please wait...
72b4a25d-970d-1351-338b-5203ac58b6ad
Passed TestMethod0
[Expected Unique GUID]
Passed Test_Add_DynamicData_Method
[Expected Unique GUID]
Passed Test_Add_DynamicData_Method (1,1,2)
[Expected Unique GUID]
Passed Test_Add_DynamicData_Method (12,30,42)
[Expected Unique GUID]
Passed Test_Add_DynamicData_Method (14,1,15)

Expected: TestResults folder to have Folder for each run of DynamicData_Method

Actual behavior

Starting test execution, please wait...
72b4a25d-970d-1351-338b-5203ac58b6ad
Passed TestMethod0
7c7826b6-350f-da93-57ca-15c3d29d9f35 [Duplicate GUID]
Passed Test_Add_DynamicData_Method
7c7826b6-350f-da93-57ca-15c3d29d9f35 [Duplicate GUID]
Passed Test_Add_DynamicData_Method (1,1,2)
7c7826b6-350f-da93-57ca-15c3d29d9f35 [Duplicate GUID]
Passed Test_Add_DynamicData_Method (12,30,42)
7c7826b6-350f-da93-57ca-15c3d29d9f35 [Duplicate GUID]
Passed Test_Add_DynamicData_Method (14,1,15)

TestResults folder has one results folder for DynamicData_Method

Environment

vstest.console.exe: Version 15.7.2
Windows 10: 10.0.16299 Build 16299
MSTest.TestFramework 1.3.2
MSTest.TestPlatform.ObjectModel 15.7.2

[jayaranigarg]

J.D. Cain (@JDCain) : Thank you for reaching out to us. The behavior you saw is expected.

In case of data-driven tests i.e. [DataTestMethod] in your case, each testmethod has a parent result and a bunch of child results depending on number of data rows. All of these results(parent as well as child) have same testcase id. This is because it is considered as one testmethod(which when ran produces multiple results) and hence one testcase id. It is by design.

In order to access unique GUID for each test result, consider using "ExecutionId" TestProperty which is unique for each test instance. You might want to replace this line of code with

            Console.WriteLine(e.Result.GetPropertyValue(TestProperty.Find("ExecutionId")));

Let us know if this works for you.

[JDCain]

Thank you for the information. My guess would be then that "datacollector://microsoft/VideoRecorder/1.0" is not using the ExecutionId to start and stop a recording since when it records it includes every run of a [DataTestMethod] in the same output video file. When it outputs the videos it puts them all in branching folders of guids which seem to match the test guids.

I am unclear on where Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder comes from. Is it part of this project in some way or is there source for it I can use to implement my own version of it to support multiple videos for each [DataTestMethod]?

[jayaranigarg]

J.D. Cain (@JDCain) :

Thank you for the information. My guess would be then that "datacollector://microsoft/VideoRecorder/1.0" is not using the ExecutionId to start and stop a recording since when it records it includes every run of a [DataTestMethod] in the same output video file. When it outputs the videos it puts them all in branching folders of guids which seem to match the test guids.

You are correct. Recording is started and stopped by testcase object in the adapter.

Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder comes from a different source(which is not open source as of now). However no changes are needed in VideoRecorder to get separate videos. Changes needs to be made in this repo itself.

Currently, one data-driven test is considered as one testcase object and therefore one testcase-start & testcase-end event is send at the beginning and finishing of a testcase. All datarows associated with that testcase executes during that timeframe generating just video.

We will have to break a data-driven test into multiple tests such that for each datarow a testcase-start and testcase-end event is send, so that each datarow has a separate video.

We will be happy to accept contribution for this. :)
Tagging pvlakshm (@pvlakshm)

[JDCain]

Thank for the information. To make sure I understand, this would mean basically taking some of the changes from RunTestMethod in #238 and and implementing it in ExecuteTestsWithTestRunner so that each data driven test is sent to UnitTestRunner.RunSingleTest?

If so would the implementation in RunTestMethod need to stay there as something else is dependent on it (#417)?

[jayaranigarg]

J.D. Cain (@JDCain) : You understand correctly. Doing the changes you mentioned, your problem should get fixed. But there are other things to be considered as well. Like, if you also want each of the datarow to display as a separate test in TestExplorer, we will have to make changes on Discovery flow as well.

This is an interesting problem. Let us know your thoughts on this. Accordingly we can publish an RFC and take it from there.

[JDCain]

Jayarani garg (@jayaranigarg) Personally I would prefer they stay merged in discovery. I only need the results video to be separate and I think having multiple in the explorer would just clutter it.

Is there a preference for how this change should be implemented. I am not sure of the style or what's
appropriate for the project. e.g. pass ITestExecutionRecorder down to the needed location, implement TypeCache at the higher level to get the separated tests, or meet in the middle by passing the ITestExecutionRecorder to UnitTestRunner and starting and stopping the recorder there while getting the sub tests?

[jayaranigarg]

J.D. Cain (@JDCain) : There will be a number of implications of this change. Will there be a performance hit after we make execution of each datarow as a separate test, how will tests appearance change in TestExplorer etx.

I would encourage you to think upon on this and come up with an RFC on which we can give our views and on which community can provide it's view.

[JDCain]

Jayarani garg (@jayaranigarg) Is the RFC something I submit via the docs repo on here or is it something I submit to you?
I was thinking that since this is a small use case I would not want to change the way it works currently as users already have expectations of how it functions. Perhaps making the treating of the DynamicData sub sets with their own start and stop optional setting the option via the Attribute.

[DynamicData(nameof(Data), DynamicDataSourceType.Property, separate: true )]
...
public DynamicDataAttribute(string dynamicDataSourceName, DynamicDataSourceType dynamicDataSourceType = DynamicDataSourceType.Property, bool separate = false )
{
...
}

[jayaranigarg]

J.D. Cain (@JDCain) : RFC is something you submit via testfx-docs here. We can discuss it further once your RFC is out.

[michalkovy]

I have one other use case for the same need.

The use case is behavior-driven development using TickSpec (https://github.com/fsprojects/TickSpec). TickSpec is a .NET framework for BDD tests that allows generating tests from feature files on runtime instead of compilation time. The benefit of runtime generation is that business people can write new tests just with feature file changes when they reuse existing steps.

When you use BDD then you have many feature files which each contain several scenarios. We are able to generate on runtime when using NUnit or xUnit tests that show in Test Explorer as a test per scenario. When we try the same with MSTest we only get one test in Test Explorer. It is unusable even with just 20 scenarios and you can have many more, thousands.

[Haplois]

This has been implemented in 2.2.5. Assuming all dynamic data can be serialized, and you're not using DataSourceAttribute all discovered tests gets a new id.