Repository navigation
In-assembly parallel execution #142
Description
Activity
Here is the RFC for this. Feedback is welcome.
Reacted by György PetőShould parallel test execution not (at least optionally) be possible for non parallel-ready code? A container with 1000 tests that can't run in parallel can be divided in chunks and run sequentially in N appdomains/processes.
We have been using https://github.com/sscobici/ParallelTestRunner for this for a while now. Can split on Class level, Method level and group tests via annotations. Would be very nice to have this build in, so there is no longer a need to run an executable to split the tests, run them and then summarize the testresults.
Reacted by codito and Roland KieslingerI'm all for letting tests run in parallel, especially since it doesn't seem to work right now in MSTestV2, at least in Visual Studio 2015. The RFC linked mentions VSTest is capable of allowing all test providers to do test parallelization by spawning multiple processes.
I know this feature is for in assembly parallelization, but for my sake, can someone confirm to me that regular old VSTest multiprocess parallelization isn't meant to work right now in MSTestV2? I would appreciate the clarity. Thank you!
AdamDotNet : The container level parallelization for vstest works irrespective of the Framework(mstestv2/mstest/xunit/nunit). If your solution has multiple test projects, it should spawn a host process for each test container and run them in parallel when enabled just like this blog details. This only works if there is more than one test container. You aren't seeing that happen in VS 2015?
Abhitej John (@AbhitejJohn) Thanks for the information. I never realized that multiprocess parallelism was dependent on having multiple projects. I just observed it working in a sample project with two test projects.
Back to this feature of in assembly parallelism, if I'm reading the RFC right, you can mark an assembly with how you want it to handle parallelism in that assembly. I'm imagining in my solution, it would be generally acceptable to mark the assembly as parallel by class by default, but it would be nice if certain classes could run faster by running the test methods in a particular class in parallel.
So, will it be possible to default an assembly to class level parallelization, but you can override that behavior by placing attributes on a specific class to enable method parallelization for that test class? Will this work with only one test project?
To illustrate, I have one test project with the desire for the following behavior:
- Classes that contain tests that are totally in memory using Moq: run as class parallelization because each test takes only a handful of milliseconds to run.
- Classes that contain a number of tests that integrate with other pieces: run as method parallelization because each test takes about 5 seconds to run.
AdamDotNet : That's a great point. This is essentially a mix and match of multiple modes in the same assembly. Our thought process was that integration/End-to-End tests would actually be split into a different assembly from unit tests, whose test mode can then be configured accordingly.
That said, for the scenario described above, as long as the fast running tests within a test class do not interfere with each other, the mode for the assembly can be made method level as a whole. That ensures that all methods irrespective of what class they are under are run in parallel.
However this does look like a valid ask and could surface more when we add in custom modes. The only choice that the RFC provides today is to opt-out of the default mode for an assembly. We hope that these scenarios however are relatively fewer in number and come under a more advanced set of capabilities that we can add in the future. Do let us know otherwise.
Reacted by AdamDotNetThis is something that is incredibly frustrating to my team. This seems like a regression in behavior from the parallelization control we got from the testsettings file. In fact, my team is still currently are using a testsettings file to get that level of parallelization of our tests because one of our test categories takes around 3 hours to complete without it, and only 20 minutes to complete with it (with a setting of parallelTestCount="18").
Unfortunately we're banging our heads against this problem right now because without using a runsettings file we can't easily use test parameters, and using a runsettings file we can't get parallel execution.
The sooner we can get support for this in MSTest while using a runsettings file the better.
Thanks for letting us know JoshCollinsMSFT . We are in the phase of collecting feedback on the RFC that Pratap posted above. Do let us know if that meets your requirements. We plan to start off on this in the weeks to come.
Abhitej John (@AbhitejJohn) How do we provide feedback to the RFC?
JoshCollinsMSFT : you can do so here :)
53 remaining items
Hi, I got some feedback.
I want to be able to mark some tests as dependent on each other. And what that means is that the dependent tests will not run in parallel, but they can run in parallel with the rest of the tests.
A better explanation, given the next tests:
- TestA (Depend on testC)
- TestB
- Test C
- Test ...
- Test Z
I want all the tests to run in parallel, but if testC is running then testA will not run until testC is complete.
Does it makes sense?
Why your tests depends on each other? Make them independent and isolated. If needed, call dependencies explicitly (make them private methods, not public test methods).
Reacted by Adrian GodongAlexander Batishchev (@abatishchev) They are dependent because they both are UI tests that send an email to a specific test account, and I need to verify each email, but if they run at the same time, and 2 emails are sent, it is really messy to identify which email belong to each test. So I want to run one first, then the other one.
The example above is just an example, they are more complicated scenarios of tests using the same test data and I think someone else might face a related issue.
Ok, then see the discussion/workaround above: how to decorate tests the way that some run in parallel, some not. P.S. I'm just a user with very similar requirements/issues, not a dev on the MsTest team.
Alexander Batishchev (@abatishchev) The workaround works fine, but all the tests decorated to not run in parallel will be executed one at the time and that add extra time to the execution. I want them to be in parallel, just some tests to run after others.
You can have different categories, like Parallel 1,2,3, and run a set of tests in parallel, then another, then another, then everything else. Again yet another workaround but it works for us.
I have recently moved our framework from mstest to mstestv2.0 using dotnetcore and observing few issues when running tests in parallel. If i execute single test i am able to connect to DB for my test data and get the data but when i run two tests in parallel i am not getting any data. Could anyone let me know what could be the issue. I am using sqldataadapter. I have been using datarow in mstest but mstest2.0 doesnot have any data row. Could this be an issue or is this related to any Sql implementation?
Most likely it's your code's issues. We're making numerous async HTTP calls from tests and all works just fine.
Post a question on Stack Overflow and provide a link here.Thanks for the update. I was able to fix it. I am new to .netcore and when i run the tessts in parallel using .netcore it is not taking two tests Any idea how we can identify the number of test cases i trigerred as i have some predefined things which we needs to run in initialize method. I have ran two test cases but it is only taking one tests in to initialize method
From the RFC it isn't clear whether a single test method with multiple DataRow attributes will be parallelized. My own testing right now seems to indicate that they are not parallelized.
- What are the considerations for allowing DataRow-annotated test methods to be parallelized?
- Do the semantics of the data source concept in MSTest not work well with parallelization?
Being able to parallelize over DataRow attributes would help me a great deal with making some of my integration tests more convenient to run.
I realize I can refactor so that the test method itself uses Parallel.ForEach (or similar) to explicitly parallelize over the test cases I have, but then I'd lose the separate-outcome-per-case display in Test Explorer. (I'd certainly consider doing this anyway if my integration test suite grows much bigger.)
I have clarified this in the RFC - indeed, parallelizing over DataRow attributes is not supported.
Thank you for bringing it up.Reacted by William RummlerThis has been shipping since v1.1.3.0 Beta, and went RTM in v1.3.0.
Accordingly, closing this issue.Reacted by Chris HawsCooooool
I recommend moving to v1.3.1. It has the fix for AppDomain creation should honor runsettings.
Description
Enable tests runs to complete faster by allowing tests within a single assembly to execute in parallel.
Please see RFC here: 004-In-Assembly-Parallel-Execution