[🚀 Feature]: [dotnet] Revisit unit tests infrastructure

[nvborisenko]

Feature and motivation

Current situation is:

Usage example

Concerns:

• It could be one test project because Chrome/Edge/Firefox drivers live in one project supported by Selenium (like built-in classes).
• Default namespace in test project is OpenQA.Selenium when usually it should be OpenQA.Selenium.Tests. I don't understand why it is unusual.
• Avoid any static context (EnvironmentManager class) - it will allow to look at parallelization. My computer has 6/12 cores, I want to utilize all of them when running Selenium unit tests.
• Provide a possibility to override driver options, like required in [dotnet] [bidi] Support WebExtension module #15850

[RenderMichael]

I don't have much of the historical context surrounding the design decisions of the test projects. There are some clear reasons I see for a heavily configuration-based EnvironmentManager. For example, the EnvironmentManager.Instance property is accessible in the AssemblyFixture types.

At the same time, I see some room for optimization.

Notably, the above AssemblyFixture's one-time setup starts the web server and the remote server, but not the driver. The one-time teardown closes whatever driver may be opened, likely as a precaution. I fully believe we can move the "current driver" management to the test-side.

Rather than using appconfig.json (and all the Runfiles.Create() and runfiles.Rlocation ceremony surrounding that), we can give DriverTestFixture a method like so:

public abstract class DriverTestFixture
{
    public abstract IWebDriver CreateDriver();
}

I can experiment with some potential solutions.

[nvborisenko]

Historical decision is very clear: reusing browser across 100 tests is faster than initiating new clean browser per test. I would agree at that long time ago, as a quick approach to run tests. But now I strongly disagree. Reusing shared context breaks main principle of testing: isolation. Tests should not be dependent on context, on browser state in our case. (here I could put a link to ignored test which should not be ignored). This is why we currently have NeedsFreshDriverAttribute, meaning we have to keep in mind the state of browser, shared between tests. And it is bad in my opinion.

Tests isolation has more priority than tests execution duration. Let's say that current total duration is 20mins, introducing tests isolation will increase total duration to 40 mins. Not a problem at all, I don't see big diff between 40 mins and 20 mins. Insight: BiDi tests are executed in parallel fully isolated: 100 tests in 1.5 minutes on my machine.

So my recommendation is also very clear: be isolated and parallelizable. Good news it is achievable.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[nvborisenko]

Next big step is to move NUnit infrastructure into dedicated reusable project.

[nvborisenko]

NUnit infra is in dedicated project (reusable), test fixtures are parallelizable. I am confident now how it works - in IDE just run test - it runs fast.

@YevgeniyShunevych @RenderMichael any leftovers you wanted to bring before closing this chapter (issue)?

[YevgeniyShunevych]

I have nothing to mention.

[nvborisenko]

Sold. Thank you all for good collaboration. Next improvements will be tweak-tuning. Closing this chapter as completed.