java.nio.file.FileSystemAlreadyExistsException during test discovery

[oscargolding]

When using JUnit 5 (and also 4) with version 1.8.2 for the launcher and 5.8.2 for jupiter I have been observing the presence of the following warning: WARNING: Error scanning files for URI jar: ... followed immediately by java.nio.file.FileSystemAlreadyExistsException emitted from the JUnit code jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:102) at org.junit.platform.commons.util.CloseablePath.createForJarFileSystem(CloseablePath.java:57).

I am making a program to run a suite of JUnit tests across multiple threads, and using the JUnit launcher to perform test execution. I suspect this issue is caused by the JUnit .exectue() method running test discovery each time I invoke the method, which causes a race condition on the java filesystem object that JUnit creates. For reference, I believe that this class is not thread safe https://github.com/junit-team/junit5/blob/main/junit-platform-commons/src/main/java/org/junit/platform/commons/util/CloseablePath.java. I expect JUnit to produce no warnings when I execute tests across threads.

Since JUnit no longer supports performing test discovery once for multiple executions (which is ideal for my use case), I am forced to perform repeated test discovery each time I invoke the JUnit launcher. Moreover, it seems that the decision to repeatedly do test discovery on each execution was made by JUnit recently #2379.

The environment I run my program in requires the tests to be in .jar files. When the tests are not in a .jar file the error coming from ZipFileSystemProvider does not happen.

Steps to reproduce

I created a simple program https://github.com/oscargolding/junit-load-test that shows the presence of the bug.

To reproduce the warning clone the project then run the following commands.

First build the project and get the resulting .jar file. Then place the file into a relevant directory (I picked /tmp/1/lib).

$ ./gradlew build
$ mkdir -p /tmp/1/lib
$ cp ./build/distributions/junit-load-tst-1.0-SNAPSHOT.jar /tmp/1/lib/

Then copy all the junit dependency .jar files (listed below) that are cached under the directory ~/.gradle/ into /tmp/1/lib/.

junit-jupiter-api-5.8.2.jar
junit-jupiter-engine-5.8.2.jar
junit-platform-commons-1.8.2.jar
junit-platform-engine-1.8.2.jar
junit-platform-launcher-1.8.2.jar
junit-platform-runner-1.8.2.jar
junit-platform-suite-api-1.8.2.jar
junit-platform-suite-commons-1.8.2.jar

Finally run the commands below to reproduce the warning.

$ cd /tmp/1
$ java -cp "./lib/*" "org.example.Main"

Context

• Used versions (Jupiter/Vintage/Platform): 5.8.2
• Build Tool/IDE: Gradle, Java 11

Deliverables

• ...

[sormuras]

An interesting use-case and thanks for raising this issue.

Seems like the code should first call FileSystem.getFileSystem(URI) in order to check whether a file system was already created for the passed URI. Not sure though, how the requested CloseablePath instance should be treated: ignore/discard it, register a duplicate, ... ?

[marcphilipp]

Since JUnit no longer supports performing test discovery once for multiple executions (which is ideal for my use case), I am forced to perform repeated test discovery each time I invoke the JUnit launcher.

Could you please elaborate? Are you running the same set of tests concurrently or repeatedly?

[oscargolding]

Thanks for the quick response. I am running the same set of tests concurrently (this is a hard requirement for my application). I believe if the tests are run in a single threaded environment this issue does not happen.

Initially I explored using a single TestPlan so that I could perform test discovery once and avoid the cost of having to do test discovery multiple times for the same test suite. What I found when I attempted to do this was an exception saying that a TestPlan can only be executed once https://github.com/junit-team/junit5/blob/main/junit-platform-launcher/src/main/java/org/junit/platform/launcher/core/InternalTestPlan.java#L43-L47 (I believe due to some state being shared between the TestPlan and engine).

I did try modifying the JUnit source code to first call Filesystem.getFileSystem(URI) when the filesystem already exists. There was then a subsequent race condition on the first thread to call .close() found here https://github.com/junit-team/junit5/blob/main/junit-platform-commons/src/main/java/org/junit/platform/commons/util/CloseablePath.java#L71-L74

[marcphilipp]

I am running the same set of tests concurrently (this is a hard requirement for my application).

What's the use case for that? Why would you run the same tests multiple times concurrently?

[oscargolding]

What's the use case for that?

I am trying to run JUnit tests at a high load, without having to go through the effort of having to rewrite the tests. I also want to use the JUnit listeners that are provided for each test execution to get summary data on the number of tests which are passing / failing / started.

Why would you run the same tests multiple times concurrently?

Similar to my response above, I need concurrency to generate a sufficient level of load to run these tests at. Some of the tests that my tool runs involve a network call. To generate a sufficient number of invocations per second multiple threads are necessary.

To help you understand my use case consider the tool JMeter https://jmeter.apache.org/ which is designed for measuring system performance. What is different about my use case is that I want to leverage the JUnit launcher to do test discovery (JMeter implements their own version of discovering tests).

Do you have any suggestions about how to reuse JUnit tests at a high load? For the most part JUnit works great for my use case, it is just this race condition during test discovery that is a problem. Thanks :)

[mpkorstanje]

@sormuras

Seems like the code should first call FileSystem.getFileSystem(URI) in order to check whether a file system was already created for the passed URI. Not sure though, how the requested CloseablePath instance should be treated: ignore/discard it, register a duplicate, ... ?

Neither would work. The life-span of different file systems usages may overlap like so:

A: |-------------|
B:          |-------------|

Here closing the file system when A is done, would result in a failure of B.

Or:

A: |-----------------------------|
B:          |-------------|

Here closing the file system when B is done, would result in a failure of A.

If this is to be solved I don't believe this can be solved in any other way then by some form of reference counting in the CloseablePath implementation.

[marcphilipp]

@oscargolding Thanks for elaborating! I wonder if you could avoid this situation by only calling Launcher.discover with classpath scanning once, inspecting the returned test plan, creating a new discovery request with class or unique id selectors, and using that one for all subsequent, concurrent calls to Launcher.execute. Would that work for you?