Async GlobalSetup does not work with InProcess toolchain

#521 added support for async GlobalSetup, but it appears that InProcess toolchain does not wait for the async setup method to finish. Simple reproducer:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Threading.Tasks;

namespace AsyncGlobalSetupRepro
{
    [InProcess]
    public class MyBenchmarkClass
    {
        private string _foo;

        [GlobalSetup]
        public async Task MyAsyncSetup()
        {
            await Task.Delay(1000);
            _foo = "hello";
        }

        [Benchmark]
        public int MyBenchmark() => _foo.Length; // NullReferenceException!
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            BenchmarkRunner.Run(typeof(MyBenchmarkClass));
        }
    }
}

It looks like MyAsyncSetup gets invoked as a regular method here, and then JITting starts immediately without waiting for it to finish:

BenchmarkDotNet/src/BenchmarkDotNet/Engines/EngineFactory.cs

Line 28 in e7ff4ce

    
           engineParameters.GlobalSetupAction?.Invoke(); // whatever the settings are, we MUST call global setup here, the global cleanup is part of Engine's Dispose

Hi, I experience the same problem here ! ✋

When using BDN with dotnet/crank, the --inProcess flag is passed to the BDN executable, and my benchmark throws a NullReferenceException. By testing my benchmarks directly with BDN (not through Crank) I realized that my async GlobalSetup method was not awaited/executed when the --inProcess flag is toggled.

Yeah, just hit this myself. Looks like #1968 will fix this so hopefully that gets merged in soon.

Are there any updates on the progress of this? I ran into this issue today with InProcess in an xunit test.

@AndreyAkinshin

I've had an open PR to fix this for a while, it just depends on when @AndreyAkinshin will get around to reviewing it.

In the meantime, you can work around this issue by synchronously waiting on your task task.GetAwaiter().GetResult().

Encountered this bug well over a year ago and found this issue. Returned to it today to see if it was fixed. Quite surprised that this issue is still open if all that's needed is for someone to review the fix.

Is there is a reason it hasn't been merged yet?

Navigation Menu

Repository navigation

Async GlobalSetup does not work with InProcess toolchain #1738

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Async GlobalSetup does not work with InProcess toolchain #1738

Description

Activity

corrieriluca commented on Oct 22, 2021

Turnerj commented on Sep 13, 2022

Rambolink94 commented on Aug 21, 2023

timcassell commented on Aug 22, 2023

agra6475 commented on Apr 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions