Skip to content

ArgumentsSource method is executed before GlobalSetup method #1408

Open

Description

@drieseng

In BenchmarkDotNet version 0.12, the method that provided arguments for a parameterized benchmark is executed before the GlobalSetup method of the benchmark.

Having the designated GlobalSetup method execute before the ArgumentsSource method is useful when the arguments are costly to create, and - for example - need to be disposed once the benchmark has completed.

Example:

using BenchmarkDotNet.Attributes;
using System;
using System.Collections.Generic;
using System.IO;

public class ArgumentsSourceBenchmarks
{
    private List<DownloadFile> _downloadFiles;
    private StreamProvider _streamProvider;

    [GlobalSetup]
    public void GlobalSetup()
    {
        _streamProvider = new StreamProvider();
        _downloadFiles = new List<DownloadFile>
            {
                new DownloadFile("A", _streamProvider.Create(2048)),
                new DownloadFile("B", _streamProvider.Create(4096))
            };
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        _streamProvider?.Dispose();

        foreach (var downloadFile in _downloadFiles)
        {
            downloadFile.DownloadStream.Dispose();
        }
    }

    [Benchmark]
    [ArgumentsSource(nameof(DownloadFileArguments))]
    public void DownloadFileBenchmark(string path, Stream stream)
    {
        // Do something
    }

    public IEnumerable<object[]> DownloadFileArguments()
    {
        foreach (var downloadFile in _downloadFiles)
        {
            yield return new object[] { downloadFile.Path, downloadFile.DownloadStream };
        }
    }

    public class DownloadFile
    {
        public DownloadFile(string path, Stream downloadStream)
        {
            Path = path;
            DownloadStream = downloadStream;
        }

        public string Path { get; }
        public Stream DownloadStream { get; }
    }

    public class StreamProvider : IDisposable
    {
        public void Dispose()
        {
        }

        public Stream Create(int initialCapacity)
        {
            return new MemoryStream(initialCapacity);
        }
    }
}

Running this benchmark class will current result in an NRE:

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at ArgumentsSourceBenchmarks.<DownloadFileArguments>d__4.MoveNext() in D:\Development\Repos\ArgumentsSource\ArgumentsSourceBenchmarks.cs:line 32
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkConverter.GetValidValuesForParamsSource(Type parentType, String sourceName)
   at BenchmarkDotNet.Running.BenchmarkConverter.<GetArgumentsDefinitions>d__9.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type containingType, MethodInfo[] benchmarkMethods, ImmutableConfig immutableConfig)
   at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
   at Program.Main(String[] args) in D:\Development\Repos\ArgumentsSource\Program.cs:line 13

Activity

  1. adamsitnik commented on Oct 21, 2020

    @adamsitnik
    Member

    Hi @drieseng

    Thank you for a very detailed bug report.

    I agree that the problem exists and should be solved.

    For a now, I can offer a workaround that I've been using myself so far:

    moving the setup logic to the ctor of the type returned by [ArgumentsSource]

    You can find the example here.

    What is important is that you can override the .ToString method in given type to change the way the arguments are represented in the results. Example: https://github.com/dotnet/performance/blob/2a4232d994b88c37093399537b0b2a62ef163aeb/src/benchmarks/micro/libraries/System.Text.Encodings.Web/Perf.Encoders.cs#L74

  2. added this to the v0.12.x milestone on Oct 21, 2020
  3. added a commit that references this issue on Jan 28, 2024
  4. ValeriyGourov commented on Dec 2, 2025

    @ValeriyGourov

    For a now, I can offer a workaround that I've been using myself so far:

    moving the setup logic to the ctor of the type returned by [ArgumentsSource]

    You can find the example here.

    Hi @adamsitnik

    How can I initialize my data for ArgumentsSource asynchronously? For example:

    using System.Diagnostics;
    
    using BenchmarkDotNet.Attributes;
    using BenchmarkDotNet.Configs;
    using BenchmarkDotNet.Running;
    
    IConfig config = Debugger.IsAttached
        ? new DebugInProcessConfig()
        : DefaultConfig.Instance;
    
    _ = BenchmarkRunner.Run<Benchmarks>(config);
    
    public class Benchmarks
    {
        private List<SomeClass> _data = [];
    
        [GlobalSetup(Target = nameof(Test1))]
        public async Task Test1GlobalSetup()
        {
            await Task.Delay(100);
    
            _data = [ new (), new () ];
        }
    
        public IEnumerable<SomeClass> TestData => _data;
    
        [Benchmark]
        [ArgumentsSource(nameof(TestData))]
        public Task Test1(SomeClass someClass) => Task.CompletedTask;
    }
    
    public class SomeClass
    {
        public override string ToString() => nameof(SomeClass);
    }

    But it doesn't work.

  5. removed this from the v0.14.x milestone on Dec 3, 2025
  6. self-assigned this
    on Aug 28, 2026
  7. added this to the v0.16.0 milestone on Aug 28, 2026
  8. timcassell commented on Sep 12, 2026

    @timcassell
    Collaborator

    How can I initialize my data for ArgumentsSource asynchronously?

    It supports IAsyncEnumerable<T> since #3248.

  9. timcassell commented on Sep 12, 2026

    @timcassell
    Collaborator

    This is a similar issue to #773. Arguments are obtained during discovery, cross-joined with params, and printed to the summary table. The example makes that type of discovery impossible, especially if the value is expected to be cleaned up by the same instance. The only way I could see this working is by introducing a new API like [DependentArgumentsSource] that only retrieves the value at execution time, not discovery time, and excludes the arguments from the summary table. Is that what you're after? Although in that case, I wonder why simple fields aren't good enough? Is the benchmark sensitive enough for fields vs parameters to make a noticeable difference?

  10. removed their assignment
    on Sep 12, 2026
  11. removed this from the v0.16.0 milestone on Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions