Repository navigation
Open
Labels
Description
In BenchmarkDotNet version 0.12, the method that provided arguments for a parameterized benchmark is executed before the GlobalSetup method of the benchmark.
Having the designated GlobalSetup method execute before the ArgumentsSource method is useful when the arguments are costly to create, and - for example - need to be disposed once the benchmark has completed.
Example:
using BenchmarkDotNet.Attributes;
using System;
using System.Collections.Generic;
using System.IO;
public class ArgumentsSourceBenchmarks
{
private List<DownloadFile> _downloadFiles;
private StreamProvider _streamProvider;
[GlobalSetup]
public void GlobalSetup()
{
_streamProvider = new StreamProvider();
_downloadFiles = new List<DownloadFile>
{
new DownloadFile("A", _streamProvider.Create(2048)),
new DownloadFile("B", _streamProvider.Create(4096))
};
}
[GlobalCleanup]
public void GlobalCleanup()
{
_streamProvider?.Dispose();
foreach (var downloadFile in _downloadFiles)
{
downloadFile.DownloadStream.Dispose();
}
}
[Benchmark]
[ArgumentsSource(nameof(DownloadFileArguments))]
public void DownloadFileBenchmark(string path, Stream stream)
{
// Do something
}
public IEnumerable<object[]> DownloadFileArguments()
{
foreach (var downloadFile in _downloadFiles)
{
yield return new object[] { downloadFile.Path, downloadFile.DownloadStream };
}
}
public class DownloadFile
{
public DownloadFile(string path, Stream downloadStream)
{
Path = path;
DownloadStream = downloadStream;
}
public string Path { get; }
public Stream DownloadStream { get; }
}
public class StreamProvider : IDisposable
{
public void Dispose()
{
}
public Stream Create(int initialCapacity)
{
return new MemoryStream(initialCapacity);
}
}
}Running this benchmark class will current result in an NRE:
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at ArgumentsSourceBenchmarks.<DownloadFileArguments>d__4.MoveNext() in D:\Development\Repos\ArgumentsSource\ArgumentsSourceBenchmarks.cs:line 32
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkConverter.GetValidValuesForParamsSource(Type parentType, String sourceName)
at BenchmarkDotNet.Running.BenchmarkConverter.<GetArgumentsDefinitions>d__9.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type containingType, MethodInfo[] benchmarkMethods, ImmutableConfig immutableConfig)
at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
at Program.Main(String[] args) in D:\Development\Repos\ArgumentsSource\Program.cs:line 13Reacted by jeremyosterhoudt, Peter Bruch, Raul Santos, jonmotos, J. Zebedee, Alexander Vinnikov and Valeriy Gourov
Activity
adamsitnik commentedon Oct 21, 2020
Hi @drieseng
Thank you for a very detailed bug report.
I agree that the problem exists and should be solved.
For a now, I can offer a workaround that I've been using myself so far:
moving the setup logic to the ctor of the type returned by
[ArgumentsSource]You can find the example here.
What is important is that you can
overridethe.ToStringmethod in given type to change the way the arguments are represented in the results. Example: https://github.com/dotnet/performance/blob/2a4232d994b88c37093399537b0b2a62ef163aeb/src/benchmarks/micro/libraries/System.Text.Encodings.Web/Perf.Encoders.cs#L74Add broken benchmarks
ValeriyGourov commentedon Dec 2, 2025
Hi @adamsitnik
How can I initialize my data for
ArgumentsSourceasynchronously? For example:But it doesn't work.
timcassell commentedon Sep 12, 2026
It supports
IAsyncEnumerable<T>since #3248.timcassell commentedon Sep 12, 2026
This is a similar issue to #773. Arguments are obtained during discovery, cross-joined with params, and printed to the summary table. The example makes that type of discovery impossible, especially if the value is expected to be cleaned up by the same instance. The only way I could see this working is by introducing a new API like
[DependentArgumentsSource]that only retrieves the value at execution time, not discovery time, and excludes the arguments from the summary table. Is that what you're after? Although in that case, I wonder why simple fields aren't good enough? Is the benchmark sensitive enough for fields vs parameters to make a noticeable difference?