Benchmark DotNet

Updated 23/02/2025

“BenchmarkDotNet helps you to transform methods into benchmarks, track their performance, and share reproducible measurement experiments.”

It outputs useful results like the below which help compare methods. The code and examples below are based on Nick Chapsas - Stop Using FirstOrDefault in .NET! | Code Cop #021, How To Use BenchmarkDotNet - A Beginner’s Guide For C# Benchmarks and the BenchmarkDotNet docs

Some pointers that I thought were useful:

  • Consider testing with different .Net versions, .Net9 out performs .Net8 with features like LINQ, this is because MS has invested in LINQ to make it better
  • When testing, use larger data sets, ie: 5000 records in an array over 5 and always target the middle, eg: 2500 when Benchmarking

Demo

These examples below are simply comparing .Find (a method from the list collection) and FirstOrDefault (a LINQ method). Although the examples from Nick’s video focused on which of these methods are faster, my interest is how to use BenchmarkDotNet itself, so I can use it to compare my own algorithms.

Starting with a clean .Net console app follow these steps, my complete example is at https://github.com/carlpaton/BenchmarkDemo/tree/main/BenchmarkDotNetDemo

  1. Install BenchmarkDotNet (at the time of writing the version was 0.14.0)
  2. Create the test class Benchmarks and decorate it with MemoryDiagnoser(false), using false will cause the diagnoser to output allocated memory data, but it will ommit the detailed generational Garbage Collection data.
1
2
3
4
[MemoryDiagnoser(false)]
public class Benchmarks
{
    private readonly List<int> _rawNumber;

  1. Use the constructor to generate the data as we dont want to test this initialisation, additionally readonly is useful because we dont want to mutate shared test data.

  2. Create the tests and decorate with [Benchmark], these are contrived but help me understand how to use BenchmarkDotNet. The suffix Raw is just to denote that this is working with Value types and not Reference Types.

1
2
3
4
5
6
7
8
9
10
11
[Benchmark]
public int FindRaw() 
{ 
    return _rawNumber.Find(x => x > 500);
}

[Benchmark]
public int FirstOrDefaultRaw()
{
    return _rawNumber.FirstOrDefault(x => x > 500);
}
  1. Run the application as Release with dotnet build --configuration Release and observe the outputs, this takes a wee while, its interesting to see the drastic improvements by just changing the .Net version, watch Nicks video to understand why, he explains it far better than I ever could.
1
2
3
4
5
6
7
8
9
10
11
.NET 9
| Method            | Mean     | Error   | StdDev  | Allocated |
|------------------ |---------:|--------:|--------:|----------:|
| FindRaw           | 423.4 ns | 5.62 ns | 5.25 ns |         - |
| FirstOrDefaultRaw | 172.8 ns | 2.29 ns | 2.14 ns |         - |

.NET 8
| Method            | Mean       | Error    | StdDev   | Allocated |
|------------------ |-----------:|---------:|---------:|----------:|
| FindRaw           |   471.0 ns |  3.65 ns |  3.41 ns |         - |
| FirstOrDefaultRaw | 1,163.8 ns | 17.95 ns | 16.79 ns |      40 B |

Summary of what these columns mean

  • Method lists the names of the methods that were benchmarked
  • Mean average execution time of the method, measured in nanoseconds (ns), lower mean indicates better performance
  • Error margin of error for the mean, represents the uncertainty in the measured mean, smaller error indicates more precise results
  • StdDev Standard Deviation measured execution times, indicates the variability or spread of the results, smaller standard deviation means the execution times were more consistent
  • Allocated amount of memory allocated by the method during its execution, “40 B” means 40 bytes allocated and dash (-) means no memory allocation