Benchmarking .Net Versions for Collection Operations

With the imminent release of .Net 8 LTS on the horizon, I decided to run a few benchmarks to measure the performance gains on .Net 8 Preview 2 over a few of the most recent releases of .Net, specifically, the versions benchmarked were:

• .Net Framework 4.8
• .Net 5.0
• .Net 6.0
• .Net 7.0

We all use collections in C# ubiquitously, specifically List<T>. I wanted to know if List<T> is the best approach. So I also benchmarked a few other collection types; the types measured were:

• Collection<T>
• List<T>
• HashSet<T>
• ImmutableHashSet<T>
• SortedHashSet<T>

I created a benchmark that adds the same 100 000 Guids represented as strings to each collection type; I referenced one of the Guids randomly from the collections and used that as my search text (_itemToFind). A snippet of the benchmarked code for each collection type looks as follows:

I decided to use the "Contains" method for each of these types as I am sure all of us have used this method at some point in our production code; knowing how it performs concerning other collection types seems helpful. Now, by no means is this benchmark the most accurate, I ran it on a reasonably new laptop, but it was only run once. Nonetheless, it gives a good overview of expected performance.

Here's the results:

You should zoom in a bit to see the detail (if you're interested). I found a surprise lurking in that result.

Don't worry too much about the column headings. The most important ones are

• "Median"(it is a more robust measure of central tendency that is less sensitive to outliers), and
• "Ratio"(showing the relative performance of each measure against the baseline, in this case, .Net Framework 4.8).

So what was the surprise here?

Look at the "List_Contains" result for the various .Net versions. The benchmarking framework (BenchmarkDotNet) does some amazing things to reduce inaccuracies, like running the benchmark 100 times for each method for each framework version and accounting for identified outliers.

It shows that .Net 5.0 is 31.43 times faster than the baseline (.Net Framework 4.8) when performing the List<T>.Contains() operation. It's not surprising that it is so fast; surprisingly, it is much quicker than .Net 6.0 LTS and even the latest (at the time of writing) .Net 8.0 Preview 2.

What version of .Net should you use to optimise for ".Contains()"?

It depends; what is the collection you are using? If, for instance, like most of us, you lean on List<T> for most of your collections in c#. The answer depends on the list's size and the types of elements in it. Still, for this sample set, which is 100 000 strings, it appears that .Net 5.0 is the answer, now there are many reasons not to use .Net 5.0 in production code, but if your primary goal is to optimise for this use case, then it is worth considering.

The fastest collection type to use when optimising this type of operation is a HastSet<T> on the latest .Net 8.0 Preview version.

The benchmark highlights that List<T> shouldn't always be the "go-to" collection type. A bit of thought into our collections' use and expected performance could point us to use better, more fit-for-purpose types in our code.

Have you run similar tests? I'd love to hear what you found and if it helped you make better decisions about your code.