How to Compensate for the Top 3 Google Test Mistakes

Google Test is Google's opensource framework for writing C++ tests. It's used widely inside and outside of Google. If you write C++, chances are you've used Google Test or have heard of it.

I was the original and primary author of Google Test - I developed it as my starter project when I first joined Google 20 years ago. It's hard to believe that this project will celebrate its 20th anniversary in June this year. How time flies!

Any non-toy software system has bugs or design defects. I've certainly made my share of mistakes while developing Google Test. Some of the mistakes were fixed, but some were left alone as fixing them was deemed too costly.

As the saying goes, hindsight is 20/20. I ask myself what I would do differently if I were to design Google Test again. Three things come to my mind immediately. Next, I'm going to explain what they are and what you can do to compensate for my mistakes. If you follow my suggestions, I believe you'll get tests that are easier to maintain and more correct.

Let's go.

Regret #1: I should've made Google Test run the test cases in a random order by default.

When I first wrote Google Test, I let it run the test cases in the order they are defined. It seemed a natural choice. After all, this behavior was easy to implement, easy to understand, deterministic, and unsurprising.

Unfortunately, this also makes it easy to let one test case depend on the state created by other test cases before it. Such dependencies are bad - with them, to understand what a test case does you will need to also understand what other test cases do.

Really, test cases should all be independent or each other - if a test case fails, I should be able to debug it by running it alone, skipping all other test cases; I shouldn't have to face the surprise when deleting one test case suddenly breaks the other test cases.

To guide people toward the good habit of writing independent test cases, I added a flag --gtest_shuffle to let Google Test run all test cases in a test program in a random order. This will pretty much shake out any hidden dependencies between test cases. I wish this were the default behavior, but changing the default when you have lots of users was hard, as the change would break many builds and people won't be happy about it. So now we are stuck with deterministic execution order.

However, I strongly suggest to update your test script to pass --gtest_shuffle to your Google Test programs. This should be the first thing you do when you start a new project using Google Test.

Regret #2: I should've made Google Test fail if no test case is linked in.

What do I mean? If you have a test program that contains 0 test cases, Google Test will tell you that everything is fine (after all, the test doesn't generate any failure).

This, however, isn't what we want most of the time. Sure, there may be legitimate uses of a test program containing no test case, but that should be pretty rare. Most likely, when no test case is found, the author has made a mistake and Google Test should help the author by letting them know about their mistake.

How could one make such a mistake? You might wonder. This can happen surprisingly easily. For example, someone may use a glob pattern (like "foo_*_test.cpp") to match test source files in their build script. If the glob pattern is written wrong (maybe the files are named like foo-abc-test.cpp instead of foo_abc_test.cpp), it might match nothing, which will silently lead to the test "passing" while no test case is being run!

Another common scenario where one ends up with 0 test cases is putting all test definitions in a library and having the test program depend on the library. When using static linking, this results in none of the test cases in the library being linked in, as Google Test relies on the TEST, etc, macros to register test cases during dynamic initialization. Since the main() function of a test program doesn't explicitly call the test cases, the linker will think that the test library is unused and strip it from the final program. If you ever put test case definitions in a library, be sure to mark the library as "alwayslink" to prevent the linker from skipping it.

So, it would be really nice if Google Test detects such programmer errors. For many years, it's impossible to configure Google Test to have this behavior, and people resort to custom build script tricks to get similar functionalities, which is honestly clunky.

Yesterday (2025/2/24) I contributed a feature to address this problem. If you upgrade to the latest Google Test, you can pass a --gtest_fail_if_no_test_linked flag to make the test fail if no test case is linked in.

Unfortunately, we cannot make it the default as it would break tons of builds, but at least you have this option now.

I recommend updating your test scripts to pass this flag to your tests.

Regret #3: I should've not provided the SetUp() and TearDown() API.

For those of you unfamiliar with Google Test, it follows the xUnit pattern where each test case runs on a new test fixture object. Before the interesting part of the test case runs, one can initialize the fixture object by overriding the fixture class's SetUp() function. Similarly, after the test case finishes, the fixture object's TearDown() function will be called to clean up the stage (releasing any resources obtained during the test case, restoring any mutated state so that everything appears nice and new, etc).

What I realized too late was that SetUp() and TearDown() are completely unnecessary. There's something better: we can just do the initialization and clean-up in the fixture class's default constructor and destructor instead, as Google Test always creates/destroys a test fixture object for each test case.

So we ended up with two ways to do the same thing. Such redundancy is unnecessary and annoying, as it leads to inconsistency. Now, it's a bit too late to remove the SetUp() and TearDown() API as too many people depend on them.

My recommendation is to avoid using SetUp() and TearDown(). The fixture constructor and destructor can do the same and have a bunch of advantages:

1. If we initialize a member variable in the constructor instead of in SetUp(), we can make it const. This makes the test easier to understand (as we know that the member variable's value will never change) and avoids bugs where one unintentionally modifies a fixture member variable.
2. When we subclass a fixture class, the subclass's constructor/destructor will always call the base class's constructor/destructor, so we don't have to remember to do it - it's done automatically by the compiler. In contrast, if we put logic in SetUp() and TearDown() and decide to override them in the subclass, we have to remember to call the base class's SetUp() and TearDown() from the derived class's SetUp() and TearDown(). It's just tedious and error-prone.

If you haven't, you probably should add a linter to your project to prevent the use of SetUp() and TearDown() in test code.

I surely have other regrets about Google Test, but 3 is a good number for one sitting. Maybe I'll follow up with another article later.

What are your biggest complaints about Google Test? What do you do to compensate for them? I'd love to hear!