Guide to C++ Code analysis tools

In our quest to standardize code analyzers at work, I tested various tools aimed at improving code quality and security. I wanted to share my findings and insights in this article.

For the purpose of simplicity, I am going to only focus on code analysis which improves code quality and & security and remove the code linters. As most of the linters focus only on code style (such as tabs vs spaces, length of line etc) and syntax rules.

Categories of Code Analysis Tools

Code analysis tools can be broadly categorized into two types:

A.) Static Analysis tools

Static analysis tools examine code for errors, bugs, and potential vulnerabilities without executing it. They analyze the code's structure, syntax, and patterns to minimize bugs during development and can detect logical errors, such as using = instead of == in conditional statements.

Examples of error messages:

The uninitialized class member 'config' is used to initialize the 'comparator' member. Remember that members are initialized in the order of their declarations inside a class.

The assignment operator should be protected from the case of 'this == &other'

B. Dynamic Code Analysis Tools

Dynamic code analyzers evaluate a binary at runtime to identify issues like memory leaks and runtime errors. While they excel at detecting memory issues, they only analyze executed code paths. For instance, in the following code snippet, if the option is false, dynamic analyzers won't catch the memory bug, but static analyzers can:

if(option){
   SomeClass* p = new SomeClass(); 
   // Memory not deallocated
}        

Static Analysis Tools

1. Cppcheck

Cppcheck is a free and open source static analyzer for C and C++. It is the de facto tool and the best free analyzer. Everyone C++ project at least should use it. GitHub link to check and contribute.

2. PVS-Studio

PVS-Studio is a modern static analyzer for C++, C, and Java, with a strong focus on C++. I discovered it through the ISO C++ website. They have excellent blogs on their website and also support open-source software by making their products free for them. The great thing about them is that in addition to cli (a standalone tool) , it's also available as plugins for Qt Creator, Clion and Visual studio. One of the demerits of the ide plug-in that I found is that when analysing a large codebase sometimes it doesn't report all the errors (in the included files vs even we scan those files separately) due to some limitations.

3. Coverity

Now there are 2 types of tools : coverity and coverity scan. Coverity scan is the free tool for open source projects (but only supports gitlab only) but with some limitations.

Coverity (full edition) Support JS, C++, Java and python. Previously, coverity was one of the top 3 static code analysers but after it was acquired by synopsys, it has made less progress in improving and adding new checks. But now it's acquired by blackDuck and would like to see it back to its spot but right now it's not in the top 3 spot.

4. Perforce Klocwork

KlocWork is one of the oldest static analysis tool and only one that I think competing with Sonar at the top enterprise level. But it was too complicated to set up and one of my coworkers who was familiar with it (used it in Nokia) set it up and the results were great. One of the benefits is that it is supported my exotic C++ compilers and even QT support out of the box. It even supports various IEC and ISO functional safety standards that are required in some critical fields such as nuclear power, railways, medical devices etc and also one of the only few to even support misraC++23 standards.

5. Parasoft

Parasoft is the only one which has a static analysis tool as well as a dynamic analysis tool. While on paper specifications, it likely goes toe to toe with klocwork, but my experience with it was terrible. I requested access to both the static analysis tool as well as its dynamic analysis tool (Insecure++) as I wanted to compare it with sanitizers and valgrind. I got a meeting link of about 1 hour, so I joined with a coworker and that's when the experience went downhill. First was a boring ppt, then a live demo but the project seems like awful C like code by a university student with raw pointers everywhere, perhaps the only C++ feature it was using was classes. One could just look at the code and easily identify most of the obvious issues with the pointer handling and we miserably watched that demo. Also, its UI was also so outdated. After that, I asked a few questions also about the Insecure++, but he said that if I inquired about it, they responded that “the other team would handle queries about Insecure++” and then they ended the meeting quickly. Probably they have to give the same presentation to another one. Then no message or trial version was sent by them. Even worse, now that the evaluation process has ended, I recently got an email about Insecure++. It seems they built a product but then didn't find enough clients, so it's now not on their priority.

6. SonarQube/SonarCloud Sonar

They are the industry leader and support 25 languages including JS, TS, C++20, Java, python and even PHP. But unfortunately their free trial edition didn't include C++, so this was the product which I couldn't test personally. Let me know if you're interested in a comparison and would try to get my hands on its license.

7. Clang Tools

Clang tidy and clang analyzer are the only FOSS static analysis tools for C++ apart from CppCheck. But in my testing, I still found the CppCheck to give more error messages , but clang Tidy still has some code quality metrics like cyclomatic complexity, incorrect use of auto etc could come in handy for setting a benchmark for code quality

Dynamic Analysis Tools

There are a lot of static code analysis tools for C++, but only a few dynamic analysis tools. Especially the commercial ones are a lot less, either there isn't enough market or the open source tools are great enough, let me know what you think is the reason.

1. Parasoft's Insure++

Although it's the only commercial dynamic analysis tools I found, (as also explained above in parasoft static analysis tool) it seems they built a product but then didn't find enough clients so, it's now not on their priority.

2. Sanitizers

Sanitizers are developed by Google as a compact and lighter version of valgrind for memory issues, only 2x to 3x slower as compared to normal run.

3. Valgrind

Valgrind is the set of tools originally used for checking bugs and profiling code. Memcheck and Helgrind are the most important ones used for finding difficult to find address and memory related issues. They are much slower for running applications and I have seen, in some cases it is found to be 10x slower than running the normal binary. but it's unparalleled in detecting memory issues if your application can run under it.

Additional Tools Worth Mentioning

As I also told above that there a lot of tools available, and I thought to also mention some of the tools which I couldn't review yet, but are still under development and some even supports C++23

1. Fb Infer

The new version v1.2 is a major change to the Facebook static analysis tool and is now gaining popularity. Although I haven't tried it out yet, it seems now it could compete with CppCheck as the best open source static analysis tools. Let me know if you want a in depth comparison

2. JetBrains Qodana

Jetbrains is also trying to make its own static analysis tool, but currently for C++ is in early access and will see if they make it can compete with the industry leaders in the future

3. Ldra

4. Astree

5. Qt Axivion

6. PC lint plus : supports even C++20

7. HCL code scan

8. cppdepend : recently, I have seen great progress in CppDepend. It support C++ 23 as well as modules and now they are doubling down in AI (windows only feature) don't know whether it's just a marketing gimmick or a really useful feature

Summary

To ensure code security and quality:

• Always use both static and dynamic analysis tools.
• For Free Tools:

- Static Analysis: Cppcheck is my top choice.

- Dynamic Analysis: Valgrind is ideal if your application can run under it; otherwise, use Sanitizers.

- Additional Tools: Clang Analyzer, Facebook Infer. In addition, I found an amazing open source tool : CodeChecker It is a static analysis infrastructure tool which helps in running various tools like clang tidy, clang analyzer, CppCheck, Fb Inder etc. It has additional functionalities like marking false positives as well as hiding some issues.

• For Commercial Tools:

- Static Analysis: I recommend PVS-Studio for its user-friendly interface and impressive IDE integration, especially since it focuses primarily on C++.

- Enterprise Solutions: If your organization requires adherence to specific functional safety standards or additional features, consider Klocwork or SonarQube/SonarCloud.

I hope this guide assists you in choosing the right code analysis tools to enhance your code's quality and security. Feel free to share your experiences or ask questions in the comments below!

If you're interested in a detailed comparison of these tools—how effectively they find bugs, their unique strengths, and which one might be the best fit for different scenarios—let me know in the comments! If I see enough interest, I'll dive deeper and create a comprehensive analysis.