Editorial Note: I originally wrote this post for the NDepend blog. You can check out the original here, at their site. While you’re there, take a look around at some of the other posts, and sign up for the RSS feed, if you’re so inclined.
First things first. I really wanted to call this post, “who will analyze the analyzer,” because I fancy myself clever. This title would have mirrored the relatively famous Latin question from Satires, “who will guard the guards themselves?” But I suspect that the confusion I’d cause with that title would outweigh any appreciation of my cleverness.
So, without any literary references whatsoever, I’ll talk about static analyzers. More specifically, I’ll talk about how you should analyze them to determine fitness for your purpose.
Before I dive into that, however, let’s do a quick refresher on the definition of static analyzer. This stack overflow question nails it pretty well, right at the beginning of the accepted answer.
Analyzing code without executing it. Generally used to find bugs or ensure conformance to coding guidelines.
Succinctly put, Aaron, and just so. Most of what we do with code tends to be dynamic analysis. Whether through automated tests or manual running of the program, we fire it up and see what happens. Static analyzers, on the other hand, look at the code and use it to make deductions. These include both deductions about runtime behavior and about the codebase itself.
What’s Your Goal?
Why rehash the definition? Well, because I want to underscore the point that you can do many different things with static analyzers. Even if you just think of them as “that thing that complains at me about the Microsoft guidelines,” they cover a whole lot more ground.
As such, your first step in sizing up the field involves setting your own goals. What do you want out of the tool? Some of them focus exclusively on code quality. Others target specific concerns, such as behavioral correctness or security. Still others simply offer so-called “linting.” Some do a mix of many things.
Lay out your goals and expectations. Once you’ve done that, you will find that you’ve narrowed the field considerably. From there, you can proceed with a more apples to apples comparison.