Microsoft Pex

I’ve recently started to play around with Microsoft Pex, and I’m discovering (1) that there is a non-trivial learning curve; and (2) that it seems like a pretty powerful tool. As best I can tell at this early stage (and admittedly, I’m certainly a neophyte), there seem to be two major components. The first is the notion of parameterized unit testing (PUT) and the second is automated test generation.

For anyone not familiar with the idea of parameterized unit testing, it’s an idea that rather than the first bit of code below, you can do something that conceptually mirrors the second bit of code, except with better syntax. In the .NET world, I believe that MBUnit supplies this syntax natively (in the form of “row test”) and that users are left to reconstruct it with NUnit and MS Test. I’m too far removed at the moment from unit testing suites in Java or C++ to comment with any authority on those languages and PUTs.

public void SingleValueTest()
{
    var myFoo = new Foo();
    Assert.AreNotEqual(1, myFoo.FooIntProperty);
}

public void ParameterizedTest(int rangeStart, int rangeStop)
{
    var myFoo = new Foo();
    for(int index = rangeStart; index < rangeStop; index++)
        Assert.AreNotEqual(index, myFoo.FooIntProperty);
}

This second example is likely something that we’ve all done, with or without some supporting syntax of the test library. Obviously, unit test methods don’t take parameters, so you might have coded up some standard helper like the method above and call it from actual, decorated test methods (or, if you’re like me, you created an assert extension class that grows as you go). Pex creates a series of method decorating attributes and some of them appeared oriented toward generating tests with these semantics–they pepper your SUT (system under test) with ranges of values and see how it behaves.

But the really snazzy part of Pex that you can’t mimic with helper methods is that it analyzes your code, looking for interesting values. That is, you don’t supply “rangeStart” and “rangeStop” to Pex–it deduces them on its own, based on static and dynamic analysis of your code, and tries them out. The first time I ran Pex, it found all sorts of exception-generating conditions in code that I had thought to be fairly bulletproof, given that I had added a whole slew of contracts using Code Contracts to check preconditions and invariants and enforce post conditions. While somewhat humbling, that is extremely helpful. It’s better than the best possible manual code review (for seeing potential exceptions, at least), and better than the manual tests you generate. It’s automatically executing your code and guiding it towards exception-generating conditions.

Making things even better is the fact that Pex presents the results and gives you the option to actually generate unit tests (in your framework of choice) based on the exceptions it finds. It refers to this as promoting the tests. So the process flow is that you run a “Pex Exploration” which does dynamic analysis (mixed, I believe, with some forms of static analysis) to find exceptional conditions. It then shows you a list of the results of its parameterized runs and whether they passed, failed, or were inconclusive (there is a configurable timeout for each exploration) and allows you to pick and choose which results that you want to promote to unit tests saved in files in your codebase.

One thing that I find a bit off-putting is the fact that the tests it generates are strewn across a number of “.g.cs” partial classes, at least with MS Test. That is really just a matter of personal taste, and I can probably get used to it. It seems generate the partials based on the method in your code that it’s exploring and handle the setup for the call there, and then invoke the main class to actually call the method in question. This is kind of confusing, as the setup occurs in a more specific class and the actual test assert is somewhere else. However, when you open the test from the test result window (which is what I’ve been doing), you can set a breakpoint and step through the heavily attribute-decorated Pex boilerplate, winding up in your own code and seeing the exceptions. And that’s really what matters.

As I play more with this apparently slick and powerful tool, I’ll have more to say about it. I’m playing with it in conjunction with Code Contracts, and there seems to be a natural synergy between these products. I’ll have more to post on both subjects in this series of posts.