MS Test Report Generator

I’ve been working on a side project for a while, and today I uploaded it to Source Forge. The project is a tool that takes XML results generated by MS Test runs and turns them into HTML-based reports. I believe that TFS will do this for you if you have fully Microsoft-integrated everything, but, in the environment I’m currently working in, I don’t.

So I created this guy.

Back Story

I was working on a build process with a number of constraints that predated my tenure on the process itself. My task was to integrate unit tests into the build and have the build fail if the unit tests were not all in a passing state. The unit test harness was MS Test and the build tool was Final Builder.

During the course of this process, some unit tests had been in a failing state for some period of time. Many of these were written by people making a good faith effort but nibbling at an unfortunate amount of global state and singletons. These tests fail erratically and some of the developers that wrote them fell out of the habit of running the tests, presumably due to frustration. I created a scheme where the build would only run tests decorated with the “TestType” attribute “Proven”. In this fashion, people could write unit tests, ‘promote’ them to be part of the build, and have it as an opt-in situation. My reasoning here is that I didn’t want to deter people who were on the fence about testing by having them be responsible for failing the build because they didn’t know exactly what they were doing.

After poking around some, I saw that there was no native final builder action that accomplished what I wanted– to execute a certain subset of tests and display a report. So I created my own batch scripts (not included in the project) that would execute the MS Test command line executable with filtering parameters. This produces an XML based output. I scan that output for the run result and, if it isn’t equal to “Passed”, I fail the build. From there, I generate a report using my custom utility so that people can see stats about the tests.

Report Screenshot

Design

During the course of my spare time, I decided to play around with some architectural concepts and new goodies added to C# in the .NET 4.0 release. Specifically, I added some default parameters and experimented with co-variance and contra-variance. In terms of architecture, I experimented with adding the IRepository pattern to an existing tiered architecture.

On the whole, the design is extensible, flexible, and highly modular. It covered about 99.7% by unit tests and, consequently, was complete overkill for a little command line utility designed to generating an HTML report. However, I was thinking bigger. Early on, I decided I wanted to put this on Source Forge. The reason I designed it the way I did was to allow for expansion of the utility into a GUI-driven application that can jack into a server database and maintain aggregate unit testing statistics on a project. Over the course of time, you can track and get reports on things like test passing rate, test addition rate, which developers write tests, etc. For that, the architecture of the application is very well suited.

The various testing domain objects are read into memory, and the XML test file and HTML output file are treated as just another kind of persistence. So in order to adapt the application in its current incarnation to, say, write the run results to a MySQL or SQL server database, it would only be necessary to add a few classes and modify main to persist the results.

Whether I actually do this or not is still up in there air, and may depend upon how much, if any, interest it generates on Source Forge. If people would find it useful, both in general and specifically on projects I work on, I’m almost certain to do it. If no one cares, I have a lot of projects that I work on.

How to Access

You can download a zipped MSI installer from SourceForge.

If you want to look at the source code, you can do a checkout/export on the following SVN address: https://mstestreportgen.svn.sourceforge.net/svnroot/mstestreportgen

That will provide read-only access to the source.

  • Kym

    I downloaded your tool and installed it, but you included no instructions on how to use it anywhere on your site, on sourceforge, or after downloading. May want to think of your users…

  • http://www.daedtech.com Erik

    My apologies. Let me check my archives and see if I have them somewhere and forgot to upload them. If not, I’ll put some instructions together and post them to sourceforge and as a follow up post here.

    In the meantime, if you have any specific questions, feel free to email me (erik at daedtech.com) or post them as comments.

  • http://www.daedtech.com Erik

    Alright, so I can’t find where I put the instructions, so I’ll throw something up here real quick to get you started, and then I’ll take care of writing them up and publishing them later.

    The MSI will install the executable “BasicReportGenerator.exe”. That executable is intended to run from the command line and it takes a single parameter: the MS Test run report. If you run MS Test, it will generate a .trx file in the test results folder. Take that file and rename it to have an XML extension — results.xml or something. From there, you run the report generator on it, and it will create a new file in that directory called results.xml.htm, and that will be your report.

    There are a couple of gotchas that resulted from this being specific to the process that I was setting up:

    (1) The compiled executable only works on source file with .XML extension. If you want it to operate on the .trx extension, go to DaoFactoryProvider.cs and add a case for “trx” to return a new DaoFactoryMsTestImpl(_targetDocument) (line 53 of that file).

    (2) The executable generates the report file in its current directory. This was perfect for what I was doing, but not so perfect when that’s in “C:\program files”. If you’re using this in a batch script as I did, that might work out fine for you as you can just add a move command if necessary. If not, Program.cs has a “SaveReport” method that you can modify to suit your needs in terms of the intended path for saving the report.

    Truth be told, this didn’t generate much interest, so I haven’t touched it since posting. I slapped the MSI installer on as part of the project to make things more convenient, but I see that it might be more of a red herring than anything at this point. I’m going to make a few changes to it to distinguish between interactive and command line mode, add some instructions to the sourceforge project, and then make a follow up post here.

    Thanks for bringing the issues to my attention.

  • http://www.daedtech.com Erik

    So, I wound up getting started down the rabbit hole here. It occurred to me that it might be nice to have a GUI for this, so I started out making a simple one and got a little carried away. I’m finished with my changes and adding some additional unit tests, so I’m hoping to have V1.1 up on the blog in the next few days, which will include instructions and a GUI from the MSI installer. I’m going to create a new post as a follow up to this one talking about the utility in more detail.

  • http://www.facebook.com/ Hippie

    Grazi for making it nice and EZ.

  • Bill N

    I’m very interested in the GUI version but haven’t seen in yet or i didn’t see it in the installer that i used. Is it still coming? I see it in your source code but I don’t have subversion and can’t download the whole solution.

  • http://www.daedtech.com Erik

    Hi Bill,

    Thanks for reading and for your interest in the tool. The GUI version is on sourceforge at http://sourceforge.net/projects/mstestreportgen/ (or, this link will take you right to the download: http://sourceforge.net/projects/mstestreportgen/files/latest/download )

    If you download v1.1 (typo-ed as “v1,1″) and run the MSI installer, you will wind up with two executables in your target directory, “CommandLineGenerator.exe” and “ReportGenerator.exe”. You want the latter for the GUI tool.

    Here is a follow up post I made after posting V1.1: http://www.daedtech.com/ms-test-report-generator-v1-1

    If this isn’t helpful or you’ve already tried this, feel free to post back here for follow up or to email me if you like.