Use MSTest To Run Your Tests Via Command Line

One step in setting up your continuous integration environment is getting your tests to run via a command line. My development environment is Visual Studio and I am using the Microsoft testing framework. I did not imagine that there would be any trickery in getting this to work. I wanted to be able to write something like:

I wasn’t really expecting this to work, because I had already seen the MSTest online command options. This is how I wanted it to work, but no luck. I was thinking that since you can run all the tests in a solution via the menu in Visual Studio, that you should be able point the MSTest executable at the solution and it would determine what tests to run. If you go to the command line options page for MSTest you will find that MSTest will take a ‘testmetadata’ option that allows you to specify a vsmdi file. In case you didn’t know the vsmdi file is the test metadata file that Visual Studio generates for your project that contains tests. Here is a screenshot of a folder containing a solution that contains tests. The highlighted file is the vsmdi file.

Great. So if I use the following at the command prompt I should get my tests to run.

Wrong! I was kind of expecting this to work. So what went wrong? “No tests to execute”…how could that be? The project has a lot of tests. It turns out that Visual Studio Professional 2008 doesn’t populate the vsmdi file without a little extra work.

First select the ‘Test List Editor’ using the following menu:

You will then the following:

Right-click the ‘List of Tests’ option in the tree view on the left and select the ‘New Test List…’ option as shown below:

Fill out the dialog with a meaningful name and description:

Click the ‘All Loaded Tests’ option in the tree view to display a list of available tests. Right click on the displayed list and select the ‘Select All’ menu item.

Drag the selected list into the ‘TestSuite’ (or what every you named yours) tree view item and drop the list. Now if you revisit your vsmdi file it will be populated with your tests.

Running MSTest at the command prompt with the ‘testmetadata’ option pointing to your new vsmdi file will run the tests.

This is great and the results are displayed in the console window after the tests complete:

Two failing tests…looks like I got some work to do still on this project.

It is important to copy and rename your vsmdi file because Visual Studio will overwrite it. It is a bit of maintenance to generate a new file when ever you add tests.

What if I want a report that will be persisted with the build. There is ‘resultsfile’ option that allows you to specify a file to save the test results.

Now you will get a results file. This file is an XML file that contains the results of your tests. Here is a screenshot from the file:

I find this very hard to read. I don’t think it was intended on being read by humans. I believe if you have the Team version of Visual Studio this may be rendered into a more usable format. If you do some searching, you can find XSLT transformations for this XML file. Here is a codeplex project that transforms this data to HTML. After this transformation the results look like the following:

The transformed results are visually more pleasing.

Summary

Now we have tests being run via the command line. I know that this is a lot easier using some of the other testing frameworks. If anyone wants to share their tricks to get MSTest to run via command line, I would like to hear.