If you're not familiar with what is mutation testing, this article give a good explanation on the java case.
NinjaTurtlesMutation is a fork of NinjaTurtles. It aims to provide a fast mutation testing tool for codebases using NUnit v2.6.4.
NTM is fast and generates easy to read reports of mutation coverage:
Table of Contents
- Visual Studio 2010+. It's recommended that you have NuGet installed.
- .NET Framework v4.5+
The followings steps assume that you are using a sh-like terminal
git clone git@github.com:criteo/NinjaTurtlesMutation.git
To build from command-line (tested with MSBuild 4.6.1 / .NET 4.0.3):
/c/Windows/Microsoft.NET/Framework/v4.0.30319/MSBuild.exe NinjaTurtlesMutation.sln
or you can open NinjaTurtlesMutation.sln and build from Visual Studio
export PATH=$PATH:"path/to/your/clone/NinjaTurtlesMutation.Console/bin/Debug"
To illustrate basic usages, I'll use a dummy solution with two projects. One for the library, one for the tests
You can find this demo project inside Demofiles/PrimeFinderMutationPlayground
| PrimeFinderMutationPlayground
| PrimeFinderMutationPlayground.IDummy
| PrimeFinderMutationPlayground.Dummy
| PrimeFinderMutationPlayground.PrimeFinder
| PrimeFinder.UTest
| PrimeFinder.UTest.T_PrimeMutationPlayground
The library has one namespace (PrimeFinderMutationPlayground) with three classes. The test
library has one namespace (PrimeFinder.UTest) with one class.
When built, this solution generates two dlls: PrimeFinderMutationPlayground.dll and PrimeFinder.UTest.dll
ntm.exe run -c PrimeFinderMutationPlayground.PrimeFinder path/to/test/assembly/PrimeFinder.UTest.dll
This command start a mutation test on all methods contained in the PrimeFinder class. During his progress, NTM displays the full name of each method tested. This output the following:
NinjaTurtlesMutation - mutation testing for .NET (version 1.0.0.0)
Copyright (C) 2016 Tony Roussel
Copyright (C) 2012 David Musgrove
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrime(Int32)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrimed(Int32, IDummy)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder..ctor
Surviving mutants:
c:\Users\t.roussel\PrimeFinder\PrimeFinderMutationPlayground\PrimeFinderMutationPlayground\PrimeFinder.cs
Line 20
Deleting sequence points from IL. (1)
000c: deleting Ldstr, Newobj, Throw
[...]
Line 26
Deleting sequence points from IL. (1)
0035: deleting Ldc_I4, Stloc, Br
Replacing writes to local variables and fields. (1)
0036: write substitution Boolean.CS$1$0000 => Boolean.CS$4$0001
Mutation score: 0,88 (77 / 88)
Mutation testing failed
TL;DR: From bottom to top, you can find mutation test's result (fail/success), the mutation score, details about surviving mutants, and methods that NTM has tested.
At the bottom, NTM displays the mutation score (0.88): which is the ratio mutants killed over mutants generated (77 / 88). The higher the better, capping at 1.
Above the mutation score, surviving mutants are described. The listing is sorted by files, lines and then mutation type. Numbers in parentheses tell you how many mutants were generated by this kind of mutation operator, at this line. For each of those mutants, details of the IL modifications are printed.
And then on top of the surviving mutants list, NTM display all tested methods. Here we can see that 3 methods were found inside PrimeFinder class.
ntm.exe run -N PrimeFinderMutationPlayground path/to/test/assembly/PrimeFinder.UTest.dll
This command start a mutation test on all methods, of every classes contained in PrimeFinderMutationPlayground namespace. The output is like the "class mode" described in the previous section, except it cover more methods:
NinjaTurtlesMutation - mutation testing for .NET (version 1.0.0.0)
Copyright (C) 2016 Tony Roussel
Copyright (C) 2012 David Musgrove
Running mutation tests for PrimeFinderMutationPlayground.Dummy..ctor
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrime(Int32)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrimed(Int32, IDummy)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder..ctor
Surviving mutants:
c:\Users\t.roussel\PrimeFinder\PrimeFinderMutationPlayground\PrimeFinderMutationPlayground\PrimeFinder.cs
Line 20
Deleting sequence points from IL. (1)
000c: deleting Ldstr, Newobj, Throw
[...]
Mutation score: 0,88 (77 / 88)
Mutation testing failed
You can generate two kind of reports: XML and HTML.
They both contain the path and code of the original source files, a mutated lines listing, and which one survived.
The html file is built from the xml. So some informations can only be found in the xml report. These informations include the start/end lines/columns of the mutations (a mutation can span on several line), and details of IL changes made for each mutants, even the one that has been killed.
Use the --output [-o] option, and give a filename with it. It should look like this:
ntm.exe run -o mutation_report.xml -N PrimeFinderMutationPlayground PrimeFinder.UTest.dll
Like the xml report, use --output [-o] to specify a filename and set the format with the --format [-f]
option as follow:
ntm.exe run -o mutation_report.html -f HTML -N PrimeFinderMutationPlayground PrimeFinder.UTest.dll
- Only compatible with tests made with NUnit (2.6.2 to 2.6.4)
- Have a namespace mode to test several classes at once
- Report of surviving mutants in the console output
- Tests run are completely separated from the NTM application. A dispatching process exchange tests queries/results with NTM. The dispatcher also handle the workload between several test runner processes
- A NUnit.Core.RemoteTestRunner instance to run tests (no longer a nunit-console process)
- Benchmark mutant's tests set to detect infinite loop quicker
- Better handling of methods with generic parameters
- The HTML report include details about mutated line that survive
- Use symlink instead of file copy for non-mutated assembly
- Stability improvement
GPL/LGPL Version 3, 29 June 2007 More details in LICENSE files
Tony Roussel
David Musgrove - Owner of the NinjaTurtles project
A list of original NinjaTurtles project's owners/contributors can be found in the codeplex page
Copyright (C) 2016 Tony Roussel
Copyright (C) 2012-14 David Musgrove and others