NUnit Support in TeamCity
There are two most common methods to set up NUnit tests reporting in TeamCity:
Besides, you can try alternative approaches or run tests in any other runner (like PowerShell or Command Line) with the TeamCity VSTest Adapter.
This article describes the specifics of the NUnit support in TeamCity and the alternative approaches to establishing it.
Supported NUnit Versions
The following NUnit versions are supported: 2.2.10, 2.4.1, 2.4.6, 2.4.7, 2.4.8, 2.5.0, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.5.6, 2.5.7, 2.5.8, 2.5.9, 2.5.10, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 3.0.
It is possible to have several versions of NUnit installed on an agent machine and use any of them in a build.
NUnit Framework Compatibility
The following table represents the compatibility of TeamCity runners with the .NET implementations:
.NET Framework 1, 1.1, 2, 3.5 | .NET Framework 4+ | .NET Core 1+ and .NET 5+ | |
---|---|---|---|
NUnit runner
| |||
.NET runner
| |||
VSTest adapter in other runners |
Alternative Approaches
If using the NUnit or .NET build runners is inapplicable, TeamCity provides the following ways to configure NUnit tests reporting in TeamCity:
The standard NUnit for NAnt Build Runner.
The NUnit Test Launcher that can be configured in the MSBuild build script or launched from the command line.
TeamCity Add-in for NUnit is available to turn on reporting on the NUnit level without build procedure modifications.
The bundled XML Test Reporting plugin allows importing any XML report to TeamCity. In this case, it is not always possible to track results on the fly.
You can add the XML Report Processing build feature to your build configuration, or use the following service message:
##teamcity[importData type='sometype' path='<path to the xml file>']
. Learn more: XML Report Processing, Importing XML Reports.Configuring test reports manually via service messages.
Feature Comparison for Alternative Approaches
Approach | Real-time Reporting | Execution in CLI-based runners | Tests Reordering | Implicit TeamCity .NET Coverage |
---|---|---|---|---|
\<nunit2\> NAnt task | /* | |||
\<NUnit\> MSBuild task | /* | |||
\<NUnitTeamCity\> MSBuild task | /* | |||
TeamCity Add-in for NUnit | ||||
TeamCity NUnit Test Launcher | ||||
XML Reporting Plugin | N/A | N/A |
* TeamCity-provided tasks may have different syntax/behavior. Some workarounds may be required to run the script without TeamCity.
In addition to the common test reporting features, TeamCity allows running NUnit tests under the x86 process on the x64 machine by introducing an explicit specification of the platform and runtime environment versions. You can define whether to use .NET Framework 1.1, 2.0 or 4.0 started under an MSIL, x64 or x86 platform.
NUnit Test Launcher
TeamCity provides its own NUnit tests launcher that can be used from the command line. The tests are run according to the passed parameters and, if the process is run inside the TeamCity build agent environment, the results are reported to the TeamCity agent.
You can pass the following command-line options to the TeamCity NUnit Test Launcher:
Option | Description |
---|---|
| Version of .NET Framework to run tests. Acceptable values are v1.1, v2.0, v4.0, and ANY. |
| Platform to run tests. Acceptable values are x86, x64, and MSIL. For .NET Framework 1.1, only MSIL option is available. |
| The test framework to use. The value has to be specified in the following format: |
| The list of categories separated by |
| The list of categories separated by |
| List of third-party NUnit add-ins to use (optional). |
| List of assembly paths separated by |
| Specify to run each assembly in a new process. |
Category Expression
Beginning with NUnit 2.4.6 and up to but not including NUnit v3.0, a category expression can be used. This table shows some examples:
Expression | Action |
---|---|
A|B|C | Selects tests having any of the categories A, B or C. |
A,B,C | Selects tests having any of the categories A, B or C. |
A+B+C | Selects only tests having all three categories assigned. |
A+B|C | Selects tests with both A and B OR with category C. |
A+B\-C | Selects tests with both A and B but not C. |
-A | Selects tests not having category A assigned. |
A+(B|C) | Selects tests having both category A and either of B or C. |
A+B,C | Selects tests having both category A and either of B or C. |
Note: As shown by the last two examples, the comma operator (,
) is equivalent to the pipe (|
) but has a higher precedence. The order of evaluation is as follows:
Unary exclusion operator (
-
).High-precedence union operator (
,
).Intersection and set subtraction operators (
+
and binary-
).Low-precedence union operator (
|
).
Since the operator characters have special meaning, avoid creating a category that uses any of them in its name. For example, the category db-tests
must not be used in the command line, as it appears to mean "run category db, except for category tests". The same limitation applies to the characters having a special meaning for the shell you are using.
The following examples assume that the teamcity.dotnet.nunitlauncher
property is set as a system property on the Parameters page of Build Configuration Settings.
Run tests from an assembly:
Run tests from an assembly with the NUnit categories filter:
Run tests from assemblies:
NUnit for NAnt Build Runner
This section assumes, that you already have a NAnt build script with the configured nunit2
task in it, and want TeamCity to track test reports without making any changes to the existing build script. Otherwise, consider adding the NUnit build runner as one of the steps for your build configuration.
To track tests defined in a NAnt build via the standard nunt2
task, TeamCity provides a custom task implementation and automatically replaces the original <nunit2>
task with its own task. When the build is triggered, TeamCity starts TeamCity NUnit Test Launcher using own implementation of <nunit2>
. This allows you to leave your the build script without changes and receive on-the-fly test reports in TeamCity.
If you don't want TeamCity to replace the original nunit2
task, consider the following options:
Use the NUnit console with TeamCity Add-in for NUnit.
Import XML test results via the XML Test Report plugin.
Use the command-line NUnit Test Launcher.
Configure reporting tests manually via service messages.
To disable
nunit2
task replacement, set theteamcity.dotnet.nant.replaceTasks
system property tofalse
.
The nunt2
task implementation in TeamCity supports additional options that can be specified either as NAnt <property>
tasks in the build script, or as system properties under Build Configuration | Build Parameters.
The following options are supported for the TeamCity <nunit2>
task implementation:
Property | Description |
---|---|
| Runs all tests regardless of the number of failed ones. Fails if at least one test has failed. |
| Sets desired runtime execution mode for .NET 2.0 on x64 machines. Supported values are x86, x64, and ANY (default). |
| Sets the required .NET Framework version. Supported values are v1.1, v2.0, v4.0. The default value is equal to the NAnt target framework. |
| Specifies which version of the NUnit runner to use. The value has to be specified in the following format: It is possible to have several versions of NUnit installed on an agent machine and use any of them in a build. |
| Specifies the list of third-party NUnit add-ins used for NAnt build runner. |
| Set |
TeamCity NUnit test launcher will run tests in the .NET Framework, which is specified by the NAnt target framework: that is, on .NET Framework 1.1, 2.0 or 4.0 runtime. TeamCity also supports test categories for the <nunit2>
task.
Adding the listed properties to the NAnt build script makes it TeamCity-dependent. To avoid this, specify properties as system properties under Build Configuration, or consider adding the <if>
task.
Examples
Start tests form a single assembly files under x64 mode on .NET 2.0.
Run all tests from category C1, but not C2.
Explicitly specify the version on NUnit to run tests with.
Note that in this case, the following property should be added before the nunit2
task call.
Using NUnit for MSBuild
This section describes how to use NUnit from MSBuild.
Working with NUnit Task in MSBuild Build
This section assumes that you already have an MSBuild build script with a configured NUnit
task in it, and want TeamCity to track test reports without making any changes to the existing build script. Otherwise, consider adding the NUnit build runner as one of the steps of your build configuration.
Using NUnitTeamCity task in MSBuild Build Script
TeamCity provides a custom NUnitTeamCity
task compatible with the NUnit
task from MSBuild Community tasks project. If you provide the NUnitTeamCity
task in your build script, TeamCity will launch its own test runner based on the options specified within the task. Thus, you do not need to have any NUnit runner, because TeamCity will run the tests.
To use the NUnitTeamCity
task correctly:
Make sure the
teamcity_dotnet_nunitlauncher
system property is accessible on build agents. Build agents running Windows should automatically detect these properties as environment variables. If you need to set them manually, see defining agent-specific properties for more information.Configure your MSBuild build script with the
NUnitTeamCity
task using the following syntax:<UsingTask TaskName="NUnitTeamCity" AssemblyFile="$(teamcity_dotnet_nunitlauncher_msbuild_task)" /> <NUnitTeamCity Assemblies="@(assemblies_to_test)" />
The NUnitTeamCity
task supports the following attributes:
Property | Description |
---|---|
| Execution mode on a x64 machine. Supported values are: x86, x64, and ANY. |
| .NET Framework to use: v1.1, v2.0, v4.0, ANY. By default, the MSBuild runtime is used. Default is v2.0 for MSBuild 2.0 and 3.5. For MSBuild 4.0 the default value is v4.0. |
| As used in the |
| As used in the |
| Version of NUnit to be used to run the tests. To use NUnit 3.0 and above, see the section below. |
| List of third-party NUnit add-ins to be used. For more information on using NUnit add-ins, refer to the NUnit Add-ins Support section. |
| True to fail task, if any test fails. |
| List of assemblies to run tests with. |
| Set |
The custom TeamCity NUnit
task also supports additional attributes. For the list of available attributes refer to this section.
If you need the TeamCity test runner to support third-party NUnit add-ins, refer to the NUnit Add-ins Support section for the details.
Example (part of the MSBuild build script):
Important notes
Be sure to replace
.
with_
when using system properties in MSBuild scripts. For example, useteamcity_dotnet_nunitlauncher_msbuild_task
instead ofteamcity.dotnet.nunitlauncher.msbuild.task
.TeamCity also provides Visual Studio Solution Runner for solution files of Microsoft Visual Studio 2005 and above. It allows you to use MSBuild-style wildcards for the assemblies to run unit tests on.
Examples
Run NUnit tests using specific NUnit runner version:
Run NUnit tests with custom add-ins with NUnit 2.4.6:
Run NUnit tests with custom add-ins with NUnit 2.4.6 in per-assembly mode:
To make a TeamCity independent build script, consider the following option:
The MSBuild property TEAMCITY_VERSION
is added to MSBuild when started from TeamCity.
Working with NUnit 3.0
Starting from version 3.0, NUnit supports TeamCity natively, so there is no need to use a special task for MSBuild as it was done for the earlier NUnit versions. The simplest way is to run the NUnit console via the standard Exec task.
The Getting Started with NUnit article contains details and examples.
MSBuild Service Tasks
For MSBuild, TeamCity provides the following service tasks that implement the same options as the Build Script Interaction:
TeamCitySetBuildNumber
allows changing the build number:
It is possible to use {build.number}
as a placeholder for older build number.
TeamCityProgressMessage
allows writing a progress message:
TeamCityPublishArtifacts
allows publishing all artifacts from the MSBuild item group:
TeamCityReportStatsValue
is a handy task for publishing statistic values:
TeamCityBuildProblem
reports a build problem which actually fails the build. Build problems appear on the Build Results page and also affect build status text:
The mandatory
description
attribute is a human-readable text describing the build problem. By default,description
appears in the build status text.identity
is an optional attribute and characterizes a particular build problem instance. It shouldn't change throughout builds if the same problem occurs: for example, the same compilation error. It should be a valid Java ID up to 60 characters long. By default,identity
is calculated based ondescription
.
TeamCitySetStatus
is a task to change current build status text.
{build.status.text
is substituted with an older status text. The status can have the SUCCESS
value.
NUnit Add-ins Support
NUnit Add-in is an extension that plugs in to the NUnit core and changes the way it operates. Refer to the NUnit add-ins page for more information. This section covers description of the NUnit add-ins support for NAnt, MSBuild, and NUnit Console Launcher.
Using Add-ins in NAnt Build Runner
To support NUnit add-ins in the NAnt build runner, you need to add the teamcity.dotnet.nant.nunit2.addins
property to your build script:
where <list>
is the list of paths to NUnit add-ins separated by ;
.
For example:
Using Add-ins in TeamCity NUnit Console Launcher
To support NUnit add-ins for the console launcher, you need to provide the /addins:<list of addins separated with ;>
command-line option.
For example:
Using Add-ins in MSBuild
This section is applicable to NUnit versions prior to 3.0.
To support NUnit add-ins for the MSBuild runner, specify the Addins
property for the NUnitTeamCity
task:
where <list>
is the list of add-ins separated by ;
or ,
.
For example:
TeamCity Add-in for NUnit
TeamCity NUnit Add-in supports NUnit prior to version 3.0. For later versions, refer to the this section.
If you run NUnit tests via the NUnit console and want TeamCity to track the test results without having to launch the TeamCity test runner, the best solution is to use TeamCity Add-in for NUnit. You can plug this add-in into NUnit, and the tests will be automatically reported to the TeamCity server.
Alternatively, you can opt to use the XML Report Processing build feature, or manually configure test reporting by means of service messages.
To be able to review test results in TeamCity:
In your build, set the path to the TeamCity Add-in to the system property
teamcity.dotnet.nunitaddin
(for MSBuild, it would beteamcity_dotnet_nunitaddin
), and add the version of NUnit at the end of this path. For example:For NUnit 2.4.X, use
${teamcity.dotnet.nunitaddin}-2.4.X.dll
(for MSBuild:$(teamcity_dotnet_nunitaddin)-2.4.X.dll
).Example for NUnit 2.4.7:
NAnt: ${teamcity.dotnet.nunitaddin}-2.4.7.dll
, MSBuild:$(teamcity_dotnet_nunitaddin)-2.4.7.dll
.For NUnit 2.5.0 alpha 4, use
${teamcity.dotnet.nunitaddin}-2.5.0.dll
(for MSBuild:$(teamcity_dotnet_nunitaddin)-2.5.0.dll
).
Copy the
.dll
and.pdb
TeamCity add-in files to the NUnit add-in directory.
Although you can copy these files once, it is highly recommended configuring your builds so that the TeamCity add-in files are copied to the NUnit add-in directory for each build, because these files could be updated by TeamCity.
The following example shows how to use the NUnit console runner with the TeamCity Add-in for NUnit 2.4.7 (on MSBuild):
If you need to configure environment variables for NUnit explicitly, specify an environment variable with the value reference of %system.teamcity.dotnet.nunitaddin%
. See this article for details.
NUnit 2.4.8 Issue
NUnit 2.4.8 has the following known issue: NUnit 2.4.8 runner tries to load an assembly according to the created AssemblyName
object. However, the addins
folder of NUnit 2.4.8 is not included in application probe paths. As a result, NUnit 2.4.8 fails to load any add-in in the console mode.
To solve the problem, we suggest that you use any of the following workarounds:
Copy the TeamCity add-in assembly both to the NUnit
bin
andbin/addins
directories.Patch
NUnit-Console.exe.config
to include the add-ins to application probe paths. Add the following code into theconfig/runtime
element:<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> <probing privatePath="addins"/> </assemblyBinding>