Investigating and Muting Build Failures

Required Permissions

To be able to manage and view investigations/mutes in a certain project, the user must have the following permissions granted in this project:

• "Mute/unmute problems in project" for managing mutes

• "Assign/unassign investigation" for managing investigations

• "View project and all parent projects" for viewing investigations/mutes

These permissions are granted to Project Administrators and System Administrators by default.

Assigning Investigations of Failed Build Configurations

To assign an investigation of a failed build configuration:

1. Navigate to the Overview tab of the Build Configuration Home page (or the Overview tab of the Build Results page) and click Assign investigation.

   

2. Select a TeamCity user from the Investigated by drop-down menu.

   

3. Select the condition to resolve the investigation:

   • When build configuration is successful

   • or, Manually; this mode is recommended for build configurations with flaky tests, when the "Success" build status does not directly indicate that the problem has been resolved.
     The user to whom the investigation is assigned can later mark the problem as fixed in this window.

4. Save the investigation.

To assign similar investigations for other failed build configurations in the same project, click more and select the configurations in the list.

Assigning Investigations of Build Problems and Failed Tests

To assign an investigation of a particular build problem:

1. Navigate to the Overview tab of the Build Results page (or the Current Problems tab of the Project Home), click the arrow next to the build problem, and then click Investigate/Mute.

   

   
   The Investigate/Mute dialog opens.

2. Select a TeamCity user from the Investigated by drop-down menu.

   

3. Select the condition to resolve the investigation:

   • Automatically when fixed (read more)

   • or, Manually

   The user to whom the investigation is assigned can later mark the problem as fixed in this window.

4. Save the investigation.

You can similarly assign an investigation of a failed test on the Overview or Tests tab of the Build Results.

Muting Build Problems

To mute a particular build problem:

1. Navigate to the Overview tab of the Build Results page (or the Current Problems tab of the Project Home), click the arrow next to the build problem, and then click Investigate/Mute.
   The Investigate/Mute dialog opens.

2. Choose where to mute the problem:

   • Project-wide, that is muting the problem in all configurations in the current project and its subprojects.

   • Selected build configurations, which allows you to select particular build configurations from the same project.

3. Select the condition to unmute the problem:

   • Automatically when fixed (read more)

   • On a specific date

   • or, Manually

4. Click Save.

Muting Tests

You can mute a failed test or several tests at once on the Overview or Tests tab of the Build Results, similarly to muting build problems.

Muting will only affect builds which fail on the "At least one test failed" build failure condition but not on the other conditions: if any other failure condition applies to your build (for example, non-zero exit code), the build will still fail even if the failing tests are muted.

This feature is useful when some tests fail for a known reason, but it is not currently possible to fix them. For example, the responsible developer is absent, or you are waiting for the system administrator to fix the environment. The feature could also be used if the test is failing intentionally: for example, if the required functionality is not yet ready.
In all these cases you can mute such failures and avoid unnecessary disturbance of other developers.

When a test is muted, it is still run in the future builds, but its failure does not affect the build status.

Your build script might need adjustment to make the build successful when there are failing but muted tests. Make sure the build does not fail because of other build failure conditions (for example, "Fail if build process exit code is not zero") in case the only errors encountered were tests failures. See the related issue TW-16784 for more details.

Changing Project Scope of Investigation or Mute

When a user assigns an investigation or mutes a problem, TeamCity suggests the scope where it has to be investigated or muted. By default, it proposes to apply the action within the current project, and users usually confirm this default scope. However, the same problem might often occur not in a single project but in multiple adjacent projects which are children of the same parent. In this case, it makes sense to investigate or mute this problem in the scope of the whole parent project.
Since version 2021.12, Project Administrators can change the value of the preferred investigation/mute scope per project. Setting the teamcity.internal.preferredInvestigationProject configuration parameter in the parent project's settings will make this project the default scope in the Investigate/Mute dialog for all its children. The value should contain the parent project ID. This property changes only the option proposed by default — users will still be able to adjust the scope in the dialog.

Marking Problems as Fixed

A user, to whom the investigation is assigned, can mark the investigated problem as fixed in any Investigate UI dialog corresponding to this problem.
The problem will be marked with on all TeamCity pages it appears. This allows identifying a potentially fixed problem in the UI, but leaves a possibility to continue the investigation if the problem persists in the following build run. If the author of the investigation has the respective notifications turned on, they will be notified about the fix.
TeamCity will remove the investigation in the following build run after the investigation is marked as fixed, if the problem is not reproduced and if this investigation has the "Automatically when fixed" resolution condition.

Viewing Investigations and Mutes

You can find all the investigations and mutes per project on the Project Home page:

• Investigations tab: see all assigned investigations in the project

• Muted Problems tab: see all muted problems and tests in the project

Here you can unmute problems and tests, and reassign the investigations to another user or mark them as fixed.

Viewing My Investigations

If one or more investigations are assigned to you, the investigation counter will appear next to your name on the TeamCity top panel:

Click it to open the My Investigations page where you can see all the investigations assigned to you in different projects. You can manage the problems directly on this page: mark them as fixed, reassign, mute, or give up the investigations.

For each failed test, you can instantly see:

• all build configurations where the test is currently failing

• current stack trace and the information about the build where the test is currently failing

• information about the first failure of the test, with the stack trace and the build

The investigations assigned to you are also highlighted in the UI if you enable the "Highlight my changes and investigations" option in your profile settings.

Viewing Investigation History

Since TeamCity 2020.1, you can view the history of actions applied to each investigation. This is most helpful for big teams and projects when it is not as easy to determine who and when changed or resolved the investigation.

To see the history of an investigation, open any UI page where this investigation is available and click Show Investigation History in its context menu.

Notes on Automatic Fix of Investigations and Muted Problems

If the "Automatically when fixed" option is select in Investigation or Mute options, TeamCity will analyze the state of each failure in different branches to resolve the investigation or mute smarter:

• If the failure occurs in the default branch and in another active branch, and an investigation (or mute) is assigned to this failure in any of these branches, it will be resolved as soon as the failure is fixed in the default branch.

• If the failure occurs in any active branch other than the default one, the assigned investigation (or mute) will be resolved only when it is fixed in all active branches in which it occurs.