Reporting Issues

If you experience problems running TeamCity and believe it's related to the software, please contact us with detailed description of the issue.

To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information.

Logging Events

TeamCity (both server and build agents) logs events and warnings using log4j.

The logging rules and log4j configuration files for TeamCity are described in Logs of Internal Events.

Before reproducing the problem it makes sense to enable 'DEBUG' log level for TeamCity classes. To do it, edit the logs/teamcity-(server|agent)-log4j.xml file by removing all the lines containing <!--DELETE THIS LINE FOR ENABLING DEBUG LOGGING--> comment. After that, DEBUG messages will go to teamcity-*.log files.

Make sure the logs are rotated by default. When debug is enabled it makes sense to increase maxBackupIndex value in the relevant appender tag to 10 or even 20 files (ensure there is sufficient free disk space available).

You can change the log4J configuration files while the server/agent is running. If it is possible (some log4j restrictions apply), the loggers will be reconfigured without process restart.

Hangs and Thread Dumps

If you experience problems with the TeamCity server or agent (e.g. no responding or working too slow) we would appreciate a thread dump of the process.

First, please indicate what process you need a thread dump for. If there is an issue with the server or build sources checkout issue, you will need a thread dump of the web server process (e.g. Tomcat), if the issue is related only to a single build, you will need an agent thread dump.

If you need agent thread dump, please note that TeamCity agent consists of two java processes: launcher and agent itself. Agent is run by the launcher process. You will usually be interested in the agent (more nested one) and not the launcher process.

To take a thread dump: Under Windows You have several options:

• if you need server thread dump and the server is run from console, press Ctrl+Break in the console window (this will not work for agent, since it's console belongs to launcher process).

• if JDK 1.6 is used, use jstack <pid_of_java_process> command (jstack is located in the "bin" directory of JDK installation).

• you can also use TeamCity-bundled thread dump tool (can be found in agent's plugins). Run the command:

  <TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>

  On Windows 2003 Server and Vista you might need to run the tool under System account to take a thread dump of ta process run as service. You can use Microsoft PsExec tool for that.

Under Linux

• run jstack <pid_of_java_process> or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity server home>/logs/catalina.out or <TeamCity agent home>/logs/output.log.

You can also use third-party GUI tool: AdaptJ StackTrace Utility. It supports Windows, Linux or Mac OS X. Choose "Launch" on the page (if you have Java installed). When the application is started, select the process id via "Process > Select", then dump its thread dump with "Process > Thread Dump".

See also Server Performance section below.

OutOfMemory Problems

If you experience problems with TeamCity "eating" too much memory (OutOfMemory errors), please do the following:

• Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent)

• Try to increase the memory for the process via '-Xmx' JVM option, like -Xmx512m. If the error message is "java.lang.OutOfMemoryError: PermGen space", the following option should be used instead: -XX:MaxPermSize=150m. The option needs to be passed to the process with problems:

  • if it is the building process itself, use "JVM Command Line Parameters" settings in the build runner. e.g. Inspections builds may specifically need to increase the parameter;

  • if it is TeamCity server, or agent, refer to TeamCity Startup Properties on how to pass additional options to the processes;

• If increasing memory size does not help, please get the memory dump and send it to us for further analysis:

  • to get a memory dump (hprof file) when an OoutOfMemory error occurs, add the following JVM option (works for JDK 1.5.0_07+): -XX:+HeapDumpOnOutOfMemoryError

  • when OOM error occurs next time, java_xxx.hprof file will be created in the process startup directory (<TeamCity home>/bin or <TeamCity Agent home>/bin);

  • archive the file and send it to us.

Version Control Debug Logging

In general, to debug VCS-related problems we need information for jetbrains.buildServer.VCS Log4j category. So please enable it in the <TeamCity home>\conf\teamcity-server-log4j.xml (for server-side checkout) and <BuildAgent home>\conf\teamcity-agent-log4j.xml (for agent-side checkout) files:

There is a set of debug options that turn on debug logging for specific version controls. If the version control you use is mentined below, run the server with the required option and send us teamcity-vcs.log file for analysis. Some version controls also have specific logs that appear after uncommenting corresponding sections form the log4j configuration file.

Subversion Integration Debug Logging

First, please enable generic VCS debug logging, as described above.

Adjust the logging configuration and restart the server or agent, correspondingly. The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log

Server

To turn on debug logging of the SVNkit library on the server side, uncomment all the lines marked with <!- UNCOMMENT FOR DEBUG -> in the <TeamCity home>\conf\teamcity-server-log4j.xml file.

Agent

To turn on debug logging of the SVNkit library on the agent (only relevant for agent-side checkout mode) uncomment the following lines in the <agent home>\conf\teamcity-agent-log4j.xml file:

CVS

For versions prior to TeamCity 4.0: Add -Dcvs.log.commands=true JVM parameter to the server and also to the agent (for agent-side checkout only). The logging will go into logs/teamcity-vcs.log.

For latter versions it is enough to enable debug logging.

ClearCase

Uncomment Clearcase-related lines in the <TeamCity home>\conf\teamcity-server-log4j.xml file. The log will be saved to logs\teamcity-clearcase.log directory.

Patch Application Problems

In case server-side checkout is used, the "patch" that is passed from server to the agent can be retrieved by:

• add property agent.save.patch=true to the build configuration properties: Administration->Edit configuration->Properties and requirement variables->Add new property...

• trigger the build.

Agent log will contain the line "Patch is saved to file ${file.name}" Get the file and provide it with the problem description.

Remote Run Problems

The changes that are sent form the IDE to the server on a remote run can be retrieved from server's .BuildServer\system\changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the from the URL of the change form the web UI). The file contains the patch in the binary form. Please provide it with the problem description.

Server Performance

If you experience degraded server performance and TeamCity server process is producing large CPU load, please take the CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what is your system setup. You can take the CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions provided on the plugin page.

Logging in TeamCity Visual Studio plugin

To capture logs from TeamCity Visual Studio plugin please do the following:

1. Close all instances of Microsoft Visual Studio.

2. Open <username-profile-folder>\Local Settings\Temp\JetLogs folder.

3. Delete all of the files in this folder.

4. Restart Microsoft Visual Studio.

5. Open a solution.

6. Try to log in to TeamCity.

7. Close Microsoft Visual Studio.

8. Navigate back to the <username-profile-folder>\Local Settings\Temp\JetLogs folder. All of the files that were created are logs.

Logging in TeamCity Eclipse plugin

Available only since TeamCity 4.5

To enable tracing for the plugin, run Eclipse IDE with the -debug <filename> program argument. The <filename> portion of the program argument is a properties file containing key-value pairs. Name of each property corresponds to the plugin module and value is either 'true' (to enable debug) or 'false'. Here is an example of enabling most common tracing options:

Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.

Sending Information to the Developers

Files under 5Mb in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit viewing the attachment to "teamcity-developers" user group only).

If the file is over 5 Mb, you can upload the archived files via https://uploads.jetbrains.com.

You can also send small files via email: teamcity-feedback@jetbrains.com Please do not forget to mention your TeamCity version and environment