Common Problems
Most user issues are related to the following topics. Before reporting your problem, check if any of these Help pages contains the solution already:
If a build fails or otherwise misbehaves in TeamCity but you believe it should not, the first thing to do is to check whether the issue is related to TeamCity or not.
To do that, follow this procedure:
Find a way to run the task from a command prompt. Make sure it works on the TeamCity agent machine, under the same user as the TeamCity agent runs under, with the same environment the agent receives. If necessary, run the TeamCity agent under a different user or tweak its environment.
When the command runs OK, configure the same command in a TeamCity build using the Command Line runner with the custom script setting.
If that works, try another runner if that feels applicable.
Here are details on the approach:
Check that the build runs fine from the command prompt when run on the same machine as the TeamCity agent and under the same user that the agent is running, with the same environment variables and the same working directory, same architecture (32/64 bit) command line.
If the TeamCity build agent is run as a service (for example, it is installed as a Windows service), try running the TeamCity agent under a regular user with administrative permissions from the command line. See also Windows Service limitations.
If this fixes the issue, you can try to figure out why running under the service is a problem for the build. Most often this is service-specific and is not related to TeamCity directly. Also, you can set up the TeamCity agent to be run from the console all the time (for example, configure an automatic user logon and run the agent on the user logon).
Here are the detailed steps you can use to run a build from the command line.
Assuming you have a configured build in TeamCity which is failing, do the following:
run the build in TeamCity and see it misbehaving
disable the agent so that no other builds run on it. This can be done while the build is still in progress
log in to the agent machine using the same user as the one running the TeamCity agent (check the right user in the machine processes list)
stop the agent
in a command line console,
cdto the checkout directory of the build in question (the directory can be looked up in the beginning of the build log in TeamCity)run the build with a command line as you would do on a developer machine. This is runner-dependent. (For some runners you can look up the command line used by TeamCity in the build log, see also the
logs\teamcity-agent.logfile for the command line used by TeamCity)if the build fails — investigate the reason as the issue is probably not TeamCity-related and should be investigated on the machine.
if it runs OK, continue
in the same console window
cdto<TeamCity agent home>/binand start TeamCity agent from there with theagent startcommandensure the runner settings in TeamCity are appropriate and should generate the same command line as you used manually. For example, use the Command Line build step with the Custom script option and the same command which can be saved in a
.shor.batfile and run from the command promptrun the build in TeamCity selecting the agent in the Run custom build dialog
when finished, enable the agent
If the build succeeds from the console but still fails in TeamCity, use a command line runner in TeamCity to launch the same command as in the console.
If it still behaves differently in TeamCity, most probably this is an environment or a tool issue.
If the command line runner works but the dedicated runner does not while the options are all the same, create a new issue in our tracker detailing the case. Please attach all the build step settings, the build log, all agent logs covering the build, the command you used in the console to run the build and the full console output of the build.
If you experience slow builds, the first thing to do is to check the build log to see if there are some long operations or the time is just spread over the entire process. You can compare build logs of slower and faster builds to figure out what the difference is. You can also run the build from the console on the same machine as detailed above to see if there is any difference between the build run from the console and the build in TeamCity.
If the slowness is spread over all the operations, the agent machine resources (CPU, disk, memory, network) are to be analyzed during the build to see if there is a bottleneck in any of those. You can enable Performance Monitor build feature in the build configuration to see how much CPU, memory and disk I/O is used by the build and at what stage.
If build process is hanging, then "View thread dump" link on the running build results page can help with understanding why it happens.
If there is some long operation and it is a TeamCity-related one (before the start or after the end of the actual build process), the TeamCity agent and server are to be analyzed (logs and thread dumps).
If you want to turn to us with the issue, make sure to describe the visible effects, detail the process of investigation and attach the build log, full agent logs and other data collected.
First start of agent after installation or TeamCity server upgrade/plugin installation can take time as agent downloads updates form the server and autoupgrades.
Regularly, agent should become connected in 1 to 10 minutes, depending on the agent/server network connection speed.
If the agent is not connected within that time, check the name of the agent (as configured in conf\buildAgent.properties file) and check the tabs under the Agents page:
the agent is under Connected — the agent is ready to run builds
the agent is under Disconnected — the agent was connected to the server, but became disconnected. Check the "Inactivity reason" in the table. If the reason is "Agent has unregistered (will upgrade)", then wait for several more minutes
the agent is under Unauthorized — all the agents connected to the server for the first time should be authorized by a server administrator
If the agent stays in the state for more than 10 minutes and you have a fast network connection between the agent and the server, do the following:
check the related agent machine to ensure that the agent process is running and
serverURLinconf\buildAgent.propertiesis correct (and that the server is reachable by that URL from the machine);check that all the related environment requirements are met;
check agent logs (
teamcity-agent.log,launcher.log,upgrade.log) for any related messages/errors;check server logs (
teamcity-server.log) for any messages/errors mentioning agent name or IP.
If you cannot find the cause of the delayed agent upgrade in the logs, contact us and provide the full agent and server logs. Be sure to check/include the state of the agent processes (java ones) on the agent machine.
If you encounter a case when artifacts are preserved while they should have been removed by the server clean-up process, check the following:
the clean-up rules of the build configuration, artifacts section
presence of the icon "This build is used by other builds" in the build history line (prior to Pin action/icon on Build History)
build's Dependencies tab, "Delivered Artifacts" section. For every build configuration, check whether "Prevent dependency artifacts clean-up" is turned ON (this is default value). If it is, then the build's artifacts are not cleaned because of the setting.
Read more on clean-up settings.
If during the TeamCity server start-up you encounter errors like: "error in script file line: ... out of memory", "java.sql.SQLException: out of memory", perform the following:
try increasing server memory. If this does not help, most probably this means that you have encountered internal database corruption. You can try to deal with this corruption using the notes based on the HSQLDB documentation.
Here is a way to attempt a manual database restore:
stop the TeamCity server
back up the
<TeamCity Data Directory>/system/buildserver.datafileremove the
<TeamCity Data Directory>/system/buildserver.datafile and replace it with zero-size file of the same namestart the TeamCity server
However, if the database does not recover automatically, chances that it can be fixed manually are minimal.
The internal (HSQL) database is not stable enough for production use and we highly recommend using an external database for TeamCity non-evaluation usage.
If you encountered database corruption, you can restore the last good backup or drop builds history and users, but preserve the settings, see Migrating to an External Database.
This error can occur with an MS SQL or Sybase database. In this case we recommend increasing the transaction log for the TeamCity database. The log size can be 1 - 16 GB depending on the number of build agents in the system and the number of tests all agents report daily.
This error can occur with a MySQL database. The error indicates that the database has run out of free space either on the disk where the database files are located or in the temp directory.
This error can occur with an Oracle database. The error indicates that Oracle could not obtain more space for a table or an index as the database has run out of space on the disk or the specified quotas are insufficient.
teamcity-server.log reports that the unsupported NOCOUNT option is enabled on the MS SQL database server. Contact your DBA to disable the setting.
The ER_NET_PACKET_TOO_LARGE error (PacketTooBigException / Packet for query is too large) is caused by the server-side max_allowed_packet configuration variable set to a low value or left at the default one.
The variable controls the maximum size of MySQL communication buffer with 4MB being the default for Windows builds of MySQL 5.6, whereas in popular Linux distributions (e. g. Debian and Fedora Core), this variable defaults to 16MB, for both i686 and amd64 architectures.
Check the value of the
max_allowed_packetconfiguration variable by either examiningmy.cnf / my.ini, or viamysql> show variables like 'max_allowed_packet';Increase (or explicitly set) the value of the
max_allowed_packetconfiguration variable inmy.cnformy.ini:[mysqld] max_allowed_packet=16M
note
Setting a client-side value (via
maxAllowedPacketconnection property) in the JDBC URL will have no effect, as this value cannot exceed the server-side limit.
TeamCity reports character set/collation mismatch error: database tables/columns have a character set or collation that is not the same as the default character set or collation in your database schema. You may see this message if you are using a non-unicode character set as default for your database as TeamCity enforces unicode charset for some of the varchar fields. Make sure you configured the database according to our recommendations.
If you want to allow your local characters in texts in TeamCity (for example, VCS messages, test names, usernames), you need to migrate to a database with the appropriate character set.
TeamCity server restoration from a backup may fail with errors like "Unique key violation" or "Duplicates found" if the character sets (or collations) of the source and destination databases are not the same, and the cardinality of the destination character set is less than that of the source database.
To resolve the problem, select and set up the proper character set (and collation) for the destination database.
As for case sensitivity, the possible transitions are:
CS > CS
CI > CS
CI > CI
However, it is recommended to always use CS.
If the source character set is Unicode or UTF, the destination one must also be Unicode or UTF. If the source character set is 8-bit non-UTF, the destination one can be the same or Unicode/UTF.
To fix a problem, perform the following steps:
Create a new database with the appropriate character set and collation. We recommend using a unicode case-sensitive collation: see instructions for PostgreSQL and MySQL. For MySQL,
utf8_binorutf8mb4_binis preferred.
See also PostgreSQL, MySQL, MS SQL documentation for details on character set.Copy the current
<TeamCity Data Directory>/config/database.propertiesfile, and change the database references in the copy to the newly created database.Stop the TeamCity server.
Use the
maintainDBtool to migrate to the new database:maintainDB migrate [-A <path-to-data-dir>] -T <new-database-properties-file>Depending on the size of your database, the migration may take from several minutes to several hours. For more information on the
maintainDB tool, see this section.Upon the successful completion of the database migration, the
maintainDBtool should update the<TeamCity Data Directory>/config/database.propertiesfile with references to the new database. Ensure that the file has been updated. Edit the file manually if the tool fails to do it automatically.Start the TeamCity server.
Check if the character set of your MySQL database. It is recommended to use the utf8 character set.
If your database uses the utf8mb4 character set (available since MySQL 5.5), set the following InnoDB configuration options (under [mysqld] section in my.cnf or my.ini) to run:
innodb_large_prefix=1for index key prefixes longer than 767 bytes (up to 3072 bytes) to be allowed for InnoDB tables that use DYNAMIC row format (deprecated in MySQL 5.7.7).innodb_file_format=Barracudato enable the DYNAMIC row format.innodb_file_per_table=1to enable the Barracuda file format
The parameters above have the following default values:
MySQL 5.5 | MySQL 5.6 | MySQL 5.7 | MariaDB 5.5 | MariaDB 10.0 | MariaDB 10.1 | MariaDB 10.2 | |
|---|---|---|---|---|---|---|---|
innodb_large_prefix | 0 | 0 | 1 | 0 | 0 | 0 | 1 |
innodb_file_format | Antelope | Antelope | Barracuda | Antelope | Antelope | Antelope | Barracuda |
innodb_file_per_table | 0 | 1 | 1 | 1 | 1 | 1 | 1 |
Depending on the database server version, the following configuration changes need to be made:
MySQL 5.5:
innodb_large_prefix=1innodb_file_format=Barracudainnodb_file_per_table=1
MySQL 5.6 and MariaDB from 5.5 to 10.1:
innodb_file_format=Barracudainnodb_file_per_table=1
For MySQL 5.7+ and MariaDB 10.2+ use the defaults, no changes are required.
Make sure the database uses the
utf8mb4character set andutf8mb4_bincollation. If that's not the case, change the DB character set and collation.If the character set and collation are set properly, but the new errors still occur, the problem is that the default charset in MySQL is not utf8mb4. Set up the MySQL server to use
utf8mb4by default by specifying the following options in the[mysqld]section ofmy.cnf:character_set_server = utf8mb4collation_server = utf8mb4_binRestart MySQL for the changes in
my.cnfto take effect.
During TeamCity installation, the following error might occur when connecting and creating the MS SQL database with Windows integrated security: "SQL error when doing: Taking a connection from the data source: This driver is not configured for integrated authentication."
The most common reason for the problem is the different bitness of the sqljdbc_auth.dll MS SQL shared library and the JRE used by TeamCity.
To solve the problem, do the following:
Make sure you use the MS SQL native driver (downloadable from the Microsoft Download Center).
Use the right JRE bitness: ensure that you are running TeamCity using Java with the same bitness as your
sqljdbc_auth.dllMS SQL shared library. By default, TeamCity uses the 32-bit Java. However, both 32-bit and 64-bit Java versions can be used.
To run TeamCity with the required JRE, do one of the following: * either set the TEAMCITY_JRE environment variable * or remove the JRE bundled with TeamCity from <TeamCity home>\jre and set JAVA_HOME.
note
Note that on upgrade, TeamCity will overwrite the existing JRE with the default 32-bit version, so you'll have to update to the 64-bit JRE again after upgrade.
See also this related external posting.
This error can occur when the Oracle JDBC driver is not compatible with the Oracle server. For example, Oracle JDBC driver version 11.1 is not compatible with Oracle server version 10.2.
In order to resolve the problem, use the Oracle JDBC driver from your Oracle server installation, or download the driver of the same version as the Oracle server.
There are two kinds of Maven-related issues commonly seen in the TeamCity build configurations:
Error message on "Maven" tab of build configuration: "An error occurred during collecting Maven project information ... "
Error message in build configuration with Maven dependencies trigger activated: "Unable to check for Maven dependency Update ..." If the build configuration produces successful builds despite displaying such error messages, these errors are likely to be caused by the server-side Maven misconfiguration.
To collect information for the Maven tab, or to perform Maven dependencies check (for the trigger), TeamCity runs the embedded Maven. The execution is performed on the server machine, and any agent-side maven settings are not accessible. TeamCity resolves the settings.xml files on the server-side separately, as described in Maven Server-Side Settings.
It makes sense to check if the server-side settings.xml files contain correct information about remote repositories, proxies, mirrors, profiles, credentials, and so on.
If you encounter the error, it means the settings stored in the TeamCity Data Directory are in an inconsistent state. This can occur after manual change of the files or if newer version of TeamCity starts to report the inconsistencies.
To resolve the issue, you can edit the file noted in the message on the server file system. (Make sure to create backup copy of the file before any manual edits). Usually server restart is not necessary for the changes to take effect.
VCS root with id "XXX" does not exist
The build configuration or template reference a VCS root which is not defined in the system. Remedy actions: Restore the VCS root or create a new VCS root with the id noted or edit the file noted in the message to remove the reference to the VCS root.
If the TeamCity web UI cannot be accessed after installation, you might be running TeamCity on a port that is already in use by another program. Check and configure your TeamCity installation.
If you are experiencing issues with partial TeamCity NuGet Feed (for example, missing NuGet packages), you might have to reindex the TeamCity NuGet Feed.
To force TeamCity to reindex all available packages and reset the NuGet package list, navigate to the server Administration | Diagnostics | Caches and use the buildsMetadata Reset link.
For earlier versions, refer to this section.
.NET Framework below version 4.0 installed on TeamCity agents may cause performance issues of .NET-related TeamCity tools due to Code access security (CAS) policy imposed by Microsoft.
To solve the issue, use one of the options:
Add the following setting described in the Microsoft documentation to the
machine.configfile on all agents:<configuration> <runtime> <generatePublisherEvidence enabled="false"/> </runtime> </configuration>You can modify the machine.config file as described in this external blog post and pass this config file to all agents, e.g. using a custom script.
Alternatively, upgrade .NET Framework on the TeamCity agents to version 4.0 and above. Details are available in the Microsoft documentation.
You might want to use tools which require some manual interaction during the build procedure executed on the TeamCity agent. This is not a TeamCity-specific problem, so it should be approached using generic means.
Under Windows, you might want to configure TeamCity agent to run not as a Service, but with access to the desktop by configuring automatic user logon, related details.
There is no simple solution for Extended Validation (EV) code signing as the feature is built in for a reason. There is some discussion on the issue on stack overflow. The appropriate solution seems to implement a dedicated service with own authorization approach and sign the binaries through it.
By default, Tomcat sets a 4096 bytes (4KB) limit to the request and response HTTP headers. If too many cookies get included into a header (for example, by the proxy servers between the end user and TeamCity), users might get the 400: Request Header too large error when browsing TeamCity.
To identify the cause of this problem, we recommend that you inspect the affected header with the browser developer console. As a temporary workaround, you can also increase the Tomcat header limit. To do this, change the $TOMCAT_HOME/conf/server.xml file and increase the value of the maxHttpHeaderSize parameter for all relevant connectors (for example, set 65536 to increase the limit to 64KB).