Build Fails on the agent with SQL Deadlock

Hi Nitin, thanks for the report. I haven't been able to find this issue before, could you send us the full log where it appears?

Thank you for the response Denis.

I do not know if I should be copying the logs here, but I could not find any other way. Below are the agent logs and the server logs. On the web page, the error that is displayed is the one in my original message.

Thanks a lot for having some time to look into this.

Agent log:

[2017-06-21 10:21:24,012] INFO - dDirectoryBasedCleanupRegistry - Removing files from C:\BuildAgent\work\.old
[2017-06-21 10:21:24,052] INFO - l.directories.DirectoryMapImpl - Cleaning up old checkout directories. Default lifetime = 192 hour(s)
[2017-06-21 10:26:23,954] INFO - r.artifacts.impl.HttpDiskCache - Cleaning up items with life time in cache greater than 172800 seconds
[2017-06-21 10:26:23,954] INFO - r.artifacts.impl.HttpDiskCache - Finished cleaning up expired items, 0 items removed
[2017-06-21 10:26:23,954] INFO - dDirectoryBasedCleanupRegistry - Removing files from C:\BuildAgent\temp\.old
[2017-06-21 10:26:23,954] INFO - dDirectoryBasedCleanupRegistry - Removing files from C:\BuildAgent\work\.old
[2017-06-21 10:26:24,084] INFO - l.directories.DirectoryMapImpl - Cleaning up old checkout directories. Default lifetime = 192 hour(s)
[2017-06-21 10:29:19,031] INFO - ldServer.AGENT.PollingProtocol - New command is received from server "runBuild" {id = 44}
[2017-06-21 10:29:19,051] INFO - jetbrains.buildServer.AGENT -

[2017-06-21 10:29:19,051] INFO - jetbrains.buildServer.AGENT - ===================================================
[2017-06-21 10:29:19,051] INFO - jetbrains.buildServer.AGENT - Starting Build {id=402058,

Teamcity server logs:

[2017-06-21 10:29:03,432] INFO - jetbrains.buildServer.SERVER - Build removed from queue (started): 402069
[2017-06-21 10:29:03,432] INFO - jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2017-06-21 10:29:09,576] ERROR - jetbrains.buildServer.SERVER - Error while creating an AgentBuild
jetbrains.buildServer.serverSide.db.DatabaseDeadlockException: Could not resolve deadlock with 7 attempts.
SQL error when doing 'Fetching query results'
SQL query: select vcs_root_id, current_version from agent_sources_version where checkout_dir = ? and agent_id = ? and vcs_settings_hash = ?
SQL exception: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
SQL exception: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:4853)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1781)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1034)
at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:208)
at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:208)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$2.process(DBVersionProvider.java:1)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$2.process(DBVersionProvider.java:6)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.processResultSet(GenericQuery.java:380)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:367)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.access$500(GenericQuery.java:27)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery$7.action(GenericQuery.java:339)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.underPreparedStatement(GenericQuery.java:407)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:336)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.execute(GenericQuery.java:78)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$13.run(DBVersionProvider.java:3)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$13.run(DBVersionProvider.java:2)
at jetbrains.buildServer.serverSide.db.InternalDBUtil.runAndRetry(InternalDBUtil.java:25)
at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.runAndRetry(TeamCityDatabaseManager.java:931)
at jetbrains.buildServer.serverSide.db.DBFacade.runAndRetry(DBFacade.java:284)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getRevisionForCheckoutDirectory(DBVersionProvider.java:66)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getVcsRootRevisionOnAgent(DBVersionProvider.java:143)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getVcsRootRevisionOnAgent(DBVersionProvider.java:61)
at jetbrains.buildServer.serverSide.impl.RunningBuildImpl.createAgentBuild(RunningBuildImpl.java:535)
at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:1)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2$1.run(BuildStarter.java:3)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2.run(BuildStarter.java:1)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
... 33 more
[2017-06-21 10:29:09,577] WARN - jetbrains.buildServer.SERVER - Error while starting build {id=402043}: jetbrains.buildServer.serverSide.db.DatabaseDeadlockException: Could not resolve deadlock with 7 attempts.
SQL error when doing 'Fetching query results'
SQL query: select vcs_root_id, current_version from agent_sources_version where checkout_dir = ? and agent_id = ? and vcs_settings_hash = ?
SQL exception: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
SQL exception: Transaction (Process ID 103) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. (enable debug to see stacktrace)
[2017-06-21 10:29:10,659] INFO - jetbrains.buildServer.SERVER - Build finished received for build: #919 {build id=402043, buildTypeId=Lte_Builds_Ls28x8_Tma}
[2017-06-21 10:29:10,944] INFO - ndexer.EventBasedBuildsIndexer - Indexing finishing build: #919 {build id=402043, buildTypeId=Lte_Builds_Ls28x8_Tma}
[2017-06-21 10:29:11,584] INFO - de.impl.history.DBBuildHistory - Created build history entry; build id: 402043, finish date: 2017-06-21 10:29:10.396
[2017-06-21 10:29:11,938] ERROR - jetbrains.buildServer.SERVER - Error while creating an AgentBuild
jetbrains.buildServer.serverSide.db.DatabaseDeadlockException: Could not resolve deadlock with 7 attempts.
SQL error when doing 'Fetching query results'
SQL query: select vcs_root_id, current_version from agent_sources_version where checkout_dir = ? and agent_id = ? and vcs_settings_hash = ?
SQL exception: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
SQL exception: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:4853)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1781)
at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1034)
at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:208)
at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:208)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$2.process(DBVersionProvider.java:1)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$2.process(DBVersionProvider.java:6)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.processResultSet(GenericQuery.java:380)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:367)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.access$500(GenericQuery.java:27)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery$7.action(GenericQuery.java:339)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.underPreparedStatement(GenericQuery.java:407)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:336)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.execute(GenericQuery.java:78)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$13.run(DBVersionProvider.java:3)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider$13.run(DBVersionProvider.java:2)
at jetbrains.buildServer.serverSide.db.InternalDBUtil.runAndRetry(InternalDBUtil.java:25)
at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.runAndRetry(TeamCityDatabaseManager.java:931)
at jetbrains.buildServer.serverSide.db.DBFacade.runAndRetry(DBFacade.java:284)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getRevisionForCheckoutDirectory(DBVersionProvider.java:66)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getVcsRootRevisionOnAgent(DBVersionProvider.java:143)
at jetbrains.buildServer.serverSide.impl.DBVersionProvider.getVcsRootRevisionOnAgent(DBVersionProvider.java:61)
at jetbrains.buildServer.serverSide.impl.RunningBuildImpl.createAgentBuild(RunningBuildImpl.java:535)
at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:1)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2$1.run(BuildStarter.java:3)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2.run(BuildStarter.java:1)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
... 33 more
[2017-06-21 10:29:11,939] WARN - jetbrains.buildServer.SERVER - Error while starting build {id=402046}: jetbrains.buildServer.serverSide.db.DatabaseDeadlockException: Could not resolve deadlock with 7 attempts.
SQL error when doing 'Fetching query results'
SQL query: select vcs_root_id, current_version from agent_sources_version where checkout_dir = ? and agent_id = ? and vcs_settings_hash = ?
SQL exception: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
SQL exception: Transaction (Process ID 140) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. (enable debug to see stacktrace)

Hi Nitin,

sorry, I usually add the comment but forgot to add it into my answer. You can use the Submit a Request button on top of the page to send us files, in case you would prefer to remove this from here and keep the logs private, other than that, it just works as well. I'm going to review it in a bit more detail and try to find the reason, as soon as we get any more info, I'll let you know.

Hi Nitin,

Ok, I've been reviewing the logs and asking around a bit. This was a known issue in 10.0.4 that got fixed, details here: https://youtrack.jetbrains.com/issue/TW-48811

If you upgrade to the newest release, it should be fixed.

Thank you Denis for getting this investigated. I saw the issue you mentioned in your comments, but thought it was for a different error (where the Admin GUI was getting stuck), and that I was not getting the exact error as in TW-48811.

I'll try and get the TeamCity server upgraded to the latest version to see if I get similar errors then.

Thanks again for your support. Much appreciated!