TeamCity upgrade fails with MySQL error

Answered
Created May 17, 2021 21:14
I'm attempting to upgrade TeamCity from 2019.2.4 to 2020.1.5. TeamCity is running as a container on Azure Kubernetes Service.
I'm using MySQL 5.7 as the database hosted by Azure Database for MySQL. I'm using the MySQL Connector/J JDBC driver 8.0.16, but have also attempted with the latest of 8.0.25.
When performing the upgrade, immediately after the database backup step, I receive the below error. I've tried upgrading the JDBC driver, and restarting the MySQL server, but am experiencing the same issue.
Can you please advise of the best next action?
jetbrains.buildServer.maintenance.exceptions.UnknownUpgradeError: Upgrading error: Could not connect to MySQL server: SQL error when doing 'Connection.setAutoCommit(false)': com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem. Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at jetbrains.buildServer.maintenance.StartupProcessor.doUpgrade(StartupProcessor.java:825)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$1600(StartupProcessor.java:466)
	at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:7)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:872)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:965)
	at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1169)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1346)
	at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
	at java.base/java.lang.Thread.run(Thread.java:834)
caused by: jetbrains.buildServer.serverSide.db.DatabaseIsNotReachableException: Could not connect to MySQL server: SQL error when doing 'Connection.setAutoCommit(false)': com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem. Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2074)
	at jetbrains.buildServer.serverSide.db.DBFunctions.setAutoCommit(DBFunctions.java:1099)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverter(ConvertPerformer.java:108)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverters(ConvertPerformer.java:87)
	at jetbrains.buildServer.serverSide.versioning.VersionManager.upgradeIt(VersionManager.java:383)
	at jetbrains.buildServer.maintenance.StartupProcessor$UpgradePerformer.doUpgrade(StartupProcessor.java:1)
	at jetbrains.buildServer.maintenance.StartupProcessor.lambda$doUpgrade$10(StartupProcessor.java:482)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:940)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:915)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.maintenance.StartupProcessor.doUpgrade(StartupProcessor.java:433)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$1600(StartupProcessor.java:466)
	at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:7)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:872)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:965)
	at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1169)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1346)
	at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
	at java.base/java.lang.Thread.run(Thread.java:834)
caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2074)
	at jetbrains.buildServer.serverSide.db.DBFunctions.setAutoCommit(DBFunctions.java:1099)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverter(ConvertPerformer.java:108)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverters(ConvertPerformer.java:87)
	at jetbrains.buildServer.serverSide.versioning.VersionManager.upgradeIt(VersionManager.java:383)
	at jetbrains.buildServer.maintenance.StartupProcessor$UpgradePerformer.doUpgrade(StartupProcessor.java:1)
	at jetbrains.buildServer.maintenance.StartupProcessor.lambda$doUpgrade$10(StartupProcessor.java:482)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:940)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:915)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.maintenance.StartupProcessor.doUpgrade(StartupProcessor.java:433)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$1600(StartupProcessor.java:466)
	at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:7)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:872)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:965)
	at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1169)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1346)
	at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
	at java.base/java.lang.Thread.run(Thread.java:834)
caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 250,005 milliseconds ago.  The last packet sent successfully to the server was 250,007 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
	at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
	at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:593)
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:921)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1165)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2064)
	at jetbrains.buildServer.serverSide.db.DBFunctions.setAutoCommit(DBFunctions.java:1099)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverter(ConvertPerformer.java:108)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverters(ConvertPerformer.java:87)
	at jetbrains.buildServer.serverSide.versioning.VersionManager.upgradeIt(VersionManager.java:383)
	at jetbrains.buildServer.maintenance.StartupProcessor$UpgradePerformer.doUpgrade(StartupProcessor.java:1)
	at jetbrains.buildServer.maintenance.StartupProcessor.lambda$doUpgrade$10(StartupProcessor.java:482)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:940)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:915)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.maintenance.StartupProcessor.doUpgrade(StartupProcessor.java:433)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$1600(StartupProcessor.java:466)
	at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:7)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:872)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:965)
	at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1169)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1346)
	at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
	at java.base/java.lang.Thread.run(Thread.java:834)
caused by: java.net.SocketException: Connection reset by peer (Write failed)
	at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
	at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
	at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:319)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1197)
	at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
	at com.mysql.cj.protocol.a.SimplePacketSender.send(SimplePacketSender.java:55)
	at com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:50)
	at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:584)
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:921)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1165)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2064)
	at jetbrains.buildServer.serverSide.db.DBFunctions.setAutoCommit(DBFunctions.java:1099)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverter(ConvertPerformer.java:108)
	at jetbrains.buildServer.serverSide.versioning.ConvertPerformer.runConverters(ConvertPerformer.java:87)
	at jetbrains.buildServer.serverSide.versioning.VersionManager.upgradeIt(VersionManager.java:383)
	at jetbrains.buildServer.maintenance.StartupProcessor$UpgradePerformer.doUpgrade(StartupProcessor.java:1)
	at jetbrains.buildServer.maintenance.StartupProcessor.lambda$doUpgrade$10(StartupProcessor.java:482)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:940)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager$3.run(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:915)
	at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.withMaintenanceDBF(TeamCityDatabaseManager.java:935)
	at jetbrains.buildServer.maintenance.StartupProcessor.doUpgrade(StartupProcessor.java:433)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$1600(StartupProcessor.java:466)
	at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:7)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:872)
	at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:965)
	at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:1169)
	at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1346)
	at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
	at java.base/java.lang.Thread.run(Thread.java:834)
2 comments
Eric Borchardt
Created May 20, 2021 21:19
It seems the connection to the database is interrupted for some reason and is not automatically reconnecting because autoReconnect=false. You may be able to resolve it by opening your <teamcity data directory>/config/database.properties file and appending
'?autoReconnect=true&tcpKeepAlive=true' to the end of your connection string.
Bokelo
Created April 22, 2024 11:09
Eric Borchardt thanks a bunch. Saved a whole lot of time.
Please sign in to leave a comment.