Heartbeat pulse error - java.sql.SQLException: HOUR_OF_DAY: 2 -> 3

Completed

Dears, 

Maybe one of you saw this error already and knows how to solve, we tried now about two days to resolve the issue. 

We updated last week our test TC to latest 2019.2.3 since this we can't run the server anymore, we stuck at this: (thousands of this messages)

[2020-05-04 16:43:44,277] WARN [:39.868), node id: MAIN_SERVER] - etbrains.buildServer.HEARTBEAT - Heartbeat pulse error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Fail to peek column 7 with type java.sql.Timestamp from result set.: java.sql.SQLException: HOUR_OF_DAY: 2 -> 3. Caused by: com.mysql.cj.exceptions.WrongArgumentException: HOUR_OF_DAY: 2 -> 3 (enable debug to see stacktrace)

We are sure the timezones in docker container the server runs and mysql native are set correctly to Europe/Vienna. Mysql server is also set correctly (collation etc.) as we used the DB now for more than 8 months and also performed TC updates. Any Idea what could be wrong there? 

This is the full start output: 

Java executable is found: '/opt/java/openjdk/jre/bin/java'
Using CATALINA_BASE: /opt/teamcity
Using CATALINA_HOME: /opt/teamcity
Using CATALINA_TMPDIR: /opt/teamcity/temp
Using JRE_HOME: /opt/java/openjdk/jre
Using CLASSPATH: /opt/teamcity/bin/bootstrap.jar:/opt/teamcity/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/teamcity/bin/../logs/teamcity.pid
$CATALINA_PID was set but the specified file does not exist. Is Tomcat running? Stop aborted.
/run-services.sh
/run-server.sh
Java executable is found: '/opt/java/openjdk/jre/bin/java'
2020-05-04 16:58:02 CEST: Starting TeamCity server
2020-05-04 16:58:02 CEST: TeamCity process PID is 123
04-May-2020 16:58:03.151 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/8.5.47
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Oct 7 2019 13:30:46 UTC
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.47.0
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.15.0-91-generic
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
04-May-2020 16:58:03.152 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/java/openjdk/jre
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_232-b09
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Amazon.com Inc.
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/teamcity
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/teamcity
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/teamcity/conf/logging.properties
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx2048m
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dteamcity.configuration.path=../conf/teamcity-startup.properties
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dlog4j.configuration=file:/opt/teamcity/bin/../conf/teamcity-server-log4j.xml
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dteamcity_logs=/opt/teamcity/bin/../logs
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/teamcity
04-May-2020 16:58:03.153 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/teamcity
04-May-2020 16:58:03.154 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/teamcity/temp
04-May-2020 16:58:03.154 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
04-May-2020 16:58:03.199 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8111"]
04-May-2020 16:58:03.206 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
04-May-2020 16:58:03.213 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 259 ms
04-May-2020 16:58:03.228 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
04-May-2020 16:58:03.228 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.47
04-May-2020 16:58:03.235 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/teamcity/webapps/ROOT]
Using logs directory "/opt/teamcity/logs"
Log4J configuration file /opt/teamcity/bin/../conf/teamcity-server-log4j.xml will be monitored with interval 10 seconds.
04-May-2020 16:58:05.977 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/teamcity/webapps/ROOT] has finished in [2,741] ms
04-May-2020 16:58:05.980 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8111"]
04-May-2020 16:58:06.017 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 2803 ms
TeamCity version: 2019.2.2 (build 71923), data format version 915
OS: Linux, version 4.15.0-91-generic, amd64, Current user: root, Time zone: CEST (UTC+02:00)
Java: 1.8.0_232, OpenJDK 64-Bit Server VM (25.232-b09, mixed mode), OpenJDK Runtime Environment (1.8.0_232-b09), Amazon.com Inc.; JVM parameters: -Djava.util.logging.config.file=/opt/teamcity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx2048m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:/opt/teamcity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=/opt/teamcity/bin/../logs -Djava.awt.headless=true -Dignore.endorsed.dirs= -Dcatalina.base=/opt/teamcity -Dcatalina.home=/opt/teamcity -Djava.io.tmpdir=/opt/teamcity/temp
[2020-05-04 16:58:06,683] WARN [:05.910), node id: MAIN_SERVER] - etbrains.buildServer.HEARTBEAT - Heartbeat pulse error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Fail to peek column 7 with type java.sql.Timestamp from result set.: java.sql.SQLException: HOUR_OF_DAY: 2 -> 3. Caused by: com.mysql.cj.exceptions.WrongArgumentException: HOUR_OF_DAY: 2 -> 3 (enable debug to see stacktrace)
2 comments
Comment actions Permalink

Hi, and thanks for your report,

 

this has been reported a few times and traced back to an issue with DST: https://youtrack.jetbrains.com/issue/TW-64914

 

Changing timezones or dropping the database table for the heartbeats seems to bring back the server, as detailed in the issue.

0
Comment actions Permalink

Thanks a lot! This worked! 

 

 

KR Andreas

0

Please sign in to leave a comment.