Stuck in Maintenance Mode after server upgrade

Hello

So I have been using https://github.com/JetBrains/teamcity-cloudformation-template in our AWS account to run teamcity as our CI server. I have performed several upgrades in the past year of the server node and the DB (upgrading version) successfully.

 

But this time when I upgraded to TeamCity 2021.1.2 (build 92869) there's seems to be an error.

 

I cannot login to the Super user page in Maintenance Mode even after using the token from teamcity-server.log. After I put the token I get redirected to the same page

Any help is appreciated. Any other log I can read to debug the issue? Can I recover my environment?

 

[2021-08-2804:54:24,834] INFO - jetbrains.buildServer.STARTUP - ===========================================================
[2021-08-2804:54:24,834] INFO - jetbrains.buildServer.STARTUP - Starting TeamCity server
[2021-08-2804:54:25,031] INFO - jetbrains.buildServer.STARTUP - TeamCity version: 2021.1.2 (build 92869), data format version 976
[2021-08-2804:54:25,036] INFO - jetbrains.buildServer.STARTUP - OS: Linux, version 4.9.85-38.58.amzn1.x86_64, amd64, Current user: root, Time zone: GMT (UTC)
[2021-08-2804:54:25,037] INFO - jetbrains.buildServer.STARTUP - Java: 11.0.11, OpenJDK 64-Bit Server VM (11.0.11+9-LTS, mixed mode), OpenJDK Runtime Environment (11.0.11+9-LTS), Amazon.com Inc.; JVM parameters: --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/opt/teamcity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager-Djdk.tls.ephemeralDHKeySize=2048-Djava.protocol.handler.pkgs=org.apache.catalina.webresources-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1850m -XX:MaxPermSize=270m -XX:ReservedCodeCacheSize=350m -Dteamcity.configuration.path=../conf/teamcity-startup.properties-Dlog4j.configuration=file:/opt/teamcity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=/opt/teamcity/bin/../logs -Djava.awt.headless=true-Dignore.endorsed.dirs= -Dcatalina.base=/opt/teamcity -Dcatalina.home=/opt/teamcity -Djava.io.tmpdir=/opt/teamcity/temp
[2021-08-2804:54:25,038] INFO - jetbrains.buildServer.STARTUP - Garbage collection: G1 Young Generation, G1 Old Generation
[2021-08-2804:54:25,038] INFO - jetbrains.buildServer.STARTUP - Web server version: Apache Tomcat/8.5.65
[2021-08-2804:54:25,039] INFO - jetbrains.buildServer.STARTUP - Server IP address: 172.17.0.2
[2021-08-2804:54:25,041] INFO - jetbrains.buildServer.STARTUP - Web server ports: 8111 (protocol: HTTP/1.1, scheme: https), 8105 (shutdown port)
[2021-08-2804:54:25,042] INFO - jetbrains.buildServer.STARTUP - Context path: "/", servlet real path: "/opt/teamcity/webapps/ROOT"
[2021-08-2804:54:25,047] INFO - jetbrains.buildServer.STARTUP - TeamCity distribution type: docker-ubuntu-aws
[2021-08-2804:54:25,065] INFO - jetbrains.buildServer.STARTUP - TeamCity server node id: MAIN_SERVER
[2021-08-2804:54:25,068] INFO - jetbrains.buildServer.STARTUP - Version file lock was successfully obtained: /opt/teamcity/webapps/ROOT/BUILD_92869
[2021-08-2804:54:25,081] INFO - jetbrains.buildServer.STARTUP - Current stage: Looking for the TeamCity Data Directory
[2021-08-2804:54:25,081] INFO - jetbrains.buildServer.STARTUP - The TeamCity Data Directory path is determined using the environment variable TEAMCITY_DATA_PATH
[2021-08-2804:54:25,086] INFO - jetbrains.buildServer.STARTUP - The TeamCity Data Directory "/data/teamcity_server/datadir" exists
[2021-08-2804:54:25,094] INFO - jetbrains.buildServer.STARTUP - Node specific data directory location: /data/teamcity_server/datadir
[2021-08-2804:54:25,095] INFO - jetbrains.buildServer.STARTUP - Will use custom path for the database properties file: /data/teamcity_server/datadir/config/database.properties
[2021-08-2804:54:25,095] INFO - jetbrains.buildServer.STARTUP - The TeamCity server caches directory is set to the path: /data/teamcity_server/datadir/system/caches
[2021-08-2804:54:25,097] INFO - jetbrains.buildServer.STARTUP - The main configuration file "/data/teamcity_server/datadir/config/main-config.xml" exists
[2021-08-2804:54:25,097] INFO - jetbrains.buildServer.STARTUP - The database properties file "/data/teamcity_server/datadir/config/database.properties" exists
[2021-08-2804:54:25,097] INFO - jetbrains.buildServer.STARTUP - The internal database data file "/data/teamcity_server/datadir/system/buildserver.data" doesn't exist
[2021-08-2804:54:25,104] INFO - jetbrains.buildServer.STARTUP - Data Directory version: 958
[2021-08-2804:54:25,104] INFO - jetbrains.buildServer.STARTUP - TeamCity main node (id: MAIN_SERVER) is starting...
[2021-08-2804:54:25,105] INFO - jetbrains.buildServer.STARTUP - Current stage: Looking for the database configuration
[2021-08-2804:54:25,105] INFO - jetbrains.buildServer.STARTUP - Database properties file "/data/teamcity_server/datadir/config/database.properties" exists
[2021-08-2804:54:25,106] INFO - jetbrains.buildServer.STARTUP - Internal HSQL database file (/data/teamcity_server/datadir/system/buildserver.data) doesn't exist
[2021-08-2804:54:25,106] INFO - jetbrains.buildServer.STARTUP - Database connection URL: jdbc:mysql://RDS/teamcitydb
[2021-08-2804:54:25,107] INFO - jetbrains.buildServer.STARTUP - Using database connection URL from the database properties file. The URL is: jdbc:mysql://RDS/teamcitydb
[2021-08-2804:54:25,108] INFO - jetbrains.buildServer.STARTUP - Using external (MYSQL) database
[2021-08-2804:54:25,108] INFO - jetbrains.buildServer.STARTUP - Current stage: Connecting to the database
[2021-08-2804:54:25,109] INFO - jetbrains.buildServer.STARTUP - JDBC loader: loading JDBC drivers (for the first time)
[2021-08-2804:54:25,199] INFO - jetbrains.buildServer.STARTUP - Effective database connection properties:
[2021-08-2804:54:25,199] INFO - jetbrains.buildServer.STARTUP - alwaysSendSetIsolation=false [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - cachePrepStmts=true [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - characterEncoding=UTF-8 [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - elideSetAutoCommits=true [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - noTimezoneConversionForDateType=false [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - noTimezoneConversionForTimeType=false [defaults]
[2021-08-2804:54:25,200] INFO - jetbrains.buildServer.STARTUP - prepStmtCacheSize=300 [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - serverTimezone=GMT [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - useGmtMillisForDatetimes=true [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - useJDBCCompliantTimezoneShift=true [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - useLegacyDatetimeCode=false [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - useLocalSessionState=true [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - useTimezone=true [defaults]
[2021-08-2804:54:25,201] INFO - jetbrains.buildServer.STARTUP - userServerPrepStmts=true [defaults]
[2021-08-2804:54:26,007] INFO - jetbrains.buildServer.STARTUP - Database server time zone: UTC (+00:00)
[2021-08-2804:54:26,008] INFO - jetbrains.buildServer.STARTUP - Database server character set and collation: utf8mb3.utf8_bin
[2021-08-2804:54:26,010] INFO - jetbrains.buildServer.STARTUP - Database character set: utf8mb4
[2021-08-2804:54:26,010] INFO - jetbrains.buildServer.STARTUP - Database collation: utf8mb4_0900_ai_ci
[2021-08-2804:54:26,015] INFO - jetbrains.buildServer.STARTUP - Database connection character set and collation: utf8mb4.utf8mb4_0900_ai_ci
[2021-08-2804:54:26,016] INFO - jetbrains.buildServer.STARTUP - Database client character set: utf8mb4
[2021-08-2804:54:26,165] INFO - jetbrains.buildServer.STARTUP - JDBC driver version: 8.0 (MySQL Connector/J)
[2021-08-2804:54:26,166] INFO - jetbrains.buildServer.STARTUP - Database system version: 8.0.25 (MySQL)
[2021-08-2804:54:26,166] INFO - jetbrains.buildServer.STARTUP - Connected to the database successfully
[2021-08-2804:54:26,166] INFO - jetbrains.buildServer.STARTUP - Current stage: Checking the database
[2021-08-2804:54:26,402] INFO - jetbrains.buildServer.STARTUP - Database contains 143 tables.
[2021-08-2804:54:26,416] INFO - jetbrains.buildServer.STARTUP - Database version is 958
[2021-08-2804:54:26,420] INFO - jetbrains.buildServer.STARTUP - Upgrade from version 958 to version 976 is required
[2021-08-2804:54:26,420] INFO - jetbrains.buildServer.STARTUP - Backup of this version is possible
[2021-08-2804:54:26,475] INFO - etbrains.buildServer.HEARTBEAT - Attempting to capture the DB lock. My session starting code: 813127524
[2021-08-2804:54:26,554] INFO - etbrains.buildServer.HEARTBEAT - The Heartbeat thread is started.
[2021-08-2804:54:26,563] INFO - etbrains.buildServer.HEARTBEAT - Exclusive DB lock was taken.
[2021-08-2804:54:26,566] INFO - jetbrains.buildServer.STARTUP - Exclusive database lock was taken
[2021-08-2804:54:26,566] INFO - jetbrains.buildServer.STARTUP - Current stage: Data upgrade is required (administrator login is required to proceed)
[2021-08-2804:54:26,566] INFO - jetbrains.buildServer.STARTUP - Administrator can login from web UI using super user authentication token
[2021-08-2804:54:26,566] INFO - jetbrains.buildServer.SERVER - Super user authentication token: 4421578681814499617 (use empty username with the token as the password to access the server)
[2021-08-2805:00:18,347] INFO - jetbrains.buildServer.STARTUP - Successful authentication token entry from IP address: /10.0.0.209 with request POST '/mnt/do/authenticate?token=4421578681814499617', from client 10.0.0.209:54595, no auth

Please sign in to leave a comment.