TeamCity High Availability Setup Issue with External MS SQL Database

We are currently facing connectivity issues with an external MS SQL Database while setting up TeamCity High Availability.

Our current setup consists of a single-node TeamCity server running with an external MS SQL Database and a shared data directory over NFS. The objective is to establish high availability by reconfiguring the existing single-node TeamCity server as the first leg and adding a new server as the second leg. To achieve this, we've directed the `teamcity.data.path` to the existing shared data directory and updated the `database.properties` file to point to the external MS SQL Database.
 

Here are the relevant configurations:

On the first leg server (leg-1-ip):
- `teamcity.data.path` in `/opt/TeamCity/conf/teamcity-startup.properties` is set to `/opt/BuildServer`.
- In `/opt/BuildServer/config/database.properties`, the database connection properties are configured with a username, password, and the connection URL to the external MS SQL Database.

After configuring the above settings, we've also set the `TEAMCITY_SERVER_OPTS` environment variable as follows:

export TEAMCITY_SERVER_OPTS="-Dteamcity.server.nodeId=1 -Dteamcity.server.rootURL=http://<leg-1-ip>:8111 -Dteamcity.data.path=/opt/Buildserver -Dteamcity.node.data.path=/opt/teamcity_node"

Then, we initiated the TeamCity services using the following command:
/opt/TeamCity/bin/teamcity-server.sh start

The problem we're encountering is that, instead of utilizing the existing external MS SQL Database, TeamCity is attempting to create a new database. We're currently uncertain about the exact point where we may be making an error. The provided logs for reference are as follows:

----------------------

[2023-11-07 04:59:39,818]   INFO -  jetbrains.buildServer.STARTUP - ===========================================================
[2023-11-07 04:59:39,818]   INFO -  jetbrains.buildServer.STARTUP - Starting TeamCity server
[2023-11-07 04:59:40,064]   INFO -  jetbrains.buildServer.STARTUP - TeamCity version: 2023.05.1 (build 129321), data format version 994
[2023-11-07 04:59:40,072]   INFO -  jetbrains.buildServer.STARTUP - OS: Linux, version 3.10.0-1160.66.1.el7.x86_64, amd64, Current user: root, Time zone: EST (UTC-05:00)
[2023-11-07 04:59:40,072]   INFO -  jetbrains.buildServer.STARTUP - Java: 11.0.15, OpenJDK 64-Bit Server VM (11.0.15+9-LTS, mixed mode, sharing), OpenJDK Runtime Environment (11.0.15+9-LTS), Red Hat, Inc.; JVM parameters: --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=640M --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/opt/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dteamcity.server.nodeId=1 -Dteamcity.server.rootURL=http://leg-1-ip:8111 -Dteamcity.data.path=/opt/Buildserver -Dteamcity.node.data.path=/opt/teamcity_node -Xmx1024m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j2.configurationFile=file:/opt/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=/opt/TeamCity/bin/../logs -Djava.awt.headless=true -Dignore.endorsed.dirs= -Dcatalina.base=/opt/TeamCity -Dcatalina.home=/opt/TeamCity -Djava.io.tmpdir=/opt/TeamCity/temp
[2023-11-07 04:59:40,075]   INFO -  jetbrains.buildServer.STARTUP - Garbage collection: G1 Young Generation, G1 Old Generation
[2023-11-07 04:59:40,075]   INFO -  jetbrains.buildServer.STARTUP - Web server version: Apache Tomcat/9.0.75
[2023-11-07 04:59:40,078]   INFO -  jetbrains.buildServer.STARTUP - Server IP addresses: leg-1-ip
[2023-11-07 04:59:40,080]   INFO -  jetbrains.buildServer.STARTUP - Web server ports: 8111 (protocol: HTTP/1.1, scheme: http), 8105 (shutdown port)
[2023-11-07 04:59:40,081]   INFO -  jetbrains.buildServer.STARTUP - Context path: "/", servlet real path: "/opt/TeamCity/webapps/ROOT"
[2023-11-07 04:59:40,092]   INFO -  jetbrains.buildServer.STARTUP - TeamCity distribution type: tar.gz
[2023-11-07 04:59:40,092]   INFO -  jetbrains.buildServer.STARTUP - TeamCity server node id: 1
[2023-11-07 04:59:40,095]   INFO -  jetbrains.buildServer.STARTUP - Version file lock was successfully obtained: /opt/TeamCity/webapps/ROOT/BUILD_129321
[2023-11-07 04:59:40,142]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Looking for the TeamCity Data Directory
[2023-11-07 04:59:40,142]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity Data Directory path is determined using the JVM property teamcity.data.path
[2023-11-07 04:59:40,148]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity Data Directory "/opt/Buildserver" exists
[2023-11-07 04:59:40,148]   INFO -  jetbrains.buildServer.STARTUP - Asking user to confirm first start with the predefined TeamCity Data Directory path
[2023-11-07 04:59:40,156]   INFO -  jetbrains.buildServer.STARTUP - Startup confirmation is required. Open TeamCity web page in the browser. Server is running at http://localhost:8111
[2023-11-07 04:59:40,157]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Confirming TeamCity first start (administrator login is required to proceed)
[2023-11-07 04:59:40,157]   INFO -  jetbrains.buildServer.STARTUP - Administrator can login from web UI using super user authentication token (better use a private browser window)
[2023-11-07 04:59:40,157]   INFO -   jetbrains.buildServer.SERVER - Super user authentication token: <9999999999999999999999> (use empty username with the token as the password to access the server)
[2023-11-07 05:01:03,914]   INFO -  jetbrains.buildServer.STARTUP - Successful authentication token entry from IP address: /client-1-ip with request POST '/mnt/do/authenticate?token=9999999999999999999', from client client-1-ip:50194, no auth
[2023-11-07 05:01:08,387]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Initializing the TeamCity Data Directory
[2023-11-07 05:01:08,387]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Confirming the TeamCity node type (administrator login is required to proceed)
[2023-11-07 05:01:19,717]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Initializing the TeamCity Data Directory
[2023-11-07 05:01:19,983]   INFO -  jetbrains.buildServer.STARTUP - Downloading JDBC driver metadata from <https://download.jetbrains.com/teamcity/database-drivers/2023.5.xml?myVersion=2023.05.1&javaVersion=11.0.15>...
[2023-11-07 05:05:35,135]   WARN -  jetbrains.buildServer.STARTUP - Network is unreachable (connect failed)
[2023-11-07 05:05:35,136]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Restore settings (administrator login is required to proceed)
[2023-11-07 05:06:25,823]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Looking for the TeamCity Data Directory
[2023-11-07 05:06:25,823]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity Data Directory "/opt/Buildserver" exists
[2023-11-07 05:06:25,824]   INFO -  jetbrains.buildServer.STARTUP - Asking user to confirm first start with the predefined TeamCity Data Directory path
[2023-11-07 05:06:25,824]   INFO -  jetbrains.buildServer.STARTUP - Startup confirmation is required. Open TeamCity web page in the browser. Server is running at http://localhost:8111
[2023-11-07 05:06:25,824]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Confirming TeamCity first start (administrator login is required to proceed)
[2023-11-07 05:06:41,886]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Initializing the TeamCity Data Directory
[2023-11-07 05:06:41,887]   INFO -  jetbrains.buildServer.STARTUP - Node specific data directory location: /opt/teamcity_node
[2023-11-07 05:06:41,887]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity server caches directory is set to the path: /opt/teamcity_node/system/caches
[2023-11-07 05:06:41,929]   INFO - er.https.HttpsConfiguratorImpl - Https is not configured.
[2023-11-07 05:06:41,937]   INFO -  jetbrains.buildServer.STARTUP - Downloading JDBC driver metadata from <https://download.jetbrains.com/teamcity/database-drivers/2023.5.xml?myVersion=2023.05.1&javaVersion=11.0.15>...
[2023-11-07 05:10:56,536]   WARN -  jetbrains.buildServer.STARTUP - Network is unreachable (connect failed)
[2023-11-07 05:10:56,537]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Setting up database connection (administrator login is required to proceed)
[2023-11-07 05:13:21,608]   INFO -  jetbrains.buildServer.STARTUP - Generated JDBC connection string: jdbc:sqlserver://external.db.com\TeamCity:dbport;databaseName=dbname;user=teamcityuser;password=********
[2023-11-07 05:13:21,608]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Creating a new database
[2023-11-07 05:13:21,609]   INFO -  jetbrains.buildServer.STARTUP - Creating a new database
[2023-11-07 05:13:21,646]  ERROR -  jetbrains.buildServer.STARTUP - No suitable JDBC driver found for database type: MSSQL
[2023-11-07 05:13:21,646]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Setting up database connection (administrator login is required to proceed)
[2023-11-07 05:13:39,104]   INFO -  jetbrains.buildServer.STARTUP - Generated JDBC connection string: jdbc:sqlserver://external.db.com:dbport;databaseName=dbname;user=teamcityuser;password=********
[2023-11-07 05:13:39,104]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Creating a new database
[2023-11-07 05:13:39,104]   INFO -  jetbrains.buildServer.STARTUP - Creating a new database
[2023-11-07 05:13:39,105]  ERROR -  jetbrains.buildServer.STARTUP - No suitable JDBC driver found for database type: MSSQL
[2023-11-07 05:13:39,105]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Setting up database connection (administrator login is required to proceed)
[2023-11-07 05:45:35,894]   INFO -  jetbrains.buildServer.STARTUP - Server shutdown event received
[2023-11-07 05:45:35,894]   INFO -  jetbrains.buildServer.STARTUP - Current stage: TeamCity server is shutting down
[2023-11-07 05:45:35,894]   INFO -  jetbrains.buildServer.STARTUP - Stopping the main application
[2023-11-07 05:45:35,894]   INFO -  jetbrains.buildServer.STARTUP - The main servlet is not initialized, nothing to shut down.
[2023-11-07 05:45:35,896]   INFO -  jetbrains.buildServer.STARTUP - Version file lock was released: /opt/TeamCity/webapps/ROOT/BUILD_129321
[2023-11-07 05:45:35,914]   INFO -  jetbrains.buildServer.STARTUP - Server shutdown complete. Uptime was 45m:56s
[2023-11-07 06:03:50,438]   INFO -  jetbrains.buildServer.STARTUP - Path to the startup configuration file: /opt/TeamCity/bin/../conf/teamcity-startup.properties
[2023-11-07 06:03:50,585]   INFO -   jetbrains.buildServer.SERVER - Using logs directory "/opt/TeamCity/logs"

--------------------------------

 

We would greatly appreciate any assistance or insights that could help us resolve this issue.