After upgrade, TeamCity service is running but not accepting requests

We had TeamCity 2020.2.4 running on Windows Server 2012 R2.  A reverse proxy server using IIS has been setup.  IIS handles the certificates.  The database is SQL Server. Creating new accounts have been turned off and all accounts are authorized with TeamCity SAML authentication plug-in. 
On the website I scheduled an update to 2021.1.4.  The update, according to what I see in the logs, went well.  The server started up and is waiting for confirmation for the Schema update page.  However, the server is not accepting requests and just returns "The service is unavailable.".  The service is running and the teamcity-server.log states that there is a data upgrade required and gives the authentication token.
  
Why would the server be unavailable?  If I reproduce the upgrade from 2020.2.4 to 2021.1.4 on a separate machine as a test, I do get similar logs.  I also can see the webpage with the information about the data migration, the option to backup the database and a button to upgrade.  

Thanks

Tim

5 comments
Comment actions Permalink

If the teamcity logs report that it's up and running, the most likely cause is that the proxy is having issues with processing the requests. Could you try accessing the server jumping over the proxy? You might also want to review in the logs that the server is going up on the expected address and port. You can also check IIS logs to see if there is anything in there that might hint at the issue.

0
Comment actions Permalink

Thanks for the reply. Can I turn off the secure and scheme?  I can't connect to 127.0.0.1:8081 to jump over the proxy because IIS is supplying the certificate.  127.0.0.1:8081 complains about the certificate.   Maybe edit server.xml to provide a port without secure="true" and scheme="https"?

 

 

0
Comment actions Permalink

I was able to get this working. I had to create a new Connector on a new port in server.xml that did not have https (this connector did not define secure nor scheme="https").  This is for localhost.  Then I could connect to http://localhost:port and see the request to upgrade scheme (after entering the administrator password token).  Once the upgrade was complete.  The IIS reverse proxy worked and TeamCity is up and running.  Something about requiring administrator to finish the upgrade wasn't able to go through the IIS proxy.

Is this normal? Will I need to do this for every upgrade that requires a schema change?

Thanks for your help.

0
Comment actions Permalink

It's not normal, to the extent of our knowledge. This is why I suggested checking IIS logs, since it's most likely that it's somehow misbehaving (as you saw, just pulling a separate endpoint works just fine), it may be misconfigured for this specific case, so the logs might have contained some additional information.

0
Comment actions Permalink

I did not setup the IIS server and not that familiar with the configuration.  The logs from IIS report attempts to talk to a build agent and get a 503 error.  The agent is on a private network separate from the company network.  Only the build agents on this network are in the logs. Here is one example.  

2022-02-17 00:00:01 W3SVC1 SRVNAME 10.240.0.101 GET /app/agents/protocols X-ARR-CACHE-HIT=0&X-ARR-LOG-ID=5bf187af-4f31-4be2-8e21-8dba1842b54d&SERVER-STATUS=503 8443 - 10.240.0.132 HTTP/1.1 TeamCity+Agent+2020.2.4 - - 127.0.0.1:8081 503 0 0 341 120 15

Once I created a new connector and finished the upgrade, these 503 errors disappeared. The build agents connected and updated then restarted themselves just fine.

 

0

Please sign in to leave a comment.