Team City Upgrade from 6.0.3 to 7.1.4


Hello

 

Based on your advice (in a previous thread), I want to upgrade my company's production TC enterprise server from 6.0.3 to 7.1.4. We would like to take advantage of all the new enhancements that JetBrains has included in this version.

 

The problem is that the last two times my colleagues tried to upgrade (to 7.1 back in Sept), we had to rollback because, in the new version, it was taking a lot longer to check for changes. Initial testing of the upgrade seemed to indicate (we’re not sure though) that this was a capacity-related issue due to the number of VCS roots in our instance.

 

To be clear, the upgrade itself did not fail, just the performance in the new version was terribly slow. Therefore, we decided to stick with 6.0 until recently when the performance is degrading our developers have not been able to take advantage of the new features that TC offers in 7.1.4.

 

I would like to get your help in possibly determining the cause behind the slowness as well as get your advice on how to proceed with the upgrade this time. So, there are actually two threads here:

 

1 -  Last Upgrade/Rollback – root cause:

 

Unfortunately, during the last upgrade attempts there were no thread dumps taken while testing.  However, I have attached some of the relevant parts of the logs (we tried upgraded on 9/10 and 9/17). My hope is that you may be able to see something that we may have missed.

 

2 – This attempt – advice:

 

In order to avoid breaking production this time I would like to test out the upgrade on QA. My hope is that if I can get QA as close to prod as possible, I can test out most of the possible breaks, thus minimizing the risk on prod. Currently, we have 24 build agents on production and we use SVN as our VCS. We also use MS SQL as our external DB. I can replicate the DB, install the new server on QA and install a new build agent that uses the new server. However, replicating SVN will be a problem.

 

My questions around this are –

 

  • How do you test performance related issues? What are some of the smoke tests you use in QA?
  • How do I replicate the “checking for changes” part without having to replicate prod SVN or having to point the prod SVN to the QA version.
  • What are some steps that I can do in QA to ensure that this time the upgrade on prod does not fail
  • Am I approaching this wrong? Should I be following a different route.
  •   Finally, does JetBrains assist in upgrades/support (remote assist possibly)? If so, this is another avenue we can explore to guardrail against failing in prod. What are the costs associated with this kind of support?

 

Thanks in advance for your time.

 

Minoo



Attachment(s):
teamcity-winservice.log.zip
teamcity-vcs.log.zip
teamcity-server.log.zip
teamcity-javaLogging-2012-09-17.log.zip
teamcity-agent.log.3.zip
1 comment
Comment actions Permalink

I received an email from Yegor:

Hi Minoo,

Looking in the logs you've sent, it looks like you copied the new TeamCity installation files over the old ones or at least preserved some of the old files under Tomcat's webapps directory. This is against our instructions on server upgrading and proper server functioning can't be guaranteed in such configuration.

BTW, please use the latest version (7.1.4) for upgrade.

Other then that I see no obvious VCS-realted issues and we would need detailed report with thread dumps to analyze.

How do you guys test performance related issues?


This is too generic a question, so I do not have an answer on this level.
We have some performance-related tests in our codebase, but that are specific to the implementation logic.
You probably mean high-level performance. Any issues seen can be investigated with the logs, thread dumps, etc.

How do I replicate the “checking for changes” part without having to replicate prod SVN or having to point the prod SVN to the QA version.


Not sure what you mean here. Are there issues with QA TeamCity server accessing production SVN? Usually we assume the version control server is performant enough to handle high load, and there is usually no need like that.

What are some steps that I can do in QA to ensure that this time the upgrade on prod does not fail


Running QA server for some time and checking the performance is usually a good enough approach. Changes detection part will work just like on a real server. If you care for checkout time, connecting several agents and running several builds will probably be necessary.

Am I approaching this wrong? Should I be following a different route.


A test server is usually the right thing to do.

Finally, does JetBrains assist in upgrades/support (remote assist possibly)? If so, this is another avenue we can explore to guardrail against failing in prod. What are the costs associated with this kind of support?


We do not have different levels of support at this time. If you are facing a major issue with a production server and ultimately need an urgent response, you can CC your message to teamcity-feedback-urgent@jetbrains.com address so that we know you need the reply ASAP and try to handle it correspondingly.

--
Best regards,

Yegor Yarko
QA Engineer (TeamCity)
JetBrains GmbH, Munich
http://www.jetbrains.com
"Develop with pleasure!"

0

Please sign in to leave a comment.