Our Enterprise TeamCity installation is growing exponentially and we are beginning to deal with performance and scale issues. We've only been running TC in this way for several months. In the past, each development team created their own TeamCity/Jenkins/CruiseControl.Net instance with their own set of (several dozen) projects. Now we are moving towards one central TC server, with many build agents (on different servers). Because of this we are finding some issues:
- The TC project based permissions are cumbersome. We need more granularity. For example, a project has a compile build-configuration that just compiles the solution and maybe runs unit tests, etc... It also has a dev-deploy build-configuration, which deploys and installs artifacts from from the compile build-configuration to some development server. There are also staging-deploy and a production-deploy build-configurations that install artifacts to respective staging and production environments. These environments are protected and deployments to them, triggered by those with access rights. In order to secure them within TC, we create seperate projects like myproject-dev, myproject-staging, etc... Without considering environments, our organisation already has hundreds of projects, when environments are considered, there's a lot more.
- The more build configurations you have, the slower TC gets. This is kind of expected. We mitigate this by making sure that checkouts happen on the agent, not the server. The other thing that causes performance problems is when the agents are sending artifacts back to the server. I don't have a workaround for this but it would be better if individual build-configurations could control where artifacts go instead of sending them back to the TC server, they could just tell the TC server where to get them from. Some artifacts are hundreds of megabytes so all that network traffic between agents and TC server make performance on the TC server quite slow. Even just the ability to have an artifact server on a different piece of hardware to the TC server, would speed things up a lot.
- Even without considering the extra projects needed to solve permissions problems, we have too many projects on the default overview screen. People can create their own visible/hidden project configurations but it would be better if there was a way to break the project list up a bit or add some hierarchy or tagging system. With a hierarchy, you would have lists of business areas that contain projects below them and just show the business areas on the overview screen. Better still a project tagging system would allow projects to have more than one tag (and so exist in more than one project list). Then, the overview screen could show the list of tags (and summary of build success/failure), with drill-down/click-through to the projects that contain those tags.
TeamCity is already more awesome than many/most other CI servers. JetBrains, please implement my wish list above and I will continue to make my company buy many hundreds more build agent licenses.