Why we are moving away from TeamCity

Completed

Hey JetBrains,

I'm sorry if this is not the place to put things like this, but i thought you might want to know, so you had a chance to improve in the future :)

We started using TeamCity in 2015 and was actually pretty happy with it, until about a year ago. 

As we started using TC for more and more things, TeamCity has gotten slower and slower (And is right now crashing several times a day, without anything discernable in the logs). We have about 90 build configuration (Which isn't that much according to the documentation), and about 10 developers accessing TC on a daily basis. According to varios posts i have found on the internet this can be solved by increasing the amount of memory TC has available, that however requires switching to x64, which will then break every time we update TC. 

Additionally we started using versioned configuration as a way of managing our configuration as code. This works reasonably well. However it has several glaring issues, mainly related to using different versions of configurations across branches, see https://youtrack.jetbrains.com/issue/TW-60493, on which there have been absolutely zero communication about efforts to fix things. 

Furthermore the versioned scripts that are generated are not generated well enought that you can actually edit them without losing things like build history, or for moving between VCS providers. 

 

TC is currently working so badly for us, that we are putting additional resources and effort into moving to something else, where things actually works. 

There are many good things that we really love about TC, however the above are bad enough deal breakers that we have now called it quits. 

 

 

Again, apologies if this is the wrong place. 

8 comments
Comment actions Permalink

I'm sorry that your experience has not been great, but I'd like to address several of your points, as they are not entirely accurate.

 

-TeamCity getting slower as time passes, to the point of crashes: Memory shortages is the most common scenario, and usage not only depends on the amount of build configurations, but on how much the server gets used, plugin usage, features, vcs roots, etc. It's not always the case that memory shortages are the problem, sometimes there is long standing production installations using the provided internal database rather than an external one which grows very large leading to similar issues. This database is provided just for evaluation purposes and while performs fine in small setups, is not meant to support production servers.

-Upgrades "breaking" java 64 bits: this is a case of only one scenario, a windows installation with a manual upgrade process. If you are running the server under windows, and run the manual upgrade process, and do not follow our upgrade instructions, then yes, the java "will break" (the current installation will be overwritten and the 32 bit will be brought back, more accurately) and the fix is just to remove the shipped jre folder. If you are not using windows or if you are using the automatic upgrade system, it will not break. If you are in that situation, you can follow our upgrade instructions, there is a specific note for it (several, actually).                                                                                                                                 

-Issue with versioned settings: I've raised a note internally that whether we support or not a feature, we should have at least answered, my apologies for that (I was not involved in that specific issue, and there was some internal discussion about it in the issue, just not made public). This said, this is not a bug, it's documented that some settings defined in branches in the DSL will not work: https://www.jetbrains.com/help/teamcity/2019.1/storing-project-settings-in-version-control.html#StoringProjectSettingsinVersionControl-DefiningSettingstoApplytoBuilds . We keep working on expanding support for feature branches in the versioned settings, but due to the internals on how this is managed, adding some of those features is not trivial and requires a large amount of rework on some parts. Usually the approach for requiring those settings in separate branches is to have separate build configurations / projects for them.

-Modifications on versioned scripts leading to lost build history: Most parts of the versioned settings scripts (everything except the IDs) can be modified just fine and the build history will be kept. If the scripts are downloaded in the portable format, then the UUIDs are not added, and as long as the ID is not modified in the script, the build history should still be kept. Furthermore, if the ID is modified and the history is lost, it can be reattached. This is described in the docs here: https://www.jetbrains.com/help/teamcity/2019.1/kotlin-dsl.html#KotlinDSL-Project.kt

 

As a last comment, I'm not sure what you mean with "moving between vcs providers". If you mean moving the settings repository from one vcs to another, then there is no streamlined way to achieve this because we don't really expect it to be a common occurence, but it shouldn't be too complex either, and history should definitely be kept through one such change as well. If you mean something else, please feel free to share the scenario you had in mind.

 

I hope I addressed most of your concerns. If there is anything else, please don't hesitate to point it out.

0
Comment actions Permalink

I concede on the first 2 points, i'm not familiar enough with the process (or java for that matter), to actually know what is going on behind the scenes, i just recall having read somewhere that TC would revert to x32 on upgrades (Especially if using the automatic process, so you would have to manually correct it after every upgrade). 

As for the versioned settings, i'm sorry to say something that might sound a bit angry: If it doesn't work versioned, don't expose it, since everything that can be edited on a per-branch basis, I expect will work there, or at least put a warning somewhere that is easy to access "some of the configured settings differs between branches, and can only be applied from the master branch", with a link to your linked documentation :)

In addition for versioned settings, adding or removing entire build configuration on per-branch basis doesn't seem to work, or I have at least had it not work as expected several times. 


In relation to the modifications, about 99.9% of my problems there could be solved, if the scripts were actually generated with the `id` and `uuid` property specified, since that will ensure the build history doesn't go away. If I then choose to change those, that is my problem. However out of box they are right now not there, which means it uses the configuration name, which changes from time to time. 
I did manage to correct and get my history back, however it's one of those lessons you kinda don't want to learn the hard way. :)

The "moving between vcs providers" means moving from for example from BitBucket to GitHub. I had to disable versioned configuration syncronization and reenable to be able to have it pick up the configuration, since it forgets where the configuraiton comes from, even if it's litterally the same repository, but at another provider. (I just did this today)

 

Thank you for getting back to me :)

0
Comment actions Permalink

Thanks for your feedback, it's important to understand what the issues are in order to improve.

I've created an issue in our tracker for the lack of uuids or any explanations about them: https://youtrack.jetbrains.com/issue/TW-63837 . You have not been the first to temporally lose the build history to replacing IDs, which clearly implies that our decision to not add them in had unexpected consequences on our end. We'll see what we can do to improve on that aspect.

 

About the upgrade as information, before we introduced the automatic upgrade process, that was true (but again, it's listed in the upgrade documentation multiple times to ensure it's performed). The automatic upgrade process does not modify the jre subfolder, so whether the old installation was kept, removed or altered, the modifications will stay. It might have been outdated or wrong information, but this is definitely the current situation. The .tar.gz package does not include the jre at all so it's not an issue in that situation.

 

About the branch-specific versioned settings, some of them do work, as listed in the link I provided. There are restrictions to what can and cannot be modified, listed in the specific section I linked, and one of the features that does not work is removing or creating full build configurations. We provide the ability to use it because the current set of working features does cover a number of use cases and we do plan on expanding those. I do agree that it might be better to warn about this directly rather than expecting the user to go directly to the documentation, but this is the reasoning it is enabled and in its state. I created an issue for it to be warned explicitly: https://youtrack.jetbrains.com/issue/TW-63838

 

About shifting providers for your versioned settings repo, at the moment there isn't much I can say, I haven't tested it yet. This said, TeamCity tracks revisions for its vcs roots, but understands that changes in some critical elements, such as url or tracked branches could lead to issues and under some of this changes it will cut its history. Versioned settings roots are not treated differently so a modification in the full URL is likely to produe such a cut, which could possibly lead to issues. Disabling the sync before doing the change, and enabling it again afterwards is likely to solve the problem, and given that this kind of changes are uncommon is probably enough for most scenarios.

Again, thanks for your feedback, if you have anything else, please feel free to discuss it here, or to open an issue in the tracker. Even if we do not answer directly, they are all handled.

0
Comment actions Permalink

Hello Rasmus,

I just wanted to clarify one thing. You said "... However out of box they are right now not there, which means it uses the configuration name, which changes from time to time. ", but if id of the build configuration is not specified it is taken from the class name of the build configuration which usually does not change that often.

So is this your case: you're losing the history when you change the build configuration class name?

0
Comment actions Permalink

Hello Pavel:

I lose it if I just change the displayed name, since that is what the ID seems to be generated from by default. I can change the kotlin class name without issues, as long as I keep the display name the same. 

0
Comment actions Permalink

This seems strange to me. As far as I understand we should not use names of entities when we generate ids. Can you attach an example of such a DSL code?

0
Comment actions Permalink

Appologies, but it has been a while since i corrected that, and it could just as well have been the class names, since i usually update the class name and the display name at the same time. But you are probably right, that it's the class names being used. 

 

0
Comment actions Permalink

Ok, I also see that our documentation is wrong about that because it claims that name can be used to generate id, but this is wrong. We'll fix it ASAP.

1

Please sign in to leave a comment.