Migration to TeamCity: Features clarification

Answered

Hi

We got extremely fed up with Jenkins, its instability, abuse of LTS definition and so on, and now considering other options, where 1 of them is TeamCity.
But before migrating to it, I would like to clarify a few features, are they all exist in TeamCity or not.

 

  • Dynamic dockerised slaves using AWS Spot fleet. 

so far we use Jenkins ECS plugin, our slaves are spawned dynamically as ECS dockers on our Spot Fleet, which scales based on CloudWatch rule based on CPU usage. So it allows us theoretically infinitive concurrent builds - where each build happened inside own dockerised isolated slave in our Spot fleet.
I do understand TeamCity pricing politics with agents and concurrent build limitations, but if we will not touch agents context, do TeamCity support dynamic dockerised slaves on ECS with SpotFleet and further scalability? 

  • Dynamic UI with reactive parameters defined inside the pipeline file (IasC - Infrastructure as Code)

As I understand TeamCity also has code pipeline definition like Jenkinsfile which is located in `.Teamcity/settings.kts`, but apart from declarative build stages, we need dynamic parameters selection before the build, considering it is IaaC they also should be defined in declarative pipeline file.

Here is an example of QA/Prod EKS deployment pipelines we have: 
We select an app from the dropdown list, we select app versions from the list dynamically queried from the app repository, then select environment and nspace of EKS where to deploy...
All those dropdowns are dynamic and populated based on previous selections. If you select one app, it will fetch versions from the corresponding repository and populate following dropdown with versions of that app, if select another app, the following dropdown list will contain other values, the same related to environment and nspaces, each environment has own corresponding list of nspaces.
Do `.Teamcity/settings.kts` allow defining such dynamic selection parameters with functions which resolve dynamic values and failover methods? 
In Jenkinsfile it is somewhat a challenge but we managed to organise it in Jenkinsfile with Active Choice Plugin, however, we are still not happy with it and looking for the new and better tool. 

Any chance I could be pointed to see some code snippets of such dynamic build parameters?
Is it base Teamcity functionality or some 3d party plugins? 


Thank you in advance for information.

5 comments
Comment actions Permalink

Hi Vadim, and thanks for your interest. I'll try to answer your questions:

-ECS Support: TeamCity integrates mainly with EC2, for ECS specific features I might need to check for which specific ones. With regards to spawning agents on demand through docker images, yes, it is supported, both for spot and non-spot instances. You can see most details of our AWS support here: https://www.jetbrains.com/help/teamcity/setting-up-teamcity-for-amazon-ec2.html

 

In particular the cloud profiles feature allows you to spawn the build agents dynamically as they are needed, also allowing you to distinguish between images (creates a clean container from the image) or instances (reuses an old container, so you can persist data between runs): https://www.jetbrains.com/help/teamcity/setting-up-teamcity-for-amazon-ec2.html#Configuring+an+Amazon+EC2+cloud+profile

 

-Dynamic parameters. If I understand correctly your need, it should be available by default. You have probably already seen the availability of build parameters. In particular, all build parameters can get a type from a subset of predefined types, and one of those types is "prompt". This type of parameter is considered as required to be "prompted" whenever a build is to be run manually, thus a dialog is presented with the different set of parameters available. In particular, the dialog is our "Run Custom Build" dialog which accepts a large number of different settings, including the "prompt" type of parameter.

 

It would look like something like this:

params {
text("name", "value", "label", "description", ParameterDisplay.PROMPT)
}

This example is obviously for a text field, but it's the same for dropdowns or any other type of supported values.

You can find more details here: https://www.jetbrains.com/help/teamcity/typed-parameters.html#Adding+Parameter+Specification . While the DSL syntax for them is not explicitly described in it, keep in mind that if at some point the IDE help is not enough, you can always set in the UI a build configuration with the setting you want.then click the "View DSL" button on the page that will give you the exact syntax for the setting as you have defined it.

 

In terms of saving this values for future runs, TeamCity stores any options modified for this dialog in the user session. This will be saved for any given user on a specific browser but other users will still see the default values (or their last stored one). If the actual default values need to be altered on every run, you would need to actively modify the build configuration itself on those runs. This is not too hard to do, it can be done via the REST API, but is not the default behavior. If you would like to have it be that way, it would be useful if you could share the full use case with as much detail as possible

 

If the default values should be fine, it's also possible to not use the prompt type, but modify them when required. Next to the "Run" button, there is an ellipsis button. Clicking on it will display the exact same dialog also remembering those fields, so you can click run and trigger with the default values, or click on the ellipsis and have the option to alter them. Possibly relevant, clicking on Run without "prompt" values will *not* use the last remembered values for the given user. It will use the defaults for the build configuration, since those values are only loaded from the user session when the dialog is opened. Also important to note, this prompt will only be shown on manually triggered builds, automatic ones will go through with the default values, just like when clicking on Run.

 

I think this should cover most options. As usual, feel free to test TeamCity on a local test installation. The Professional edition is free of charge and has all features, just limits the number of different build configurations.

 

if you have any further questions, feel free to ask.

0
Comment actions Permalink

Hi, @Denis Lapuente thank you for your fast response!

Regarding agent spawned dynamically on-demand trough ECS and Spot Fleet - your link to documentation seems to describe cases with Spot instances where build happening on the instance itself, where I was talking more about ECS based on Spot Fleet cluster, but anyway after a bit of googling I found this plugin: https://github.com/JetBrains/teamcity-amazon-ecs-plugin seems you have this feature and what is more - it is your home plugin, not 3d party what is very good and promises quality.

Regarding dynamic fields - it seems I didn't explain it correctly, as filed type with

ParameteIrDisplay.PROMPT

is not really what I was looking for.

Let me explain in more details:

Here is our UI screenshot from Jenkins: 

As you can see we have multiple dropdowns
For example, first ENV_TYPE has 2 values [qa, prod]
and based on its selection - the content in the following dropdown IP_2ND_OCTET will be different
if ill select "qa" in ENV_TYPE, then IP_2ND_OCTET will be [54, 56]
but if ill select ENV_TYPE to be "prod", the IP_2ND_OCTET will be [250]
NSPACE values also depend on ENV_TYPE choice

and what is more important if APP_DOCKER_TAG will be selected option "other", then new (hidden) dropdown will appear, it will fetch all available tags of the selected app (APP_NAME field) from AWS ECR and populate it. if il change app name, it will fetch other tags from the corresponding repository and re-populate dropdown content.

It means that just ParameteIrDisplay.PROMPT is not enough, it is not user something enter, but rather programmed functionality with the possibility to fetch content from REST URL, and form the response generate desired dropdown list content, if the previous selection will be changed, the dependant dropdown have to be re-generated
So it is a reactive dynamic UI which is programmed inside the pipeline file attached to the project. 

here is an example of simple but ugly Jenkins code related to "Active Choice Plugin" which allow such reactive and dynamic UI:

properties([
	parameters([
		[
			$class: 'ChoiceParameter',
			choiceType: 'PT_SINGLE_SELECT',
			description: 'Select Development Environment Type',
			filterable: false,
			name: 'ENV_TYPE',
			script: [
				$class: 'GroovyScript',
				fallbackScript: [
					classpath: [], sandbox: true, script: 'return ["script exception"]'
				],
				script: [
					classpath: [], sandbox: true,
					script: '''return ["qa", "prod"]'''
				]
			]
		],
		[
			$class: 'CascadeChoiceParameter',
			choiceType: 'PT_SINGLE_SELECT',
			description: 'Select Subnet IP 2nd Octet',
			name: 'IP_2ND_OCTET',
			referencedParameters: 'ENV_TYPE',
			script:  [
				$class: 'GroovyScript',
				fallbackScript: [
					classpath: [], sandbox: false, script: 'return ["script exception"]'
				],
				script: [
					classpath: [], sandbox: false,
					script: '''
					if (ENV_TYPE.equals("qa")) {
						return ["54", "56"]
					} else if (ENV_TYPE.equals("prod")) {
						return ["250"]
					} else {
						return ["unknown"]
					}'''
				]
			]
		],
		[
			$class: 'CascadeChoiceParameter',
			choiceType: 'PT_SINGLE_SELECT',
			description: 'Select Number Space',
			name: 'NSPACE',
			referencedParameters: 'IP_2ND_OCTET',
			script:  [
				$class: 'GroovyScript',
				fallbackScript: [
					classpath: [], sandbox: false, script: 'return ["script exception"]'
				],
				script: [
					classpath: [], sandbox: false,
					script: '''
					if (IP_2ND_OCTET.equals("54")) {
						return ["nspace20", "nspace21", "nspace22", "nspace23", "nspace24", "nspace25"]
					} else if (IP_2ND_OCTET.equals("56")) {
						return ["nspace30", "nspace31", "nspace32", "nspace33", "nspace34", "nspace35"]
					} else if (IP_2ND_OCTET.equals("250")) {
					return ["nspace10"]
					} else {
						return ["unknown"]
					}'''
				]
			]
		]
	])
])


Note: that there is simple ChoiceParameter - which is simple dropdown and CascadeChoiceParameter which allow observing dependant filed.
As you see "CascadeChoiceParameter" has filed "referencedParameters" which points to the observable object and its value will be recalculated on observable state change. 
Additionally, it has body for an executable script which takes 2 values, "script" and "fallbackScript" which called in case of "script" failure. 

So I'm looking for programmable dynamic UI for build parameters with a reactive update on observable filed state change, with various filed types (text, checkbox, radio button etc)


P.S. regarding agents and JB pricing politics - the world is moving into a more dynamic and on-demand world, slaves or agents (in JB terminology) are spawned and killed dynamically and the whole infrastructure also scales and shrinks dynamically. Therefore charging for each next concurrent agent (after first 3 free) is putting you into a weaker position as now parallel builds are so dynamic, nobody even knows how many builds are running concurrently, they just run.
So it maybe makes sense for you to revisit your pricing politic and think how you can keep charging for a licence but avoid limiting concurrent builds as soon it might bite you back in the competitive market. As Jenkins already allow theoretically infinitive concurrent builds and infrastructure will scale up and down also dynamically without extra charges for idle slaves.     

0
Comment actions Permalink

Thanks for the clarification. We don't really have an embedded way of creating dynamic popups like that, although there is a relatively well known third party plugin that covers some of it: https://plugins.jetbrains.com/plugin/9076-web-parameters

 

This said, we typically recommend against selecting environments in popups in many scenarios. While the alternative is creating more build configurations, it brings increased readability and a much simpler way of rerunning anything required, as well as more granular permission control (you might want your QA team to be able to push to QA and not prod, for example). The dependency system typically takes care of not duplicating unnecessary work, since you typically will be running a build that generates the required artifacts, and a separate one that does the deployments, or runs whatever required on the remote servers. It is commonly said that this selectors make it easier to manage, but I would argue that the snippet you shared explicitly display that it's hardly the case. Whatever we could implement that would allow for dynamic selectors would eventually need to allow for user's logic which needs to be maintained as well. On the other hand, having a build config per environment would not require any of that logic, just set the individual values for each of the configs, which can be set statically, dynamically using the plugin above, and can also be set through code via our DSL.

 

That's without considering that it would just take a single mistake to push to prod what was meant for QA. This mistake is not impossible to make with separate configs, but it's much more obvious and, again, can be removed via the permissions system. There are scenarios where this dropdowns aren't this problematic and using them might be fine, but they are quite rare, while the multiconfig setup always works and brings advantages.

 

We also appreciate the suggestions about pricing. There has been a long standing discussion about it and since we have recently opened a public beta for a SaaS model we are finalizing pricing options in that direction. We have a request to introduce a cloud-friendly model in general here: https://youtrack.jetbrains.com/issue/TW-48045. It's not a trivial task, since our licensing model was meant for on prem, where the servers might never have access to the internet, and the licensing model should still cover that case, but cloud profiles don't just work with cloud-based services, they also work with tools like docker, kubernetes, or vmware which can all be run in the internal network, never letting the server get out. We still have a non-trivial amount of installations like that, so whatever we do has to support them. If you have a specific scenario in mind, feel free to share it in the comments of that request in our tracker, they will be appreciated by the people making the calls.

 

I think this should cover most points, if you still have any question feel free to follow up

0
Comment actions Permalink

Denis Lapuente thank you once again for such detailed response, it seems we could attempt to set up TeamCity and implement some sort of POC.

Regarding multiple pipelines with pre-set/hardcoded values for jobs instead of dynamic UI selection. I suspect here we facing another issue...

As I understand Teamcity support only one pipeline file per project (.Teamcity/settings.kts) and it could be a limitation for us as we have made in-house framework to deal with EKS (clean/deploy/info etc)
Means we could create separate pipelines for each app deployment individually and pre-set some params per each, but all of them still will refer to some git project and as I understand we cant create multiple settings file per each pipeline as TeamCity support only 1 per project, plus it will be a lot of code duplication. The same is related to QA or Prod separation. I do understand your security concerns and it is reasonable to separate QA and Prod deployment into separate pipelines and tune the access for the team members, but we still facing problem with one config per 1 git project, where the whole deployment happens trough our framework in single git project.
2nd problem is that we still need dynamic UI, as during deployment we need to pull all available artefact versions from the repository and allow selection of version before deployment. I hope that the link u published above to "Web-parameters" plugins could solve the problem, but we need POC to confirm that.

Regarding the pricing model - thank you for commenting on that, didn't know there is opened feature request. 
I don't have a ready plan for you, all I'm saying is that those times when you had dedicated set up of X amount machines dedicated to 1 master and multiple slaves which are up and running 24/7 is going away, a lot of businesses are moving to the cloud where you can dynamically request resource and kill them when u not need it anymore. So paying for 24/7 idle boxes is kind of expensive. Considering AWS Spot instances or what even better Spot Fleet you getting dynamic boxes for about ~92% cheaper than simple EC2 instance. for example, our current Jenkins set-up with infinitive scale and unlimited parallel build cost us just about 36$ p/m in terms of infrastructure pay, thanx to Spot Fleet. Our builds are spawned dynamically in the concurrent mode as many as there are jobs in the Q, therefore your slave's limitation because of price politics might bite you back in terms of features and market competition.
I would suggest charging for nr of builds or overall build time rather than agents concurrency.
I'm sure you have some sort of telemetry and you could estimate average usage of free TeamCity set-up with 3 agents, so you would know rough nr of builds per month made on those 3 agents or overall build time, so you can work from that information. for example, allow first 1000 builds free, and then charge for bundles (1k to 2k - X amount, 2k - 5k - XX amount and so on) but limiting concurrency in cloud set-up will definitely play against you. 

0
Comment actions Permalink

I'll try to focus on the "one pipeline per project" for this one. In terms of licensing, there is a wide array of factors to take into account. From a support perspective, we can only recommend sharing your use case in the tracker if you think it has some special factors on the issue I mentioned. We do have some telemetry from the installations that opt in into sending anonymous data, and we are using the cloud beta information as well to determine which factors make the most sense, but I don't really have much to share in that regard other than we are aware and working on it.

 

With regards to the "one pipeline file per project", it is technically correct to some extent. First, a project does not have to be determined by a VCS Root or repository, it is merely a hierarchical structure made to enable inheritance of a multitude of features. Of course, most of the time this correlates with projects, but does not have to be the case. You can have a single project with 10 different VCS Roots or a single VCS Root shared among 10 different projects, but only one of those repositories will be able to contain the "settings.kts" file. Branching is an option here but it has currently some limitations as to what can be included into a branch. We are working on removing them but they are due to our current architecture so getting rid of them requires a number of modifications in many places.

 

With this in mind, any given project can have:

-Multiple build configurations and build chains (the TeamCity term for what is commonly referred to as pipelines)

-Multiple subprojects with independent settings (which can be set in the same settings.kts file or each having their own). Each subproject will inherit all the VCS Roots from the parent, permissions and project features.

-Build templates which allow to reuse settings on a build configuration level (https://www.jetbrains.com/help/teamcity/build-configuration-template.html)

-Since Kotlin is a fully featured language, you can actually have code that will generate some or all of them based on shared code, be it on the same file, on other files or even on libraries.

-You can reuse the same files and use some variables we call "Context parameters", defined at the project level, which will be taken into account by your dsl code to generate (https://www.jetbrains.com/help/teamcity/kotlin-dsl.html#How+to+Read+Files+in+Kotlin+DSL)

And of course all of the above can be combined as desired. A common scenario here would be to have your project as the parent project containing the settings and VCS Roots, it contains build templates for your build settings, then you create subprojects for the different environments or requirements, that add build configurations an inherit from the build templates so all the settings can be shared without repeating them while retaining the separate setup.

 

About the dynamic UI, there is some valid use cases for sure, so feel free to test the plugin, and if it works for you that's fine. This said, TeamCity can still provide this functionality in a different fashion, as long as the artifacts have been generated by a TeamCity build (there might also be some workarounds to load artifacts from external places but this is more complex). In a common scenario, you will have some "Compile" build that will generate the artifacts. This artifacts can be published to teamcity itself as artifacts, or can be published to an external repository. Having this artifact generated by the build, it is possible to access the build that generated it, and under its "Actions" menu, select "promote". Every build that has this one as a dependency will be displayed as a promotion candidate. For example, if you have "deploy to staging" and "deploy to production", both having "compile" as a snapshot dependency, both will be displayed and  you can choose there where to deploy. Your deployment build can then pick up the artifact from TeamCity via an artifact dependency, if it's stored there, or if you have an external repository, the build could have published a parameter with the artifact name for the deployment to ensure that it picks the correct one.

 

Now, this obviously doesn't work for artifacts that weren't published by teamcity, and here the plugin would be more useful, although all this artifacts can still be added to teamcity (either the files or the references as mentioned above) with some scripting enabling the workflow mentioned above. Simply have once the "compile" build triggered via a script for each artifact, and either have it download the artifact and publish it as an artifact or simply set the parameter as mentioned above. Then it can become the proper build config while keeping everything together.

 

Hope this helps.

0

Please sign in to leave a comment.