Cloud agent priority, to manage costs
I'm exploring TeamCity cloud profiles (on GCE) using preemptible instances.
My 18-agent server has a number of inter-dependent build configurations which are used together as preconditions to integrating a feature branch. These are triggered -
(a) by developers, when they are in a hurry
(b) by a nightly schedule, without auto integrating.
Cloud costs vary massively depending on the amount of horsepower I give the agents. So I'd like there to be 2 or 3 different cloud profiles / cloud images, which vary the CPU count (e.g. 2 to 12) and disk types (e.g. regular, SSD, local-ssd, ramdisk) of the agents.
(1) If it's a nightly schedule, or through choice, it would use the lowest spec cloud image configuration:
2 CPU, regular persistent disk, pre-emptible.
(2) If a developer triggers the tests, by default it uses some intermediate spec config:
6 CPU, local SSD, pre-emptible.
(3) If it's an emergency (critical hotfix), you can pick some "on steroids" config:
12+ CPU, ramdisk, not pre-emptible.
I've seen this post: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206886525-Setting-a-build-agent-s-priority
and the plugin: https://github.com/grundic/teamcity-agent-priority
I think this would allow me to create different classes of agents and choose which classes get prioritised for each build. However I don't think it would allow me to have different priorities for different invocations of a given build - whether via schedule or through user choice in the Run dialog.
Any suggestions?
Please sign in to leave a comment.
Hi Steve,
Thanks for the detailed explanation. Running custom builds you can choose on which agent to run them, but I can imagine that you would like to tackle this in a more automated way.
There should be a possibility without external plugins: https://confluence.jetbrains.com/display/TCD10/Assigning+Build+Configurations+to+Specific+Build+Agents
Using this, you can extract a build template of the build configurations, and apply it into different build configurations that use the different build agents. It would make it a bit more complex as "same" builds would be gathered in three different places, although on the other hand, it would also grant visibility to the different kinds of builds and deployment, granting an easy oversight on whether special builds have happened, and special control, on not just the agents, but other kinds of characteristics, or even the build process itself.
Hi Denis,
Thanks for the suggestions. Unfortunately we already have a complex set of configurations including extensive use of templates and dependencies to achieve parallelization of unit tests and other builds across agents to maximise performance. What you suggest would be good for a simple build but unimaginably complex to maintain with our setup.
Steve
Hi Steve,
I understand, unfortunately there is no direct way to prioritize different build agents. They can be selected manually on the runs you perform manually, but that's about it. Please open a feature request in our tracker for it: https://youtrack.jetbrains.com/issues/TW
Done. https://youtrack.jetbrains.com/issue/TW-54048
I've had some more thoughts on this subject, and now I've moved my Teamcity farm wholly to the cloud my priorities/requirements have become clearer.
Currently I have a set of about 18 builds which are now wholly linux based, so run on cheaper linux agents. However I have a few other less frequently used builds which require Windows or Mac agents. I'd like to be able to run 18 linux agents when that's what I need, but automatically downsize the linux pool to allow Windows or Mac agents to be started (I have 15 build agent licenses). Any suggestions?
My fallback is to have a dedicated cloud profile for Windows and Mac with just 1 agent in each pool, and 16 in the linux pool, but it's going to be less efficient, more expensive, and I'm going to need to figure out how to prioritise the linux pool over the win/mac ones (or prevent regular builds running on the win/mac ones).
also see https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360000120410-Agent-licenses-and-cloud-profiles
Hi Steve, thanks for the extra feedback, although I would suggest posting it in the issue in the tracker, so the devs will be able to actually see it and work with it.
Thanks Denis. Done - https://youtrack.jetbrains.com/issue/TW-54463