parallelizing tests on teamcity

Answered

Hi,

we use TC heavily but due to our builds taking long time I was trying to figure out what can be changed to improve it. Appears that some of our Build Configuration were building the same stuff. For example, we have these build configs: [Windows, WindowsTests, WindowsTestsFull, Xbox, Android, iOS, Mac]. These run obviously on different machines and so do WindowsTests and WindowsTestsFull. WindowsTests or WindowsTestsFull each take over 20 minutes to complete and temporary build object files and binaries are like 5GB-10GB. It appears that in both of these configs they build pretty much the same stuff, which ads extra weight on our buildfarm and out of 20+ minutes of the total execution time 18 minutes or so are taken by tests that run sequentially and aren't using multiple cores.

Obvious solution is to 1) combine two build configs WindowsTests and WindowsTestsFull into one TC build config (this way we don't build the same stuff twice using two different machines) and then 2) run tests in parallel on that same machine that built them. Build steps for this combined WindowsTests config would be something like Build test1, build test2, build test3, ... build testN, then sequentially run these tests. Some of the tests take over 5 minutes and aren't CPU intensive and we would like to run them in parallel on the same PC. How can I do that, so that we could perhaps even see that green bar per each sub-config of our WindowsTests config? If not green bar then at least some way to do that  where logs can be checked separately?

 

 

12 comments
Comment actions Permalink

 I just tried to google and I cannot believe what I see. Tickets hanging for almost 10 years with high priority, multiple are assigned and still nothing is done about it, still no freaking support for parallel execution... because somebody in TC believes that it's not a good idea?! WTF?

I was absolutely sure that such obvious thing is possible out of the box, I guess my opinion was way off about TC.

 

What's possible other than switching from TC or writing something ourselves? In my case build has to be done on one PC (because of size and time), then if because of some dumb decision in TC I have to run these test on different machines to get parallel execution then I'm ok with that. If that's possible, how can I do that? If that's possible then while each of our multicore build agents run 5+ minute long tests will that build agent become unavailable for other builds because 1 core out of 8+ cores is waiting on IO?! This kind of dumb limitation make people turn away looking for alternatives.

0
Comment actions Permalink

Here's some visual approximation what I mean:

Right now it looks like this:

 

But if that "Release Mac Build" config had a build step that was "run multiple commands in parallel, complete all or fail" and then it would look something like that:

0
Comment actions Permalink

Hi Pablo,

I'm not entirely sure that I understand what you mean. You say "parallelizing tests" in the title. Parallel tests is a feature of the test runners themselves, and are supported where the test runner supports parallel tests. For example, JUnit has no native support, while NUnit does, and so does TeamCity respectively.

What you seem to describe is instead parallelizing different build steps (what you call tasks) inside a single build configuration. This is not directly possible as build steps are supposed to be executed in order. If you have different build steps that can be run in parallel, you should convert them into different build configurations which can be run in parallel, then set up snapshot dependencies so that they run in the appropriate order if required. This is our approach to different tasks which depend on each other.

Different build configurations can run in parallel in different build agents, as you have already mentioned. You can also set up multiple agents in a same device so that the results from one can be used by the other agents in the same device without having to transfer large loads of information.

0
Comment actions Permalink

@Pablo XX: Sorry, Pablo, you are losing the point! Look: https://blog.jetbrains.com/teamcity/2012/04/teamcity-build-dependencies-2/
Especially: "TeamCity Snapshot Dependencies" - so it is possible to do parallel steps in this way, but it requires:

a) extra build configurations 

or (likely and)

b) extra agents

So TC sales would not be interested in parallel steps within the same build configuration - as TC pricing is based on number of agents and build configurations.

If real parallel steps functionality developed - it means Jetbrain is losing money! Hopefully I am wrong.

0
Comment actions Permalink

@Alexopenmind:

I would like to point out that I had explained how it works in the response above yours. Multiple build configurations running in several agents is the way TeamCity is set up to run tasks in parallel. Also, with the Professional Edition, free of charge, we allow for up to 20 build configurations and 3 agents, which should cover most small projects.

So the "parallel steps" is already developed, implemented and working, and (to a certain extent) available for free. The admin just needs to adapt and move from multiple build steps, to multiple build configurations.

0
Comment actions Permalink

@Denis Lapuente

We have many cases where we would like to run build steps in parallel on the same agent. Often it's a good way to make full processing power of an agent. Sometimes you have build steps like uploading large debug symbols or packaging and uploading files to an FTP somewhere, that can be run while something else like unit tests are being run.

Sure you can install multiple agents on one host, but in a lot of cases creating multiple build configurations with snapshot dependencies becomes impracticable. This is especially true when you use branch builds functionality. It's convenient when you have one build configuration off one repo able to build any dev branch automatically from a naming pattern, but if you split this config up into half a dozen, it becomes a nightmare. It's also a problem when you want to branch the project itself for release versions.

I just end up having to manually create parallelisation using my own scripting.

0
Comment actions Permalink

Hi Anuradha,

The situation you describe in the first paragraph is almost by definition what TeamCity considers a build chain: A series of build configurations tied with snapshot dependencies that do independent things and can be run in parallel.

Creating snapshot dependencies should take care of the branches by itself. Once you set up a snapshot dependency, triggering a build on a specific branch will carry that branch throughout the chain (in all builds that have a VCS set up, even through multiple different VCSs, if the branch exists in them).

Maintaining the different builds and their dependencies does have some maintenance overhead, but "manually create parallelisation using own scripting" also has quite a bit, if not more, while doing the maintenance for the dependencies allows to:
-parallelize tasks not just on a same agent's machine but also on multiple agents, be it on the same machine or on multiple.
-have a more granuralized overview of the different components that form part of a complex build.
-reuse components in multiple builds
-reuse components from many other builds, not just those on the chain

If the steps are independent to each other (or just have common dependencies), and you want to run them in parallel, a build configuration has a very small overhead in comparison to a build step and offers parallelization along all the advantages mentioned above.

This said, we provide scripting runners precisely for the situations where teamcity's functionality doesn't match your scenario. If you consider it easier or better for your workflow to have a single build with scripts performing all tasks rather than multiple builds, please feel free to do so.

Also, please consider voting and watching this issue: https://youtrack.jetbrains.com/issue/TW-13849. Feel free to join the discussion adding any comment you might consider relevant.

0
Comment actions Permalink

@Denis Lapuente The use case I have for wanting parallelised builds (and this is what I suspect other that ask for it have) in one configuration can't really be addressed by multiple build  configurations though:

  1. We want to make full use of the CPU power of modern physical agents. A decent physical machine will be using less than a quarter of its capacity for most of the time during a one hour build of our product. Splitting large legacy builds across multiple build configurations and agents will require a lot of resources. If we had a well architectured product that could build on Linux and use, say Docker containers, then it would be better to spin up (or down) lots of tiny VMs to meet demand. But with a product that's been around a long time and builds on Windows, this is not so easy.
  2. We need to support several previous releases, with dev branch builds on all of them. If a single release required half a dozen smaller build configurations, multiplied out to half a dozen supported releases, all with dev branches, the number of agents required to get decent throughput is not practical. We already have close to 100 build configurations to manage as it is.

We script our own parallel building solutions, but it would be very useful if TeamCity had the functionality.

0
Comment actions Permalink

Hi Anuradha,

Thanks for the comments, but I'd like to point out a couple things:
-Docker also works on Windows and we now have support for it: https://blog.jetbrains.com/teamcity/2017/10/teamcity-docker-images-for-windows/ and https://confluence.jetbrains.com/display/TCD10/Integrating+TeamCity+with+Docker
-Other Windows-capable services, such as VMWare, are also supported and provide very similar functionality.
-Parallelization works in the extent of which the tools that run the builds support it. We don't restrict tools that support it from using it. This obviously extends to custom scripts such as what you are using.

-If you need to maintain a very large amount of build configurations for different branches, products and versions, you can use versioned settings: https://confluence.jetbrains.com/display/TCD10/Storing+Project+Settings+in+Version+Control, which might be more helpful. Older versions can be stored in the versioned settings if you need, so that you can simply modify them at need.

Also we don't deny its utility, that's why it works with the different build configurations running in multiple agents system.

As I already mentioned, please refer to the issue in the tracker. Any discussion towards its request should go there.

0
Comment actions Permalink

Hi Dennis,

Running steps in parallel is always in high demand and must be as easy as in Jenkins, e.g.:

  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }

No non-obvious and not immediately visible snapshot dependencies - in one quick look I see the whole parallel graph of executions.

It's quite strange that you are trying so hard not to see this obvious lack of very useful functionality in TC.

0
Comment actions Permalink

Dear Serguei,

If you re-read the conversation, I've already pointed out that we already have a feature request to implement "it", have linked to the request, and have spent the rest of the time explaining how the requirement can be approached with the current tools already implemented, so your personal attack is uncalled for.

Also, the functionality is available in TeamCity, as I've already explained through the rest of the topic. Maybe not as "easy" as in jenkins, but available. The Kotlin DSL, with proper enough handling, will even let you have a similar syntax. Change "parallel steps" for "build configurations", and you can achieve basically the same rather easily, setting up a chain. Do not translate jenkins "steps" to TeamCity's "steps", because they aren't the same base concept.

If you have further comments about the feature itself, and not on how to achieve the specific task with TeamCity, please forward them to the feature request in the tracker (https://youtrack.jetbrains.com/issue/TW-13849). Discussion about its merits and drawbacks will happen there with the developers.

0
Comment actions Permalink

Hi Denis,

Let me just to quote a guy Pablo XX who started this topic:
"I just tried to google and I cannot believe what I see. Tickets hanging for almost 10 years with high priority, multiple are assigned and still nothing is done about it, still no freaking support for parallel execution... because somebody in TC believes that it's not a good idea?! WTF?"

That's basically it. :-)

2

Please sign in to leave a comment.