Teamcity parallel tests for C++

Hi! I've read https://www.jetbrains.com/help/teamcity/cloud/parallel-tests.html however I'm not clear if it is possible to create parallel testing for C++ projects, either natively or by creating custom tasks or custom execution configuration.

If it is not yet possible, it it planned, is there a timeline when this feature might be available?

0
2 comments
Hi Zoltan,

Currently, no built-in build runners for C++ are available in TeamCity. However, there is a plugin that enables CMake and GNU Make tools support: https://plugins.jetbrains.com/plugin/20316-cmake-and-gnu-make-tools-support.
Automatic execution of tests in parallel is available only for Maven, Gradle, and .NET build runners. When using other built-in build runners, or third-party build runners via plugins, the custom execution of paralleliized tests setup is required.

TeamCity makes this setup possible by providing special parameters. In general, custom parallelization of the tests works as follows:
1. TeamCity checks the duration of the tests and calculates which tests can be split between the batches. For TeamCity to be able to do that, it is required to make TeamCity aware of the tests by reporting them to TeamCity. That can be done by either:
• making your testing tool send service messages (https://www.jetbrains.com/help/teamcity/service-messages.html#Reporting+Tests) to stdout;
• importing an XML with the test results (https://www.jetbrains.com/help/teamcity/xml-report-processing.html).
Once the tests are reported to TeamCity and can be found on the 'Tests' tab of the build results page, you can proceed to the next step.
2. TeamCity will create a text file for each batch containing the names of the tests to exclude from the batch. This file can be referred in a testing tool/script to exclude the tests. TeamCity will run the testing tool/script on different agents and will provide each agent with a different file containing the names of the tests to exclude from the current batch. 

Let me give you an example of such implementation. It does not use any real testing tool but rather aims to give a general idea of how TeamCity can help parallelize the tests run by any custom tool. For this example, PowerShell build runner will be used. The test logic is irrelevant, so instead of running actual tests, the `sleep` command to simulate test execution will be used.

```
echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"

echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"

```

After the Parallel Tests build feature (https://www.jetbrains.com/help/teamcity/parallel-tests.html#Run+tests+in+parallel) is enabled, TeamCity will provide the `system.teamcity.build.parallelTests.excludesFile` parameter. The value of the parameter points to an automatically created file named `excludedTests.txt`. That file contains the names of the excluded from the current batch tests. In a case of 2 batches, the tests can be run in parallel on 2 agents. The content of the `excludedTests.txt` file for one batch is:

```
#version=1
#algorithm=duration
#current_batch=1
#total_batches=2
#creation_date=2022-08-18 15:38:28
#suite=
MyTestSuite.Class1

```

And for the other batch, it is:

```
#version=1
#algorithm=duration
#current_batch=2
#total_batches=2
#creation_date=2022-08-18 15:38:28
#suite=
MyTestSuite.Class2

```

Note how the file on each agent excludes different classes. The first file excludes `MyTestSuite.Class1`, and the second one - `MyTestSuite.Class2`. Still, both agents run both tests, so there is no parallelization yet. That's because the test tool (i.e. PowerShell script) doesn't make use of the `excludedTests.txt` file. So it is needed to make the test tool aware of it. This is where the `system.teamcity.build.parallelTests.excludesFile` parameter should be used. It can be used in the script to decide whether or not the test should run:

```
if ((cat %system.teamcity.build.parallelTests.excludesFile%) -notcontains "MyTestSuite.Class1") {
echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something here
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"
}

if ((cat %system.teamcity.build.parallelTests.excludesFile%) -notcontains "MyTestSuite.Class2") {
echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else here
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"
}

```

As a result, each agent running will only run the test from its batch, but not the test from another batch, which splits the tests between the agents, allowing running the tests in parallel.

Alternatively, the test tool/script can decide whether to run a test on the current agent by referring to the current batch number (and perhaps to the number of total batches) instead of reading the `excludedTests.txt` file. You can refer to the `%teamcity.build.parallelTests.currentBatch%` and `teamcity.build.parallelTests.totalBatches` parameter values in TeamCity for this. Example:

```
if (%teamcity.build.parallelTests.currentBatch% -eq 1) {
echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something here
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"
}

if (%teamcity.build.parallelTests.currentBatch% -eq 2) {
echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else here
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"
}

```

I hope this answers your question.

Best regards,
Anton
0

Please sign in to leave a comment.