How to handle several tests that are connceted to the same repository
Completed
So i managed to create several tests in teamcity, where each test is triggered by a different change in my git repository.
All is working fine regarding that, if i create a commit that changes just one folder, the specific test would run.
My issue comes when i want to use "Require status checks to pass before merging".
If the pull request only triggering test X and not test Y, i would only want to block the merge on test X. But i don't see any option to do so in git and teamcity.
Anyone might have any idea?
thanks!
Please sign in to leave a comment.
Hello Tamir!
TeamCity allows to alter how check is reported to GitHub by using the following build parameter:
Therefore, the options I could see so far are:
a) if each test is a separate configuration, you may report them all under the same name (parameter above), say, TeamCityTestCheck. Downside is that each configuration will overwrite the check status, so if a commit triggers multiple tests, for instance, test A and B, test A runs first and fails and test B runs second and is successful, in the end you`ll see a green check with no hint of A failure.
b) you would need an extra step to be taken; GitHub allows to report statuses via their API, so you could also report the tests that were not triggered as green. For instance, you could do a script which checks the latest revision to see what tests should have been triggered, and reports the rest of tests as auto-success, then add it to each test configuration.
c) this approach is a bit like b) option, but allows to do the trick without custom scripts. If you are currently running tests linked to VCS trigger and checkout rules (so that folder-specific commit triggers specific test), you could add a set of tests with inverted checkout rules (for instance, test A triggers on change at folder1, and configuration Anti_A would trigger on anything but folder1). These "anti" configurations would not do anything but report the status as success, with the help of above parameter - so that if the test is not triggered by a commit, you will still see the status correctly. Of course, this will require that for each pair of test and "anti" configuration, single status check name is used.
May I ask how exactly are the tests set up? Is there a separate build configuration for each test? With this, we might be able to suggest more specifically.
first of, thanks for the help!
my configuration is quite simple. i have a repository that contains tests for several groups, lets say A,B,C.
as i set it up now, each time someone changes the test file for A, or another A related folder, Teamcity will trigger the tests set up i have for A.
What i do in the teamcity tets is pretty much just call nosetest on my A python tests.
Option B sounds a bit complicated. C i dont quite understand to be honest, but it sounds a bit dirty.
A sounds the best, but i wonder if there's any way to overcome the issue you pointed there
About C, if i understand correctly, i would have Test A, test anti A, test B, test anti B. etc...
and Test A and anti A would have the same
githubContextAAnd to sum it up, my git repository would be dependant on githubContextA, githubContextB. etc..
?
on the same subject, im trying to search how exactly do i use "githubContext", but can't really find any example online.
edit: never mind, i think ive found it
I have to agree that option C is a bit of a hack; at the other hand, Commit Status Submitter functionality has been built just to report that the build is successful.
Yes, that is correct - essentially, for each context you run either test or an "anti" test configuration which will populate the status.
For the community reference, this build parameter (https://www.jetbrains.com/help/teamcity/configuring-build-parameters.html) could be set as follows: