disable automatic test detection?
I am running ScalaTest tests in TeamCity and my test information gathered via the TeamCity built-in test detection is not accurate:
- tests are not grouped by Suite
- the times reported for tests are not accurate, all of my tests are reported as running in 1-3 ms (which is definitely not the case. the tests are run in parallel, so I imagine that is confusing the test reporter).
If I enable the "Build Feature" for "XML report processing", the correct information (tests are grouped in suites, and the test times are accurate) is shown in the tests tab alongside the previously reported data, so now my tests get double-counted. The configuration page for "XML report processing" has a message on it reading "Please make sure that tests are not detected automatically before using this feature."
Is there a way to disable tests being "detected automatically" so my tests tab will only report the data collected in "XML report processing". I have searched the docs and the forum and haven't found any way to do this.
Please sign in to leave a comment.
Hi Dara,
how are you running ScalaTest? The test reporting can go through two different means, as you have exprienced: the build runner, and the XML Report processing. If you are running your tests through a specific build runner, you would need to check which one it is. Scala is, as far as I know, not a default runner from us, so I suppose that you are either running a third party plugin, in which case you would need to request them a way to ignore tests, or running it with a specific runner from the ones we provide. In that case, we would need to know which runner you are using to say whether disabling test detection is an option, and if so, how to do it.
Hi Denis, thanks for your response.
I'm running my ScalaTest tests via sbt. I don't have a third party plugin for my test runner. I am running the tests in a "Command Line" build step with this format:
sbt "project <project name>" "<test target>"
Hi Dara,
For some reason, I totally missed that we do have the "Simple Build Tool (Scala)" runner. You might want to try that one first. It should integrate with our usual features nicely.
Other than that, there is a third possibility for test reporting I didn't mention. The test runner (in your case, sbt, not the build runner) might provide a TeamCity reporting method named "service messages" (More info here: https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity). If sbt is reporting the tests via this method, or through another report different to the one you load on the XML Report Processing, this might be the issue. You would need to ask the devs of sbt in this case whether they generate automatically a report (different to the one you load with the XML Report Processing), or whether they do the automatic teamcity report.
If none of that is the case and the sbt runner doesn't help, please send us a build log with the "Submit a request" button on top of the page for review.
Hi Denis. I have tried both "Command Line" and "Simple Build Tool (Scala)" as my runner and the test run times are identically inaccurate in either runner. I will submit a request as you suggest. Thank you.
Denis Lapuente is there a way to disable the automatic test detection in a Gradle runner? It's faulty when using parallel test execution.
Additionally, you may be interested in upvoting this request: https://youtrack.jetbrains.com/issue/TW-49874