Test migration from VSTest to MTP
Hello,
I'm trying to migrate our automated tests from VSTest to MTP, which was introduced with .net10.
For that I changed the outputtype of all testprojects to “exe”, added “<EnableMSTestRunner>true</EnableMSTestRunner>” to the Directory.Build.props file responsible for the tests and added an global.json with following content
{ "test": { "runner": "Microsoft.Testing.Platform" } }
According to this Migration Guide from Microsoft. After those changes the unit test execution fails. I tried to execute the command stated in the build log locally and on the build agent server. In both cases commandline throws errors regarding the ";verbosity=normal" for the logger argument:
"C:\Program Files\dotnet\dotnet.exe" test <path to dll> /logger:logger://teamcity /TestAdapterPath:D:\BuildAgents\Agent3\plugins\dotnet\tools\vstest15 /logger:console;verbosity=normal --Filter ClassName~UnitTest
Before making these changes the command looked like this and was executed as expected (besides, that since migration to .net10 test execution through this way got muuuuch slower than with .net8):
"C:\Program Files\dotnet\dotnet.exe" test <path to dll> /logger:logger://teamcity /TestAdapterPath:D:\BuildAgents\Agent3\plugins\dotnet\tools\vstest15 /logger:console;verbosity=normal /TestCaseFilter:"ClassName~UnitTest"
I'm not sure where this comes from. If I change the log verbosity inside the build step this will be ignored and the command looks the same.
I'm also not sure where the TestAdapterPath-Argument comes from, but it looks quite outdated since vstest15 points to Visual Studio 2017 which we are not using anymore for years. I'm not able to find the location where this is configured and would like to get rid of it, but on command line this causes no issues.
I migrated the TestCaseFilter-Argument to the more general Filter-Argument since it was vstest-specific.
The build step configuration looks like this (there is only this one) and did not change except for the command line parameter for class filtering:
Step 1: Execute Unit Tests
Runner type: .NET (Provides .NET toolchain support for .NET projects)
Execute: If all previous steps finished successfully
Command: test
Paths: Tests\UnitIntegrationTests\**bin**Debug\*Tests.dll
Command line parameters: --filter "ClassName~UnitTest"
Code Coverage: JetBrains dotCover
Filters: <some of our assemblies listed>
Docker Settings
Docker Image: unset
Run build step under JetBrains dotMemory Unit: false
Path to dotMemoryUnit.exe: <empty>
Path for storing workspaces: <empty>
Do you have any ideas where this adendum for the logger argument comes from and how to remove it?
How can I remove the TestAdapterPath Argument, since it is outdated?
Otherwise any additional hints what could be wrong for the migration to MTP?
Thank you & best regards
Daniel
Please sign in to leave a comment.
Hi Daniel,
This is a known limitation of the current TeamCity .NET runner, rather than a setting that can be changed in the build step.
When the
testcommand is selected, the runner generates a VSTest-oriented command line and adds TeamCity’s VSTest integration arguments, including/logger:logger://teamcity,/TestAdapterPath, and the console logger. These arguments are not compatible with Microsoft Testing Platform (MTP).Native MTP support in the .NET runner is tracked as TW-90891. There is no fix version assigned yet.
In the meantime, you can the following work around:
Use the .NET runner with Command: custom. Leave Executables empty and specify, for example:
test --solution "solution.slnx" --configuration Debug --results-directory "testreport" -- --report-trx --filter “ClassName~UnitTest”Then add the XML Report Processing (https://www.jetbrains.com/help/teamcity/xml-report-processing.html) build feature with:
Report type: TRXMonitoring rules: testreport/*.trxPlease also ensure the test projects include the
Microsoft.Testing.Extensions.TrxReportpackage required to generate TRX reports.Since your current step also uses dotCover, please note that its configuration may need to be adjusted and tested separately after switching away from the standard
testcommandHi Tom,
thank you for your response.
I already found the mentioned issue on my own on friday and commented it accordingly, but your answer clarifies that MTP being not supported by the runner is indeed the actual problem.
I hope that this issue now gets high(er) priority since MTP is now released for nearly 2 years and in the passt few days several comments have been added that people want/need this feature.
I'll try out the Custom Command approach, but this seems to come with some tradeoffs:
- at least missing live testing reporting
- the need to rework all the testing with adding additional extensions to the projects and giving the build steps a revision
Some background info why we want to try or need MTP:
After switchting to .net10 the .Net runner with test command resulted in extremely slow testing on TeamCity. The test build configurtations derive from the same templates as the build configurations that currently run smoothly with .net8, so there was no change in configuration itself. The only difference I could spot is, that for .net8 we have an installed VS2022 instance on the machine, which should not have any effects since we're using the test command insted of vstest command, while for .net10 we're using the VS2026 Buildtools.
Just an example how critical the issue is:
- we have ~3000 unit tests which ran within ~35min, with .net10 the same unit test build configuration run in our 1h timeout with only ~1000 test being executed → ~6 times slower
- we have ~800 integration & ~550 functional tests which ran within ~25min each, with .net10 the same integration test build configurations took ~50min each → ~2 times slower
Without this issue no need to use/try MTP would exist on our side. So hopefully with MTP everything is fast again (or even a bit faster from what Microsoft is promoting)
Best regards
Daniel
TW-90891 (https://youtrack.jetbrains.com/issue/TW-90891) currently has no committed timeline. Votes, comments, and concrete use cases such as yours are reviewed by the team responsible for prioritization, although I cannot promise when native MTP support will be implemented.
The significant slowdown after moving to .NET 10 appears to be a separate issue from the .NET runner’s lack of MTP support. I could not find an existing TeamCity issue matching this performance regression, so it would be worth investigating separately.
Could you please provide:
1. The exact .NET SDK versions used by the .NET 8 and .NET 10 configurations, ideally together with the output of `dotnet --info` from the relevant agents.
2. Full build logs from comparable .NET 8 and .NET 10 test runs.
3. The result of running the equivalent `dotnet test` command directly on the affected agent, outside the TeamCity .NET runner.
The third comparison will help us determine whether the slowdown originates in the TeamCity runner integration or in the underlying .NET 10 test execution environment.
In the meantime, the Custom Command approach remains the available workaround for running the tests through MTP, with the test-reporting and project-configuration trade-offs you mentioned. Please also let us know whether it restores the expected test performance.
Hello Tom,
1. We are just specifying the TargetFramework in form of net8.0/10.0 everything else we let visual studio / dotnet decide. the “dotnet --info” output of the build agent machine:
So there seems no .net8 SDK directly being installed anymore, testing on net8.0 still works as expected with the times told previously.
2. I collected some logfiles that are comparable as possible, but they're quite big (4 files ~ 9mb) and I don't want to share them publicly. How can I provide them?
3. Unfortunately I'm not able to do this. I tried to run a history build on TeamCity with a CommandLine Step containing the command extracted from the build logs but it's not accepted and fails. It would be a huge effort to revert everything to the state where I ran into the problem before switching the testplatform (and I'm not even sure if the commandline approach would then work).
4. Test performance update:
Since MTP is doing parallelization at assembly level as default it's a not straight forward comparison, but so far are the current times:
Unit Tests: ~8min (~35min / 1h timeout)
Integration Tests: ~2min (~25min / ~50min)
Function Tests: ~6min (~25min / ~50min)
So parallelization speeds everything massively up but from my point of view the (overhead) slowdown from the vstest execution is gone as well, because the biggest test assembly takes approximately 5min if only these tests are executed locally in visual studio. So combined with some assemblies running previously before this one is taken and a bit of overhead these ~8min don't look like being affected by a slowdown.
I did also a test run with disabled parallelization and it took ~18min. This may come from MTP itself but it's also possible that the change from test discovery based on name to discovery based on testcategory may have accelerated execution.
So overall we have now much faster test execution with the MTP workaround. There's still one strange thing: Teamcity now says that ~2560 unit tests are executed, but the summary of the dotnet test command in build log itself says ~3640 Tests. The difference may be caused by different counting behavior regarding data tests wich use data row or dynamic data. Anyway overall the same amount of tests should be run.
Since no .NET 8 SDK is installed and no SDK version is pinned, both the `net8.0` and `net10.0` test projects are being processed by the .NET 10.0.400 SDK. The `net8.0` tests then run against the installed .NET 8 runtime. Therefore, the original comparison was primarily between the target frameworks and their test execution paths, rather than between two installed SDK versions.
There is no need to restore the previous configuration solely to reproduce the VSTest slowdown. Your MTP results show that the workaround is performing well, even allowing for assembly-level parallelization and the change in test filtering.
Regarding the difference between approximately 3,640 tests reported by `dotnet test` and 2,560 displayed by TeamCity: your assumption about data-driven tests is plausible.
TeamCity generally counts multiple invocations with the same test identity as one test within a build, while the MTP summary may count every data-row invocation separately.
To verify this, please also include the generated `.trx` report files and build log files. We can compare the individual results and their test names/IDs with what TeamCity imported and determine whether this is expected deduplication or whether some results are not being reported.
For the files you do not want to share publicly, please upload files via https://uploads.jetbrains.com/.
and let us know the exact id after the upload.
Please, excuse the delay. Those were quite busy days!
I uploaded all data under following Upload id: 2026_09_02_AuyFe5PLCwwLzM1WpmLzCd (file: logs_net8_vs_net10.zip)
The 4 log files with net8 and net10 suffix are quite comparable teamcity buildlogs from the unit and functional tests running under mstest/vstest. Please note that, the .net10 unittest ran into our timeout limit of 1 hour therefore it's not showing all tests compared to the net8 version. Also between those builds development progressed so in the net8 log there may be tests noted that are not showing up in the net10 version. But there should be enough examples for analysing
Inside the subfolder is the teamcity buildlog with the according trx-files of our current state of the MTP-testing with the testcount difference for our unittests. In the meanwhile numbers slightly increased and teamcity says 2686 executed tests, the buildlog says 3790.
Just one more remark: The workaround with trx-files has also the disadvantage that we can't use fail-conditions based on the appearance of log entries inside the buildlog. Those entries are now in the according trx file. We have at least one test which relies on this mechanic. For the current state I'll create a workaround.