Yesterday, I upgraded our TeamCity server from 6.5 to 7.0. All of our MSTest biuld steps have now stopped working.
I have specified 10 test assemblies to run, for a total of around 400 tests. What I have found is that if I reduce the number of assemblies, it will run, but the more assemblies that add, the slower the step runs, until it finally stops working.
Our build times prior to the upgrade were around 3 minutes. With a couple assemblies in the MSTest step, it will run, but take twice the time. It doesn't appear to matter which assemblies I add and remove, only the number which are present.
Has anybody else encoutered a similar issue?
With all 10 assemblies specified, after about 20 minutes the MSTest step will fail with the following output:
[10:50:54][Step 2/3] Starting: D:\TeamCity\buildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe #TeamCityImplicit
[10:50:54][Step 2/3] in directory: D:\TeamCity\buildAgent\work\47d0ecb36bbf7ade
[11:08:08][Step 2/3] 2012-03-01 11:08:08,804 [1484] ERROR JetBrains.BuildServer.NAntLoggers.NUnitLauncher2 - Exception of type 'System.OutOfMemoryException' was thrown.
[11:08:08][Step 2/3] System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.Wildcard.IsMatch(String str) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\Wildcard.cs:line 80
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.AntPatternState.Enter(String component, MatchResult& match) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\AntPatternState.cs:line 96
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.Any(AntPatternState[] state, String component, AnyPredicate predicate, AntPatternState[]& newState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 89
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 138
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFiles(IFileSystem fs, IDirectoryEntry root, String[] includes, String[] excludes) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 27
[11:08:08][Step 2/3] at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFiles(String root, String[] includes, String[] excludes) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 19
[11:08:08][Step 2/3] at JetBrains.BuildServer.CommonLoggers.Impl.DirecrotyWildcardResolverImpl.FindFiles(String[] includeRules, String[] excludeRules) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\CommonLoggers\src\Impl\DirecrotyWildcardResolverImpl.cs:line 23
[11:08:08][Step 2/3] at JetBrains.TeamCity.MSTestSupport.Executable.MSTestTargetGenerator.GenerateMSTestCommand(IMSTestExeArguments msTestArgs, String resultsFile) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestTargetGenerator.cs:line 68
[11:08:08][Step 2/3] at JetBrains.TeamCity.MSTestSupport.Executable.MSTestTargetGenerator.GenerateTargetContent(IMSTestExeArguments mstestArgs) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestTargetGenerator.cs:line 47
[11:08:08][Step 2/3] at JetBrains.TeamCity.MSTestSupport.Executable.MSTestExeTestRunnerFactory.CreateRunner(ITestRunArguments args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestExeTestRunnerFactory.cs:line 29
[11:08:08][Step 2/3] at JetBrains.BuildServer.NAntLoggers.RunnerFactory.FindTestRunner(ITestRunArguments arguments) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\RunnerFactory.cs:line 45
[11:08:08][Step 2/3] at JetBrains.BuildServer.NAntLoggers.RunnerFactory.CreateTestRunner(ITestRunArguments arguments) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\RunnerFactory.cs:line 30
[11:08:08][Step 2/3] at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run2(String[] args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\NUnitLauncher2.cs:line 98
[11:08:08][Step 2/3] at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run(String[] args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\NUnitLauncher2.cs:line 54
[11:08:08][Step 2/3] Exception of type 'System.OutOfMemoryException' was thrown. at JetBrains.TeamCity.Utils.FsScanner.Wildcard.IsMatch(String str) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\Wildcard.cs:line 80 at JetBrains.TeamCity.Utils.FsScanner.AntPatternState.Enter(String component, MatchResult& match) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\AntPatternState.cs:line 96 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.Any(AntPatternState[] state, String component, AnyPredicate predicate, AntPatternState[]& newState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 89 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 138 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFilesRec(IDirectoryEntry directory, ArrayList result, AntPatternState[] includeState, AntPatternState[] excludeState) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 148 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFiles(IFileSystem fs, IDirectoryEntry root, String[] includes, String[] excludes) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 27 at JetBrains.TeamCity.Utils.FsScanner.DirectoryScanner.FindFiles(String root, String[] includes, String[] excludes) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\Utils\src\FsScanner\DirectoryScanner.cs:line 19 at JetBrains.BuildServer.CommonLoggers.Impl.DirecrotyWildcardResolverImpl.FindFiles(String[] includeRules, String[] excludeRules) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\CommonLoggers\src\Impl\DirecrotyWildcardResolverImpl.cs:line 23 at JetBrains.TeamCity.MSTestSupport.Executable.MSTestTargetGenerator.GenerateMSTestCommand(IMSTestExeArguments msTestArgs, String resultsFile) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestTargetGenerator.cs:line 68 at JetBrains.TeamCity.MSTestSupport.Executable.MSTestTargetGenerator.GenerateTargetContent(IMSTestExeArguments mstestArgs) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestTargetGenerator.cs:line 47 at JetBrains.TeamCity.MSTestSupport.Executable.MSTestExeTestRunnerFactory.CreateRunner(ITestRunArguments args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\MSTestSupport\src\Executable\MSTestExeTestRunnerFactory.cs:line 29 at JetBrains.BuildServer.NAntLoggers.RunnerFactory.FindTestRunner(ITestRunArguments arguments) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\RunnerFactory.cs:line 45 at JetBrains.BuildServer.NAntLoggers.RunnerFactory.CreateTestRunner(ITestRunArguments arguments) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\RunnerFactory.cs:line 30 at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run2(String[] args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\NUnitLauncher2.cs:line 98 at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run(String[] args) in c:\BuildAgent\work\dc6fdb6ba3889d18\src\NUnitBootstrap\src\NUnitLauncher2.cs:line 54
[11:08:10][Step 2/3] Process exited with code -1
This issue is addressed below:
http://youtrack.jetbrains.com/issue/TW-20482