We’ve got a problem with our NUnit tests hanging periodically when run in TeamCity and I’ve run out of ideas.
- This is running on an Azure VM. I mention this first, because it is the first time I've tried this. Normally I run TeamCity on physical hardware or Hypervisors.
- The configuration database is a local SQL Server Express instance on the VM. So no external network call.
- It hangs about 1 out of every 20 builds.
- In my investigation, I’ve looked at which unit tests it hangs on and there is no obvious pattern. Looking at 10 tests, they are from different parts of the system, some are complex, and some are incredibly simple.
- There is nothing in the build log. The last line always looks something like: [01:37:17] : [Ce.Tests.UnitTests.dll] Ce.Tests.UnitTests.Business.Budget.TestBudgetBL.BeforeSaveEntityValidation_BudgetLineItemBudgetQtyWithinLimit_DoesNotThrowException (running for 7h:28m:31s)
- We’ve never seen this behavior when running tests locally
- Failing the build and re-adding it will always make the build pass
- TeamCity 9.1.1 (build 37059)
- There is a single build agent
I’ve gotten to the point where I’ve set “Fail build if it runs longer than specified limit in minutes to 30 minutes”, so at least it doesn’t prevent other builds from running. But that doesn’t fix the root issue.
Anybody have any suggestions on how to diagnose this issue further?