NUnit tests hang the build
Answered
We’ve got a problem with our NUnit tests hanging periodically when run in TeamCity and I’ve run out of ideas.
Details:
- This is running on an Azure VM. I mention this first, because it is the first time I've tried this. Normally I run TeamCity on physical hardware or Hypervisors.
- The configuration database is a local SQL Server Express instance on the VM. So no external network call.
- It hangs about 1 out of every 20 builds.
- In my investigation, I’ve looked at which unit tests it hangs on and there is no obvious pattern. Looking at 10 tests, they are from different parts of the system, some are complex, and some are incredibly simple.
- There is nothing in the build log. The last line always looks something like: [01:37:17] : [Ce.Tests.UnitTests.dll] Ce.Tests.UnitTests.Business.Budget.TestBudgetBL.BeforeSaveEntityValidation_BudgetLineItemBudgetQtyWithinLimit_DoesNotThrowException (running for 7h:28m:31s)
- We’ve never seen this behavior when running tests locally
- Failing the build and re-adding it will always make the build pass
- TeamCity 9.1.1 (build 37059)
- There is a single build agent
I’ve gotten to the point where I’ve set “Fail build if it runs longer than specified limit in minutes to 30 minutes”, so at least it doesn’t prevent other builds from running. But that doesn’t fix the root issue.
Anybody have any suggestions on how to diagnose this issue further?
Please sign in to leave a comment.
Hello Randar,
Did you try to run the same build via the command line in the working directory on the TeamCity agent machine under the same user that the agent is running?Please follow the guidelines. Does the build hang intermittently?
If the build runs successfully on the agent via console, but still hangs in TeamCity, then please attach complete build log and teamcity-agent.log from the agent where this build executed. Also please take several agent thread dumps while it's in the hanging state. You can take thread dumps from agent details page.
Unfortunately, since it fails intermittently, I can't say how many times to run it locally before I can say "it behaves differently".
I've turned the timeout off so that the hang will not clear on it's own. When it hangs next, I will include agent thread dumps and logs.
Thank you for details, waiting for the update from your side.
I think you can consider this issue resolved. It ends up my statement of "We’ve never seen this behavior when running tests locally" was not true. It's just not very often. Once I reproduced it locally, I was able to see that the problem was in some setup code that ran on every unit test, but had a rare and infrequent case that could never return. So the problem was not on the TeamCity side after all.
Thanks for the help.
Thank you for the update! Glad that you found the root of the problem.