"Failed to clean files from" only when builds run from queue

We have a Java web project with two build configurations in TeamCity: a continuous integration (CI) build and a quality assurance (QA) build. The CI build simply builds and runs unit tests whenever it has been at least 20 minutes since the latest check-in. The QA build builds, runs unit tests, upgrades QA databases and deploys the project onto QA servers on a set schedule. The QA build depends on the CI build so that it will not begin until a successful CI build has completed with the same source control snapshot.

Up until today, we have never had a problem. When QA ran at its scheduled time, CI ran first, completed and then QA ran (unless, of course, CI failed). However, today something strange happened. We had just made some upgrades to the build machine that significantly improved performance (build times have been cut in half) and then when we kicked off the QA build today, CI built and passed, QA started, and then QA failed with the error "Failed to clean files from C:\xxx\xxx." Now, we have seen this problem before when someone logged into the server and had a window open to that directory (Windows can't delete a directory when another user is browsing it). But this time nobody was logged in to the server. On a hunch, I removed the dependency between CI and QA and kicked off a QA build and it worked! It cleaned the directory without issue, built and passed. When I added the dependency back and CI ran first, QA failed with the error "Failed to clean files from C:\xxx\xxx."

After a lot of trail and error, I have determined that a build fails with that error if it is already queued up and waiting on another build to complete and then starts immediately after the other build completes (and, yes, our agent is set to never run builds simultaneously). I thought there was a way in TeamCity to configure a pause between queued builds, but I can't find it. Does anyone know where this setting is, if it exists, or how else I can solve this problem? I'm very convince that's what the problem is. After the performance upgrades, it's going so fast now, the previous build still has the directory locked when the next build attempts to build it. This is a problem...

Any help would be appreciated!

Comment actions Permalink

Anyone? Any ideas? All automated functions of our build machine are pretty much disabled now, rendering TeamCity halfway useless. Any insights would be appreciated.

Comment actions Permalink

Sorry for the delay in replying.

It looks like the reason for "Failed to clean files" error is still some process that is locking some files in the directory. My guess is that your CI build forks some process that is still running/locking files after the build finishes. And this became an issue after agent became faster so QA build starts faster.

I would recommend to deal with this in the scope of your build (ensure there are no processes left when the build script exits). This way you will be able to address it in every environment the script runs (e.g. also on developer's machines) and address it where you have sufficient knowledge to tackle it.

I'd start running Process Explorer just after CI build on agent to see what processes are still locking  files in the build's directory.

There are several related improvements in TeamCity that you might want to vote for: TW-4430, TW-8625.

As to your original question on whther there is an ability to start a new build after some timeout, there is no such ability for the time being. We have a request for it, but so far we are not convinced this is indeed a good feature to add.

Comment actions Permalink

Thanks, you were right. We had a process that was forking off. It wasn't part of the original build configuration; somebody made an unauthorized change and added a call to a target that is normally run in a more controlled manner but instead was being called directly from TeamCity. When I removed that target call from the build configuration, all was well again. Now, if only I could track which users made what changes to which builds...



Please sign in to leave a comment.