My Windows agent keeps an open file handle on maven-core-3.0.3.jar.

Hi,

I'm having a bit of trouble with my Windows build agent.  I have a module in my project that is set up to get all of my build tools from my local Nexus repository (maven, jdk, installer builders, etc.).  Most of my build configurations consist of two steps:

1) Use the built in Maven and Java to execute 'clean install' on my build tools module which unpacks the whole build environment for my project onto the agent.
2) Use the Maven and JDK from the unpacked build environment to run the build.

After the second step, a sub-process of the windows build agent (agent > java.exe > java.exe) has an open file handle on the maven-core-3.0.3.jar file from the Maven install (my custom unpacked one) used to run the build step.  I don't have any problems when I run the build on a Linux agent or when I run the build manually from my IDE (in Windows).  I want to clean and re-install my build environment on all agents nightly, but the open file handle causes the build to fail on my windows agent because it can't delete that file.

I have TeamCity build 17985 and did a clean re-install of my agent.

It could be Maven or a misbehaving plugin causing my problems, but I'd like to figure out why my Linux agent works while my Windows agent doesn't before I really start to dig into it.  Is there a difference in the way Maven is invoked / killed on each platform?

I've tried everything I can think of to find the cause for now.  I tried profiling the agent with YourKit.  I connected to the same PID that ProcessExplorer reports as holding the open file handle (YourKit lists it as AgentMain), but the YourKit inspections don't find anything and I can't find anything by hand.  Any suggestions?



Ryan
5 comments
Comment actions Permalink

Hi Ryan,

Normally the agent doesn't have any child processes except those executed during a build. If any child processes are still present in memory after a build, they all must have been created by that build. So my guess is that those two java.exe below the agent process are Maven executions. But this is quite strange.

Can I see the build log?

I also suggest you trying Swabra (http://confluence.jetbrains.net/display/TCD6/Adding+Swabra+as+a+Build+Feature), which is designed for cleaning up files and killing processes left by builds.

0
Comment actions Permalink

Hi Sergey,

I'm attaching some (debug level) logs for two agents; one windows, one linux.  I stopped both agents, moved old log files, started both agents and ran the same build configuration twice in a row on both agents.  I've included the raw logs from the agents' log directory and the logs that can be downloaded for each build using the web interface.  The build configuration I used has two steps:

1) clean package my build tools module.  This deletes the dir %system.teamcity.agent.dir%/.build-tools-itma, retrieves artifacts for the JDK and Maven from my Maven repo (or Nexus proxy if needed).  Those artifacts are unpacked into %system.teamcity.agent.dir%/.build-tools-itma/jdk and %system.teamcity.agent.dir%/.build-tools-itma/maven.  This is done using the built in JRE and Maven (or whatever JDK happens to be on the build agent).

2) clean package my project parent module.  This uses the JDK and Maven that were unpacked in step 1.  I've stripped it down to almost nothing, so it only builds my project parent and organization parent.

Based on timing (~90s for step one vs ~1s for step two), I'm fairly sure the file handle is getting opened in step 2.  I've also included a screenshot of the java command that ProcessExplorer shows for the PID that has a reference to the open file handle.  I couldn't figure out how to get plain text for it, so I apologize for making you squint :-)

I forgot to delete the .build-tools-itma dir on my linux agent before the first run.  That's the reason linux log for the first run is much bigger than the windows log for the first run.  The complaints at the end about missing artifacts are because I modified an existing build configuration by pointing it to a different POM.

My underlying goal is to get a consistent build environment that I can easily re-create for release builds.  I didn't want to put the JDK and Maven into SVN and putting them into Nexus seemed like a decent compromise.  I'm not married to the idea and if my approach is misguided you're welcome to say so.

Let me know if you'd like any more information,
Ryan



Attachment(s):
tc-agent-logs.rar.zip
0
Comment actions Permalink

Hi Ryan,

Looks like I've found out the cause. Before running Maven the agent looks into Maven's jars to determine its version using URLClassLoader, which frees file handles only when being garbage collected. On Linux it seems to happen earlier than on Windows. Is it really critical for you?

0
Comment actions Permalink

Hi,

Yes, you've solved it!  The file handles are very long lived on my agents though.  The linux agent works because linux allows the deletion (unlinking) of open files.  The agent process holds a reference to the deleted file.

After one run of my build configuration:

lsof -p 7963 | grep itma java    7963 teamcity  mem    REG    252,1   556983   539744 /opt/teamcity-agent/.build-tools-itma/maven/lib/maven-core-3.0.3.jar java    7963 teamcity  234r   REG    252,1   556983   539744 /opt/teamcity-agent/.build-tools-itma/maven/lib/maven-core-3.0.3.jar



Subsequent runs:

lsof -p 7963 | grep itma

java    7963 teamcity  DEL    REG    252,1            539744 /opt/teamcity-agent/.build-tools-itma/maven/lib/maven-core-3.0.3.jar java    7963 teamcity  234r   REG    252,1   556983   539744 /opt/teamcity-agent/.build-tools-itma/maven/lib/maven-core-3.0.3.jar (deleted)



It's not too critical.  Knowing the cause is the most important thing.  I have very few builds and, since I'm using the free version of TC, only 3 agents.  Using linux agents only is a perfectly fine workaround for me.

I was also able to find a bug report / RFE (http://bugs.sun.com/view_bug.do?bug_id=4950148) for the underlying issue which has been improved in Java 7 (http://blogs.oracle.com/CoreJavaTechTips/entry/closing_a_urlclassloader).

Thank you very much for your help.

Ryan
0

Please sign in to leave a comment.