TeamCity 6.0.1 agent - locking junit files

We're seeing this behavior occasionally with TeamCity 6.0.1 where it seems to lock junit files for use.

In our build step, we're using the XML reporting option, specifying junit and having these as the parameters:

%system.dxsi.view.pname%/vobs_dx_qa/si/target/docs/junit/data/dbsetup/*.xml
%system.dxsi.view.pname%/vobs_dx_qa/si/target/docs/junit/data/eclipse/*.xml

This is because TeamCity cannot detect these junit results using its normal mechanism so we need to specify them. At the start of the next build, we try to remove the ClearCase view which is located at %system.dxsi.view.pname% but we're unable to do so because the TeamCity agent still has some files in use:

D:\views\bsmith_dx_8.5_tc>rd /s/q vobs_dx_qa
vobs_dx_qa\si\target\docs\junit\data\eclipse\TEST-com.progress.dataxtend.si.ui.compare.adapter.TestResourceDiffNodeInput.xml - The process cannot access the file because it is being used by another process.
vobs_dx_qa\si\target\docs\junit\data\eclipse\TEST-com.progress.dataxtend.si.ui.views.TestDeleteAnalysisView.xml - The process cannot access the file because it is being used by another process.

Using NT SysInternal tools (handle.exe), I'm able to determine that it's the TeamCity agent process that is doing this.  This seems like a bug to me.

Any thoughts on how we can track down this defect?

I'm going to try to implement a workaround where I copy the junit results directory to the                        %system.teamcity.build.tempDir% directory and change the junit xml reporting to use this instead so it won't hold onto file handles when a new build tries to delete the view.

14 comments
Comment actions Permalink

I tried a simple ant script to copy the junit results to the buildtmp directory.  It worked the first time. It shows the results but when I run it again, it doesn't show the results.

Here's my ant file:

<project name="ci-master-build" default="ci-copy-dir" basedir=".">
    <target name="ci-copy-dir" description="Copy a directory to some destination">
        <fail message="Need to define dxsi.src.dir.to.copy for target ci-copy-dir" unless="dxsi.src.dir.to.copy"/>
        <fail message="Need to define dxsi.dest.dir.for.copy for target ci-copy-dir" unless="dxsi.dest.dir.for.copy"/>
                <delete dir="${dxsi.dest.dir.for.copy}" quiet="true"/>
                <copy preservelastmodified="true" todir="${dxsi.dest.dir.for.copy}" overwrite="true">
                   <fileset dir="${dxsi.src.dir.to.copy}"/>
                </copy>
    </target>
</project>


I setup a build with a single step to call the target ci-copy-dir.  The parameters I pass are:

<project name="ci-master-build" default="ci-test-runtime" basedir=".">

-Ddxsi.src.dir.to.copy=%system.dxsi.view.pname%/vobs_dx_qa/si/target/docs/junit -Ddxsi.dest.dir.for.copy=%system.teamcity.build.tempDir%/junit

The ant target just copies some junit results from the view to the buildtmp directory in order to work around an issue where the TC agent locks some files to prevent their deletion.

Under the XML Reporting, I specify Ant JUnit with the value of:
%system.teamcity.build.tempDir%/junit

So it works the first time I run it.  Then it doesn't work again even though the build deletes the report directory and copies it into it again.

Seems like a bug with the TC agent.

0
Comment actions Permalink

I found the issue with copying the junit result files to the agent buildtemp directory. The
issue had to do with my using preservelastmodified="true" in the copy task. It seems that although files were getting copied, TC seems to expect the file timestamp to be newer than the start of the monitoring period.

Is this behavior to be expected?  Once I remove the preservelastmodified attribute, then TC was able to detect the copied junit files.

My original issue of the TeamCity agent holding onto file handles of junit report files still exist.  Would like to know what info you need to track down the issue.

0
Comment actions Permalink

Xml test reporting plugin check files timestamp to avoid importing junit reports from older builds. To make importing work files timestamp should be newer the time of build start.

0
Comment actions Permalink

Hello, Bill,

Could you please enable debug logging on agent (uncomment the corresponding lines in <Build agent home>/conf/teamcity-agent-log4j.xml), try to reproduce the problem and attach build log and <Build agent home>/logs/teamcity-agent.log.

The whole handle.exe output for D:\views\bsmith_dx_8.5_tc\vobs_dx_qa\si\target\docs\junit\data could also be helpful.

0
Comment actions Permalink

We experience the same problem. Has this issue already been resolved?

0
Comment actions Permalink

Hello, Johannes,

We've made several fixes concerning this issue in TeamCity 6.5.2 and TeamCity 7.0.
What version are you running?

0
Comment actions Permalink

we are currently running 6.5 (build 17795) but I will update today to the latest 6.5.2 release.

0
Comment actions Permalink

Johnannes, please let me know if you face the problem after the upgrade.

0
Comment actions Permalink

Sure. Its just hard to reproduce. It happens from time to time.

0
Comment actions Permalink

Johannes, by the way we've just released teamCity 6.5.3, which has more fixes than 6.5.2. :)

0
Comment actions Permalink

Also after upgrading to 6.5.3 the problem still exists. What kind of information fo you need? Still I could not see any pattern when this error occurs.

0
Comment actions Permalink

From time to time we still facing this problem that the junit report file, which teamcity is monitoring for ant junit xml processing is locked. Only a restart of the agent solves the problem. I also enabled swabra together handle.exe to monitor this directory. But no helpful information is gathered anyhow. I ensured that no other detection of the junit tests took place before.

0
Comment actions Permalink

Johannes,

The next TeamCity 6.5.x release and TeamCity 7.0 EAP contain fixes.
Please, upgrade when possible.

0
Comment actions Permalink

We're planning to publish 6.5.4 in about a week.

0

Please sign in to leave a comment.