TeamCity Build Failing: Cannot delete file....

One of our builds get the Cannot delete file and only way to fix it is by going onto the buildAgent box and deleting the directory it mentions.



[16:21:18]: [Updating sources] Repository sources transferred: 3413.52Mb total
[16:21:18]: [Updating sources] Average transfer speed: 8.28Mb per second
[16:21:21]: [Updating sources] Removing /opt/choicestream/apps/buildAgent/work/b641db99760b3607
[16:21:22]: [Updating sources] Error while applying patch: Cannot delete file /opt/choicestream/apps/buildAgent/work/b641db99760b3607


Have tried changing the directory to have 777 permissions and the build still fails.  Any ideas?




34 comments
Comment actions Permalink

Hello Jason,

  Please make sure build agent has full access to /opt/choicestream/apps/buildAgent/work/ directory, i.e. it can create/delete files in them.
  You should change access rights to this directory, not to directory being deleted.

  Hope this helps,
  KIR

0
Comment actions Permalink

Hi,

I tried what you suggested and the build again ran into this problem:

[08:35:41]: Checking for changes
[08:35:44]: Clearing temporary directory: /opt/choicestream/apps/buildAgent/temp/buildTmp
[08:35:44]: Checkout directory: /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
[08:35:44]: Updating sources (5m:45s)
[08:35:45]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[08:35:45]: [Updating sources] Transferring cached clean patch for VCS root: CSA Build Root for GCC 64-Bit Build on 64-Bit Reco Server (Mainline)
[08:37:45]: [Updating sources] Transferring repository sources: 1002.0Mb so far...
[08:39:45]: [Updating sources] Transferring repository sources: 2333.73Mb so far...
[08:41:23]: [Updating sources] Building incremental patch over the cached patch
[08:41:25]: [Updating sources] Repository sources transferred: 3366.73Mb total
[08:41:25]: [Updating sources] Average transfer speed: 9.88Mb per second
[08:41:28]: [Updating sources] Removing /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
[08:41:29]: [Updating sources] Error while applying patch: Cannot delete file /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
[08:41:30]: Build finished
0
Comment actions Permalink

I have the same problem here on Linux also. All folders are created by
same user, a special teamcity user, and sometimes folder cannot be
deleted, so I have to do it manually.

Le 23/09/2009 17:22, Jason Perry a ecrit :

Hi,

I tried what you suggested and the build again ran into this problem:

: Checking for changes
: Clearing temporary directory: /opt/choicestream/apps/buildAgent/temp/buildTmp
: Checkout directory: /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
: +Updating sources+ (5m:45s)
: Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
: Transferring cached clean patch for VCS root: CSA Build Root for GCC 64-Bit Build on 64-Bit Reco Server (Mainline)
: Transferring repository sources: 1002.0Mb so far...
: Transferring repository sources: 2333.73Mb so far...
: Building incremental patch over the cached patch
: Repository sources transferred: 3366.73Mb total
: Average transfer speed: 9.88Mb per second
: Removing /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
: Error while applying patch: Cannot delete file /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
: Build finished

---
Original message URL: http://www.jetbrains.net/devnet/message/5245756#5245756

0
Comment actions Permalink

Hello Jason,

  Could you please run commands
   ls -al /opt/choicestream/apps/buildAgent/work/ and
   du -sk /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792

   So far it is unclear why the directory cannot be deleted, given that build process has appropriate access permissions.

  Regards,
  KIR

0
Comment actions Permalink

We have seen this quite often in the past and it has been related to a process that is still running from a previous build.  In our case it was usually an orphaned Tomcat process because a build failed somehow before a stop command was made.  I have since written a custom Ant builder extension which guarantees targets to be called after a build completes regardless of outcome so that we can ensure that Tomcat is always stopped.

Not sure if it is related but figured it was worth sharing.

0
Comment actions Permalink

The ls -al

drwxrwxrwx  5 csuser csuser 4096 Sep 23 22:00 .
drwxrwxr-x 11 csuser csuser 4096 Aug 12 17:58 ..
drwxrwxr-x  8 csuser csuser 4096 Sep 23 22:47 86f0113228255dda
drwxrwxr-x  8 csuser csuser 4096 Sep 23 09:50 9f3179dbf35c6792
-rw-rw-r--  1 csuser csuser  647 Sep 25 15:20 directory.map
drwxrwxr-x  7 csuser csuser 4096 Sep 23 22:18 .old

The du -sk

3894184    /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792

This is after hitting this:  Error while applying patch: Cannot delete file /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792 which we just got again.

0
Comment actions Permalink

Here is what was in agents log file:

[2009-09-25 15:47:07,133]   WARN -    jetbrains.buildServer.AGENT - Exception occurred while patch applying:Cannot delete file /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792
java.io.IOException: Cannot delete file /opt/choicestream/apps/buildAgent/work/9f3179dbf35c6792        at jetbrains.buildServer.vcs.patches.PatcherImpl.deleteFile(PatcherImpl.java:150)
        at jetbrains.buildServer.vcs.patches.AbstractPatcher$1.delete(AbstractPatcher.java:53)        at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:93)
        at jetbrains.buildServer.vcs.patches.AbstractPatcher.applyPatch(AbstractPatcher.java:50)        at jetbrains.buildServer.agent.impl.patch.ApplyPatch.execute(ApplyPatch.java:20)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.applyPatch(ProjectSourcesGetter.java:258)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.applyPatchToDisc(ProjectSourcesGetter.java:231)        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.loadAndApplyPatch(ProjectSourcesGetter.java:218)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.access$400(ProjectSourcesGetter.java:31)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter$4.execute(ProjectSourcesGetter.java:282)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.executePatchProcess(ProjectSourcesGetter.java:159)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.applyPatch(ProjectSourcesGetter.java:280)
        at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.execute(ProjectSourcesGetter.java:119)
        at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doGetSources(CheckoutSourcesStage.java:45)
        at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doRecoverableStage(CheckoutSourcesStage.java:31)
        at jetbrains.buildServer.agent.impl.runStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:61)
        at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:135)
        at jetbrains.buildServer.agent.impl.BuildRunAction.access$000(BuildRunAction.java:21)
        at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:91)
        at java.lang.Thread.run(Thread.java:619)

0
Comment actions Permalink

Le 25/09/2009 22:00, Tom Ruggles a ecrit :

We have seen this quite often in the past and it has been related to a process that is still running from a previous build.  In our case it was usually an orphaned Tomcat process because a build failed somehow before a stop command was made.  I have since written a custom Ant builder extension which guarantees targets to be called after a build completes regardless of outcome so that we can ensure that Tomcat is always stopped.

>

Not sure if it is related but figured it was worth sharing.


I've seen that on Windows, but I thought on Linux filesystem was able to
delete a folder even in that case.

I'll check more carefully next time.

Gérald

0
Comment actions Permalink

You may try to identify what could have an handle on that folder, or file in that folder...

Olivier.

0
Comment actions Permalink

I'd like to state this a linux box so deleting a file when something may be using it is allowed.  I think this a teamcity agent problem.

0
Comment actions Permalink

Jason could you please list the content of the directory which cannot be deleted?


0
Comment actions Permalink

I will the next time it happens.

0
Comment actions Permalink

I have exact same problem, that happens more or less everyday :-(

[2009-10-05 18:21:49,994]   WARN -    jetbrains.buildServer.AGENT - Exception occurred while patch applying:Cannot delete file /home/teamcity/work/parisrd_quick
java.io.IOException: Cannot delete file /home/teamcity/work/parisrd_quick
    at jetbrains.buildServer.vcs.patches.PatcherImpl.deleteFile(PatcherImpl.java:150)
    at jetbrains.buildServer.vcs.patches.AbstractPatcher$1.delete(AbstractPatcher.java:53)
    at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:93)
    at jetbrains.buildServer.vcs.patches.AbstractPatcher.applyPatch(AbstractPatcher.java:50)
    at jetbrains.buildServer.agent.impl.patch.ApplyPatch.execute(ApplyPatch.java:20)

I join the agent log.

And here is the list of files in the folder :

fauvelle@lanthane:/home/teamcity/work/parisrd_quick$ ls -l *
P_FieldReader:
total 24
drwxr-xr-x 2 bbslave bbslave  4096 2009-09-29 19:02 A2iARC
drwxr-xr-x 3 bbslave bbslave  4096 2009-09-29 19:02 A2iAX
drwxr-xr-x 2 bbslave bbslave  4096 2009-09-29 19:02 ObjectCOM
-rw-r--r-- 1 bbslave bbslave 10872 2009-07-30 17:36 P_FieldReader.bcf

ProjectSpace:
total 72
-rwxr-xr-x 1 bbslave bbslave 2555 2009-08-14 14:40 build_doc.sh
-rwxr-xr-x 1 bbslave bbslave 2098 2009-08-14 11:30 build_parisrd.sh
-rwxr-xr-x 1 bbslave bbslave 1819 2009-08-14 14:40 build_prod.sh
-rwxr-xr-x 1 bbslave bbslave 2286 2009-08-14 14:40 build_spb.sh
-rwxr-xr-x 1 bbslave bbslave 1393 2009-08-14 14:40 build_trunk.sh
-rw-r--r-- 1 bbslave bbslave  335 2009-01-30 13:41 Checkout - dd - ConfigTools.meta.bat
-rw-r--r-- 1 bbslave bbslave  307 2008-11-24 16:58 Checkout - Jph - Core + nrt.meta.bat
-rw-r--r-- 1 bbslave bbslave  497 2009-01-30 13:41 Checkout - Jph - MRM + nrt.meta.bat
-rw-r--r-- 1 bbslave bbslave 4806 2009-06-29 10:43 _CreateMetaProject.py
-rwxr-xr-x 1 bbslave bbslave   90 2009-06-05 15:32 nrt_doc.sh
-rwxr-xr-x 1 bbslave bbslave  179 2009-09-10 14:58 nrt_parisrd.sh
-rwxr-xr-x 1 bbslave bbslave  262 2009-08-10 20:43 nrt_prod.sh
-rwxr-xr-x 1 bbslave bbslave  129 2009-06-01 14:35 nrt_spb.sh
-rwxr-xr-x 1 bbslave bbslave  118 2009-06-01 14:35 nrt_trunk.sh
-rw-r--r-- 1 bbslave bbslave  171 2009-08-17 15:33 Products.ver
drwxr-xr-x 2 bbslave bbslave 4096 2009-09-29 19:02 script
-rw-r--r-- 1 bbslave bbslave  224 2009-01-30 13:41 __wsmgr.bat

PyModules.nrt:
total 168
-rw-r--r--  1 bbslave bbslave    96 2009-10-02 19:00 binfile.dat
-rwxr-xr-x  1 bbslave bbslave    64 2009-08-07 17:04 CMakeLists.txt
drwxr-xr-x  2 bbslave bbslave  4096 2009-09-29 19:03 coverage
-rw-r--r--  1 bbslave bbslave 21812 2008-08-22 15:49 coverage.py
-rw-r--r--  1 bbslave bbslave  9513 2009-10-02 18:48 dependencies.html
-rw-r--r--  1 bbslave bbslave 29189 2009-10-02 18:48 dependencies.png
-rw-r--r--  1 bbslave bbslave  1855 2009-10-02 18:48 dependencies.txt
-rw-r--r--  1 bbslave bbslave 13184 2007-09-12 16:31 details.xsl
drwxr-xr-x  4 bbslave bbslave  4096 2009-09-29 19:21 DST_data
-rw-r--r--  1 bbslave bbslave    50 2009-10-02 18:59 KORN_bidon.mwfi
drwxr-xr-x  2 bbslave bbslave  4096 2009-09-29 19:03 profiling
drwxr-xr-x 16 bbslave bbslave  4096 2009-09-29 19:19 PyExports
drwxr-xr-x  3 bbslave bbslave  4096 2009-10-02 19:00 PyIMDBRelease
-rw-r--r--  1 bbslave bbslave  1125 2009-07-30 17:36 PyModules.nrt.bcf
drwxr-xr-x 13 bbslave bbslave  4096 2009-10-01 18:51 PyNRTLib
drwxr-xr-x  3 bbslave bbslave  4096 2009-10-02 19:00 PyVFSRelease
-rw-r--r--  1 bbslave bbslave  3743 2008-10-09 17:52 RunCoverage.py
-rwxr-xr-x  1 bbslave bbslave   117 2009-05-28 15:22 RunTests_buildbot.sh
-rw-r--r--  1 bbslave bbslave   129 2008-03-25 16:57 RunTests_py.bat
-rw-r--r--  1 bbslave bbslave  2088 2009-10-02 18:53 SBT
-rw-r--r--  1 bbslave bbslave     0 2009-09-29 19:20 stdout.txt
drwxr-xr-x  2 bbslave bbslave  4096 2009-09-29 19:03 TestCases
-rw-r--r--  1 bbslave bbslave   711 2008-06-09 17:27 testNRTTools.py
-rw-r--r--  1 bbslave bbslave  6700 2008-06-09 17:27 testPythonDependencies.py
-rw-r--r--  1 bbslave bbslave  1348 2009-10-02 18:53 tot_train_.dat
fauvelle@lanthane:/home/teamcity/work/parisrd_quick$

They all belong to bbslave user, that is the user for TeamCity build agent, and all these files have been checked out by TeamCity agent, so I don't understand why they cannot be deleted.

The Linux have just been rebooted, so I think there no problem of a process that would have an handle on some files.



Attachment(s):
delete.log.zip
0
Comment actions Permalink

Hi,

Does someone from Jetbrains have more info about this issue?

Thanks,

Gérald

0
Comment actions Permalink

What was the version of TeamCity?

0
Comment actions Permalink

Le 07/10/2009 16:15, Eugene Petrenko a ecrit :

What was the version of TeamCity?


I use 4.5.3.


0
Comment actions Permalink

We use 4.5.3 also.

0
Comment actions Permalink

Just to add some more context, this is happening when two builds share the same VCS Root and using the same build agent.  Hope this helps to diagnose the problem as I think it is an agent bug.

0
Comment actions Permalink

Le 07/10/2009 18:40, Jason Perry a ecrit :

Just to add some more context, this is happening when two builds share the same VCS Root and using the same build agent.  Hope this helps to diagnose the problem as I think it is an agent bug.


Yes, I can say the same, and I had the error today also, on another
build, but with same VCS Root, and coincidence or not, the files and
folders that remains and cannot be deleted are the same between the 2
builds.

Gérald

0
Comment actions Permalink

I've checked the code. The only reason it failed was java failed to delete one of the files under the folder.
What version control do you use? Do you use agent-side checkout? What os is used to run server?
Does the build change file attributes/rights?
What is the user account that is running build agent?

Is this folder mounted to some real drive or is it a network drive? Please check build agents are not using same folder for build checkouts.

0
Comment actions Permalink

Could there be any hidden files under the path?

Please try using the command:
ls -l -a -r .

Please attach the output of  'ps -auxf' too.

Do you use any symlinks?

0
Comment actions Permalink

Le 08/10/2009 12:46, Eugene Petrenko a ecrit :

I've checked the code. The only reason it failed was java failed to delete one of the files under the folder.


Can we know the file that cannot be deleted ?

What version control do you use?


Subversion 1.6

Do you use agent-side checkout? What os is used to run server?


No, I use "Automatically on server", Server is Windows XP.

Does the build change file attributes/rights?


No, it shouldn't. Ah... I think I know. Some tests create and modify
some output files in source folder, as I see in the directory listing.
Could this be the problem ?

What is the user account that is running build agent?


It's a special user account created for TeamCity. When login with that
account and doing a rm -rf on folder, there's no problem.

Is this folder mounted to some real drive or is it a network drive?


It's a real drive

Please check build agents are not using same folder for build checkouts.


I don't understand this last point.

Gérald

0
Comment actions Permalink

Delete of the folder fail if any file under it can not be deleted.

Starting from TeamCity 5.0 EAP we added more detailed reporting of files that were failed to delete.

Created file or symbolic link with wrong rights may be the issue. Please make sure tests removes all
files at the end even in case of failure.
Please check there is no stale processes after some build finish. Such processes may keep creating files
under checkout directory and thus disallow removing files by the agent.

Does your build agents use different drives/folders to checkout the project and run the build?
Please check buildAgent.properties files workDir setting.

0
Comment actions Permalink

Here is the ls and ps output.
I don't see any hidden files. There are some files with strange extension (.¤m), but that shouldnd't be a problem.

There are also python compiled .pyc files, that are automatically generated by python.

And we don't use sym links, at least not in sources.



Attachment(s):
ps.log.zip
ls.txt.zip
0
Comment actions Permalink

But why using a simple rm, with same user, would work then?

The sources are checked out in /home/teamcity/work/..., compilation is done in /home/teamcity/build/..., tests are runned in /home/teamcity/work/..., and some files are created, I can avoid some, but not all, like .pyc files made by python.

And there's no process running, as I said the other day, after a reboot, and launching the build in TeamCity, it fails for same reason.

0
Comment actions Permalink

I've created a patched version of TeamCity with more logging for your issue. Please update TeamCity to
newer version. Please note, this version is build on top of TeamCity 4.5.5, thus, downgrade to TeamCity 4.5.3 or TeamCity 4.5.4
may not be possible. Remember to backup database and TeamCity data directory as usual.

ftp://ftp.intellij.net/pub/.teamcity/TeamCity-9108.war

To apply the patch, you may simply stop TeamCity server,
backup database and data dir, open installation folder and replace webappas/root
with contents of the war file.

Thanks!

0
Comment actions Permalink

Hi,

Thanks for this version. Update run with no problem, and the problem is
still here, with more information :


: //Updating sources//_ (10m:22s)_
: Will perform clean checkout. Reason:
Agent doesn't have any version of the project sources
: Building clean patch for VCS root:
SVN/parisrd
: Transferring cached clean patch for VCS
root: SVN/parisrd
: Repository sources transferred:
1117.51Mb total
: Average transfer speed: 34.51Mb per second
: Removing
/home/teamcity/work/parisrd_nightly
: /[Updating sources] /Error while applying patch: Failed to
delete: /home/teamcity/work/parisrd_nightly. Cannot delete file
/home/teamcity/work/parisrd_nightly/PyModules.nrt/DST_data/sn, children:
[/home/teamcity/work/parisrd_nightly/PyModules.nrt/DST_data/sn/mdc5aldil3dfjntut79ggsk4.�m,
/home/teamcity/work/parisrd_nightly/PyModules.nrt/DST_data/sn/mdc5aldil3dfjntut79ggsk4.�o]
: Build finished

So it seems some data files with strange filename (.¤m or .¤o) are not
handled by TeamCity... These files are generated during some tests.

Is it a bug of TeamCity ?

Anyway I'll see if I can remove them on my side after testing, I don't
need them once the test is done.


Gérald

0
Comment actions Permalink

Hello,

This could be an issue related to locale that was set to TeamCity agent. Please attach output of command 'locale'.
You know, Linux stores file names as chains of bytes, but java uses String objects from it.

Does the build changes locale?


Thanks!

0
Comment actions Permalink

Le 12/10/2009 19:39, Eugene Petrenko a ecrit :

Hello,

>

This could be an issue related to locale that was set to TeamCity agent. Please attach output of command 'locale'.
You know, Linux stores file names as chains of bytes, but java uses String objects from it.


$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Does the build changes locale?


No, as far as I know, we don't use specific locale, it's always default.



Gérald

0
Comment actions Permalink

Please check the created extension names are truly UTF-8.
You may run ls -bl in the broken folder to make ls escape all non-graphical chars with it's codes.

Please add 'locale' command call near the generation of those files in the build script too.
Thanks!

0

Please sign in to leave a comment.