Publishing artifacts on physical agent takes too long

hi,

I'm facing this issue for which I cannot understand the possible cause. Our build uses two agents on Teamcity, one is a VM and other a physical machine. I have seen the issue that publishing artifacts takes way too long on the physicl machine(About 30 mins) and it takes 15 seconds on the VM. The artifacts produced are msi files and few jar and batch files. If the artifacts are something like just jar files on the phycal machine agent, it does it quickly.

This is my build log for the physical machine.

Publishing artifacts (19m:58s)
[10:56:49]: [Publishing artifacts] Paths to publish: [build/**/setup.msi, build/**/setup_instrument.msi, build/**/emma_report.bat, build/**/*.war, build/**/setup_test.msi, build/metrics.txt]
[10:56:50]: [Publishing artifacts] Sending files
[11:16:47]: Build finished

This is my build log for the VM agent.
17:45:55]: Publishing artifacts (15s)
[17:45:55]: [Publishing artifacts] Paths to publish: [build/**/setup.msi, build/**/setup_instrument.msi, build/**/emma_report.bat, build/**/*.war, build/**/setup_test.msi, build/metrics.txt]
[17:45:58]: [Publishing artifacts] Sending files
[17:46:10]: Build finished


the physical machine is a remote agent in my case.

Is there something missing in the physical machine agent configuration that needs to be taken care of? I cannot understand the possible cause of it and would want to rectify this problem.

Thanks

5 comments
Comment actions Permalink

Hi

How fast is the network link between your TeamCity server and remote agent?
It looks like a connection issue. How much time does take to transfer these artifact files manually?

0
Comment actions Permalink

I'm seeing a similar issue on my server, except in opposite.  For no obvious reason a build using a VM agent is taking forever to publish it's artifacts.  In fact, it seems to be stuck in a loop of some kind as the is continually hitting Sending Files according to the build log.  The artifacts aren't that large, just a handful of .dlls, .libs and .exes totalling about 15MB.  This build spent about 25 seconds in the publishing step previously, now it's taking an hour or more.  There's a build on another project that has 47MB worth of artifacts and it doesn't seem to be having any problems, it's using a physical machine currently.

It should be going through as our IT admin is monitoring the traffic and tells me the agent was hitting the network bandwidth at 34 megs/min.

Looking at the log from the agent it seems something is wrong with the connection and it's getting continually reset.

Edit:
I can supply the server log via email, rather not post it publicly since it has project names in it.  I'm seeing socket write errors over and over in it.

[2012-07-30 09:15:45,156]   WARN -   jetbrains.buildServer.SERVER - Failed to upload artifact, due to error: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Connection reset 
[2012-07-30 09:15:45,156]   WARN -   jetbrains.buildServer.SERVER - Failed to send response to agent: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error 


Attachment(s):
teamcity-agent.log.zip
0
Comment actions Permalink

We got the problem fixed for our VM that was having this problem.  The solution ended up being to update Java and to set the timeout for the jre to 40000 instead of 10000.

0
Comment actions Permalink

What I thought was a fix for the VM turned out to only be a band aid.  The VM eventually started having problems so I spun up a fresh new physical agent as a temporary solution and now that physical agent is having the same problem.  At this point I'm leaning more and more toward this being a bug in TeamCity introduced in either 7.02a or 7.03, I never had this problem before one of those two versions.  I updated to 7.1 last night to see if it fixed it but only partially fixed it, or the reporting in the build log has just improved so I'm seeing how far it actually got in the upload before failing out.  I'm still getting the same errors in the log that I posted previously.

I really need a solution or workaround for this ASAP as this has become a blocker for our development pipeline.

I should add that I have tested transferring the files manually, which takes seconds, so this is not an issue at the network level.

0
Comment actions Permalink

Hello Scott,

Please,

  • enable debug logging on both, server and agent (instructions for server and agent),
  • reproduce the issue
  • send both logs to teamcity-feedback@jetbrains.com (add a link to this forum thread in your message body)
0

Please sign in to leave a comment.