Teamcity issue with large artifacts
Hi all!
I'm experiencing bad issues with publishing large artifacts ( 1..10Gb )
My installation looks like this : [ TeamCity Agent (physical machine) ] <-> [ nginx <-> TeamCity Server (amazon ec2 instance) ].
All settings are tuned to store large artifacts (artifact size, nginx post data size etc...).
And artifact publishing goes well while I store them on my system disk on the server.
But this is obviously bad practice so I create another volume and mount it on the artifact storage path.
>>>
ubuntu@aux:~$ mount | grep xvdf
/dev/xvdf on /opt/TeamCity/.BuildServer/system/artifacts type xfs (rw,noatime)
<<<
After that magic happens : small artifacts keep publishing well, but large artifacts eem to fail publishing
Nginx logs are pretty ok
Agent tries to publish artifacts in an infinite loop (and fails)
[20:19:47]Publishing artifacts (14h:13m:52s)
[20:19:47][Publishing artifacts] Collecting files to publish: [tourdb.pgdump.gz, media.tar.gz]
[20:19:48][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [ArtifactsCachePublisher]
[20:19:48][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [WebPublisher]
[01:06:56][Publishing artifacts] Recoverable problem publishing artifacts (will retry): Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out)
[01:07:49][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [ArtifactsCachePublisher]
[01:07:49][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [WebPublisher]
[05:51:43][Publishing artifacts] Recoverable problem publishing artifacts (will retry): Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out)
[05:52:35][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [ArtifactsCachePublisher]
[05:52:36][Publishing artifacts] Publishing 2 files [tourdb.pgdump.gz] using [WebPublisher]
[10:32:01][Publishing artifacts] Recoverable problem publishing artifacts (will retry): Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out)
...
Agent log contain something like
[2016-09-26 01:39:16,779] WARN - jetbrains.buildServer.AGENT - Failed to publish artifacts: Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out), will
try again
[2016-09-26 01:39:17,207] INFO - jetbrains.buildServer.AGENT - Recoverable problem publishing artifacts (will retry): Failed to publish artifacts, see teamcity-agent.log for details. Server status: 50
4 (Gateway Time-out): jetbrains.buildServer.agent.ArtifactPublishingFailedException: Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out) (enable debug to
see stacktrace)
[2016-09-26 01:39:17,207] INFO - r.agent.impl.AgentLogProxyImpl - Failed to perform remote command Publishing to for build 159, error: jetbrains.buildServer.agent.ArtifactPublishingFailedException: Fail
ed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out): jetbrains.buildServer.agent.ArtifactPublishingFailedException: Failed to publish artifacts, see teamcity-
agent.log for details. Server status: 504 (Gateway Time-out) (enable debug to see stacktrace)
Server log looks like
try again
[2016-09-26 01:39:17,207] INFO - jetbrains.buildServer.AGENT - Recoverable problem publishing artifacts (will retry): Failed to publish artifacts, see teamcity-agent.log for details. Server status: 50
4 (Gateway Time-out): jetbrains.buildServer.agent.ArtifactPublishingFailedException: Failed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out) (enable debug to
see stacktrace)
[2016-09-26 01:39:17,207] INFO - r.agent.impl.AgentLogProxyImpl - Failed to perform remote command Publishing to for build 159, error: jetbrains.buildServer.agent.ArtifactPublishingFailedException: Fail
ed to publish artifacts, see teamcity-agent.log for details. Server status: 504 (Gateway Time-out): jetbrains.buildServer.agent.ArtifactPublishingFailedException: Failed to publish artifacts, see teamcity-
agent.log for details. Server status: 504 (Gateway Time-out) (enable debug to see stacktrace)
Server log looks like
[2016-09-25 16:53:19,039] WARN - jetbrains.buildServer.SERVER - Failed to upload artifact for build with id: 159, due to error: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Proce
ssing of multipart/form-data request failed. Unexpected EOF read on the socket
[2016-09-25 16:54:11,521] WARN - jetbrains.buildServer.AGENT - Agent "ip_172.31.31.4" {id=5} tries to register while it is already registered.
[2016-09-25 21:39:18,388] WARN - jetbrains.buildServer.SERVER - Failed to upload artifact for build with id: 159, due to error: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Proce
ssing of multipart/form-data request failed. Unexpected EOF read on the socket
[2016-09-25 21:40:10,879] WARN - jetbrains.buildServer.AGENT - Agent "ip_172.31.31.4" {id=5} tries to register while it is already registered.
ssing of multipart/form-data request failed. Unexpected EOF read on the socket
[2016-09-25 16:54:11,521] WARN - jetbrains.buildServer.AGENT - Agent "ip_172.31.31.4" {id=5} tries to register while it is already registered.
[2016-09-25 21:39:18,388] WARN - jetbrains.buildServer.SERVER - Failed to upload artifact for build with id: 159, due to error: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Proce
ssing of multipart/form-data request failed. Unexpected EOF read on the socket
[2016-09-25 21:40:10,879] WARN - jetbrains.buildServer.AGENT - Agent "ip_172.31.31.4" {id=5} tries to register while it is already registered.
...
And this generates an ugly infinite loop of generating traffic without any useful result.
I've suspected some nginx-related issues but everything is ok while TeamcityServer stores artifacts on its system disk.
And this generates an ugly infinite loop of generating traffic without any useful result.
I've suspected some nginx-related issues but everything is ok while TeamcityServer stores artifacts on its system disk.
This is TC Server 9.1.6 (build 37459)
Upgrade to TC10 is planned but impossible just now.
Any suggestions?
Any suggestions?
Please sign in to leave a comment.
Upgrade to TC10 didn't help.
Now I see messages in nginx err.log
2016/10/01 03:00:07 [error] 1585#0: *149406 upstream timed out (110: Connection timed out) while sending request to upstream, client: 109.173.122.219, server: teamcity, request: "POST /httpAuth/artefactUpload.html HTTP/1.1", upstream: "http://127.0.0.1:8111/httpAuth/artefactUpload.html", host: "teamcity.alterra.cc"
How can I profile TC server artifact saving routine?