TeamCity canceling build

Permanently deleted user

Created March 03, 2015 16:54

Hello,

In the past couple of weeks, a certain build gets automatically canceled by TeamCity on a regular basis after running anywhere from 90 to 240 minutes. The entire build takes about 6 hours to run. It is run using several custom script build steps and fails at different build steps but tends to fail by the 3rd, 4th or 5th step. It hangs for an hour or more and then gets automatically canceled. We have seen this error in the build agent log:
[2015-03-03 15:37:12,727] out - Process exited with code 137

Here are the build failure conditions:

Does this seem like a memory crash? If so, why wouldn't the build fail rather than being automatically canceled?

Thanks,
-Dave

1 comment

Permanently deleted user

Created March 06, 2015 20:30

This seems to have been caused by an overload on the TeamCity server when all 14 of our build agents are building at the same time.

Here is the output of top at the time:

top - 02:45:05 up 56 days,  6:37,  1 user,  load average: 5.06, 5.48, 6.06

Tasks: 216 total,   2 running, 214 sleeping,   0 stopped,   0 zombie

Cpu0  : 51.0%us,  1.3%sy,  0.0%ni, 47.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Cpu1  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Cpu2  : 99.7%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st

Cpu3  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Cpu4  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Cpu5  : 52.8%us,  2.0%sy,  0.0%ni, 45.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   5968396k total,  5925124k used,    43272k free,  1322796k buffers

Swap:  3080184k total,   262912k used,  2817272k free,  1566148k cached

The TeamCity 8.1.5 server is running on a CentOS6 VM and as you can see above had 6 vcpus. After we doubled the number of vcpus to 12, the problem seems to have gone away.

-Dave

Please sign in to leave a comment.