Slow resolving artifact dependencies

Hi,

We are seeing artifact resolution taking much longer than previously (6 minutes before, and now 20 minutes).

For example this step used to take half a minute and now its taking 2 minutes:

[16:03:01]Resolving artifact dependencies

[16:05:06][Resolving  artifact dependencies] 236 files retrieved for  [**/TX.2012.05.0/**/install/common/lib/*.*=>otclcm] downloading  pattern


At the same time we see high CPU load from the teamcity server process.

We switched off the "Show changes from snapshot dependencies" because there is a known issue with this feature in TC7 that causes slowness, but this did not improve the artifact resolution times.

Attached is a thread dump taken at the time of the slowness.

Can you assist in diagnosing this problem please?

Thanks,
Sadegh



Attachment(s):
threadDump-2012-05-15_15.39.33.zip
10 comments
Comment actions Permalink

Sadegh

What version of Teamcity are you using?
When did you start to face this issue?

0
Comment actions Permalink

Hi Nikita,



Version 7.0.2

We have seen this issue once since upgrading to version 7.0.2 *(from 6.5.x). On that day we did have a lot of builds running - but no more than was typical when we were running version 6.5.x.

0
Comment actions Permalink

Sadegh,

How many artifact dependencies do you have for the build configuration? E.g., for the one which changed artifact resolution time from 6 to 20 minutes.
Could you also post several agent-side thread dumps during artifacts resolution. You can find instructions here: http://confluence.jetbrains.net/display/TCD7/Reporting+Issues#ReportingIssues-Agentthreaddump

0
Comment actions Permalink

There is 11 artifact dependencies and 1 snapshot dependency on this particular build configuration.

Do we need to take the agent thread dump while its taking a long time to resolve?

0
Comment actions Permalink

Yes, please.

So artifact resolution is not always slow? Can you think of anything specific about when it happens (e.g., onle on one agent, or smth else...)?

0
Comment actions Permalink

Nothing specific comes to mind. Its not always slow, but we do see it like today.

I've attached the thread dump from the server and 3 agents (while they were slowly resolving dependencies) from today.



Attachment(s):
threadDump-server-2012-05-17_11.31.43.zip
threadDump-agent-4002-2012-05-17_11.26.05.zip
threadDump-agent-4001-2012-05-17_11.30.24.zip
threadDump-agent-3003-2012-05-17_11.23.58.zip
0
Comment actions Permalink

Thank you for thread dumps.

So, does this happen to all the agents and build server together at the same time?

0
Comment actions Permalink

Yes. All of them have slow resolving dependency while they are all running builds at the same time, causing a queue to build up.

0
Comment actions Permalink

Sadegh,

It looks like the disk IO is the bottleneck, as TeamCity server spends time delivering the lists of artifacts.
What's is your harddrive configuration? Do you have other applications heavely utilizing the filesystem containing artifacts?  

0
Comment actions Permalink

Hi Nikita,

We upgraded to 7.0.3 and switched off the "Show changes from snapshot dependencies" option (was creating heavy load on the server) and its been fine so far.

We'll keep an eye out for further issues, nothing has come up yet so I'd consider this resolved.

Thanks!

0

Please sign in to leave a comment.