diagnosing issues with VCS roots and build scripts

I have a few questions about how to diagnose/debug VCS issues and build scripts.
I am currently migrating several builds into TeamCity and am having some difficulties.

1. What is the best way to diagnose issues with VCS roots?  One of my builds that uses CVS sometimes takes minutes to to collect changes and sometimes it takes an hour.
I have turned on debugging and looked at the logs but I don't see any errors.

For instance:

[2010-07-26 15:25:05,998]   INFO [loader 5 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for cvs: :pserver:cvsuser@cvs.xxx.com:/usr/cvs CodeBranch/ {id=2}; 0 changes collected 0 changes reported, time spent: 1m:42s,204ms
[2010-07-26 15:30:17,396]   INFO [loader 4 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for cvs: :pserver:cvsuser@cvs.xxx.com:/usr/cvs CodeBranch/ {id=2}; 0 changes collected 0 changes reported, time spent: 1m:48s,567ms
[2010-07-26 15:33:16,229]   INFO [oader 2 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for cvs: :pserver:cvsuser@cvs.xxx.com:/usr/cvs CodeBranch/ {id=2}; 0 changes collected 0 changes reported, time spent: 43m:26s,558ms
[2010-07-26 15:35:21,073]   INFO [loader 2 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for cvs: :pserver:cvsuser@cvs.xxx.com:/usr/cvs CodeBranch/ {id=2}; 0 changes collected 0 changes reported, time spent: 1m:42s,235ms

Is there any way to get verbose information about what is going on during the "Checking changes" phase of the build?

I also see that sometimes it doesn't pick up code changes - I find this particulary happens with the CVS VCS roots.  We have many builds running on SVN without the same problems.

2.  What is the best technique to debug a new build?  Is there a way to initially check-out the code but ignore the check-out while you work on the build logic?  The check-out takes 30 minutes for a particular build I am working.  If I miss one variable and have to restart, it is very painful. I tried switching the "VCS" mode to "don't checkout", but it still does the collecting changes step and switches working directories because of the VCS change.


thanks in advance.
NSN

5 comments

Hello,

What is the best way to diagnose issues with VCS roots?  



To debug VCS problems you need to enable debug mode in the <TeamCity home>\conf\teamcity-server-log4j.xml and <BuildAgent home>\conf\teamcity-agent-log4j.xml files:

<category name="jetbrains.buildServer.VCS" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>


Also in the <TeamCity home>\conf\teamcity-server-log4j.xml for ROLL.VCS replace line <param name="maxBackupIndex" value="3"/> with unncomment line <param name="maxBackupIndex" value="20"/> to store more logs.

I also see that sometimes it doesn't pick up code changes - I find this particulary happens with the CVS VCS roots.  We have many builds running on SVN without the same problems

Is this issue reproducible ? We will need more information to investigate this issue. Could you please add issue in our tracker http://youtrack.jetbrains.net/issues/tw. Please provide screenshots of the Changes tab of the builds that should include the changes but did not (with revisions table expanded), screenshot of the Change Log tab of the affected build configuration,  settings of the CVS VCS root and checkout rules of the build configuration. We might also need teamcity-server.log and teamcity-vcs.log.

What is the best technique to debug a new build?  Is there a way to initially check-out the code but ignore the check-out while you work on the build logic?  The check-out takes 30 minutes for a particular build I am working.  If I miss one variable and have to restart, it is very painful. I tried switching the "VCS" mode to "don't checkout", but it still does the collecting changes step and switches working directories because of the VCS change.



You could check out sources in certain directory, remove VCS root from build configuration, switch "VCS checkout mode" to "Do not checkout files automatically" and explicitly specify the checkout directory to that already contain that sources.


Kind regards,
Marina

0

Thanks Marina.  I will investigate your solutions.

-nsn

0

Marina,

You could check out sources in certain directory, remove VCS root from build configuration, switch "VCS checkout mode" to "Do not checkout files automatically" and explicitly specify the checkout directory to that already contain that sources.


Your suggestion worked perfectly for me.

What is the best way to diagnose issues with VCS roots?

I enabled the debug mode as you outlined and I am still not able to figure out what the issue is.  My original configuration was pulling source from multiple CVS ans SVN repos.  I have narrowed down the problem to one particular CVS location.  The CVS repository in extremely large, but I have applied 29 check-out rules to the VCS, one directory and 28 individual files.  When I check-out these files from the command line it takes 2-3 minutes.  I have only got these files checked out one time with success in TeamCity.  I have pulled out the the VCS root into a test configuration and it literally runs for hours and never seems to complete. It checks out most of the files but never gets all of the files.  I searched the log files for any instance of the particular VCS and it doesn't show up.  I have tried running the check-out on both Windows and Linux agents with the same results.
When I dump the threads I get the following message: "TeamCity was unable to locate any processes of this build."
This is a brief dump of the output

[20:49:06]: Updating sources: Agent  side checkout...
[20:49:06]: [Updating sources: Agent side  checkout...] Will perform clean checkout. Reason: Checkout directory is  empty or doesn't exist
...
...
...

[20:53:13]: [VCS Root: ansysDV] Updating from CVS  repository
[20:53:13]: [VCS Root: ansysDV] cvs update:  Updating .
[22:40:23]: [VCS Root: ansysDV] Updating from CVS  repository
[22:40:37]: [VCS Root: ansysDV] cvs update:  Updating .

Do you have any other suggestions?

0

I am now only checking out a subset of the files I need as a test.
I have cut down the check-out rules to only 6 and the check-out did complete in 8-9 minutes ( the full checkout on the command line takes 2 minutes).
The subset is 15 MB with 1942 files and 4 folders.
How can I diagnose why it takes so long in TC and hangs on the full  29 rules and takes 4 times as long for the subset?

0

Hello,

Сan you wait for the build to finish normally? If it is hanging you can try to see if it actually is doing something (e.g. loading CPU, files uploaded)? If it is not, take a thread dump from of the build agent process (please see http://confluence.jetbrains.net/display/TCD5/Reporting+Issues#ReportingIssues-HangsandThreadDumps).

Kind regards,
Marina

0

Please sign in to leave a comment.