SVN build triggering has stopped working - stuck with pending changes

Hi,


SVN build triggering stopped working today on our TeamCity server (6.0.2 build 15857). On the project overview page, the changes show up as a yellow "pending" box next to the Run button - but the build doesn't trigger automatically. If I manually click the Run button, it correctly builds the project with the latest changes - the part that's not working is the automatic triggering.


I turned on debug logging and I'm seeing the following in the teamcity-server.log:

[2011-04-27 23:47:38,796]  DEBUG - ains.buildServer.util.WatchDog - Build Triggers Monitor start: 0 msec
[2011-04-27 23:47:38,796]  DEBUG - Side.impl.BuildTriggersChecker - Start triggers polling for build configuration: IMS :: Development - CI Build {id=bt2}
[2011-04-27 23:47:38,797]  DEBUG - Side.impl.BuildTriggersChecker - Finished triggers polling for build configuration: IMS :: Development - CI Build {id=bt2}, # of successfully processed triggers: 0

.... more build configurations all with # successfully processed=0 ...
[2011-04-27 23:47:38,800]  DEBUG - ains.buildServer.util.WatchDog - Build Triggers Monitor done: 4 msec


Let me know if you want the full log files.


We didn't change the build configuration today. The build configuration has a quiet period of 60 seconds, and the VCS root has a check interval of 60 seconds (using the global setting). So the max delay should be two minutes. Currently it doesn't trigger at all though, even after a few hours.

We did upgrade our SVN server last week... but if that's the cause, I assume the problem would have happened sooner. We upgraded from an SVN 1.4 server running svn:// protocol to a 1.6 server accessed over HTTP. I updated the repository URL in the VCS root, and everything worked fine in TeamCity until today.

Let me know if you have any advice.


Thanks for your help,
Richard
8 comments
Comment actions Permalink

Hi Richard

Couuld you also post teamcity-vcs.log please.

0
Comment actions Permalink

Hi Michael,


Thanks for your help. I've attached a zip file containing teamcity-vcs.log, as well as all the other logs I could find.

We have two build agents running on the same machine: BuildServer and BuildServer2. I've put the logs for these agents in subfolders inside the zip file.

teamcity-vcs.log has a lot of errors from the time we were upgrading SVN. It also has some more recent errors, which make me suspicious - e.g.:

[2011-04-28 03:03:20,961]   WARN [loader 3 {id=2}] -      jetbrains.buildServer.VCS - Error while loading vcs version for root 'svn: http://inflection2:81/svn/IA/GUI/V4/releases/4.0.1/ {id=2}', id=2: org.tmatesoft.svn.core.SVNException: svn: connection refused by the server
svn: OPTIONS request failed on '/svn/IA/GUI/V4/releases/4.0.1'


But this error is not from the time when changes were checked in - it's at 3AM when no one was working. Maybe there was a temporary problem with the SVN server or the network connection.


Here's an example of the triggering problem:

Our "IMS :: Development - CI Build" build configuration has a VCS root of http://inflection2:81/svn/IA/GUI/V4/src. It's configured to trigger the build on every checkin.

According to our svn logs, someone checked in revision 3928 to the repository at 2011-04-27 16:49:03. All the files checked in were under http://inflection2:81/svn/IA/GUI/V4/src. For example, file http://inflection2:81/svn/IA/GUI/V4/src/Client.Core/ViewModels/RefreshableViewModel.cs was changed.

teamcity-vcs.log shows this checkin here:

[2011-04-27 16:49:41,983]   INFO [loader 2 {id=1}] -      jetbrains.buildServer.VCS - Finish collecting changes for svn: http://inflection2:81/svn/IA/GUI/V4/src/ {id=1} from version 3927_2011/04/27 16:48:41 -0700 to version 3928_2011/04/27 16:49:41 -0700; 1 changes collected 1 changes reported, time spent: 436ms


But the "IMS :: Development - CI Build" build configuration was never triggered by this checkin. Around half an hour later, I manually clicked the Run button to trigger a build.



Thanks,
Richard

Attachment(s):
teamcity-logs.zip
0
Comment actions Permalink

Hi,

Do you have any suggestions for further investigating this issue?

Thanks,
Richard

0
Comment actions Permalink

Filed by Richard as TW-16608 (duplicate of TW-13114).

0
Comment actions Permalink

In case anyone else finds this discussion thread:

Yegor suggested stopping the server, backing up the .BuildServer\system\pluginData\customDataStorage directory, and then deleting the directory. (This directory is under c:\users\<username>\ on our installation.) This has solved the problem for us - build triggering works properly now.

Thanks,
Richard

0
Comment actions Permalink

Worked for us.
Thanks for sharing Richard!

0
Comment actions Permalink

Alex,
If you are using TeamCity 6.5.x and you have either the content you deleted or teamcity-server.log covering the issue, you are welcome to share them for our investigation.

0
Comment actions Permalink

Looks like this thread is more active than the other one (http://devnet.jetbrains.net/message/5446285), so I'll post here.

This issue showed up again for me today right after I "cleaned" the Oracle database (not a big database guy, I just dropped all the tables and let it rebuild [It should be noted, I am just messing around with TeamCity at this point.  I'd never do this for real]).  Poking around in the folder described above (\.BuildServer\system\pluginData\customDataStorage) I found a folder that corresponded with the build configuration that was giving me problems (\.BuildServer\system\pluginData\customDataStorage\buildTypes\bt2).  Inside of that folder are three files, two with what appear to be randomly generated numerical file names and one that is idmap.properties.  The contents of one of the numeric files looked like this:

<?xml version="1.0" encoding="UTF-8" ?>
<map>
  <entry>
    <string>lastTriggeredBy</string>
    <string>85</string>
  </entry>
</map>



The 85 appears to be the MODIFICATION_ID of the last change that triggered the build.  This number has been reset in the Database so new changes have started with MODIFICATION_ID of 1.  Looking at the contents of a similar file in a different buildType folder which is set up exactly like this build configuration, but had not triggered anything yet, I noticed the contents were as follows:

<?xml version="1.0" encoding="UTF-8" ?>
<map/>



I then edited the file in the original build configuration to match and builds are pushing correctly now.
0

Please sign in to leave a comment.