TeamCity Stops Checking For Changes (VCS - Visual Source Safe)

Avatar
William Roush
Created

I'm having an odd issue with 6.0.3, though it started to happen in 6.0.2 before our upgrade.

Every once in awhile, TeamCity just stops checking for new changes, all my projects will have 0 pending changes, clicking "check for pending changes" does nothing (not even anything in the log) and restarting TeamCity causes all the changes to show up!

Files I checked in 2 hours ago just showed up with the most recent TeamCity restart, check interval is 60 seconds.

Where should I start gathering logs other than teamcity-vcs.log in order to figure out why this isn't working? I just turned on debug-vcs and will be getting dumps when this happens again, if it's a problem with TeamCity and not my config I'll submit a bug report.


TC Server Info:

TeamCity 6.0.3 (also did it in 6.0.2)
OS: Windows Server 2008 R2
VCS: Visual Source Safe


Thanks. :)

Votes

0

Share

7 comments
Avatar
William Roush
Created

Alright, this is interesting:



[2011-03-22 10:25:59,953]  DEBUG [loader 5 {id=2}] - .vss.VssChangeCollectionPolicy - version from 1300803894953 to 1300803959953
[2011-03-22 10:25:59,953]  DEBUG [loader 5 {id=2}] - cs.vss.runner.AbstractSSRunner - Run VSS Command: c:\TeamCity\temp\TC-VSS-NATIVE-1300466190570.exe C:\TeamCity\temp\TC-VSS-14518-VSS\TC-VSS-0.output C:\TeamCity\temp\TC-VSS-14518-VSS\TC-VSS-1.error \\vss_server\SourceSafe\srcsafe.ini /login:***** /pwd:***** history "$/Project A (MVC 2.0)/Project A 1.0 Test/" 1300803894953 1300803959953
[2011-03-22 10:26:00,656]  DEBUG [loader 5 {id=2}] -      jetbrains.buildServer.VCS - VCS plugin vss reported changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2}; 0 changes collected
[2011-03-22 10:26:00,656]  DEBUG [loader 5 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2} from version 1300803894953 to version 1300803959953; 0 changes collected 0 changes reported, time spent: 703ms
[2011-03-22 10:26:00,656]  DEBUG [loader 5 {id=2}] -      jetbrains.buildServer.VCS - vss, set version: 1300803959953 for root vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2}
[2011-03-22 10:26:01,437]  DEBUG [t files cleanup] - .vss.runner.FilesCleanupThread - C:\TeamCity\temp\TC-VSS-14518-VSS
[2011-03-22 10:26:39,953]   INFO [loader 3 {id=3}] -      jetbrains.buildServer.VCS - Collecting changes: root=vss: \\vss_server\SourceSafe\srcsafe.ini $/Project B {id=3}, fromVersion=1300803934953, toVersion=1300803999953
[2011-03-22 10:26:39,953]  DEBUG [loader 3 {id=3}] -      jetbrains.buildServer.VCS - root details: Project B {id=3; ini-path=\\vss_server\SourceSafe\srcsafe.ini; module-name=$/Project B; secure:vss-password=*****; teamcity:vcsRootName=Project B; vcs=vss; vss-username=my_user; ; ver:2}
[2011-03-22 10:26:39,953]  DEBUG [loader 3 {id=3}] -      jetbrains.buildServer.VCS - Combined checkout rules: =>

[2011-03-22 10:26:39,953]  DEBUG [loader 3 {id=3}] - .vss.VssChangeCollectionPolicy - version from 1300803934953 to 1300803999953
[2011-03-22 10:26:39,953]  DEBUG [loader 3 {id=3}] - cs.vss.runner.AbstractSSRunner - Run VSS Command: c:\TeamCity\temp\TC-VSS-NATIVE-1300466190570.exe C:\TeamCity\temp\TC-VSS-14519-VSS\TC-VSS-0.output C:\TeamCity\temp\TC-VSS-14519-VSS\TC-VSS-1.error \\vss_server\SourceSafe\srcsafe.ini /login:***** /pwd:***** history $/Project B/ 1300803934953 1300803999953
[2011-03-22 10:26:40,421]  DEBUG [loader 3 {id=3}] -      jetbrains.buildServer.VCS - VCS plugin vss reported changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project B {id=3}; 0 changes collected
[2011-03-22 10:26:40,421]  DEBUG [loader 3 {id=3}] -      jetbrains.buildServer.VCS - Finish collecting changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project B {id=3} from version 1300803934953 to version 1300803999953; 0 changes collected 0 changes reported, time spent: 468ms
[2011-03-22 10:26:40,421]  DEBUG [loader 3 {id=3}] -      jetbrains.buildServer.VCS - vss, set version: 1300803999953 for root vss: \\vss_server\SourceSafe\srcsafe.ini $/Project B {id=3}
[2011-03-22 10:26:40,484]  DEBUG [t files cleanup] - .vss.runner.FilesCleanupThread - C:\TeamCity\temp\TC-VSS-14519-VSS
[2011-03-22 10:26:49,984]   INFO [loader 5 {id=1}] -      jetbrains.buildServer.VCS - Collecting changes: root=vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 {id=1}, fromVersion=1300803934953, toVersion=1300804009984
[2011-03-22 10:26:49,984]  DEBUG [loader 5 {id=1}] -      jetbrains.buildServer.VCS - root details: Project A 1.0 {id=1; ini-path=\\vss_server\SourceSafe\srcsafe.ini; module-name=$/Project A (MVC 2.0)/Project A 1.0; secure:vss-password=*****; teamcity:vcsRootName=Project A 1.0; vcs=vss; vss-username=my_user; ; ver:1}
[2011-03-22 10:26:49,984]  DEBUG [loader 5 {id=1}] -      jetbrains.buildServer.VCS - Combined checkout rules: =>

--- RESTART TC ---

[2011-03-22 10:26:49,984]  DEBUG [loader 5 {id=1}] - .vss.VssChangeCollectionPolicy - version from 1300803934953 to 1300804009984
[2011-03-22 10:26:50,015]  DEBUG [loader 5 {id=1}] - cs.vss.runner.AbstractSSRunner - Run VSS Command: c:\TeamCity\temp\TC-VSS-NATIVE-1300466190570.exe C:\TeamCity\temp\TC-VSS-14520-VSS\TC-VSS-0.output C:\TeamCity\temp\TC-VSS-14520-VSS\TC-VSS-1.error \\vss_server\SourceSafe\srcsafe.ini /login:***** /pwd:***** history "$/Project A (MVC 2.0)/Project A 1.0/" 1300803934953 1300804009984
[2011-03-22 10:27:04,984]   INFO [loader 3 {id=2}] -      jetbrains.buildServer.VCS - Collecting changes: root=vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2}, fromVersion=1300803959953, toVersion=1300804024984
[2011-03-22 10:27:04,984]  DEBUG [loader 3 {id=2}] -      jetbrains.buildServer.VCS - root details: Project A 1.0 Test {id=2; ini-path=\\vss_server\SourceSafe\srcsafe.ini; module-name=$/Project A (MVC 2.0)/Project A 1.0 Test; secure:vss-password=*****; teamcity:vcsRootName=Project A 1.0 Test; vcs=vss; vss-use-ssexe-patch=true; vss-username=my_user; ; ver:1}
[2011-03-22 10:27:04,984]  DEBUG [loader 3 {id=2}] -      jetbrains.buildServer.VCS - Combined checkout rules: =>

[2011-03-22 10:27:04,984]  DEBUG [loader 3 {id=2}] - .vss.VssChangeCollectionPolicy - version from 1300803959953 to 1300804024984
[2011-03-22 10:27:05,921]  DEBUG [loader 5 {id=1}] -      jetbrains.buildServer.VCS - VCS plugin vss reported changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 {id=1}; 0 changes collected
[2011-03-22 10:27:05,921]  DEBUG [loader 5 {id=1}] -      jetbrains.buildServer.VCS - Finish collecting changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 {id=1} from version 1300803934953 to version 1300804009984; 0 changes collected 0 changes reported, time spent: 15s,937ms
[2011-03-22 10:27:05,921]  DEBUG [loader 5 {id=1}] -      jetbrains.buildServer.VCS - vss, set version: 1300804009984 for root vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 {id=1}
[2011-03-22 10:27:05,921]  DEBUG [loader 3 {id=2}] - cs.vss.runner.AbstractSSRunner - Run VSS Command: c:\TeamCity\temp\TC-VSS-NATIVE-1300466190570.exe C:\TeamCity\temp\TC-VSS-14521-VSS\TC-VSS-0.output C:\TeamCity\temp\TC-VSS-14521-VSS\TC-VSS-1.error \\vss_server\SourceSafe\srcsafe.ini /login:***** /pwd:***** history "$/Project A (MVC 2.0)/Project A 1.0 Test/" 1300803959953 1300804024984
[2011-03-22 10:27:06,015]  DEBUG [t files cleanup] - .vss.runner.FilesCleanupThread - C:\TeamCity\temp\TC-VSS-14520-VSS
[2011-03-22 10:27:06,593]  DEBUG [loader 3 {id=2}] -      jetbrains.buildServer.VCS - VCS plugin vss reported changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2}; 0 changes collected
[2011-03-22 10:27:06,593]  DEBUG [loader 3 {id=2}] -      jetbrains.buildServer.VCS - Finish collecting changes for vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2} from version 1300803959953 to version 1300804024984; 0 changes collected 0 changes reported, time spent: 1s,609ms
[2011-03-22 10:27:06,593]  DEBUG [loader 3 {id=2}] -      jetbrains.buildServer.VCS - vss, set version: 1300804024984 for root vss: \\vss_server\SourceSafe\srcsafe.ini $/Project A (MVC 2.0)/Project A 1.0 Test {id=2}
[2011-03-22 10:27:07,078]  DEBUG [t files cleanup] - .vss.runner.FilesCleanupThread - C:\TeamCity\temp\TC-VSS-14521-VSS



What I immediately notice is that the ChangeCollectionPolicy isn't being updated until after I restart, of course there are no changes when comparing a version with itself.


Is this a bug that has to do with TeamCity? Or is the VSS component more hands off than that?

0
Avatar
William Roush
Created

Ok, it's confirmed, the trigger just stops running.

I have on 3/22 the vss runner running every minute (or so... there are two projects).

Then nothing after 10:28 on 3/22.

We get one call at midnight at 3/24 and one at 3/26.

Basically the VCS triggers are messed up.

0
Avatar
Eugene Petrenko
Created

Please try checking you VSS database.
Have you changes/update VSS related software on TeamCity server?
Check server time to be the same as VSS share holding machine.

0
Avatar
William Roush
Created

Please try checking you VSS database.

Everything seems in order, we're constantly using it and haven't had issues outside of TeamCity.

Have you changes/update VSS related software on TeamCity server?

We haven't made any  changes to our VSS server, however I did go ahead and reinstall the VSS tools to confirm it isn't an issue there.

Check server time to be the same as VSS share holding machine.


Both are observing DST and are the same.



I think the biggest flag is that TeamCity isn't even checking anymore, VCS log is blank for days on end, when TeamCity is restarted it checks the VSS fine.

0
Avatar
Eugene Petrenko
Created

Please check Checkout rules in the build configurations. Checkout rules are processes with respect to case, while VSS is case _in_sensitive. If you used wrong case in checkout rules, that could make all changes filtered away.

Could you please check VSS database with VSS admin tools. There could be some errors in the database that make TeamCity fail to detect changes.

0
Avatar
William Roush
Created

Wow, I feel bad, just because I thought they both were pointed at the same network time server doesn't mean I should believe the time was the same. I don't actually have access to our VSS server (it does various other things of company importance), so I had to kind of make an assumption when I asked if they both were observing the same time frame and was set "nearly" to be the same, didn't think the constraints on the VSS checkout were that tight (apparently by domain default the variance in that can be up to 5 minutes which is unacceptable for me, but that is something I can fix myself).


Alright so here is the situation:

TeamCity box is virtualized (on ESXi), and is pointed to a network time server that it CANNOT connect to (whoops, by default).

Over the course of the past few months and various downtime for upgrades on the hypervisor the kernel time clock has drifted due to virtualization, the clock has been moving forward.

Being as it checks between the latest minute or so interval, it wasn't reaching far back enough to find the checkins.


Pointed it at the domain controller for our network, everything is fine now.



Of course though, doesn't that mean that VSS won't work across a VPN that may span a timezone?


Everything seems to be working top notch now, sorry for the trouble, hopefully if this happens to anyone else they'll stumble on this thread.

0
Avatar
Eugene Petrenko
Created

Glad to read the problem is resolved.

TeamCity uses current time to query VSS history. If you run it over VPN connection with different timezones you need to ensure VSS report date and time in server's time zone.
It seems VSS2005 had such option in the settings.

0

Please sign in to leave a comment.