Questions about TeamCity changes gathering and dates with ClearCase (base) SCM

Hi!

As I see no activity on these questions I'm wondering... am I right ?

Regards,

Olivier.

Olivier,

(a). Yes, the changes are grouped now by user name, date and time. There is a request in our tracker for grouping changes by activity (http://jetbrains.net/tracker/issue/TW-4760) - you can vote for it or create your own request.

(b). All dates came from the result of the command "cleartool lshistory -all -since <DATE> -fmt %u#--#%Nd#--#%En#--#%m#--#%Vn#--#%o#--#%e#--#%Nc#--#%[activity]p###----###\n <PATH>" ("%Nd" key returns the timestamp of the operation). TC does not change the result of this command. What does this command return in your case? And where do you see another date?

(c), (d). Can you please provide me TC debug logs containing the timestamps of these errors and VCS root settings for investigating these problems?

(e). Labeling problems are already fixed in TC 4.5.3.

There are also many ClearCase integration bugs that were fixed in TC 4.5.3 so I recommend you to upgrade your TC and try to reproduce your problems after it.

Hi Maxim and thanks for your answer,

(a) It's OK. In fact, for grouping changes, ClearCase UCM activity feature should be supported by TeamCity.

So I voted for it (even if we do not use UCM yet, but only ClearCase base!).


(b) On my server, it seems that this command returns a file list with element that comply with the since date attribute, no?

I used to noticed that change time presented in TeamCity interface is ClearCase creation version time + 1 second (seen in ClearCase GUI).

(c), (d) I do not have these log any more but I will kept them if it would be reproduced.

And I think that I will make some tests again after the server be upgraded to 4.5.4.

(e) It wasn't a labelling problem, but a cache problem. When removing an element version in ClearCase, TeamCity seems lost.

See http://www.jetbrains.net/devnet/message/5241352#5241352

Regards,

Olivier.

Olivier,

(b) Can you please provide me the result of this command (that includes right change date) and some screenshots of TC UI where the wrong date appear?

(e) Did you test it on TC 4.5.3 - 4.5.4?

Hi Maxim,

(e) I did not test with TeamCity 4.5.3+ yet.

(b) I reproduced the test, just making a minor change on a file. So here is some more info.

On ClearCase snapshots, the version creation date is 10:39:48.

On TeamCity snapshots, pending changes and build associated changes, the "date" is 10.39.49 (+1 second).

Here is the result of the command "cleartool lshistory" that retrieve an element:
[2009-07-15 10:40:00,222]   INFO [VCS changes loader 1] - .clearcase.ClearCaseConnection - simple execute: cleartool lshistory -all -since 15-July-2009.10:38:19 -fmt %u#--#%Nd#--#%En#--#%m#--#%Vn#--#%o#--#%e#--#%Nc#--#%[activity]p###----###\n E:\CCase\odassaud_tcdev_view\SET\Sandbox\AnotherConsoleApplication
[2009-07-15 10:40:00,456]  DEBUG [VCS changes loader 5] - .clearcase.ClearCaseConnection - result: ErrCode=0
StdOut:------------
odassaud#--#20090715.103948#--#E:\CCase\odassaud_tcdev_view\SET\Sandbox\AnotherConsoleApplication\Build\ProjectSpecific.proj#--#version#--#\main\3#--#checkin#--#create version#--#Another change only for testing changes in TeamCity#--####----###

StdErr:------------

End----------------

Full log files are attached also.

Hope it will help.

Regards,

Olivier.

Olivier,

Sorry, I have a vacation for two weeks. I will continue the investigation of your problem when I return.

OK, it' no problem for me.

Have a nice time. Enjoy !

Olivier.

Hi,

I've just got a problem with TeamCity 4.5.4 on Windows Server 2008 x64 and ClearCase 7.1.0.0.

It's about the first change on a new TeamCity platform.

TeamCity reported the change but with the wrong comment. Instead of the last comment, it took the previous one.

After restarting the server, the same pending change stayed.

So I stopped the server, cleared BuildServer\system\caches\clearCase

and updated ClearCcase snapshot view manually

After restart, I had got the same problem.

I then do the same but cleared also search and sources in system caches folder

Same problem.

I launched the build.

After, I've made another change.

And this one is well taken into account by TeamCity.

As, I used to raised this kind of problem before, I took time to gather some log and snapshots for investigation.

Here are 4 snapshots with comments:

On "Detect change pb Clearcase version.gif", the new version is "Remove comments" (\main\6). The previous one is "Basic simple comment..." (\main\5).

On "Detect change pb TC interface.gif", a change has been detected

2009-07-29 11h08 "Basic simple comment..."

The comment description doesn't match the change date (see ClearCase snapshot later)

On "Detect change pb TC interface with file version.gif", the file version corresponds to the last change (\main\6) but not match the comment.

On "Detect change pb Diff seems OK.gif", diff changes is OK between 5 and 6 versions.

I attached the "teamcity-vcs.log" file zipped.

I tried to understand what happened, see the teamcity-vcs-extracted.log file.

I wonder why the first "cleartool describe" (simple execute) has a parameter with the version "...ProjectSpecific.proj@@\main\6"

and that the folowing "cleartool describe" (interactive execute) has only an element name.

That last command returned the wrong comment! Is there a loss of information that caused the problem?

simple execute: cleartool describe -s -pre E:\ccase\ic_prod_set_view\SET\Legacy\GOOSEObs\Build\ProjectSpecific.proj@@\main\6

interactive execute: cleartool describe -fmt %c -pname E:\ccase\ic_prod_set_view\SET\Legacy\GOOSEObs\Build\ProjectSpecific.proj

output line read: Basic simple comment modification to validate labeling with TeamCity new ClearCase plugin

output line read: Command 7 returned status 0

What about your analysis ;-)

Regards,

Olivier.

Here is the 6th attachement. The complete log file.

Hi!

Here is anothe sample of change comment concatenation with TeamCity 4.5.4 and ClearCase 7.1.0.0.

Regards,

Olivier.

Hi,

Here is some more info on the wrong comment concatenation in changes.

I’ve noticed that this happens when snapshot view accessed by TeamCity server isn’t up to date. So the comment that is (abnormally) added to the last change comment comes from the last version loaded in that snapshot view.

For example let’s take a file element in \main\2 version, added with “comment 2” description, and two snapshot views up to date, one on a developer workstation and the other on TeamCity hosted server.

A \main\3 and a \main\4 element version with “comment 3” and “comment 4” are created on workstation view.

So two Build are successively launched by TeamCity, respectively with \main\3 and a \main\4 versions, and “comment 2comment 3”, then “comment 2comment 4” description.

After updated manually the snapshot view on server, next builds are OK (synchronized): \main\5 version with “comment 5”…

So my theory is that sometimes ClearCase snapshot view update isn’t performed by TeamCity plug-in for ClearCase, or it does fail.

Hope it would help you (with the previous log files and screen snapshot files).

Regards,

Olivier.

Hi!

I've reported this problem as a bug in the tracker:

TW-9043 VCS update problem with ClearCase causes change description anomaly

http://www.jetbrains.net/tracker/issue/TW-9043

...and maybe it can be linked to another one about ClearCase support:

TW-9089 TeamCity seems not care ClearCase config spec labels

http://www.jetbrains.net/tracker/issue/TW-9089

Regards,

Olivier.

Hi!


I've just investigated why TeamCity plug-in didn't make some update on the server snapshot view... that seems a cause of my problems:
see TW-9043 http://www.jetbrains.net/tracker/issue/TW-9043 and http://www.jetbrains.net/devnet/message/5242893

It seems that the only code that performs a ClearCase update is dead, as the method "public static void updateView(final String viewPath) throws VcsException" from "jetbrains.buildServer.buildTriggers.vcs.clearcase.ClearCaseConnection" class is never called.

When looking at the "public void updateCurrentView() throws VcsException" method code, I've seen that it is empty, as the call to the previous method is commented:
//updateView(getViewWholePath());

I activated this code again (note that I do not know why it was commented as I didn't analyzed all source code).

Then I changed the jar on the server: it now performs snapshot view updates. Changed comments concatenation problem seems solved TW-9043.

So, could you explore this way as a corrective patch for ClearCase plug-in? and make this patch available as I do not have a released build (see the following explanations)


Also, regarding the build of the ClearCase plugin I had two problems:

a) First, in "jetbrains.buildServer.buildTriggers.vcs.clearcase.CCPathElement" and in  "jetbrains.buildServer.buildTriggers.vcs.clearcase.versionTree.VersionTree" some code has been changed to use a new split method, change on July 29, 2009.

The problem is that I couldn't find the corresponding StringUtil.split() method from TeamCity 4.5.4 or 5.0 EAP (that from  July 21, 2009). Maybe it's not yet available to the community.

So I had to get back on previous revisions of these files (735 and 1035) to make a build.

b) Second, the build isn't operational when using Ant files. After that the path.variable.teamcitydistribution property has been set, some libraries are missing:

I added the following pathelements:
${path.variable.teamcitydistribution}/buildAgent/lib/jdom.jar
${path.variable.teamcitydistribution}/lib/servlet-api.jar
${path.variable.teamcitydistribution}/webapps/ROOT/WEB-INF/lib/spring.jar
${path.variable.teamcitydistribution}/webapps/ROOT/WEB-INF/lib/spring-webmvc.jar
${path.variable.teamcitydistribution}/webapps/ROOT/WEB-INF/lib/server.jar

Also, as I compiled with a 1.6 JDK, I had to set the flag target="1.5" in the javac task (in clearcase-standalone.xml) or the plug-in wasn't loaded by the TeamCity server.

I used to raised this problem before. See: http://www.jetbrains.net/devnet/thread/281359


Best regards,

Olivier.

Hello,

I notice also that changes time in TeamCity is set to one second later than the "create version" time. So where this value does-it come from?

It seems that TC really adds a second to the time got from ClearCase but I still can not understand why this is done.

I've just investigated why TeamCity plug-in didn't make some update on the server snapshot view... that seems a cause of my problems:
see TW-9043 http://www.jetbrains.net/tracker/issue/TW-9043 and http://www.jetbrains.net/devnet/message/5242893

It seems that the only code that performs a ClearCase update is dead, as the method "public static void updateView(final String viewPath) throws VcsException" from "jetbrains.buildServer.buildTriggers.vcs.clearcase.ClearCaseConnection" class is never called.

When looking at the "public void updateCurrentView() throws VcsException" method code, I've seen that it is empty, as the call to the previous method is commented:
//updateView(getViewWholePath());

I activated this code again (note that I do not know why it was commented as I didn't analyzed all source code).

Then I changed the jar on the server: it now performs snapshot view updates. Changed comments concatenation problem seems solved TW-9043.

So, could you explore this way as a corrective patch for ClearCase plug-in? and make this patch available as I do not have a released build (see the following explanations)

Thank you for the investigation. I think this code was commented by mistake. I attached the patch (clearcase.jar) and the "common.jar" file that contains new "split" method. Please try to replace your "<TeamCity Home>\webapps\ROOT\WEB-INF\plugins\clearcase\server\clearcase.jar" and "<TeamCity Home>\webapps\ROOT\WEB-INF\lib\common.jar" with attached files. Please backup your versions of these files to be able to restore them.

Hi!

Thanks for the patch. I've just tried it. I've copied the two jar files (clearcase.jar as a replacement and common.jar as a new file).

It seems KO: no such method error with the split() method.

Maybe I should remove the existant common-api.jar file, but it seems that some packages are missing in the common.jar (in comparison with the common-api.jar file) so that I could have problems on other parts...

So I think that I will let my patch in place, that seems OK, till TeamCity 4.5.5, no?

Regards,

Olivier.

Hi Maxim & Oliver,

Thanks so much for posting this thread I thought I was the only one having this problem! We have only recently upgraded our large projects to TeamCity to 4.5.4 and have been having all sorts of ClearCase issues (we use BASE not UCM)..

After applying one patch TeamCity will now detect changes that occur on the snapshots, but as you mention it will not perform an update on the snapshots before doing the builds so while it knows it needs to rebuild it uses the same stale source files so build issues do not get resolved.

I've put in the new ClearCase.jar (and added Common.jar as a new file) as Maxim suggested but then get the following:

Unable to collect changes for [Trade Capture Shared Solution]
jetbrains.buildServer.vcs.VcsException: Error collecting changes for VCS root 'clearcase: C:\TeamCity_Snapshots\abowker\snap_TeamCity_Shared.abowker\coretech_scm\CORETECH_TRADE_CAPTURE_SHARED' #6
java.lang.NoSuchMethodError: jetbrains.buildServer.util.StringUtil.split(Ljava/lang/String;Z[C)Ljava/util/List;

    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:43)

So then as Oliver has suggested tried backing up my common-api.jar and renaming Maxim's attached common.jar as common-api.jar but then I couldn't even start the server.

Does anyone else have any suggestions on how to fix this?

Surely other people must be reporting this issue? Everyone I know that uses Team City in a corporate environment uses ClearCase with BASE and TeamCity 4.5.4 is currently broken for everyone with this configuration?? When can we expect a patch?

Oliver is there any way you can supply your patched ClearCase.jar that doesnt use the split method for us to use in the meantime?

Any help would be most appreciated.

Thanks in advance

Regards
Ainsley

Hi!

Here is the jar file I've built and that seems to work for teamCity 4.5.4 and ClearCase base.
I've attached also the modified source code file.

Now, the update is called: updateView(getViewWholePath());
I do no more delete ClearCase log file about update: FileUtil.delete(new File(viewPath, UPDATE_LOG));
And there is some more log messages.

Regards,

Olivier.

Hi Olivier,

Thanks very much!! Your patched ClearCase.jar works perfectly for us and has resolved our issues - the extra logging is also very useful.

I hope this gets incorporated into a 4.5.5 patch as I dont think its an issue that can wait until TeamCity 5.0 is released to be fixed.

Much appreciated.

cheers
Ainsley

Hi Olivier,
Dumb question maybe. I'm running 4.5. I have snapshot views for clearcase base. I've been having problems where if I click on 'check for pending changes' after I've checked in sources (35 files), that no changes are being found. Sometimes teamcity finds the changes but sometimes it doesn't. Would your fix help this? Or should I wait for 5.0.

Thanks in advance.

Nick.

Hi Nick!

I do also get some problems with TeamCity plug-in for ClearCase (just see this forum or JetBrains tracker!) and TeamCity source files caching.
I do not know if the fix would help you for that specific problem.

My opinion is that this plug-in lacks maturity right now.
And, as a separated open-source plug-in (shipped with TeamCity), I don't know if version shipped with TeamCity 5.0 will be really different.
I HOPE IT WILL!

See activity on the SVN plug-in repository and forge your own opinion.

Regards,

Olivier.

Hi Olivier,
I'm gonna download the 5.0 EAP build and give it a kick. I've there's any improvement I'll post it here.

Have a good weekend!

Nick.

OK. Thanks. Nice week-end too.