AccuRev VCS plugin for TeamCity

Thank you Francois!

Have been waiting a long time for AccuRev support in TeamCity so I'm very much looking forward to trying this.

Will let you know how I go.

Hey Francois,

My initial findings with using the plug-in:

Setup was pretty straightforward but I did have one issue on our Windows 2003 server, around authentication with AccuRev: it complained that the AccuRev session hadn't been authenticated yet despite the fact I had provided the appropriate username/password in the VCS root settings. I managed to get it to work by logging onto the server console and manually doing a "accurev login ".

After that I had no trouble setting up a project and build configuration using the new AccuRev VCS root. After the initial population of the work area, it was extremely fast in retrieving changes when I promoted to the stream specified in the VCS root settings. Check-out rules and trigger patterns worked a treat which was very encouraging.

However when promoting to any parent streams of the specified stream it didn't pick up the changes. I presume this is by design, as usage of the HIST command against a stream will only ever return changes directly on that specific stream. Is that the intended behaviour? If so, had you any thoughts about practises within TeamCity to support the AccuRev stream inheritance model?

Wish list for the plug-in:

• Ability to specify an optional "watch" Stream in addition to the Stream setting which the plug-in uses as a basis of the working area it populates. For example, say I wanted to watch for changes to Stream A, but I want TeamCity to actually populate the work area from a child stream; Stream B. In this scenario, Stream A would be the "watch" Stream.

• Throw error on overlap detection. I'm not sure how possible this is but it would be great if TeamCity could show an error in the project list if one of the VCS roots has an overlap (i.e. conflicting changes) that will require a merge in AccuRev. The idea being that overlaps can very often cause issues in builds.

• Poll based on AccuRev trigger. This may not be possible due to limitations in TeamCity, however it would be nice for the plug-in to only look for changes when told to do so by AccuRev, as opposed to polling the server at regular intervals for each VCS root (not sure if there are any issues yet with continuous polling in AccuRev).

I'm sure I'll think up some more later but again I'm really encouraged with the alpha and look forward to seeing this develop!

Hello Daniel,

Thanks for the feedback.

The way the AccuRev login works is a pain. Can you indicate which part complains about the login session? (e.g. change checking, checkout, etc.) The plugin checks if it is logged in and if not, it does a "accurev login ]]>" command.

The plugin use the "accurev hist" command to get the list of changes. So not getting upstream changes is an AccuRev limitation. I like your idea about the watch stream. That means we have a watch stream where we look for changes and a build stream that we check out.

The TeamCity server handles the polling for changes in VCS, the plugin just do the check when told to do so. But according to the TeamCity documentation a build can be triggered via a http GET command. One could add a GET to a server-post-promote-trig trigger on the AccuRev server to kick off a build in TeamCity.

Francois, thanks for this information. I wanted to give you the heads-up that AccuRev 4.7.1 will have enhancements to the command-line which will help you detect changes in a stream that have come through inheritance. This is really geared toward continuous integration applications and will greatly assist in plug-ins like yours.

Keep an eye out for the 4.7.1 release in the next couple of weeks.
Cheers,
~James Talbott
Senior Systems Engineer
AccuRev, Inc.

Thanks for the heads-up James! I was just about to upgrade the whole team to 4.7 so now I'll hold off a bit for this feature.

Hi Francois,

fgretief wrote:

The way the AccuRev login works is a pain.  Can you indicate which part complains about the login session? (e.g. change checking, checkout, etc.)  The plugin checks if it is logged in and if not, it does a "accurev login " command.

It seems to be every command that fails, even the "Test connection" button on the VCS root settings page. Is it possible that there are trailing spaces or carriage return from the CLI command under Windows that are affecting the condition? (if I look at the tomcat console there is a CR present in the response from secinfo).

fgretief wrote:

The TeamCity server handles the polling for changes in VCS, the plugin just do the check when told to do so. But according to the TeamCity documentation a build can be triggered via a http GET command.  One could add a GET to a server-post-promote-trig trigger on the AccuRev server to kick off a build in TeamCity.

That is definitely an idea. The only issue being that none of the checkout or trigger rules apply when you call that http GET command; you only seem to get that functionality if you use the poll-based approach (the http command is just like clicking the "Run" button in TeamCity). Given what you've said I might raise a feature request for the TeamCity guys to have a look at.

Daniel,

Actually, we already have the feature filed for VCS changes pushing. Please also see the issue comments for a way to configure the approach with the current TeamCity version.
Also, feel free to vote for the feature.

--
Best regards,

Yegor Yarko
Project Manager (TeamCity)
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

That's fantastic... until TeamCity has the smarts to trigger at the VCS root level I can extract the build type ID from the project/build configuration file anyway.

Thanks for pointing out that (presumably) undocumented feature!

Hi James,

Did this feature make it into 4.7.1? I can't see it mentioned in the release notes.

Thanks,
Jamie.

I might have a scenario that was never thought through.  I work in an environment which has multiple AccuRev servers (different machines).  The only way I can see a single TeamCity installation working with both servers is if every call included the server name parameters ("-H server:port").  In the AccuRev code for RunProcess, there exists a setter method called setSessionToken().  If this parameter is set, the server name gets appended to every AccuRev call in the method addSecurityArgsToCommand().  Without it being set, no server information is appended.  In the TeamCity AccuRev plugin, I see no reference to this method call.  Is there a reason why I can't find it?  Can someone look into this and add it?  I may end up playing around with adding it myself locally.

Thanks

Hello Derrick,

I have not given that  scenario much through because we only have one server with a single depot.  Anyway, the TeamCity plugin is working with the AccuBridge Java SDK v4.6.0, and there is no setSessionToken() method in the documentation of the RunProcess() class that I can find.  Also, I see no reference to addSecurityArgsToCommand() either. Do you have a newer version of AccuBridge Java SDK?  That is the main reason why it's not part of the TeamCity plugin, I don't know it exist. ;-)

You are welcome to adding it to the TeamCity plugin. I can help if you need any. I would love to see how it works. The whole login/authentication scenario of AccuBridge is still a bit unclear to me due to lack of examples and documentation. Only got the AccuBridge API reference docs to work with.

Cheers
  Francois

Hi Francois,
I rarely read documentation for the reason that it is oh so often incomplete.  I go straight to decompiling.  I am using the exact same version of the AccuBridge library that you've included in the plugin.  I'll email you directly my code.  I worked on this yesterday and got it working.  Look for files with a July 13th timestamp and do a diff between them and your version to see what I've changed.  If you merge the changes into the trunk and create a new build, let me know so that I can convert back to your build.

thanks

Hi Francois,

Regarding the "inherited changes do not appear in the stream" issue, have you had a chance to look at implementing the enhancements to the ACCUREV DIFF command?

As of 4.7.2 you can now get a full list of changed elements betwen two transaction references, including all of the inherited changes. The command you use goes something like this:

accurev diff -a -fx -i -v [STREAM_NAME] -V [STREAM_NAME] -t[TRANS_START]-[TRANS_END]

e.g.

accurev diff -a -fx -i -v MyStream -V MyStream -t10001-10003

This will return an XML list of elements from the specified stream as well as all parent streams. Note that the same [STREAM_NAME] is specified twice in the command.

The workaround we've been using for this has been to write our own "check out" script using an AccuRev Reference Tree, but these are not without their problems!

If you're time poor at the moment I might be tempted to have a go at this myself. Overall my team has found this plugin invaluable, particularly in unlocking the build triggering/vcs root rule functionality in TeamCity. Thanks again for your efforts on this.

Dan

Hello Daniel,

No I have not looked at it until now. I was still on v4.7.1.  Thanks for mentioning it. Looks like just the feature we need.

Where is it documented that you can add the -t for the transactions?  I looked in both the online help as well as the AccuRev CLI pdf, but there is no mention of the -t option.

Also, do you by chance know what values are possible for the What attribute in the Change element of the XML output? I found: "created", "version", "moved", "not found". Is that all?

I hope to work on this during the comming week-end, but I must admit, time is a bit tight.

Cheers
  Francois

Hi Francois,

They're all the possible What values that I could find, looking at our own stream histories.

It is kind of strange that AccuRev have failed to document such an important feature properly. I think I got the syntax from the release notes.

This may just be me venting but overall there seems to be a bit of a lack of a focus on Continuous Integration at AccuRev, as evidenced by the fact the company hasn't released a single AccuBridge for any CI server.  I find this particularly ironic considering the software is promoted as a best-of-breed product for "multi-stage Continuous Integration".

Having said that I would never want to use another SCM tool.

Dan

Francois,

I've had a try at incorporating the AccuRev DIFF command into the plugin.

One challenge I encountered is that the command actually returns a flat list of changes for a given Transaction range. What we really want to know is on what stream(s) the changes were made, when, by whom, etc.

The solution I implemented is as follows:

1. When the "CollectBuildChanges" method is called, it performs the DIFF command using the "from Transaction" to the most recent Transaction ID within the entire depot.
   
2. If any Changes are returned, it performs the HIST command on the specified stream (as per the current version) using the same Transaction ID range, and attempts to match each DIFF Change against a Transaction that is returned. Any found Changes are flagged as "matched" and added to the Modification collection.
3. If there are any Changes left that are not flagged, then it recurses up a level in the stream hierarchy and repeats Step 2. The recursion continues until either the root stream is encountered or all Changes in the collection have been flagged.
4. If there are still Changes left after encountering the root node, it adds the remaining Change items to a single Modification item with the description "Other Inherited Changes". This will tend to happen if you make a change to a stream, e.g. changing the stream's basis time.

We've been trialling/testing the changes in our shop for the past week and so far so good. In particular the devs really appreciate being able to see the inherited parent stream changes flow down to their builds. One nice side effect of this is that having a "watch stream" is now possible, due to the fact the plugin will now collect changes all the way up the stream hierarchy. Also, using AccuRev include/exclude rules on the watch stream has become a very nice alternative to TeamCity VCS checkout rules because it will directly affect what gets populated in the TeamCity work areas and also in the AccuRev DIFF lists.

One last thing I need to do is to get my site working with Derrick's "set token" changes (for some reason it won't authenticate for me at the moment). Please let me know if you'd like to try the changes and possibly put them into the trunk.

Cheers
Dan

Pleased to announce the following changes have now been committed:

- INHERITED CHANGES SUPPORT ADDED
Promoted changes inherited by the VCS root stream are now included in the TeamCity changes list (previously only promotes directly affecting the VCS root stream were included). Any other changes, such as reverts, stream basis time modifications, or changes resulting from re-parenting of a stream are grouped into a generic change group with the comment "OTHER INHERITED CHANGES".

- DEFUNCTED/REVERTED ELEMENTS SUPPORT ADDED
Versions of files or directories that are removed from a stream will now show as removed or edited (depending on the outcome) in TeamCity.

BUG FIXES
- AccuRev plugin error "Index: 0, Size: 0"
http://youtrack.jetbrains.net/issue/TW-6883

- AccuRev plugin error "Unable to find the last transaction ID!"
http://youtrack.jetbrains.net/issue/TW-6884

- (TBC) AccuRev plugin update problems
http://youtrack.jetbrains.net/issue/TW-8017

- Changes to existing elements in AccuRev were sometimes showing as "added" elements instead of "edited" (occurred the first time an existing element was promoted to a stream).

I just tried upgrading our TeamCity server to use the newest version of the AccuRev plugin (we were previously using 0.2), and this version breaks for me.  When it tries to run the "accurev update" command, the command fails and prints its help page as though it doesn't recognize the arguments that are being passed to it; I think it doesn't recognize the "-p" option (at least, I don't see that option in any of the official documentation, and it fails when I try to run it manually).  This leads to the AcRunCommand class getting an IndexOutOfBounds exception when it tries to parse the XML output.  I made a bug case here: http://youtrack.jetbrains.net/issue/TW-9872

Any idea what might be going on?

This may have something to do with the plug-in's handling of the "Clean all files before build" option on the VCS settings page, but I'll need to confirm that after some investigation.

Confirmed the issue was in the 'clean build' option; was a victim of some last minute refactoring of mine.
Now fixed in latest build: http://teamcity.jetbrains.com/viewLog.html?buildId=26515&buildTypeId=bt138&tab=artifacts

The fix worked for me, and everything seems to be fine now. Thanks!

A bug which can cause failure of change list population and prevention of build execution has just been fixed.
We discovered the issue this week when element ID of a directory/file changed during a 'move' operation.

Build available here: http://teamcity.jetbrains.com/viewLog.html?buildId=29123&buildTypeId=lastSuccessful&tab=artifacts

I found and submitted another bug a few weeks ago: http://youtrack.jetbrains.net/issue/TW-10438

I was just curious if you had seen that, since there haven't been any official comments on the case.

Hi Phillip,

Apologies for that, unfortunately I don't have a way of subscribing to those YouTrack bugs at the moment.

Regarding the issue, I'm not entirely sure what's going on here. Can I suggest you try manually running this command on your TeamCity build server and let me know if the permissions are retained?

accurev pop -H [accuRevServerName] -fx -R -O -v [streamNameToPopulateFrom] -L [directoryToPopulateInto] \.\

e.g. accurev pop -H accurev:5050 -fx -R -O -v FR -L E:\Program Files\TeamCity\temp\accurev6205070683 \.\

This is the same command that the plugin uses to initially populate the vcs root's working area.

By the way I've noticed that TeamCity seems to populate a separate directory to where the build actually takes place, i.e. it first populates a TEMP directory similar to the one mentioned here and then transfers those files over to WORKING. Not sure if anything is being lost in that file transfer but it's something to bear in mind.

Dan

Manually running "accurev pop" does produce files with the right permissions.

I'm not really familiar with how TeamCity moves around files internally, but we have a central build server and several build agents that connect to it; based on what I've observed, it looks like the "accurev pop" command is run on the TeamCity server and then it uses some other mechanism to copy the files over to the agents where the compilation is actually done.  Perhaps the executable flag is being lost during that transfer?

That's interesting. Just to be absolutely sure about this though, do you use the "clean all files before build" option on the Version Control Settings tab (this causes the ACCUREV POP command to be called every time, instead of an UPDATE)? If not if you could try that and confirm the permissions are still being lost... that would be a good case for JetBrains to take this up.

Currently the "Clean all files before build" checkbox is disabled for all of our configurations.  For what it's worth, I can verify that in all of my normal AccuRev workspaces, running the "update" command doesn't cause files to lose their permissions.

I can try it, but even if it works, I don't think that will really be feasible for us in the log run... we have several projects with multiple build configurations each that have source trees that are several GBs in size, and repopulating the tree every time a build is triggered could cause some serious performance issues.

That's cool, I was mainly interested just to try it to see if the problem was limited to the UPDATE command.

So if its not the update that is responsible, the culprit is more likely to be the plumbing between the agents and the central build server. Unfortunately this is all controlled by TeamCity so I'm not sure how much can be done with the plug-in. Perhaps this is worthy of a broader post on the TC forums?

Daniel,

Actually, there seems to be a way to subscribe to issue events in YouTrack.

You can try the following:

• Search for the issues you are interested in (e.g. #{3rd party: AccuRev plugin} #Unresolved)
• Click "Save this search" below the search field and give a name to the search.
• Go to your Profile, Filters and Notifications and tick necessary options for the Saved Search.

Please have a look to the comment for the mentioned issue at
http://youtrack.jetbrains.net/issue/TW-10438#comment=27-92865

Actually, TeamCity provides PatchBuilder interface for tha plugins to build a patch.
This is done on the server.
The patch is send to the build agent and that executed in the same order as it was built.

The command of updating a file contains fileMode arguement. If that argument != null, build agent will run chmod command for the local file.
Does AccuRev plugin use this attribute?

On the other hand, agent side checkout will let plugin do update local filesystem.