AccuRev VCS plugin for TeamCity

I've seen another issue occur where occasionally, the sources on the build agent will stop updating, even though the TeamCity GUI indicates that the sources have been updated and rebuild.  I went ahead and entered a case here: http://youtrack.jetbrains.net/issue/TW-10825

Unfortunately, I haven't yet figured out how to reliably reproduce it, but I have seen it happen a few different times, so it definitely wasn't a one-time fluke.

Has anyone tested the latest plugin with TeamCity 5? We recently setup a TeamCity 5 server, running under Linux, I have installed the latest AccuRev plugin, but it doesn't seem to detect changes anymore. When I click build it does seem to pick up the correct source code, but the monitoring of changes doesn't appear to be working.

Previous I was using TeamCity 4.5 on a Windows box, so there are a number of things that have changed now - not sure which one has caused this issue?

Any ideas?

Steve

Hi Steve,

Actually I'm not sure myself since we haven't upgraded to TC 5.0 yet (we're still on 4.5).

I've just installed 5.0 on a test Windows server so will set it up and see if it still works. Will let you know how I go.

Cheers
Dan

Fantastic.

Thank you.

Hopefully your endeavours with your test Windows box will issolate if the issue is with TeamCity 5 or with the fact I am now running a Linux based server now.

Steve

Just to keep you in the loop, we have tested it on a Windows test server (running 5.x) and it appears to be working, so I wonder if the problem is specific to Linux. Do you know if anyone has run this plugin with a Linux server before? We have compiled the plugin ourselves, are there any areas that spring to mind that might have issues with Windows/Linux differences that we could look at?

Steve

For what it's worth, we're running TeamCity 4.5 on an Ubuntu Linux 8.04 server with the AccuRev plugin, and we haven't encountered the issue mentioned above.  In fact, the issue we have is practically the opposite -- sometimes it will detect changes in the source stream but not actually download the new source code. :-)  The case is detailed here: http://youtrack.jetbrains.net/issue/TW-10825?query=%23{3rd+party%3A+AccuRev+plugin}+%23Open

Doh, got to the bottom of it, I was pointing it at our old AccuRev server, Oops!

Sorry for the confusion, we can now successfully debug the AccuRev plugin in Eclipse now though.

We don't actually use the AccuRev plugin to checkout our code for us, as it takes ages (about 20 minute), we just use it to monitor streams and decide when builds should run, then from inside the build runner we checkout the code manually (which takes about 40 seconds) - so we may extend the plugin to add client side checkout support. If we get this done in a half decent manner I will try and get it shared back on here so others can make use of it, should they wish to.

Steve

I'm glad to hear that Steve, thanks for the update!

Regarding the checkout problem, I have finally got off my lazy behind and raised this with AccuRev.

One thing that may really help is if anyone experiencing the problem could post snippets of their "stdout_yyyymmdd.log" file (available in the "<TeamCity Data Dir>\Logs") from around the time that the bug occurs. It might be a bit of to-and-fro to get this one resolved.

For anyone interested, the workaround for this we use is to create a sub-stream under the stream we want to checkout from. We then add include/exclude rules to that sub-stream so that only the elements that are required for that particular build are included in the checkout. We then setup the TeamCity VCS root to monitor that sub-stream, which in turn watches all of its parent streams. This generally results in much faster build times because only the required files are monitored for changes and retrieved during the "clean build", rather than the entire depot. This has helped us keep all of our build processing times < 4 minutes total. I can understand that this may not work for everyone, though.

What we do is create a temporary stream (or multiple temporary streams if there are multiple concurrent checkouts), and then parent said stream to the stream we are interested in, and then set a time basis to the transaction ID we are interested in. We then use the populate command to get a copy of the code. As I mentioned, this whole process normally takes around 40 seconds on our new server, where as we find the plugin takes substantially longer. We have always found this with the plugin, I wondered if it was to do with the fact our checkout side is about 1 GB so it just took a while to do what ever it is it currently does (as it is doing it server side, I am presuming it has to send the files back over to the agent, not sure what overhead is there?).

Steve

I have augmented to AccurevPlugin to support agent side checkout.
This has reduced the amount of time required to perform a checkout from 16 Minutes to under 2 minutes.
We would like to get theses changes integrated into the official plugin and were wondering how to proceed.

Please advise:
Thank you

Hi Abraham,

That's fantastic, if you'd like to send on your source code changes to me I can check them into the trunk.

Thanks,
Dan

For what it's worth, I would be eager to help test this if any help is needed. :-)

A quick question about the implementation of agent-side checkout: does it account for AccuRev possibly being installed in different locations on different build machines?  On some of ours it's in /usr/local, on others it's in /opt... it'd be great if it looks for it either in the PATH or if there's an agent-specific property that can be set to indicate where it is.

the Build agent expects accurev to be on the path.

We are just putting the finishing touches to the plugin.
I will send you the source code once we have eliminated all the major bugs.
Thanks

I have attached all the files to this post

Thanks!  I upgraded the plugin on our server and so far everything seems to be working well.

Fantastic. We have had it running on our test server for more than a day now, and so far, so good. Up until now we have had all the AccuRev populate code living in our build runner, so it is nice to now have everything done in the right place. A few of the changes Abe has made are reasonably specific to our requirements (being able to have separate watch/populate streams, being able to specify a populate subdir in the VCS node, doing an agent side populate from a time based stream) but hopefully they'll work well for everyone else too.

Initially I wasn't very keen on having these 'temporary' streams created for Teamcity agents, but now I find it quite useful as you can see (in AccuRev) exactly what code an agent is checking out, and even attach a workspace to the stream and get the code yourself.

The two remaining things on our potential improvements list is:

- Being able to trigger an AccuRev promote upon successful build/tests, although right now we haven't found a way to do this.
- Adding personal build support - right now it seems this is quite linked in to IDE's so isn't so straight forward to do. But at some point we'd like to be able to send up a users workspace for building/testing

If anyone has any other ideas of useful features it'd be worth while posting on here, be interested in seeing what other things could be handy!

Steve

Based on my observations, it seems like when build configurations are set to checkout-on-agent mode, every time a build runs it re-populates the directory from TeamCity's build stream.  This can take a while for large streams, and waiting a few minutes for it to repopulate when only a few files have changed seems a bit excessive.

It seems to me like it would be a good idea to, in addition to creating a stream for each build agent and each VCS root, also create a reference tree off of that stream so that the "accurev update" command can be used to quickly change between revisions without repopulating the directory.

We haven't used reference trees for anything here; we have about a 1GB populate from AccuRev, which takes around 1 minute; I have experimented with using a workspace instead, but for us, the amount of time it takes to completely revert the workspace (and remove external files and so on), to get the workspace back to its populated state, took as long or longer than doing a fresh populate. From what I have read about reference trees so far, it sounds like they'd have the same issue.

What size checkout do you have? How long does it take? How long does it take you to fully revert a reference tree to get its back to it's populated state? It'd be interesting to compare notes.

Steve

Not all of our build agents are on the same network segment as our AccuRev server (in fact I don't know if any are...), and some of them have to go through 100 Mb/s links, so it's not necessarily very fast -- we've got one 1.1 GB source tree that I regularly see 3-4 minutes to populate, and a 3.6 GB tree that takes 10-12 minutes.  There are other trees, too, but those are getting the most activity at the moment.  That same 3.6 GB tree has 61 build targets that depend on it, so it takes a long time to do a complete build if it repopulates the work directory for every build. :-)

Maybe I'm not clear on something -- is the AR plugin cleaning out the source tree between every build?  I wouldn't expect that to happen unless the "Clean all files before build" flag is explicitly turned on.  In fact, we have a few build scripts that expect the artifacts from the previous build to still be there.  (I know, it's bad, but they're legacy scripts that were in place long before we started using TC, and we haven't had the time to rewrite them.)  If the agent-side checkout code was using a reference tree for building, I would expect it to just do an "update" between builds and leave any modified or external files in place unless it was specifically told to clean the agent's sources.

In practice, if we ignore any modified or external elements and simply use "accurev update" to move between different versions of a workspace or reference tree, how long it takes only depends on how much has changed; it only takes maybe 10-15 seconds to update a workspace that's had only a few changes.  In the worst-case scenario, it takes about as long as a normal full populate would.  We have our build configurations in TC set up to automatically rebuild whenever something in the appropriate directories has changed, so usually changes to the stream between builds are minimal; when using the checkout-on-server mode, it usually takes maybe 10-15 seconds for the server to acquire the changes and copy them over to the build agent.  I haven't done any specific testing with reference trees, but I would expect them to be about the same; from what I understand, they're basically like read-only workspaces that are specifically intended for automated building.

If you actually need to do a complete wipe and repopulate of a workspace with the latest code from its parent stream, probably the easiest way to do that is to just delete the whole directory, then run "accurev update -9" and then do a populate.  If you're not familiar with "update -9", it's an undocumented flag that marks a workspace as being "updated" in AR's database but does not actually change any files on the disk.  Its use is usually not recommend, but sometimes it's helpful.

Just out of interest Steve, what is the specific requirement you have for different streams for populate vs watch?

I only ask because the plugin already checks all of the parent streams (presuming your populate stream is a child of your watch stream). Is the goal to avoid detection of changes on the populate stream?

We have a reasonable sized code base in AccuRev, so we have filter streams to get rid of a lot of unnecessary content for a given project, and then have everyone parent off of the filter stream. The problem we found previously was when attached to the filter stream we didn't see changes from above, but as you say, this has now been fixed, but something else we (well Abe) is adding for support for now is the ability to promote on success (optional), which wouldn't work if we just pointed directly at the filter stream (there'd be nothing to promote in the filter stream). So we end up having it something like:

Watch : ProjectA_GAMEBUILD
Populate : ProjectA_GAMEBUILD_Filter

And if that succeeds, it automatically promotes the code in ProjectA_GAMEBUILD, up to the transaction tested.

Maybe there is a better way to do all this? It is what we have been doing up till now (with out own batch files to handle the AccuRev side), so we are just trying to get it a bit more integrated with the plugin.

On a side note, we have noticed the 'seeing changes from above streams' doesn't stop at snapshots (not sure about time based streams) - we were going to have a look ourselves at fixing this, unless it is something you are already fixing, or know can't be done due to AccuRev limitations etc.?

Steve

I have made some enhancements to the accurev plugin.

I have attached the changelog and the source code to this post.

Appologies for the lack of details in the change log.
I made the changes over many weeks/months and some of the details escape me.

Any problems just let me know.
Thanks

Thanks Abraham,
I'll take a look at the changes you've provided.

By the way, I've meant to mention that I've been in contact with AccuRev regarding the incremental workspace update problem (i.e. the issue that occurs when the "clean all files before build" option is not checked).

They have indicated that the method that was implemented in the plugin is not supported. The method that is used is basically a single "ACCUREV UPDATE" CLI command with a transactional range (From Transaction A to Transaction B/C/D/E/etc), however AccuRev have indicated that only one transaction can be specified at a time using this command.

This would explain why on some occasions changes don't get retrieved from the repository, because in order to do so you would need to make individual calls to "ACCUREV UPDATE"  to retrieve changes in Transactions B, C, D and E.

I'll have a look at implementing this change when I have some downtime, unless of course someone else is keen to have a go!

We are just getting started with TeamCity but we use Accurev very heavily so are interested in contributing to development of the Accurev Plugin.

We have around 10GB of source code/data that we push around via (yeah, I know, too much really) so having efficient Agent-side updates is going to be a big concern for us.

We're still getting set up for plugin development etc but if there are pending things to do that people haven't got time to look at, please let me know about them as we will have some resources to contribute to this :)

Cheers,

Mark

10 GB, wow that's quite a lot. What is it you work on, if you don't mind me asking - I'm just curious!

The agent side plugin was just developed to meet our needs initially, which is to do a fresh AccuRev populate of the given stream at the given transaction. We have about 1 GB of source code and it takes a minute or two to populate it, I don't think you'd want to do that every time with 10 GB of data. So I think the first thing you'd want on the agent side checkout side of things is to implement an incremental checkout option. We were going to look into this by using reference trees, but as of yet we haven't started on this task (and hadn't got it penciled in for anytime too soon). Ideally this would also make sure the source tree ends up correct, e.g. if the build processes modified a sources file, that source file should be reverted etc.

There are probably a few other enhancements to the plugin that could be made, if I think of any good ones I'll post here!

Steve

We are part of Disney and make Racing-type games on PS3 and Xbox 360. Thus far (for Disney) we've done Split/Second: Velocity and Pure: http://www2.disney.co.uk/disneyinteractivestudios/blackrockstudio/

My role as part of that is being Lead of the Infrastructure team that supports our game teams with code/data builds, web apps and maintains a pool of consoles and blades for lots of automated builds and tests. This is a bit of a growth area for us as we've gone from 4 build machines about 3 years ago to 40+ build blades and 24 test consoles. We do a lot of automated testing ;-)

A lot of our accurev workspace size is taken up by 3rd party libraries, which we tend to check in for easier deployment (e.g. things like Havok, Qt, Fmod and platform SDKs). Right now that's about 70% of the workspace, so we do have a plan to move over to a more linux-style 'package' system that would manage installing those libs on-demand on a machine and sharing them between multiple workspaces.  When you have multiple workspaces on a single machine, it eats up a lot of space right now.

We have a lot of our own source code too, as well as some (small) test assets that get checked in to be used in unit tests, so that all adds up and we'd still have about 3GB of source 'code' even after stripping out out 3rd party libs.

We do have an existing continuous integration system that is a bespoke one - it's basically a bunch of python scripts (we really like python) that run on each build machine and periodically grab code from Accurev to test. It has worked well for us in the past but as we want to do more advanced things (like splitting builds between machines and automatically grabbing a 'free' machine to use, recovering from machine failure) and need a better UI for changing build configurations, we've decided to move over to TeamCity.

Incremental checkout (or 'update' in Accurev terminology) is going to be a big issue. To start with we are going to lock each build configuration to a specific Agent, so incremental checkout would be a big win for us. (also this mirrors what we've done in the past with our bespoke system). This will need an accurev Workspace be created per Configuration/Agent combination, but that shouldn't be a big problem as they are pretty cheap in Accurev.


The same is going to be needed for our Perforce data, but I believe that plugin already supports it to some extent (at least in the beta 6.0 version if nothing else). Our perforce data (which is the game's raw data) will REALLY need incremental update because its over 100GB for a full set of data.


In the longer term I hope to move both Accurev/Perforce data into more of an 'on demand pull' system that grabs files from the server only when they are opened by a build process, rather than syncing everything up front. This would save a huge amount of bandwidth!

Hi Folks,

I have tried to enable the VCS labeling option for Accurev and it seems to have no effect. I have selected the "Successful only" radio button and checked the VCS root to label.
Are snapshots supported in this plugin?

Best Regards,

Daniel

Hi all,

I'm very happy to say that we've just started work on improving the Accurev plugin to cover our use cases.
Initially that is to add a couple of things:

Agent-side checkout
This is a must-have for us; right now we are doing this using a python script on the agent, which works but isn't neat and throws off build timing (as the update/sync time is counted as part of the build time)
We handle building from a specific Accurev Transaction by creating a timelocked stream and workspace, as this is currently the only way to get this behaviour in Accurev.

Limited support for checkout rules
We want to specify that Accurev syncs to a sub-folder of the TeamCity Checkout directory, because for some builds we have checkouts from both Accurev and Perforce and obviously don't want them both writing to the same directory.
Right now the Accurev plugin doesn't seem to support the rule we want, so we'll be adding support for it.
In the future we might add more checkout rule support e.g. for including/excluding things from a Workspace based on the checkout rules, but for now we'll be implementing that by using additional Streams in Accurev and applying the include/exclude rules to them.

I'm hoping to have both of these working in the next couple of weeks, so should be able to update the plugin with our changes soon after that.

Cheers,

Mark

Hi Daniel,

I've never actually used this option so I'm not sure about the support aspect, but looking at the codebase there's definitely code implemented for the "label" function.

It looks as though it outputs some debugging information in the TeamCity logs. Is there anything illuminating in there?

Dan