Clearcase integration

Hello,

I work on a large project, and i'm wondering why you issue tons of "cleartool lsvtree -all" commands (one for each file in the project...) ?

Gilles

12 comments

We have to list all file versions just to select the version for the date for which the sources are being loaded (at least for clean checkout).

0

I'm sorry Olesya, but what are you talking about exactly ?

I don't understand why you have to use some date to load versions.
It's an extremely bad practice IMHO to use dates to select versions. We have an overwhelming amount of projects here, and i have never seen our release managers use dates for any operation whatsoever.

At work we use Clearcase in UCM mode for like 200 components, with thousands and thousands of files. A file here can have up to a thousand versions spread over all streams (branches), maybe 80 or 90 for a single stream. Teamcity is literally crawling under the charge trying to list the versions for each file (it means about 100 lines of output to parse for each file). I set up TC to manage the build for 10 components ( 10 000) files... Do the maths. I do understand now why i have to wait like 2 hours for TC to update everything. By comparison, an update of the whole project in IntelliJ takes 1 or 2 mins.

What i'd like to do ideally in TC :
- declare the roots of my projects (i have multiple views, it works at this time)
- have TC update those views (should be 1 or 2 minutes)
- optionally baseline everything (if so, should be 1 minute) (baseline is the UCM equivalent of tag, it's a list of activities).
- build, test, inspect everything (long time, that's ok)
- report results

That should be 3 minutes in ClearCase overall.

Now if you want to see some specific versions of a stream, clearcase has a language for that : the config spec. So to do that :

  • have CC create the needed views with custom config spec (which is just plain easy, cleartool mkview and so on)...

  • provide an option to specify a config spec (that's the Hudson project does)

  • do not ask 1000 versions and so on, just tell CC the load rules


If you need to have the changes between two tags, use cleartool history (with some options you can do that). If you use CC UCM, and want to list the new activities, use cleartool diffbl...

Please could you explain in detail what you're trying to achieve with those dates and i'll give you the best way to do that with cleartool commands. Maybe you need that to do something that can't be done another way, but then make it optional because i'm pretty sure that clearcase integration is doomed on large projects (i mean with a LARGE version history, not only by the count of files)...

I'm a huge fan of your products, i'm an IntelliJ client, i'm a project manager at work and have to deal with a lot of regression and lack of quality, and i'm really willing to make TC work, and next, sell it to my manager and show the value add. Do not hesitate to bother me with questions, i'll be glad to help you, now that i have time to test things ( just take look at my recent exchanges with Michael : http://www.jetbrains.net/jira/browse/IDEA-16326 ;) )

Gilles

0

TeamCity uses vcs-independent agents to provide easy installations of the agents.

While checking for changes it detects some "version" of the repository and when build is being built on an agent with already checked out version A server has to provide information about what files changed from A to current build version B. (B could be no the latest repostory version).

As a version we're using time because I could not find sime kind of repository version (like in Perforce or subversion, for example).

If no version is checked out on the agent server has to send all files of the version B.

Several agent can load source files for one configuration but different versions.

What could you suggest for such requirements?

0

Hello Olesya,

Now i understand better why you're using the date.
As i believed, you should not use that information at all.

First of all, there is no such thing as a repository "version" in Clearcase. All you have is :

  • In Base clearcase : branches and tags

  • In UCM clearcase : streams and baselines (stream is like a branch, baseline is like a tag)


Now there are two things to know if you want to handle Clearcase integration and build agents :
- Do you use "server side checkout" or "agent side checkout" ?
- Is your root dir a base clearcase view or a UCM view ?

Note : when you're using UCM, if you create an UCM view, usually you see the latest files. You cannot override this and load specific versions. If you want to see some specific version of a clearcase UCM stream, you have to create a non-UCM view (like in Base Clearcase), and then manually set the config spec with rules like :
element * /main/LATEST load \myproject ]]>


Scenario A : "server side checkout"
===================================
TC should :
1 - Update views directly on the server (Base clearcase or UCM clearcase) with log.
2 - Tag the files in Base Clearcase (cleartool mklbtype + cleartool mklabel -r), or make a baseline on the stream in UCM Clearcase (cleartool mkbl).
3 - Detect the changes since last update. Parse the update log to compute the delta (ie the files to add, create, delete). If you want to know the versions, you can use cleartool lshistory -recurse.
4 - Send all the files for a new agent or the delta for an existing agent. You can list what has changed since the last update by comparing the labels or baselines.


Scenario B : "agent side checkout"
==================================

Provide a configuration option to specify a config spec "template" if you use UCM mode (due to the note above)

TC should :
2 - Update views directly on the server (Base clearcase or UCM clearcase).
3 - Tag on the server the files in Base Clearcase (cleartool mklbtype + cleartool mklabel -r), or make a baseline on the stream in UCM Clearcase (cleartool mkbl).
4 - Makeup a config spec like the above and replace LATEST with the tag you just created
5 - Send the config spec to the clients and tell them to set this config spec on their view with cleartool setcs the_new_config_spec.txt. Then make them update their view, they'll get all tagged versions.


Make scenario A work first as it's the easiest, and later on you can add support for scenario B if users request it. Personnaly i'm not interested in scenario B.

Do not hesitate to ask questions :)

Gilles

0

Hello Gilles,

>First of all, there is no such thing as a repository "version" in Clearcase.

I know... But there is repository version in TeamCity abstract version control and I have to support it somehow to implement clearcase support for teamcity. I cannot update current view and use output information because there are several agent and they have different versions now on these agent and I cannot update current version from yesterday version to current to send changed files to agent A and at the same time update the same files from day before yesterday version to the current one because agent B didn't build builds last two days.

Anyway, we won't change clearcase right now for certain, because we're going to release TeamCity 3.0. :)

For version controls like clearcase I wanted to support something like local vcs, which registers changes after every update and provides changed source files between any two versions (version is "update number"). I would like to implement this in the next TeamCity major version.

0

Olesya,

If this whole issue is just about sending deltas to client, could we have at least a configurable option to always send all files to an agent, not delta. ?

I really don't mind if there is more network traffic or if it takes more time until an agent has all files and can start the build, at least for now.

I just want to avoid 2 hours of cleartool lsvtree, currently that makes TC just plain unusable and untestable.

Adding this option would not disrupt anything for other users...

Another solution, would be to store the deltas after each update, and track which deltas were sent to which agent, and then you can resend as much delta that an agent need. That would be needed only for BASE clearcase VCS + Checkout Server side only mode, which might be a rare combinaison.

Gilles

0

In the last solution i described above, you could of course set a max number a delta per project, if an agent needs more ancient delta, you can resend him all the files...

0

It is not only about deltas (because for the whole checkout we have to send files for some fixed "version", not last file versions).

What TeamCity version do you use now? There is a cache for clearcase support since one of the latest EAPs which allows not to list versions for each file.

0

Hi Gilles,

I fully concur with you. We also use ClearCase with upwards of 25,000 files. I've tried to use TeamCity, but I now see that it issues lsvtree on each file. It's totally unusable for us. I launched a build which crashed after nearly 5 hours of "getting sources", and it crashed on yet another lsvtree command.

Is there a JIRA issue open for this?

Cheers!

0

Hello Gilles, Sebastien,

we decided to implement initial checkout by getting all files directly from the snapshot. It could cause problem that some change will be actually built but TeamCity will show that it belongs to the next build (if snapshot was updated before "get sources" call). This optimization will be available through special VM option because of this. I'll create the build with the change and let you know where it is available in a few days.

In TC 3.1 possibly some fair performance fix will be implemented.

Thanks for the feedback!

0

Hi Olesya,

I'm looking forward to give it a try when you release the build. If I must, I'll even log in over my vacation in order to try it out! :) Will your change work on both UCM and BASE types of configuration?

Thanks,

Sebastien

0

Here it is: ftp://ftp.intellij.net/.1283517263/TeamCity-6006.war
You have to add JVM property -Dclearcase.optimize.initial.checkout=true

Also please remove caches on the server ($/system/sourcesCache and $/system/clearCaseCache). Also please invoke "Actions/Enforce clean checkout" for the corresponding configuration.

Please let me know how it is going.

Thanks!

0

Please sign in to leave a comment.