ClearCase get latest stops working

I have set up a continuous integration build with TeamCity that gets source files from ClearCase.
This set up has seemed to work fine for a couple of months.

Now, it seems that TeamCity does not always get all files it should from ClearCase. Sometimes old files are missing.
My first workaround was to uncheck Clean all files before build. But that worked only until even updated files started missing from what TeamCity got.

I have spoken to the ClearCase administrators and they claim that they have made no changes to the environment.

I don't really know even how to start analyzing this problem. It seems that the ClearCase connection has become progressively worse for no apparent reason.

Any ideas?
Thanks!
Klas Mellbourn

12 comments
Comment actions Permalink

Please enable debug logging according to this document: http://www.jetbrains.net/confluence/display/TCD4/Reporting+Issues (pay especial attention at the ClearCase specific section), try to reproduce bug and submit a bug report to our tracker: http://jetbrains.net/tracker/workspace/TW with logs attached.

0
Comment actions Permalink

Now I actually got the following error:

[13:25:23]: Checking for changes
[13:25:30]: Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[13:25:30]: Checkout directory: C:\BuildAgent\work\XXXX_only_compilation
[13:25:30]: Getting project sources
[13:26:36]: Could not get project sources: for build #590
[13:26:36]: java.lang.NullPointerException
java.lang.NullPointerException
at jetbrains.buildServer.buildTriggers.vcs.clearcase.CCParseUtil.getVersionInt(CCParseUtil.java:219)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.structure.WriteCorrectingVersionProcessor.findLatest(WriteCorrectingVersionProcessor.java:59)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.structure.WriteCorrectingVersionProcessor.<init>(WriteCorrectingVersionProcessor.java:41)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.structure.CacheElement.loadDifferences(CacheElement.java:121)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.structure.CacheElement.processAllVersions(CacheElement.java:84)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.ClearCaseConnection.processAllVersions(ClearCaseConnection.java:613)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.CCPatchProvider.buildPatch(CCPatchProvider.java:50)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.ClearCaseSupport.buildPatch(ClearCaseSupport.java:221)
at jetbrains.buildServer.vcs.VcsSupportUtil.buildPatch(VcsSupportUtil.java:84)
at jetbrains.buildServer.buildTriggers.vcs.clearcase.ClearCaseSupport.buildPatch(ClearCaseSupport.java:390)
at jetbrains.buildServer.serverSide.impl.projectSources.SmallPatchCache.getCachedCleanPatch(SmallPatchCache.java:12)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchCacheImpl.buildCleanPatch(PatchCacheImpl.java:12)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:95)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:29)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:93)
at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.buildPatch(BuildTypeImpl.java:326)
at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$FastClassByCGLIB$$a84db719.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:32)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:582)
at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$EnhancerByCGLIB$$529531fe.buildPatch(<generated>)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2.call(BuildStarter.java:9)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

[13:26:36]: Could not get project sources
[13:26:36]: Loading changed files from server...
[13:26:36]: Patch is broken, can be found in file C:\BuildAgent\temp\cache\temp53115patch590
[13:26:36]: Error while applying patch: java.lang.NullPointerException
at jetbrains.buildServer.buildTriggers.vcs.clearcase.CCParseUtil...

[13:26:36]: Build finished
0
Comment actions Permalink

And what version of TeamCity you are using?

0
Comment actions Permalink

I am using Windows XP SP3, TeamCity 4.0.1 (build 8171), Visual Studio 2005, ClearCase 7.0.1.1

Thanks,
Klas Mellbourn

0
Comment actions Permalink

I should clarify that the exception above may be a clue, but it is not my main problem. The exception occurrs very infrequently.
What is persistent is that TeamCity does not get the latest files from ClearCase. It gets some, but not all, and it reports no errors.
Thank you,
Klas Mellbourn

0
Comment actions Permalink

Just so we are completely clear on the priority of this:

I am currently consulting at a large organization handling huge amounts of money
This organization is in the process of restructuring their development environment.
I am in a position where I can strongly influence the future choice of tools for continuous integration.
I have introduced TeamCity here and to my embarrassment it has permanently stopped working due to the problems above.
If this problem (with clearcase no longer getting files) is not fixed TeamCity will certainly not be used by this organization.

0
Comment actions Permalink

In this thread I asked to enable debug logging. Could you please do this and then try to reproduce the problem. Once it happens please send server logs to teamcity-feedback [at] jetbrains.com. Without logs we cannot analyze what happens at your side.

0
Comment actions Permalink

I have now mailed the log files to the given adress. Please let me know if you need anything else.
Thanks,
Klas Mellbourn

0
Comment actions Permalink

Klas,

According to the logs the xslt file you mentioned in the message was got from ClearCase, so we need the agent patch file for further investigation. To get this file please do the following steps:

1. Add new system property with name "agent.save.patch" and value "true" to your build configuration.
2. Make some changes in ClearCase and commit them.
3. Run build.
4. Find the line "Patch is saved to file <fileName> (property agent.save.patch is set)" in the build log.

The patch file if file <fileName>. Can you please provide us this file?

Also tell us please whether the changes you made on step 2 are shown on TeamCity UI or not? (And was it collected for the build at step 3?)

And also can you please provide us your checkout rules and agent checkout directory?

0
Comment actions Permalink

I have sent what you request to teamcity-feedback (at) jetbrains.com.
Thanks,
Klas Mellbourn

0
Comment actions Permalink

Maxim solved the problem!

The solution is to remove caches by removnig the following directories:

<user home>\.BuildServer\system\caches\sources

and

<user home>\.BuildServer\system\caches\clearCase

For the future you can disable ClearCase caching with "-Dclearcase.disable.caches=true" JVM option.

Thanks Maxim!
/Klas

0

Please sign in to leave a comment.