Git Submodule Issues

I have mulitple projects that utilize git submodules.  Teamcity/Jetbrains Git is able to properly handle initial checkouts/builds, as well as subsequent commits to the parent git repository, but as soon as the parent git repo updates it's reference to a newer revision of the submodule, teamcity freaks out and the only thing I have figured out that can fix it is to force a clean checkout by changing the "Clone Repository To" field on the VCS settings to something new.  FYI, I have the submodules setting on the VCS set to "Checkout".  Here is the error I get when this happens:

java.util.concurrent.ExecutionException:  jetbrains.buildServer.vcs.VcsException: Problem collecting changes for  'MyRepo:: MyBranch' : Error collecting changes for VCS root 'My VCS Name'

The collecting changes failed:

org.eclipse.jgit.errors.CorruptObjectException: Object  71d3d2d44c878520513c80cb3988ec628e82ca9c is corrupt: Commit could not be  resolved

java.io.IOException: The commit  AnyObjectId[71d3d2d44c878520513c80cb3988ec628e82ca9c] (referenced by MySubmodule ) is not found in repository:  Repository[C:\Users\Administrator\.BuildServer\system\caches\git\git-FC4488F5.git]

java.util.concurrent.ExecutionException:  jetbrains.buildServer.vcs.VcsException: Problem collecting changes for  'MyRepo::  MyBranch' : Error collecting changes for VCS root 'My VCS Name'

The collecting changes failed:

org.eclipse.jgit.errors.CorruptObjectException: Object  71d3d2d44c878520513c80cb3988ec628e82ca9c is corrupt: Commit could not be  resolved

java.io.IOException: The commit  AnyObjectId[71d3d2d44c878520513c80cb3988ec628e82ca9c] (referenced by MySubmodule ) is not found in repository:  Repository[C:\Users\Administrator\.BuildServer\system\caches\git\git-FC4488F5.git]

at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205)
at java.util.concurrent.FutureTask.get(FutureTask.java:80)

at  jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:31)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:25)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:3)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:2)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java)

at  jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:34)

at  jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:21)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:42)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:38)

at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:17)

at  jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:10)
at  jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:13)

at  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)

at  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

at java.lang.Thread.run(Thread.java:595)

Caused by: jetbrains.buildServer.vcs.VcsException: Problem collecting  changes for 'MyRepo:: MyBranch' : Error collecting changes for VCS root  'My VCS Name'

The collecting changes failed:

org.eclipse.jgit.errors.CorruptObjectException: Object  71d3d2d44c878520513c80cb3988ec628e82ca9c is corrupt: Commit could not be  resolved

java.io.IOException: The commit  AnyObjectId[71d3d2d44c878520513c80cb3988ec628e82ca9c] (referenced by MySubmodule ) is not found in repository:  Repository[C:\Users\Administrator\.BuildServer\system\caches\git\git-FC4488F5.git]

at  jetbrains.buildServer.vcs.impl.ChangesCollectorResultImpl.rethrowIfOurBuildType(ChangesCollectorResultImpl.java:43)

at  jetbrains.buildServer.vcs.impl.ChangesCollectorResultImpl.checkNoProblemsInMyBuildTypes(ChangesCollectorResultImpl.java:45)

at  jetbrains.buildServer.vcs.impl.ChangesCollectorResultImpl.checkingForChangesFinished(ChangesCollectorResultImpl.java:25)

at  jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:8)

at  jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:28)

... 5 more

Caused by: jetbrains.buildServer.vcs.VcsException: Error collecting  changes for VCS root 'My VCS Name'

The collecting changes failed:

org.eclipse.jgit.errors.CorruptObjectException: Object  71d3d2d44c878520513c80cb3988ec628e82ca9c is corrupt: Commit could not be  resolved

java.io.IOException: The commit  AnyObjectId[71d3d2d44c878520513c80cb3988ec628e82ca9c] (referenced by MySubmodule ) is not found in repository:  Repository[C:\Users\Administrator\.BuildServer\system\caches\git\git-FC4488F5.git]

at  jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.processException(GitVcsSupport.java:159)

at  jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.collectChanges(GitVcsSupport.java:221)

at  jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:202)

at  jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:71)

at  jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChangesForRule(VcsChangesLoader.java:154)

at  jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges4CommonCheckoutRule(VcsChangesLoader.java:188)

at  jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:213)

at  jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:446)

at  jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:42)

at  jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1)

at  jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java)

... 5 more

Caused by: org.eclipse.jgit.errors.CorruptObjectException: Object  71d3d2d44c878520513c80cb3988ec628e82ca9c is corrupt: Commit could not be  resolved

at  jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleAwareTreeIterator.movedToEntry(SubmoduleAwareTreeIterator.java:148)

at  jetbrains.buildServer.buildTriggers.vcs.git.submodules.DirectSubmoduleAwareTreeIterator.next(DirectSubmoduleAwareTreeIterator.java:71)

at  org.eclipse.jgit.treewalk.AbstractTreeIterator.skip(AbstractTreeIterator.java:557)

at  org.eclipse.jgit.treewalk.TreeWalk.skipEntriesEqual(TreeWalk.java:883)

at org.eclipse.jgit.treewalk.TreeWalk.next(TreeWalk.java:495)

at  jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCommitChanges(GitVcsSupport.java:302)

at  jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.addCommit(GitVcsSupport.java:256)

at  jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.collectChanges(GitVcsSupport.java:191)

... 14 more

Caused by: java.io.IOException: The commit  AnyObjectId[71d3d2d44c878520513c80cb3988ec628e82ca9c] (referenced by MySubmodule ) is not found in repository:  Repository[C:\Users\Administrator\.BuildServer\system\caches\git\git-FC4488F5.git]

at  jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleResolver.getSubmodule(SubmoduleResolver.java:74)

at  jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleAwareTreeIterator.movedToEntry(SubmoduleAwareTreeIterator.java:146)

... 21 more

4 comments

From the execption it looks like the commit is not in the local submodule repository at time of collecting changes. The teamcity plugin runs jgit analog of "git fetch origin +ref/heads/*:ref/heads/*" on submodule repository prior to resolving commit. So the problem could be one of the following:

1. The commit has not been pushed to the respository referred by .gitmodules yet. This could happen due to the bad timing of pushes (root is pushed earlier than submodules) or when referred submodule repository and repository where submodule commits are pushed are different (for example when periodic replicaton to public repository referred by .gitmodules is used).
2. The commit is not under branch heads (it might be referred only by tags).
3. The branch for the commit is somehow not fetched (for example, jgit fails to match it using all branches mask)
4. The relative paths for specifying submodules were used (there is a known bug with it).

After error occurs again, please find the submodule repository in "${user.home}\.BuildServer\system\caches\git\" and check if the reffered commit presents in that repository (the repository url is stored in teamcity.remote property).

If it does not please compare outputs of "git ls-remote ." and "git ls-remote <submodule-ur-as-referred-teamcity.remote>". Also try to run "git describe --all --contains <commit>" while all tags in the repository are temporary removed.

If the problem is not the case 1, please fill the bug to youtrack.

Also please fill the bug about lack of UI to recover from problems during change collecting.

0

Thank you very much for your very detailed response.

I will do some debugging the next time this behavior occurs as you suggested, however, I can fairly certainly rule out #1 and #2.  I'm betting that the issue lies within your suggestion #4, however I'm not quite sure exactly what you mean when you say relative paths for submodules.  My .gitmodules file in the parent directory looks as follows:
[submodule "MySubmodule"]
    path = MySubmodule
    url = git@git.server.com:mysubmodule.git

I have multiple developers working on this project, and we all have different paths.  Are you suggesting that if we standardized our directory structure and had something like the following, that it would solve our problem?

[submodule "MySubmodule"]
     path = C:\ParentRepo\MySubmodule
     url = git@git.server.com:mysubmodule.git

Is this a known issue with jgit or with teamcity's submodule implementation? Is there a bug ticket you could point me to?

Thanks again, I'm so appreciative when people give consise, knowledgeable help.

0

I meant absolute and relative url to repository. For example, submodule repository could be referred as

[submodule "MySubmodule"]
    path = MySubmodule
    url = ../mysubmodule.git

And this was not supported. Original bug were http://youtrack.jetbrains.net/issue/TW-9292 . But fix was broken until recently. So this is definitely is not your issue. Please check if the commit is in the repository managed by TeamCity.

0

Please sign in to leave a comment.