Git submodule issues

It is located under <TeamCity Server home>/logs directory (see http://confluence.jetbrains.net/display/TCD65/Viewing+TeamCity+Server+Logs for details).

I've stumbled upon this bug again just now and I've got the log for last few days.  Can I send this to you privately? I don't see any option to send message to you as a user of this forum.

Hi

Please send them to teamcity-feedback@jetbrains.com (mention a link to this forum thread).

But Dmitry is on vacation for next two weeks, and we'll need his return to debug this.

Thanks

I've sent the logs with debug messages enabled two days ago.  The issue still occurs regularly.
Does Dmitry's vacation mean that this issue will be fixed not earlier than 7th of July?

Sorry for delay. Please try plugin attached to this issue: http://youtrack.jetbrains.net/issue/TW-17333

I'm seeing the same (a very similar?) issue. I tested with the attached plugin (SNAPSHOT-20110616044913), but no luck.

For me, the thread attempting to fetch the updates hangs forever, always with a stacktrace that looks sort of like:
"14:00:53 Collect changes for ssh://git.spotify.net/libspotify.git#master {instance id=11, parent id=1}; revs: [52f414c47c384d5a51d39f644e237fe6e0a3d13a@13c3ed6c8a8->d20d5b73d0a6f052389539242adc63c9f74fa96c@1311e8f3900]; 14:00:50 Loading VCS changes for ssh://git.spotify.net/libspotify.git#master {instance id=11, parent id=1}; VCS executor 5 {id=11} {id=11}" group="main" prio=5 tid=108 runnable

   java.lang.Thread.State: RUNNABLE
    at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
    at java.io.File.lastModified(File.java:795)
    at org.eclipse.jgit.storage.file.FileSnapshot.isModified(FileSnapshot.java:117)
    at org.eclipse.jgit.storage.file.FileBasedConfig.isOutdated(FileBasedConfig.java:210)
    at org.eclipse.jgit.storage.file.FileRepository.getConfig(FileRepository.java:319)
    at org.eclipse.jgit.storage.file.FileRepository.getConfig(FileRepository.java:97)
    at jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleResolver.ensureConfigLoaded(SubmoduleResolver.java:136)
    at jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleResolver.containsSubmodule(SubmoduleResolver.java:103)
    at jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleAwareTreeIterator.createSubmoduleAwareTreeIterator(SubmoduleAwareTreeIterator.java:350)
    at jetbrains.buildServer.buildTriggers.vcs.git.submodules.SubmoduleAwareTreeIterator.createSubtreeIterator(SubmoduleAwareTreeIterator.java:299)
    at org.eclipse.jgit.treewalk.AbstractTreeIterator.createSubtreeIterator(AbstractTreeIterator.java:527)
    at org.eclipse.jgit.treewalk.TreeWalk.enterSubtree(TreeWalk.java:893)
    at org.eclipse.jgit.treewalk.TreeWalk.next(TreeWalk.java:559)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getPreviousCommitWithFixedSubmodule(GitVcsSupport.java:542)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCommitChanges(GitVcsSupport.java:455)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.createModificationData(GitVcsSupport.java:361)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.collectChanges(GitVcsSupport.java:194)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:33)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:78)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChangesForCheckoutRules(VcsChangesLoader.java:119)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:31)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader._loadChangesForRoot(VcsChangesLoader.java:81)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:169)
    at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:156)
    at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:76)
    at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:11)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:17)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
    at java.lang.Thread.run(Thread.java:595)

I also get the type of errors

[2011-07-12 14:19:08,360]   WARN [{id=11} {id=11}] - les.SubmoduleAwareTreeIterator - Ignore submodule error: "Object 30dcc9b7c2ffa7de380b58e87ecda5219e19a362 is corrupt: Commit could not be resolved: Missing unknown 30dcc9b7c2ffa7de380b58e87ecda5219e19a362". It seems to be fixed in one of the later commits.

Actually seems it's still no luck. After I removed the git checkout, it took a very long time (a few hours) with "Checking for changes", but  I was hopeful as the number of changes kept increasing, and there was an active thread on the server that seemed to be doing processing.

Now all my builds pass the "Checking for changes", goes into "running", but there is no build output there. The thread output of the server shows no threads that seem to be related, and getting the process listing from the client lists nothing.

This happens consistently, and I haven't been able to get any builds from this repository since the problem started. What information can I provide to help debug this?

Thanks
/Johan

Hi Johan,

please enable a debug-vcs loggin preset, reproduce the problem and provide a teamcity-vcs.log/

Mailed log to teamcity-feedback.

Let me know what else I can provide to help debug this, or any workarounds.

-Johan

Hi guys,

recently I fixed several bugs in the cleanup and submodules logic in the git-plugin. You can download build of plugin with fixes from http://teamcity.jetbrains.com/viewType.html?buildTypeId=bt245&tab=buildTypeStatusDiv , its number is '#90'.

It seems that the fix really works. We had a lot of troubles with false submodules errors. We had just one and I decided to install this update. After starting team city with a new plugin the source codes re-fetch correctly and build finished successfuly. With the previous version we would need to delete caches and we had to do that quite offten.

Thank you Dmitry for that fix it will make our life a lot easier :)

I have the same problem, but the new plugin (snapshot 99) doesn't help.

Error collecting changes for VCS root 'NameOfProject'
Collecting changes failed: org.eclipse.jgit.errors.CorruptObjectException: Object 87a86811ba7cf76bbdaa0c378d0bfb0d38ae5a29 is corrupt: Commit could not be resolved: Missing unknown 87a86811ba7cf76bbdaa0c378d0bfb0d38ae5a29

Local checkout works, git caches are deleted, teamcity rebootet...

Nothing helps...

TeamCity 6.5.4

Hi Felix,

is you repository public, so I can try to reproduce that? Bug that I fixed have to do with clean up process, does your teamcity-server.log contains any records on removing unused git repositories?

We've been struggling with this exception for a couple of days now.  Our issue has a different cause, but Google kept sending us here, so I'll reply here in case someone else is in the same boat.

For us, the problem was that we had a submodule pointing to a branch/commit that no longer existed on GitHub.  It had been merged and deleted (Developers, please don't do this :()

Figuring out this was the problem and figuring out which submodule was causing it were challenges, and the TeamCity stack trace provided almost no help in sorting things out.  We were further thrown off by the fact that we had just updated an unrelated submodule.  It would be nice if TeamCity could provide better diagnostic information, including the repository that's missing the commit.  A slightly better log could have saved us a full day.

Hi Patrick,

sorry for painful troubleshooting, what error did you get?

Resurrecting a dead horse here but I keep getting git issues almost every time I alter the submodules of our projects. The issues are usually resolved by shutting the server down, clearing the cache and doing a clean rebuild on the agents. This one, however, seems to be permanently stuck:

Hi Marco,

such an error usually happen when the main repository refers to a commit in subrepo which wasn't pushed. Please send your teamcity-vcs.log from the server to teamcity-feedback@jetbrains.com.

Thanks, sent.