Using 2 VCS roots with different manual selected branches

Anton, thanks!

But I'll be lacking great cache featuring TeamCity do with submodules and cleaning cache automatically, I wanted to avoid it with all costs. Also, there will be no list of branches to select from, this makes running the build just a nightmare on scale.

Thanks again for the details.

Yes, we want to perform a build composing 2 VCS roots with separate selected branches from the drop-down in Run Custom Build dialog, put one in the root of the build directory and the second in one of subfolder (supported already).

But for example, a custom parameter with type “VCS REF/BRANCH” also is a solid addition, which can fetch and show the list of refs.

Also, one more questions: how I can get GitLab oauth token during build process to clone the repositories with custom script? GitlabCI has a Job Token, do we have something similar? Don't found any info. 

Anton Vakhtel Hey!

Thanks for your response. We came back to this topic, and decided to try the easiest option we have: use 1 VCS with submodule and after Teamcity own checkout, checkout the same branch in the submodule. But we have no access to the repository from our script.

You mentioned the solution above, but it's highly unscalable and requires creating hundreds of access tokens, which now leaves only 1 year in the GitLab.

 In GitLab CI you have access to Job Token which you can use to perform any authorization to the git with the right the same as the person who triggered the build. This token leaves until build finishes. This solution resolve 99% of the cases.

Do we have some similar options in the TeamCity? I'm sure we are not the first with this kind of question.

Hey,

For context about our VCS composition and processes:

• We have Unity game engine based projects hosted in Git (with LFS) repositories on the Self Hosted GitLab.
• Each game has one main Git repository with game engine files, project dependent game play code and build in resources.
• Each main Git repository can contain up to 100 Git submodules of two main assets types: code library or game resources (images, sounds, etc.).
• Each main Git repository mostly contain ~5 resources submodules, and each can have thousands of branches and hundreds of gigabytes of LFS files.
• Each day, each project can have tens of new features with only resources changes, but with the same project configuration and code. So we create a feature branch in a selected resource submodule and build resources archive using a main (last release) branch of our main Git repository.
• Each build needs to check out the main project branch, checkout on of selected resources submodules with selected branch, build the archive and deploy to a CDN folder.
• Worth to note that each first build with main project open can take up to 1–2 hours to build a cache of all build in resources and code compilations. So having a lot of folders on the Agents for the same project is no go for us. That cache can be up to 200gb per project. So we make clean checkout only for the release builds. Imagine having 30 projects and 30 agents.

For this feature, we are trying the following temporary approach:

• We've deleted submodules inside the project.
• We've added them as 3 secondary VCS.
• We've configured them to check out inside the main VCS folder structure (to be a part of Unity project).
• We'll try to change development process to use same branch in main repository as on secondaries. But in 90% cases we are using main branch in main repository and a feature branch in submodules (sometimes it's 10–20 branches per day to build resources with same code base). Sometimes we need to build some older feature, and keeping all branches is a hell.

I know, here isn't a place to discuss Teamcity solutions for our personal purposes, but maybe our case can help you grow in some directions, merging our cases and knowledge with your outstanding solution.

Yes, we use connection and Commit Status Pusher is essential for our processes, not sure if it'll work with SSH only VCS. Also, SSH key require creation of a service account with read-only access to all our repositories and adding the new projects to it. Not perfectly scalable and introduces another block - point. Also for SSH the repo remote URL need to be adjust by adding ssh variant in the TeamCity or updating during build process.

Anton,

Thanks a lot for your time and support!

We are still experimenting with our case and trying different options. I'll share results later.

While you are still here, do you have any recommendations how to work with big git repositories? Cloning one of them can take 4 hours due to the amount of commits and branches, even with full LFS. Do we have some options to clone only with latest 100 branches? Or provide depth?

We are using the own fleet of macOS + Windows machines set upped with the Puppet. Also, only the agent checkout with the mirrors. Our repositories are huge, and keeping the cache on the machine in the same directory is essential to keep building times under 30 minutes. That cache can be 30–200 GB, so we can't move it around or use short-lived agent. Also, all build steps should be done on the same agent and same folder to maximize speed (what we do).

Git mirroring is great, but it still clones the whole git history in the agent cache, which takes up to 5 hours due to huge amount of commits and branches (90% of which we use as an archive s to fork from, or to update older feature to new version of code).

Unfortunately, the behavior is the same after starting the build with teamcity.git.use.shallow.clone=true on the machine without any cache. 1 hour and 7% in progress.

I think the total performance will be worse, because we'll lose mirrors. I'll try it later, but not sure if that is a solution.