How to enable multiple VCS roots checkout in parallel to save time

Unfortunately, parallel checkout is not implemented as of now. It is possible to develop a custom plugin which would check out the roots and merge the resulting sources at a single directory. You could also disable automatic checkout and replace it with an additional step where a custom script would check out the sources in a parallel fashion. Both changes would mean a good part of your existing setup would require amendment.
We have created a feature request regarding this functionality which you may find at https://youtrack.jetbrains.com/issue/TW-65213. Could you please provide some details on why your setup requires 6 different roots to be checked out? That would be really helpful for our development team understanding of the usecase.
To keep in mind, while parallel checkout would save time, it could be problematic under scenarios where there are conflicts. As an example, if you check out some repository that contains a "tools" subfolder, but another one of the repositories is checked out into the same "tools" subfolder, sequential checkout would overwrite the content just fine, while parallel checkout might leave it in an inconsistent, maybe corrupted state, if not crash processes if multiple processes attempt to write to the same files at the same time.

Hi Fedor,

>Could you please provide some details on why your setup requires 6 different roots to be checked out? 

The 6 different root have different release/distribution/license models. Some of them are open-source project repos, the others are close-source project repos. So we have to keep them separated and don't prefer to mix them in one repo.

>To keep in mind, while parallel checkout would save time, it could be problematic under scenarios where there are conflicts.

I understand. We already guarantee the 6 repos have no conflicts to each other and can be safely check-outed in parallel

>You could also disable automatic checkout and replace it with an additional step ...

No. I don't hope to disable the VCS. It is too risk for us.

>It is possible to develop a custom plugin which would check out the roots and merge the resulting sources at a single directory. 

Is there any lightweight hook point before the VCS to run custom tool? If there is, then I could introduce a new Pre_VCS step before the VCS running, which will run a multiple-thread sync tool to parallel update all repo local caches and finish all repo code checkout in workspace. Then, in the next real VCS root step, the VCS service will find all local cache and workspace repos code have already been the latest version and will do nothing to skip. In this way, the VCS root just become a quick double-checking step, not the real slow checkout worker. And all the repo checkout can be finished in parallel way.

Thanks

Steven

BTW, in our current Teamcity build flow, there are two VCS checkout steps in one build. One is in a control agent, the other is in a build worker agent. So our overall VCS checkout time is +12mins. I hope to enable the parallel checkout which can save us  +8mins in every build.

Hello Steven,

Thank you for the responses, it`s appreciated! In regard to the workaround, let me suggest to move this discussion to the feature request page at the issue tracker: https://youtrack.jetbrains.com/issue/TW-65213