Only checkout files from the VCS that triggered the build.

Could you explain what the use case is? Currently, TeamCity assumes all dependencies are mandatory. All VCS Roots are included as well, considering them required to perform a successful build.

If you want to poll different VCS and perform separate builds for each root, even if they follow the exact same steps and have the same dependencies, you'll need a separate build configuration.

For optional artifacts, you can vote here: https://youtrack.jetbrains.com/issue/TW-19132 Other than that, artifact dependencies are also supposed to be all needed to perform a successful build.

If you specified why you need different roots to be built with the exact same configuration, and not a template, it might be easier to figure out a solution.

So my specific use case is that I have Different Ember apps in Different Repositories. The steps to build the ember apps are not different. The different Ember apps all depend on a lib repo and a modules repo. the two lib/modules repo's are snapshot dependant on our api repo. They move horizontally across diff web servers as a part of their own complicated build chain (simplified below). Each web server in that chain has a build config for its current lib/modules revision.

The Ember apps have their own Project and build chain. At certain points in that build chain, when it needs the lib/modules for that particular web server, it has an artifact dependency(no snapshot dependancy).

I hope this simplified visual helps frame the relevant parts of what I have...

api -> -> -> ->  (web-server1) -> -> -> -> -> (web-server2) -> -> -> -> -> (web-server3)

                     ^verticleDepUp^                ^verticleDepUp^                 ^verticleDepUp^

lib/module -> (build-config-rev3) -> -> -> (build-config-rev2) ->  -> -> (build-config-rev1)

                      ^vertDepDown^               ^vertDepDown^                ^vertDepDown^

EmberApps ... -> (rev3-artifact)  -> ... -> (rev2-artifact)  -> -> ... -> -> (rev1-artifact)

... What I didnt show above is that the begining of the lib/modules chain there are initial build configs containing all the Ember VCS roots. It extracts their package.json and bower.json files into an artifact. Each repo having its own folder, in the artifact, containing their .json files. then along the way it runs the npm and bower installs,  makes a tar of the node_modules and bower_components and caches the tar balls for future use (significant time saver). First blocker is here, where I dont know which folder has new .json files and which cached packages to retrieve so I dont have to do a complete npm install (adds 5min+ per app) or install every Ember app. It does this at each web server stop and creates the artifacts of tar packages shown above. Thats where it passes the artifacts down to the ember apps, but each app uses different components and modules from the lib and modules repo. Keep in mind these ember apps all have diff branches that are using snapshot dependencies to ensure everything is ran on the right source/branches as well. Also the lib/modules artifacts that are passed down have snapshot dep pointing up which makes them in diff revision states. This is what makes areas, like were I take the .json files and where I move the Ember apps horizontally, such a hassle to have to make new build configs. Every thing is already very deeply connected by dependancies and build chains that adding more and more horizontally linked build configs requires more and more vertical linked configs which leads to a lot of headache and un-intended triggering of build configs that are very destructive to the environment that needed that config to not have accidently triggered.

Being able to change parent project params from one sub project so that other sub project build configs can reference their new values as config params, sys params, and env params seems like a no brainer feature and easy solution to this and many other problems, like working with immutable systems. I feel like i must be missing a currently present feature or not understand teamcity parameters correctly. Why wouldn't teamcity allow scoped variables that can be referenced and changed in the same way most proograming languages scope variables.

I should also add that this only became an issue for me when it was decided that our buildAgents are going to move from an "always on" server to a cloud profile that launches and terminates EC2 Spot Instances. This means there is no server provided environment to reference. TeamCity parameters appear to intentionally not offer the ability to pull out build data and continue using it. Which I can start to think makes more sense now. Is there anything currently in TC that allows build specific data to be pulled outside its configuration? A feature that allows you to create a dependency data structure that is defined at the project level, to be instantiated at the start of a build chain would be a possible feature request. Once instantiated it would act as a "back-plane" for only the build chain that initialized it. Allowing configurations with snapshot dependencies to be passed the data reference as a mutable object. While configurations with artifact dependencies get passed only the chosen key -> values to be used as parameters. The build chain's start and end points determining the range of configurations inside the scope. This would allow multiple vcs roots, like in my example, to be the source of "module like" build chains which can be assigned to a cloud profile with aws resources configured and optimized to the chains purpose.

Thanks for the detailed explanation. There are several things that fit what you have commented in your last 2 messages but definitely have little to do with the initial request, so I'll expand on them.

Parameters (as in parameters you use through the build configurations) are scoped hierarchically. Set them at the project level and they can be referenced through the whole project, its subprojects and builds. Change them at the build configuration (or subproject) level, and that build configuration or project will use the updated value but the rest of the builds/projects will use the original. You can also use the parameters of previous builds of the own chain using "dep.<build_config_id>.parameter_name", which will return the value they had at the end of the build (should you have changed it through the build).

Build Templates are another way of abstracting the same logic to a series of configurations. Every configuration that inherits from a template will inherit all the behavior of the template. In this sense, they are more like "scaffolds" rather than templates. It's possible to then add new behavior to the build configuration, but not to remove from the template. Also, if you modify the template, you instantly modify all the configurations that inherit from it. This is the best approach to having multiple build configurations that share a lot of configuration.

Artifacts can be any kind of file that you deem appropriate. This means, artifact dependencies can be another form of passing along data through dependent builds. If you generate runtime data that you can't predict where it's going to end, you can collect them at the end of a build inside a specific data format, post it as an artifact, then load it in the next step (next build configuration) by taking it as an artifact dependency and running a build step to load them into variables.

Getting build chains to work with sets of configuration, as you mention, is precisely the reason for which the recommendation is to separate build configurations for different objects even if the steps and configuration are the exact same. The main advantage it brings is that for every build, you know exactly what was going on. If a single build configuration "processes" 3 different types of builds, when one has failed, you don't know which of the three has failed until you go into the details, then you can check why it has failed. Having a different configuration for each app/project/whatever you want to call it, allows you to instantly know which one has failed, and then just check why.

While there is no automated way to modify build chains on the fly, if you really need it, you can use the REST API to change dependencies. You can set up a build configuration that modifies some dependencies on the fly, but be warned, a build chain isn't calculated on the fly, but when the first build of the chain is triggered. The option here would be to move the triggers to a separate, not a part of the chain build configuration, that modifies the dependencies as required, then triggers manually a build at the end of the chain that then pulls the full chain. More info here: https://confluence.jetbrains.com/display/TCD10/REST+API#RESTAPI-BuildConfigurationAndTemplateSettings

With this in mind, build chains are slowly becoming more prominent, the upcoming 2017.1 release includes special build chain views and displays on the dependency pages when a build is part of a chain. More work will be dedicated towards improving build chains in the future, as they are taking over individual builds as the core. If you have specific comments or suggestions towards it, we are open for feedback on our tracker https://youtrack.jetbrains.com/issues/TW

Thank you for the detailed response. I am in the process of refactoring most of my build chains and slightly different variations of the same problem keep coming up some of which can be handled by what you said. But the really tricky scenarios are when you don't the value of a required variable until the build chain is already started and it's value needs to be retrieved from the result of a build step and referenced, later down the chain, inside a separate build configuration.

Can an undefined parameter be defined inside a build step then passed using dep.vcsid.param to a later configuration? If so can u please explain the part on how you might change the value of a param inside a build step?

It is possible to create a parameter dynamically and reference it later. To add or change parameters, check how to use our service messages here: https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-changingBuildParameterAddingorChangingaBuildParameterfromaBuildStepAddingorChangingaBuildParameter (Basically you need to make sure a specific message appears in the build log)

It is better to predefine the parameter and simply populate it though, using undefined parameters can sometimes be inconsistent. Normally a build will force you to define it or it won't start, even if you are setting it on code every time.

That looks like it will work, didn't see this page looking through the docs. Thanks for providing the link.

It sounds like the only limitation is it wont allow you to Trigger a build configuration in a runtime environment (JavaScript style). Even if you are using a scripting runner like node.js

That is mostly correct, but there is some possibilities. We offer a REST API that, among other things, includes the possibility of triggering builds. You could add a last build step at some point that triggers a specific build configuration based on the information you have. While it wouldn't immediately respect the branch, you can add the branchName to the trigger request, which you can simply get from one of the build parameters.