How to best implement upstream gradle enterprise remote build cache jobs

Hi,

First, I'm sorry but I'm not particluarly familiar with Gradle Enterprise, so my comments will be based on what I've been able to gather and your description. Should I misinterpret something, please don't hesitate to point it out.

The build cache seems to be just the same thing as teamcity artifact dependencies. Unfortunately, this dependencies are exclusive to teamcity and cannot be configured to download from a remote cache, but I would think that Gradle should pick them up automatically if the "test jobs" would have the remote cache configured to pull the dependencies from.

Furthermore, both for gradle enterprise "remote cache" and for artifact dependencies on teamcity, snapshots would seem to be the right approach. Normally in this case, you would either have build OS and test OS which has a snapshot dependency on build OS, and move the VCS Trigger to test OS, and it would run both builds consecutively, waiting for the first to finish before starting the second. Whether you use the remote cache (via gradle) or simply use artifact dependencies as well, this should do the trick. If you use artifact dependencies, TeamCity would guarantee they're using the same commit (if you set the dependency to use the artifact from the same build chain). Unfortunately, by gradle being a third party tool, I'm currently not aware that this guarantee would happen as I'm not particularly sure of how gradle manages this. It would make sense that the artifact had some tag with a build number or commit, that the next build can use to pull from (just like regular dependencies), but, again, not familiar with it.

With a more concrete example I might be able to point out limitations, but in the general case, I'm not aware of any. A single build can both trigger multiple via snapshot dependencies or can be triggered by many and be shared if it has been built previously (configure the snapshot dependencies to reuse the build when possible).

There is more information on build chains and snapshot dependencies in here: https://www.jetbrains.com/help/teamcity/snapshot-dependencies.html

Hope this helps.

Thanks for the response and filling in some of the background.

For background: from the gradle enterprise server point of view, remote cache works by taking a signature snapshot of the inputs and outputs of a task when it is executed and recording it in the gradle daemon.  On subsequent builds before executing the task if either the inputs or outputs of the task are changed, the task will be executed, if not the task is skipped.  When the task executes the local and remote cache are checked to see if one of the build outputs in the caches match the source signatures of the task and if so those outputs are used instead of running the task again to re-generate them.  In the case of java compile, this results in downloading a bundle of the class files compiled previously for a particular gradle project (projects can be nested). The upstream 'Build' job on team city would simply be the first execution of a job that would fill the remote cache so that when downstream developers or team city test jobs run, the remote cache will be full and gradle will be able to draw on the compiled classes cached there. Typically developers don't fill the remote cache since their changes are not necessarily shareable with the team or team city. Until they commit their changes are local to them.

The part I still have questions on is how the triggering works with these chained snapshot jobs. I am not quite sure how this works for some different scenarios from the documentation. We currently have team city jobs in the following categories.

1. Triggered based off commit with an N minute delay to wait for additional changes before scheduling a run.

2. Triggered based off a time of day (possibly multiple specific times of day)

So what I want to achieve is...

1. Run the 'build' job immediately after every commit (or as close to that as possible).

2. Run the existing 'test' jobs with their current triggers (at their current frequency).

I believe from reading the TC docs that if a job at the end of a build chain is triggered, Team City will automatically run the 'upstream' dependent jobs in the chain, so that seems fine. I also believe that triggering the 'build' job at the head of the build chain will not cascade to starting every job that is dependent on it (i.e. running the 'build' job will not trigger the 'test' jobs). From the documentation I also believe this is what should happen. 

There is one scenario though which I am not not clear. I think typically events will trigger as follows, (1) the 'build' job runs after commit, then(2) a test job is triggered based on commit delay or schedule sometime later.   When job 2 launches and no commits have been made in the interim, will the output of job 1 be used as part of the snapshot build chain or is the 'build' job rerun again?  If the 'build' job runs again I think it will be running a lot more than it needs to and I am concerned that it could put unneeded additional overhead into the system as a whole.

Your understanding for the initial scenarios is correct, that's how it works. Whether "build" will be rerun if there are no new commits is configurable at the snapshot dependency level. You can select to reuse if a suitable build exists (the definition of suitable build is in the same snapshot dependency page as well), and for the most part it should be reused based on the set conditions.

Thanks for the advice and confirmation, I will try to set it up.