Deploy pipeline with large artifacts Follow
Completed
In my scenario, I have two parts for a configuration:
- Build
- Deploy
In the build step, an approx ~60GB build will be generated.
The deploy step will then upload these artifacts to a windows file share.
With the concept being to have the deploy step free up the agent whilst a file transfer occurs to network storage. However the archive artifacts on Build is taking a very long time. So any benefit is pretty much lost in having them as separate jobs.
I can forsee the following solutions:
- Remove the artifacts feature from this job, add a build step directly to do the SMB upload in Build.
- The deploy configuration can read direct from the workspace of the dependency Build configuration. (I've not seen this as any variable for the path). This also requires locking the original workspace in some way (So another job can't come along and taint the assets)
- The configuration of the artifacts on Build needs to be improved. Perhaps I'm using something super slow?
- It's a resource problem, there's a blocker within my hardware resources.
Please sign in to leave a comment.
Hi Peter,
When setting artifacts as such in teamcity, this means that the build agent will pick them up at the end of the build and transfer them all the way to the server. This is normally desirable, it allows for distribution of the build process, but having 60gb of artifacts makes it rather slow certainly, requiring a 120gb transfer (60 agent-server for the first build, 60 server-agent for the second). There are several possibilities to improve on this, those you mentioned, some similar and some slightly different:
-Make them a single build, just different steps. This will remove the server as an intermediate step and the build will not be interrupted. Reduces flexibility but is the most straightforward approach.
-Set the snapshot dependency to "Always run on the same build agent". This will still allow for the independence between the two builds and will still upload the artifacts to the server, but will allow you to remove the artifact dependency as the artifacts will be almost guaranteed in the agent. It would be recommended to store them outside the work directory to ensure that they are not cleaned by teamcity's tools.
-This would be a complex task, but you could set up an external artifacts storage plugin that basically eliminates the second step entirely. You would just define them as artifacts and teamcity would store them there directly: https://www.jetbrains.com/help/teamcity/configuring-artifacts-storage.html#ConfiguringArtifactsStorage-ExternalArtifactsStorage . It can be set at the project level so it should be very easy to make it not impact other projects/builds, simply isolate this build in its own project.
-There might be something you can do to make those 60gb be able to work in an incremental fashion, moving deltas rather than the full assets. This depends on your toolkit and project, so it would be outside of our scope to help with it, but it might be an option to investigate.
-Similar to your project, but simplifying a bit, it should be possible to mount the share and force the output to go there. It would be similar but would reduce the requirement of having to handle the project around after it's been created.
Hope this helps.