Publish build artifacts from Kubernetes
Hello,
We are going to be running multiple (~50) tests/tasks in parallel per pipeline and are using Kubernetes and AWS to run the tests/tasks. The test/task jobs are submitted to Kubernetes from the teamcity build. We are using AWS S3 as the artifact storage for the pipelines.
How do we retrieve the artifacts from the Kubernetes jobs in a performant manner?
I tried uploading a file to the s3 bucket directly while a pipeline was running but that file was not visible from the TeamCity artifacts tab. If that had worked we could just have written to the build's bucket directly from each Kubernetes worker node.
I also tried adding an artifact path to the uploaded file in the hopes that it would modify the .teamcity/artifacts.json manifest but it didn't work.
Thanks,
Cameron
Please sign in to leave a comment.
Hi Cameron,
I'm not sure I follow. As long as you have set up the Artifact paths in the build configuration, the artifacts will be automatically published to the build results and they will be moved to S3 if that's what you are using, then they should just be available there. Are you trying any other approach instead?
If you consider this to not be performant enough, that's a separate issue that we might want/need to tackle on its own, but we would need to have data about performance.
If this is not what you are looking for, could you please specify exactly what you are trying?
I’m not sure how to directly access files on Kubernetes worker nodes from TeamCity as we haven’t found any Rest API functions that write artifacts.
How would you use TeamCity to run several hundred jobs (say 700 for now) in parallel.
These Kubernetes worker nodes shutdown after they’ve executed their task.
What would the artifact path be in this case?
Thanks so much,
Cameron
Hi Cameron,
I might be missing something here. Are the build agents running on Kubernetes? If that's the case, the agents will automatically pick up the files from the working folder on the relative path you set using the "Artifact Paths" configuration on the General Settings of the build configuration. You don't need to access any of them manually.
I'm also not sure what you call "700 jobs". TeamCity runs builds, and often we see builds being referred as "jobs". Running them in parallel requires TeamCity agents to be spun up and run those builds. If that's the case, then what is mentioned above should work just fine, but I get the impression it's not the case and you are doing something different, so you would need to be much more specific about exactly what it is for us to be able to provide guidance.
If the agents aren't running on Kubernetes but on AWS or similar, and those are just pushing tasks into Kubernetes by a different approach, then you will need to bring the files to the build agent, and publish them via the artifact paths or any of the prepared methods for adding artifacts to a build, be it our meta runner: https://github.com/JetBrains/meta-runner-power-pack/tree/master/artifacts-publisher or a service message: https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-PublishingArtifactswhiletheBuildisStillinProgress
About other methods to add build artifacts to an existing build, copying the files into the correct artifact folder should work just fine, but keep in mind that artifacts are written at the end of a build, which means that TeamCity might overwrite its contents if you are adding them during the build unexpectedly. We have a request in our tracker to be able to, asynchronously have a way to add artifacts to a build: https://youtrack.jetbrains.com/issue/TW-11277