Build cache outside of agent directory

Hi,

I was testing out the new experimental Build Cache feature and noticed the docs said the rules are relative to the checkout directory.

Does this imply that it's impossible to add folders in the user's homedir?

For example, if I want to keep Maven and sbt caches, I'd love to be able to cache files in the homedir:

```
%env.HOME%/.m2/repository/
%env.HOME%/.cache/coursier/
```

This doesn't seem to work. Will this be supported or is there a way to keep Maven or sbt / Coursier caches already?

0
7 comments

Hi! Currently, as of TeamCity Cloud 2023.03, only relative paths inside the checkout directory are supported. This means sharing the Maven cache from the user's home directory is not supported yet. We plan to support it in the future. In the meantime, you can do the following:

 

1. Add the -Dmaven.repo.local=%teamcity.build.checkoutDir%/.m2 argument to the Additional Maven command line parameters field in the Maven step settings.
2. Set the Artifact repository to Maven default in the Maven step settings.
3. Specify .m2/repository in the Publishing rules field of the Build Cache feature settings.

 

Steps 1 and 2 will make Maven to save it's cache in the .m2 directory under the checkout directory during the TeamCity build. Step 3 will point the feature to that directory.

Please note: the current state of the feature is experimental. That means the functionality and UI will change in the future.

0

Thanks for the help here. That was great! My Maven build went from 15 minutes to 2 minutes.

I tried this with npm and sbt as well, but fetching the cache archive is a bit slow so it was actually about the same or a few percent slower than the uncached builds for those. Hopefully the transfer performance can be improved in the future, but this is great so far.

0
Thanks for the feedback. Could you share what the sbt and npm cache sizes are?
0

Is it possible to do the same for docker images ?

Usually they are stored in

 

/var/lib/docker/overlay2
/var/lib/docker/image

0
Hi Mukundhan,

Could you provide more details on your current setup and what you want to achieve? 

Thanks,
Anton
0

Hey Anton,

 

Background is, we are using docker images in the build steps and they are being pulled on each run as we are using cloud agents.

 

We would like to reduce the time spent on pulling the images and cache the images using, the build cache feature.

 

The images and volumes are mostly stored in the above location along with the checksums.

 

When we set the path of caching outside of the build directory they are not being cached.

 

Is there a suggested ways of caching docker images in teamcity builds ?

 

TIA,

Mukundhan

 

 

0
Dear Mukundhan,

The build cache build feature caches the files produced by the builds, so it won't work for Docker images, as they're pulled (and cached) by Docker.
I assume that by Cloud agents, you mean the self-hosted Cloud agents, not JetBrains-hosted agents available only in TeamCity Cloud. If that's the case, I suggest re-creating the VM image after pulling the required images. The Docker will then reuse it.

Best regards,
Anton
0

Please sign in to leave a comment.