Resolving Build Artifacts takes a very long time

Hi,

I've been trying out TeamCity for a few days, and I'm having some troubles with build artifacts.
I have the following build configurations in my project:
- Boost: responsible for the compilation of the boost library. The generated libs are defined as Build Artifacts.
- Continuous build: The actual project build, which depends on the Boost build configuration, and its artifacts.

I have two issues:
- The transfer of the artifacts takes a very long time (about 20 mins) when the boost headers are included (~ 4000 files)
- The artifacts from the Boost configuration are transferred to the build agent each time a continuous build is triggered, even if they didn't change since the last continuous build on the same agent.

This means that each the continuous build is triggered, event for a very small change that would takes a few seconds to compile, the retrieval of the boost libraries takes ages, so the actual build always takes at least 20 mins.
Is there any way to improve this? I'm afraid there isn't much that can be done to transfer the boost artifacts more quickly (or maybe by compressing them?), but isn't there a way to prevent the build agent to download artifacts it already has?

Thanks,


Nicolas

6 comments

I think it is not a good idea to skip downloading files if they are already on the agent. First of all we don't really know what build script is doing with these files. It does not look good if files will be changed accidentally and then you will deliver a release build to your customers based on these files. Moreover one working directory can be shared among several build configurations. You'll never know what these build configurations actually do with these files.

It looks like the only way to reduce build duration is to pack these files into the one archive and write script to unpack it on the agent.

--
Pavel Sher

0

Hi,

Thanks for your reply.
Indeed it makes sense to always download all the dependencies when doing a release build to be shipped to client.
However this is really inconvenient for a continuous build system, where you really want the build to be done as quickly as possible. Couldn't this be an option in the Artifact Dependencies page?

As it is we can't really use this for continuous builds. Even if we pack/unpack the lib, it should still take at least 10-15 mins to get them (it's more than 1.5Gb). That's not acceptable when the actual build usually takes a few seconds.

I had another idea, which works quite well if a single build agent is involved:
I use the same Perforce source for my 2 build configuration, so that both configurations are built in the same checkout folder. This way I can refer to the files generated by the "Boost" build configuration in the second one.
This works quite well with one agent. I can modify the build trigger so that Boost doesn't get built if the change didn't occur in the Boost source tree, so the continous build effectively takes only a few seconds.

Unfortunately this won't work if several agents are involved, since there's no guarantee that the two configurations are built on the same server.
Is there a way to make that setup work with several agents? For instance, can I make sure that the latest Boost configuration has been built on the agent that builds the continuous build?

0

Indeed it makes sense to always download all the dependencies when doing a release build to be shipped to client.
However this is really inconvenient for a continuous build system, where you really want the build to be done as quickly as possible. Couldn't this be an option in the Artifact Dependencies page?


Yes this could be an option and I would suggest you to submit a feature request to our tracker: http://jetbrains.net/tracker

As it is we can't really use this for continuous builds. Even if we pack/unpack the lib, it should still take at least 10-15 mins to get them (it's more than 1.5Gb). That's not acceptable when the actual build usually takes a few seconds.


Not sure I understand why it is not acceptable, for example, our builds take minimum 40 minutes to complete and anyway we prefer to use continuous integration just because it provides builds and changes history. Moreover if you have more than one agent you will get builds with new changes faster.

I had another idea, which works quite well if a single build agent is involved:
I use the same Perforce source for my 2 build configuration, so that both configurations are built in the same checkout folder. This way I can refer to the files generated by the "Boost" build configuration in the second one.
This works quite well with one agent. I can modify the build trigger so that Boost doesn't get built if the change didn't occur in the Boost source tree, so the continous build effectively takes only a few seconds.

Unfortunately this won't work if several agents are involved, since there's no guarantee that the two configurations are built on the same server.
Is there a way to make that setup work with several agents? For instance, can I make sure that the latest Boost configuration has been built on the agent that builds the continuous build?


So if I understand you right you do not want to start dependent configuration on an agent until Boost configuration is built there. In this case you probably want dependent configuration to be built on the latest version of Boost lib. It means that when dependent configuration build starts it must make sure that Boost library on this agent is up to date. Is it possible to always build Boost library in the build script of dependent configuration?

If there is no requirement to always use latest version of Boost then you probably could build Boost lib in the manually specified checkout directory and then share this directory by network among your agents. However in this case you'll have to provide some synchronization to disallow cases when dependent configuration starts building on Boost library which is not fully built yet.

--
Pavel Sher

0

Yes this could be an option and I would suggest you to submit a feature request to our tracker: http://jetbrains.net/tracker

Will do.

Not sure I understand why it is not acceptable, for example, our builds take minimum 40 minutes to complete and anyway we prefer to use continuous integration just because it provides builds and changes history. Moreover if you have more than one agent you will get builds with new changes faster.

Well ok, it might be acceptable; let's just say it would be much better not to waste 20 min to download something that was already there. If I break the build by commit a change, I'd rather be notified 10s later than 20 minutes...

Is it possible to always build Boost library in the build script of dependent configuration?

Yes, that could be a possibility. The problem is that it also takes quite a lot of time to build the Boost configuration (which actually doesn't only contain boost library), so I'd rather not do that for each build either. I might end up doing this, though, since it should still be quicker than downloading the artifacts from the other build.

If there is no requirement to always use latest version of Boost...

Unfortunately there is, so I can't build Boost lib in the manually specified checkout directory and share it.

Thanks for your help!

Nicolas

0

Was a feature request ever added for this?  If so what is the bug #, I'd like to monitor this if possible, and maybe even add some of my own input.

0

Seth,

Yes, this feature request was added to the issue tracker, please watch/vote for it.
If you need another approach, feel free to file a feature request in our tracker.
If you do please describe the reasons why do you need this so we can better understand your needs.

0

Please sign in to leave a comment.