Debugging TeamCity cleanup

I understand that a build, say X, will not be cleaned up if another build is pinned and depends on build X. Are there other rules that prevent a build from being cleaned up?

We have a number of old builds which are not being cleaned up and I do not understand why. Is there some debug option that can be turned on during cleanup to display why a particular build is not being cleaned up?

7 comments
Comment actions Permalink

William,

Turning server debug logging for the cleanup time can provide somewhat more information in the logs. However, the logging might be too verbose.

For the dependencies that might hold the build, please check the Dependencies tab of the build that was not cleaned up.

If there are no dependencies and the build falls under cleanup rules, we would be interested in more details to investigate the issue.

0
Comment actions Permalink

Thanks Yegor. There are dependencies and I have "Prevent dependency artifacts cleanup" marked, so it is explainable and my confusion is the misunderstanding of how this works.

I'm still not 100% sure of what should be happening and I would request that you enhance the documentation with some more examples. I think there is also an error in the documentation which states (http://confluence.jetbrains.net/display/TCD65/Clean-Up):

Say, a build configuration A has an artifact dependency on B. If Prevent dependency artifacts clean-up option is enabled for A, the builds of B that provided artifacts for  the builds of A will not be processed while cleaning artifacts, so that  artifacts will be preserved.

My observation is:
Say, a build configuration A has an artifact dependency on B. If Prevent dependency artifacts clean-up option is enabled for B, the builds of B that provided artifacts for  the builds of A will not be processed while cleaning artifacts, so that  artifacts will be preserved.

I suggest clearer and corrected text and an additional example:
Example 1:
A build configuration A has an artifact dependency on B. If the Prevent dependency artifacts clean-up option is enabled for B, the builds of B that provided artifacts for  the builds of A will not be processed while cleaning artifacts. Hence the artifacts in build B are always preserved even if the clean-up rules for A result in the artifacts for A not being preserved.

Example 2:
A build configuration A has an artifact dependency on B. If the Prevent dependency artifacts clean-up option is not enabled for B, the builds of B that provided artifacts for  the builds of A will be processed while cleaning artifacts. Hence the artifacts in build B will not necessarily be preserved even if the artifacts of A are preserved.


Furthermore, it is not clear to me in the examples above what will happen when build configuration A is pinned and B is not pinned. Does it make any difference to the above?

Finally, please also make clear what would happens in the examples above if B is pinned and A is not pinned.

0
Comment actions Permalink

William,

Sorry fort the delay in replying.

Actually, the intended behavior is the one correctly described in the documentation example.
The point is that if you have an important build using artifacts of other builds and you turn "Prevent dependency artifacts clean-up" option for it, all the builds providing artifacts are preserved.
As a side note, it seems that we should rework the approach as the current one does not cover all the cases, so suggestions as to how you would like to see it are welcome. However, at this time it should be as described in the doc.

Can you please doublecheck your observations and make sure they are not influenced by other settings?

> Furthermore, it is not clear to me in the examples above what will  happen when build configuration A is pinned and B is not pinned. Does it  make any difference to the above?
When a build of A is pinned, it is not cleaned up. As it is not cleaned up and has "Prevent dependency artifacts clean-up" option set, all the builds that provided artifacts for the build are not cleaned up either.

> Finally, please also make clear what would happens in the examples above if B is pinned and A is not pinned.
If a build of B is pinned, it will ot be cleaned up no matter what other options are set. This will not affect cleanup of the builds that depend from B's artifacts.

0
Comment actions Permalink

Thanks that definitely helps. Still my problem is artifacts are not being deleted when I'd like them to.

Back to the example,  neither A nor B are pinned, yet the artifacts for B are not being deleted. I expect you to say they should be deleted - is that right? If so I think there is a bug. Our dependencies are actually a lot bigger than a simple two build dependency, hence I'd like to debug what is preventing the deletion of B artifacts.

We actually have X depends on Y depends on Z and Prevent dependency artifacts clean-up is turned on for all 3 builds. Case 1: When just Y is pinned, I'd now expect from the documentation that artifacts from Z are kept and artifacts from X are deleted. Case 2: When nothing is pinned I'd expect that all artifacts would be deleted. What I observe in Case 1 is as expected, but for Case 2: artifacts from X are deleted but artifacts from Y and Z are kept. How can I find out why artifacts from Y and Z are being kept? These artifacts are filling up our disk causing lots of disk space issues! What I did was look at a build Z that I expected to be deleted and  looked at its 'delivered artifacts' (Y) and this was not pinned and  likewise if I walk up the entire tree of delivered artifacts.

With respect to suggestions to improve, I'd be mostly be content if the above worked as I expected in the previous paragraph. A 'rather nice to have', would be an additional option to keep the artifacts from X if Y is pinned. In real life, Y is a package to be deployed and X are some tests run against the package Y and we need to keep the test results. There are also various tests U and V also depending on Y and so we'd like to keep all the test results for pinned Y builds.

William

0
Comment actions Permalink

William,

Your approach to determine why build of Z is not cleaned is right: looking at delivered artifacts and checking that neither of the build configuraitons mentioned has "Prevent dependency artifacts cleanup" set to true.

If this method confirms that all the build configurations using the artifacts have the option set to off
and the build is not pinned
and the artifacts should be cleaned according to the artifact cleanup rules
and these artifacts are not hidden ones (not under .teamcity directory in artifacts)
then it's about time to investigate the logs/settings as this can be a bug.

As to retaining builds depending on the pinned one:
this is doable, but we should devise a way to introduce the option and not make the entire cleanup settings even more complicated then they are now.
There is a related issue that you can vote on: TW-15524.

0
Comment actions Permalink

I mentioned that all the builds are showing Prevent dependency artifacts clean-up, presumably that means set to true. What you are saying is that if Z has Prevent dependency artifacts clean-up showing and because Y is dependent on Z, there is no way to get rid of the artifacts from Z even if Y is not pinned.

I was assuming that if the artifacts from Y are cleaned, then it would break the dependency, but clearly not. Will the dependency be broken if "Clean Everything" or "Clean History" is used instead for Y? I was actually hoping to keep old unpinned builds of Y, but get rid of the artifacts.

My use case seems simple enough and TeamCity doesn't quite meet it: I would like to pin a build and keep all dependent builds up and down the chain. If builds are not in the chain, then the artifacts should be deleted as we don't have the disk space for these old artifacts. I guess, I will have to remove all the Prevent dependency artifacts clean-up options and manually pin every build in the dependency chain.

Perhaps there is a tool that will pin all dependent builds instead? If not it would be a very nice option to add to TeamCity: Click on a build and it'll display all the builds up and down the dependency chain in one dialog/web page with a tickbox against each to pin or not pin.

0
Comment actions Permalink

> I mentioned that all the builds are showing Prevent dependency artifacts clean-up

Sorry, I was trying to be a bit generic there, probably.
If the option is set to ON (say, in Y), then no artifacts will be cleaned in the builds that delivered them (builds of Z), until the build of Y is present in the history. When the build (of Y) is cleaned up, only then the artifacts will lose "protection" provided by "Prevent dependency artifacts clean-up" option.

> if Z has Prevent dependency artifacts clean-up showing
The option in Z has no significance since it is Y that is consuming artifacts, not Z.

> I was assuming that if the artifacts from Y are cleaned, then it  would break the dependency, but clearly not.

You are right, option/change in the behavior is requested by TW-5509.

> Will the dependency be  broken if "Clean Everything" or "Clean History" is used instead for Y?

Yes, when there is no nbuild of Y, nothing will "hold" the build of Z with artifacts.


> Perhaps  there is a tool that will pin all dependent builds instead? If not it  would be a very nice option to add to TeamCity: Click on a build and  it'll display all the builds up and down the dependency chain in one  dialog/web page with a tickbox against each to pin or not pin.

There is no such feature yet, but this sounds exactly as TW-15524.

0

Please sign in to leave a comment.