Snapshot dependency cleanup

Hi all,

I am experiencing the following problem with TeamCity 9: my snapshot dependencies are being cleaned up once a week!
The documentation explicitly says that it should not happen.

I used all default settings, and it seems indeed that the files should be kept forever.

My best shot this time is that TeamCity 'forgets' the build after reboot. Can it be my issue? If I reboot a server, will it still remember that a certain folder contains a dependency?

Next, is it possible to force building a dependency in case of need? What happens now is just the dependent build fails after a cleanup. I would like to setup the system in such a way that it first checks that a dependency is available, and if not, it builds it first.
Unfortunately, it seems that now it performs some checking, and it thinks that there is a ready build available, but this is not true :(

8 comments
Comment actions Permalink

Hi Maxim,

By default, TeamCity does not prevent dependency artifacts cleanup. If you want to prevent dependencies clean-up for all projects then open Root project Clean-up Rules and select "Prevent clean-up" in "<Default cleanup rules for all projects>".
If you want the build chain to rebuild if there is no suitable build, then you need to use snapshot dependencies.

0
Comment actions Permalink

Hi,

My problem is that I am already using snapshot dependencies...
And once in a week I get them erased.

0
Comment actions Permalink

It sounds strange.

>I would like to setup the system in such a way that it first checks that a dependency is available, and if not, it builds it first.
Snapshot dependencies works exactly as you described. By setting a snapshot dependency of a build (e.g. build B) on other build's (build A's) sources, you can ensure that build B will start only after the one it depends on (build A) is run and finished. So when you start build on B, the whole build chain is added to the queue A > B. If a suitable build A is found it is skipped. If not found (for example was deleted), it will be build before build B.

It is not clear why the build are deleted. Could you please attach screenshots of the configured clean up rules for affected projects? Also please attach screenshots illustrating the issue and teamcity-cleanup.log.

0
Comment actions Permalink

Well, in my case I tend to believe that the system thinks there IS a suitable build in a chain, while actually it is not true.
So the build was cleaned up, but TeamCity doesn't know it; maybe it is related to reboots between the sessions.

I am not sure what kind of information can be helpful, but I will try to illustrate. There is a project "TennisAC" with a snapshot dependency "SharedLibs" (see Attach 1).
The cleanup rules are set to defaults (see Attach 2).

Now, let's compare builds 145 and 146 (Attach 3).

In build 145 the system thought that there is a build of SharedLibs available, and started building TennisAC, which resulted in a "missing file" error.
Naturally, I went to the build log, then Parameters, and there was a line

dep.SharedLibs_BuildAll.system.teamcity.build.checkoutDir C:\Utils\TeamCity\buildAgent\work\3560295a39953123

When I checked this folder, I found it being empty. So I manually rebuilt SharedLibs, and voila, build 146 is just fine. (The previous build 144 was also fine, since it happened before the cleanup).

The clean up log of SharedLibs is not very informative, as it does not say what exactly was deleted:

[2015-07-16 03:00:00,026]   INFO -  jetbrains.buildServer.CLEANUP -  Processing SharedLibs :: Build All {id=SharedLibs_BuildAll, internal id=bt8}
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP - Executing 22 global cleaners
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (1 of 22): BackgroundBuildDataCleaner
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (2 of 22): TestFailureRateCollector
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (3 of 22): ObsoleteFilesCleaner
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP - Searching for obsolete log files (not used by any build) under build logs directory: C:\Users\rg_software\.TeamCity\system\messages
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP - Cannot find obsolete build log directories in old format
[2015-07-16 03:00:00,041]   INFO -  jetbrains.buildServer.CLEANUP - Searching for obsolete artifact directories (not used by any build) under directory: C:\Users\rg_software\.TeamCity\system\artifacts
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP - Cannot find obsolete artifact directories
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (4 of 22): InspectionDataCleaner
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP - Start cleaning inspection_data dictionary...
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP - No unused hashes.
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP - Finished cleaning inspection_data dictionary
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (5 of 22): BuildDataStorage
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (6 of 22): PersonalBuildCleaner
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (7 of 22): jetbrains.buildServer.controllers.login.RememberMe$2
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (8 of 22): CustomDataStorageManager
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP - Removing old files in custom data storage...
[2015-07-16 03:00:00,119]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (9 of 22): OldAgentsCleaner
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (10 of 22): AgentTypeDatabaseStorage
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (11 of 22): TestName2Index
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP - Start cleaning unused test names ...
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (12 of 22): ResponsibilityCleaner
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP - Starting a clean up of responsibility entries...
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP - Responsibility entries to process: 0
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP - Completed a clean up of responsibility entries: 0
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (13 of 22): TestHistoryCacheCleaner
[2015-07-16 03:00:00,135]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (14 of 22): BuildArtifactsCleaner
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (15 of 22): InvalidHistoryRecordsCleaner
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning invalid history records without build_state peers ...
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning invalid history records without build_state peers for 0 builds, build ids: []
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (16 of 22): TestMutesCleaner
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (17 of 22): AuditLogCleaner
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP - Removing audit log records older than 365 days ...
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP - Removing audit log records for obsolete tests ...
[2015-07-16 03:00:00,166]   INFO -  jetbrains.buildServer.CLEANUP - Removing audit log records for obsolete builds ...
[2015-07-16 03:00:00,182]   INFO -  jetbrains.buildServer.CLEANUP - Finish removing audit log records.
[2015-07-16 03:00:00,182]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (18 of 22): DuplicatesDataCleaner
[2015-07-16 03:00:00,182]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (19 of 22): ClearcaseCacheGeneralDataCleaner
[2015-07-16 03:00:00,182]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (20 of 22): PatchCache
[2015-07-16 03:00:00,182]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (21 of 22): MetadataStorageBean
[2015-07-16 03:00:01,086]   INFO -  jetbrains.buildServer.CLEANUP -  Executing global cleaner (22 of 22): InvestigationTestRunsHolderCleaner
[2015-07-16 03:00:01,086]   INFO -  jetbrains.buildServer.CLEANUP - Global cleaners finished
[2015-07-16 03:00:01,086]   INFO -  jetbrains.buildServer.CLEANUP - Clean-up took 1s
[2015-07-16 03:00:01,102]   INFO -  jetbrains.buildServer.CLEANUP - Lock for the maintenance process released



Attachment(s):
Clipboard03.png
Clipboard02.png
Clipboard01.png
0
Comment actions Permalink

Do I correctly understand that you want BuildAll build configuration to use some files (artifacts) produced by SharedLibs build configuration? If yes, then you have to configure artifact dependency in addition to snapshot dependency.
As far as I understand you use checkout directory to share files between these two build configurations. It is not recommended way because checkout directory can be deleted and also no previous version of artifacts is stored.
If you configure SharedLibs to publish artifacts, then they will be sent from agent to server and stored on server according to clean-up rules. If you configure artifact dependency from BuildAll to SharedLibs, then it will download needed artifacts before the build from the server.
Please find setup examples in this section: https://confluence.jetbrains.com/display/TCD9/Build+Dependencies+Setup.

0
Comment actions Permalink

Well, it DOES use some artifacts, but not only. SharedLibs is a collection of C++ libraries, for example, including boost (which contains 10,000+ header files).

So naturally I want to share files between the configurations using snapshot dependencies (this IS the reason why snapshot dependencies exist, isn't it?)

If I follow the 'artifact' method, I will have to publish thousands of artifacts.

You say that 'checkout directory can be deleted', but the documentation also says that 'TeamCity always preserves builds which are used as snapshot dependencies in other builds. These builds and their associated data (e.g. artifacts) are never deleted by the clean-up procedure.'
So isn't it true that the checkout directory in this case shall not be deleted?

And what is the recommended way of sharing data between the configurations if this data consists of enormous number of files (as in case of any reasonabley large C++ libraries)?

Sorry for so many questions, but I am really puzzled.

0
Comment actions Permalink

>So naturally I want to share files between the configurations using snapshot dependencies (this IS the reason why snapshot dependencies exist, isn't it?)

No, snapshot dependency is not intended to share files. A snapshot dependency allows launching builds from several build configurations in a specific order and ensure they use the same sources snapshot from VCS (sources revisions correspond to the same moment).

>You say that 'checkout directory can be deleted', but the documentation also says that 'TeamCity always preserves builds which are used as snapshot dependencies in other builds. These builds and their associated data (e.g. artifacts) are never deleted by the clean-up procedure.'

So isn't it true that the checkout directory in this case shall not be deleted?

Clean-up is performed on TeamCity server (not agents). The “build” means the build artifacts and other data that are stored on server and in database related to the build.

Checkout directories on agents are clean by there own rules: after some time or if there is not enough space on agent machine. Find more details in section Automatic Checkout Directory Cleaning.

>And what is the recommended way of sharing data between the configurations if this data consists of enormous number of files (as in case of any reasonabley large C++ libraries)?

If you want to share files between build configurations then use the artifact dependency or for .NET NuGet Feed can be used. If you do not need to store old artifacts you can configure more strict cleanup rules.

Also you can set the following parameter system.teamcity.build.checkoutDir.expireHours for SharedLibs build configuration to "never", so checkout directory will be never deleted. However using artifacts dependency is a recommended way.

0
Comment actions Permalink

I see. OK, finally it becomes clearer :)
Thanks a lot. It wasn't obvious.

0

Please sign in to leave a comment.