Teamcity AWS ECR integration is issuing deletes to the registry and we can't find where

Answered

We had the "registry cleanup" for ECR turned on in a couple of jobs (the `On server clean-up, delete pushed Docker images from registry` in the docker integration tab), and then we realized it was deleted our production images.

So we turned it off everywhere we can find it.

But it's still deleting our production images.

It feels like the deletes are queued up; it seems to ONLY be issuing deletes for the image tags we use for production at this point.

So, questions:
1) Is there any way to like..list everything the server cleanup is doing and preferably modify it?
2) Is there any way to figure out if we've missed a docker integration that might still be firing someplace?
3) how do I troubleshoot this?

5 comments
Comment actions Permalink

Hello!

Cleanup may be debugged using <TeamCity Server Home>/logs/teamcity-cleanup.log(.X) files. As per my check, Docker cleanup extension should log INFO-level messages starting with "Removing docker images" and "Removed docker image", for the start and finish of the cleanup batches, respectively. You can look up those entries on your logs - in case of any doubt, please feel free to share them with me via https://uploads.jetbrains.com/. 

An easy way to find all instances of Docker Support build feature is to perform file search in the <Data Directory>/config/projects (and its subdirectories). Every build configuration/project is stored as an .xml file internally, so you could look them up by searching for similar fragments:

      <extension id="DockerSupport" type="DockerSupport">
<parameters>
<param name="cleanupPushed" value="true" />
</parameters>
</extension>

Please let me know if this helps to pinpoint the source of issue, or if there is anything else I can assist with. 

0
Comment actions Permalink

So none of the configs have cleanupPushed in them, and yet we're still issuing batch deletes to aws. I could try manually adding `cleanupPushed value="false"` but that is also painful.

The logs do show teamcity issuing those deletes and the deletes failing because it tries to delete the same image many times and only the first attempt succeeds.

But it shouldn't be deleting them at all.

0
Comment actions Permalink

Is there a way to globally disable `org.jetbrains.teamcity.docker.cleanup.PushedImagesRegistryCleaner`

0
Comment actions Permalink

Hello!

I have reached out to the development team; they have confirmed that the logic works as following - if a build has pushed an image *when this feature was turned on*, then it will cache the feature configuration in the hidden artifacts and will be a subject to cleanup. New builds which were started when the cleanup feature was disabled are not affected.

The devs are currently working on a fix for this issue (registered as https://youtrack.jetbrains.com/issue/TW-71285); as an immediate workaround, you could consider amending the cached settings for the affected builds. 
Every build in TeamCity has hidden artifacts; these are stored alongside the usual artifacts, in the .teamcity folder. The full path to the cached build settings file of a specific build is as following:

<Data Directory>\system\artifacts\<project ID>\<build configuration ID>\<build ID>\.teamcity\settings\buildSettings.xml

It looks like you could work around the issue by searching for the 

<param name="cleanupPushed" value="true" />

in the 

<Data Directory>\system\artifacts

and, if there are any buildSettings.xml files found which contain this line, amend them to remove the parameter. (For example, notepad++ has a "replace in files" functionality which could help with this).

I am most sorry for the inconvenience this issue might have caused. If there is anything else I can assist with, please let me know.  

0
Comment actions Permalink

Ah ha!  ok, great. I did that.

`cd ${TEAMCITY_DATA_DIR}/system/artifacts; sudo grep -l -R cleanupPushed * | sudo xargs -d '\n' sed -e 's|<param name="cleanupPushed" value="true" />||'` if you want to give folks a one liner in the future. :-)

Thank you!

0

Please sign in to leave a comment.