Artifact packaging with Octopus push step now intermittently fails with 2023.05

We had to work around this issue by adding a delay step in before the Octopus step - once we did this the problem has gone. Which all but confirms my theory above - that there is some sort of caching by ArtifactsBuilder and in particular circumstances that cache is not being refreshed quickly enough for services that subsequently make use of it (an Octopus Deploy plugin in this case).

It is a crappy workaround - but at least it stops the builds failing until someone from JetBrains bothers to responds to this post.

Hi Kirill,

Shrinkwrap was indeed the first thing I looked at. However all it is doing is simple File.Copy, File.Delete operations within a C# executable that completely exits prior to TeamCity starting the next step. As I mentioned in the original post this process has worked flawlessly for years - the only thing that obviously changed was the latest TeamCity version upgrade (I continuously upgrade each time TeamCity reminds me to, so not like we were coming from an old version).

We have got our reliability of builds working back by keeping that delay step in I mentioned above - it is not ideal but as we have a workaround it isn't something we are going to invest a lot more time on. With no visibility to your source code I can make no other suggestions - whether that code is relying on file system change event notifications which are not happening, it is using cached data or some other bug that has introduced this behaviour.

Hi Kirill - not sure how to determine what the previous version was? As I mentioned I always install within a day or two of TeamCity notifying me that a new version is available, so it would have been whatever version you guys released publicly before 2023.05 build 129203 which has had the issue. I see there is now 129321 available so that will be installed today.

And yes the files not found are indeed removed by Shrinkwrap. Our TeamCity release build compiles various C# service executables into a series of subfolders. We then want to zip that whole parent output folder up for handing off to Octopus, however to reduce the zip size down we have a de-duplication step performed by our Shrinkwrap tool. It iterates across all those output subfolders, finds any duplicated dlls between them, moves one copy of each into a "common" folder, records the original positions in a manifest file and then deletes all the remaining "duplicates" from those folders.

So in the error stack I posted, the "RegulatoryFeedEngine" subfolder containing the binaries for that service included a "Rebex.Networking.dll" when it was compiled by the TeamCity build step. However as other services also had that particular .dll, Shrinkwrap would have done either a .NET File.Move() to the shared  folder or a File.Delete() on it. There is no background threading in Shrinkwrap, it is a synchronous process with very simple file operations.

However when the Octopus step executes and invokes the TeamCity Java api for archiving, for whatever reason it is thinking that there should still be a Rebex.Networking.dll in that Services folder, tries to add it to the archive it is generating, and of course fails because the file is not physically present in that folder. You can see the Shrinkwrap process executable executed at 10:33:29 in that log above, then a second later the Octopus step starts trying to archive the folder contents and boom...

It didn't happen "every single time without fail" - if a developer manually triggered the build it usually succeeded. Yet the scheduled builds were failing "most" days after the TeamCity upgrade. It was very intermittent - but adding a 15 second delay step in between Shrinkwrap and Octopus allowed "something" on the server to get itself up to date such that it has worked flawlessly again ever since.

Very confusing given you are saying there is no caching. We don't use symlinks that I am aware of. On our Windows 2019 build server we have both TeamCity itself and 3 TeamCity agents all on that same machine. 

Hi Kirill - sure that sounds an interesting idea. I will see what it reports the next few days (as this was an intermittent issue) across the builds to see if at any stage files appear listed which should not be and let you know.

Hi Kirill, just a small update for you. I added a Powershell step to list out the files, and as expected it shows them deleted from the folder after the Shrinkwrap step. I have now disabled the "delay" step I added (keeping the Powershell list step) - and it has run successfully all the way through which is not overly helpful in pointing out the culprit!

Of course two things have now changed since the original problem above - a subsequent TeamCity update installed, plus adding this Powershell step which itself will be causing a small delay between Shrinkwrap finishing executing and the Octopus packaging step running.

Will give it another day or two to see if the problem reoccurs with the current setup, if not then try disabling the Powershell list step and see how it behaves then. It always was an intermittent problem unfortunately so don't want to celebrate it quietly fixing itself without a longer run...