Publishing to AWS S3 suddenly failed
We've been successfully publishing build artifacts to Amazon S3 for years using the plugin. Just like week this process started to fail, with the following error:
[Publishing artifacts] Failed to publish artifacts: Unable to access AWS. Check your network settings and try again.
As far as I can tell there is nothing wrong with our network settings. The publish fails intermittently. There is no other information given.
It's annoying that, if the upload to AWS fails, TC will simply lose the artifacts completely. I'd like to have the option to store them locally if the upload fails. Is there any way to do this?
But of course the publish failure itself is the main problem. Is there a way to get any further diagnostics on why this is happening?
I am using Team City Professional 2019.1.5 (build 66605)
Would upgrading to the latest version of Team City fix this? Or maybe I need to upgrade the AWS plugin?
Please sign in to leave a comment.
Unfortunately, until we have a better picture on why AWS connection began to fail, it`s not clear whether TeamCity or plugin update would help (albeit TeamCity update would not hurt either).
A similar issue about S3 has been reported recently; it appeared like S3 configuration did not have enough bandwidth to handle the data transmission before timing out. Could I ask you if number of artifacts/their size grew recently? Do you have any logs from AWS side regarding the upload failure? When the upload fails, do any of the artifacts make it to the storage?
I suggest to enable the debug logging for the agent(s) which are handling the build in question first (please see the details here: https://www.jetbrains.com/help/teamcity/viewing-build-agent-logs.html#ViewingBuildAgentLogs-GenericDebugLogging). When the issue reproduces, I would like to ask you to upload teamcity-server.log and the build log (former found at <teamcity-installation-home>/logs, latter available via "Download full build log" at the Build Log tab) by clicking on "Submit a Request" and attaching the files.
In regards to the "backup" step covering local artifact storage, here are the options:
1) as per https://www.jetbrains.com/help/teamcity/teamcity-configuration-and-maintenance.html, there can be multiple artifact directories you can configure. It is possible to save all artifacts locally, and then push them to S3. You could use a background task periodically running to check if there were new builds, or, possibly, a server-side custom plugin which could hook to a "build finished" event to start the artifact migration.
2) there is a feature request regarding migration of locally stored artifacts to S3, somewhat similar to artifact cleanup (https://youtrack.jetbrains.com/issue/TW-52778).