Beware! Bulk registrations
Hey JB
I do not know if this is the right place to share this. But today I learned that in the past 3 days someone bulk registered like 30 fake accounts on my self-hosted TC instance. I already secured my roles and access to read-only for new users in the past. So aside from being able to bulk-download my artifacts, I could not see any harm done.
Truth to be told, I am surprised someone even bothered to go this far considering there was nothing to gain except for making a bit of a mess. I tightened my security even further and made it impossible to sign up manually. Removed the basic auth and token-based auth for scripts. 2FA was and remains optional but users now must connect with either a GitHub or GitLab account.
Logs didn't indicate any login attempts with my own admin account and the audits did not show any alterations to any configurations. Although, there is one file that I know nothing about that kept being updated after each user registration with no changes. This file is called internal.properties and is located in the data/config folder. Other than that I have noticed nothing that should lead to concern.
With this post I like to raise awareness and report a security issue. Although I think I fixed my problem I am still happy to share logs to the JB team if it would help them.
With kind regards,
Ruben Labruyere
Please sign in to leave a comment.
Hi Ruben,
I’m sorry to hear your server was compromised. We published a security announcement on Monday of this week, advising customers of two critical security vulnerabilities. We released a fixed version on Monday (version 2023.11.4) and provided a security hotfix plugin for those customers who couldn’t upgrade immediately.
Unfortunately, attackers were able to exploit the vulnerabilities shortly after the release of our fix because full exploit examples had been published online by a third-party.
Please review the following blog post for the full details: https://blog.jetbrains.com/teamcity/2024/03/additional-critical-security-issues-affecting-teamcity-on-premises-cve-2024-27198-and-cve-2024-27199-update-to-2023-11-4-now/.
We did email all customers on Monday (along with our Security Bulletin subscribers), but if you’re using the free TeamCity Professional license unfortunately we may not have had your contact details on file in order to notify you accordingly.
We recommend taking the server offline to isolate it from both external and internal networks.
We also recommend double-checking the Audit screen on the TeamCity server to ensure no suspicious actions have been performed, such as any unknown user access tokens created or deleted, any unknown users created, login actions, etc. These can be filtered by the “User actions” category on the Audit screen.
There is also a “teamcity-activities” log file that can provide further information. It’s recommended to review installed/deleted plugins on the server, and the presence of any unknown projects and build configurations (plus any changes to existing projects - the Audit screen can also help with this).
Event logs should be checked on the host operating system to spot any unknown events logged by external processes. Checks should be carried out for the presence of any unrecognized processes running in the background on the host server, such as malware, cryptomining, and other unauthorized tools.
In case any suspicious actions are detected, ensure the server is still isolated and start to rotate all credentials for users. There will also be a varying number of credentials for any external services that TeamCity and its host operating system had access to, such as repositories and connections (e.g. Slack, AWS, Docker Registries, etc), the TeamCity database connection, and network drives.
If the server has been compromised, any build agents connected to the server during that period should also be considered compromised. Any build artifacts generated from when the server was attacked to now should also be considered unsafe.
Importantly, the server should be upgraded to the latest version or appropriately patched.
If you have any other immediate concerns or wish to reach out to us privately to discuss your situation, please send an email to teamcity-support@jetbrains.com and we will advise further.
Hi Daniel
Thanks a lot for your extended answer. I went through all the things you recommended and as far as the logs tell, nothing else but the users were created. Based on the logs of my server I was able to determine the users were created using the API and with a HTTP client from Python.
I also noticed I received several TC server updates in a row. I also updated the TC server instance a couple of days ago. That explains the timing of these attacks.
That said. I am grateful for your help here, but I do not agree with the fact that I will not be notified of critical security issues just because I am not paying for TC. If I am not mistaken this is even illegal since it violates the GDPR law which clearly states in case of critical security issues, you have to notify all your users. What worries me the most however is that my server, which hosts other websites as well, could have been badly damaged because I was not given the opportunity to prevent prevent this in time. I was lucky this time so I am cool but who knows what will happen in the future.
At the other hand, I understand the issue. Because while I may not be paying for TC, I do pay for a subscription for other JB products. So I am a customer. You guys just could not know I was using TC too. Is there some way, despite I am using the free license, to make me part of the mailing list should such issues happen in the future so in the very least I will be notified? Or, maybe you could suggest to your development team to introduce some sort of feature to connect free services to an account? That way you guys know I use TC and have access to my email. If none of this is possible, I understand. But that could make me consider looking elsewhere for a CI/CD solution. And I hope it will not get to that.
Hi Ruben,
I’m glad to hear the damage wasn’t more serious.
Unfortunately we haven’t previously captured the contact details for users who downloaded or installed TeamCity Professional, as it’s been freely available for download. It was only a few months ago that we added an email address capture form to the download page.
We will be providing a mechanism later this year to associate TeamCity with your JetBrains Account, making it easier to receive these types of important notifications in the future.
In TeamCity 2023.11.4 we have also added the ability for a TeamCity server to show critical security alerts in the user interface (in the notification bar at the top of the screen).
In the meantime I recommend subscribing to our Security Bulletin by visiting this page: https://www.jetbrains.com/privacy-security/subscribe/.
Thanks for your understanding.
Hi Daniel
Thanks a lot for the heads up. I subscribed to the Security Bulletin and I am glad to hear the mechanism will be added! That is good news. Basically the feature I suggested.
As for the security alert in the TC interface. I do not open TC every day, nor look at it. If it builds I get an email and I am good. It could have been the reason why I missed it, otherwise, it was not shown. Does this happen automatically or do I need to enable a setting to make this happen? Thanks in advance.
EDIT:
I just checked. I updated TC yesterday for the last time I think. It is possible that the notification was not there yet.
Hi Ruben,
The capability for TeamCity to show critical security notices directly in the UI is automatic, and it's a feature that was only added in 2023.11.4. Previous versions didn't have the capability. The initial implementation of this capability is designed so that users currently within the TeamCity user interface will see the notification. This capability may be extended further in the future.
The upcoming mechanism to associate TeamCity with your JetBrains Account will allow you to receive email alerts for these types of vulnerabilities (along with subscribing to the Security Bulletin).
Thanks.
I also encountered this same issue at around the same time as the OP - a bunch of user accounts had been created.
I too never received any notification about the critical nature of this issue, although I am paying for an additional agent license.
I only became aware something was up when I got a bunch of returned mail notifications from the email account I had setup Team City to output notifications. It was obviously trying to send notifications about my builds to non-existent email addresses, and that's what alerted me to the rogue user accounts. It took me a day or two to actually register that these emails weren't just regular SPAM somehow finding its way into my inbox.
I generally use TC every day and usually patch right away when there is an option. The last few patches have worked fine, but I think last year one of the updates required me to reconfigure the DB connection (nothing was lost) so for a while after that I was a bit hesitant to install the updates.
I love using TC but this has dented my confidence a bit in the product, if I'm honest.
Hi Oliver,
I'm sorry to hear you were also impacted. Please could you reach out to us via the Get In Touch page, letting us know the details of your agent license (e.g. provide your associated email address or license key) - I can help make sure your JetBrains Account containing the TeamCity agent license is subscribed to receive future security notifications.
We've also published this blog post outlining some recommendations for investigating a compromised TeamCity On-Premises server and some possible remediation actions.
Thanks.