Very slow artifact downloads
I'm in charge of a TeamCity server that's experiencing a significant slowdown when downloading artifacts (clicking on the artifact link in the browser, and downloading the zip/tar file)
Server specs:
Win2K12R2 Hyper-V VM on our Host at the data center, which we access over a 1G metro line.
4 GB RAM
2 Procs
1G NIC (The Hyper-V host itself has dual paired 1G NICs)
Storage for Hyper-V is provided by an EMC VNX over a 1G iSCSI connection.
Data files live on the VM itself. Database lives at AWS.
When we first began using the server, users would consistently get download speeds of 40M+ and now it's dropped down to about 8-10M. Our builds are typically 9-15GB so this makes a big difference for our devs. I played with the Java heap settings and added this to the 2K12 server's env variables:
TEAMCITY_SERVER_MEM_OPTS -Xmx1024m -XX:MaxPermSize=270m
Reboot, and speeds jumped up again, but only for a while. Now they're back down again.
I realize there are many components in play here (Hyper-V host, VM, mulitple links, etc.) but as I mentioned, everything was speedy at one point.
After more troubleshooting, I've eliminated it having anything to do with Java. I run an nginx proxy in front of the server and I've added a location that allows me to browse the artifact directories. When I download directly from the artifacts, while on the TeamCity host, I get the same, 8-10M speeds. Same thing goes for downloading on the TeamCity host, through the regular TeamCity artifact dropdown.
Could it be disk I/O? Another idea: the TeamCity disk is currently 1.5 TB and has about 230GB free. Would TeamCity slow down as the disk fills up?
Thanks.
Please sign in to leave a comment.
Hello Ben,
Please check the related section in the documentation: https://confluence.jetbrains.com/display/TCD9/Known+Issues#KnownIssues-SlowdownloadfromTeamCityserver.
Could you try to run a simple http server (non-TeamCity related, e.g. a clean Tomcat server ) side by side on the same machine as TeamCity server and measure the download speed you get from this test server?
Sorry, forgot to mention that I'd already read that doc and confirmed the server settings are correct.
I was a bit unclear in my sentence above about downloading directly from the artifact links. I added a "location" to my nginx webserver/proxy that pointed directly to the artifacts directory, thus avoiding TeamCity altogether and I get the same slow downloads. My inclination is that it's a disk I/O issue, but then that doesn't explain why the speeds were high for the first month, unless it's related to how TC caches, etc.
I have a ticket open with support and they've responded with some ideas. I'll try them and post back here if I come up with anything. Thanks.
Another thought: the Hyper-V disks are VHDX dynamic disks. I wonder if converting them to fixed disks will improve performance? I know this is recommended by MS.
I'm ready to move our TeamCity installation to a dedicated machine because running it on Hyper-V seems to be a lost cause. Anybody else out there running on VMs and running into performance issues?
Hi Ben! Did you get figured it out? I have the same issue. I run teamcity on google cloud platform on Ubuntu 14.04. It's a bit different for me though. It stuck when a file size exceeds 1GB. If a file lets say 1.01GB it will download it fast until 1GB but will be stuck on the last 10MB