Very slow artifact downloads

I'm in charge of a TeamCity server that's experiencing a significant slowdown when downloading artifacts (clicking on the artifact link in the browser, and downloading the zip/tar file)

Server specs:
Win2K12R2 Hyper-V VM on our Host at the data center, which we access over a 1G metro line.
4 GB RAM
2 Procs
1G NIC (The Hyper-V host itself has dual paired 1G NICs)

Storage for Hyper-V is provided by an EMC VNX over a 1G iSCSI connection.
Data files live on the VM itself. Database lives at AWS.

When we first began using the server, users would consistently get download speeds of 40M+ and now it's dropped down to about 8-10M. Our builds are typically 9-15GB so this makes a big difference for our devs. I played with the Java heap settings and added this to the 2K12 server's env variables:
TEAMCITY_SERVER_MEM_OPTS -Xmx1024m -XX:MaxPermSize=270m

Reboot, and speeds jumped up again, but only for a while. Now they're back down again.

I realize there are many components in play here (Hyper-V host, VM, mulitple links, etc.) but as I mentioned, everything was speedy at one point.

After more troubleshooting, I've eliminated it having anything to do with Java. I run an nginx proxy in front of the server and I've added a location that allows me to browse the artifact directories. When I download directly from the artifacts, while on the TeamCity host, I get the same, 8-10M speeds. Same thing goes for downloading on the TeamCity host, through the regular TeamCity artifact dropdown.

Could it be disk I/O? Another idea: the TeamCity disk is currently 1.5 TB and has about 230GB free. Would TeamCity slow down as the disk fills up?

Thanks.

5 comments

Hello Ben,

Please check the related section in the documentation: https://confluence.jetbrains.com/display/TCD9/Known+Issues#KnownIssues-SlowdownloadfromTeamCityserver. 

Could you try to run a simple http server (non-TeamCity related, e.g. a clean Tomcat server ) side by side on the same machine as TeamCity server and measure the download speed you get from this test server?

0

Sorry, forgot to mention that I'd already read that doc and confirmed the server settings are correct. 

I was a bit unclear in my sentence above about downloading directly from the artifact links. I added a "location" to my nginx webserver/proxy that pointed directly to the artifacts directory, thus avoiding TeamCity altogether and I get the same slow downloads. My inclination is that it's a disk I/O issue, but then that doesn't explain why the speeds were high for the first month, unless it's related to how TC caches, etc.

I have a ticket open with support and they've responded with some ideas. I'll try them and post back here if I come up with anything. Thanks.

0

Another thought: the Hyper-V disks are VHDX dynamic disks. I wonder if converting them to fixed disks will improve performance? I know this is recommended by MS.

0

I'm ready to move our TeamCity installation to a dedicated machine because running it on Hyper-V seems to be a lost cause. Anybody else out there running on VMs and running into performance issues? 

0

Hi Ben! Did you get figured it out? I have the same issue. I run teamcity on google cloud platform on Ubuntu 14.04. It's a bit different for me though. It stuck when a file size exceeds 1GB. If a file lets say 1.01GB it will download it fast until 1GB but will be stuck on the last 10MB

0

Please sign in to leave a comment.