Over the many years we've used TeamCity we've frequently noticed that artifact upload performance is much slower than expected, often by at least an order of magnitude. For example, uploading a ~200MB artifact typically takes ~45 seconds across a low-latency Gigabit network. I took some time to look into this today and found some interesting results it'd be great to get some input on.
Testing configuration for reference:
- TeamCity Server 2020.1 on Ubuntu Server 16.04 x64
- TeamCity Agent on Windows Server 2012 R2 x64
- Apache 2.4.43 as a reverse proxy w/ SSL termination
- Using bundled JRE on both systems
Monitoring the system calls on the Windows system via ProcMon shows the Java process corresponding to the agent is reading the artifacts in 4KB chunks and seemingly transmitting the data in 4KB chunks as well. Importantly, there appears to be effectively no buffering, as each 4KB read almost immediately results in a 4KB TCP transmission (slightly larger in practice w/ headers). We suspect that the transmission of fixed 4KB chunks combined with the lack of any buffering of the artifact file may be the culprit of the poor upload performance. Can anyone from JetBrains engineering comment on this?
Thanks in advance!