TeamCity slow artifact upload performance

Over the many years we've used TeamCity we've frequently noticed that artifact upload performance is much slower than expected, often by at least an order of magnitude. For example, uploading a ~200MB artifact typically takes ~45 seconds across a low-latency Gigabit network. I took some time to look into this today and found some interesting results it'd be great to get some input on.

Testing configuration for reference:

  • TeamCity Server 2020.1 on Ubuntu Server 16.04 x64
  • TeamCity Agent on Windows Server 2012 R2 x64
  • Apache 2.4.43 as a reverse proxy w/ SSL termination
  • Using bundled JRE on both systems

Monitoring the system calls on the Windows system via ProcMon shows the Java process corresponding to the agent is reading the artifacts in 4KB chunks and seemingly transmitting the data in 4KB chunks as well. Importantly, there appears to be effectively no buffering, as each 4KB read almost immediately results in a 4KB TCP transmission (slightly larger in practice w/ headers). We suspect that the transmission of fixed 4KB chunks combined with the lack of any buffering of the artifact file may be the culprit of the poor upload performance. Can anyone from JetBrains engineering comment on this?

Thanks in advance!

3 comments
Comment actions Permalink

Just wanted to follow-up if anyone from JetBrains can comment on this?

0
Comment actions Permalink

Hi, and sorry for the delay,

 

there is actually a buffering mechanism in place. By default, it caches up to 64kb, but we don't really use anything particularly special about our transfer, it's common java calls, and it shouldn't really need much tuning. We still do offer the ability to modify the size of the cache for both send and receive via setting the following internal properties on the agent:

teamcity.httpClient.connection.http.socket.sendbuffer=xxx
teamcity.httpClient.connection.http.socket.receivebuffer=xxx

Where "xxx" should be the size in bytes.

 

We have seen some specific setups in the past where there were impacts on performance during artifact transfer, such as https://youtrack.jetbrains.com/issue/TW-44031, but most of those were fixed when we introduced the 64k buffer and the parameters mentioned above. We have since seen issues with artifact transfers on edge cases, typically related to huge amounts of very small files, while other issues were mostly tied to environmental issues (network setup, infrastructure, issues with the JVM, etc).

 

We can recommend tuning the parameters above, and while we are not aware of specific issues with Server 2012 or the current JVM, we could try suggesting testing with other OS or JVM. We would also recommend testing jumping over the proxy and see if that helps.

0
Comment actions Permalink

Thanks Denis, that was really helpful. We ended up upgrading the JVM on the host from Oracle JDK 8u161 to Amazon Corretto 11.0.8.10 which has resulted in a substantial performance improvement. It may be worth documenting this somewhere w.r.t. potential performance gains?

Regarding the above referenced agent properties, I had trouble getting them to work, but after some additional poking around determined they seem to only be applied if teamcity.httpClient.connection.patch is also set to true. Does that sound right or am I completely missing something here? After doing so, we were able to obtain some additional performance improvements by tweaking the send/receive buffer sizes per your referenced properties.

0

Please sign in to leave a comment.