Artifacts publishing using WebPublisher have poor transfer speeds.

Hi Victor,

> How WebPublisher, I assume this is the name of the component that performs the publishing from agent to server, works? Protocol, speeds etc.
Java.exe on the Agent reads the artifact from Agent's file system and sends it to the Build Server via an HTTP POST request on port 80 (default). As for the speeds, we don't have any speed benchmarks for that, so performance would always strongly depend on the given environment. There is also no throttling mechanism in TeamCity, that could impose a speed limitation, and no known bottleneck in the logic of WebPublisher per se. So, to tell something about where the bottleneck could be, some additional information is required.

> What else should I try or investigate to troubleshoot this problem.
I would like to take a look at the exact results of the measurements you've taken, when trying to identify the bottleneck, can so can you share them below?
If you are willing to conduct the tests again, you can use the tools I'm referring to further in this post. In that case, I would suggest that you do these steps during when the artifact publishing slowness is observed.

1. Check disk read performance on the Agent using Diskspd:

diskspd.exe -b4K -h -L -d30 X:\Agent\checkout\directory\arficat_file_name.xxx

By default, agent checkout directory is <Build agent home>/work/<VCS settings hash code>.

2. Check disk write performance on the Build server using Diskspd:

diskspd.exe -b4K -w100 -c1G -h -L -d30 X:\path\to\artifacts\storage\testfile.dat

Default artifact storage on Build server is <TeamCity Data Directory>/system/artifacts. Make sure there is no such file as testfile.dat before you run this test, otherwise it will overwrite your data.

3. Check network throughput between the Agent and the Build server using iperf:
On the Build server:

iperf.exe -s

On the Agent:

iperf -c <Build server IP address> -t 30

4. Check latency between Build server and Agent using ping.

5. Create a 3 Build server thread dumps and 3 Agent thread dumps with a little bit of a pause between creating consecutive dumps. It is essential that you perform this step during when the publishing slowness is observed.

Also, right after the artifact is published, check performance of non-buffered SMB/CIFS copying of the same artifact file using Robocopy on the Build server:

robocopy.exe \\Agent\X$\checkout\directory\ X:\path\to\artifacts\storage\ artifact_file_name.xxx /J /IS

This is different from simply copying files using File Explorer, because File Explorer may utilize buffering to transfer files at the background after showing that the process has finished in the UI, while /J switch in Robocopy will disable it. The /IS switch will make Robocopy to overwrite the existing file in the artifacts storage. Notice the space between X:\path\to\artifacts\storage\ and artifact_file_name.xxx.

> Is it possible to have a build step using SMB Upload to copy the files that I want as artifacts to the TC server and later at the build finishing step, just tell the server which one to use as artifacts, the files already being on the server FS.
> Give the user the power to choose what to use for artifacts publishing: SMB, FTP...
Technically it is possible to upload artifacts from Agent checkout directory to an SMB share using 'SMB upload' build step, and, if the files are put into the artifacts storage, TeamCity should recognize the them as the build artifacts. You could use that as a workaround, however I would like to understand why you want to do that, because without knowing what the actual root of the issue is, the new setup might suffer from exactly the same problems.

Regarding the first suggestion you made, about the diagnostics pane for Agents, that looks like a valid feature request to me. Feel free to submit it to our issue tracker: https://youtrack.jetbrains.com/issues/TW