Symbol server unusably slow w/ high CPU usage

We have a medium sized MSVC/C++ application building in TeamCity, and we store the installers and debug symbols for easy debugging of nightly builds.

Lately the symbol server has become completely unusable. A single user fetching the symbols for the application will put one of the cores on the server to 100% utilization for over 10 minutes. Multiple users being unlucky enough to start debugging around the same time will be waiting for a better part of an hour to get going.

Granted it's not exactly a tiny application and we do store quite a bit of builds (around 300GB as of this morning), but I don't think fetching 100MB of PDBs should take this long nor cause such a spike in CPU usage.

What would be a reasonable course of action to start investigating this? Surely this cannot be working as intended?

 

This is on TC 10.0.3, and the plugin version seems to be 10.0.4

Please sign in to leave a comment.