Symbol server unusably slow w/ high CPU usage

We have a medium sized MSVC/C++ application building in TeamCity, and we store the installers and debug symbols for easy debugging of nightly builds.

Lately the symbol server has become completely unusable. A single user fetching the symbols for the application will put one of the cores on the server to 100% utilization for over 10 minutes. Multiple users being unlucky enough to start debugging around the same time will be waiting for a better part of an hour to get going.

Granted it's not exactly a tiny application and we do store quite a bit of builds (around 300GB as of this morning), but I don't think fetching 100MB of PDBs should take this long nor cause such a spike in CPU usage.

What would be a reasonable course of action to start investigating this? Surely this cannot be working as intended?

This is on TC 10.0.3, and the plugin version seems to be 10.0.4