TeamCity Symbol Server stopped working. Not finding any pdb files
Hi,
I'm trying to get the TeamCity Symbol Server plugin to work and are experiencing some strange behaviour. I followed the guide for installing WinDbg on a build agent and enabling the build feature. And I actually managed to get everything working in the first attempt. Visual Studio downloaded the pdb file from TeamCity and I was able to step into the source files.
At this point I added symbol files indexer to several other builds and run one more build of the package that I got to work previously.
Now TeamCity reports "Not Found" on all pdb files. I've tested both with Visual Studio and see that it attempts to download the pdb file from TeamCity(this woked at one time for one of the packages). I've also tested with symchk and it also report that it can't find any pdb files.
From the build log of the NuGet packages that I've added symbol files indexing everything looks correct. It updated the source files in the pdb file, run the JetBrains.CommandLine.Symbols.exe command and creates the symbol-signatures-XXXXX.xml file. It also reports that it's publishing the xml file to .teamcity/symbols. So everything in the build looks fine.
I've downloaded the pdb file from the build artifacts and looked at the content. It's correctly updated with the correct TeamCity url for source files. If I add the pdb file to the local symbol directory it's picked up by Visual Studio and I can step into the source files. The source files are downloaded from TeamCity.
So the only thing that don't work is downloading pdb files from TeamCity. It's always come back as Not Found.
SYMSRV: http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/A5906DAE16CE482090A3B2E3395638FA1/Confirmit.Configuration.pdb not found
The extra frustrating thing is that it actually worked very well after the first build. I was able to demo it and other developers where able to configure Visual Studio to download symbols and source files from TeamCity. This functionallity is very nice when it's working so I'm hoping for some help to get it working again.
Please let me know if you need any aditional information to help me resolve this.
Regards,
Jan
Please sign in to leave a comment.
Jan, sorry for the delay in reply,
i'm pretty busy working on upcoming 9.1.5 bugfix. i'll do my best to get back to this issues ASAP.
Please remember that this plugin is not bundled yet. And TeamCity team do not provide an official market for TeamCity plugins.
I'll definitely provide a fix. But some period of time is required. Thank you for your patience.
Hi Evgeniy,
Hope you have time to look into the symbol server issues now that TeamCity 9.1.5 is released
Regards,
Jan
I'm currently experiencing similar scenario where sometimes or most times the PDB files are not indexed, although the xml artifact is published, the server returns nothing at all when requesting a pdb with the same guid.
Any news Evgeniy?
Any chance of a fix soon?
Hi,
I've tried to use your plugin and I've met similar issues as Jan. TeamCity PDBs have wrong ages - DLL age and PDB age don't match. When I tried to find a reason I found out:
- For same PDB file, JetBrains.CommandLine.Symbols.exe tool returns age "2" but MS tools (i. e. SymStore) return "1". (DLL has age 1 according dumpbin tool)
- Because we use same a build script for manual and TC build I can compare results from same environment. Both tools return age "1" for PDBs from manual build. I guess TC plugin modifies PDBs during publishing to update a source path (URL to TC etc.). As PDB format newbie I see that one of age records is 2 (field between timestamp and Guid). That is maybe reason why JetBrains.CommandLine.Symbols.exe tool thinks that it should be 2. But PDB format is more complex probably... 2 places where this data can be, maybe some checksum...
I'm sure that own implementation of PDB parser is really hard task. Maybe you can use some existing one - DIA or public MS sources - https://github.com/Microsoft/microsoft-pdb/tree/e6b1dec61e154b568357537792e1d17a13525d5d
Regards,
Michal
It would be great if this plugin could be made open source and we try to fix it. Even just pushing the current state of code to github, we'll clean up, etc.
Hi, Shawn. Sources of the plugin are actually available here http://hg.jetbrains.org/hg/tc-symbol-server/
Pull requests or any other option to collaborate are not available at the moment, But I plan to put sources to github soon.
I've hoped that new version 10 could fix these issues... Unfortunately not. Age value of all records in .teamcity/symbols/symbol-signatures-local-xxxxxxx.xml is "2" although real "age" of pdb files in Artifacts zip is "1".
I give it up. :(
Hi Evgeniy,
Do you have any plan to fix the issue, have got the same issue, and found this thread?
Daniel
Any news on this issue? We are impacted as well