TeamCity Symbol Server stopped working. Not finding any pdb files

Hi,

I'm trying to get the TeamCity Symbol Server plugin to work and are experiencing some strange behaviour. I followed the guide for installing WinDbg on a build agent and enabling the build feature. And I actually managed to get everything working in the first attempt. Visual Studio downloaded the pdb file from TeamCity and I was able to step into the source files.
At this point I added symbol files indexer to several other builds and run one more build of the package that I got to work previously.

Now TeamCity reports "Not Found" on all pdb files. I've tested both with Visual Studio and see that it attempts to download the pdb file from TeamCity(this woked at one time for one of the packages). I've also tested with symchk and it also report that it can't find any pdb files.

From the build log of the NuGet packages that I've added symbol files indexing everything looks correct. It updated the source files in the pdb file, run the JetBrains.CommandLine.Symbols.exe command and creates the symbol-signatures-XXXXX.xml file. It also reports that it's publishing the xml file to .teamcity/symbols. So everything in the build looks fine.

I've downloaded the pdb file from the build artifacts and looked at the content. It's correctly updated with the correct TeamCity url for source files. If I add the pdb file to the local symbol directory it's picked up by Visual Studio and I can step into the source files. The source files are downloaded from TeamCity.

So the only thing that don't work is downloading pdb files from TeamCity. It's always come back as Not Found.
SYMSRV:  http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/A5906DAE16CE482090A3B2E3395638FA1/Confirmit.Configuration.pdb not found

The extra frustrating thing is that it actually worked very well after the first build. I was able to demo it and other developers where able to configure Visual Studio to download symbols and source files from TeamCity. This functionallity is very nice when it's working so I'm hoping for some help to get it working again.

Please let me know if you need any aditional information to help me resolve this.

Regards,
Jan

41 comments
Comment actions Permalink

Jan, sorry for the delay in reply,
i'm pretty busy working on upcoming 9.1.5 bugfix. i'll do my best to get back to this issues ASAP.
Please remember that this plugin is not bundled yet. And TeamCity team do not provide an official market for TeamCity plugins.
I'll definitely provide a fix. But some period of time is required. Thank you for your patience.

0
Comment actions Permalink

Hi Evgeniy,

Hope you have time to look into the symbol server issues now that TeamCity 9.1.5 is released

Regards,
Jan

0
Comment actions Permalink

I'm currently experiencing similar scenario where sometimes or most times the PDB files are not indexed, although the xml artifact is published, the server returns nothing at all when requesting a pdb with the same guid.

0
Comment actions Permalink

Any news Evgeniy?

0
Comment actions Permalink

Any chance of a fix soon?

1
Comment actions Permalink

Hi,

I've tried to use your plugin and I've met similar issues as Jan. TeamCity PDBs have wrong ages - DLL age and PDB age don't match. When I tried to find a reason I found out:

- For same PDB file, JetBrains.CommandLine.Symbols.exe tool returns age "2" but MS tools (i. e. SymStore) return "1". (DLL has age 1 according dumpbin tool)

- Because we use same a build script for manual and TC build I can compare results from same environment. Both tools return age "1" for PDBs from manual build. I guess TC plugin modifies PDBs during publishing to update a source path (URL to TC etc.). As PDB format newbie I see that one of age records is 2 (field between timestamp and Guid). That is maybe reason why JetBrains.CommandLine.Symbols.exe tool thinks that it should be 2. But PDB format is more complex probably... 2 places where this data can be, maybe some checksum... 

I'm sure that own implementation of PDB parser is really hard task. Maybe you can use some existing one - DIA or public MS sources - https://github.com/Microsoft/microsoft-pdb/tree/e6b1dec61e154b568357537792e1d17a13525d5d

 

Regards,

Michal

0
Comment actions Permalink

It would be great if this plugin could be made open source and we try to fix it. Even just pushing the current state of code to github, we'll clean up, etc.

0
Comment actions Permalink

Hi, Shawn. Sources of the plugin are actually available here http://hg.jetbrains.org/hg/tc-symbol-server/
Pull requests or any other option to collaborate are not available at the moment, But I plan to put sources to github soon.

0
Comment actions Permalink

I've hoped that new version 10 could fix these issues... Unfortunately not. Age value of all records in .teamcity/symbols/symbol-signatures-local-xxxxxxx.xml is "2" although real "age" of pdb files in Artifacts zip is "1".

I give it up. :(

0
Comment actions Permalink

Hi Evgeniy,

Do you have any plan to fix the issue, have got the same issue, and found this thread?

 

Daniel

0
Comment actions Permalink

Any news on this issue? We are impacted as well

0

Please sign in to leave a comment.