TeamCity Symbol Server stopped working. Not finding any pdb files
Hi,
I'm trying to get the TeamCity Symbol Server plugin to work and are experiencing some strange behaviour. I followed the guide for installing WinDbg on a build agent and enabling the build feature. And I actually managed to get everything working in the first attempt. Visual Studio downloaded the pdb file from TeamCity and I was able to step into the source files.
At this point I added symbol files indexer to several other builds and run one more build of the package that I got to work previously.
Now TeamCity reports "Not Found" on all pdb files. I've tested both with Visual Studio and see that it attempts to download the pdb file from TeamCity(this woked at one time for one of the packages). I've also tested with symchk and it also report that it can't find any pdb files.
From the build log of the NuGet packages that I've added symbol files indexing everything looks correct. It updated the source files in the pdb file, run the JetBrains.CommandLine.Symbols.exe command and creates the symbol-signatures-XXXXX.xml file. It also reports that it's publishing the xml file to .teamcity/symbols. So everything in the build looks fine.
I've downloaded the pdb file from the build artifacts and looked at the content. It's correctly updated with the correct TeamCity url for source files. If I add the pdb file to the local symbol directory it's picked up by Visual Studio and I can step into the source files. The source files are downloaded from TeamCity.
So the only thing that don't work is downloading pdb files from TeamCity. It's always come back as Not Found.
SYMSRV: http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/A5906DAE16CE482090A3B2E3395638FA1/Confirmit.Configuration.pdb not found
The extra frustrating thing is that it actually worked very well after the first build. I was able to demo it and other developers where able to configure Visual Studio to download symbols and source files from TeamCity. This functionallity is very nice when it's working so I'm hoping for some help to get it working again.
Please let me know if you need any aditional information to help me resolve this.
Regards,
Jan
Please sign in to leave a comment.
Thank you for the feedback, Michal. Please watch for the following issue https://youtrack.jetbrains.com/issue/TW-44106.
Hello, Jan
Please enable debug logging on TeamCity server, trigger downloading of pdb on VS side and share resulted logs with me.
Hi Evgeniy,
Thanks for looking into this. I've sent you the log files on e-mail.
Regards,
Jan
Jan,
I see numbers of lines like this in logs you've provided
Looks like build metadata index on TeamCity server is corrupted. Please go to Administration > Diagnostic > Caches and use reset action against buildsMetadata cache. TeamCity will start re-indexing of build metadata after that. So all builds published info about indexed symbol files fill be re analized as well. This process details will be captured in server logs. Debug logging level is required to capture all the details.
I've now cleared the buildsMetadata cache. It's now marked as empty. pdb files are still not loading, but it probably takes some time to re-index. Any way I can see that re-indexing is complete?
There no any kind of indicator in web UI. All the details are capturing in logs.
Re-indexing have now had all night to complete. Looking in the logs I can only see indexing of new builds. What should I look for in the logs to see if reindexing is still working on old builds?
I'm still getting "Not Found" when trying to download symbols from Visual Studio. So either re-indexing is not complete or something else is wrong.
Re-Indexing is processing build history from newest to oldest builds.
Following messages should appear in the logs
-Enqueued {number of} builds for indexing, builds left: {number}, last build id: {id}"- Finished re-indexing builds, total indexed {number of builds}
The 10 teamcity-server log files are overwritten every few minutes because of the amount of logging done in debug. I searched the current set of files and can't find any of the messages you decribe. That and the fact the the builds containing the pdb files are quite new I now assume that reindexing is complete.
Download of all the pdb files still fails with the same error message:
Please start new build in configuration which is producing pdb's and capture server logs.
Please also turn off debug-all logging preset and turn on debug logging for jetbrains.buildServer.symbols.* category only.
ok. I'll try that.
How do I turn on debug logging for jetbrains.buildServer.symbols category.* only? There is no active logging preset for this. Do you have a debug conf file I can upload to TeamCity with the configuration you want?
I've added log4j config to plugin sources http://hg.jetbrains.org/hg/tc-symbol-server/file/0cd726efa1aa/tools/debug-symbols.xml
file and copy/rename the debug-symbols.xml file over before the server start.Please backup the
I sent you a new e-mail with logs
Hi Evgeniy, I am having the same kind of problem. Source server works fine but the symbol server is not finding the pdb's. I belive my problem is related to the artificat publishing step. I changed the debug log level to the file you included above. Is there a way i could send the files to you privately?
Please send logs to "evgeniy dot koshkin at jetbrains dot com"
Looking at the last bunch of logs i see no builds provided symbol data.
I see number of lines like this in teamcity-symbolserver.log file
Build logs send by you have an old id, its indexing details is not captured in logs. Could you please start fresh build in this config and send teamcity-symbolserver.log to me with whole build processing captured.
I've sent you an email with new logs. Looking forward to your response
Hi Evgeniy,
I hope the latest logs I sent you contain some useful information. Any news?
/Jan
Jan, looks like it is a kind of race conditions in build metadata index on TeamCity server. I'll add some logging and will ask you to try new version of the plugin during this week.
ok. Great that you found something. I'm looking forward to a fix so just let me know if I can do anything to help out.
/Jan
Hi Evgeniy,
Any progress on the new version?
/Jan
Any news? I really need to get this issue resolved
Sory for delay. It was a busy time since we've released 9.1.4 update. I'll do my best to send new version of the plugin with improved logging today,
Jan. i've published new version of the plugin with more logged details. Please try it with debug enabled and share new logs with me.
Jan, what i see from logs is that pdb file signatures requested by VS were never processed by TeamCity.
Looks like there are several copies of pdb with the same name in every particular build. One copy is indexed by TeamCity another one appeared on developer machine.
Such a behaviour can be observed when for example pdb was published as TeamCity artifact from build agent and also is published as a part of NuGet package and downloaded later by VS.
Is it your case?
A few weeks back the pdb files was part of the nuget package. As part of the job of introducing the symbol server the pdb files has been removed from the nuget package. So the build that produce the nuget package don't put the pdb file in the nuget package.
I'm not sure I understand exactly what is happening. Visual Studio looks for a pdb file with a particular signature. The TeamCity server responds with Not Found. But the pdb file that is published as a build artifact and updated by the symbol server plugin is the correct one. If I manually download it from TeamCity and place it in a local symbol server path then Visual Studio finds it and uses it. Why is not this pdb file processed by TeamCity so that it can be served by the symbol server?
As an example:
Visual Studio looks for this pdb file.
SYMSRV: http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/DB44780813134986AA4D5C93F7A60A5C1/Confirmit.Configuration.pdb not found
The build that produced Confirmit.Configuration nuget package(without the pdb file inside the nupkg) publish this symbol signatures xml file.
</file-signs>
So you see that the signature matches (except for the last digit). But TeamCity don't seem to process this pdb file/signature
Looks like binary file loaded by VS during debug session references version of pdb file which wasn't indexed by TeamCity.
Could you please install the newest plugin version where even more details are logged. I hope it would be last iteration)
Could you please also provide symbol-signatures file published from build agent via email?
Hi Evgeniy,
I am also having an issue with Symbol Server, it seems to index them ok but fails when trying to retrieve them from VS.
which version of Team City is this latest build of Symbol Server targetting? I am on 8.0.6.
Thansk,
Simon
Hi Evgeniy,
I would really like an update on the progress of fixing the issues with this plugin. Currently it's not working. After several days with investigations and e-mails with you I see these issues with the plugin. And I believe that the behaviour has been documented in the e-mails and log's I've sent you.
1. Pdb files are most of the time not indexed at all by TeamCity. Even several days after the build has completed the pdb file are still not found when requested from Visual Studio
2. Build configurations that don't have the symbol-indexer feature configured will sometimes, when they execute on a build agent with WinDbg installed, incorrectly index all pdb files in the build artifacts
3. Indexing of pdb files seem to be very slow in the few cases where it works. I've seen the examples for the builds that incorrectly index hundreds of pdb files that TeamCity still log indexing work several hours after the build completed
After spending so much time trying to get this plugin to work in a stable and predictable way I really feel that you need to come up with a fix. If you don't want to fix it then please remove the plugin from the market so that potential users of it don't waist a lot of time trying to get it to work.
Regards,
Jan