TeamCity Symbol Server stopped working. Not finding any pdb files

Hi,

I'm trying to get the TeamCity Symbol Server plugin to work and are experiencing some strange behaviour. I followed the guide for installing WinDbg on a build agent and enabling the build feature. And I actually managed to get everything working in the first attempt. Visual Studio downloaded the pdb file from TeamCity and I was able to step into the source files.
At this point I added symbol files indexer to several other builds and run one more build of the package that I got to work previously.

Now TeamCity reports "Not Found" on all pdb files. I've tested both with Visual Studio and see that it attempts to download the pdb file from TeamCity(this woked at one time for one of the packages). I've also tested with symchk and it also report that it can't find any pdb files.

From the build log of the NuGet packages that I've added symbol files indexing everything looks correct. It updated the source files in the pdb file, run the JetBrains.CommandLine.Symbols.exe command and creates the symbol-signatures-XXXXX.xml file. It also reports that it's publishing the xml file to .teamcity/symbols. So everything in the build looks fine.

I've downloaded the pdb file from the build artifacts and looked at the content. It's correctly updated with the correct TeamCity url for source files. If I add the pdb file to the local symbol directory it's picked up by Visual Studio and I can step into the source files. The source files are downloaded from TeamCity.

So the only thing that don't work is downloading pdb files from TeamCity. It's always come back as Not Found.
SYMSRV:  http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/A5906DAE16CE482090A3B2E3395638FA1/Confirmit.Configuration.pdb not found

The extra frustrating thing is that it actually worked very well after the first build. I was able to demo it and other developers where able to configure Visual Studio to download symbols and source files from TeamCity. This functionallity is very nice when it's working so I'm hoping for some help to get it working again.

Please let me know if you need any aditional information to help me resolve this.

Regards,
Jan

1
41 comments
Avatar
Permanently deleted user
Official comment

Thank you for the feedback, Michal. Please watch for the following issue https://youtrack.jetbrains.com/issue/TW-44106.

Avatar
Permanently deleted user

Any chance of a fix soon?

1
Avatar
Permanently deleted user

Hello, Jan
Please enable debug logging on TeamCity server, trigger downloading of pdb on VS side and share resulted logs with me.

0
Avatar
Permanently deleted user

Hi Evgeniy,

Thanks for looking into this. I've sent you the log files on e-mail.

Regards,
Jan

0
Avatar
Permanently deleted user

Jan,
I see numbers of lines like this in logs you've provided

DEBUG - bols.DownloadSymbolsController - Symbol file requested. File name: {name}. Guid: {guid}.
DEBUG - bols.DownloadSymbolsController - There is no information about symbol file with id {guid} in the index.

Looks like build metadata index on TeamCity server is corrupted. Please go to Administration > Diagnostic > Caches and use reset action against buildsMetadata cache. TeamCity will start re-indexing of build metadata after that. So all builds published info about indexed symbol files fill be re analized as well. This process details will be captured in server logs. Debug logging level is required to capture all the details.

0
Avatar
Permanently deleted user

I've now cleared the buildsMetadata cache. It's now marked as empty. pdb files are still not loading, but it probably takes some time to re-index. Any way I can see that re-indexing is complete?

0
Avatar
Permanently deleted user

There no any kind of indicator in web UI. All the details are capturing in logs.

0
Avatar
Permanently deleted user

Re-indexing have now had all night to complete. Looking in the logs I can only see indexing of new builds. What should I look for in the logs to see if reindexing is still working on old builds?
I'm still getting "Not Found" when trying to download symbols from Visual Studio. So either re-indexing is not complete or something else is wrong.

0
Avatar
Permanently deleted user

Re-Indexing is processing build history from newest to oldest builds.
Following messages should appear in the logs

 
-Enqueued {number of} builds for indexing, builds left: {number}, last build id: {id}"

- Finished re-indexing builds, total indexed {number of builds}

0
Avatar
Permanently deleted user

The 10 teamcity-server log files are overwritten every few minutes because of the amount of logging done in debug. I searched the current set of files and can't find any of the messages you decribe. That and the fact the the builds containing the pdb files are quite new I now assume that reindexing is complete.

Download of all the pdb files still fails with the same error message:

[2015-11-06 12:25:14,321]  DEBUG - bols.DownloadSymbolsController - Symbol file requested. File name: Confirmit.Configuration.pdb. Guid: a5906dae16ce482090a3b2e3395638fa. 
[2015-11-06 12:25:14,336]  DEBUG - bols.DownloadSymbolsController - There is no information about symbol file with id a5906dae16ce482090a3b2e3395638fa in the index. 
0
Avatar
Permanently deleted user

Please start new build in configuration which is producing pdb's and capture server logs.
Please also turn off debug-all logging preset and turn on debug logging for jetbrains.buildServer.symbols.* category only.

0
Avatar
Permanently deleted user

ok. I'll try that.

How do I turn on debug logging for jetbrains.buildServer.symbols category.* only? There is no active logging preset for this. Do you have a debug conf file I can upload to TeamCity with the configuration you want?

0
Avatar
Permanently deleted user

I've added log4j config to plugin sources http://hg.jetbrains.org/hg/tc-symbol-server/file/0cd726efa1aa/tools/debug-symbols.xml
Please backup the

conf/teamcity-server-log4j.xml
file and copy/rename the debug-symbols.xml file over
conf/teamcity-server-log4j.xml
before the server start.
0
Avatar
Permanently deleted user

I sent you a new e-mail with logs

0
Avatar
Permanently deleted user

Hi Evgeniy, I am having the same kind of problem. Source server works fine but the symbol server is not finding the pdb's. I belive my problem is related to the artificat publishing step. I changed the debug log level to the file you included above. Is there a way i could send the files to you privately?

0
Avatar
Permanently deleted user

Please send logs to "evgeniy dot koshkin at jetbrains dot com"

0
Avatar
Permanently deleted user

Looking at the last bunch of logs i see no builds provided symbol data.
I see number of lines like this in teamcity-symbolserver.log file

DEBUG [und Build Indexer (metadata) 1] - Build with id {build id} doesn't provide symbols index data.

Build logs send by you have an old id, its indexing details is not captured in logs. Could you please start fresh build in this config and send teamcity-symbolserver.log to me with whole build processing captured.

0
Avatar
Permanently deleted user

I've sent you an email with new logs. Looking forward to your response

0
Avatar
Permanently deleted user

Hi Evgeniy,

I hope the latest logs I sent you contain some useful information. Any news?

/Jan

0
Avatar
Permanently deleted user

Jan, looks like it is a kind of race conditions in build metadata index on TeamCity server. I'll add some logging and will ask you to try new version of the plugin during this week.

0
Avatar
Permanently deleted user

ok. Great that you found something. I'm looking forward to a fix so just let me know if I can do anything to help out.

/Jan

0
Avatar
Permanently deleted user

Hi Evgeniy,

Any progress on the new version?

/Jan

0
Avatar
Permanently deleted user

Any news? I really need to get this issue resolved

0
Avatar
Permanently deleted user

Sory for delay. It was a busy time since we've released 9.1.4 update. I'll do my best to send new version of the plugin with improved logging today,

0
Avatar
Permanently deleted user

Jan. i've published new version of the plugin with more logged details. Please try it with debug enabled and share new logs with me.

0
Avatar
Permanently deleted user

Jan, what i see from logs is that pdb file signatures requested by VS were never processed by TeamCity.
Looks like there are several copies of pdb with the same name in every particular build. One copy is indexed by TeamCity another one appeared on developer machine.
Such a behaviour can be observed when for example pdb was published as TeamCity artifact from build agent and also is published as a part of NuGet package and downloaded later by VS.
Is it your case?

0
Avatar
Permanently deleted user

A few weeks back the pdb files was part of the nuget package. As part of the job of introducing the symbol server the pdb files has been removed from the nuget package. So the build that produce the nuget package don't put the pdb file in the nuget package.
I'm not sure I understand exactly what is happening. Visual Studio looks for a pdb file with a particular signature. The TeamCity server responds with Not Found. But the pdb file that is published as a build artifact and updated by the symbol server plugin is the correct one. If I manually download it from TeamCity and place it in a local symbol server path then Visual Studio finds it and uses it. Why is not this pdb file processed by TeamCity so that it can be served by the symbol server?

0
Avatar
Permanently deleted user

As an example:
Visual Studio looks for this pdb file.
SYMSRV:  http://co-osl-teamcity/app/symbols/Confirmit.Configuration.pdb/DB44780813134986AA4D5C93F7A60A5C1/Confirmit.Configuration.pdb not found

The build that produced Confirmit.Configuration nuget package(without the pdb file inside the nupkg) publish this symbol signatures xml file.

<file-signs>
<file-sign-entry file="Confirmit.Configuration.pdb" sign="DB44780813134986AA4D5C93F7A60A5C2"/>

</file-signs>

So you see that the signature matches (except for the last digit). But TeamCity don't seem to process this pdb file/signature
0
Avatar
Permanently deleted user

Looks like binary file loaded by VS during debug session references version of pdb file which wasn't indexed by TeamCity.
Could you please install the newest plugin version where even more details are logged. I hope it would be last iteration)
Could you please also provide symbol-signatures file published from build agent via email?

0
Avatar
Permanently deleted user

Hi Evgeniy,

I am also having an issue with Symbol Server, it seems to index them ok but fails when trying to retrieve them from VS.

which version of Team City is this latest build of Symbol Server targetting? I am on 8.0.6.

Thansk,

Simon

0

Please sign in to leave a comment.