Symantec Endpoint Protection conflicting with TeamCity?

We have two machines that are running TeamCity server that are crashing seemingly due to a problem with Symantec Heuristics driver. Has anyone else experienced this? We are waiting for Symantec support to get back to us, but in the mean time these machines occasionally crash and reboot. Obviously not good for production TeamCity installation. TeamCity installation directory is excluded from the SEP scan. Thanks.

Symantec Endpoint Protection version:  12.1.671.4971

From the Windows dump file:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86

Copyright (c) Microsoft Corporation. All rights reserved.

 

 

Loading Dump File [C:\1\082712-65531-01.dmp]

Mini Kernel Dump File: Only registers and stack trace are available

 

Symbol search path is: C:\WINDOWS\Symbols

Executable search path is:

Unable to load image \SystemRoot\system32\ntoskrnl.exe, Win32 error 0n2

Windows 7 Kernel Version 7600 MP (4 procs) Free x64

Product: Server, suite: TerminalServer SingleUserTS

Built by: 7600.17017.amd64fre.win7_gdr.120503-2030

Machine Name:

Kernel base = 0xfffff800`01667000 PsLoadedModuleList = 0xfffff800`018a3e70

Debug session time: Mon Aug 27 10:32:15.313 2012 (GMT-4)

System Uptime: 1 days 17:04:33.007

Unable to load image \SystemRoot\system32\ntoskrnl.exe, Win32 error 0n2

Loading Kernel Symbols

...............................................................

................................................................

................

Loading User Symbols

Loading unloaded module list

...........

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

 

Use !analyze -v to get detailed debugging information.

 

BugCheck 3B, {c0000005, fffff88007c810c0, fffff8800744eca0, 0}

 

Unable to load image \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys, Win32 error 0n2

*** WARNING: Unable to verify timestamp for BHDrvx64.sys

*** ERROR: Module load completed but symbols could not be loaded for BHDrvx64.sys

Probably caused by : BHDrvx64.sys ( BHDrvx64+370c0 )

 

Followup: MachineOwner

---------

 

2: kd> !analyze -v

*******************************************************************************

*                                                                             *

*                        Bugcheck Analysis                                    *

*                                                                             *

*******************************************************************************

 

SYSTEM_SERVICE_EXCEPTION (3b)

An exception happened while executing a system service routine.

Arguments:

Arg1: 00000000c0000005, Exception code that caused the bugcheck

Arg2: fffff88007c810c0, Address of the exception record for the exception that caused the bugcheck

Arg3: fffff8800744eca0, Address of the context record for the exception that caused the bugcheck

Arg4: 0000000000000000, zero.

 

Debugging Details:

------------------

 

 

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

 

FAULTING_IP:

BHDrvx64+370c0

fffff880`07c810c0 4c8918          mov     qword ptr [rax],r11

 

CONTEXT:  fffff8800744eca0 -- (.cxr 0xfffff8800744eca0)

rax=0000000000000000 rbx=fffff8a01262ea20 rcx=fffff8a011f04590

rdx=fffff8a00c32f990 rsi=fffff8a0025b4070 rdi=fffff8a0025b4070

rip=fffff88007c810c0 rsp=fffff8800744f670 rbp=fffff8a0057fe970

r8=0000000000000003  r9=fffff88007d67670 r10=fffff88001e5d900

r11=0000000000000000 r12=0000000000000001 r13=0000000000000003

r14=fffff88002bd8740 r15=fffff8a00f50b630

iopl=0         nv up ei pl zr na po nc

cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246

BHDrvx64+0x370c0:

fffff880`07c810c0 4c8918          mov     qword ptr [rax],r11 ds:002b:00000000`00000000=????????????????

Resetting default scope

 

CUSTOMER_CRASH_COUNT:  1

 

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

 

BUGCHECK_STR:  0x3B

 

PROCESS_NAME:  `„Í

 

CURRENT_IRQL:  0

 

LAST_CONTROL_TRANSFER:  from 0000000000000000 to fffff88007c810c0

 

STACK_TEXT:  

fffff880`0744f670 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : BHDrvx64+0x370c0

 

 

FOLLOWUP_IP:

BHDrvx64+370c0

fffff880`07c810c0 4c8918          mov     qword ptr [rax],r11

 

SYMBOL_STACK_INDEX:  0

 

SYMBOL_NAME:  BHDrvx64+370c0

 

FOLLOWUP_NAME:  MachineOwner

 

MODULE_NAME: BHDrvx64

 

IMAGE_NAME:  BHDrvx64.sys

 

DEBUG_FLR_IMAGE_TIMESTAMP:  50236349

 

STACK_COMMAND:  .cxr 0xfffff8800744eca0 ; kb

 

FAILURE_BUCKET_ID:  X64_0x3B_BHDrvx64+370c0

 

BUCKET_ID:  X64_0x3B_BHDrvx64+370c0

 

Followup: MachineOwner

---------

 

2: kd> lmvm BHDrvx64

start             end                 module name

fffff880`07c4a000 fffff880`07da1000   BHDrvx64 T (no symbols)           

    Loaded symbol image file: BHDrvx64.sys

    Image path: \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys

    Image name: BHDrvx64.sys

    Timestamp:        Thu Aug 09 03:14:17 2012 (50236349)

    CheckSum:         0015DF3B

    ImageSize:        00157000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

2: kd> lmvm BHDrvx64

start             end                 module name

fffff880`07c4a000 fffff880`07da1000   BHDrvx64 T (no symbols)           

    Loaded symbol image file: BHDrvx64.sys

    Image path: \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys

    Image name: BHDrvx64.sys

    Timestamp:        Thu Aug 09 03:14:17 2012 (50236349)

    CheckSum:         0015DF3B

    ImageSize:        00157000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

15 comments
Comment actions Permalink

Oleg,

We did not receive any reports on confilict with SEP before.

Windows dump file does not show anything related to Teamcity. Nevertheless, I advise to also exclude TeamCity data directory from scan

0
Comment actions Permalink

Nikita,
Thanks for your reply. Our data directory lives under TeamCity installation directory and should be covered by the exclusion. We are still waiting to hear from Symantec. In the mean time we removed SEP from TeamCity server in production and have not had it crash for 2nd day now. I'll update this post once we know more.

0
Comment actions Permalink

I wanted to add some details in case someone runs into a similar problem. The problem appears to have been caused by an obsolete Symantec Heuristics driver. We switched the SEP client from version 12.1.671.4971 to 12.1.1101.401 (12.2 RU1 MP1). The driver is now in this folder:
C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.1101.401.105\Data\Definitions\BASHDefs\20120823.013\BHDrvx64.sys.

Looking at file details in Windows, I see file version 6.6.3.3 with modification date 08/28/2012 3:56 PM.

We did not get any information from Symantec as to why the old driver crashed, so we don't know if it's at all related to TeamCity. We've been running with the new driver for a few days without a single crash.

0
Comment actions Permalink

Oleg, thank you for the follow up!

0
Comment actions Permalink

Nikita,

Our problem came back. :-(

Symantec support pointed us to this error message from the heuristics driver (also called SONAR):

Severity: Critical, Source: SONAR, Description: [SONAR heuristic Submission] Submitting file to Symantec failed.  File : 'c:\teamcity\buildagent\temp\globaltmp\teamcity(RandomNumber)jetregistry\jetregistry.exe'.,

This is from our dev TeamCity machine that has both TC server and agent installed. Our production TC machine does not have an agent, but jetRegistry folders/files are also there in C:\teamcity\temp\teamcity(RandomNumber)jetregistry\jetregistry.exe. I also see similar files in  C:\teamcity\temp\teamcity(RandomNumber)ps\JetBrains.TeamCity.ps.exe. What are these files and how are they generated?

Symantec support suggested that jetregistry.exe might not be signed and signing them might remedy the problem. Can you check if executables are signed?

It turns out there is a separate setting that controls folder exclusions for SONAR, which we did not know about. We excluded C:\teamcity from the scan in dev to see how it performs.

All of this might be related to http://youtrack.jetbrains.com/issue/TW-19320. Perhaps the files are leaking when they should not.

Please let me know if you can shed more light on the issue. Thanks for your help!

Regards,

Oleg.

0
Comment actions Permalink

Oleg,

These excutables are part of Teamcity. JetRegistry is used to access Windows registry and JetBrains.Teamcity.ps.exe is used to manage processess (e.g., to find processess that has locked some files).

Indeed, the executables were not signed. This is now fixed. Thank you for reporting.

0
Comment actions Permalink

Sounds great! Would you like me to file an issue in YouTrack or would you open one yourself? I would like to track the version of TeamCity that will include this fix and be notified by YouTrack.

0
Comment actions Permalink

Oleg, I've create an issue here: http://youtrack.jetbrains.com/issue/TW-23861 please watch it to get notifications.

0
Comment actions Permalink

Thanks! Can you tell me if agentInstaller.exe is signed? This is the file I can pull down from "Install Build Agents" popup. SEP flagged it and quarantined, but did not crash my computer.

0
Comment actions Permalink

I wonder, will this fix resolve the issue I am seeing where MSTest runner is failing because Symantec is blocking NUnitLauncher.exe?  (Our IT dept has confirmed by uninstalling SEP from the server to validate, they put it back on due to InfoSec rules so we are still hosed)  - We have had no luck with adding SONAR exceptions or folder exceptions to C:\TeamCity directory...

Here is the post I had going about our issue: http://devnet.jetbrains.com/message/5509327#5509327

If this will fix my issue, any idea when a new build will be available with the fix?

0
Comment actions Permalink

Hi Darron,
We haven't seen the problem with NUnitLauncher and SEP. We are on version 8.0.6 right now. Are you on a recent version of TeamCity? Perhaps your IT department can work with Symantec to identify why SEP has an issue with the executable. Perhaps it's unsigned or there is some other issue.

If someone on TC dev team could confirm that NUnitLauncher is signed, that removes one possibility.

0
Comment actions Permalink

Our IT dept is working with Symantec with little luck.  

I noticed a newer version of TeamCity just released (v8.1) and upgraded to the newer version hoping this would address the issues, it did not.  If we can't get this worked out we'll have to switch to Jenkins for CI.  I demo'd both to our team the week prior and we all agreed to go with Team City. Unfortunately, if we can't get this resolved we would be forced to switch to Jenkins. Unit Testing with CI is critical and we can get this working in Jenkins with Symantec SEP.

:(

0
Comment actions Permalink

Oleg,

Do you recall what the resolution was for this issue? We're having similar trouble (even when Symantec protection is disabled, while the Symantec Windows service is running, while the build is running, the machine eventually goes down).

Take care,

Kyle

0
Comment actions Permalink

Kyle,

I don't remember exactly what we did back then, but as of now we are running TC 10.0.3 with SEP policy having an exclusion for the drive where TeamCity is installed. My understanding is that new files appear during build execution. If SEP locks them up, builds can fail. So we just opted to exclude the entire TeamCity drive. It's not an ideal solution, but was acceptable to us.

0
Comment actions Permalink

Oleg, thank you very much for the response! We have a few TeamCity folders excluded from SEP, and will consider blocking the drive too. Much obliged for the tip.

Cheers!

0

Please sign in to leave a comment.