Symantec Endpoint Protection conflicting with TeamCity?
We have two machines that are running TeamCity server that are crashing seemingly due to a problem with Symantec Heuristics driver. Has anyone else experienced this? We are waiting for Symantec support to get back to us, but in the mean time these machines occasionally crash and reboot. Obviously not good for production TeamCity installation. TeamCity installation directory is excluded from the SEP scan. Thanks.
Symantec Endpoint Protection version: 12.1.671.4971
From the Windows dump file:
Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\1\082712-65531-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: C:\WINDOWS\Symbols
Executable search path is:
Unable to load image \SystemRoot\system32\ntoskrnl.exe, Win32 error 0n2
Windows 7 Kernel Version 7600 MP (4 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 7600.17017.amd64fre.win7_gdr.120503-2030
Machine Name:
Kernel base = 0xfffff800`01667000 PsLoadedModuleList = 0xfffff800`018a3e70
Debug session time: Mon Aug 27 10:32:15.313 2012 (GMT-4)
System Uptime: 1 days 17:04:33.007
Unable to load image \SystemRoot\system32\ntoskrnl.exe, Win32 error 0n2
Loading Kernel Symbols
...............................................................
................................................................
................
Loading User Symbols
Loading unloaded module list
...........
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 3B, {c0000005, fffff88007c810c0, fffff8800744eca0, 0}
Unable to load image \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for BHDrvx64.sys
*** ERROR: Module load completed but symbols could not be loaded for BHDrvx64.sys
Probably caused by : BHDrvx64.sys ( BHDrvx64+370c0 )
Followup: MachineOwner
---------
2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff88007c810c0, Address of the exception record for the exception that caused the bugcheck
Arg3: fffff8800744eca0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
Debugging Details:
------------------
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".
FAULTING_IP:
BHDrvx64+370c0
fffff880`07c810c0 4c8918 mov qword ptr [rax],r11
CONTEXT: fffff8800744eca0 -- (.cxr 0xfffff8800744eca0)
rax=0000000000000000 rbx=fffff8a01262ea20 rcx=fffff8a011f04590
rdx=fffff8a00c32f990 rsi=fffff8a0025b4070 rdi=fffff8a0025b4070
rip=fffff88007c810c0 rsp=fffff8800744f670 rbp=fffff8a0057fe970
r8=0000000000000003 r9=fffff88007d67670 r10=fffff88001e5d900
r11=0000000000000000 r12=0000000000000001 r13=0000000000000003
r14=fffff88002bd8740 r15=fffff8a00f50b630
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010246
BHDrvx64+0x370c0:
fffff880`07c810c0 4c8918 mov qword ptr [rax],r11 ds:002b:00000000`00000000=????????????????
Resetting default scope
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0x3B
PROCESS_NAME: `„Í
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from 0000000000000000 to fffff88007c810c0
STACK_TEXT:
fffff880`0744f670 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : BHDrvx64+0x370c0
FOLLOWUP_IP:
BHDrvx64+370c0
fffff880`07c810c0 4c8918 mov qword ptr [rax],r11
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: BHDrvx64+370c0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: BHDrvx64
IMAGE_NAME: BHDrvx64.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 50236349
STACK_COMMAND: .cxr 0xfffff8800744eca0 ; kb
FAILURE_BUCKET_ID: X64_0x3B_BHDrvx64+370c0
BUCKET_ID: X64_0x3B_BHDrvx64+370c0
Followup: MachineOwner
---------
2: kd> lmvm BHDrvx64
start end module name
fffff880`07c4a000 fffff880`07da1000 BHDrvx64 T (no symbols)
Loaded symbol image file: BHDrvx64.sys
Image path: \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys
Image name: BHDrvx64.sys
Timestamp: Thu Aug 09 03:14:17 2012 (50236349)
CheckSum: 0015DF3B
ImageSize: 00157000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
2: kd> lmvm BHDrvx64
start end module name
fffff880`07c4a000 fffff880`07da1000 BHDrvx64 T (no symbols)
Loaded symbol image file: BHDrvx64.sys
Image path: \??\C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.671.4971.105\Data\Definitions\BASHDefs\20120823.012\BHDrvx64.sys
Image name: BHDrvx64.sys
Timestamp: Thu Aug 09 03:14:17 2012 (50236349)
CheckSum: 0015DF3B
ImageSize: 00157000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Please sign in to leave a comment.
Oleg,
We did not receive any reports on confilict with SEP before.
Windows dump file does not show anything related to Teamcity. Nevertheless, I advise to also exclude TeamCity data directory from scan
Nikita,
Thanks for your reply. Our data directory lives under TeamCity installation directory and should be covered by the exclusion. We are still waiting to hear from Symantec. In the mean time we removed SEP from TeamCity server in production and have not had it crash for 2nd day now. I'll update this post once we know more.
I wanted to add some details in case someone runs into a similar problem. The problem appears to have been caused by an obsolete Symantec Heuristics driver. We switched the SEP client from version 12.1.671.4971 to 12.1.1101.401 (12.2 RU1 MP1). The driver is now in this folder:
C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.1101.401.105\Data\Definitions\BASHDefs\20120823.013\BHDrvx64.sys.
Looking at file details in Windows, I see file version 6.6.3.3 with modification date 08/28/2012 3:56 PM.
We did not get any information from Symantec as to why the old driver crashed, so we don't know if it's at all related to TeamCity. We've been running with the new driver for a few days without a single crash.
Oleg, thank you for the follow up!
Nikita,
Our problem came back. :-(
Symantec support pointed us to this error message from the heuristics driver (also called SONAR):
Severity: Critical, Source: SONAR, Description: [SONAR heuristic Submission] Submitting file to Symantec failed. File : 'c:\teamcity\buildagent\temp\globaltmp\teamcity(RandomNumber)jetregistry\jetregistry.exe'.,
This is from our dev TeamCity machine that has both TC server and agent installed. Our production TC machine does not have an agent, but jetRegistry folders/files are also there in C:\teamcity\temp\teamcity(RandomNumber)jetregistry\jetregistry.exe. I also see similar files in C:\teamcity\temp\teamcity(RandomNumber)ps\JetBrains.TeamCity.ps.exe. What are these files and how are they generated?
Symantec support suggested that jetregistry.exe might not be signed and signing them might remedy the problem. Can you check if executables are signed?
It turns out there is a separate setting that controls folder exclusions for SONAR, which we did not know about. We excluded C:\teamcity from the scan in dev to see how it performs.
All of this might be related to http://youtrack.jetbrains.com/issue/TW-19320. Perhaps the files are leaking when they should not.
Please let me know if you can shed more light on the issue. Thanks for your help!
Regards,
Oleg.
Oleg,
These excutables are part of Teamcity. JetRegistry is used to access Windows registry and JetBrains.Teamcity.ps.exe is used to manage processess (e.g., to find processess that has locked some files).
Indeed, the executables were not signed. This is now fixed. Thank you for reporting.
Sounds great! Would you like me to file an issue in YouTrack or would you open one yourself? I would like to track the version of TeamCity that will include this fix and be notified by YouTrack.
Oleg, I've create an issue here: http://youtrack.jetbrains.com/issue/TW-23861 please watch it to get notifications.
Thanks! Can you tell me if agentInstaller.exe is signed? This is the file I can pull down from "Install Build Agents" popup. SEP flagged it and quarantined, but did not crash my computer.
I wonder, will this fix resolve the issue I am seeing where MSTest runner is failing because Symantec is blocking NUnitLauncher.exe? (Our IT dept has confirmed by uninstalling SEP from the server to validate, they put it back on due to InfoSec rules so we are still hosed) - We have had no luck with adding SONAR exceptions or folder exceptions to C:\TeamCity directory...
Here is the post I had going about our issue: http://devnet.jetbrains.com/message/5509327#5509327
If this will fix my issue, any idea when a new build will be available with the fix?
Hi Darron,
We haven't seen the problem with NUnitLauncher and SEP. We are on version 8.0.6 right now. Are you on a recent version of TeamCity? Perhaps your IT department can work with Symantec to identify why SEP has an issue with the executable. Perhaps it's unsigned or there is some other issue.
If someone on TC dev team could confirm that NUnitLauncher is signed, that removes one possibility.
Our IT dept is working with Symantec with little luck.
I noticed a newer version of TeamCity just released (v8.1) and upgraded to the newer version hoping this would address the issues, it did not. If we can't get this worked out we'll have to switch to Jenkins for CI. I demo'd both to our team the week prior and we all agreed to go with Team City. Unfortunately, if we can't get this resolved we would be forced to switch to Jenkins. Unit Testing with CI is critical and we can get this working in Jenkins with Symantec SEP.
:(
Oleg,
Do you recall what the resolution was for this issue? We're having similar trouble (even when Symantec protection is disabled, while the Symantec Windows service is running, while the build is running, the machine eventually goes down).
Take care,
Kyle
Kyle,
I don't remember exactly what we did back then, but as of now we are running TC 10.0.3 with SEP policy having an exclusion for the drive where TeamCity is installed. My understanding is that new files appear during build execution. If SEP locks them up, builds can fail. So we just opted to exclude the entire TeamCity drive. It's not an ideal solution, but was acceptable to us.
Oleg, thank you very much for the response! We have a few TeamCity folders excluded from SEP, and will consider blocking the drive too. Much obliged for the tip.
Cheers!