Windows Server - Server 2008 x64 crashing

Asked By Zachary
19-Nov-09 11:10 AM
I have a 2008 server that has crashed 5 times this morning.  The event logs
show nothing right before the crash to point me in the right direction.  All
I have to go on is the Blue Screen.  I am currently downloading the symbols
needed to analyze a server 2008 crash file.  Once I get that downloaded I
might know more but I need some preliminary help on this.  To start I want
to make everyone aware, no hardware changes were made recently, no driver
updates or installs were done recently, and no windows updates were done
recently.  Here is the BSOD info:



SYSTEM_SERVICE_EXCEPTION



STOP: 0X0000003B (0x00000000C0000005, 0XFFFFFA6008B18726,
0XFFFFFA600BBC4B30, 0x0000000000000000)



VSApiNt.sys - Address FFFFFA6008B18726 base at FFFFFA6008A0E000, DateStamp
4ad30768



Any help would be appreciated.
Windows Server 2008
(1)
Windows Vista
(1)
VSScanVirusInMemory
(1)
PsLoadedModuleList
(1)
SingleUserTS
(1)
VSApiNt
(1)
LanManNt
(1)
SMTP
(1)
  Zachary replied to Zachary
19-Nov-09 11:37 AM
Here is my crash analysis:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Temp\AGRADC2-926-Crash.DMP]
Kernel Summary Dump File: Only kernel address space is available

WARNING: Inaccessible path: 'D:\I386'
Symbol search path is:
SRV*c:\temp\symbolstore*http://msdl.microsoft.com/download/symbols
Executable search path is: D:\I386
Windows Server 2008/Windows Vista Kernel Version 6001 (Service Pack 1) MP (8
procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Built by: 6001.18304.amd64fre.vistasp1_gdr.090805-0102
Machine Name:
Kernel base = 0xfffff800`01a06000 PsLoadedModuleList = 0xfffff800`01bcbdb0
Debug session time: Thu Nov 19 09:26:47.649 2009 (GMT-6)
System Uptime: 0 days 1:20:02.331
Loading Kernel Symbols
...............................................................
................................................................
.................
Loading User Symbols
PEB is paged out (Peb.Ldr = 000007ff`fffd5018).  Type ".hh dbgerr001" for
details
Loading unloaded module list
.............
*******************************************************************************
*
*
*                        Bugcheck Analysis
*
*
*
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c0000005, fffffa6008b18726, fffffa600bbc4b30, 0}

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for
VSApiNt.sys -
*** ERROR: Module load completed but symbols could not be loaded for
TmXPFlt.sys
Page c74fc not present in the dump file. Type ".hh dbgerr004" for details
Page c7dea not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 000007ff`fffd5018).  Type ".hh dbgerr001" for
details
PEB is paged out (Peb.Ldr = 000007ff`fffd5018).  Type ".hh dbgerr001" for
details
Probably caused by : VSApiNt.sys ( VSApiNt!VSScanVirusInMemory+4eb6 )

Followup: MachineOwner
---------

1: kd> !analyze -v
*******************************************************************************
*
*
*                        Bugcheck Analysis
*
*
*
*******************************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffffa6008b18726, Address of the exception record for the exception
that caused the bugcheck
Arg3: fffffa600bbc4b30, Address of the context record for the exception that
caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
------------------

Page c74fc not present in the dump file. Type ".hh dbgerr004" for details
Page c7dea not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 000007ff`fffd5018).  Type ".hh dbgerr001" for
details
PEB is paged out (Peb.Ldr = 000007ff`fffd5018).  Type ".hh dbgerr001" for
details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".

FAULTING_IP:
VSApiNt!VSScanVirusInMemory+4eb6
fffffa60`08b18726 458b4820        mov     r9d,dword ptr [r8+20h]

CONTEXT:  fffffa600bbc4b30 -- (.cxr 0xfffffa600bbc4b30)
rax=fffff88016a84940 rbx=fffff88027e7d040 rcx=fffff880263c6520
rdx=0000000000000007 rsi=0000000000000030 rdi=0000000000000000
rip=fffffa6008b18726 rsp=fffffa600bbc5390 rbp=0000000000000007
r8=00000000876402c0  r9=fffffffff528975c r10=fffff8803113c036
  Zachary replied to Zachary
19-Nov-09 01:43 PM
I contacted Trend Micro and we rolled back the trend micro scan engine and
we are monitoring the situation.
  Joe replied to Zachary
20-Nov-09 01:37 PM
---Joe   I got the exact same thing on a 2003x64 file server (Dell PE2950).
It crashed yesterday morning during the scheduled scan.  Windbg showed pretty
much what you have here.  It crashed agin this morning at about the same
time.  Ran Windbg on the dump and got the exact same thing.  I have uninstalled
Trend Micro for the moment.  I am real interested to see what Trend Micro
tell's you.  ---Joe
  Gregg Hill replied to Zachary
26-Nov-09 12:25 AM
Zachary,

Trend released a new scan engine, version 9.000.1003, on 11/17/09. As of
today, 11/24/09, it is version 9.100.1001. What version of the scan engine
did you have when it crashed?

Gregg Hill
  Gregg Hill replied to Zachary
26-Nov-09 12:56 PM
I just heard back from Trend. The scan engine 9.000.1003 that was released
11/17/09 had the crash problem. They now have 9.100.1001 that is the fix for
that problem.

Gregg Hill
  Pegasus [MVP] replied to Gregg Hill
26-Nov-09 02:10 PM
The sad bit is that exactly the same thing happened with Trend about five
years ago. Their scan engine would cause a spontaneous reboot on SBS Servers
(and perhaps others) whenever a certain "net use" command was executed,
either from the console or in a batch file. The Trend engineers had  known
about the issue for several months but forgot to tell their own Help Desk .
. .
  Gregg Hill replied to Pegasus [MVP]
26-Nov-09 06:49 PM
I moved from Symantec to Trend around that time because of their new version
causing so many BSODs. Everyone has problems, I guess. The bummer is that I
love the features of Trend, but they REALLY need to work on their catch
rates.

With the newer versions, having URL filtering and Web Reputation enabled
should keep them away from bad guys. However, for those too-new-to-be-listed
sites, I still recommend a WatchGuard firewall to my clients. AV's inability
to detect new threats is precisely why I like my WatchGuard that will not let
the executable through in the first place, whether from HTTP, HTTPS, FTP, or
SMTP traffic.

The way I look at it, letting it in via the front door, then tackling it and
inspecting it, hoping that you are better at recognition than the bad guy is
at hiding, is not as good as looking through the peephole, seeing it is
executable, flipping the trap door, and dropping it.

I have my WatchGuard set up to allow executables from Microsoft and Trend
Micro (after virus scan from the WG), maybe one or two others, but only to
certain IP addresses, mainly servers. I have bypass passwords that allow
managers to download truly needed executables from sites where they expect
the file but where I do not globally trust the site, and even then, they
still go through the virus scan of the WG (it uses AVG).

Of course, I also have Trend WFBS installed on all computers for threats
from other sources.

The best of both worlds! That is, IF I can convince my clients to buy the
firewall.

Gregg Hill
Create New Account
help
Windows server 2008 registry value for MaxUserPort Windows Server What is the equalvilent registry value for MaxUserPort (which is considered in Windows Server 2003) under Windows Server 2008? Windows Server Discussions Windows Server (1) MaxUserPort (1) Vista (1) Windows (1) WS
Windows Server 2008, Intel Xenon 7500 series SQL Server 2008? Windows Server What challenges will the Intel Xeon 7500 series together with Windows Server 2008 R2 and SQL Server 2008 R2 help address for my business? Windows Server
Functional Level Windows Server Can a domain that is built using purely Windows Server 2008 R2 DC's be rolled back to Windows Server 2008? No Recycle Bin. Windows Server Active Directory Discussions Windows Server 2008 R2 (1) Windows Server
Windows Server 2008 R2 Windows Server Windows Server 2008 R2 and Windows 7 share the same code? how is that possible when Windows 7
sending mails in windows server 2008 Windows Server Hi Does windows server 2008 accepts outlook express?if yes where can i download it?if not what is the