Random Windows 2003 server freeze

[A.G.]

Hello,

our problem is that multiple of our server machines freeze at random.
In some cases background processes (like data base, web service) continue to work but are slowed down.
Logon (via remote desktop or directlz at the machine) is not possible. Hard-reset is the only option to restart the machines.

System environment:
Machines: 6 HP Proliant ML370 G5 and Proliant DL380 G5
Operating system: Windows 2003 server R2 SP2
Applications: Varies from machine to machine (MSSQL 2005, IIS, 3rd Party software). No common application on all servers above the OS-level.

Remarks:
- Machines were installed in April 2009.
- Of the 6 servers only 4 crash frequently, 2 of them are not affected
- Freezes are not simultaneous. When one servers freezes, others mostly do not at the same time.
- The same software project is installed at a different site (different HP Proliant series used) and works without any problems.
- Freezes seems to occur more and more frequently (no freezes in April/May/June, few freezes in July/August, weekly in September/October, almost daily in November)
- Windows event log and application logs do not show any suspicious entry.

Actions taken so far:
- Ran hardware diagnostics tool: no problems detected
- Re-installed Windows SP2 and MS Patches: no effect
- Scanned for viruses: nothing found
- Installed newest HP-Support-Pack: no effect

Now, we are running out of ideas.
Any help or hint is appreciated!
Thanks in advance!

Andre

[Michal Kapalka (mikap)]

hi,

what about some infected network traffic ???

check what type of connection are to the servers, it seems to be made everithing, what i will do but, if the other servers are working what is the difference ???

somethings must by other.


mikap

[Edgar Zapata]

Hi Andre,

I would definitely break this down in the following:

1. Either it is a system process or software component that's causing system degradation or
2. This is being caused by any kind of malware (Virus/attacks/spyware)

Remove any suspicios software (any that could exploit the system)
I understand you've tried to establish a pattern to account for that behavior.
All I can think of is try to isolate one of the servers that have freezed:
a. Unplug the network wire and see if it still freezes.
b. Run msconfig to disable any services that are not essential.

- Run System Management Homepage and see if the logs in there give you a hint.
- Run chkdks (read-only) to discard disk errors.
- Dump the contents of the memory the moment the server starts freezing. This can be done throught the keyboard.
Here's HOW:
http://support.microsoft.com/kb/244139/en-us
This can be done through the iLO too:
Here's HOW:
http://support.microsoft.com/?kbid=927069
- Run System monitor (perfmon.msc) and see if there's any suspicous process.
- Run procexp.exe and see if there's any process going high on CPU or Memory usage.
You can find the tool here:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
- Check for any events that may indicate System degradation (like 2019 or 2020. Source: SRV) I know you have already.
- Check for any LEDs at the front pane.
- Check if any of the servers has a USB memory stick plugged in.
- Run the AT command and see if there's any scheduled task you're not aware of.

Run a RootKit revealer tool.

Hope to hear back from you.

Regards.
Edgar

[Phil.Howell]

You should also run netstat -e
and check that errors = 0
Phil

[A.G.]

Thanks for all the advice so far :)
I will try and let you know.

[A.G.]

So, I tried many things, none with success:

- netstat -e: no errors
- AT: no unwanted jobs
- msconfig: nothing suspicious, only MS and HP stuff
- rootkit revealer: nothing suspicious
- chkdsk: no erros
- process explorer: nothing suspicious (high CPU usage, RAM/VirtualMemory usage, number of threads etc.)
- poolmon: no memory leaks
- net work traffic: seems to be normal
- HP Systems Management Homepage: no aberrations
- USB sticks or strange devices: none

Yet to be checked:
- Memory dump at moment of freeze
- Check if server freezes in stand-alone mode (network cable unplugged)

Any other ideas?
Thanks
Andre