ML350 G4 server with SBS 2003 - Server freezing

[Abid-sof]

Hi,

We have a SBS 2003 server that has some serious issues for nearly 2 months now. The problem is that server freezes for a short span of time and comes back to life. Freeze happens many times every hour and lasts 50-60 secs each time and then come back to life. During the freeze:

• Users get the popup balloon in Outlook telling them that Exchange is trying to retrieve data.
• Shared drives cannot be opened.
• Softwares retrieving data from the server also appear hang/freeze.
• Remote Desktop connection to the server continues but cannot perform any actions (only mouse pointer moves but cannot select anything).
• During the time of freeze following Events are generated in System log:
• Going through the logs Event ID: 7011 always seems to be recorded around the time the freeze happens. It says, 'Timeout (60000 milliseconds) waiting for a transaction response from the NtFrs service'.
• Also in Applications Log This error is found: Event Source: MSSQLSERVER - The Scheduler 0 (and other times 1) appears to be hung. SPID 0, ECID 0, UMS Context 0x03640BC8

• When the server is booted it takes over 40 mins on 'Applying computer settings'.
• When the server is restarted or shutdown it takes about 15-20 mins.

A bit about the server:

OS: Windows SBS 2003 Premium Edition – SP2, with Exchange 2003 SP2, SQL 2000,
SharePoint Services 2.0
Hardware: HP ProLiant ML350 G4
Smart Array 641 Controller
6 x 146.8GB HDD (10K rpm) configured into 3 Logical drives
RAID 1+0 is used
x1 Intel Xeon 3.0GHz Processor
4GB of RAM installed – SBS 32-bit only uses 3.5GB
Antivirus: CA Threat Manager
Backup: on Tape using Yosemite software

Hardware diagnostics (using HP diag tools) show no issues. Green LEDs are shown on all 6 Hard Drives. I have already updated all device drivers and system firmware using HP tools. HP Array Configuration Utility shows no issues at all.

Thanks for your time.

Abid

[Abid-sof]

Hi,

I have calculated that it started happening since I followed CA Tech's suggestion on how to fix the CA Anti-virus on the server to enable users saving documents directly on the shared network drives. CAâ s fix had its side effects. Over a period of 2 month there were over half a million of 4KB files generated in Windows Temp folder. Now temp was so full in terms of number of files that every time server had to do anything in temp it would freeze.

Windows would not let me empty the temp folder. It did not even show the files. We could only see and calculate files in DOS mode. UBCD4Win was used and slip streamed with Windows XP setup to create a bootable file. After the server boot from teh bootable media, all files from Temp were deleted from command-line. It took over an hour only to delete all those files. After emptying the temp server booted withing 10-15 mins.

Then the next step was to locate the program which was creating 6 temp files each minute. Process Explorer and Process Monitor was used to identify the culprit. It was CA Host-based Intrusion Prevention Manager which was bought as part of CA eTrust Threat Manager package. It was found using an instance of Java under a process Wrapper and creating hss_tmp**************.dat files in C:\Windows\Temp. Disabling CA HIPS service resolved again filling up of temp. There are no freeze issue anymore and server works fine.

Disabling all CA package didn't stop the freeze from occuring because the over-grown temp folder was still in place and SBS OS was still accessing it thus halting on times.

I am hoping that this may help someone in future.

Abid

[Abid-sof]

Found a solution to the issue and has already posted it on the forum. This thread can now be closed.

Thanks,
Abid