ML310 G5 Server not responding at the end of Exchange Backup

[BMason]

Hi All,

I have an ML310 G5 server which has been hanging at the end of the nightly backup. The hang is to the point of only responding to pings and file shares with everything else not working. The only fix is to cold boot the server as you can get to the desktop or send a reboot command. I found that I could recreate the problem when copying files above 4gig to the external backup Harddrive where it would start the copy but halfway through the time would jump from say 10 minutes to 30 to 60 to 160mins before locking up. I then worked with HP and found enabling the write behind cache and change the stripe size from the default 128 to 256 seemed to fix the large file copy problem. I've now just had the server lockup again overnight. It only locks up at the end of backing up the Exchange database which is a 14gig file. The backup actually worked fine for two days before locking up again so it's somewhat intermitant.

The server is running SBS2003 R2 Premium, 3gig ram and two 250 SATA HD's in RAID 1.


Any ideas?

[Rob Buxton]

Anything in any of the event logs?
You say ping works, try pinging with larger packet sizes - say 1500, sometimes shows issues. Also check the duplex settings, although if that was wrong I'd expect the issues to be there all of the time.
How full are the disks? fragmented / full disks can cause similar issues.

You might want to set up perfmon logging to a file. Then check it after it's locked up - look at Memory usage, Page Read I/Os etc.

[BMason]

Thanks for the reply. It doesn't appear to be anything network related as even at the console you cannot get it responding. It's almost like the RAID is timeing out. The server is brand new with only 20% of the drive in use in the RAID 1 mirror. The event logs start reporting errors like the domain controller is not responding and exchange is no longer responding up until you reboot the server. I've just turned off the Exchange side of the backup and will complete manually every couple of days to see if the standard backup crashes the server. It's strange that the hanging of the server appears immediatley after the event log states Source:ESE EventID: 213 the information store first storage group has been successfully completed. Within a minute after that I receive the first of many errors indicated Process MAD.exe All domain Controller Servers in use are not responding. Perhaps the verify is doing it?

I've got the same software setup on over 50 other clients server from ML110, ML 350G5 to DL 380's without problem. Seems to be something specific to the ML 310 G5..

[Rob Buxton]

Having it not be able to contact the DC's could be the network.But the other area is still fragmentation / storage issue.

The only time I've seen similar is on our SAN where we had synchronous data replication. A large copy could swamp the link and the wait on IOs caused problems.

Where are you copying the files from / to in the test? locally within the server or across the network? Maybe try locally within the server. This would exclude any effect from the network.

And are the backups across the network or is the tape device attached directly to the server?

[BMason]

this is a sbs2003 box so it's the only DC. it's copying to a usb 1.5TB drive. only related to copying large files over 12 gig it seems.

[Rob Buxton]

OK - although even with an SBS it may still use IP ports to access the DC, even though its on the same server.

But copying to a usb device shouldn't involve the network.
Do you have any other PC's external to the SBS that you could try copying large files to?

Just a process of elimination.

How much free space on the server do you have in percentage terms?

Maybe try defragmenting the drive.