ML 370 G5 BSOD, Freezes,Crashes when doing backup

[johnwanderson]

I have just installed an ML370 G5
4gb of Ram- 2 logical drives
2 x 146gb hdds Raid 1 and
4 x 146gb hdd in Raid 5 with hot spare
Windows 2003 R2 SP2 with all latest windows updates
Using Smartstart Version 8.0 and PSP 8.0
Using Firmware Maintenance 8.0 but have already upgraded to Firmware maintenance 8.1 due to the problems i have been having


The problem i experience is the server freezes when doing a backup
I have an Adaptec 29160 and Quantum LTO3 Half height drive that worked in a DELL Poweredge successfully
The same problem occurs when using both Symantec Backup Exec 12 or Arcserve 11.1 with SP2
I have tried swapping out SCSI controllers and cables I have tried a HP SC11 EX adapter
If the backup is not scheduled The server will not crash
I have also disabled the onboard Network controllers and replaced them with an Intel branded card but the same problem continues

I have logged a call with HP but they cant seem to own up to the problem
I have ran the HP SRP Version 8.0 reporting tool but this stalls when gathering ACU details

Can anyone assist me as it has been going on some weeks now

Just to let you know

If i take the SCSI controller and Tape unit out of server and put it into another DELL
The backup runs fine
If i add the HP server with remote agent The HP server will also crash

The reason the customer put this HP in was to replace a DELL that was causing problems so i need to get some stability and a good backup to keep my head high

Please get back to me as soon as possible

REgards

John

[tomasz.puchalski]

I can tell You that in similar situation i always make backup (for test only) to file not to backup device. after that i know more -

[KarloChacon]

hi

have you installed the storport fix?

http://support.microsoft.com/kb/941276/


what about the errors in windows event viewer?
source?
event id?

regards

[johnwanderson]

I have installed the latest HP storport driver as requested by HP and it hasnt made any difference KB932755
and an update to turn off SNP KB948496
Have run full diagnostics from Smartstart 8.0 CD and looped it three times and it doesnt come up with any info
Please find log attached
I did note when running the backup last night that when the server froze overiusing HP branded HP SC11 EX Scsi host bus
No keyboard or mouse was available Task manager was open and nothing was spiking CPU or memory
The lights on the 2x Raid 1 disks were flashing on and off and lights on 4x Raid 5 disks had lights on steady -Not flashing

[James ~ Happy Dude]

Hello John,
If the server encounters a BSOD, I am sure there is a DUMP file. What does that say ?

Regards,

[johnwanderson]

The server freezes and then you have to physically switch off the server
The last minidump was on the 18th of June and it pointed to bxnd52x.sys which is a broadcom driver related to HP373 integrated NIC

I have since disabled both hp 373 nics and put in a third party intel nic and have not had a minidump since 16th of June but yet the server still freezes on backup

Could it be related to big load on raid 5 drives on p200 with 128mb battery backup with cache??

[KarloChacon]

well

I know there is like an issue with ML350 G5s with that e200 controller

usually the fix was:
latest firmware for e200
latest driver for e200
and the storport update

but you already applied those

BTW interested that dump indicates an issue with NICs I was confused why HP told you to turn off SNP

I would say e200 it's an entry level smartarray even should work right

is that the only server around?
it is new I mean I new deployment - Os installation?

regards

[johnwanderson]

All the latest updates are applied with Firmware 8.1 and smartstart 8.0

I would have sent them the minidump that had the broadcom file in it

I had a similar problem with NIC in another ML370 G5 earlier in the year but disabling TOE resolved it
It is a brand new server with fresh installation of windows 2003 r2
No extra third party apps installed only Arcserve 11.1 SP2
Again I had originally installed Symantec Backup Exec and it had the same problem

I think it is pointing towards the Raid Controller now but i cant be certain

Any more ideas how to sheck this out
After doing a full diagnostics through smartstart 8.0 with no errors
It is becoming very frustrating

Regards

John

[James ~ Happy Dude]

Hello John,

First, Check compatibility list : http://www.hp.com/products1/storage/compatibility/tapebackup/ISS/13-0003-0002.html#matrixtable
Click on the "TICK" for the appropriate Server & Drive; & on the next page you should see the compatible controllers.
If its NOT supported, Then its just one of those cases, where one runs out of luck (of having "X" hardware work with "Y" Hardware) !!

You have already mentioned about using the latest Firmware & PSP.

Steps to try :
1) Download HP CreateData utility from :
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&swItem=co-10838-1&jumpid=reg_R1002_USEN

This Utility will help you create blank data. Create Data of 10 GB.

2) Use NTBACKUP to back up this "HPDATA". Make sure u try this on a new media.

Wait n watch :
(a)If the above action fails. Then Issue is with the Hardware.(subject to compatibility as mentioned before)

(b)If this works fine, it will conclude that the HP Server, controller, tape, media, Cables... etc are FINE !!
Then the problem is either with your DATA or with the Software you use(arcserve/symantec).Certain software WOULD require the DATA to be FREE of any activities while backup.

(b1)Then, you may try a backup of YOUR DATA using NTBACKUP. If this succeeds, then 100% issue is with the Software. If this Fails, then Issue is your data.

Please let us know how this works out. we'd be curious.

Regards,

[James ~ Happy Dude]

Sorry, For windows the link is :
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=US&swItem=co-9515-2

Regards,

[johnwanderson]

Hi

I have checked the compatibility on HP website previously and because the LTO 3 is not HP (It is a Quantum LTO3 Half Height TCL32BX)They will not guarantee it is compatible while quantum say the LTO3 is compatible with an Adaptec 29160 (this combination worked fine in a Dell poweredge)but so far not this model HP
The scsi controller is now working with the Quantum with Cable in another third party server

Just to say i did note that since having the backup on another server It does not crash as regularly with the remote agent now
I have excluded some pst files from the backup which you could see in the backup log that the server was crashing when backing these files up

I will wait to see tonight whether or not the server crashes and if not Put the controller and unit back onto the HP server and backup excluding the pst files to see what result i get
Again the RAID1 c:drive of 146gb of the server always backs up ok regardless of amount of data and it would always crash when starting the RAID5 d:drive which has 420gb of diskspace using around 190gb
About 80 gbs of this are backup pst files for around 50 users

I will let you know how i get on
Thank you all for your help

Regards

John

[RaMpaNTe]

Hi, the Adaptec card let's say is out of the question since is a non supported card in the server, so any BSOD caused when in use of said card will not be supported. This applies also to any TAPE device not listed in the compatible options for the server.

In the other hand, this little friend the HP SC11Xe is well known for causing not only BSODs but also causing some servers not to power on at all. I have dealt with it in the past.

The best thing to do here is to get rid of that card and use the following instead 64-Bit/133-MHz Single Channel Ultra320 SCSI HBA G2 374654-B21

There is a FW version for the bad boy if you wanna give it a last chance. It is located here: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3191201&swItem=MTX-4f16c14c1aa54618a879868cd2&prodNameId=3191202&swEnvOID=1005&swLang=8&taskId=135&mode=3

Also there is a updated driver at
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3191201&prodNameId=3191202&swEnvOID=1005&swLang=8&mode=2&taskId=135&swItem=MTX-7dbe36462d5c490e89abc5cff0


Thanks and I hope this can solve your problem.

\RaMpaNTe




PS: say thanks with points :)

[johnwanderson]

I have the latest firmware and drivers for the sc11ex installed
I had tried also a HP branded LSI Logic 20320 SCSI controller and the HP guys said it was a Raid controller but it had the same effect on the system as the other HP SC11 Ex

The latest comment from HP was to disable ASR which i have done, so iam waiting for a crash

What i would like to know now is if the ASR is disabled Will the server reboot after its next crash as i dont have a good backup of the system state yet and am worried that it may require a rebuild

Again this is a live production server so please let me know the consequences

Regards

John

[RaMpaNTe]

ASR stands for Automatic Server Recovery and it has a threshold of 10 minutes, so if the server freezes for more than 10 minutes of if there is any other problem the ASR reboots the server so it won't be stuck for a long time.

By disabling the ASR in the next crash you will be able to "see" what caused the crash by checking the event logs IML and others.

[Bobby Campbell_1]

I have had the exact same problem on a load of G5 servers. The fix we used was changing the page file to a fixed size and only on 1 disk. It generally always happened on backup !

[johnwanderson]

There is 4gb of ram in this server and the page file is set to 4096 to 4096
Can you recommend the size of the pagefile as i would be interested in trying this

From last mail
Just a note to say Excluding pst files did not make any difference The srever froze over again and it had to be physically switched off and there was no mini dump file for that day
The hp engineer has now asked for a kernel dump file which we are now waiting for server to freeze again
Will keep you posted

I also created the data file requested earlier and it didnt seem to freeze

Your thoughts please


Regards

John

[johnwanderson]

The latest details are
I have the SCSI Controller and Tape Unit in a third party server and am running backups using Arcserve 11.1 with latest service packs and hotfixes with remote agent on four other Dell servers and the HP ML370G5 in question
The backup agent backs up all other servers without any problems
The remote agent will never fail backing up HP ML370 G5 Raid 1 Logical drive which is the c:\drive-
Again this in a P200 Raid controller with 128mb BBWC
Somewhere through the backup of the HP Server d:\drive (Again this is a 2nd logical Drive in a Raid 5 formation with 4 x146gb Disks)
The server will freeze (No keyboard,mouse or task manager) Push button for 10 seconds reboot is required As a result no minidump or as requested by hp engineer kernel memory dump is not created I have sent the HP Engineer on the Insight Management Logs for viewing but he hasn't any ideas as yet

Any assistance would be appreciated
Maybe more information regarding Page files or p200 issues

Regards

John

[johnwanderson]

Just to let you know the latest

I changed the memory in the server and also changed the pagefile size
I also disabled all the HP programs in the services tab in the control panel to see if that would make a difference
Again We got one good backup but the server froze again on friday nights backup

I will be putting the rebuilt DELL back in place tomorrow night with the view to wiping the HP server and rebuilding it
Any other thoughts since

Regards

John

[dcolpitts]

We ran into a similar issue with both an ML350G4p and a ML370G4 using the HP Ultra320 controller that is based on the LSI chipset. We did alot of troubleshooting, including reloading the server, replacing the tape drive, tapes, scsi cable and controller.

Finally we built another new box, patched via Windows Updates, installed KB932755, then installed the Ultra320 card into it, followed by BackupExec 12. We were able to get successful backups, but then we discovered we were using the generic Microsoft LSI SYMPCI.SYS driver for the Ultra320 card. As soon as we loaded both CP008498 (lsi_scsi.sys - 2007.02.09), we started getting backup errors again.

After doing some Googling, we found a Symantec forum posting detailing the issue, and stating there is a bug with the current HP scsi drivers and that you need to use the generic Microsoft drivers. Rolling back the driver to the Microsoft driver solved our problem.

When we had previously reloaded our servers in the troubleshooting process, the Ultra320 controller was in the servers, and when we ran the PSP setup, it installed the HP drivers. As we didn't have a spare Ultra320 controller, we built our test box first, then pulled the controller out of a production box, but we forgot to re-apply the PSP, so that is how it ended up with the generic Microsoft driver instead of CP008498's driver.

dcc