SAN Attached Host Freezing.

[Pshedges]

Has anyone experienced SAN Boot Blade servers 'Freezing'.

We currently have the following:
HP BL20p G3 SAN Boot Server
Windows Server 2003
Connected to an EVA8000 XCS 5.110
Through Brocade Director 12000's Dual Fabrics.
Utilising MPIO Full Featured
Blade server has Qlogic HBA's installed.
All drivers and firmware are up to the required level for connection to the EVA8000 at XCS Version 5.110.

Server freezes then ASR's for no particular reason.

As this is SAN boot we cannot force a dump file which makes any further investigation difficult.

This is not Hardware specific as we have failed this over to our DR site (separate Blade, EVA8000, Blade chassis, Switches etc..) and it still does it.

This is not application specific as this has occurred on a File Server and a Landesk Application Server.

If anyone has come across this before any help would be appreciated.

[Ivan Ferreira]

I got server freezes and ASR in the following ocations:

- Current Windows service pack not applied.

- Latest HP Support pack not installed.

- Incorrect zoning configuration in the SAN.

[da1nonly1]

Has there been a resolution to your issue?


I found this link...
http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=1009259&admit=-682735245+1160580868092+28353475


We also have the same issue of a server freezing then we can't do anything to it until it is rebooted.

we use an EVA5000

[Pshedges]

Thanks for the reply's.

I am not sure of the issue has been resolved yet or not.

We recently had issues with the Bracoade Director 12000 switches, we were getting dropped paths on multiple servers, this required the firmware to be upgraded to 5.05a, since then we have only had a single occurrence of a server freeze.

HP have checked ther servers/san infrastructure and all firmware/drivers and service packs are at the supported levels so this is not the issue.

da1nonly,

If you are having issues with non-blade servers let me know, you can force an NMI dump when a server freezes, requires a registry change and when the server freezes you need to open it up and press and hold the NMI button on the system board, if you require full instructions let me know.

[da1nonly1]

My issue is with a blp G3 blade connected to the EVA5000.

The server freezes almost every 2-3 days. I was hoping HP had some insight on this problem and had seen it before. appreciate the quick response.

[Pshedges]

We currently have an analyser attached to the host and eva8000, the host finally failed at the weekend so I am hoping we have captured a trace log that we can send to HP.

I will let you know any outcome of HP's investigations.....

[da1nonly1]

Thanks. Please let me know. Very interested on finding out what you have discovered.

[da1nonly1]

Hi Pshedges,

Did you get any results from the analyzer results you sent to HP?

Because of the instability issues... we completely rebuilt the server on a different and older hardware platform provider. And so far we haven't had any issues. This isn't where we want to be in the end but for now it will get us going until we can figure out what causes the freeze up issue with the blade and san.

[Pshedges]

Trace logs were no good, but we have had another trace generated this weekend and I am in the process of sending these to HP.
I will update you with any progress.

[Pshedges]

We have sent a good trace to HP.
They have come back suggesting we set the Port Speed on the Fabric to 1GB, this is by no means a fix but should reduce the occurences of the reboots..

From HP:

"The trace clearly shows everything on the
SAN behaving as you would expect until data returns from the storage to the host. At this point the host sends an ABTS frame, the suspicion is that the CRC errors seen point to some kind of data corruption at the HBA interfacel."



This is still being investigated.

We are currently moving the server to Local Boot to see if we still get the error.

I will let you know when I have more information.

[da1nonly1]

thanks for the feedback... please keep me updated on how things run after the change...

is hp still analysing your data and trying to find other inconsistencies? or is this all they were able to discover and suggest as a resolution to your issues?

[Pshedges]

Investigation is on going..

We are still going to capture trace's, we are now on local boot so we will see if the problem persists

[Pshedges]

Application Server went to local boot, this worked without any issues for 2 weeks, although the analyser was picking up ABTS errors these did not cause the server to 'Freeze'.
We changed (at the request of HP) back to SAN boot on Tuesday, set the switch port speeds to 1GBps and the HBA cards to Auto.
Wednesday - server freezes.

I have sent the traces to HP for investigation.

HP state that if a server starts to experience the freezing issue then to set the Switch Port speeds to 1GBps and the HBA's to Auto, this does not rectify the issue but significantly reduces the occurences.

We had a second server start to suffer freezes, we changed the port and HBA speeds and so far it has been stable.

Investigation ongoing, will update when more is known.

Is anyone still interested in the progress on this or should I close this thread ?

[da1nonly1]

I'm still very interested...

our configuration has always been a local boot and SAN attached configuration but we haven't had the same stability as you have had. freezing/unresponsiveness was/is a day to day
thing with this file server.

What is the average user load on your server? 100+ users accessing it and the SAN at throughout the day?

you mentioned that setting Switch port speeds to 1GBps and HBAs to auto does not rectify the issue but reduces it... does HP know why it reduces the issue? is it something they are trying to address

[Pshedges]

There is still an ongoing investigation.

It sounds as though your config matches the pattern more than ours.

I would suggest (if you can) trying the switch port speeds on a failing host to see if you get any benefits.

The hosts we have failing have low usage, it seems lots of small i/O's cause the freezing/failure.

Feedback on the trace shows that the EVA is functioning as it should and there is still communication between the host and EVA even when the host has frozen.

Will nknow more on Monday after US engineering team have had a look at the trace.

Will update then.

[Pshedges]

UPDATE:
We have install all the latest PSP's, firmware etc.. which have not resovled the issue.

HP are stating that this is most likely a host related OS issue.

We have now had another server fail, this has been moved to the EVA5000 and has also failed, we have now made this local boot.

It appears that as soon as the server goes local boot the issues do not arise again, therefore only SAN Boot blades effected.

Will update again soon....

[chris barnett_2]

I would make sure the port parameters are set to 2Gb and F port on the switch. The Qlogic mez cards are configured to 2Gb and point to point connection. and move the page file off the EVA and onto local SCSI disk. This is especially important if using SCSIport drivers, pagefile's are not supported using SCSIport drivers.

[Pshedges]

Thanks for your reply.

We have been asked by HP to set port speeds on fabric to 1Gbps as this should reduce the number of outages.

We have also had failures when running local boot so this is not a driver/pagefile issues.

[Pshedges]

We have just had another update from HP 3rd Level Support:

"It appears, even though Microsoft provides this functionality of
booting from SAN for Windows 2003 operating systems; they heavily
discourage their customers from actually implementing the same as it has
underlying problems in terms of achieving high availability solutions
and capturing correct information for troubleshooting purposes"

We are currently discussing with Microsoft.

[da1nonly1]

our issues are occurring with a local boot SAN attached model. so i don't think it is exclusive to a SAN boot model.

[Pshedges]

We have finally discovered what our issues were.

LOng story - but it all boiled down to a contractor who was building multiple servers, he ctreated an image and deployed using RDP then used Ghostwalker to rename/regenerate the CID.... this is what was causing our issues, we have now rebuilt all efected servers and have had no more problems.