disk enclosure problem on EVA4400?

[iulianv]

briefly, the history is as it follows (the setup is an EVA4400 with one double-controller HSV300 enclosure and three disk enclosures):

a few days ago, one FATA disk simply disappears from its group - i only noticed it yesterday, re-inserted the disk and added it to its group again

before the leveling was complete, the disk disappears again, and all other disks in the same enclosure (some FATA, some 15K, so different disk groups) only remain reachable through a single port - to be more accurate, CV-EVA says that "Loop A" is unavailable for every disk in the enclosure; the other two enclosures are fine, and the problematic one reports both its I/O ports as "Good" in CV-EVA

i'll try replacing the disappeared disk soon, but i'm more concerned about the "missing loop" for all the disks - what might have caused that, what options do i have (besides CV-EVA event log) to further debug it, and what can i do to fix it? all these, as much as possible, without powering down the enclosure (i guess this would require shutting down the blade rack completely, since the disk groups will be unusable without one third of the physical volumes)

thanks in advance for any hints on this

[Rob Leadbeater]

Hi,

The failing disk is probably causing some disturbance on the fibre channel loop, however your EVA4400 is almost certainly under support with HP so why speculate in a forum ?

Log a fault call !

Cheers,

Rob

[iulianv]

it is indeed under support - the whole rack was purchased about two months ago - i'll contact HP support after i reach some conclusion about the disappearing disk...

as i said, i just replaced it - the new one is reachable through both Loop A and B, but all the other disks in the enclosure still have Loop A not operational; i'll see what happens after leveling - is there any manual operation i have to do to "revive" Loop A on the rest of the disks, or it should go up automatically in case the FATA disk was the cause of the problem?

[IBaltay]

Hi,
can you look at the controller log for determining the root cause of this behaviour pls? Can you list the main (critical) controller events here to seee what happened and to be able to interprete?

[iulianv]

i'm attaching the related events, in reverse chronological order, as copy-pasted from CV-EVA's event log

leveling after replacing the "disappeared" disk completed normally - the new disk is reachable through both its ports, while all the other disks in the same enclosure still have LoopA unavailable; i'm thinking about restarting Controller 1 in the controller enclosure, in order to "force" port rediscovery of the devices

[IBaltay]

Hi,
can you insert the problematic FATA disk in any other disk bay to see if the problem persists pls?

[patbourdon]

hi, i just want reply to Rob, ok to "log a cal to support", but when support takes weeks to resolve a problem, you have to try to do the job yourself... i've some calls waiting for an good answer from hp support, sent a mail to support manager for speed up the call, but no solution to this day, so JulianV is right to try here... ;) (i'll log my problem in another thread in case somone can help me...).

[Mark...]

Hi,

A few suggestions for you to try:
I often find that a problem is fixed but the warning triangle is still there on several items. Try clicking on every item with a triangle, in this case your disks and you MAY find that that will resolve your problem once you have clicked on all the items concerned. If this is the problem you should find that the warning triangle goes away when you click on each item but make sure you start at the lowest level ie the disks in this case.

If not, have you looked at the properties of the controllers and the "device ports" to see if there is a little tab that says "enable" as this will / should restart the loop concerned.

You can also open up the WOCP and look at the option "Configure controller device ports" which also has an option to re enable the device loops.

In the WOCP there are also some log files and controller information in the bottom two options. This may also give you some information about your current problem.

Finally I would suggest that if the disk disappeared from the loop/config in the first place then it could well be faulty and pulling it out and then putting it back in will not "make it better". If you have the capacity and you still have this "suspect" drive installed then I would suggest you use the "ungroup" tab to ungroup the drive correctly, then use the "remove" tab to take the drive physically out of the EVA and check the above suggestions again.

Mark...

[iulianv]

on Monday i'll try inserting the problematic disk in another disk enclosure, to see if it "contaminates" that one as well; i'll also go wire the controller's management module's ethernet port, to see what WOCP has to say about this

as for clearing the yellow marks by clicking the items, what happens here is the other way around - when i open CV-EVA, all disks in the problematic enclosure are green under the "Hardware" tab; once i start clicking them, they get the yellow mark; but they are always yellow under the "Disk Groups" tab

i couldn't find any option to re-enable or restart disk ports or disk enclosure ports - just the "Shutdown" button for the two controllers

[Mark...]

Hi,

The "enable" the loops tab only normally shows when a loop is down.

FYI - the default I/P address for the WOCP is 192.168.0.1
def user = "admin"
def password = NOTHING, just hit enter/return to login.

Mark...

[iulianv]

i know, just that the ethernet port of the management module is not connected anywhere right now, so i'll have to get to work first to wire it, in order to make it's IP address available for access...

[IBaltay]

Hi,
and did the "problematic" disk spoil also the other enclosure pls?

[iulianv]

i don't know yet - as i said, i'll try all these on Monday, when i get to work, right now i'm enjoying the week-end at home ;)

[iulianv]

ok, these is the current status and the events that led to it:

i went to the datacenter and plugged the "disappearing" hard drive into another disk enclosure (i didn't add it to a group though) - nothing happened; i plugged another spare hard drive into the problematic enclosure - again, everything ok (at least judging by the LEDs, as i don't have easy access to CV-EVA there)

i connected the laptop to the management port of the controller enclosure, logged into WOCP, and performed "unbypass all" to both controllers - again, no change in LED configuration (still orange LEDs on for 10 out of 12 hard drives in the problematic enclosure)

i then disabled automatic loop recovery, and found out that i cannot enable it again - it says "operation successful", but when checking again, it's still disabled - this _really_ annoyed me

i came back to the office, logged into CV-EVA, and found everything green (?!)

now, i'm too lazy to go back downstairs and see if the hard drive LEDs are still as orange as i left them; apart from that, i'm still confused about automatic loop recovery not coming back again, as well as about whether the "disappearing hard drive" is indeed faulty or not...

a pretty frustrating experience alltoghether :)

[IBaltay]

Hi,
can you review the controller to see what is going on in your EVA 4400 controllers?

[iulianv]

what is going on is that, although everything's green in CV-EVA (i even have log messages saying that FC devices previously known with only one port have been fixed, and redundancy restored), the orange LEDs on the hard drives are still on. nice...

[Richard Choi (Ultima)]

I have a similar problem, error suggestions something wrong with the Fibre loop switch, with FATA disks missing and multiple FATA disks are getting hardware failure. Would be interested to learn what is the underlying problem

[iulianv]

well, it happened again for me - this time another FATA disk failed in a different enclosure, and all hard-drives in that enclosure lost one loop and got the amber LED lit

i performed an "unbypass all" from WOCP, and their status "turned green" in CV-EVA, but the amber LEDs are still on - the first time it happened they turned green when i upgraded the hard-drives' firmware - now how do i turn them off?

i replaced the faulty FATA disk with the one that seemed faulty last time - that one looked OK while ungroupped, so i hesitated replacing it, this one keeps reporting check condition errors every minute, and i'll definitely replace it

by the way, what check condition error rate could be considered "normal" for FATA disks?

[CSA Support]

Check the EVA 4400 firmware if you are running 900000 this firmware had issues with the 1TB fata drives, update to latest firmware 09004000 could help you will also need CV 8.02 update before installing this firmware revision.

THere is also a firmware release for the WOCP coming soon that resolves a comms issue beetween WOCP and Controllers.

[iulianv]

i was running 09001000 when the problems occured - now i've upgraded (by HP recommendation) to 09003000 (with CV-EVA 8.0.1)

i'll see what happens... :)

[Dr_No]

Its ben OCT since last reply, but let me take a shot.

Maybe is a firmware issue?

On the 1TB FATA Hard Drives, firmware HP03 was causing send/receive issues to the controller and management module.

They said they have fixed this in firmware HP04.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=co-61563-1〈=en&cc=us&idx=0&mode=4&

Also, they recommend to upgrade to firmware 09004000 on the controllers.