"Volume is Missing" EVA 4k SAN

[Brett Barton]

Good morning, our school district has an EVA 4k series SAN that consists of two HSV200 controllers as well as 4 disk enclosures containing a mix of FC and FATA drives.

Last week some additional disks were added too quickly to the SAN in the # 4 and 5 enclosures causing a loop interruption. That issue was quickly corrected but we now have an error with two drives in enclosures 1 and 2, which we not touched when the new drives were added.

Currently there are warning signs icon on the "Disk Enclosure 1" and "Disk Enclosure 2." When I expand both enclosures I see 1 disk in each group that has the "red x" icon. IN each enclosure it is Bay 14.

When expanding Bay 14 the Condition/State reported by CV is "Populated (data disk)" with a red X icon. When I click on the Disk Drive tab I receive the error message "Operation Failed! Volume is Missing."

I also managed to locate the same disks in the Disk Group folder, both disks are within the same disk group. The affected disks also show up here as having a red x icon. When I click on the disk under the disk group I get the same message "Operation failed! Volume is Missing."

Once again all this happened after having a loop interruption on enclosure 5, but the two disks having the issues are on enclosure 1 and 2.

The physical drives do not show any error LEDs, nor is there anything interesting in the logs. The VDISKS still show up with green check marks and CV reports "All of your Vdisk data is protected by redundant disk drive storage."

However the messages are still concerning to me. Any pointers would be great.

Thanks!

[SUBHAJIT KHANBARMAN_1]

Can you please share the controller event logs and configuration log ?

[Brett Barton]

Thanks for the response, for the controller event log can you use the .idi log file? Also where would I find the configuration log?

I am relatively new to this SAN. My coworker responsible for this SAN recently accepted a new job and can no longer help.

Thanks!

[SUBHAJIT KHANBARMAN_1]

What is the CV EVA version ?

Attached is the document in order to collect logs.

[Johan Guldmyr]

That guide is only for CV 9.x.

Here is the web version of the same guide:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c01775768

--

To get the logs if you have a lower version, you can look in this thread for the configuration file: http://h30499.www3.hp.com/t5/Storage-Area-Networks-SAN/SSSU-config-capture/m-p/4470195#M39512

-- and yes, it is the .idi file. You can get it from the file system or eva -> events -> controller events -> get log file. (not the parse file) ;)

[Brett Barton]

Alright, we are running version 7. So I had to use the SSSU to get the configuration. From the post the second link that Johan provided I ran the script I also did a capture configuration and saved the results into a separate file.

Also I pulled the controller log from CV. Any idea on how far back this log records? The trouble started last week Friday.

I attached a zip file containing 3 logs.

Thanks a lot!

[Johan Guldmyr]

Brett: I don't have access to the log analyzing tool anymore but can't you sort by date in Command View - to see how far back they go?

[Brett Barton]

Well if I go back to the longest "Display Range" I can see all the way back to 2007 which is plenty far.

I just didn't know if the .idi file kept the same date range. I would assume it would.

[Johan Guldmyr]

It does, it's the same :)

[SUBHAJIT KHANBARMAN_1]

Thank you for sharing the logs. However config log is not proper.

On 13th May there are hardware errors captured for few drives.

As per my understanding I can suggest you to re-seat the drives at the location enc 1 bay 14, enc 2 bay 14, enc 5 bay 1 and enc 5 bay 2 one by one.

Suggest you to log case with HP because you may have to replace drives.

[SUBHAJIT KHANBARMAN_1]

I forgot to mention that the EVA is running on old firmware means inactive firmware. Please update it to latest firmware(XCS6.240).

Please find the advisory,
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c02747476&jumpid=reg_R1002_USEN

and check for "EVA XCS support matrix"

[Brett Barton]

Thanks for taking a look. Would it be worth doing a "refresh" on the cell before removing and re-adding the drives?

[SUBHAJIT KHANBARMAN_1]

Oh I thought you have already done that.

Please go ahead and restart the CV EVA service and check.

At the same time suggest you to re-seat the suspected drives within 10 seconds each one and one by one.

After that share the log again.

[Johan Guldmyr]

You have 60 seconds per drive, but have it out at least 10 seconds before putting it back in (to let it spin down).

After each drive I would suggest that you check command view to make sure everything is OK(so wait a minute or 10, up to you ;) before proceeding to the next.