CommandView EVA lost communication with the Storage System

[Ahmed BH]

I am getting the following error:

---------- Problem Found: : EVA: CommandView EVA lost communication with the Storage System. at Thu 28 May 2009 09:16:03 GMT+03:00 ----------

Problem Report Times:
Event Time: Thu 28 May 2009 07:42:41 GMT+03:00
Report Time: Thu 28 May 2009 09:16:03 GMT+03:00
Expiration Time: Fri 29 May 2009 07:42:41 GMT+03:00

Managed Entity:
SANworks Management Appliance Name: HO-GIS-STRG
WWN & Storage System Name: 5000-1FE1-500D-BEB0 EVA
Entitlement Information:
Storage System WWN: 5000-1FE1-500D-BEB0
Storage System Name String: EVA
Product ID: -- Not Available
Serial Number: -- Not Available


Service Obligation Data:

Service Obligation: Valid
Service Obligation Number: silent
System Serial Number: silent
Service Provider Company Name: Hewlett-Packard Company


Device Type:
EVA

Brief Description:
: EVA: CommandView EVA lost communication with the Storage System.

Rule ID:
Storage_Lx_CV -- V 1.24 Event Code: 5010

Rule Severity:
Critical

Reporting Node:
EVA

Full Description:
CommandView EVA is unable to communicate with the indicated Storage
System.

This can be caused by:
- CommandView losing Fibre Channel connectivity to the Storage
System.
- The Storage System is down.
- CommandView is unable to maintain communication with
the Storage System due to an unknown condition.

The Storage System identification information is reported in the
FRU List.

ELC: If ISEE is enabled, an Event Log Collection (ELC) was sent
with this report.


Failed Component Type:


Recommended Action:
Investigate further: Review the Evidence Section for
more detail.



FRU List:
System Identification --
Storage System Name: EVA
Storage System World Wide ID: 5000-1FE1-500D-BEB0
OS Version: Windows 5.2 Service Pack 1 build 3790
Device Type: EVA_Mgt
Logging Controller Serial Number: - NA - Requires CommandView V7.0 or higher
CommandView Version: - NA - Requires CommandView V7.0 or higher




FRU Location(s):
Storage System Name: EVA
Storage System World Wide ID: 5000-1FE1-500D-BEB0



Evidence:
Corrective Action:
Browse to CommandView EVA and verify the status of the
Storage System. The icon may be displayed as Unknown.
Select the Storage System icon. If CommandView displays
normal Storage System status, the communication has been
restored.

Further analysis of event data, configuration information and
visual indicators may be required.

The Evidence section of this report identifies the
Storage System and time of the event. Multiple Storage
Systems reporting this event at or near the same time
may be an indication of a common path fabric failure.


CommandView EVA is unable to communicate with the indicated Storage
System.

This can be caused by:
- CommandView losing Fibre Channel connectivity to the Storage
System.
- The Storage System is down.
- CommandView is unable to maintain communication with
the Storage System due to an unknown condition.

The Storage System identification information is reported in the
FRU List.
Last Time Stamp: Thu 28 May 2009 07:42:41 GMT+03:00
CommandView Time Stamp: 5/28/2009 07:42:41
Summary of possible Communication Events on this Storage System:
Event Code key
5010 = Lost Communications with EVA
5011 = Restored Communications with EVA
5012 = Restoreed Communications with EVA
8090 = NO Devices Found On Loop
8092 = The Storage Array's name has changed
8093 = A New Storage Array has been discovered
8094 = The EVA Storage Cell was Uninitialized
8095 = The EVA Storage Cell was Initialized
8097 = The EVA Storage Cell is now being managed by this instance of Command View EVA
8098 = This instance of Command View EVA is no longer managing the storage cell
8099 = This instance of Command View EVA has lost communication with the storage cell
8100 = This instance of Command View EVA has no path to the master controller

EC Storage System WWN Storage System Name Event Time
------ -------------------- ------------------------------ -------------------
5010 5000-1FE1-500D-BEB0 EVA 5/28/2009 07:42:41
5014 5000-1FE1-500D-BEB0 EVA 5/28/2009 07:42:36


Notifications:
All


Analysis Mode:
Automatic


SEA Version:
System Event Analyzer for Windows V5.5 (Build 61)


WCC Version:
Web-Based Enterprise Services Common Components for
Windows V5.5 (Build 61), member of Web-Based
Enterprise Services Suite for Windows V5.5 (Build
61)


Could you please advise what is the couse of this error???

P.S. this thread has been moved from General to Storage Area Networks (SAN) (Enterprise). - Hp Forum Moderator

[marsh_1]

hi,

have you checked the status of your hba(s) on your management server ? is this the only eva you are managing with cv , if not is all ok with the others ? have you checked switches for port failures ? physically checked eva status ?

hth

[DanE_5]

Have you gotten a fix for this? I am having a similiar issue. CV works fine then after some arbitrary amount of time (less than 24 hours) it stops working saying:

Storage System Not Responding: Comes up on on the CV page and this from the Management Server Event Log.
This instance of Command View EVA has lost communication with the storage cell.


Any info would be great as I have been working with HP on this for anout a week to no avail.

Thanks


Dan

[Denz]

Hi Guys,

I also have a similar problem and using the following firmware and CV version:

XCS09006000
Management module 1.00.00.080916
CommandView 8.00.02.080909

At the time of this problem, has anyone tried installing CV on a different machine to isolate the problem? By the way, all access to the LUN(s)/Vdisks are still ok.

Any suggestion/comment is greatly appreciated. Thanks.

[Denz]

@DanE

Any updates on how you go about the CV connectivity problem? THanks.

[J.Kool_1]

I got a similar issue here with CV 9.1 on Windows 2008/Sp2/X32 and two EVA 8100 (XCS CR1363xcsp-6220), its getting quite annoying having to reboot my SMA every other day.

Anyone heard back from HP on this?

[Jim Eastman]

I am getting the same issue with Commandview 9.1, on windows server 2008 SP2 X64.
Restarting the commandview server re-enables communication. It is not possible to Start/Stop the commandview service on the server.

[kunalsahoo]

Hi,

do you have 2 HBA cards in the server ?and both mapped to the EVA ?

Please check the following as these are the reason for lost communication.

1 Switch port error details on which server ports are connected

2 if you get high error counts on switch ports,check which parameter is high.

3 If all looks green, then reboot the master controller of the EVA, so the secondary becomes master..and monitor if you still get the same issue, as CVEVA only talks to the Master controller.

[Jim Eastman]

I have a single connection, with single HBA. Not seeing errors on the switch port.

[kunalsahoo]

Hi Jim,

first thing i would suggest to have redundant setup by adding 1 more card.

is this card mapped to both the controller ?

as mentioned earlier , cveva only talks to master controller so the setup you have, it should be mapped/zonned to both the controllers..

[Jim Eastman]

The card is mapped to both controllers.
The SAN is not yet in full production but we will be adding redundancy.

[Jim Eastman]

The following entry is showing up in the system event log:
Reset to device, \Device\RaidPort0, was issued.

[IvanForceville]

We're experiencing more or less the same problem.

2x EVA4400 with the latest firmware replicated with CA and 2x CV EVA9.1 each located at one site.

EVA9.1 installed on Win2003R2-x32

First, the GUI was very slow (5-15 seconds) waiting after every click.
Now, When we open the GUI the EVA's seem to be dissapeared. Discovery can 't find the EVA's...

Call has been logged today, so waiting for feedback... will keep you guys posted.

[SDrake_1]

I'm getting the same issue. My setup is:

2 x EVA4400 (f/w: 0952200)
Command View 9.1

I have lost communication to one of the EVAs but i can manage it through the WOCP. The other Command View server cannot see the problem EVA.

HP want me to reboot the Master Controller but this seems like a Microsoft style of fix! We bought this kit for uptime!

We have a call open with HP too so if i hear anything i will let you know

Simon

[Víctor Cespón]

Change your zoning so each zone contains only ONE EVA port.

This is a known firmware bug. Do not include two ports from the same EVA on one zone, they do not need to talk to eachother.

[SDrake_1]

Thanks vcespon

We have two zones setup thus:

Fabric A:

Controller 1 Host Port 1
Controller 2 Host Port 1

Fabric B

Controller 1 Host Port 2
Controller 2 Host Port 2

------------------

Are you saying that this is incorrect (or incompatible with the current firmware)?

Thanks

Simon

[Víctor Cespón]

Losing communication from external Command View while the internal Command View keeps working has happened on several customers. We had to reboot or resync the controllers to recover access.

The workaround for the moment is to configure one zone for each EVA port, since each port does not have to talk to the others.

[SDrake_1]

@ vcespon

There are 4 device ports on an EVA4400 are you saying we need to configure 4 zones?

Thanks

Simon

[Víctor Cespón]

4 host ports

Controller 1 FP1
Controller 1 FP2
Controller 2 FP1
Controller 2 FP2

FP1s will be connected to one switch
FP2s will be connected to another switch

In each switch, each zone must include only one EVA port, so 4 zones per HBA.

This is how it works now, what we have been told to do by HP L2...

[Johny Root]

I had the same problem. Eva was turned off and opened, now everything is working properly.

[SDrake_1]

I rebooted both controllers (through WOCP) last night and i managed to get management back.

Unfortunately before we rebooted the controllers we had an ESX host fail yesterday morning due to HBA timeouts (to the same EVA). It took HP two hours to get the VMs which were on this host back up and running.

Not only did the host become unusable (until the controllers were rebooted last night - go figure) but vCenter went wonky and we could not see any LUNs on any host managed by vCenter.

We had to log directly into the hosts to add VMs back and view datastores. (you have to be able to view a datastore to add a VM back to a host or the inventory - if using the GIU)

All a bit of a pain and HP have not covered themselves in glory.

Awaiting answers from support.

Simon

[SDrake_1]

Reply from L2 team:

"With respect to EVA loosing management access..
Command View communicates with EVA through LUN 0 which is a command console lun, and most management SCMI commands are executed through master controller. As with most arrays the storage commands are given more priority over management commands. The CV would have issued some commands & probably they may have been of lower priority. Until the command is finished, SCMI will not be able to process next command (in this case, the next command is to view EVA in CV). In these cases, we will need to wait for SCMI commands to complete. The other alternative option is to reboot the master controller, so that the earlier command is aborted. The next command (view EVA from CV) will be processed by the then master controller."

We also had a lot of HBA timeout errors on the host which lost access to all LUNs.

So basically the EVA had a SCMI command pileup and couldnt sort itself out.

Is a manual reboot of the controllers then only way to fix this problem?

And, can i expect this problem to happen again?

BTW, there was nothing happening on the EVA when the initial management was lost

Thanks

Simon

[Kritao]

hi we have the same problem. the command view on the management module able to detect the controllers whereas the cv9.2 installed on the management server does not detect the eva4400.

we managed to bring the eva4400 with the one controller and upgraded the firmware also to the latest. unfortunately when we inserted the second controller, it is not replicating the firmware image to the slave controller.

we had a hard time with eva4400, any workaround to resolve these issues.

[Uwe Zessin]

> unfortunately when we inserted the second controller

STOP! STOP! STOP! STOP! STOP! STOP! STOP! STOP! STOP! STOP!

If the system contains valuable data:

Please pull the second controller and stop playing around *NOW*.
You do risk losing *ALL* data!

Call your service provider and you both develop a plan how to exit from this critical situation.