From: uksoeva01@skandia.co.uk Sent: 21 October 2010 09:42 To: UK DL ITS Server Team; UK DL ITS Datacomms Team; UK DL ITS Storage Team Subject: HP SEA: CUST: CA warning: UKSOEVA02: Excessive out of order message rate on the inter site link., System Event Analyzer Detected Fault Attachments: ATT1803375.txt ---------- Problem Found: CUST: CA warning: UKSOEVA02: Excessive out of order message rate on the inter site link. at Thu 21 Oct 2010 09:42:21 GMT+01:00 ---------- Problem Report Times: Event Time: Thu 21 Oct 2010 09:38:58 GMT+01:00 Report Time: Thu 21 Oct 2010 09:42:21 GMT+01:00 Expiration Time: Fri 22 Oct 2010 09:38:58 GMT+01:00 Managed Entity: Storage Management Server: uksoevamglv01 Controller Type: HSV450 Firmware Version: CR18CB 09534000 WWN & Storage System Name: 5000-1FE1-5023-E1F0 UKSOEVA02 Entitlement Information: Product ID: AJ758A Serial Number: SGA92401D4 Service Obligation Data: Service Obligation: Valid Service Obligation Number: GB483400GB System Serial Number: GB483400GB Service Provider Company Name: Hewlett-Packard Company Device Type: EVA Brief Description: CUST: CA warning: UKSOEVA02: Excessive out of order message rate on the inter site link. Rule ID: Storage_Lx_DRM -- V 1.19 Event Code: (0C 59 69 0C) Rule Severity: Critical Reporting Node: UKSOEVA02 Full Description: Excessive out of order message rate on the inter site link is impacting replication throughput. The Device location, reporting controller, and corrective action is reported in the FRU list. Other information about the event is reported in the evidence section The UUID of the remote DR group and the UUID of the Peer storage system is stored in the FRU section. A summary of events have been consolidated in the evidence section. This is an Customer only notification. A notification will not be sent to HP Services. Failed Component Type: Continuous Access Recommended Action: Check fabric switch settings and inter site link quality between the Data Replication Source Site controllers and the Data Replication Destination Site controllers. FC Loop: UUID of Group Name: N/A UUID of Remote/Peer Storage System: 6001-4380-05DE-D298-0000-5000-0002-0000 UUID of Data Replication Group: N/A UUID of Source Virtual Disk: N/A UUID of Destination Virtual Disk: 5000-1FE1-5024-482A-0000-0000-0000-0000 FRU List: System Identification -- Storage System Name: UKSOEVA02 Storage System World Wide ID: 5000-1FE1-5023-E1F0 OS Version: Windows 5.2 Service Pack 2 build 3790 Device Type: HSV Controller Logging Controller Serial Number: PAFURB49SXN02G CommandView Version: 9.02.01.100426 EVA User Initiated Service Mode Flag: 0 Detail Location of problem: FC Loop: UUID of Group Name: N/A UUID of Remote/Peer Storage System: 6001-4380-05DE-D298-0000-5000-0002-0000 UUID of Data Replication Group: N/A UUID of Source Virtual Disk: N/A UUID of Destination Virtual Disk: 5000-1FE1-5024-482A-0000-0000-0000-0000 FRU Location(s): DRM Location: FC Loop: UUID of Group Name: N/A UUID of Remote/Peer Storage System: 6001-4380-05DE-D298-0000-5000-0002-0000 UUID of Data Replication Group: N/A UUID of Source Virtual Disk: N/A UUID of Destination Virtual Disk: 5000-1FE1-5024-482A-0000-0000-0000-0000 Evidence: Local Event Time : Thu 21 Oct 2010 09:38:58 GMT+01:00 Controller Report Time: 21-Oct-2010 09:35:32.464 EVA Log Description: 0c59690c: CAC=69 - Data Replication Manager State Change A Data Replication Manager state change occurred. Excessive out of order message rate on the inter site link is impacting replication throughput. Sequence Number: 30978 Rule#: Storage_Lx_DRM -- V 1.19 Event Code: (0C 59 69 0C) Full Corrective Action Statement: Check fabric switch settings and inter site link quality between the Data Replication Source Site controllers and the Data Replication Destination Site controllers. FC Loop: UUID of Group Name: N/A UUID of Remote/Peer Storage System: 6001-4380-05DE-D298-0000-5000-0002-0000 UUID of Data Replication Group: N/A UUID of Source Virtual Disk: N/A UUID of Destination Virtual Disk: 5000-1FE1-5024-482A-0000-0000-0000-0000 ---------------------------------------------------------- Related information: Number of like events in the last hour: 1 Affected Virtual Disk(s): Seq # Controller Report Time Local Event Time ~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 30978 21-Oct-2010 09:35:32.464 Thu 21 Oct 2010 09:38:58 GMT+01:00 CUST: CA warning: UKSOEVA02: Excessive out of order message rate on the inter site link. FC Loop: UUID of Group Name: N/A UUID of Remote/Peer Storage System: 6001-4380-05DE-D298-0000-5000-0002-0000 UUID of Data Replication Group: N/A UUID of Source Virtual Disk: N/A UUID of Destination Virtual Disk: 5000-1FE1-5024-482A-0000-0000-0000-0000 Notifications: Customer,Script,Browser,Console Analysis Mode: Automatic SEA Version: System Event Analyzer for Windows V6.1 (Build 136) WCC Version: Web-Based Enterprise Services Common Components for Windows V6.1 (Build 136), member of Web-Based Enterprise Services Suite for Windows V6.1 (Build 136)