"Enclosure Interlink" cabling question

[just_testing]

Hi all,

We have inherited some C7000s which I've never used and need some advice on.

At one site, we have 2 x chassis, setup as a "full stacking" domain.

From what I tell from the photos I have (it's at a remote site) it looks like the cable goes between the UPLINK "Enclosure Interlink" port on BOTH chassis.

From what I have read you should connect the uplink on the first chassis to the downlink on the second.

Will it work OK if cabled liked this?

I ask because in Virtual Connect Manager, the Domain Status shows as "Critical" and says that it cannot communicate with the modules in the second enclosure.

I do know that the VC modules in the second enclosure are on a newer version of the firmware (see healthcheck below)

So, is the problem is the downrev modules in the first enclosure, or the cabling between the two UPLINK ports?

Any thoughts or suggestions would be appreciated.

A healthcheck with vscu shows the difference in versions and the "Module Configuration" on the second chassis as "Not In Sync"

-------------------------------------------------------------------------------
Virtual Connect Domain Information
-------------------------------------------------------------------------------
Stacking Connection : OK
Stacking Redundancy : OK

-------------------------------------------------------------------------------
Bay 1 : HP VC FlexFabric 10Gb/24-Port Module
-------------------------------------------------------------------------------
Power : On
Health : Ok
IP Address : confidential information removed
IP Connectivity : Passed
Version : 4.30
Mode : Backup
Domain Configuration : In Sync
Module Configuration : In Sync

-------------------------------------------------------------------------------
Bay 2 : HP VC FlexFabric 10Gb/24-Port Module
-------------------------------------------------------------------------------
Power : On
Health : Ok
IP Address : 192.168.103.112
IP Connectivity : Passed
Version : 4.30
Mode : Primary
Domain Configuration : In Sync
Module Configuration : In Sync

-------------------------------------------------------------------------------
Bay 1 : HP VC Flex-10 Enet Module
-------------------------------------------------------------------------------
Power : On
Health : Ok
IP Address : confidential information removed
IP Connectivity : Passed
Version : 4.45
Mode : Backup
Domain Configuration : In Sync
Module Configuration : Not In Sync <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

-------------------------------------------------------------------------------
Bay 2 : HP VC Flex-10 Enet Module
-------------------------------------------------------------------------------
Power : On
Health : Ok
IP Address : confidential information removed
IP Connectivity : Passed
Version : 4.45
Mode : Primary
Domain Configuration : In Sync
Module Configuration : Not In Sync <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

[Kashyap02]

Please refer to the below guide which has different scenarios for multi enclosure VC stacking.

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c02102153

Page: 13 to 17.  This has the cable diagrams as well.

I recommend you to update the firmware of the VC modules which are running at 4.30. As we have seen a lot of issues on 4.30 and it has been removed from the website. I recommend to keep the firmware at 4.50 atleast.

As you have mentioned that the domain status is critical, can you please attach the screenshot of the error?

[just_testing]

Thanks for the response

I'll try to upgrade the 4.30 VC modules

I can't seem to upload the images to this site, but here is the equivalent from the vcsu report

VCSU 

*******************************************************************************
ENCLOSURE INFORMATION
*******************************************************************************
ID : enc0
Enclosure Name : ENC-01
Description : BladeSystem c7000 Enclosure G3
Import Status : Imported
Overall Status : OK
Part Number : 686611-B21
Asset Tag :
Num Device Bays : 16
Num I/O Bays : 8
Manufacturer : HP
Spare Part Number : 689229-001
Spare Rack Name : 103
Primary : true
OA1 IP Address : confidential information removed
OA2 IP Address : confidential information removed
OA1 IPv6 Address : -- --
OA2 IPv6 Address : -- --
Comm Status : OK
Operational Status : OK
OA Status : OK
Status Cause : Enclosure enc0 is normal
Root Cause :
-------------------------------------------------------------------------------
ID : enc1
Enclosure Name : ENC-02
Description : BladeSystem c7000 Enclosure G2
Import Status : Imported
Overall Status : Failed
Part Number : 507019-B21
Asset Tag :
Num Device Bays : 16
Num I/O Bays : 8
Manufacturer : HP
Spare Part Number : 519345-001
Spare Rack Name : 103
Primary : false
OA1 IP Address : confidential information removed
OA2 IP Address : confidential information removed
OA1 IPv6 Address : -- --
OA2 IPv6 Address : -- --
Comm Status : OK
Operational Status : Failed
OA Status : OK
Status Cause : All Ethernet modules are abnormal in enclosure enc1
Root Cause : Module enc1:iobay1 unable to communicate with enclosure
enc1 (powered off); Module enc1:iobay2 unable to
communicate with enclosure enc1 (powered off)

FIRST ENCLOSURE

->show version
HP Virtual Connect Management CLI v4.30
Build: 4.30-356 (r301774) Jul 8 2014 14:58:41
(C) Copyright 2006-2014 Hewlett-Packard Development Company, L.P.
All Rights Reserved

->show enclosure
==========================================================================
ID Name Import Status Serial Number Part Asset Tag
Number
==========================================================================
enc0        ENC-01 Imported xxxxxxxCJR 686611-B21
enc1         ENC-02 Imported xxxxxxxKLN 507019-B21

->show interconnect
==========================================================================
ID Enclosure Bay Type Product Name Role Status
==========================================================================
enc0:1 ENC-01 1 VC-ENET+ HP VC Standby OK
FC FlexFabric
10Gb/24-Port
Module
--------------------------------------------------------------------------
enc0:2 ENC-01 2 VC-ENET+ HP VC Primary OK
FC FlexFabric
10Gb/24-Port
Module
--------------------------------------------------------------------------
enc1:1 ENC-02 1 VC-ENET+ HP VC Subordinate No Comm
FC FlexFabric
10Gb/24-Port
Module
--------------------------------------------------------------------------
enc1:2 ENC-02 2 VC-ENET+ HP VC Subordinate No Comm
FC FlexFabric
10Gb/24-Port
Module
--------------------------------------------------------------------------
enc1:5 ENC-02 5 Unknown HP BLc -- -- Unknown
QLogic 4X
QDR IB Switch
--------------------------------------------------------------------------
enc1:7 ENC-02 7 Unknown HP BLc -- -- Unknown
QLogic 4X
QDR IB Switch
--------------------------------------------------------------------------

->

SECOND ENCLOSURE

->show version
HP Virtual Connect Management CLI v4.45
Build: 4.45-59 (r318084) Jul 20 2015 18:15:56
(C) Copyright 2006-2015 Hewlett-Packard Development Company, L.P.
All Rights Reserved

->show enclosure
==========================================================================
ID Name Import Status Serial Number Part Asset Tag
Number
==========================================================================
enc0        ENC-02 Imported xxxxxxxKLN 507019-B21

->show interconnect
=====================================================================
ID Enclosure Bay Type Product Name Role Status
=====================================================================
enc0:1 ENC-02 1 VC-ENET HP VC Standby OK
Flex-10 Enet
Module
---------------------------------------------------------------------
enc0:2 ENC-02 2 VC-ENET HP VC Primary OK
Flex-10 Enet
Module
---------------------------------------------------------------------
enc0:5 ENC-02 5 Unknown HP BLc -- -- Unknown
QLogic 4X
QDR IB Switch
---------------------------------------------------------------------
enc0:7 ENC-02 7 Unknown HP BLc -- -- Unknown
QLogic 4X
QDR IB Switch
---------------------------------------------------------------------

->

[Kashyap02]

Looking at the logs which you shared, there is a disconnection between enc0 (Primary) and enc1. this could be because of the firmware mismatch.

Try updating the firmware and check. If that fails. you can try

1. Physically re-seat the OA modules and OA tray on both enclosures.

- Remove OA 1 --> OA 2 --> OA tray.

- Install OA tray --> OA 2 --> OA 1

2. Physically reseat the VC modules ( 2nd enclosure)

As you do not have access to enclosure physically, please perofrm the EFUSE from OA CLI

Reference: https://support.hpe.com/hpsc/doc/public/display?docId=c00702815 (Page 208) - poweroff / poweron / restart.

If you still get the error, i recommend you to log a support case with HPE as this needs to be verified by collecting different logs.

[just_testing]

Thanks very much for the suggestions, I'll give them a try and report back.

[just_testing]

The forced upgrade changed the messages to indicate that we could not run a multi-enclosure setup with different VC module types

I think we need to run the same type Flex modules in both enclosures and not have HP VC FlexFabric 10Gb/24-Port Modules in one and HP VC Flex-10 Enet Modules in the other

For now, I have split them into two separate domains and all of the errors have cleared