Virtual Connects are unable to access OAs: "Would you like to re-authenticate"

[jmir]

HP c7000 enclosures, Onboard administrators, administrator tray. Using only CLI access to VCs, ssh into those works fine.

VCs are unable to connect to the OA. There are 2 enclosures and the VCs on either one are unable to access their OAs. On both VCs, we get a note that the VC is unable to connect to the OA using the configured address from earlier (?) . Then next we get a string query first asking to re-authenticate:

     "Would you like to re-authenticate to this OA" Yes/No

If answer Yes, next it asks for an IP address for the OA to get to it. We enter the correct address of the Active OA of that enclosure, and request re-authentication. Depending on which VC, the response is either:

1) "the object is already in use" [sorry didn't catch the exact wording of this error]

2) "ERROR: Error enclosure serial number mismatch"

We have tried reseating OA1/OA2 on each enclosure, along with the tray.  Tried this with only 1 OA in the slot and leaving the other OA out. Tried failing over between active/standby VCs. We have tried almost everything except actually restarting the VCs, which is service-impacting and requires moving traffic. Verified logins are as Administrator and correct credentials.

Anything else to check or other ideas before we impact the network traffic?

Traffic is actually running fine, it's just that the VC cannot communicate with OAs.

------------------------------------------------------------

HPE Virtual Connect Management CLI v4.63

Build: 4.63-8 (r327000) Aug 21 2018 13:59:19

(C) Copyright 2006-2018 Hewlett Packard Enterprise Development LP

All Rights Reserved

[feigenL]

Hello there, any recent hardware changes?

maybe any OA module got replaced? Any recent firmware upgrade on either the OA's or the VC's?

Are you able to access, from the OA interface the VC modules? what about the iLO's?

Have you make sure that the VC's IP addressess are asgined by EBIPA? Can you double check that the network settings, such as netmask, gateway, DNS are correct?

If there are 2 VC modules on both enclosures, there shouldn't be a problem to reseat them, just doing it one by one.

Check the following documents/advisories with issues that somehow are similar to the issue reported by you:

https://support.hpe.com/hpesc/public/docDisplay?docId=c02720395

https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0105502

Regards

[jmir]

@feigenL Thank you for the tips regarding this issue. We have looked into these extensively and also had a lengthy troubleshooting session with a hardware vendor about this that included failing over OAs, removing and reseating both OAs and removing / reseating the OA tray also. The issue still exists and has been escalated with vendor. After these troubleshooting events, the status of "show interconnect" command for the enclosure returns encl:1 and encl:2 as "Unknown". 

Some answers to your questions below.

Any recent hardware changes? - none that we are aware of.

maybe any OA module got replaced? - No, There were blades replaced within the chassis but not OA.

Any recent firmware upgrade on either the OA's or the VC's? - no, we are locked into the firmware version per customer, legacy system and we are not allowed to update firmware.

Are you able to access, from the OA interface the VC modules? - tried logging into the OA from the VC. On the primary VC in first enclosure, get error "object is in use"; on primary VC in second enclosure, get error "Error enclosure serial number mismatch" . Failing between active/standby VCs and retesting shows same results.

what about the iLO's? - We can get to the OA no problem. We can get to the iLO no problem. On the GUI the OA populates the VC entry. Due to lack of license, we are not able to access VCs from GUI and CLI is the only means available. The VC and the OA are both working fine oherwise, traffic is running on the chassis - they are just not talking to each other at all.

Have you make sure that the VC's IP addressess are asgined by EBIPA? - all addresses are correct per premises specs.

Can you double check that the network settings, such as netmask, gateway, DNS are correct? - all addresses correct per specs.

If there are 2 VC modules on both enclosures, there shouldn't be a problem to reseat them, just doing it one by one. - Because VC reseating or reboot is considered service-impacting, this has not been done yet and requires another maintenance window and some approvals. This is a hard sell when the traffic is running without issues and everything else appears to be okay.

We will look over those links but since Firmware update is not an option, this rules many things out.

[jmir]

@feigenL 

After recent extended troubleshooting, we are still getting errors on this system, only after logging in it is now always either "Error enclosure serial number mismatch" or "The specified object is in use and the requested operation cannot be completed.". Tried restarting VCS but this did not help. Issue being escalated to our vendor as they have been unable to tell us what these errors mean. We played with the OAs a lot and tried everything the vendor engineers, including using vcsu tool to get everything connected to both VCs. These two VCs are still not in sync.

Recall that this is a situation with 2 stacked enclosures. We have seen that most community activity is from members familiar with 1 enclosure setups, but this is a little different and we are unable to find much expertise in this area. 

The latest response from the vendor is as follows. We already verified matching credentials and health check passed. 

-----------

The second enclosure is not being authenticated correctly.   

The trick is that there must be both an OA Admin and VC Admin with matching creditentials to log into both enclosures and VCs.  Now the OA and VC user names don’t have to match, they just need to match the other enclosure:

Ie enc0 OA_Admin/password1   

Enc1 OA_Admin/password1    

Enc0 VC_Admin/password2   

Enc1 VC_Admin/password2 

So if we run the following on each enclosure, getting the same type of output should show us that the created Admin accounts have all the necessary permissions for both the OA and the VC.

vcsu -a collect -i <OAIP> -u <OA_admin> -p <OA_pw> -vcu <VC User> -vcp <vcm pass> 

Then once the admin permissions match, do a healhcheck.

vcsu -a healthcheck -i <primary OA> -u <user> -p <pwd> -vcu <vcm user> -vcp <vcm pass> 

If this healthcheck still shows the subordinate modules out of sync, then we will need to reset the vcm.  NOTE:   This will impact data access, do this in a maintenance window:

vcsu -a resetvcm -i OA211_IP -u Administrator -p password 

If that reset does not work, a physical reseat would be next and then replacement of the VC that does not reset properly.

[Vinky_99]

Hello @jmir,

It sounds like the issue could be related to authentication and credentials, specifically with the second enclosure not being authenticated correctly. Based on the vendor's response, it appears that both the OA and VC Admin accounts must have matching credentials on both enclosures for proper authentication.

The vcsu tool can be used to collect information and verify that the created Admin accounts have all the necessary permissions for both the OA and VC. After verifying the permissions match, a health check can be run to confirm if the subordinate modules are in sync. If the subordinate modules are still out of sync, resetting the vcm may be necessary, but this will impact data access, so it should be done during a maintenance window.

If resetting the vcm does not work, a physical reseat or replacement of the VC that does not reset properly may be needed.

It's important to note that this is a complex issue and the vendor may need to provide further assistance.,