Add Node to Management Error

[IWem812]

Hello,

We have an existing cluster of (2) P4500 G2 system running LHOS 12.5, with FOM running on an ESXi server with local storage.  We recently acquired a 3rd node with matching specs.

The problem I've encountered is that when I attempted to add the new node to the existing management group, I received an error and it resulted in the attached screenshot.  The status of the 3rd node is "Joining/leaving management group, Storage system state missing".  The 3rd node isn't running a manager.

If I attempt to log into the 3rd node, the CMC is unable to do so.  If I attempt to log into the 3rd node via the console, I'm prompted for a user and password, but it doesn't accept the mgmt group's username and password.  I did, however, find that admin/admin works.

If I right-click on the node in the CMC and select 'Remove from Management Group' I initially received an error stating that I couldn't do so because the management group was running a FOM.  I believe that I now receive an error indicating that it is unable to login.

I think my only option at this point is to use the admin/admin credentials to remove the node from the management group.  However, I've never used that option, so I'm unclear on the potential impact.  This issue never arose in the lab, so I'm a little gun-shy about trying it on a production system.

Any advice?

[IWem812]

Sorry, forgot the screenshot.  Attached now.

My suspicion is that the use of a split network on the 3rd node is partially to blame. (192.168.170.x for mgmt, 192.168.160.x for iSCSI and LHOS).  All 3 nodes had 10 Gb cards and addtional memory installed over the last week, but only the third node is utilizing the 10Gb ports.  The plan was to use the built-in ports in a bond (bond0) for management and the new 10 Gb ports in a bond (bond1) for LHOS and iSCSI.  If I'm able to resolve this issue, I think I'll move all of the operations to the 10 Gb bond and abandon the idea of using the built-in ports, as that doesn't seem to be fully baked in 12.5 at this point.

Anything suggestions or thoughts would be appreciated.

[BenLoveday]

Hey there,

I would say it is your split networking definitely. We've seen this before where we've incorrectly set the wrong nic to be LHOS and the other nodes are unable to talk to it. 

If you can get back into the new node, make sure the LHOS connection is on the same network that the other nodes are using and you should be fine.

Also might pay to check that the new node (and existing ones) are fully patched or at least at the same patch level first too.

Cheers,

Ben

[BenLoveday]

I forgot to mention, you can always move the LHOS to the 10Gb ports afterwards once you have them all working together. Just be careful when changing the LHOS interface on each node as this is what does the inter-node comms and data replication.

Cheers,

Ben

[IWem812]

Ben,

Thanks for the feedback.  As it stands, I'm not able to manage the new node using the CMC.  I'm able to login to its console using admin/admin, so I'm thinking that my best option at this point is to remove the node from the management group via the console - it just makes me a little nervous =)

The new node doesn't appear in the CMC unless I add it to the Find list.  When I do, it tells me that the node believes that it is in the management group, but the management group doesn't agree.

[IWem812 · Accepted solution]

To follow-up on this, I decided to disable the network cards for the 3rd node and then used the console to remove the node from the management group.  That all went well., so I re-added the 3rd node to the CMC, which I used to remove the split-networking configuration.  After removing the split network, I was able to add the 3rd node to the management group.