network connectivity lost after adding two new FlexLOM connections

[tvtue]

Hello Community,

I hope I can describe the szenario detailed enough out of my mind. This keeps me busy in my head even after work hours. :\

We have a c7000 enclosure with 16 G9 Blade servers. Each of them having two FlexLOM adapters. The enclosure holds two FlexFabric 20/40 VC switches. I configured an active/active uplink configuration with the typical *_A and *_B networks for each VLAN using OneView 2.0. The blades have RHEV-H (Red Hat Enterprise Virtualization Hypervisor) installed as OS and the network interfaces are bonded (802.3ad). Everything was working so far as I could see.

Then I got the task to connect an iscsi storage box to the enclosure. The iscsi box is a P4300 with one dual port emulex 10 GbE card. 

First thought: In order to keep redundancy, I could connect each VC switch at a time with one port on the iscsi box and configure LACP over those two ports. Then add another VLAN to the network set, configure ip addresses and done. But then I read that LACP is not recommended for ISCSI storage connections in virtual environments like RHEV. Instead one should use multipath. 

Ok, thus next thought: I could add two VLANs on the bond, one for each connection. etc. After configuring that I had strange behaviour when pinging from the hypervisor to the storage and vicy versa. Some addresses were pingable others won't. I tried figure out a pattern in the error, but I wasn't able to. My assumption was that bonding and multipathing together on the same interfaces doesn't work well, too.

Then I had the idea to add two new NICs in each blade to separate bonding from multipathing in hardware. When done with the changes in OneView, I lost network connectivity from the blades to the RHEV management server, which is located out of the enclosure on a different hardware.

Only a subset of the 16 blades shall have that iscsi connection. Surpisingly those blades that became new network cards can ping each other, but not the other blades and also not out of the bladecenter.

The mac addresses haven't changed and the bond in the RHEV-H is still configured and up.

If I remove the new network connections again, the connectivy stays lost. 

I even reinstalled one blade freshly, but it keeps the same way. 

Any idea where I went wrong here? Any hints are greatly apreciated.

Regards

Timo

[ChrisLynch]

LACP, Aka Mode 4, is not supported on BladeSystem where Virtual Connect is present on the downlinks only. You must change the mode to 1, 2 or 5. This has been case, and documented.

[tvtue]

Dear Chris,

thank you very much for your reply! I will try another bonding mode on monday when I am back at work. But I wonder why the other blades are working with mode 4. 

Can you point me to that documentation that you mentioned?

And is that case publicly available or at least to HP customers?

Best regards

Timo

[tvtue]

Hello Chris, 

now I remember that I even removed the bonding configuration and set up only one network interface with the required VLAN for communication with the management server. Also that didn't work. This gives me the impression that the problem is more related to the action of "adding another nic in oneview". 

However I will - as I already said - definetely try another bonding mode. 

Thanks!

Timo

[tvtue]

Hi, 

here is a follow to my problem. I found a HPE webpage which you probably were refering to. It states the bonding modes that are supported by Virtual Connect.

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0129462&sp4ts.oid=4144084

Modes 1, 5 or 6 are supported. 

RHEV on the other hand supports modes 0 to 4. Thus the poor intersecting set is mode "1".

I changed the bonding mode but keeps not working. :( 

I am still wondering why I can ping the one blade that I also changed (not the recent mode change done) whereas I can't ping the others.

Any further hints?

Best regards

Timo