C Class enclosure interconnect slots

[S Petersen]

Greetings,

I found that we have been given some Cisco 9124e switches so I thought I should give them a go. We already have SAN pass through modules in interconnect 3 and 4. I read some vague document that suggested that you could have a mixture of SAN Pass through modules, 9124e switches and Virtual Connect modules and they could all work..

So, I put the 9124e's into interconnect 7 and 8 in one enclosure with little in it and nothing untowards happened. I then found out that the particular enclosure has a network problem and I needed to move the 9124e's to another enclosure in interconnect 7 an 8. As soon as they went in, a 680C complained about a pot mismatch. So I moved them to interconnect 5 and 6 and another server complained about a mismatch.

Some of the system log says.

A port mismatch was found with server blade bay 8 and interconnect bay 6.
...

Blade in bay #8 status changed from OK to Degraded.
Mismatching I/O was detected on Blade 8, Mezz card 2, Port 3.

Once I withdrew the two 9124e's all seems to be ok again. The actual servers did not notice anything that I can see.

The dumb thing about this is that Mezz card 2 does not have anything in it. However, looking at the Graphical View, there is a tick in the Mezz slot 2. The ports that are not even there in my opinion say they are using the bays 5, 6, 7 and 8.

The other blade in another slot did the same thing. It complained about a mezz card that is not there.

What could be the problem? Is the solution just unticking a box in Port Mapping?

Regards

Stephen

[JKytsi]

Do You know the c-class port mapping basics ?

What mezzazine cards You have in the server and in which slots ?

[S Petersen]

---- Do You know the c-class port mapping basics ?

I expect that by that you mean that the onboard nic's go to interconnect bays 1 and 2.

Mezz card 1 goes t interconnect 3 and 4. That leaves mezz card 2 going to 5 and 6 adn 7 and 8.

---- What mezzazine cards You have in the server and in which slots ?

All our servers have mezz cards in slot 1. Generally for normal usage, we only have interconnect bays 1, 2, 3 and 4 in use.

I had a look at another of the blades in the enclosure that did not complain. In OA, it clearly states that Mezz slot 2 has no card present. The two blades that complained look different in OA. The Mezz Slot 2 "tick Box" is ticked and the grayed out ports are mapped to the right bays. What I did not realise is that two of the blades actually have cards there. Table view shows very clearly that Mezz slot 2 has some quad port 1 Gb NICs.

This changes my plans a lot. The servers that complained are Hyper V and obviously need these extra NICs.

Thanks for making me look in the right spot.

Stephen

[Adrian Clint]

The 9124 SAN switches in a c7000 enclosure in interconnects 7&8 will only connect to full height blades if you have a HBA in mezz3 and a 2 port NIC in mezz2.

They will not connect to half height blades at all.

[S Petersen]

--- The 9124 SAN switches in a c7000 enclosure in interconnects 7&8 will only connect to full height blades if you have a HBA in mezz3 and a 2 port NIC in mezz2.

--- They will not connect to half height blades at all.

Going by the diagram for port mapping of the half-height servers, Mezz 2 gives an indication that it can connect to 7/8. The OA admin guide shows that a Mezz 2 that uses a PCIe x8 (interesting) slot can address interconnect 5 and 6 as well as 7 and 8? The full height blades are even more interesting.

My main error was that as there is nothing physically in the interconnect bays 5 and 6, I assumed they were not being used. Someone has put NC326m Dual Port 1 Gb NIC for c-Class Bladesystem cards in some of the servers. They use interconnect 5 and 6 (half height and 7 and 8 full height) even though there is nothing can be seen in the interconnect bays.

Anyway, I have learnt a valuable lesson on this and it changes our architecture design for SAN a lot. VC bandwidth for virtualisation systems just can't offer the bandwidth especially in a full C Class running hot little 490's. This was not so much about the 9124e, it was about using the interconnects.

[JKytsi]

We have two enclosures full of 490 servers and we don't see any problems with VC.

You need to have quad-port mezz card to reach interconnect bays 7/8 from half-height machine

[The Brit]

As Jarkko (and the c7000 wiring diagram indicates),

The only way a halfheight blade can make use of IC Bays 7 and 8, is if there is a quad port NIC in MEZZ 2. In which case NIC port 3 maps to Bay 7 and NIC port 4 maps to Bay 8.

Dave.

[S Petersen]

Quad port mezz card?

Are you satisfied with the VC performance with so many systems using it? VC with only 4 x 4 Gpbs is not what I would think enough for heavy usage. I admit that its probably unlikely that all 16 servers would be pushing past 1 Gbps at the same time but I don't want that hassle if I can design something else.

VC looks great and if they could use 8 x 4 Gpbs FC ports (or even 4 x 8 Gpbs), it would sell me. However our network people would rather loose a limb than use VC. I don't know why and it gives me some impression its a loss of "individuality".

[S Petersen]

Hi Dave,

--- The only way a halfheight blade can make use of IC Bays 7 and 8, is if there is a quad port NIC in MEZZ 2. In which case NIC port 3 maps to Bay 7 and NIC port 4 maps to Bay 8

Are you saying that in that scenerio, the card in Mezz 1 would see the 9124e in interconnect 5 and 6?

I have a 460 and a 680 in this rack that first identified me of this problem. When I put the 9124e into interconnect 7 and 8, the 680 complained. It has a NC325m Quad port NIC in Mezz slot 2 and it uses interconnects 5,6,7 and 8.

I then put the 9124e into interconnect 5 and 6 and the 460 also complained. It has a NC326 Dual port NIC in Mezz slot 2 and it uses interconnect 5 and 6.

I guess the two systems made for an interesting day. I just don't have a 460 with a Quad port NIC in Mezz slot 2 to see what happens.

I really have to reread those manuals.

Thanks for our help.

Stephen

[JKytsi]

well ...we have 8Gb VC FC modules (8Gb FC HBAs and 8Gb director core switches)but so far using only 1 uplink per module and not seeing any performance problems. But that of course depends of usage (we are running ESX vSphere)