X4 InfiniBand Interconnect - Support for linked enclosures

[chuckk281]

Norman had an InfiniBand interconnect question:

*****************

First Question:

Customer is inquiring as to whether two c7000 enclosures can be linked via their respective X4 IB interconnects to permit traffic between all 32 nodes.

Hardware:

2x           c7000 enclosures

32x         BL460c G7 servers (serving as compute nodes)

2x           QDR (X4) IB Interconnects (one per enclosure in bay 5/6)

Second question:

Are there any special considerations, thoughts or caveats when using IB enclosure-to-enclosure communication?

(I would suspect that this would work, but also may require utilizing one blade to support the management of the IB network infrastructure.)

Follow on question:

Is there a practical or theoretical maximum number of c7000 enclosures that can be joined without using an external IB switch?

(I expect that multiple enclosures could be supported, but may degrade IB network depending on number of “hops” to reach other nodes.)

****************

Randy replied:

**************

1) Yes you can connect two enclosures back to back.

2) The primary consideration is that the blade enclosure IB switches are dumb so you need to make sure that you include a software Fabric/Subnet manager.  It will have to run on one of the nodes to keep track of the routing table. This is not the most scalable way to do things as you will be limited to how many enclosures you can connect this way. It could also lead to odd non-deterministic performance depending on the connections.

3) The external switches are not that expensive ~$13k list. I would just sell the external switch and be done with it. The fabric management will be handled in hardware, plus the scalability arguments go away. As the customer adds enclosures, depending upon the required oversubscription, you simply add more 36-port switches and form a fat-tree topology for the IB fabric that will handle virtually any number of nodes they will need.

*********

Any other comments or questions?