topic Re: C7000, Bl860c,Infiniband in BladeSystem - General

C7000, Bl860c,Infiniband

Thanos Demetropoulos_1 — Thu, 29 Nov 2007 06:16:59 GMT

The Blade node BL860c Mezz. Card (P/N: 409377-B21) cannot connect with the Infiniband Module with P/N:410398-B21 .The ports in Module are inactive. OS is Redhat ES v4.5. I Cannot find
any sw manual or configuration pdf for Infiniband module
We have installed the same Module with BL460c nodes without problem. Please help

Re: C7000, Bl860c,Infiniband

Víctor Cespón — Thu, 29 Nov 2007 07:03:29 GMT

The truth is that the information on web and Quickspecs is confusing. The web gives the part number 409377-B21, but in quickspecs, only appear:

HP 4X DDR IB Dual-port Mezzanine HCA for HP BladeSystem c-Class 448262-B21
HP 4X DDR IB Mezzanine HCA for HP BladeSystem c-Class (single-port) 410533-B21

But http://h18000.www1.hp.com/products/blades/components/infiniband/questionsanswers.html says that this mezzanine is compatible with all blade models.

Re: C7000, Bl860c,Infiniband

James ~ Happy Dude — Thu, 29 Nov 2007 07:26:15 GMT

There is no Confusion.

InfiniBand Mezzanine Options:
HP 4X DDR IB Mezzanine HCA (no Software) 410533-B21
HP 4X DDR IB Mezzanine HCA (with Voltaire Software) 409377-B21
HP 4X DDR IB Mezzanine HCA (with Cisco Software) 438670-B21
As Mentioned at:
http://h18004.www1.hp.com/products/quickspecs/12671_div/12671_div.html

HP 4X DDR IB Switch Module for HP BladeSystem c-Class 410398-B21
As mentioned at:
h18004.www1.hp.com/products/quickspecs/12586_na/12586_na.pdf

Perhaps We are Forgetting that the Switch Module that you are trying to use is For HP Bladesystem C-Class. & BL860c is an HP Integrity BL860c Server Blade.

There is an Architecture Difference & this Combination will not work.

http://h18004.www1.hp.com/products/quickspecs/12586_div/12586_div.html

;) Regards.

PS: If You get this to work, Please let me know the secret trick !!

Re: C7000, Bl860c,Infiniband

Víctor Cespón — Thu, 29 Nov 2007 08:51:39 GMT

Hi, but BL860c is a c-Class blade, and the HP 4X DDR IB Switch Module is for c-Class enclosures.

Re: C7000, Bl860c,Infiniband

James ~ Happy Dude — Thu, 29 Nov 2007 08:58:15 GMT

No vcespon,

BL860c is an INTEGRITY server - Integrity blade; Which is NOT the same category as other Blade c-class servers. As I mentioned, Architecture is different. Hence this combination will not work, as far as I know.

Again, if you can make this work, let me know. I will learn something new too.

;) Regards.

Re: C7000, Bl860c,Infiniband

rick jones — Fri, 30 Nov 2007 13:02:38 GMT

While in broad handwaving terms, PCIe is PCIe, unless there is an explicit statement of support for that card in an Integrity blade, don't just assume that being a c-class means it is "supported." Now, it could be the case that it might "work" but it is important to remember that "works" and "supported" mean different things in HPspeak:

Supported, known to work -> warm fuzzies all around
Supported, not known to not work -> an HPite may be in trouble
Supported, known to not work -> an HPite is in trouble
Unsupported, known to work -> lucky today, unlucky tomorrow?
Unsupported, not known to not work -> there but for the grace of Turing
Unsupported, known to not work -> no, it was not deliberate ;-)

I myself have tried to take some ProLiant Blade mezzanine Gigabit NICs and use them in an Integrity Blade. None were/are "supported" on the Integrity Blade and I only got one to "work" - the other two prevented the BL860c from even powering-on fully...but I digress.

Now the usual litany of questions, keeping in mind that what you are trying to do may indeed be "unsupported":

If you have installed the mezz card and the system boots, does it appear in the output of lspci?

Is there a driver installed on the RHEL4.5 bits on your BL860c for that mezz card?

Are you going to be doing "native" IB stuff, or IPoIB? I'm guessing you are just using what is in RH4.5 rather than grabbing later OFED bits? (Which would again take us into the realm of works vs supported, this time including RedHat I suspect :)

I just checked http://www.hp.com/go/bl860c (any integrity system will have a URL like that to get to online stuff for it). Then going to the quickspecs, it suggests there is a 4X IB card supported on the BL860c. However, I have yet to find the product number - for all I know it could be the one you have, it could be something else. Ah wait, down towards the bottom of the quick-specs - a different product number - 410533-B21 for the bare HCA, and two others for either Votaire or Cisco SW and a link about linux "support."

Re: C7000, Bl860c,Infiniband

rick jones — Tue, 04 Dec 2007 21:28:46 GMT

Thanos - I have gotten in contact with some of the IB people in HP HPC. Their response is encouraging, of course they have a litany of questions:

1. What slot is the Mezz card installed
2. Which bay is the switch installed
3. What is the OS
4. What drivers are installed
5. What version of OA software is running
6. What is the blade switch connected to?
7. What SM is running on the cluster? (Vendor, type and version)
8. What is the output of #lspci
9. Assuming OFED drivers, What is the output of #mstflint -d XX.00.0 query
10. What is the output of ibstat

I think some of that is boilerplate already covered in the thread (eg OS), but feel free to repeat it.