- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- Infiniband Part 2 !
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-07-2010 12:31 AM
тАО10-07-2010 12:31 AM
Infiniband Part 2 !
If I start a subnet manager on one of then I can ping them both so happy with that part !
My issue is that I have to connect them into another switch in an exadata rack.
So from the HP BLc 4X QDR IB Switch in the enclosure I have cables connected to the switch in the Exadata.
These do not show an active link at either end
Do I have to enable something on the switch for it to become active ?
This is the first time dealing with IB and also blade enclosures so sorry if this is a noddy question !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-08-2010 06:15 AM
тАО10-08-2010 06:15 AM
Re: Infiniband Part 2 !
My IB connections did not come ready/active link until I loaded the drivers and configured the interfaces.
e.g. either manually looking at the lights or looking at the switch port status, there was no link lights until I configured the HCA from the OS.
you can review the HCA status by either using your ibstat diags or by cat /sys/class/infiniband/mlx4_0/ports/*/state
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-13-2010 02:26 AM
тАО10-13-2010 02:26 AM
Re: Infiniband Part 2 !
Sorry havent got back sooner!
Found issue was that when someone built the hardware for me they decided to put the cables in the switch upside down!
Ok so next problem !!
When I start up the IB on blade servers it causes the subnet manager to die on the switch with a mem segfault - but can run the subnet manager on one of the servers and it is all ok ?
Would ideally like to sort this out
Problem started when I moved cables from the blade switch to other switches in the exadata for resilience
I have powered off/on all the switches involved
Now it is getting annoying !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-13-2010 09:55 AM
тАО10-13-2010 09:55 AM
Re: Infiniband Part 2 !
I initially was testing using a server as the subnet manager. I then realized that if the server went down the whole IB network would stop.. not sure why anyone would want that.
so I have my switches be the subnet manager ( voltaire 2046s. and do not let the server(s) start a subnet manager. disable with chkconfig opensm (i think)
I would not think that starting a SM on a server connected to switches that already have a master / slave SM running would effect them but you never know.
these are just thoughts.. and may only lead you to a solution..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-13-2010 10:04 AM
тАО10-13-2010 10:04 AM
Re: Infiniband Part 2 !
Is this the switch in the Exadata rack, or the HP switch?
Regardless, perhaps a tangent, but in my opinion, if indeed the subnet manager on the switch dies with a mem segfault, you should go ahead and exercise your support contract and get a defect filed. Of course, the first question out of the support folks will probably be to ask if the switch is running the latest bits...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-14-2010 12:39 AM
тАО10-14-2010 12:39 AM
Re: Infiniband Part 2 !
I dont start the SM on the server it is as soon as IB runs on the blades server it kills the SM on all 3 of the exadata switches so IB goes down so have to run it up on server to get the fabric up.
If I take the IB down on the 2 blade servers the SM on the switches springs to life.
So far I have tried 3 versions on ib on the blades and all of them cause the issue so at least it is consistent.
Rick
Dont think it is an issue with exadata unfortuantely as it works ok when the blades are not running IB
Problem seems to have happened after I moved 2 of the blade cables from one of the exadata switches to put 1 in each of the other 2 switches in the exadata.
Is there some sort of arp/routing table that needs resetting on the switch ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-14-2010 09:11 AM
тАО10-14-2010 09:11 AM
Re: Infiniband Part 2 !
That said, I could see where say dueling subnet managers might cause one or another to decide to take themselves offline, but I see that as being very different from going down with a segfault.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-14-2010 09:11 AM
тАО10-14-2010 09:11 AM
Re: Infiniband Part 2 !
maybe check out the doc on the switches and exadata config ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-22-2010 12:17 AM
тАО10-22-2010 12:17 AM
Re: Infiniband Part 2 !
found this util on one of the exadata servers - /opt/oracle.SupportTools/ibdiagtools/verify-topology
when I ran this it showed an error that I had 2 external switches connected together ie the blade switch and the spine switch in the exadata
I disconnected this connection and hey presto SM on the switches now working !!
Thanks for all your input
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-22-2010 09:51 AM
тАО10-22-2010 09:51 AM