Server with more than 2 FC-HBAs - LoadBalance/Presentation of LUNs from an Array

[Alzhy]

Those that have large server configurations and more than 1 pair of FC-HBAs (or more than 2 FC-HBA's) - do you present/access your LUNS accross however many HBAs or do you simply dual path each LUN and divvy up the LUNS accross however many pairs of FC-HBAs?

As I am preparing to cutover big servers (8-HBAs) to an XP12K - I am faced with a dilemma whether to present each LUN accross all 8 HBAs hence having 8 paths for each LUN! Or simply use a pair for each LUN.

For example:
I have 80 LUNs to a server that has 8 HBAs. On an EVA, the traditional practice in our site was to preent each LUN to each HBA - so each LUN has 8 paths. Would it not be efficient on an XP to instead present 20 LUNs each to each pair of HBA's? Or it really does not matter?

Thanks for any advice.

[RAC_1]

If I understood your posting correctly, you have 8 HBAs. It means that a pv has 8 alternate paths. without a software like secure path, you can just make use of it as alternate paths and not load balancing in it's real term.

Say for a LUN, you have three pvs.

Now, I will set primary path different for all three PVs going through three different HBAs and alternate paths to another three HBAs

Anil

[Alzhy]

I plan to use both VxVM DMP (active/active) as well as SecurePath for XP (actually AutoPath -- which is also Active/Active pathing).

So is there an advantage to say (in my example) multipath each of the 80 LUNs to all 8 FC-HBAs?

Or should I reap better results if I say simply use a pair of FC-HBas (4 of them in this case) and present/dual-path each LUN to each pair (20 to each pair) ?

One advantage that I can see with the latter is I may be able to isolate FC-PATHs/LUNS say for really heavy hitting volumes/filesystems -- ie. redo logs, temp tablespaces.. etc...

[Geoff Wild]

Unless you have software like EMC Powerpath - load balancing will have to be manual...

One way to do it, is after you create your vgs, reduce a path - then vgextend it back in....the primary path will now be the next path....

Example:

Say I have 4 paths:

PV Name /dev/dsk/c60t0d5
PV Name /dev/dsk/c62t0d5 Alternate Link
PV Name /dev/dsk/c56t0d5 Alternate Link
PV Name /dev/dsk/c58t0d5 Alternate Link
PV Status available
Total PE 4314
Free PE 0
Autoswitch On

PV Name /dev/dsk/c60t0d7
PV Name /dev/dsk/c62t0d7 Alternate Link
PV Name /dev/dsk/c56t0d7 Alternate Link
PV Name /dev/dsk/c58t0d7 Alternate Link
PV Status available
Total PE 4314
Free PE 0
Autoswitch On



I could reduce /dev/dsk/c60t0d5 then extend it back in:


--- Physical volumes ---
PV Name /dev/dsk/c60t0d5
PV Name /dev/dsk/c62t0d5 Alternate Link
PV Name /dev/dsk/c56t0d5 Alternate Link
PV Name /dev/dsk/c58t0d5 Alternate Link
PV Status available
Total PE 4314
Free PE 0
Autoswitch On

PV Name /dev/dsk/c62t0d7
PV Name /dev/dsk/c56t0d7 Alternate Link
PV Name /dev/dsk/c58t0d7 Alternate Link
PV Name /dev/dsk/c60t0d7 Alternate Link
PV Status available
Total PE 4314
Free PE 0
Autoswitch On


A lot of what you want to do depends on number of vg's - if you have 1 big one - then put all the paths in it....if you have multiple vg's - then you might want to split them up - across say pairs of hba's...

Rgds...Geoff

[Florian Heigl (new acc)]

the maximum we see on our hosts would be 4 paths and we use all of them, interleaving the primary path to achieve a bit of loadbalancing.

with eight paths, I'd definitely do it the same way but built some scripts around it to avoid human errors (i.e. one device having 5, one 8, and one 7, with the primarys overloading only the first three), as this might have You end up with hundred or more disk paths.
also emphasize deleting old device files
(showing up as "at ???" in lssf /dev/dsk/ctd...) to prevent false alarms from EMS.

And actually I'd stop using pvlinks in favour of a more transparent solution like powerpath, where a simple command will supply a good overview of the link states.

[RAC_1]

If you are going to use VxVM DMP and secure, then certainly go for all 8 HBAs, in effect you are distributing the i/os over the 8 HBAs, which is cerainly good.

I hope there is no restriction on no.of paths over which you can load balance in DMP or with secure path. And even if it is there, I do not think it would be that low.

Anil

[TwoProc]

Nelson,
Strictly from a maintenance perspective - I feel that 8 is too many to keep alive to each LUN. I've got the same setup as you do - but have elected to follow the logic that I'd like one connection to each ACP in the XP *per LUN* (we still use all 8 connections considering all LUNs). I make sure that when creating a VG, I've got a nice rotation of the HBA's and the SAN switch ports too (rotate the paths on your switch - or at least make them one-to-one with your HBA's). That way, when the LV's are created you can see that there is a multiple path (if you SAN switch is handled logically as one-to-one to the HBAs) rotation of hardware as you progress down the extent list. Keep in mind that each LUN with have 4 different HBA of it's down, but since you've got many LUNS, you should end up with all of the HBAs in play in each lvol of each vg - so that you'd be hitting a different hba/sanport in/sanport out/CL in path/ACP cpu for each next grab of data...

Of course, the LUNs I've built work the same way, I allocate space for LUNS in a circular queue - one from ACP0, then next from ACP1, the next from ACP2, then next from ACP3, then next from ACP0... etc...
Actually, it's a quite a lot of work to get this all done...

If you rotate and match HBA to SAN port to CLI to ACP to LUN in rotation - You'll end up with a nice, redundant and well-balanced I/O load on the XP. Be aware that from what I'm told - if you've got AutoPath - that all this work is not necessary. Still, even if you have AutoPath - I like the idea of having an alternate path to each CLI following on to each different ACP (4 ACPs is the max - but you can certainly have less which might mean you want a different rotation method).

Anyway, what this method above should do is give lots of available hardware width to get I/O transfers done, and I believe is a better solution than just two. Of course, following the same logic, one could say the same of using 8 ports, or even reducing to two and spreading your I/O using other balancing methods.

[Alzhy]

Messr Joubert:

For my VxVM 3.5 DMP systems, I actually use an ASL (array support library) that removes the manual work of picking and ensuring my VxVM volumes get built so the component LUNS are taken cosnidering the arrar's innards -- ACP, array group, CHiP presentation and HBA paths.


What I am am simply unsure is the wisdom and necessity and performance need of presenting each LDEV/LUN to 8 front-ends (ChiP port) and to 8 HBAs. Am I better off presenting each LDEV/LUN to pairs of ChiP ports and hence to pairs of HBAs - so 80 LDevs will mean 20 Luns presented to each pair of 4.

What do you think?

In an AutoPath environment which does active/active as well - I think the same argument applies.

[TwoProc]

OK, so to put simply - faced with the same issue - I elected to present 4 paths to each LUN - it matched the hardware I had for a round-robin path allocation strategy.

I like the idea of two per LUN for administrative simplicity - however, I use four to get an edge in performance. I think managing 8 is too many, however, if you've no problem managing that many - than I would use all 8.

This one (for me) would follow the Cajun saying
"De More Dat - De More Better", but its FAR (oh so far) from the last word on things - it's just how I felt I could best utilize the resources I have.

[Tim D Fulford]

I think you are possibly looking at this upside down... loadsharing your luns over all 8 HBA's will mean that you are unlikely to suffer a HBA/port bottleneck due to throughput/bandwidth.

If you are sure that there in no chance of a bottleneck happening at the HBA/ports with 2, or 4 ports then you can use that number; though I would keep it an even number.

I for one would probably not opt for the latter as you are more likly to wish that you load shred the LUNs over more ports than over less.... but that is my opionion...

Regards

Tim