XP12000/24000 autolun performance and issues

[Niksa Franceschi]

Hi,

our current environment is SAN with several UNIX boxes and two XP12000 storages, of which most UNIX boxes are Oracle database servers.
One of our main performance bottlenecks are CPU's on ports on XP12000 storage. We have even situations where 1 LUN (~70Gb size) can saturate CHIP port, as it goes above 1000 IOPS. When CPU load on that port goes above some 65-70% we can notice drop in performances.
Due to high loads, we do use several ports, but as load on some LUN's can go quite high we would like to use autopath.
Now, problem is that apparently autopath active-active type of load balancing is not recommended by HP (at least we are so told by our support).
We did try to use it, and when using 2 path for balancing (be it RRD or SQL type), we did not notice performance drops - but when doing so over 4 or more paths performance drops were significant (access times rocketed from 4-8ms to 15+ms).

So, my question is if anyone managed to use load balancing using autopath with XP12000, and have good performance, and how?
Is it or will it be properly supported by XP12k+ boxes?
This is quite significant issue, as CPU's on XP12k box are quite weak for our circumstances.

Also, in approx 1 year we might move to XP24000 box, but so far from our HP support info we got is that it does not support properly autopath again.
I do find it a bit ridiculous that XP24000 which is supposed to support visualization and all other bells&whistles still cannot properly support load balancing, and thus needs to be done completely manually on host side.

[Niksa Franceschi]

Sorry for wrong topic.. was supposed to say autopath not autolun :)

[IBaltay]

Hi,
what is your backend raid group sizes?
Is it 3D+1P or 7D+1P or are you using the PG interleaved concatenation (14D+2P or 28D+4P)

[Niksa Franceschi]

We mostly use either RAID6 (6D+2P), or RAID1 (4 spindles).
RAID6 did prove to be more stable at higher IOPS per raid group (as expected due to more spindles).

[IBaltay]

RAID6 is the less powerfull. 2D+2D are only 4 spindles. Are you using the LVM on UNIX with the PVs from the different parity groups to have as many spindles in the VGs/Logical volumes as possible?

[Niksa Franceschi]

We have over 50TB on each storage.
On databases that have high load we try to use each LUN on different raid group, due to load.
Yes, LVM is used, as we often need to extend filesystems too (add new LUN's).
However, problem is that due to load on some LUN's it gets quite hard to balance paths manually, while using autopath is not recommended, nor is actually possible if using 4 or more paths per LUN, which actually one would expect to be possible to use on high-end storage.

[krusty]

Hi Niksa,

To answer your question on load balacing on the XP12000, the answer (as with all performance related questions) is "it depends". Load balancing is fully supported on the XP12000, however there are cases where it may not be the best solution. Sometimes it is better to static load balance across the CHIP ports. A MP on a XP12000 should be able to get I believe around 3000 IOPS. On some of the CHIPs on the XP12000, the ports share an MP. You may have the combined IOPS on the two ports greater than this number.

What type of load balancing did your use? You may need to try different policies to get the best one for your environment. The reason it is sometimes not recommended is due to the prefetching algorthims built into the CHIP ports. When the processor on the CHIP port senses sequential reads coming in, it will start reading ahead on the backend and pulling that data into cache to get a better read hit ratio. This is can be an issue with round robin load balancing. This issue is resolved on the XP24000.

I would dig a little bit more into what other hosts are sharing (if they are) the CHIP MP for that port. Next I would try different load balancing policies (start with SQST). As with the other suggestions, you may need to look at the backend Array Groups to see if the bottleneck is coming there also.

HTH,

Curt