Performance issues with Smartarray 6402 controller

[John Oberlander]

We're running a universe database on a HP4440 11iV1 and having some performance issues with this new raid controller we installed. I have 10 146G drives in slots 1-10 of the MSA30 with 2 cables coming from the 6402. I configured all 10 drives to be raid 5 and presented 1 1TB lun to the box. We also increased the memory to 512 and increased the read cache to 80/20. Were using the default scripe size of 16K. The raid card is in slot 7, which is suppose to be the PCI-X 133 slot.

I know that i need to spread the disks out 5/5 on each side, but is there anything else i can reconfigure this weekend to get the best out of this controller?

[Steven E. Protter]

Shalom John,

To adequately answer that question, I would need to know what the system is doing. If its doing a heavy write oracle or db app on the disk, you are best to arrange for logs and data and index be on raid 1, not raid 5.

You might be able to spread around the I/O if you analyze the I/O.

Perhaps tell us about the application environment and collect some data to see which disk is the bottleneck. It might be a local disk.

http://www.hpux.ws/system.perf.sh

SEP

[John Oberlander]

The night processing is where were having problems, which are mostly reads. I attaced the output from your script.

I know for a fact that the disk is bottlenecked, i dont think you can ever give universe enough IO. The queues run around 80, and sar always reports 100%. But as far as the hardware config, i have something wrong cause it should be faster.

I think its beause im striping across channels, i should have put 5 disks on ch1 and 5 on ch2 and created 2 raid 5 luns, one on each channel.

We dont have enought space to go raid 0+1.

[Steve Lewis]

One LUN only?

Thats a bottleneck because HP-UX default queue depth is 8. I suggest to try increasing the scsi_max_qdepth parameter, but don't get your hopes up too far, because if the array controller cannot service current i/o fast enough, then pushing more i/o requests on to it won't help much.

I would definintely recommend at least 4 LUNs with your set-up. maybe 8 Luns. Put luns from both controllers into your volume group(s) and try LVM striping.

Maybe the stripe size should have been a bit bigger. I am not an expert on RAID 5, apart from I don't like it because it is diabolically SLOW. Have you seen
http://www.baarf.com ?

The next thing you should look into is the application code and database optimisation. Check your DB statistics are up-to-date and identify any unnecessary sequential scans. Maybe create new indexes. Bad code is the cause of 99% of performance problems.

Then check your DB storage model. It is raw LV or cooked files? If cooked files, what is the mount option and HP-UX buffering settings. If raw LVs then what is the Universe buffering/cache setting?

Lots more to investigate...

[John Oberlander]

I created 1 lun because i thought it would be best to stripe across all 10 disks. How can i create 4 luns when you need 4-5 disks to create a raid 5 lun? I'd need 20 disks, 5 per lun to create 4 luns. Right?

I only have 1 raid controller, it has 2 channels.

Should i switch from raid 5 to HP's ADS?

[Steven E. Protter · Accepted solution]

Shalom,

You don't need me to read the output of my(HP developed it, I fixed it) problem.

Look at the summary of the disk section.

You have a horrible io/wait problem.

1 Lun striped over all the disks. Thats how the text books say to do it.

I think the real problem here is you may be asking too much I/O of this system.

SEP

[John Oberlander]

Ok, so whats your recomendation on how we would rebuild it up with RAID0/1 this weekend?

Like i said eariler we have 10 disks, 1 controller with 2 channels.

put 4 disks on ch1 and 4 disks on ch2 as Raid 0/1, then create 2 luns and make one vg from it?

[Steven E. Protter]

The real bad news is:

I'm not sure rebuilding with raid 1 will help.

It surely would help a write intensive situation, because stuff gets written in less places with raid 1 than raid 5.

But this is mostly read, which means taking load off the entire system should help.

A bigger buffer cache on the OS dbc_max_pct higher may help.

If I was organizing I/O, I'd try and get the files or application with heavy reads onto different smaller raid 1 luns, in the hope that we reduce disk thrashing, ie the heads needing to be at more than one place on the same disk at the same time.

SEP

[John Oberlander]

I think the main issue that i have is that my raid 5 is spread across 2 channels on the controller. HP recommended that we create 2 luns, one for each channel.

We're not going to have the time to move all the universe files around and figure all that out.

I think ill have to go RAID1/0, 6 drives on channel A, and 4 drives on channel B. That will give us 1 438G lun and 1 292G lun.

Should i use PE distribution across these when i create the VG?

[John Oberlander]

We already increased the bufferecache up to 2.5G, any more and we'll be pressed on memory.

[Steven E. Protter]

No, I agree buffer cache is big enough.

Make sure dbc_min_pct is set clsoe to dbc_max_pct so the system doesn't waste a lot of cycles changing the buffer cache.

If its oracle data, consider radically reducing the buffer cache and using the SGA configuration to do Oracle buffering. Oracle is better at it for its own database. Same is true of other database vendors.

SEP

[John Oberlander]

Its a universe databsase, and alot of bufferecache is the only way i can get my boxes to run right. We have 6G of bc on another system running universe.

Thanks for the help, we'll figure something out.

[John Oberlander]

Thx