RAID 4si slowed down OS environment

[cam9269]

Hi Guys,

I recently installed a RAID 4si on an L-class server with 2 x440Mhz CPU and 4GB mem

The controller is attached to 8 x 300GB MSASB ext storage; configured as RAID5 (7disks + 1 spare)

The problem is, whenever, the disks get busy, the OS is severely affected and exhibits major slowdown, which makes the users complain about it.

From this point on, I have no ideas as to where to start troubleshooting nor implementing tune-ups.

Hoping for your valued inputs

[Michael Steele_2]

Well you're going to have to send some performance data:

sar -b 5 5
sar -v 5 5
vmstat 5 5
sar -uM 5 5
swapinfo -tam
sar -d 5 5

You may end up with an issue like new or unstructured (* not in a database *) problem.

Also provide:

cat fstab

Need to see your vxfs switches

[cam9269]

Hi Michael,

Thanks for the quick reply, here's the data, though not sure if these will be conclusive enough, because the system is not as busy as i wanted it to be, i just ran a copy of a large file to create I/O.

cat /etc/fsatab

/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vg00/lvol9 /var/adm/crash vxfs delaylog 0 2
/dev/datavg/drdata /u02/CMDR vxfs delaylog,nodatainlog 0 2
/dev/datavg/devdata /u02/CMDEV vxfs delaylog,nodatainlog 0 2
/dev/datavg/drappl /u01/DRora vxfs delaylog,nodatainlog 0 2
/dev/datavg/devappl /u01/DEVora vxfs delaylog,nodatainlog 0 2
/dev/datavg/drloglv /logs/DRLogs vxfs delaylog,nodatainlog 0 2
/dev/datavg/devloglv /logs/DEVLogs vxfs delaylog,nodatainlog 0 2
/dev/datavg/tstloglv /logs/TSTLogs vxfs delaylog,nodatainlog 0 2
#/dev/datavg/u3201tst /u3201 vxfs delaylog,nodatainlog 0 2
/dev/datavg/u3201tst /u3201TST vxfs delaylog,nodatainlog 0 2
/dev/datavg/u3201dev /u3201DEV vxfs delaylog,nodatainlog 0 2
/dev/datavg/u3201dr /u3201DR vxfs delaylog,nodatainlog 0 2
/dev/datavg/tstdata /u02/CMTST vxfs delaylog,nodatainlog 0 2
/dev/datavg/bkuplv /OraBkp vxfs delaylog,nodatainlog 0 2


> sar -b 5 5

HP-UX B.11.11 U 9000/800 03/09/09

12:52:54 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
12:52:59 1610 1731 7 1630 3260 50 0 0
12:53:04 902 1061 15 1730 2690 36 0 0
12:53:09 819 789 0 1391 2154 35 0 0
12:53:14 1421 1544 8 1490 2928 49 0 0
12:53:19 1752 1746 0 1573 3309 52 0 0

Average 1301 1374 5 1563 2868 46 0 0

> sar -v 5 5

HP-UX B.11.11 U 9000/800 03/09/09

12:53:26 text-sz ov proc-sz ov inod-sz ov file-sz ov
12:53:31 N/A N/A 182/4420 0 1347/37408 0 1242/86790 0
12:53:36 N/A N/A 182/4420 0 1347/37408 0 1243/86790 0
12:53:41 N/A N/A 182/4420 0 1347/37408 0 1246/86790 0
12:53:46 N/A N/A 182/4420 0 1347/37408 0 1246/86790 0
12:53:51 N/A N/A 182/4420 0 1347/37408 0 1246/86790 0

> vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
3 1 0 691638 84846 156 43 8 1 0 0 61 2257 7955 675 16 8 76
3 1 0 691638 84815 19 1 2 0 0 0 0 867 1353 339 0 7 92
2 0 0 638728 84822 37 4 1 0 0 0 0 865 2214 337 1 6 93
2 0 0 638728 84767 12 0 0 0 0 0 0 863 1467 343 0 5 95
4 0 0 658498 84767 3 0 0 0 0 0 0 861 1137 319 0 6 94


> sar -uM 5 5

HP-UX B.11.11 U 9000/800 03/09/09

12:54:32 cpu %usr %sys %wio %idle
12:54:37 0 1 2 12 85
1 1 11 57 32
system 1 6 34 59
12:54:42 0 2 4 24 70
1 0 9 50 42
system 1 6 37 56
12:54:47 0 0 5 26 69
1 1 6 49 45
system 1 5 37 57
12:54:52 0 0 3 9 88
1 1 11 60 28
system 1 7 34 58
12:54:57 0 0 0 8 91
1 1 13 65 21
system 1 7 37 56

Average 0 1 3 16 81
Average 1 1 10 56 34
Average system 1 6 36 57

> swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 275 3821 7% 0 - 1 /dev/vg00/lvol2
reserve - 2319 -2319
memory 2704 817 1887 30%
total 6800 3411 3389 50% - 0 -

> sar -d 5 5

HP-UX B.11.11 U 9000/800 03/09/09

12:55:17 device %busy avque r+w/s blks/s avwait avserv
12:55:22 c1t15d0 1.00 0.50 6 49 0.00 1.66
c6t0d0 99.80 0.50 497 49815 0.00 47.67
12:55:27 c1t15d0 0.20 0.50 7 45 0.00 0.38
c6t0d0 100.00 0.50 496 48828 0.00 48.03
12:55:32 c1t15d0 1.80 0.50 15 113 0.00 1.33
c6t0d0 100.00 0.50 501 49587 0.00 46.94
12:55:37 c6t0d0 99.60 0.50 496 50097 0.00 46.91
12:55:42 c1t15d0 0.80 0.50 6 85 0.00 1.89
c6t0d0 100.00 0.50 493 48857 0.00 47.82

Average c1t15d0 0.76 0.50 7 58 0.00 1.29
Average c6t0d0 100.04 0.50 497 49437 0.00 47.48

[Michael Steele_2]

Refer first to sar -d avwait and avserv. When avwait is greater than avserv a disk bottleneck is identified. You have no problems here. Your array is not the problem.


Refer now to you buffer cache under sar -b and %wcache and %rcache. Your average read cache is 5. It should be 100. Here is your problem.

You %wcache averages 46 when 75 is the rule of thumb. This could be because you're writing new flat files to disk instead of structured data into a database. But, 46 could also be your norm is you have a lot of flat files, else, your database needs to be rehashed / reindexed / etc.

[TTr]

> The controller is attached to 8 x 300GB MSASB ext storage

What kind of storage enclosure exactly are you using and how is it connected to the 4Si? With a 4Si card you should try and use all 4 SCSI buses and spread the disks out so that you get much higher throughput.

[Michael Steele_2]

How could it be a through put problem when the wait time is zero?

[cam9269]

Thanks for the inputs Michael, could this be a result of a freshly imported database? because we recently imported a prod database onto this box to be a test env. when you say the database needs to be rehashed/reindexed, is that similar to dropping and re-creating all indexes? I also did a gather schema stats on one of the users. not sure if i need to continue with the rest of the users or do a full db gather schema stats.

[Michael Steele_2]

Freshly installed database. Well, the reindexing is what I'm talking about. You're fstab says that you are mounted on vxfs file system. What I can tell you about the file system is that they are as ordered and as contiguous as they will ever be since they are new and newly filled with data. So I would definitely be looking at the database indexing.