Disk Usage is always 100%

[Jhun_2]

We have 4 CPU rp4440 server with HPUX 11.11. The RAM is 20 GB. This server is running Oracle Database along with Oracle Concurrent managers, the application (Oracle Apps 11i) is running in different server. On average basis the disk utilization is 100%. Even users are less the utilization is reaching 100%. Oracle support (thru metalink) already fine tuned our database and ask to check the OS side. Can anybody suggest a solution for this? Our system is deadly slow because of this. Thanks in advance.

[Raj D.]

Jhun,

Use iostat command and find out the disks io for the particular disks, and vg and fs .
The disk usage shows by glance is not correct. So use iostat command or use extract utility to find it out.

Cheers,
Raj.

[Jhun_2]

Hi Raj,

Here is the ouput of iostat.

device bps sps msps

c2t1d0 51 9.6 1.0
c2t0d0 18 4.1 1.0
c4t1d2 29450 1220.1 1.0
c5t1d2 0 0.0 1.0
c4t0d4 386 21.9 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 1005 56 0 10 34

device bps sps msps

c2t1d0 129 28.2 1.0
c2t0d0 44 8.0 1.0
c4t1d2 31207 1389.2 1.0
c5t1d2 0 0.0 1.0
c4t0d4 883 50.8 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 61 31 0 7 62

device bps sps msps

c2t1d0 41 10.0 1.0
c2t0d0 9 2.6 1.0
c4t1d2 24395 1152.8 1.0
c5t1d2 0 0.0 1.0
c4t0d4 1095 80.1 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 940 44 0 5 51

device bps sps msps

c2t1d0 356 33.4 1.0
c2t0d0 48 10.4 1.0
c4t1d2 33590 1353.2 1.0
c5t1d2 0 0.0 1.0
c4t0d4 1248 83.0 1.0
c5t0d4 0 0.0 1.0

and sar output

Average c2t1d0 9.41 0.50 17 259 4.72 8.92
Average c2t0d0 3.72 0.50 6 71 4.78 8.59
Average c4t1d2 99.98 8.56 1234 32307 11.47 6.17
Average c4t0d4 27.78 0.53 61 1394 5.10 6.44

Do I need to adjust some parameters?

Thanks in advance
Jhun

[Jeeshan]

Please use the command and send me the feedback

sar -d 5 5

here u can get the %busy of disks

Note: Sometime i have seen in my experience i have facing this kind of probelm. So give the output that i can help u.

Jeesun

[Jhun_2]

Dear Jessun,

Here is the output of sar -d 5 5

[root@orasrvr1]sar -d 5 5

HP-UX orasrvr1 B.11.11 U 9000/800 12/02/07

11:40:40 device %busy avque r+w/s blks/s avwait avserv
11:40:45 c2t1d0 9.15 0.50 24 198 5.23 6.11
c2t0d0 3.58 0.50 8 83 3.15 7.31
c4t1d2 99.60 10.55 1610 63688 14.62 7.62
c4t0d4 49.70 0.81 173 6925 5.51 7.97
11:40:50 c2t1d0 8.80 0.50 24 228 4.38 5.37
c2t0d0 2.40 0.50 5 51 3.17 9.16
c4t1d2 100.00 5.73 1702 75972 8.82 5.81
c4t0d4 74.60 0.70 319 12181 5.31 6.05
11:40:55 c2t1d0 9.98 0.50 18 184 4.37 8.79
c2t0d0 2.99 0.50 7 46 4.04 7.33
c4t1d2 99.80 7.30 1902 83225 9.96 5.06
c4t0d4 76.65 0.63 208 5544 6.40 6.53
11:41:00 c2t1d0 6.80 0.52 12 82 4.23 19.77
c2t0d0 4.00 0.50 8 66 3.43 14.15
c4t1d2 100.00 8.54 1428 56309 10.91 6.24
c4t0d4 58.60 0.68 118 3760 4.80 8.61
11:41:05 c2t1d0 8.43 0.50 13 137 4.79 10.39
c2t0d0 2.81 0.50 4 61 4.36 7.72
c4t1d2 100.00 12.06 1485 54376 23.75 7.34
c4t0d4 4.02 0.51 14 357 4.95 5.44

Average c2t1d0 8.63 0.50 18 166 4.64 8.82
Average c2t0d0 3.16 0.50 6 61 3.57 9.39
Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35
Average c4t0d4 52.76 0.70 167 5759 5.54 6.92

Thanks in advance

Jhun

[Jeeshan]

hi

your disk is in danger

Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35

its wait time is too much high

when did u bounce your database last?

and after tune did u restart the machine as with as database?

please check the buffer cache.

Jeesun

[Jeeshan]

what the disk write status in database?

if database has async. and OS has sync it will make pressure your disks

[Jhun_2]

Hi Jeesun,

Thanks for the help. We are bouncing our database daily and we just recently reboot the servers. The buffer cache is fine, but I checked the status, database is async while the OS is in sync. This may be the problem? Please advise. And also our HP engineer set the wait to 180 seconds. Is it okay?

Thanks in advance

Jhun

[Bill Hassell]

As mentioned before:

> Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35

This is a bad disk. The numbers are far too high for a normal disk. Do you have errors in syslog?

> I checked the status, database is async while the OS is in sync. This may be the problem?

No. The sync/async access methods are just methods used by the applications to obtain data.

> our HP engineer set the wait to 180 seconds

This is normal when connecting to high performance disk arrays. There are circumstances where the disk array simply stops responding for several seconds. Setting the timeout to a larger value keeps these maintenance periods from reporting an error in the OS.

What is the c4t1d2 disk? Is it part of a disk array? If so, these consistently high numbers could mean that the array has an internal problem.

[Jeeshan]

Hi Jhun

sync async is ok in your system.I think the problem is in your disk array.

Call your storage vendor and ask them to analyze the disks and also ask for report then you can realize

Jeesun