Disk Usage is always 100%

[Jhun_2]

We have 4 CPU rp4440 server with HPUX 11.11. The RAM is 20 GB. This server is running Oracle Database along with Oracle Concurrent managers, the application (Oracle Apps 11i) is running in different server. On average basis the disk utilization is 100%. Even users are less the utilization is reaching 100%. Oracle support (thru metalink) already fine tuned our database and ask to check the OS side. Can anybody suggest a solution for this? Our system is deadly slow because of this. Thanks in advance.

[Raj D.]

Jhun,

Use iostat command and find out the disks io for the particular disks, and vg and fs .
The disk usage shows by glance is not correct. So use iostat command or use extract utility to find it out.

Cheers,
Raj.

[Jhun_2]

Hi Raj,

Here is the ouput of iostat.

device bps sps msps

c2t1d0 51 9.6 1.0
c2t0d0 18 4.1 1.0
c4t1d2 29450 1220.1 1.0
c5t1d2 0 0.0 1.0
c4t0d4 386 21.9 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 1005 56 0 10 34

device bps sps msps

c2t1d0 129 28.2 1.0
c2t0d0 44 8.0 1.0
c4t1d2 31207 1389.2 1.0
c5t1d2 0 0.0 1.0
c4t0d4 883 50.8 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 61 31 0 7 62

device bps sps msps

c2t1d0 41 10.0 1.0
c2t0d0 9 2.6 1.0
c4t1d2 24395 1152.8 1.0
c5t1d2 0 0.0 1.0
c4t0d4 1095 80.1 1.0
c5t0d4 0 0.0 1.0


tty cpu
tin tout us ni sy id
0 940 44 0 5 51

device bps sps msps

c2t1d0 356 33.4 1.0
c2t0d0 48 10.4 1.0
c4t1d2 33590 1353.2 1.0
c5t1d2 0 0.0 1.0
c4t0d4 1248 83.0 1.0
c5t0d4 0 0.0 1.0

and sar output

Average c2t1d0 9.41 0.50 17 259 4.72 8.92
Average c2t0d0 3.72 0.50 6 71 4.78 8.59
Average c4t1d2 99.98 8.56 1234 32307 11.47 6.17
Average c4t0d4 27.78 0.53 61 1394 5.10 6.44

Do I need to adjust some parameters?

Thanks in advance
Jhun

[Jeeshan]

Please use the command and send me the feedback

sar -d 5 5

here u can get the %busy of disks

Note: Sometime i have seen in my experience i have facing this kind of probelm. So give the output that i can help u.

Jeesun

[Jhun_2]

Dear Jessun,

Here is the output of sar -d 5 5

[root@orasrvr1]sar -d 5 5

HP-UX orasrvr1 B.11.11 U 9000/800 12/02/07

11:40:40 device %busy avque r+w/s blks/s avwait avserv
11:40:45 c2t1d0 9.15 0.50 24 198 5.23 6.11
c2t0d0 3.58 0.50 8 83 3.15 7.31
c4t1d2 99.60 10.55 1610 63688 14.62 7.62
c4t0d4 49.70 0.81 173 6925 5.51 7.97
11:40:50 c2t1d0 8.80 0.50 24 228 4.38 5.37
c2t0d0 2.40 0.50 5 51 3.17 9.16
c4t1d2 100.00 5.73 1702 75972 8.82 5.81
c4t0d4 74.60 0.70 319 12181 5.31 6.05
11:40:55 c2t1d0 9.98 0.50 18 184 4.37 8.79
c2t0d0 2.99 0.50 7 46 4.04 7.33
c4t1d2 99.80 7.30 1902 83225 9.96 5.06
c4t0d4 76.65 0.63 208 5544 6.40 6.53
11:41:00 c2t1d0 6.80 0.52 12 82 4.23 19.77
c2t0d0 4.00 0.50 8 66 3.43 14.15
c4t1d2 100.00 8.54 1428 56309 10.91 6.24
c4t0d4 58.60 0.68 118 3760 4.80 8.61
11:41:05 c2t1d0 8.43 0.50 13 137 4.79 10.39
c2t0d0 2.81 0.50 4 61 4.36 7.72
c4t1d2 100.00 12.06 1485 54376 23.75 7.34
c4t0d4 4.02 0.51 14 357 4.95 5.44

Average c2t1d0 8.63 0.50 18 166 4.64 8.82
Average c2t0d0 3.16 0.50 6 61 3.57 9.39
Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35
Average c4t0d4 52.76 0.70 167 5759 5.54 6.92

Thanks in advance

Jhun

[Jeeshan]

hi

your disk is in danger

Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35

its wait time is too much high

when did u bounce your database last?

and after tune did u restart the machine as with as database?

please check the buffer cache.

Jeesun

[Jeeshan]

what the disk write status in database?

if database has async. and OS has sync it will make pressure your disks

[Jhun_2]

Hi Jeesun,

Thanks for the help. We are bouncing our database daily and we just recently reboot the servers. The buffer cache is fine, but I checked the status, database is async while the OS is in sync. This may be the problem? Please advise. And also our HP engineer set the wait to 180 seconds. Is it okay?

Thanks in advance

Jhun

[Bill Hassell]

As mentioned before:

> Average c4t1d2 100.00 8.70 1626 66727 13.32 6.35

This is a bad disk. The numbers are far too high for a normal disk. Do you have errors in syslog?

> I checked the status, database is async while the OS is in sync. This may be the problem?

No. The sync/async access methods are just methods used by the applications to obtain data.

> our HP engineer set the wait to 180 seconds

This is normal when connecting to high performance disk arrays. There are circumstances where the disk array simply stops responding for several seconds. Setting the timeout to a larger value keeps these maintenance periods from reporting an error in the OS.

What is the c4t1d2 disk? Is it part of a disk array? If so, these consistently high numbers could mean that the array has an internal problem.

[Jeeshan]

Hi Jhun

sync async is ok in your system.I think the problem is in your disk array.

Call your storage vendor and ask them to analyze the disks and also ask for report then you can realize

Jeesun

[Jhun_2]

Hi,

We used to get this kind of error in syslog:

Nov 26 12:43:17 orasrvr1 syslog: CVSDM; INFORMATION Event Code=400; Description=348:FRONTEND_FC_ABTS_EVENT_EH This error code indicates that the Host sent a Fibre Channel ABTS (Abort Sequence) BLS frame to the abort an IO. The array will log this event for informational and debug purposes only. It does not necessarily indicate a problem with the array.; 0x17028a: 0x8044; frontend osPortCB: ABTSAbort-JCB; NPortID=0x010200,OXID=0x8044,LUN=0x400a; CDB=0x28000f4e928000001000000000000000; et=10.1 qd=17 hwm=39 jl=0x00002000 dl=0x00002000 ro=0x00000000 js=1; enclosureId/slot/component/subcomponent : 0x00/0x00/0x70/0xff; controller tick : 22916704206211; serialNum/moduleId/processId : 00PR00D50026/0x47/0xffffffff; ; Hardware Address=0/5/2/0.1.0.255.0.0.0; FRU Location=M/C1.H1; Vendor ID=HP; Model ID=A6189B; Product S/N=00DEH4409AP3; Latest information on this event at http://docs.hp.com/hpux/content/hardware/ems/RemoteMonitor.htm#400

Our vendor and HP Engineer analyze this problem several times but no result. Yes, c4t1d2 is part of the array. Do you think it is the problem of the whole array or just a particular disk in the array? Futhermore whenever we do some operation such copying huge amount of data within the array the system performance is going down drastically.

Thanks

Jhun

[leelangco_1]

why not check the lv information on the disk?

[Bill Hassell]

The syslog message indicates a minor symptom, basically a handshake condition. Did the engineer run cvui or armdsp to list out the logs and diagnostic state of the VA 7110? syslog is a catch-all for failure messages but it will not record internal states on the array. All the LUNs on the array will respond at about the same speed and this one LUN is way out of line. The VA 7110 is a sophisticated array with a lot of logs and diagnostics. Something is going on inside the array that is slowing the response time.