topic Re: I/O Bottleneck - Need help in Operating System - HP-UX

I/O Bottleneck - Need help

cheesytime — Wed, 25 Nov 2009 19:44:53 GMT

On a Itanium 11.31 box I suspect a lot of I/O activity that slowing down a few apps.

swapinfo output:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4227072 190572 4036500 5% 0 - 1 /dev/vg00/lvol2
reserve - 1790884 -1790884
memory 3969772 1894196 2075576 48%

top output:

System Page Size: 4Kbytes
Memory: 1299660K (638332K) real, 3042484K (2010240K) virtual, 89692K free Page# 1/5

This is sars output:

sar -d 5 50

HP-UX dragma B.11.31 U ia64 11/25/09

16:35:47 device %busy avque r+w/s blks/s avwait avserv
16:35:52 c2t3d0 22.65 0.50 43 547 0.00 5.65
disk5 22.85 0.50 47 539 0.00 6.00
disk9 22.65 0.50 43 547 0.00 5.65
16:35:57 c2t3d0 13.80 0.50 26 362 0.00 5.71
disk5 9.40 0.50 20 255 0.00 7.31
disk9 13.80 0.50 26 362 0.00 5.71
16:36:02 c2t3d0 9.58 0.50 18 301 0.00 7.11
disk5 9.18 0.50 23 328 0.00 6.28
disk9 9.58 0.50 18 301 0.00 7.11
16:36:07 c2t3d0 14.57 0.50 28 359 0.00 6.05
disk5 11.38 0.50 21 260 0.00 6.16
disk9 14.57 0.50 28 359 0.00 6.05
16:36:12 c2t3d0 19.68 0.50 38 457 0.00 5.82
disk5 14.86 0.50 29 310 0.00 5.26
disk9 19.68 0.50 38 457 0.00 5.82
16:36:17 c2t3d0 19.12 0.50 36 443 0.00 6.12
disk5 15.94 0.51 30 364 0.12 8.30
disk9 19.12 0.50 36 443 0.00 6.12
16:36:22 c2t3d0 18.00 0.50 35 394 0.00 5.78
disk5 12.60 0.50 31 331 0.00 5.55
disk9 18.00 0.50 35 394 0.00 5.78
16:36:27 c2t3d0 3.41 0.50 6 84 0.00 6.74
disk5 3.41 0.50 6 99 0.00 9.12
disk9 3.41 0.50 6 84 0.00 6.74
16:36:32 c2t3d0 2.20 0.50 4 63 0.00 7.53
disk5 2.20 0.50 4 61 0.00 8.11
disk9 2.20 0.50 4 63 0.00 7.53
16:36:37 c2t3d0 12.60 0.50 24 258 0.00 5.36
disk5 8.80 0.50 23 253 0.00 4.92
disk9 12.60 0.50 24 258 0.00 5.36
16:36:42 c2t3d0 15.43 0.50 27 295 0.00 6.85
disk5 18.04 0.50 55 640 0.00 3.76
disk9 15.43 0.50 27 295 0.00 6.85
16:36:47 c2t3d0 7.78 0.50 17 168 0.00 6.15
disk5 44.31 0.50 243 2579 0.06 2.98
disk9 7.78 0.50 17 168 0.00 6.15
16:36:52 c2t3d0 1.20 0.50 3 30 0.00 7.87
disk5 2.61 0.50 4 69 0.00 14.49
disk9 1.20 0.50 3 30 0.00 7.87
16:36:57 c2t3d0 1.60 0.50 2 34 0.00 10.07
disk5 2.40 0.50 4 61 0.00 14.61
disk9 1.60 0.50 2 34 0.00 10.07
16:37:02 c2t3d0 1.60 0.50 3 33 0.00 13.93
disk5 2.20 0.50 3 53 0.00 12.20
disk9 1.60 0.50 3 33 0.00 13.93
16:37:07 c2t3d0 1.60 0.50 2 45 0.00 9.04
disk5 2.80 0.50 4 72 0.00 14.18
disk9 1.60 0.50 2 45 0.00 9.04
16:37:12 c2t3d0 1.40 0.50 2 39 0.00 6.69
disk5 1.60 0.50 2 44 0.00 9.83
disk9 1.40 0.50 2 39 0.00 6.69
16:37:17 c2t3d0 2.59 0.50 3 46 0.00 14.79
disk5 2.59 0.55 4 63 1.29 19.66
disk9 2.59 0.50 3 46 0.00 14.79
16:37:22 c2t3d0 2.21 0.50 4 69 0.00 8.58
disk5 2.81 0.50 5 98 0.00 10.01
disk9 2.21 0.50 4 69 0.00 8.58
16:37:27 c2t3d0 2.20 0.50 4 60 0.00 7.53
disk5 2.59 0.50 4 65 0.00 16.21
disk9 2.20 0.50 4 60 0.00 7.53
16:37:32 c2t3d0 2.40 0.50 4 66 0.00 6.03
disk5 8.58 0.50 25 312 0.00 4.54
disk9 2.40 0.50 4 66 0.00 6.03
16:37:37 c2t3d0 1.60 0.50 2 45 0.00 7.55
disk5 2.00 0.50 3 56 0.00 10.33
disk9 1.60 0.50 2 45 0.00 7.55
16:37:42 c2t3d0 2.81 0.50 5 73 0.00 8.56
disk5 2.81 0.50 6 87 0.00 10.96
disk9 2.81 0.50 5 73 0.00 8.56
16:37:47 c2t3d0 2.19 0.50 4 45 0.00 12.01
disk5 3.98 0.52 8 111 0.77 13.25
disk9 2.19 0.50 4 45 0.00 12.01
16:37:52 c2t3d0 1.60 0.50 3 46 0.00 6.57
disk5 5.01 0.50 7 105 0.00 23.38
disk9 1.60 0.50 3 46 0.00 6.57
16:37:57 c2t3d0 3.00 0.50 4 74 0.00 7.94
disk5 2.00 0.50 3 56 0.00 9.03
disk9 3.00 0.50 4 74 0.00 7.94
16:38:02 c2t3d0 3.80 0.50 6 80 0.00 19.16
disk5 16.40 4.99 108 1381 21.14 8.21
disk9 3.80 0.50 6 80 0.00 19.16
16:38:07 c2t3d0 2.40 0.50 4 60 0.00 7.53
disk5 5.80 0.50 123 1497 0.00 0.54
disk9 2.40 0.50 4 60 0.00 7.53
16:38:12 c2t3d0 1.80 0.50 3 46 0.00 6.66
disk5 5.40 0.50 125 1505 0.00 0.53
disk9 1.80 0.50 3 46 0.00 6.66
16:38:17 c2t3d0 3.40 0.50 4 58 0.00 16.32
disk5 10.00 0.52 69 917 0.18 2.79
disk9 3.40 0.50 4 58 0.00 16.32
16:38:22 c2t3d0 2.20 0.50 4 51 0.00 10.96
disk5 2.20 0.50 3 63 0.00 9.30
disk9 2.20 0.50 4 51 0.00 10.96
16:38:27 c2t3d0 1.81 0.50 3 39 0.00 8.37
disk5 2.61 0.50 20 236 0.00 1.66
disk9 1.81 0.50 3 39 0.00 8.37
16:38:32 c2t3d0 2.40 0.50 4 57 0.00 8.35
disk5 3.99 0.50 11 157 0.00 6.58
disk9 2.40 0.50 4 57 0.00 8.35
16:38:37 c2t3d0 1.20 0.50 2 42 0.00 7.04
disk5 2.00 0.50 3 48 0.00 9.39
disk9 1.20 0.50 2 42 0.00 7.04
16:38:42 c2t3d0 2.00 0.50 4 48 0.00 13.55
disk5 3.40 0.50 6 95 0.00 5.62
disk9 2.00 0.50 4 48 0.00 13.55
16:38:47 c2t3d0 3.21 0.50 5 67 0.00 11.51
disk5 3.21 0.53 6 88 0.67 16.98
disk9 3.21 0.50 5 67 0.00 11.51
16:38:52 c2t3d0 2.20 0.50 3 53 0.00 8.02
disk5 5.79 0.50 13 202 0.00 6.98
disk9 2.20 0.50 3 53 0.00 8.02
16:38:57 c2t3d0 1.60 0.50 3 34 0.00 7.97
disk5 1.80 0.50 3 44 0.00 8.67
disk9 1.60 0.50 3 34 0.00 7.97
16:39:02 c2t3d0 1.80 0.50 3 41 0.00 15.47
disk5 2.20 0.50 4 62 0.00 8.60
disk9 1.80 0.50 3 41 0.00 15.47
16:39:07 c2t3d0 1.60 0.50 2 40 0.00 7.99
disk5 2.60 0.50 4 62 0.00 13.96
disk9 1.60 0.50 2 40 0.00 7.99
16:39:12 c2t3d0 1.80 0.50 3 42 0.00 15.27
disk5 2.20 0.50 3 59 0.00 12.32
disk9 1.80 0.50 3 42 0.00 15.27
16:39:17 c2t3d0 4.19 0.50 6 123 0.00 10.10
disk5 2.99 0.54 5 74 0.69 17.63
disk9 4.19 0.50 6 123 0.00 10.10
16:39:22 c2t3d0 1.80 0.50 3 40 0.00 14.81
disk5 1.20 0.50 2 36 0.00 8.82
disk9 1.80 0.50 3 40 0.00 14.81
16:39:27 c2t3d0 1.40 0.50 2 26 0.00 9.13
disk5 3.00 0.50 9 140 0.00 5.59
disk9 1.40 0.50 2 26 0.00 9.13
16:39:32 c2t3d0 1.79 0.50 2 41 0.00 10.32
disk5 21.51 0.50 120 1185 0.00 2.76
disk9 1.79 0.50 2 41 0.00 10.32
16:39:37 c2t3d0 1.61 0.50 2 40 0.00 7.36
disk5 2.41 0.50 4 65 0.00 11.03
disk9 1.61 0.50 2 40 0.00 7.36
16:39:42 c2t3d0 2.40 0.50 4 52 0.49 12.96
disk5 2.00 0.50 3 46 0.00 10.22
disk9 2.40 0.50 4 52 0.49 12.96
16:39:47 c2t3d0 3.20 0.50 5 71 0.00 12.25
disk5 3.80 0.53 6 90 1.15 20.33
disk9 3.20 0.50 5 71 0.00 12.25
16:39:52 c2t3d0 49.80 0.50 252 19807 0.00 4.04
disk5 8.17 0.50 32 577 0.00 3.50
disk9 49.80 0.50 252 19807 0.00 4.04
16:39:57 c2t3d0 14.06 0.61 176 8853 0.05 1.70
disk5 2.61 0.50 5 80 0.00 11.45
disk9 14.06 0.61 176 8853 0.05 1.70

Average c2t3d0 6.01 0.52 17 686 0.01 5.19
Average disk5 6.69 0.87 26 320 1.79 4.58
Average disk9 6.01 0.52 17 686 0.01 5.19

Re: I/O Bottleneck - Need help

Michael Steele_2 — Wed, 25 Nov 2009 20:18:38 GMT

Hi

You don't have a swap problem nor do you have a disk I/O problem. So lets see what the cpu is doing.

sar -u 5 5

UNIX95=1 ps -ef -o pcpu,pid,ppid,state,wchan,args | sort -rn | head -15

UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,args | sort -rn | head -15

cstm<<-EOF
runutil logtool
rs
EOF

Use attachements since the data will be long.

Re: I/O Bottleneck - Need help

cheesytime — Wed, 25 Nov 2009 20:27:28 GMT

#sar -u 5 5

HP-UX testbox B.11.31 U ia64 11/25/09

17:23:11 %usr %sys %wio %idle
17:23:16 1 0 1 98
17:23:21 2 1 1 97
17:23:26 1 0 1 97
17:23:31 2 1 0 98
17:23:36 1 0 1 98

Average 1 0 1 97
#:/# UNIX95=1 ps -ef -o pcpu,pid,ppid,state,wchan,args | sort -rn | head -15
3.76 3486 1 R - /opt/java1.5/bin/IA64N/java -Xmx64m -jar bigsm.jar
0.53 29635 1 S e00000016a1ca608 oraclebigsm (LOCAL=NO)
0.37 64 0 R - vxfsd
0.36 29564 22606 R - /opt/wbem/lbin/cimprovagt 0 4 8 root SFMProviderModule
0.28 1668 1 R - /opt/hpservices/jre/opt/java1.4/jre/bin/IA64N/java -Xmx512m -DRunnerValuePairFile=/var/opt/runner/data/valuePairs -cp /opt/runn
0.21 24624 1 R - /oracle/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER -inherit
0.21 3534 1 S e00000016ad6fe88 oraclebigsm (LOCAL=NO)
0.18 29960 1 S e000000169efaec0 ora_j000_bigsm
0.16 22607 22606 R - cimservermain --executor-socket 4
0.16 1915 1 R - vxsvc -r /opt/VRTSob/config/Registry -e
0.16 68 0 R - pagezerod
0.15 29964 1 S e000000404373f08 oraclebigsm (LOCAL=NO)
0.12 1745 1 R - /usr/sbin/swagentd -r
0.12 1530 1 R - /opt/dce/sbin/rpcd
0.10 66 0 R - pm_schedcpu
#:/home/uxadm/sysadm/STAST# UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,args | sort -rn | head -15
205104 24501 1 S e000000190618b40 ora_dbw0_bigsm
201472 3534 1 S e00000016ad6fe88 oraclebigsm (LOCAL=NO)
201216 29654 1 S e00000016a740b88 oraclebigsm (LOCAL=NO)
201216 29635 1 S e00000016a1ca608 oraclebigsm (LOCAL=NO)
201216 27148 1 S e000000400ebdc08 oraclebigsm (LOCAL=NO)
201216 6818 1 S e00000016ac33908 oraclebigsm (LOCAL=NO)
201216 6814 1 S e00000016ac33c08 oraclebigsm (LOCAL=NO)
200832 29984 1 S e000000404373f08 oraclesbigsm (LOCAL=NO)
200832 6816 1 S e000000169efcb88 oraclesbigsm (LOCAL=NO)
200816 24495 1 S e000000140070200 ora_pmon_bigsm
199424 24513 1 S e000000169efa240 ora_mmon_bigsm
199424 24511 1 S e000000190618440 ora_cjq0_bigsm
199168 24637 1 S e00000040a43ee40 ora_q000_bigsm
199168 24507 1 S e000000400c05b40 ora_smon_bigsm
199104 24509 1 S e0000004029e0bc0 ora_reco_bigsm
root#:/home/uxadm/sysadm/STAST# cstm<<-EOF
> runutil logtool
> rs
> EOF
Running Command File (/usr/sbin/stm/ui/config/.stmrc).

-- Information --
Support Tools Manager

Version D.06.00

Product Number B4708AA

(C) Copyright Hewlett Packard Co. 1995-2009
All Rights Reserved

Use of this program is subject to the licensing restrictions described
in "Help-->On Version". HP shall not be liable for any damages resulting
from misuse or unauthorized use of this program.

cstm>runutil logtool
To View a Summary of Events in a Raw Log

1. Select a raw (unformatted) log file. (File Menu -> "Select Raw")
The current log file ends in ".cur", e.g. "log1.raw.cur".
You do not have to switch logs.

2. View the summary of the selected log file. (View Menu -> "Raw Summary")

To Format a Raw Log

1. Set the format filter for the types of entries you want to see.
(Filters Menu -> "Format"). To see all entries, skip this step.

2. Format the raw log file. (File Menu -> "Format Raw")

3. Display the formatted file. (View Menu -> "Formatted Log")

4. To further narrow the entries displayed, set a display filter.
(Filters Menu -> "Display" -> "Formatted")

For more information, use the on-line help (Help Menu -> "General help").

-- Logtool Utility --
Logtool Utility>rs
Select Raw processing file /var/stm/logs/os/log2.raw.cur
Number of entries analyzed is 1.
Number of entries analyzed is 51.
Number of entries analyzed is 101.
Number of entries analyzed is 124.
The Select Raw operation completed.

The Display Raw Summary operation is currently in progress.

-- Converting a (1156) byte raw log file to text. --
Preparing the Logtool Utility: View Raw Summary File ...

.... testbox : 192.168.1.5 ....

-- Logtool Utility: View Raw Summary --

Summary of: /var/stm/logs/os/log2.raw.cur

Date/time of first entry: Tue Jun 30 22:55:36 2009

Date/time of last entry: Wed Nov 25 15:45:24 2009

Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 0

Number of I/O Error entries: 124

Device paths for which entries exist:

(1) 0/2/1/0
(123) 64000/0xfa00/0x0

Products for which entries exist:

(1) MPT SCSI Adapter
(123) SCSI Tape

Product Qualifiers for which entries exist:

(1) HP_A7173A_SCSI320
(123) HPSDLT600

Logger Events for which entries exist:

(1) mpt
(123) estape
-- Logtool Utility: View Raw Summary --
View - To View the file.
Print - To Print the file.
SaveAs - To Save the file.
Enter Done, Help, Print, SaveAs, or View: [Done] #.

Re: I/O Bottleneck - Need help

Michael Steele_2 — Wed, 25 Nov 2009 21:02:35 GMT

Hi

I don't see anything. I see better than normal.

Perhaps this is not the window to be running these tests. Why don't you set up a 15 cron with the sar -d 5 5, sar -u 5 5, and the two UNIX95 commands and see if it only shows up at a certain time of the day. Typically, after 11:00 AM workdays, after 4:40 PM workdays.

Re: I/O Bottleneck - Need help

Dennis Handly — Wed, 25 Nov 2009 22:19:40 GMT

>swapinfo output:

FYI: The "swapinfo -tam" output is more useful.

Re: I/O Bottleneck - Need help

cheesytime — Thu, 26 Nov 2009 11:59:40 GMT

Nov 19:
-----------------------------------------------------------------------------
TIME DISK BUSY
Average 15:14 disk5 56.07
Average 15:30 c2t3d0 34.95
Average 15:30 disk5 80.48
Average 15:30 disk9 34.95
Average 16:00 c2t3d0 31.82
Average 16:00 disk5 54.23
Average 16:00 disk9 31.82
Average 16:30 c2t3d0 31.02
Average 16:30 disk5 63.81
Average 16:30 disk9 31.02
Average 17:00 disk5 59.02
Average 17:30 disk5 69.78
Average 21:00 c2t3d0 49.21
Average 21:00 disk5 97.77
Average 21:00 disk9 49.21
Average 21:30 c2t3d0 39.13
Average 21:30 disk9 39.13
Average 22:00 c2t3d0 93.20
Average 22:00 disk9 93.20
--------------------------------------------------------------------------------
Nov 20:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 00:30 disk5 31.96
Average 08:00 disk5 72.30
Average 08:30 disk5 70.00
Average 09:00 disk5 62.62
Average 09:30 disk5 42.51
Average 10:00 disk5 71.49
Average 10:30 disk5 50.92
Average 11:00 disk5 63.95
Average 11:30 disk5 57.67
Average 12:00 disk5 53.27
Average 12:30 disk5 55.25
Average 13:00 c2t3d0 65.72
Average 13:00 disk5 93.80
Average 13:00 disk9 65.72
Average 13:30 disk5 63.22
Average 14:00 disk5 43.80
Average 14:30 c2t3d0 38.90
Average 14:30 disk5 82.01
Average 14:30 disk9 38.90
Average 15:00 c2t3d0 33.99
Average 15:00 disk5 34.19
Average 15:00 disk9 33.99
Average 15:30 disk5 54.31
Average 16:00 disk5 73.77
Average 21:00 c2t3d0 61.95
Average 21:00 disk5 78.70
Average 21:00 disk9 61.95
Average 21:30 disk5 38.44
Average 22:00 c2t3d0 92.92
Average 22:00 disk5 34.31
Average 22:00 disk9 92.92
--------------------------------------------------------------------------------
Nov 21:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 06:00 c2t3d0 83.99
Average 06:00 disk9 83.99
Average 06:30 disk5 39.25
Average 13:00 c2t3d0 81.58
Average 13:00 disk5 67.80
Average 13:00 disk9 81.58
Average 14:30 disk5 37.14
Average 21:00 c2t3d0 74.62
Average 21:00 disk5 59.84
Average 21:00 disk9 74.62
--------------------------------------------------------------------------------
Nov 22:
--------------------------------------------------------------------------------
TIME DISK BUSY
--------------------------------------------------------------------------------
Nov 23:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 00:30 disk5 31.26
Average 08:00 disk5 61.27
Average 08:30 c2t3d0 41.82
Average 08:30 disk5 72.68
Average 08:30 disk9 41.82
Average 09:00 c2t3d0 48.09
Average 09:00 disk5 79.39
Average 09:00 disk9 48.09
Average 09:30 c2t3d0 46.98
Average 09:30 disk5 88.39
Average 09:30 disk9 46.98
Average 10:00 disk5 79.33
Average 10:30 c2t3d0 98.28
Average 10:30 disk5 93.85
Average 10:30 disk9 98.28
Average 11:00 c2t3d0 87.47
Average 11:00 disk5 91.00
Average 11:00 disk9 87.47
Average 12:00 disk5 76.69
Average 12:30 disk5 45.48
Average 13:00 c2t3d0 90.42
Average 13:00 disk5 97.58
Average 13:00 disk9 90.42
Average 13:30 c2t3d0 33.43
Average 13:30 disk5 70.99
Average 13:30 disk9 33.43
Average 14:00 c2t3d0 68.96
Average 14:00 disk5 72.07
Average 14:00 disk9 68.96
Average 14:30 c2t3d0 35.69
Average 14:30 disk5 73.15
Average 14:30 disk9 35.69
Average 15:00 disk5 91.65
Average 15:30 disk5 95.50
Average 16:00 disk5 76.50
Average 17:00 disk5 66.14
Average 17:30 disk5 75.30
Average 18:00 c2t3d0 33.43
Average 18:00 disk5 80.82
Average 18:00 disk9 33.43
Average 18:30 disk5 66.62
Average 19:30 disk5 34.07
Average 21:00 c2t3d0 56.24
Average 21:00 disk5 98.10
Average 21:00 disk9 56.24
Average 21:30 disk5 40.78
Average 22:00 c2t3d0 88.62
Average 22:00 disk9 88.62
--------------------------------------------------------------------------------
Nov 24:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 07:30 disk5 59.47
Average 08:30 disk5 34.69
Average 09:00 disk5 56.42
Average 09:30 disk5 71.84
Average 10:00 disk5 33.55
Average 12:00 disk5 69.93
Average 12:30 disk5 63.84
Average 13:00 c2t3d0 86.58
Average 13:00 disk5 96.85
Average 13:00 disk9 86.58
Average 14:00 disk5 68.20
Average 15:00 disk5 38.71
Average 15:30 c2t3d0 63.93
Average 15:30 disk5 44.57
Average 15:30 disk9 63.93
Average 16:30 c2t3d0 38.26
Average 16:30 disk5 36.97
Average 16:30 disk9 38.26
Average 21:00 c2t3d0 69.40
Average 21:00 disk5 56.52
Average 21:00 disk9 69.40
Average 22:00 c2t3d0 91.30
Average 22:00 disk5 35.12
Average 22:00 disk9 91.30
--------------------------------------------------------------------------------
Nov 25:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 08:00 disk5 57.98
Average 08:30 disk5 69.21
Average 09:00 disk5 59.71
Average 09:30 c2t3d0 82.96
Average 09:30 disk5 73.74
Average 09:30 disk9 82.96
Average 10:00 c2t3d0 45.93
Average 10:00 disk5 75.60
Average 10:00 disk9 45.93
Average 10:30 disk5 65.33
Average 11:00 disk5 66.04
Average 11:30 disk5 65.84
Average 12:00 disk5 68.28
Average 12:30 disk5 42.25
Average 13:00 c2t3d0 61.75
Average 13:00 disk5 94.75
Average 13:00 disk9 61.75
Average 13:30 c2t3d0 31.05
Average 13:30 disk5 60.80
Average 13:30 disk9 31.05
Average 14:30 disk5 55.11
Average 16:00 disk5 68.83
Average 17:00 disk5 40.94
Average 21:00 c2t3d0 72.87
Average 21:00 disk5 39.19
Average 21:00 disk9 72.87
Average 22:00 c2t3d0 92.42
Average 22:00 disk9 92.42
--------------------------------------------------------------------------------
Nov 26:
--------------------------------------------------------------------------------
TIME DISK BUSY
Average 00:30 disk5 50.85
Average 08:00 c2t3d0 45.93
Average 08:00 disk5 96.87
Average 08:00 disk9 45.93
Average 08:30 disk5 65.48

Hope this helps?

Re: I/O Bottleneck - Need help

cheesytime — Thu, 26 Nov 2009 13:32:23 GMT

Any leads? please advice!

Re: I/O Bottleneck - Need help

Michael Steele_2 — Thu, 26 Nov 2009 16:07:53 GMT

Hi

Yes your %wio is high. But it could be CPU I/O waiting time as well as disk I/O waiting time.

Refer to the two UNIX95 commands and my recommendation to collect data for a week every 15 minutes. You may not need a week. You may see it after a day or two.

These are process analysis commands. pcpu tells you the top 15 process cpu consumers. vsz the virtual memory consumers.

Note the pid, ppid, wchan, and args arguements, especially state and wchan. state will be run, sleep, wait, and others you can refer to the man page about. wchan is what the pid is waiting on.

From here, it becomes more of an application analysis. Take the top five consumers and break them down with 'lsof -p pid'. See what they are writing too.

wio means writing io. So you've got processes that are doing something like, a 'dd' of one disk to another, for example. Some long time consumming write.

Refer to all of your file system kernel parameters like the following:

bufpages "Pages of static buffer cache"
dbc_min_pct "Minimum dynamic buffer cache"
dbc_max_pct "Maximum dynamic buffer cache"
nbuf "Number of static buffer headers"

maxfiles "soft limit for open files"
maxfiles_lim "hard limit for open files"
nfile "system-wide open-files limit"
nflocks "system-wide file-lock limit"
ninode "Maximum open inodes in memory"

Paste in their values and we'll see how they compare to what other people are doing.

Defrag your file systems unless you're using raw file systems or hfs. Rules to defrag, once started don't ever kill, let the defrag complete gracefully. Run daily or weekly for awhile until the reports no longer indicate fragmentatin, then may only monthly. 'fsadm' has options for number of passes to make in a file system. Default is 5. Change this to 1 until the time of execution has reduced, for the first time you defrag, it may be over an hour for just one pass.

So, you are going to perform kernel tuning, process analysis and start defragging.

Okey dokey?

Re: I/O Bottleneck - Need help

Michael Steele_2 — Thu, 26 Nov 2009 16:16:07 GMT

Hi again

Sorry about the grammer, I try to preview but it always locks up.

Another comment about defragging, don't worry about before or after backups. If you ever have to restore an entirer file system from tape, the restoration will write the data in its fastest, most contiguous form.

In fact, you make want to consider restoring all these file systems from tape, just to get the data in its fastest, most contiguous format.

Good luck.

Re: I/O Bottleneck - Need help

Michael Steele_2 — Thu, 26 Nov 2009 16:22:21 GMT

HI again

...and correction. wio means wait time, but I'm trying to get you to look for writing operations that take a long time.

Sorry for the confustion, if any.

Re: I/O Bottleneck - Need help

Jean-Luc Oudart — Thu, 26 Nov 2009 17:49:10 GMT

Hi

May be you could give some imformation regarding the type of applications that run on this box. From the ps command it looks like you have java and oracle.
I've seen the listener using quite some cpu ! do you have a big rate of db connection ?

Regards
Jean-Luc