BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

HP Blade Server running very slow I/O !

sreejithsna
Occasional Visitor

HP Blade Server running very slow I/O !

We have a new HP 8 core Blade server with SATA disks. The server is partitioned using VM ware to 4 2 Core servers, each with 12 GB RAM. I have noticed that the server is running very slow and on analysis, it was noted that disk I/O was taking considerable amount of time. I am attaching the statistics below.
Please advice on the bottleneck here.

N260 oracle [BADEV1]:free -m
total used free shared buffers cached
Mem: 12011 11971 39 0 182 10915
-/+ buffers/cache: 873 11137
Swap: 20481 43 20437
N260 oracle [BADEV1]:vmstat 5 10
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 9 44656 53416 187156 11164376 0 0 4 37 23 8 0 0 79 20
0 8 44656 52008 187172 11165788 0 0 130 43 1066 1141 0 0 47 53
0 5 44656 50728 187180 11167140 0 0 128 502 1071 1154 0 0 63 37
0 6 44656 49384 187188 11168424 0 0 128 51 1124 1147 0 0 50 50
0 7 44656 48104 187196 11169708 0 0 129 54 1069 1151 0 0 50 50
0 13 44656 47640 187200 11169704 0 0 0 25 1056 1133 0 0 50 50
0 13 44656 47640 187200 11169704 0 0 0 316 1056 1131 0 0 30 69
0 13 44656 47576 187200 11169704 0 0 0 8 1116 1127 0 0 25 75
0 9 44656 47448 187204 11169836 0 0 26 18 1066 1142 0 0 25 75
0 11 44656 43344 187208 11170172 0 0 26 20 1059 1133 0 0 48 52
N260 oracle [BADEV1]:iostat -x 5 10
Linux 2.6.9-78.ELsmp (N260) 10/25/2010

avg-cpu: %user %nice %sys %iowait %idle
0.28 0.00 0.13 20.18 79.41

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.47 31.64 0.85 5.67 35.44 298.49 17.72 149.24 51.17 4.76 729.76 92.25 60.20
sda1 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 28.86 0.00 5.65 4.94 0.00
sda2 0.01 4.56 0.01 0.14 1.94 37.54 0.97 18.77 267.16 0.63 4253.47 115.85 1.71
sda3 0.42 12.03 0.63 1.40 22.55 107.48 11.27 53.74 63.86 0.82 404.96 153.77 31.31
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 1.00 1.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.36 0.00 14.59 14.59 0.00
sda6 0.01 11.33 0.13 3.40 8.60 117.86 4.30 58.93 35.86 1.58 449.45 158.56 55.91
sda7 0.01 0.10 0.01 0.13 0.41 1.91 0.20 0.96 16.14 0.07 516.91 409.80 5.90
sda8 0.00 0.95 0.07 0.48 1.88 11.39 0.94 5.69 24.29 0.32 592.58 299.39 16.35
sda9 0.00 1.02 0.00 0.07 0.04 8.69 0.02 4.35 126.09 0.56 8021.72 531.44 3.68
sda10 0.01 1.65 0.00 0.06 0.02 13.61 0.01 6.81 242.90 0.77 13734.96 155.49 0.87

avg-cpu: %user %nice %sys %iowait %idle
0.40 0.00 0.15 25.09 74.36

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 42.17 0.40 3.41 102.81 369.48 51.41 184.74 123.79 12.04 1004.37 263.21 100.42
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.20 0.40 0.40 102.81 4.82 51.41 2.41 134.00 1.22 1001.00 1250.25 100.42
sda3 0.00 0.80 0.00 0.60 0.00 11.24 0.00 5.62 18.67 1.20 981.33 1667.00 100.42
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda6 0.00 6.43 0.00 1.81 0.00 64.26 0.00 32.13 35.56 4.01 1015.22 555.67 100.42
sda7 0.00 0.00 0.00 0.20 0.00 1.61 0.00 0.80 8.00 1.00 1016.00 5001.00 100.42
sda8 0.00 3.61 0.00 0.20 0.00 30.52 0.00 15.26 152.00 0.39 991.00 1926.00 38.67
sda9 0.00 31.12 0.00 0.20 0.00 257.03 0.00 128.51 1280.00 4.23 991.00 5001.00 100.42
sda10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.05 0.00 0.00 25.31 74.64

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 13.17 0.20 4.99 51.10 137.33 25.55 68.66 36.31 14.71 4508.04 192.35 99.82
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.20 0.20 51.10 1.60 25.55 0.80 132.00 2.00 6468.50 2500.50 99.82
sda3 0.00 4.59 0.00 0.60 0.00 44.71 0.00 22.36 74.67 1.22 3318.67 1667.00 99.82
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda6 0.00 7.78 0.00 2.40 0.00 79.84 0.00 39.92 33.33 5.37 3218.17 416.75 99.82
sda7 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.99 8947.00 4941.00 98.62
sda8 0.00 0.00 0.00 0.40 0.00 1.60 0.00 0.80 4.00 0.99 2956.00 2488.50 99.34
sda9 0.00 0.80 0.00 1.20 0.00 9.58 0.00 4.79 8.00 4.15 6806.50 833.50 99.82
sda10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.10 0.00 0.10 30.13 69.67

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 40.08 1.20 8.02 258.12 378.36 129.06 189.18 69.04 8.93 930.02 108.72 100.22
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.20 1.00 0.40 256.51 4.81 128.26 2.40 186.29 1.21 878.86 714.29 100.20
sda3 0.00 8.62 0.20 1.80 1.60 80.16 0.80 40.08 40.80 1.82 917.90 500.00 100.20
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda6 0.00 5.41 0.00 3.81 0.00 70.54 0.00 35.27 18.53 3.81 1008.89 263.21 100.22
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda8 0.00 0.20 0.00 0.40 0.00 4.81 0.00 2.40 12.00 0.40 997.50 997.50 39.98
sda9 0.00 25.65 0.00 1.60 0.00 218.04 0.00 109.02 136.00 1.68 785.75 625.12 100.22
sda10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0

avg-cpu: %user %nice %sys %iowait %idle
0.35 0.00 0.20 29.94 69.52

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.40 33.80 1.00 5.60 257.60 326.40 128.80 163.20 88.48 10.67 998.03 151.58 100.04
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.40 0.20 1.00 0.20 257.60 3.20 128.80 1.60 217.33 1.21 1004.67 833.33 100.00
sda3 0.00 1.80 0.00 1.20 0.00 25.60 0.00 12.80 21.33 1.21 1005.17 833.50 100.02
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda6 0.00 7.20 0.00 4.20 0.00 92.80 0.00 46.40 22.10 4.17 994.10 238.19 100.04
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda9 0.00 24.60 0.00 0.00 0.00 204.80 0.00 102.40 0.00 4.09 0.00 0.00 100.04
sda10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.30 0.00 0.20 33.55 65.95

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 8.20 1.20 6.00 259.20 107.20 129.60 53.60 50.89 13.06 984.17 138.89 100.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.00 1.00 0.40 256.00 1.60 128.00 0.80 184.00 1.40 1004.57 714.29 100.00
sda3 0.00 2.20 0.20 1.60 3.20 28.80 1.60 14.40 17.78 1.80 1003.11 555.56 100.00
sda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda6 0.00 5.80 0.00 3.60 0.00 72.00 0.00 36.00 20.00 3.51 978.11 277.78 100.00
sda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda8 0.00 0.20 0.00 0.40 0.00 4.80 0.00 2.40 12.00 0.35 882.00 882.00 35.28
sda9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.00 0.00 0.00 100.00
sda10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.10 49.97 49.92

An excerpt from 'top'

top - 09:53:14 up 2 days, 22:23, 7 users, load average: 11.63, 9.82, 9.07
Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.6% us, 0.1% sy, 0.0% ni, 92.9% id, 6.2% wa, 0.1% hi, 0.1% si
Cpu1 : 0.1% us, 0.1% sy, 0.0% ni, 97.3% id, 2.5% wa, 0.0% hi, 0.0% si
Cpu2 : 0.2% us, 0.0% sy, 0.0% ni, 63.8% id, 35.9% wa, 0.0% hi, 0.0% si
Cpu3 : 0.2% us, 0.0% sy, 0.0% ni, 63.6% id, 36.1% wa, 0.0% hi, 0.0% si
Mem: 12299332k total, 12249500k used, 49832k free, 187396k buffers
Swap: 20972816k total, 44656k used, 20928160k free, 11160600k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5772 oracle 16 0 6304 1224 852 R 0.5 0.0 0:00.01 top
1 root 16 0 4772 564 468 S 0.0 0.0 0:01.28 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.65 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.38 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0.0 0.0 0:00.26 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
8 root RT 0 0 0 0 S 0.0 0.0 0:00.21 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
10 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/0
11 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/1
12 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/2
13 root 5 -10 0 0 0 S 0.0 0.0 0:01.62 events/3
14 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
15 root 14 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
93 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
94 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/1
95 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/2
96 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/3
97 root 25 0 0 0 0 S 0.0 0.0 0:00.00 khubd
134 root 15 0 0 0 0 S 0.0 0.0 0:01.67 kswapd0
135 root 8 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
136 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 aio/1
137 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 aio/2
138 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 aio/3
284 root 24 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
537 root 15 0 0 0 0 S 0.0 0.0 0:00.02 scsi_eh_0
567 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
568 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 ata/1
569 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 ata/2
570 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 ata/3
571 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 ata_aux
588 root 15 0 0 0 0 S 0.0 0.0 0:01.12 kjournald
1247 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kauditd
2120 oracle 16 0 740m 130m 125m S 0.0 1.1 0:04.68 oracle
3152 root 16 0 72296 2284 1600 S 0.0 0.0 0:00.00 cupsd
3901 root 6 -10 3624 456 372 S 0.0 0.0 0:00.00 udevd
3948 root 19 0 0 0 0 S 0.0 0.0 0:00.00 pciehpd_event
4304 oracle 16 0 740m 56m 51m S 0.0 0.5 0:01.21 oracle
4341 root 15 0 0 0 0 D 0.0 0.0 0:03.47 kjournald
4342 root 19 0 0 0 0 S 0.0 0.0 0:00.00 kjournald




1 REPLY
Sivaramakrishna_1
Honored Contributor

Re: HP Blade Server running very slow I/O !

Hi,

Some of the blade serves will not be shipped with BBWC which enables the Write Cache on the array controllers so that the performace can be improved. (i am assuming that you have hosted the vms in SATA drives itself).

Generally it is recommanded to host IO on either SAS driver or in External Storage (FC) to get the better performance.

More over subjected to 4*2 core VMs: - Virtual machine will not dedicate complete core, it will allocate some % of cpu core, so you can allocate more CPU per VM, for eg: if you have total 8 cores, you can assign 4 cores to each VM also.

Did the VMware tools installed on the guest OS?These will improve the some perofmance on the Virtual Disks!