%CPU 100% with %SYS > 70%

[Eric Antunes]

Hi again,

Did you tried to measure bread/s and lread/s?

#sar -b 6 200

For example, for your c10t6d4 and c2t2d0:

#dd if=/dev/dsk/c2t2d0 of=/dev/null bs=64k

(interrupt after 30 sec.)

and:

#dd if=/dev/dsk/c10t6d4 of=/dev/null bs=64k

(interrupt after 30 sec.)

What are your bread/s and lread/s averages?

Eric

[Eric Antunes]

Hi Pedro,

Look in here (http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=469286), Bill Hassel's explanation. That may be your case if you bought a brand new DMX for an old CPU(s) box...

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi Eric,

Interesting...:
root@poru9002:/# time dd if=/dev/dsk/c10t6d4 of=/dev/null bs=64k
7220+0 records in
7220+0 records out

real 30.0
user 0.0
sys 3.8
root@poru9002:/tmp# time dd if=/dev/dsk/c10t6d4 of=/dev/null bs=64k
8670+0 records in
8670+0 records out

real 29.9
user 0.0
sys 5.6

Attached, find the sar statistics.

Thks

[Eric Antunes]

Hi Pedro,

Did you collected those "sar -b" statistics while doing the dd to the disk?? If so, you have really bad statistics here! My average is this with much older disks:

15:24:13 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
15:24:19 3562 4101 13 7 12 43 0 0
15:24:25 3358 4458 25 13 198 93 0 0
15:24:31 3164 3559 11 35 51 31 0 0
15:24:37 3340 3795 12 29 40 28 0 0
15:24:43 3899 3794 0 14 28 50 0 0
15:24:49 4234 4334 2 10 14 29 0 0
15:24:55 4282 4768 10 6 6 0 0 0
15:25:01 3311 3706 11 10 22 57 0 0
15:25:07 4161 4469 7 8 7 0 0 0
15:25:13 4008 4043 1 4 44 90 0 0
15:25:19 2641 4125 36 32 212 85 0 0
15:25:25 4616 4477 0 7 21 66 0 0
15:25:31 4386 4493 2 11 11 3 0 0
15:25:37 4628 4928 6 7 9 22 0 0
15:25:43 3053 3058 0 26 48 45 0 0
15:25:49 4395 4509 3 5 10 46 0 0

Average 3815 4164 8 14 46 69 0 0


You may have also a too variable buffer cache: what values do you have for dbc_min_pct and dbc_max_pct?

What is the frequency of your CPU's??

Best Regards,

Eric Antunes

[System Support_1]

I faced this problem before, and usually it happens because some applications are looping and wasting your CPU power, I think your only choise will be to reboot your server.

[Tim D Fulford]

Hi

What is on disk c10t6d6 ? to me that is the problem disks. It may be that you simply have lots of LVs on C10t6d6. I prefere RAID1+0 over as many disks/vDisks as possible.

Regards

Tim

[Tim D Fulford]

Sorry I realised I missed off whole chinks of my expaination...

C10t6d6 has very high queues and the service time is very large. 13ms is very poor for this type of disk subsystem. I would have expected 2-3ms (depending on the RAID level it is set at). You are also doing quite large ammount sof IO per IO (this will chew up CPU!!) some 100 blocks or 50kB. I'm guessing this is actually large sequential writes/updates. If it is large sequential updates, then RAID5 may not be the best suited to this (depending on how the array does the update). The reason for this is that an update can mean that 4 IOs are required. the first two to read in the data block & parity blaock. Then another two writes to update the data block and parity. In the "efficient" arrays this can be reduced to 2 IOs as the data and parity blocks are in memory.... Either way all this activity is behind the scens and all the host sees is a fast/slow array.... yours seems to be very slow.

Regards

Tim

[Pedro Cirne]

Hi Guys,

I've just returned from holidays, thank you for your posts.

Those boxes are connected to a brand new EMC DMX Storage, with other servers running without this problem.

I really think this a problem related with I/O and some kernel/IO parameter that's not correct. Any ideias...?

Thks

[Tim Sanko]

Having gone back to the drawing board and finding an irrelevant message telling me to pick the dog up at the groomers, I thought about your system settings. Do you have the storage to play with to do things this way?

If you have a bunch of small metas (hypers) in the 8 GB size range, and if you could consolidate under a large meta and reduce the system load... If you had for example
186 8 GB metas you could reduce your system
overhead with 4 256 GB metas.

To keep the logical layout identical, you would need to put symbolic links in place, but that could help a bit...

Tim

[Pedro Cirne]

Hi,

Thank you all for your support.

Enjoy :)

Pedro