system performance.

[jack wu_2]

platform: V5.1A+PK6
1: how to identify the reason of IOwait is very high, the IOwait > 30
2: the output of vmstat -w 3
display fault cow pin very high, and I hope to find the root cause of this.

> vmstat -w 3 7
Virtual Memory Statistics: (pagesize = 8192)
procs memory pages intr cpu
r w u act free wire fault cow pin pout in sy cs us sy id iowait
18 478 154 350K 297K 120K 13G 1G 3G 0 6K 12K 14K 27 31 37 12
19 475 150 351K 296K 120K 13974 1336 2772 0 6K 87K 10K 50 50 0 30
21 474 149 352K 296K 120K 13190 721 1528 0 7K 112K 11K 51 48 0 34
16 474 151 350K 298K 120K 11560 798 2159 0 4K 113K 9K 55 45 0 21
13 477 153 352K 295K 120K 13728 891 2071 0 7K 121K 12K 53 47 0 9
17 474 149 350K 297K 120K 10470 696 1680 0 7K 100K 12K 58 42 0 16
16 474 150 352K 295K 120K 12065 750 1936 0 4K 116K 10K 62 38 0 5


Thanks
Jack

[Hein van den Heuvel]

Interesting.

First of all, what makes you believe the IOwait is bad.
By explaining that we may be able to help you better.
I am not saying it is good, but it may be 'par for the course'.

For example. If the application mostly does random reads over a large (many GB) database, then IO wait is what you expect and what you will get. Nothing wrong with that.

Ditto for the 'cow'. If your application is constantly forking, then the cow is the natural and expected way to fill the address space for the data for the child.

The 'pin' rate would worry me most, but there again, with lots of forks, that would be the way to load the code pages for the children.

So what is your process creation rate?

What kind of box is this?(Wildfire?)
Roughly what is it supposed to be doing (Oracle?)

Did it ever work to your satisfaction? Is this new behaviour, or something that gradually went worse? Other then the vmstat observations, is there anything actually hurting?

I would start to run a quick kprof / dcpi / or trace or such tool to see what those 100K+ system calls / second are all about, and where all the system time is being burnt.

And I would run iostat (or monitor) to see which disks / domains the IOs are going to, at what rates/sizes.

Good luck!
Hein.

[jack wu_2]

got it, Thanks

Jack