Performances analysis

[Delphine TALLARON]

Hi all,

I am new to HP-UX but have some experience with solaris and linux. We have a PA-RISC, 2 cores, 6 Go of RAM, HP-UX 11.23. Applications (Oracle Application) sometimes hang. Usual work: where is the problem? System, applications?

I have used commands like vmstat, sar, swapinfo ... And I have some questions around their interpretation under HP-UX. (Note: I know Glance exists. But I don't use it for the moment as we have no license and no knowledge). You will find in attached text file some time stamped extracts.

My first question is about memory: avm and free. The values I use are coming from attached file at 10:15:39:

- What is the value of a page size? I think it is 4 Ko. Right?
- So does "avm = 1842748" mean "my virtual memory size is about 7198 Mo?
- And free = 16592 means "I have something like 64 Mo free physical memory"?
- May I say: "since virtual memory is 7198 Mo it would be a good thing that the system has 8 Go of physical RAM"?
- Yes but at the same time, swapinfo -ta shows (details in attachment) 14013680 Mo of total KB used? So should I have 14 Go or 8 Go of physical RAM? May be the answer is: swapinfo shows reserved memory, so potential needs, and vmstat shows real needs through "avm"?

My second question is still about memory, at 09:59:29. I was used to consider high "sr" values as memory bottleneck symptom. As my system has only 6 Go of RAM, I am not very surprised by the high "sr" value at 09:59:29. I can see "po" is also high. Ok. But...:

- There is no activity in "si/s"o columns. man pages say that "si/so" are "processes swapped in/out". The facts are that my system has a lack of memory (I think ...) and there is high "sr" and page out activity at this time. So paging/swapping should occur (I think again ...). Why "si/so" are null? Does it take care only of text segments, not of datas? I can see with "sar -w" that there is also no swap activity.

- Despite I suspect a lack of memory, there is not so much moment where "po" and "sr" have high values: something like 10 times per hour... So do I really miss memory?

==> In summary, I am confused.

Third one, about disks:

- What about "%wio" in output of sar -u, what is it doing here??? Is it really a metric that shows disk bottleneck as some said? Well, when I compare vmstat and sar, I can see that idle cpu from vmstat becomes now %wio cpu with sar. Through many readings (too much!) I understand that %wio is not really cpu time consuming. It's just a state in which a process is. So when there is a lot of IO and few CPU activity, %wio can be high but it does not necessary mean that there is a disk bottleneck. Conversely, if there is still a lot of IO but high CPU activity, %wio can be low despite we could have at this moment a real disk bottleneck. Right?

- So, if %wio is not a good metric for "disk bottleneck", do you think that column "procs/b" in vmstat is a better one? In my example, I find at least 55, in a constant way. I think it's very high. But is it due to lack of memory, though there is not so much paging activity, or disk saturation?

- And first, is a high "procs/b" value really a problem? I took a look at output of "sar -d" (no output in the attachment : 37 disks, it would be too much !!!) : on the 2 internal SCSI system disks (vg00) very good avserv, most under 10ms, but many avwait peak till 40 ms, some at 100 ms and corresponding avque peak above 10, sometimes 50 ! On the 35 others SAN disks, I find relatively good avserv, most below 10ms with many between 10ms and 20ms, and very good low avwait and avque.

==> In summary, I obtain with sar a rather correct disc activity but sometimes disrupted by peaks, against a strong but constant value with proc/b. I am unable to reconcile peaks from sar and constant "procs/b" from vmstatâ ¦

Well, I will stop there. I have still many questions. But I give up, I will ask an other dayâ ¦

I think you have many readings to advice. But I have also many. So what I expect is your personal advices and experiences rather than new abstract readings.

Thxs in advance for your valuable help

[Hasan Atasoy]

hi delphine ;

what about dbc_min_pct and dbc_max_pct kernel parameters ?

[Delphine TALLARON]

Bonjour Hasan,

Big problems to access forum today :-(

dbc_min_pct is 5% and dbc_max_pct is 15%. I can see that during working hours bufpages is ~300 Mo and reachs 820 Mo from 17h00. So the fact that this value fluctuates strongly means (I think) that there is a memory pressure.

Thanks Hasan for your answer. But what I need, is more understanding some metrics rather than directly solving my problem. If we can solve, nice, but it is not the main goal of this thread.

In a summary, i need to understand :

Memory needs :

- "avm" and "free" from vmstat
- swapinfo with focus on "reserve" and "memory". I show at the end of this post why I am confused with swapinfo

Swapping, paging :

- "si/so", "pi/po", "sr" from vmstat.
- at which moment can i say there is a memory problem.

Disk bottleneck or not :

- "%wio" in sar and "proc/b" in vmstat
- what means high "avwait" in sar -d (OS bottleneck ?) and high "avserv" (storage bottleneck ?) ?

Here is why I am confused with swapinfo. On an Itanium box HP-UX 11.23, I can play with, I did the following test :

- system doing noting, "swapmem_on 0", got the following swapinfo output :

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 435668 -435668
total 4194304 435668 3758636 10% - 0 -


- system still doing noting, "swapmem_on 1", got the following swapinfo output :

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 403708 -403708
memory 8358072 968352 7389720 12%
total 12552376 1372060 11180316 11% - 0 -

If I think to "Total KB used" in terms of "here are my potential memory needs" there is a mistake somewhere, no ?

DT

[Dennis Handly]

(Please provide the swapinfo -tam values. Not using -m makes it harder to understand. :-)

>If I think to "Total KB used" in terms of "here are my potential memory needs" there is a mistake somewhere, no?

Yes, the total includes the reserve which is VM that is reserved but may or may not be used yet. What mistake do you think there is? It says you "need" 1.37 Gb.

[Olivier Masse]

Salut

I won't directly answer your question, but... first of all, I'd strongly recommend learning to use glance/gpm as these tools are more easy to use than sar, iostat, vmstat etc (in my opinion). Especially vmstat which is not well documented. GPM is the graphical version of glance, it's bulkier but easier to use.

If you have the Enterprise OE or more, it comes without any need to purchase a license. I require EOE as a minimum for all my systems for this reason among others. I think there is a time-limited evaluation of glance available, look at www.hp.com/go/softwaredepot.

Also take a look at the following whitepaper, it gives a quick approach to tuning your system:
http://h21007.www2.hp.com/portal/download/files/unprot/devresource/Docs/TechPapers/UXPerfCookBook.pdf

Good luck

[Delphine TALLARON]

Bonjour,

Olivier, I definitively can't use glance at the moment, see my first post. You said vmstat, and others in my opinion, are not fully discussed. That's why I am here. I had read the Performance cookbook before, but I had left it in a corner since it's a general discussion and glance is the tool used to get metrics. But I do like the conclusion : "Performance tuning is not a science; it is more like a mixture of art, witchcraft, a little smoke (and mirrors), and a dash of luck (possibly drugs)." I do agree !

Denis, would you be a little bit lazy ;-) ? See attached document with freshly collected values.

"What mistake do you think there is?"

Well, on the same system, doing nothing, following "swapmem_on" is 1 or 0, I see 1271 Mo or only 322 Mo as potential need. I can't imagine one second that it is the truth, ant that my system will suddenly need 949 more Mo. What is exactly the meaning of "used Mo" on the line "memory" ?

DT

[Victor BERRIDGE]

Greetings Delphine,

What are your possibilities of testing? (compiling a new kernel and reboot taht is...)
From what I see I would start by:
leave dbc_min_pct at 5% but let dbc_max_pct be 8%.
Then swapmem_on = 0

And see what results you get here...
Next, you didnt mention the sga size or where the memory is used... If oracle is using loads of buffers there is no need for the system to also and here you can tune by reducing what has been done above and also at filesystem level if you are using vxfs and have (you should...) online-jfs

All the best
Victor

[Delphine TALLARON]

Well, thanks every body but your answers do not give me valuable help to interpret some metrics in a reasonable way. Anyway, I think that my initial approach was weak : too many questions, too many attached data. Shame on me

So I close this tread and will open new ones with fewer questions in each.

TD.

[Dennis Handly]

>See attached document with freshly collected values.

Thanks.

>following "swapmem_on" is 1 or 0, I see 1271 Mo or only 322 Mo as potential need.

Ah, with the new values it is obvious now.

>So I close this tread and will open new ones

That's the following link and it looks like Don has answered you questions.
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1182252

[Delphine TALLARON]

Thanks Dennis, got it this morning in my mailbox ;-)

DT