Hi,
> Great find Chris!
> Please explain how you concluded that,
> such that we can all know next time.
> What were the specific clues I missed in
> the data presented?
first hint.
Customer describes the application as "hanging".
K> Our banking application hangs during peak
K> hours
"hanging" equals in most customer
"speaking" to "slow application response time.
Second hint, memory utilisation shows at the time of the hang, via glance, a percentage of 99%.
K>And the memory ultilisation rises to 99% when monitered with glance.
Together the hints are strong enough, to check if there is some sort of memory bottleneck going on..
So check if diskspace was used as RAM..
> TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
> dev 16384 2972 13412 18% 0 - 1 /dev/vg00/lvol2
USED lvol2 device space is different from 0, so at one time, there was certainly a problem with not enough RAM available..
However a definitive indication that the system is paging is the vmstat output.
> #vmstat
> procs memory page
> faults cpu
> r b w avm free re at pi po fr de sr in
> sy cs us sy id
> 1 2 0 1174466 47810 124 2 10 10 1 0 84 5840
> 45456 1522 7 5 88
pi/po equals to 10 pages, free memory equals to 47810 pages or 186Mbyte.
With regards to how the scheduler behaves in low memory situations, at least on HP-UX 11.11 and Im assuming it remains the same on HP-UX 11.23 and only has changed with HP-UX 11.31, there are 4 variables who are important.
lotsfree
gpgslim <-- vhand starts paging "processpages" out to disk (device swapspace)
desfree
minfree <-- processes gets deactivated
lotsfree is the only "constant" value and is set at 1/64th part of physical ram. i.e. in this case 16GB * 1024 / 64 = 256Mbyte.
desfree value, is at least 4 times less then lotsfree, so at most 256Mbyte/4 = 64Mbyte,
minfree is still a lot lower then the desfree value..
gpgslim, varies between lotsfree and desfree, and as when vmstat was measured, pi/po was different of 0, gpgslim is probably set a bit higher then the 186Mbyte mark..
So was the system paging, pages out, from processes, at the time of the "hang", certainly, will that cause a sudden drop of application response time, if the "wrong" pages of the wrong "process" get paged out, and now must be serviced from disk instead of RAM, offcourse..
What would temporarly cause the problem not to occur at the same point in time as now, is setting the dbc_max_pct to lower then 1gb, i.e. dbc_max_pct = 6%. (as at the time of the "hang", swapinfo ,looks to indicate that the buffercache still was around 1.5Gbyte in size).
What will find out the rootcause of the drop in memory, is probably executing kmeminfo/kmeminfo -user/vmstat 1 60 every minute to see where the memory "goes" to just before the "hang" occurs.. i.e if it goes to kernel or to buffercache/user processes, if the latter then needs to be checked if its "normal increased operation" that caused the free memory to drop beneath gpgslim and then probably additional memory needs to be added to cope with it..
Greetz,
Chris
PS. Its a call, not a complaint. ;)
