Please help with this swap informations

[Stefan Paitoni]

We have a D-class HP-UX 10.20 with our application running. It's a production system and it's not in our premises so we can not do what we like to trace our problem.
Sometimes (let say every month) the system must be rebooted because the performance is dramatically decreasing without any increase in the workload. So we decide to collect some informations using vmstat. 'vmstat -s' is launched everyday at midnight followed by 'vmstat -z' (to reset the counters). You can see in the attachement a chart designed from those collected data. Can someone help me to interprete those data ?

Thanks.

[Ian Dennison_1]

From where I sit, I can only offer an informed guess. I have done capacity planning and performance analysis on HP-UX before, but with much more site knowledge and information to hand.

What I see is the workload increasing in a linear fashion (constantly) for the period involved. The Operating System is doing progressively more and more work, allocating and freeing up more and more pages (I guess the light orange and mid orange lines are pages allocated and pages freed.)

My suspicion is that the Application is 'leaking'. It is not cleaning up old unix processes / memory and is asking unix for more and more resources, and consequently UNIX is having to do more and more work.

My first port of call would be to patch everything at the OS and firmware level as high as you can (Everyone points the finger of blame at the OS first). Check ITRC for latest releases of patches, for keywords such as 'memory', 'leak', etc.

Once these are sorted, get your Application patched to the latest level, or even register yourself on the Application's own forum and search for problems with similar symptoms.

Share and Enjoy! Ian

[keith persons]

Stefan,

If I'm interpreting the colors correctly and the drop is the reboot, it looks like memory bottleneck. Since this seems to creep up it could be a memory leak, excessive values for caching, or possibly zombie processes.

What's your kernel config? Check for the default value of dbc_max_pct. Is there some consistency or pattern between the required reboots? Also check for zombie or processes tho they typically cause a strain on the cpu not memory, but if processes are terminated with kill -9 they do not release their allocated resources. When were the last patches applied?

Keith

[Sanjeev Kohli_1]

Hi Stefan

What I understand from your question is that the application running on the system is not releasing physical memory .

You need to check whether there is swapping activity happening on the system
# swapinfo -mt

you can run the following command to get a detailed report of the server activity

# sar -o sarfile 5(secs) 10(no. of times)
# sar -AF sarfile > sar.file
# pg sar.file
The first command runs the sar command with all the options.

check the following kernel parameters also.
dbc_min_pct
dbc_max_pct
nfile
ninode
nflocks

have fun,

Sanjeev

[Stefan Paitoni]

Sanjeev and all,

we submit also 'swapinfo' every day at midnight, here is the file :

[V. Nyga]

Hi Stefan,

I can't open your last file. Is he 'gzipped'?

Volkmar

[Stefan Paitoni]

Volkmar,

yes, it's gzipped.