high swap allocations

[Scott D. Allen]

I have a problem with a K360 server and performance. The machine averages ~60 users during the day, running Visibilty on Oracle. 1.6GB, 3x180Mhz CPU, bunch-o-disks (SCSI). I'm seeing (during working hours), pretty high swap allocation, ~95% at peak times, ~85% average. Is this normal? What's the fix, if there is one or am I just being paranoid? It appears that the performance suffers when there are a few reports running and I know that we have a disk bottleneck when that is happening (100% disk util and high CPU) but I'm starting to wonder if swap alloc is an issue also. Suggestions? I'm open to anything.

--Scott

[Rick Garland]

If you have Glance look and see what is doing this.
Also, sar and vmstat would be some good choices (if you don't have Glance)

[Stefan Farrelly]

If you can, install Measureware. This will allow you to see historical data in graphical form showing swap usage which you can then drill down to see which processes are running which could be hogging it. The best way to view these graphs is to use PerfView. HP should let you have copies on a free trial.

[Scott D. Allen]

I have glance and am using it, but it doesn't really help with determining who's using swap allocation. Is there a way to see that with glance?

[Stefan Farrelly]

Hi Scott,

Glance will give you a rough view of how much swap a process is using; Go to the view showing processes (g) and look at the size in memory of each one (RSS). The larger a process is in memory, the more swap it is reserving (not using - remember HP needs to reserve swap in case it needs it later). Anyway, for your purposes the bigger the process the more of your 'Mb/Pct USED' swap its using.

[Anthony deRito]

Scott, use swapinfo -tma and check out your total figures. use the PCT USED column and see if any number are high. Your primary swap devices should all read very low PCT USED numbers. If these are high (>50%) then you are experiencing some page out activities.

Next check vmstat. Look at the procs colums (the r,b&w colums). This will tell you how many processes are currently in the run queue and how many of them are blocked for resources.

If you notice many processes blocked on resources, use the Process List window of GlancePlus and check out the process wait states. You chould be able to see what type of events the process is blocked on. Depending on what you find here, you may or may not determine it is VM related.

Do not be concerned about the 100% disk utilization so much as to concentrate on the application and how much time the process is spending in kernel or system mode.


When you say "high CPU" on reports, it is important to determine what is actually happening while creating the reports and what resources is the process waiting for. It does not have to be the disk.

When you say "high CPU", is the CPU in user mode or system mode? sar -u will tell you this. If it in kernel mode, use Glance in the Process Resource screens to find out what types of system calls are being made.

It is not enough to look at disk utilization. What is more important is to compare the service time with the wait time of each request to your disk. Use sar -d for these figures.

Hope this info helps.

Tony

[Scott D. Allen]

Thanks guys. I'm starting to think a little more clearly about this.

I found the Global Wait State screen in glance and it's showing
~22% Sleep, ~14% System, ~2% IO, ~4% Semaphore, ~4% Pipe, ~1% Terminal, ~1% Socket and ~53% Other. That, to me, seems pretty acceptable, I'll just watch that and as we see a dip in performance, check back to see if Disk IO or VM goes up and go from there.

--Scott

[Scott D. Allen]

So what does it mean if I see a process producing a whole lot of blocks from vmstat and then in glance it shows "wait reason: pri" and/or "wait reason: cache"?

[James R. Ferguson]

Scott:

"PRI" (in Glance) means the process (thread) was waiting for its PRIority to become high enough to get the CPU.

"CACHE" means the time the process (thread) was waiting for the filesystem buffer cache to be updated.

GLANCE has some very good drill-down, inbuilt help. The above is lifted from it.

...JRF...