Performance and Memory utilisatiion check?

[rveri]

Hi all ,

I am getting report from users that server is become very slow and application is running very slow.

How to check the performance of memory , as through glance I am getting 100% memory utilisation.

We are having 8 CPU, 16GB RAM

Here is the glance output,
--------------------------

B3692A GlancePlus C.03.50.00 16:23:39 hpsap02 9000/800 Current Avg High
---------------------------------------------------------------------------------------------
CPU Util S SU U | 30% 35% 75%
Disk Util F F | 49% 54% 100%
Mem Util S SU UB B |100% 99% 100%
Swap Util U UR R |100% 99% 100%
--------------------------------------------------------------------------------------------

sar -u 2 5 output.
------------------
HP-UX hpsap02 B.11.11 U 9000/800 08/15/05

16:24:15 %usr %sys %wio %idle
16:24:17 48 2 2 48
16:24:19 38 1 5 57
16:24:21 50 2 1 47
16:24:23 50 0 0 49
16:24:25 44 1 3 52

Average 46 1 2 51
--------------------------------------------

top output
==========
System: hpsap02 Mon Aug 15 16:24:59 2005
Load averages: 0.49, 0.50, 0.49
341 processes: 307 sleeping, 33 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.19 53.2% 0.0% 1.0% 45.8% 0.0% 0.0% 0.0% 0.0%
1 0.41 19.2% 0.0% 2.4% 78.4% 0.0% 0.0% 0.0% 0.0%
2 0.61 17.1% 0.0% 5.2% 77.8% 0.0% 0.0% 0.0% 0.0%
3 0.52 17.1% 0.0% 5.6% 77.4% 0.0% 0.0% 0.0% 0.0%
4 0.60 32.1% 0.0% 5.6% 62.3% 0.0% 0.0% 0.0% 0.0%
5 0.39 29.6% 0.0% 5.0% 65.5% 0.0% 0.0% 0.0% 0.0%
6 0.51 44.2% 0.0% 3.2% 52.6% 0.0% 0.0% 0.0% 0.0%
7 0.66 47.6% 0.0% 5.0% 47.4% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.49 32.5% 0.0% 4.2% 63.4% 0.0% 0.0% 0.0% 0.0%

Memory: 13828264K (3511180K) real, 22813920K (5942588K) virtual, 63552K free Page# 1/27

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 18342 oradav 240 20 2544M 8064K run 65:40 69.06 68.94 oracleDAV
6 ? 25143 oradav 149 20 2544M 8832K sleep 18:51 67.77 67.65 oracleDAV
4 ? 18340 oradav 156 20 2552M 9096K sleep 58:29 67.74 67.62 oracleDAV
0 ? 18252 davadm 155 20 5661M 87796K sleep 4:23 15.36 15.33 dw.sapDAV_DVEBMGS01
1 ? 12218 oradaq 154 20 2616M 12420K sleep 7:14 12.41 12.39 oracleDAQ
1 ? 12256 daqadm 155 20 5636M 73756K sleep 1:14 6.58 6.57 dw.sapDAQ_DVEBMGS00
3 ? 12208 daqadm 128 20 5674M 104M sleep 16:24 6.29 6.28 dw.sapDAQ_DVEBMGS00
5 ? 21847 oradaq 154 20 2616M 13012K sleep 1:30 4.55 4.54 oracleDAQ
5 ? 18253 davadm 155 20 5726M 152M sleep 2:03 4.32 4.31 dw.sapDAV_DVEBMGS01
3 ? 21845 daqadm 154 20 5651M 82020K sleep 2:42 4.11 4.11 dw.sapDAQ_DVEBMGS00
1 ? 16223 daqadm 154 20 5635M 76040K sleep 10:37 3.29 3.29 dw.sapDAQ_DVEBMGS00
2 ? 18358 oradav 154 20 2544M 7032K sleep 5:51 2.92 2.92 oracleDAV
-------------------------------------------

swapinfo -tm
-------------
# swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 16384 2280 14104 14% 0 - 0 /dev/vg00/lvol11
reserve - 18147 -18147
total 20480 20427 53 100% - 0 -
-----------------------------------------

sar -v 2 4
--------------
16:26:57 text-sz ov proc-sz ov inod-sz ov file-sz ov
16:26:59 N/A N/A 341/3620 0 3236/6000 0 8786/16010 0
16:27:01 N/A N/A 341/3620 0 3236/6000 0 8786/16010 0
16:27:03 N/A N/A 346/3620 0 3241/6000 0 8796/16010 0
16:27:05 N/A N/A 341/3620 0 3236/6000 0 8786/16010 0
----------------------------------------------


Please suggest where its going wrong ,and also how to check Physical memory utilisation.

Many Thanks,

R.Veri.
---------

[rveri]

Also attaching the vmstat output:

# vmstat 4 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 1312865 16969 61 20 1 2 0 0 356 2956 12885 1258 8 3 90
1 0 0 1312865 17023 182 21 0 0 0 0 0 2262 5948 413 1 2 97
1 0 0 1312865 16921 138 22 0 0 0 0 0 2265 4686 399 1 1 98
3 0 0 1315226 16897 56 8 0 0 0 0 0 2261 3709 483 6 1 93
3 0 0 1315226 16897 23 2 0 0 0 0 0 2212 2904 445 0 0 99

-------------------------------------------

Can any one help in finding the problem.
Thanks,

[Patrick Wallek]

If glance says you are using 100% memory, then you are.

I also notices that you are paging out to disk to the tune of 2,280 MB (more than 2GB) on your vg00/lvol11 swap area. That is going to hurt performance.

At this point, you need to figure out what is using RAM and either 1) shut some things down so you are not using as much RAM and not paging out, or 2) get more RAM.

[Patrick Wallek]

Your PO rate in the first 3 lines is high (20+), so you are definitely paging out to disk due to a shortage of RAM. So this is definitely hurting performance.

[rveri]

Hi Patrick ,

Do you think , the Physical RAM is falling short. Its 16GB , and swap is using much.

Please suggest.
Many thanks,
R.Veri.

[rveri]

Hi Patric ,

in the vmstat output , the PO rate is in the 9th column , and its not as high as 20.

But its 2 only.

Here is given in more correct way:(pls check 9th column for PO activity. )
------------------------------------
# vmstat 10 6
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 1497120 16223 60 20 2 2 0 0 357 2955 12840 1254 8 3 90
1 0 0 1199946 16151 21 3 3 0 0 0 0 2243 1166 311 0 1 99
1 0 0 773917 15767 56 12 0 0 0 0 0 2222 2708 333 1 1 99
1 0 0 719531 16162 20 0 0 0 0 0 0 2213 1280 305 0 0 99
1 0 0 926867 16158 5 0 2 0 0 0 0 2217 771 295 0 0 99
1 0 0 807955 16158 0 0 0 0 0 0 0 2211 1049 322 1 0 99
-----------------------------------------

9th Column is the PO column. and its less than 20 , its 2 and most of the time 0.

Pls suggest further.

Many Thanks.

[rveri]

Now getting this message . Unix admin as well SAP admin while logging or while running command:

---------------------------------------------------------------------


# swapinfo -tm
sh: The fork function failed. There is not enough memory available.


----------------------------------------------------------------------
You have mail.
${HOME:-.}/.profile: cannot fork: no swap space

You have mail.
${HOME:-.}/.profile: cannot fork: no swap space

----------------------------------------------------------------------
login: davadm
Password:
Please wait...checking for disk quotas
/usr/lib/dld.sl: Call to mmap() failed - DATA /usr/lib/librpcsvc.1
/usr/lib/dld.sl: Not enough space
sh: 12387 Abort(coredump)
--------------------------------------------

Can any one explain or tell , whats the wrong.

many Thanks,

[Patrick Wallek]

You are running out of memory and swap space.

[Sameer_Nirmal]

rveri,

With those statistics , it's very clear that you are running out of memory and swap as well. System can't fork any process anymore as there is no memory left over at all on the system.

I am suspecting that something new has started running on your server. I see 2 oracle instances and 2 SAP processes.
2 Oracle instances are using around 2.5 and 2.6GB of memory (SGA) and SAP processes are consuming 5.6GB memory each. Thus system is running out of memory.

Secondaly , the page-out activity is low(2)which says that paging is not major cause.

As I said before , the new thing has got its own requirement of addressing virtual memory, which it is doing . Thats why you are running out of swap as well.

Do you know any very recent changes being applied to this server????

[Bill Hassell]

You can eliminate the memory errors (fork function failed, no swap space, etc) by adding (temporarily) about 6-8Gb of swap space by creating an lvol that size and using swapon to add the space. That will keep the system running.

Now there is nothing wrong with using dozens of Gbytes of RAM--unless there isn't enough and swapinfo indicates that there is a lot of usage and reservation. The database administrators are taking advantage of large RAM to improve performance but all the processes plus all the shared memory (Oracle's SGA) plus the kernel's buffer cache exceed RAM and thus paging (swap) will tae place, slowing some prcesses drastically.

How do you fix it? Double the amount of RAM (or at least add another 8Gb). Of course, management may be very reluctant to buy more RAM, so choice #2 is to tell the Oracle admins to reduce their SGA or not run so many instances on this one machine. Or the third choice is to add lots more sweap space and accept the inevitable slowdown due to lack of RAM.

[rveri]

Hi Sameer ,

First of let me thank you for very good reply. Well we are running out of memory.
And there is no recent changes from Application side , and from unix side as well.

Recently last week ,the server started to hang at 4pmEST near about and giving error at syslog "File table full" . And no option left except to reboot it .

After that we have increased the kernel parameters , nfiles: 8212-> 16000 , maxfiles 200-> 2048 , nproc 1628-> 3620 , ninode 4000-> 6000. And two/3 days server worked fine and again today users started saying its getting slow down. and then the error message came at 5.00pm EST today.

And also wondering why the Primary swap is not using by the system.

Here is the output:

# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 16777216 2631220 14145996 16% 0 - 0 /dev/vg00/lvol11
reserve - 18283728 -18283728
-------------------

1) Phy. Mem is = 16GB , But Primary swap configured is only =4GB. Does that make major impact. ?

2) Can adding a Filesystem swap , improve the performance and eliminate it.


3)What step can be taken to resolve this.

Expecting your valuable answer on this..

Many Thanks,

RVeri.
-------

[Bill Hassell]

The primary swap won't be used for a while because it's priority is 1 and the second swap area is priority 0, so all of it must be occupied before the primary swap will be used. But used is different than reserved. The kernel will reserve swap space just in case it may be needed and this is what Glance is reporting. The area between the U----U is actually occupied, but the area between R-----R is reserved. So you need more swap space as mentioned. I would not use filesystem swap as it is slower than a raw lvol, but if you have a filesystem with several Gb of unused space, use swapon to add it temporarily.

With a server this big, the system administrator must be monitoring nfile, nproc, memory and swap usage, disk and LAN usage, semaphores and shared memory (and many other items) all the time. And there's security and patches... It is not a good idea to be reactive, that is, wait until something breaks. You probably need a couple of weeks of sysadmin training and help from an HP-UX performance specialist to keep this environment stable.

[rveri]

Bill many thanks , also for your excellent reply and suggestions ,

i)Well , why the primary swap is not using by the system at all.

ii)And how to convince the application (SAP/Oracle) that , Physical Memory is needed to increase or to reduce the Oracle SGA parameter. Is there any calculation.!

iii) If I need to add a extra swap , does it needs a reboot. vg00 is having only 451 Free PE of 8MB size.

Pleaes suggest.

Many Thanks,

Regards,
R.veri.

[rveri]

Thanks Bill,
I was posted the last reply , before I saw your last reply , so can be ignored.

Well , good suggestion.And appreciate it .

As you mentioned that , with a server with this big , the kernel parameters , the semaphores and shared memory , should be monitor. Yes it is monitored. And also tuned recenetly , as per my last post.

But after "file table full" , swapspace error came. And posted this to itrc for guide.

Many Thanks and will be keep on updating.

Thanks Sammer , Bill and all who replied.

[Sameer_Nirmal · Accepted solution]

As said rightly by Bill , primary swap won't be used as it has priority set as 1. Swap device with priority as 0 considered as the first place to be used by the system. When it is fully utilized either the way of reserving or actual usage , secondary swap would be used thereafter. Size os promary swap ( 4GB ) doesn't matter as long you have suffient swap configured in total.
In your case , someone had configured 4GB swap as primary initially who then added 16GB more to take care of swap requirement. Since it is always good to have big contigeous swap at a time , he made the secondary device as promary swap by changing its priority as 0.

Since you have some space on the vg00 , I would advise you to create a lvol and add it using swapon command, You don't need to reboot the system. Having file swap is not as efficient as lvol beacuse it has its own overhead.

Did you change those paramters blindly without knowing the impact of them ? If you increse kerenel parameters too high , then
kernel itself would consume a substantial memoery to cater those changes. Hence memory
for user processes would be less.

How much buffer cache is configured on your system? i.e check dbc_max_pct parameter.

You should pro-actively monitor the system to know its resource utilization using sar , vmstat , iostat ,netstat and GlancePlus , top etc..
Based on those statistics collected by you ,understand them very well and then you can show it to application guys.

I would say increasing swap is short-term resolution. You have to increase system memoery anyhow to take care of application load.

Your oracle guy would know if the SGA could be reduced. But I think that maynot be possible if your application really need that much SGA .

[rveri]

Hi Sameer ,
Thanks once again,

we have changed the abocve parameters based on calculation and formula:
like nproc=20 + (8*MAXUSERS) etc.

here
The dbc_max_pct is :

# kmtune -q dbc_max_pct
Parameter Current Dyn Planned Module Version
===============================================================================
dbc_max_pct 20 - 20
--------------------

Thanks ,

[Bill Hassell]

The formula values are generally useless for large servers and should be replaced by specific numbers. The formulas are just guesses and based on nothing that you are currently running. Glance (use the "t" command) and sar -v will show you the current and maximum values. Based on the sar -v lines from above, both nproc (341 used, 3620 max) and nfile (8786 used, 6010 max) are adequate although the the current number of processes (341) seems low for such a big server. Be sure to take your measurements when the system is very busy.

dbc_max_pct is 20 which means the buffer cache is more than 3Gb in size--far too large. Change your dbc_min_pct to 2 (800 megs) and dbc_max_pct to 4 (1600 megs). That may get you an extra 1000 to 2000 megs of RAM.

[rveri]

Many many Thanks , Bill. And will update you.

[Mahesh Kumar Malik]

Hi

Your environment is more of memory and IO intensive, since disk utilsation also shows 100%. Suggest to ugrade System memory. Swap utilsation of 100% is also alarming , it can be observed further after memory upgrade

Regards
Mahesh

[Raj D.]

Many Thanks Mahesh. I will update further.

[Geoff Wild]

Definitely drop your min_dbc_pct and max_dbc_pct as Bill suggests....I would even drop them both to 2 (Oracle has it's own buffer cache).

As far as swap space - have you had SAP involved? Particularly have you had an Early Watch report done on this system?

Usually, for SAP, you want about 24GB of swap - irregardless of the amount of ram.

Swap can be added without a reboot.

The SAP Early Watch will suggest kernel parameters as well - talk to your Basis folks about having one done - if you havn't already.

Rgds...Geoff

[Victor BERRIDGE]

Hi,
I agree with George (and Bill of course but like George I would reduce further...)

And yes 24 GB swap space would be of some help here...



I would also set the kernel parameter swapmem_on 0



I found that in heavy load, set to 0 is far better


All the best
Victor

[Victor BERRIDGE]

Sorry..
I meant Geoff...

[rveri]

Hi Geoff , and Victor, and all,

Many Thanks for replying on this thread.

Well, we have added a device swap of 8GB , and found performace has improved.

Following observed:
---------------------
1)swap space utilisation , has now reduced from 99% to 71%.
2)Memory utilisation is not showing 100% and its 98% ( Including used and the reservation , usuage is nearly 60% ( U -U , in glance is usage) ) , though will be monitoring further.
3)In vmstat pi , po (Paged in/out ) values reduced to zero , indicating no excesseive swap activity.

Here is the output :
-------------------

ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
B3692A GlancePlus C.03.50.00 15:19:15 ddsapd02 9000/800 Current Avg High
-------------------------------------------------------------------------------------------
CPU Util S | 1% 1% 1%
Disk Util F | 1% 2% 4% 2%
Mem Util S SU UB B | 98% 98% 98%
Swap Util UR R | 71% 71% 71%
-------------------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 2

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 3620 342 9 9
File Table (nfile) 16010 8611 54 54
Shared Mem Table (shmmni) 1024 76 7 7
Message Table (msgmni) 128 2 2 2
Semaphore Table (semmni) 1024 148 14 14
File Locks (nflocks) 1024 623 61 61
Pseudo Terminals (npty) 200 0 0 0
Buffer Headers (nbuf) na 634134 na na
-------------------------------------------


# vmstat 15 10 ; date [ 8th column is pi value ]
Tue Aug 16 14:30:01 EDT 2005
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
3 0 0 1107320 102260 58 20 2 2 0 0 306 3031 13530 1312 8 3 89
4 0 0 846149 102220 136 16 0 0 0 0 0 2130 10015 386 25 2 73
4 0 0 1348310 102222 12 0 0 0 0 0 0 2977 7290 562 27 2 71
4 0 0 2379036 102221 49 25 0 0 0 0 0 3136 9481 714 29 3 67
4 0 0 2374664 102203 14 2 0 0 0 0 0 3349 7391 560 28 2 70
5 0 0 900614 101701 259 40 0 0 0 0 0 3230 12841 665 28 3 69
4 0 0 1159574 102208 31 4 0 0 0 0 0 3052 9657 558 27 3 70
6 0 0 1142202 102070 64 15 0 0 0 0 0 4461 12614 852 28 4 67
3 1 0 1124386 102208 25 5 0 0 0 0 0 3226 10331 581 28 4 69
4 0 0 1004441 102114 16 3 0 0 0 0 0 2835 10045 628 26 3 71
----------------------------------------
Earlier pi/po values used to go high as 6,31,87 .

swapmem_on is 0 .

Many Thanks for all your contributoins and supprot.

Thank you all.

[Bill Hassell]

From your last note:

> SYSTEM TABLES REPORT Users= 2
>
> CPU Util S | 1% 1% 1%
> Disk Util F | 1% 2% 4% 2%

At this moment, your system is doing nothing, no CPU, no disk. This isn't the best time to measure performance.

> swapmem_on is 0 .

That is a major problem! What this means is that you must waste 16Gb of swap space just covering the current RAM. The meaning of the 0 setting is that *every* process, library, shared memory area, etc must have a corresponding location in the swap area or the process will not run. So 16Gb RAM with 16Gb swap means that only 16Gb of processes can run and there can be no additional memory usage.

This 1:1 ratio was part of HP-UX from the beginning but made no sense at all with large RAM systems because in order to use all of RAM, you had to have at least as much swap, none of which could ever be used. That's where the concept of 1.5 to 2x RAM was the recommended swap space.

If you change swapmem_on=1 (the HP-UX default for many years), now your swap space is in addition to RAM (well, a 20% reservation still exists). By setting swapmem_on=1, you now can freely run 30+ Gb of process space. Glance and swapinfo will immediately report a much smaller usage of swap space and the extra 8Gb of swap space won't be needed anymore.