Server performance very slow

[Pulse001]

I have a rp3440 server which is working quite well but at times is hanging and during the periods of bottleneck i am not able to login as well as in previously logged in sessions i am not able to execute any command at all. The server comes back to normal operation after 10 mins without any intervention from my side.After that the server works fine for 2-3 hrs.

Can this error possibly bcoz the system is using up space from Hard disk for use in memory ?

[OFC_EDM]

Is anything filling up / (root)?

There may be a process filling up root with tempory files and preventing login. Then when the process cleans up the temporary files you can log back in.

It can also be a hardware error. Check your logs for any space issues and your EMS event logs for any memory errors...or any hardware errors. I've seen a post where a problem with a DIMM caused the server to hang up sporadically.

Cheers

[Jeeshan]

you need to have a checklist for investigating the hang issue

1. check all hardware logs.
2. if any SAN attached, then check the controller, disks or disk bottleneck issue
3. check the kernel parameters and its current usage.
4.check the application software and OS patch requirements.

[Dennis Handly]

>system is using up space from Hard disk for use in memory?

If you are talking about swapspace, you can use "swapinfo -tam" to see how much is being used.
And "sar -w" for paging stats.

[Ganesan R]

Hi,

Your symptoms looks like memory bottleneck. When the system consumes all of its memory, then its starting paging in and out. If you try to login or start new process when the system heavily doing paging, you may not get any result. System act like freezed.

Open a session and keep monitoring the resource utilisation using glance or command line utilities like swapinfo, vmstat till you encounter the issue. Also monitor other resources(CPU, disk, etc) utilisations. Either one could be the issue.

[prasadb]

you can check with the disk by

sar -d 1 5
and also use, vmstat -nd 1 5
it will Display the default output and disk transfers

[Pulse001]

But i can't run any command when the system is freezed. How can i check the commands lika sar, vmstat when i am not able to execute any command ?

[Pulse001]

Is there any kernel parameter i can change regarding paging ? i.e page_file_size

[Suraj K Sankari]

Hi,
If possible check which are the unwanted process are running and which are taking more memory.
If those process are not reqired or old porecess try to kill those process.

If Oracle or SAP is working then check the oracle dead oracle process.

Suraj

[Ganesan R]

Hi,

I agree that you cannot run any commands when the system is already hungry for memory.

If you have glance installed, start the tool and keep monitor which resource is being hitted and reaching maximum. When system freeze still you will be able to see the results but may be little bit slow since this process is already running. This way you can narrow down the issue. Even you can run top command to continiously monitor the CPU usage.

Sometime improperly configured kernel setting may cause issue. best example is dbc_min_pct and dbc_max_pct. These parameter define the physical memory allotted for filesystem buffer. If your application is not utilisting the amount alloted then it is wast of memory. This again depends on your requirement. Check these values and tune as per your requirement. In idle situation min 5% and max 10% is sufficient.

[Pulse001]

I am attaching sar o/p





ggnqgldd1: /> sar -u 5 5

HP-UX ggnqgldd B.11.31 U 9000/800 09/29/08

16:29:13 %usr %sys %wio %idle
16:29:18 46 14 2 38
16:29:23 56 27 1 16
16:29:28 50 27 1 22
16:29:33 49 26 1 24
16:29:38 36 27 1 36

Average 47 24 1 27
ggnqgldd1: /> sar -v 5 5

HP-UX ggnqgldd B.11.31 U 9000/800 09/29/08

16:36:50 text-sz ov proc-sz ov inod-sz ov file-sz ov
16:36:55 N/A N/A 599/4096 0 1154/34816 0 3334/2147483647 0
16:37:00 N/A N/A 599/4096 0 1154/34816 0 3334/2147483647 0
16:37:05 N/A N/A 601/4096 0 1156/34816 0 3336/2147483647 0
16:37:10 N/A N/A 601/4096 0 1156/34816 0 3336/2147483647 0
16:37:15 N/A N/A 601/4096 0 1156/34816 0 3336/2147483647 0

ggnqgldd1: /> sar -w 5 5

HP-UX ggnqgldd B.11.31 U 9000/800 09/29/08

16:37:27 swpin/s bswin/s swpot/s bswot/s pswch/s
16:37:32 0.00 0.0 0.00 0.0 352
16:37:37 0.00 0.0 0.00 0.0 3325
16:37:42 0.00 0.0 0.00 0.0 1790
16:37:47 0.00 0.0 0.00 0.0 527
16:37:52 0.00 0.0 0.00 0.0 977

Average 0.00 0.0 0.00 0.0 1394
ggnqgldd1: /> vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
3 0 0 254345 3551 1 0 4 0 0 0 0 994 1217 161 1 2 97
2 1 0 242829 3473 0 0 3 0 0 0 0 1222 5043 549 4 24 72
2 1 0 242829 3780 0 0 0 0 0 0 0 1095 2400 364 2 25 73
6 0 0 198708 3856 0 0 0 0 0 0 0 1057 1548 313 3 26 72
6 0 0 198708 4277 0 0 0 0 0 0 0 1903 15624 1516 62 16 22
ggnqgldd1: /> swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 594 7598 7% 0 - 1 /dev/vg00/lvol2
reserve - 1887 -1887
memory 3894 1715 2179 44%
total 12086 4196 7890 35% - 0 -

[Pulse001]

Hi,

Ganesan my OS is 11.31 and i am not able to find dbc_min_pct and dbc_max_pct in kctune o/p. Are these parameters not available in 11i v3 ?

[Dennis Handly · Accepted solution]

>vmstat 5 5

I don't see that many Page Outs.

>swapinfo -tam
>total 12086 4196 7890 35%

Were these outputs during the slowdown?

>my OS is 11.31 and iI am not able to find dbc_min_pct and dbc_max_pct in kctune. Are these parameters not available in 11i v3?

These were obsoleted. See dbc_max_pct(5). This points you to the replacement parms:
filecache_max and filecache_min

[Mike Miller_8]

Try running 'crontab -l' and compare any scheduled jobs to the times that the system hangs(becomes slow) and see if you have any scheduled jobs that are the cause.

[Pulse001]

Friends i came to know that the user's were running some processes which were consuming a lot of memory and thus the server was undersized.