HP UX very slow / hangs

[DnD_1]

Hi guys,

Need help,

Our HP UX 11.11 running oracle, and others charging application constantly hangs, when you issue command to list, move, see or zip large files.

It started to become very slow, then at the end it hangs !

I have looked at the syslog, there's strange messages about

----EMS Monitor Restart ----

Do you think it might be the problem ?

[Kenan Erdey · Accepted solution]

Hi,

check memory usage if it's swapping with glance or with vmstat 5 10 and look at po column. your server may be under memory pressure. also look at top command and if some process is taking extremely high cpu time.

Kenan.

[Sushil Srivastava]

Hi
ems - Description of Event Monitoring Service (EMS).

You must check the load of the server by top or w command .

EMS files live in /etc/opt/resmon and /opt/resmon. The following is a
description of files and directories that might help you as you
navigate through EMS.

/etc/opt/resmon/config

A file that sets the restart interval for monitor persistence.

/etc/opt/resmon/dictionary

Even you need to check the IO wait . Ma any disk is having any proble.

[DnD_1]

hi.

these are what i get from top and vmstat.

top seems to be normal though.

# vmstat 5 10
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 75919 1066 26 18 20 10 5 0 6049 208 3753 174 6 3 91
1 0 1 74352 1140 1 0 9 0 1 0 142 382 417 100 0 0 100
1 0 1 74352 1124 0 0 6 0 0 0 45 375 182 63 0 0 100
1 0 0 74413 1120 0 0 1 0 0 0 13 286 123 43 0 0 100
1 0 0 74413 1330 116 102 33 14 40 0 10602 346 2575 122 2 8 90
1 0 0 96136 1348 39 32 10 3 12 0 3472 361 892 70 0 0 100
1 0 0 96136 1499 14 9 2 0 3 0 1136 250 334 44 0 0 100
1 0 0 92061 1483 6 2 -1 0 0 0 371 158 145 32 0 0 100
1 0 0 92061 1469 2 0 -1 0 0 0 120 127 78 27 0 0 100
1 0 0 74544 1465 0 0 0 0 0 0 38 115 84 27 0 0 100


# top
System: md2 Thu Sep 25 11:44:49 2008
Load averages: 0.61, 0.49, 0.53
96 processes: 93 sleeping, 3 running
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0.61 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 1038812K (203372K) real, 2105952K (307488K) virtual, 17660K free Page# 1/2

Anything else I can check please ?

[SKR_1]

give swapinfo -tam output

Thanks

SKR

[SKR_1]

When you start any command such as ll,mv etc then check from another window with glance foe the disk utilization, which disk is getting highly utilized.

After that run DD commmand on that disk

Thanks

SKR

[DnD_1]

Hi SKR,

The swpainfo,

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 188 1860 9% 0 - 1 /dev/vg00/lvol2
reserve - 1481 -1481
memory 730 143 587 20%
total 2778 1812 966 65% - 0 -


Anyway we can increase the swap ?

For glance, I don't understand what is that ? Do we have to install the glance ? or it came with HP UX ?

Thanks again !!!

[sreekanthtm]

Hi,

Your server is running out of memmory.
You have to find out the process, which is eating up ur valuable resources.

# glace -m
will give a detailed o/p of memory utilisation.
You can use TOP also to find out the process with high memory utilization.
post # swapinfo -atm o/p.


Rgds
Sreekanth

[DnD_1]

Hi...

When I enter glance , it say glance : not found.

Help :(

It already hangs twice today....have to reboot it manually.....

[sreekanthtm]

Reboot will do the job.
But its not the solution, the problem will follow you. Search in google for glance and get installed in your server.
Whats OS version 32/64 bit??

# getconf KERNEL_BITS

If anybody have glance depot please share....

[DnD_1]

hi..

after rebooted, it will come again :(

getconf KERNEL_BITS
64

helps :(

[Ganesan R]

Hi,

Glance is the priced product and you should get it from HP only. You need to identifiy which is causing the system hung. There may be many reasons. System hung mainly due to the CPU/Memory/disk bottlenecks.

Open a seperate session and keep monitor the resource utilisation till you encounter this issue. If you do not have glance use standard HP-UX commands.CPU utilisation can be monitored by top or sar -u commands. Use swapinfo -mat to monitor memory and swap.
For disk i/o use sar -d command.

And also make sure system is really hung or any network chock. Try to ping the server from network. Try to access the server on console.

This way you can narrow down the issue. If you still not able to identify the root cause, take system dump and involve HP to analyse.

Hope this helps

[shankar!]

Hi

I've had the same problem before with one of HP 11.11 server, its under development env so I reinstalled with hp ux 11.23 and it's working perfectly.

[Jeeshan]

you need to have a checklist for investigating the hang issue

1. check all hardware logs.
2. if any SAN attached, then check the controller, disks or disk bottleneck issue
3. check the kernel parameters and its current usage.
4.check the application software and OS patch requirements.