- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- System load is very high (Load averages: 15.28, 15...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-10-2010 05:34 AM
тАО05-10-2010 05:34 AM
There is one HP-UX server (10.20) running on K-580 series hardware in our environment.
For a past few days I am seeing that the system load is more and more sleeping process (217) but CPU is normal.
Example:
# top
Load averages: 15.28, 15.27, 15.27
218 processes: 217 sleeping, 1 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 3.15 0.2% 0.0% 0.4% 99.4% 0.0% 0.0% 0.0% 0.0%
1 15.19 0.0% 0.0% 2.0% 98.0% 0.0% 0.0% 0.0% 0.0%
2 20.00 0.2% 0.0% 0.2% 99.6% 0.0% 0.0% 0.0% 0.0%
3 22.76 0.0% 0.0% 7.5% 92.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 15.28 0.0% 0.0% 2.6% 97.4% 0.0% 0.0% 0.0% 0.0%
And there is a following issue with sendmail in that same server.
ps -ef | grep -i sendmail
root 10308 1 0 Apr 28 ? 0:22 sendmail: rejecting connections on port 25: load average: 15
root 15884 1 0 Dec 13 ? 0:00 sendmail: BAA15880: from queue
root 3816 24919 1 23:00:22 pts/54 0:00 grep -i sendmail
How to resolve the issue.
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-10-2010 06:41 AM
тАО05-10-2010 06:41 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
It will be useful if you provide the complete output of "top" (I mean the 1st screen), "vmstat 5 5" and "swapinfo -tam" to see what's going on.
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-10-2010 07:07 AM
тАО05-10-2010 07:07 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
load average is calculated by the average number of processes waiting for CPU.
This figure is kind of high.
Bill Hassell provides a good example in his presentations of a system with very high load average and no other issues.
So lets go through a process to see if anything needs to be done at all.
1) Is there a response complaint? Do users report slow access or access difficulty.
If no, consider doing nothing. If yes, continue.
2) Does the system need to receive mail? 99% of all systems running sendmail daemon don't need to because they receive no inbound mail.
If no, consider doing nothing. If yes, identify the processes using up CPU time, identify their application and do further analysis.
Quite often bouncing a service resolves this problem.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-10-2010 10:05 PM
тАО05-10-2010 10:05 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
http://it.toolbox.com/wiki/index.php/Determining_the_Cause_of_System_Performance_Problems_for_HP-UX . It could provide something useful.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-10-2010 11:46 PM
тАО05-10-2010 11:46 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
System load is 15.28 & cpu utilization free ....
Means processes are getting blocked somewhere else..
It can be memory,disk or network.
Find out where you have bottleneck ??
You can use these utilities for this sar,vmstat,iostat,galnce,gpm,ovpm
http://h71028.www7.hp.com/enterprise/w1/en/os/hpux11i-kod-v3-performance-troubleshooting.html
this is very good training performance troubleshooting.
Gudluck
Prasanth
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 02:05 AM
тАО05-11-2010 02:05 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Check vmstat, top, swapinfo -tam , what is going on there and also check from sar output.
Cheers//
Taifur
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 09:21 AM
тАО05-11-2010 09:21 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
#vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 61 0 5498 747277 78 96 0 0 0 0 0 0 211 15 2 10 88
0 61 0 5249 747247 149 201 0 0 0 0 0 736 1535 174 1 4 95
0 61 0 5549 747247 50 64 0 0 0 0 0 720 519 92 0 1 99
0 61 0 5327 747247 16 20 0 0 0 0 0 716 183 70 0 0 100
0 61 0 3554 747247 5 5 0 0 0 0 0 721 80 64 0 0 100
#sar
HP-UX lgprime B.10.20 A 9000/800 05/10/10
00:00:00 %usr %sys %wio %idle
00:10:01 0 2 0 98
00:20:00 0 1 0 98
00:30:00 0 2 0 98
00:40:01 0 2 0 98
00:50:00 0 2 0 98
01:00:00 0 1 0 98
01:10:00 0 2 0 98
01:20:00 0 2 0 98
01:30:00 0 2 0 98
01:40:00 0 1 0 98
01:50:00 0 1 0 98
02:00:00 0 2 0 98
02:10:00 0 2 0 98
02:20:00 0 2 0 98
02:30:00 0 1 0 98
02:40:00 0 2 0 98
Average 0 2 0 98
#sar -d
HP-UX lgprime B.10.20 A 9000/800 05/10/10
00:00:00 device %busy avque r+w/s blks/s avwait avserv
00:10:01 c2t6d0 1.04 1.60 1 12 10.01 16.93
c2t5d0 0.86 0.75 1 10 5.88 17.00
c5t0d0 0.01 0.50 0 0 1.42 17.48
c6t1d1 0.18 0.50 0 4 2.22 20.55
00:20:00 c2t6d0 1.12 0.55 1 15 3.51 19.99
c2t5d0 0.95 0.55 1 14 3.56 19.33
c5t0d0 0.03 0.50 0 0 1.00 15.47
c6t1d1 0.10 0.50 0 2 0.55 18.70
00:30:00 c2t6d0 1.12 0.51 1 14 3.41 13.60
c2t5d0 0.84 0.50 1 9 3.24 14.51
c5t0d0 0.11 0.50 0 0 3.20 20.83
c6t1d1 0.10 0.50 0 2 0.56 18.94
00:40:01 c2t6d0 0.85 0.52 1 9 3.07 14.85
c2t5d0 0.69 0.50 1 8 2.99 14.80
c5t0d0 0.03 0.50 0 0 3.38 18.54
c6t1d1 0.09 0.50 0 2 0.43 19.44
00:50:00 c2t6d0 1.25 0.67 1 17 4.03 19.95
c2t5d0 1.05 0.64 1 15 4.16 19.54
c5t0d0 0.03 0.50 0 0 0.53 18.93
c6t1d1 0.18 0.50 0 4 1.35 20.20
01:00:00 c2t6d0 0.77 0.52 1 8 2.85 14.55
c2t5d0 0.61 0.50 1 7 2.70 13.97
c5t0d0 0.02 0.50 0 0 1.73 16.26
c6t1d1 0.09 0.50 0 2 0.36 20.17
01:10:00 c2t6d0 1.42 1.12 2 16 7.11 14.47
c2t5d0 1.22 0.86 2 14 6.14 13.64
c5t0d0 0.06 0.50 0 0 2.79 17.06
c6t1d1 0.09 0.50 0 2 1.09 19.17
01:20:00 c2t6d0 2.45 0.57 3 22 4.63 12.41
c2t5d0 1.01 0.66 1 15 3.98 19.08
c5t0d0 0.01 0.50 0 0 0.96 16.96
c6t1d1 0.52 0.61 1 38 5.37 7.96
01:30:00 c2t6d0 0.85 0.56 1 9 3.05 15.02
c2t5d0 0.72 0.50 1 8 2.66 14.57
c5t0d0 0.03 0.50 0 0 1.33 19.54
c6t1d1 0.13 0.50 0 4 1.45 20.00
01:40:00 c2t6d0 0.73 0.56 1 8 3.03 14.82
c2t5d0 0.60 0.50 1 7 2.72 14.59
c5t0d0 0.01 0.50 0 0 1.35 15.06
c6t1d1 0.09 0.50 0 2 0.36 19.76
01:50:00 c2t6d0 1.29 0.93 2 17 5.85 18.91
c2t5d0 1.05 0.96 1 15 5.77 18.63
c5t0d0 0.02 0.50 0 0 0.97 16.66
c6t1d1 0.46 0.50 0 6 2.98 28.75
02:00:00 c2t6d0 1.11 0.51 1 15 3.14 14.00
c2t5d0 0.81 0.50 1 9 2.88 14.34
c5t0d0 0.10 0.50 0 0 2.97 21.38
c6t1d1 0.09 0.50 0 2 0.35 19.76
02:10:00 c2t6d0 1.07 1.10 1 12 7.87 18.31
c2t5d0 0.89 0.66 1 10 4.96 17.12
c5t0d0 0.03 0.50 0 0 1.69 18.45
c6t1d1 0.11 0.50 0 2 0.66 20.33
02:20:00 c2t6d0 1.19 0.58 1 16 3.49 20.38
c2t5d0 1.00 0.55 1 14 3.52 19.16
c5t0d0 0.01 0.50 0 0 1.87 17.74
c6t1d1 0.09 0.50 0 2 0.43 18.65
02:30:00 c2t6d0 0.78 0.50 1 8 2.61 15.53
c2t5d0 0.70 0.50 1 8 2.70 15.00
c5t0d0 0.02 0.50 0 0 0.69 17.93
c6t1d1 0.10 0.50 0 2 0.33 18.71
02:40:00 c2t6d0 0.80 0.52 1 8 2.87 14.53
c2t5d0 0.62 0.50 1 7 2.80 13.78
c5t0d0 0.02 0.50 0 0 1.53 18.35
c6t1d1 0.11 0.50 0 2 0.87 18.25
Average c2t6d0 1.11 0.72 1 13 4.65 15.94
Average c2t5d0 0.85 0.63 1 11 4.03 16.47
Average c5t0d0 0.04 0.50 0 0 2.17 18.78
Average c6t1d1 0.16 0.54 0 5 2.75 16.08
#top
System: lgprime Mon May 10 02:45:50 2010
Load averages: 15.27, 15.27, 15.27
221 processes: 220 sleeping, 1 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 3.67 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 16.31 0.0% 0.0% 4.0% 96.0% 0.0% 0.0% 0.0% 0.0%
2 17.06 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
3 24.03 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 15.27 0.0% 0.0% 1.0% 99.0% 0.0% 0.0% 0.0% 0.0%
Memory: 39172K (23996K) real, 48156K (33168K) virtual, 2988932K free Page# 1/19
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 615 root 154 20 180K 232K sleep 1542:07 0.51 0.51 syncer
1 ? 3 root 128 20 0K 0K sleep 405:47 0.14 0.13 statdaemon
1 ? 7 root -32 20 0K 0K sleep 395:11 0.13 0.13 ttisr
1 ? 941 root 154 20 396K 236K sleep 70:26 0.11 0.11 rpc.statd
0 ? 19 root 100 20 0K 0K sleep 286:04 0.07 0.07 netisr
0 ? 1166 root 154 20 8568K 1896K sleep 12:13 0.06 0.06 rpcd
0 pts/54 10981 root 178 20 984K 212K run 0:00 1.00 0.05 top
#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1536 0 1536 0% 0 - 1 /dev/vg00/lvol2
reserve - 74 -74
memory 2880 349 2531 12%
total 4416 423 3993 10% - 0 -
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 10:15 AM
тАО05-11-2010 10:15 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Lets see some vmstat output if that command is available on 10.20 HP-UX.
Average c2t6d0 1.11 0.72 1 13 4.65 15.94
Average c2t5d0 0.85 0.63 1 11 4.03 16.47
Average c5t0d0 0.04 0.50 0 0 2.17 18.78
Average c6t1d1 0.16 0.54 0 5 2.75 16.08
This shows the disks listed above are being worked a little hard.
Problem seems to be heavy writes. Combine that with a near full file system and you will get severe delays.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 10:26 AM
тАО05-11-2010 10:26 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Please post:
# UNIX95=1 ps -e -o pcpu,pid,ppid,args | sort -rn | head -n 20
# UNIX95=1 ps -e -o vsz,pid,ppid,args | sort -rn | head -n 20
# uptime
# ps -ef | wc -l
# who -u | wc -l ; netstat -n | grep EST | wc -l
# glance # first page.
# top # first page.
# sar -u -M 2 3
# vmstat 3 5
# sar -d 3 10
Cheers,
Raj.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 10:32 AM
тАО05-11-2010 10:32 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Well, from above data , disks are showing medium high busy , but that would not cause that much of load average that goes 15+,
.
>>>
2t5d0 0.86 0.75 1 10 5.88 17.00
c5t0d0 0.01 0.50 0 0 1.42 17.48
c2t5d0 0.95 0.55 1 14 3.56 19.33
c5t0d0 0.11 0.50 0 0 3.20 20.83
c6t1d1 0.10 0.50 0 2 0.56 18.94
>>>
- avwait and service time more than 100 can cause little impact, but this is medium high.
How is the uptime and performance today.
Hth,
Raj.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 10:43 AM
тАО05-11-2010 10:43 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Since the data you're showing indicates nothing that would generate that high of a cpu load, not in memory, not in cpu, not in i/o...
Consider a scheduled rebooting of the server. I think some tables are a bit out of whack and have possibly overflowed their numeric boundaries, which have make your email system think that it should not deliver email because the load is too high. A reboot will probably fix the problem. Especially if this system has been up for a long period of time, like I suspect it has.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 08:01 PM
тАО05-11-2010 08:01 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
You can also check that issue by rebooting server. But before rebooting server , take down time.
Cheers/
Taifur
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-11-2010 08:09 PM
тАО05-11-2010 08:09 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
What is uptime saying?
How long this server is up?
If server is running more then 100 or 200 then better go for bounce the server once and monitor the server for load average.
Suraj
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-12-2010 06:24 AM
тАО05-12-2010 06:24 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
#uptime
11:39pm up 212 days, 13:36, 1 user, load average: 15.27, 15.27, 15.27
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-12-2010 06:51 AM
тАО05-12-2010 06:51 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-23-2010 08:59 PM
тАО05-23-2010 08:59 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
Check the attached performance tunning guide as attached which will help you.
Rgds-Kranti
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-23-2010 09:07 PM
тАО05-23-2010 09:07 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
After rebooting the system, please collect the following output and post
# uptime
# swapinfo -tam
# sar -Mu 5 10
# sar -d 5 10
# sar -v 5 10
# iostat 5 10
# top
# top -d2 -s10 -f /tmp/toperf.txt
# exit
Rgds-Kranti
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-24-2010 03:14 AM
тАО05-24-2010 03:14 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
I will update you after doing diagnoses.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-24-2010 04:39 AM
тАО05-24-2010 04:39 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
http://hpux.ws/system.perf.sh
You might find these collection scripts originally written by HP somewhat more comprehensive.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-25-2010 02:01 PM
тАО05-25-2010 02:01 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
61/4 = 15.25
who -u
check which processes, increases, the process list, day after day (ps -el)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО06-10-2010 04:42 AM
тАО06-10-2010 04:42 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
1) I am not able to understand "61/4 = 15.25", please explain this?
2)Why do you want to run "who -u"?
3)what is the difference between "ps -ef" and "ps -el"?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО06-10-2010 04:55 AM
тАО06-10-2010 04:55 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
> 1) I am not able to understand "61/4 = 15.25", please explain this?
Look at your 'vmstat' data and *read* the manpages for the meaning of the values!
> 2)Why do you want to run "who -u"?
What happens if a client session (as from a terminal emulator) aborts? Do you have old sessions, perhaps consuming resources running?
> 3)what is the difference between "ps -ef" and "ps -el"?
Again, run the commands; look at the different output; *read* the manpages and see what you might learn.
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО06-10-2010 05:16 AM
тАО06-10-2010 05:16 AM
Solution> 61/4 = 15.25
"vmstat 5 5" output showed 61 blocked processes.
4 is the number of processors in the system.
61/4, equals, to the part in the load average that, is the result of the blocked processes.
I.e. if you wouldnt have had any blocked processes, the load average would have been
load average: 15.28 (-15.25= 0.03), 15.27 (-15.25 = 0.02), 15.27 (-15.25 = 0.02)
i.e.
load average: 0.03, 0.02, 0.02
who -u
Im pretty sure the blocked processes are shell processes, that couldnt exit "normally", thats why, who -u would show them.
ps -el / ps -elf, gives the wait reason in hex, not that interesting, but just a hunch from me.. anyway blocked processes should there be allready for a long long time, doing nothing .. and as you mentioned that the amount of processes constantly increases, this is probably because of the number of blocked processes increasing, checking for a few days which processes are causing the process list to increase, should also lead you to the blocked processes..
killing, kill -9, these blocked processes, once you find them, would then 'workaround' the problem..
Greetz,
Chris
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО06-10-2010 04:19 PM
тАО06-10-2010 04:19 PM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
A sleeping process is doing nothing so they do not contribute to the load -- unless these processes wake every few milliseconds, run a few instructions and then go to sleep again. That may be exactly what is happening. What are the 217 processes? Are they shells? If so, these may be orphaned logins caused by users that do not logout properly and the SIGHUP normally enabled in /etc/profile has been turned off.
Because of the long time (200+ days), it is highly likely that these processes weren't terminated when the user disconnected and they are spinning around trying figure out what to do. The load factor is a measure of the number of processes waiting to run or actually running. These orphaned processes are probably running for a short time several times a second. That's why there is almost no user time recorded.
I have also seen this as a largepages issue with 10.20 coupled with a fairly messed up patch problem with disk mirroring. The symptom is that the largepages table is constantly filled. The fix is to disable large pages in the kernel (after a reboot).
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО06-21-2010 07:21 AM
тАО06-21-2010 07:21 AM
Re: System load is very high (Load averages: 15.28, 15.27, 15.27) and more sleeping process (217)
#who -u
ze4egi pts/56 Jun 19 22:35 . 18532 10.151.0.24
#ps -ef | grep -i sh
root 4 0 0 Oct 10 ? 28:26 unhashdaemon
root 38 0 0 Oct 10 ? 65:55 vx_iflush_thread
root 42 0 0 Oct 10 ? 16:03 vx_logflush_thread
he4b2s 5441 1 0 Apr 7 pts/27 0:23 -sh
medcpx 1677 1 0 Apr 28 pts/50 0:16 -sh
he4b2s 2447 1 0 Apr 7 ? 0:25 -sh
ze4egi 17305 1 0 Jan 20 ? 0:58 -sh
root 1982 1 0 Oct 10 ? 0:00 /sbin/sh /usr/dt/bin/dtrc
te9euu 12057 11469 0 Oct 20 ? 1:28 -sh
yezkh1 4681 1 0 Oct 10 ? 1:27 -sh
ze4egi 12156 1 0 Apr 11 ? 0:23 -sh
root 16460 1 0 Jan 20 ? 0:00 -sh
yezkh1 5346 1 0 Oct 10 ? 1:28 -sh
ze4egi 17403 1 0 Jan 24 ? 0:49 -sh
he4b2s 14177 1 0 Mar 31 ? 0:24 -sh
yezkh1 5878 1 0 Oct 10 ? 1:28 -sh
ze4egi 19037 1 0 Apr 7 ? 0:24 -sh
medcpx 27347 1 0 Apr 28 ? 0:17 -sh
root 1810 1 0 Oct 10 ? 47:07 /bin/ksh /usr/local/vaxprint/vpdaemon.sh
he4b2s 15157 1 0 Mar 31 ? 0:29 -sh
ze4egi 11703 1 0 Oct 19 ? 1:19 -sh
ze4egi 21160 1 0 Nov 15 ? 1:14 -sh
ze4egi 5692 1 0 Apr 11 ? 0:21 -sh
ze4egi 16150 1 0 Jan 20 ? 0:57 -sh
ze4egi 21024 1 0 Oct 19 ? 1:24 -sh
he4b2s 7411 1 0 Mar 31 ? 0:29 -sh
medcpx 19129 1 0 Apr 28 ? 0:16 -sh
ze4egi 16257 1 0 Oct 19 ? 1:22 -sh
he4b2s 24449 1 0 Apr 4 ? 0:24 -sh
medcpx 1640 1 0 Apr 28 pts/49 0:16 -sh
ze4egi 20107 1 0 Jan 20 ? 0:49 -sh
he4b2s 5618 1 0 Apr 7 pts/28 0:26 -sh
medcpx 19538 1 0 Apr 28 ? 0:15 -sh
ze4egi 21184 1 0 Dec 5 ? 1:04 -sh
he4b2s 2878 1 0 Apr 5 ? 0:25 -sh
ze4egi 3729 1 0 Oct 20 ? 1:20 -sh
medcpx 14521 1 0 Apr 28 pts/42 0:20 -sh
he4b2s 3028 1 0 Apr 7 ? 0:25 -sh
ze4egi 8183 1 0 Apr 8 ? 0:23 -sh
he4b2s 24603 1 0 Mar 31 ? 0:29 -sh
root 18047 1 0 Oct 19 ? 0:00 -sh
ze4egi 10615 1 0 Oct 18 ? 1:25 -sh
ze4egi 23396 1 0 Apr 8 ? 0:24 -sh
medcpx 4661 1 0 Apr 26 ? 0:19 -sh
root 11469 1 0 Oct 20 ? 0:00 -sh
medcpx 19474 1 0 Apr 28 ? 0:19 -sh
ze4egi 11632 1 0 Oct 20 ? 1:31 -sh
medcpx 12828 1 0 Apr 28 pts/53 0:19 -sh
medcpx 1184 1 0 Apr 28 pts/47 0:17 -sh
root 20553 18532 0 22:40:52 pts/56 0:00 -sh
ze4egi 14359 1 0 Oct 20 ? 1:27 -sh
medcpx 23647 1 0 Apr 28 ? 0:16 -sh
ze4egi 18532 18530 0 22:35:29 pts/56 0:00 -ksh
ze4egi 7204 1 0 Feb 9 ? 0:46 -sh
pe45my 28841 1 0 May 23 ? 0:09 -sh
medcpx 10906 1 0 Apr 28 pts/52 0:18 -sh
medcpx 3898 1 0 Apr 28 pts/51 0:18 -sh
medcpx 12077 1 0 Apr 27 ? 0:17 -sh
medcpx 20616 1 0 Apr 27 ? 0:17 -sh
ze4egi 10065 1 0 Apr 11 ? 0:23 -sh
medcpx 6372 1 0 Apr 26 ? 0:23 -sh
medcpx 1478 1 0 Apr 28 pts/48 0:16 -sh
medcpx 9335 1 0 Apr 28 pts/55 0:15 -sh
medcpx 24261 1 0 Apr 26 ? 0:20 -sh
medcpx 6621 1 0 Apr 28 ? 0:17 -sh
root 29243 20553 1 00:29:02 pts/56 0:00 grep -i sh
he4b2s 4142 1 0 Apr 7 ? 0:24 -sh
medcpx 3640 1 0 Apr 26 ? 0:18 -sh
he4b2s 26069 1 0 Apr 7 ? 0:29 -sh
medcpx 1024 1 0 Apr 28 pts/46 0:17 -sh
medcpx 849 1 0 Apr 28 pts/45 0:18 -sh
Note: I have tried to kill "sh" but not able to kill.
For example,
# ps -ef | grep -i ze4egi
ze4egi 17305 1 0 Jan 20 ? 0:58 -sh
ze4egi 12156 1 0 Apr 11 ? 0:23 -sh
ze4egi 17403 1 0 Jan 24 ? 0:49 -sh
ze4egi 19037 1 0 Apr 7 ? 0:24 -sh
ze4egi 11703 1 0 Oct 19 ? 1:19 -sh
ze4egi 21160 1 0 Nov 15 ? 1:14 -sh
ze4egi 5692 1 0 Apr 11 ? 0:21 -sh
ze4egi 16150 1 0 Jan 20 ? 0:57 -sh
ze4egi 21024 1 0 Oct 19 ? 1:24 -sh
ze4egi 16257 1 0 Oct 19 ? 1:22 -sh
ze4egi 21632 16460 0 Jan 20 ? 0:47 -su
ze4egi 20107 1 0 Jan 20 ? 0:49 -sh
root 4212 20553 1 00:42:33 pts/56 0:00 grep -i ze4egi
ze4egi 19175 18047 0 Oct 19 ? 1:26 -su
ze4egi 21184 1 0 Dec 5 ? 1:04 -sh
ze4egi 3729 1 0 Oct 20 ? 1:20 -sh
ze4egi 8183 1 0 Apr 8 ? 0:23 -sh
ze4egi 10615 1 0 Oct 18 ? 1:25 -sh
ze4egi 23396 1 0 Apr 8 ? 0:24 -sh
ze4egi 11632 1 0 Oct 20 ? 1:31 -sh
ze4egi 14359 1 0 Oct 20 ? 1:27 -sh
ze4egi 18532 18530 0 22:35:29 pts/56 0:00 -ksh
ze4egi 7204 1 0 Feb 9 ? 0:46 -sh
ze4egi 10065 1 0 Apr 11 ? 0:23 -sh
#kill -9 17305
#ps -ef | grep -i 17305
ze4egi 17305 1 0 Jan 20 ? 0:58 -sh
My Questions:
1)How to solve the issue?
2)How to enable SIGHUP in /etc/profile?