A500 Suddenly running slow. Can't Figure out Why.

[Eric Bakken]

Hello all,

Yesterday I found that my Test/DR system was running very slow while trying to do most things. It's timing out when daily automated jobs try to authenticate and send files from another server. At first I thought this would suggest a network error but I've found that I get the same latency when logging in via the console. It takes about 3 minutes.

Some other examples would be that it takes about 3 mintues to get a response after issuing a ping command.

And it takes about 5 - 10 minutes for sam to start up and then after that it takes a long time to "initialize sam"

There aren't any processes hosing up the system that I can see so I'm wondering if anyone had any other suggestions. Thanks in advance,

-Eric

[TTr · Accepted solution]

This sounds like a classic case of heavy disk swapping.

Check the ps listing for runaway processes and stop any non critical apps and daemons even temporarily.


Check environment with
swapinfo
swapinfo -tam
vmstat

[Eric Bakken]

Well, I don't see any runaway processes:

System: orion Thu May 8 14:30:11 2008
Load averages: 0.01, 0.01, 0.01
127 processes: 111 sleeping, 16 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
1 0.02 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 130644K (90776K) real, 384980K (312580K) virtual, 3900308K free Page# 1
/12

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 37 root 152 20 1120K 1120K run 0:28 0.43 0.43 vxfsd
0 ? 1812 root 152 20 215M 11456K run 1:21 0.36 0.36 prm3d
1 pts/0 6608 root 178 20 11796K 9856K run 0:00 0.59 0.25 top
1 ? 1936 root 152 20 9852K 2112K run 0:06 0.20 0.20 rds
0 ? 1811 root 127 20 32828K 2876K sleep 0:17 0.16 0.16 scopeux
1 ? 1332 root 152 20 15124K 3300K run 0:01 0.16 0.16 dmisp
1 ? 1863 root 152 20 13428K 2316K run 0:01 0.16 0.16 agdbserve
1 ? 1862 root 152 20 16384K 4060K run 0:05 0.14 0.14 rep_serve
0 ? 6602 sshd 154 20 6756K 520K sleep 0:00 0.12 0.10 sshd:

#
Here is the output form your suggested commands. I'm not sure what to make of it.
#
#
#
# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 0 1048576 0% 0 - 1 /dev/vg00/lvol2
reserve - 179120 -179120
memory 3678004 207976 3470028 6%
#
#
#
#
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
reserve - 175 -175
memory 3592 203 3389 6%
total 4616 378 4238 8% - 0 -
#
#
#
# vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 77326 978118 3 1 0 0 0 0 0 413 13063 132 2 1 97
#
#
#

[Eric Bakken]

Attempt at reformatting:



System: orion Thu May 8 14:34:34 2008
Load averages: 0.01, 0.01, 0.01
126 processes: 110 sleeping, 16 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 130272K (88860K) real, 384308K (311000K) virtual, 3900548K free Page# 1
/12

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 37 root 152 20 1120K 1120K run 0:28 0.35 0.35 vxfsd
0 ? 1812 root 152 20 215M 11456K run 1:21 0.33 0.33 prm3d
1 ? 1936 root 152 20 9852K 2112K run 0:06 0.18 0.18 rds
1 ? 1332 root 152 20 15124K 3300K run 0:01 0.16 0.16 dmisp
1 ? 1863 root 152 20 13428K 2316K run 0:01 0.16 0.16 agdbserve
1 ? 1862 root 152 20 16384K 4060K run 0:05 0.14 0.14 rep_serve
1 ? 1864 root 152 20 14424K 2392K run 0:04 0.10 0.10 alarmgen
1 ? 1798 root -16 20 27012K 6552K run 0:35 0.10 0.10 midaemon
0 pts/0 6625 root 178 20 11796K 9856K run 0:00 0.10 0.09 top




#
#
#
#
# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 0 1048576 0% 0 - 1 /dev/vg00/lvol2
reserve - 181568 -181568
memory 3678004 208112 3469892 6%
#
#
#
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
reserve - 178 -178
memory 3592 203 3389 6%
total 4616 381 4235 8% - 0 -
#
#
#
# vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
3 0 0 77901 977687 3 1 0 0 0 0 0 413 13177 132 2 1 97
#
#
#
#

[TTr]

Your system looks clean. The next things that come to mind are name resolution and flaky hardware that delays I/O.
Can you look into that?

nslookup
nslookup
ping
ping
check /etc/nsswitch.conf, /etc/resolv.conf, errors in syslog etc.

Can you correlate with something that happened when all this started?

[Eric Bakken]

Turns out it was my /etc/resolv.conf

Looks like the nameserver got changed to a completely bogus IP.

I've modified resolv.conf and everything is faster. Thanks a bunch for your help!

-Eric