Possible Server Performance Issue

[Steve Burt_1]

Hi Rick, I know its strange but sadly true....

Tasks: 94 total, 2 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 85.1% us, 0.5% sy, 0.0% ni, 14.1% id, 0.1% wa, 0.0% hi, 0.2% si
Mem: 4086548k total, 2618700k used, 1467848k free, 134508k buffers
Swap: 2031608k total, 0k used, 2031608k free, 1098932k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9453 admin 15 0 2916 884 692 R 2 0.0 0:00.01 top
1 root 16 0 2892 552 472 S 0 0.0 0:01.22 init
2 root RT 0 0 0 0 S 0 0.0 0:01.48 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.27 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.53 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.28 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3
10 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/0
11 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/1
12 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/2
13 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/3
14 root 6 -10 0 0 0 S 0 0.0 0:00.49 khelper
15 root 15 -10 0 0 0 S 0 0.0 0:00.00 kacpid
47 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0
48 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1
49 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/2
50 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/3
60 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
61 root 15 0 0 0 0 S 0 0.0 0:00.25 pdflush
63 root 7 -10 0 0 0 S 0 0.0 0:00.00 aio/0
64 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1
65 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/2
66 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/3
51 root 15 0 0 0 0 S 0 0.0 0:00.00 khubd
62 root 25 0 0 0 0 S 0 0.0 0:00.00 kswapd0
139 root 25 0 0 0 0 S 0 0.0 0:00.00 kseriod
217 root 5 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/0
218 root 7 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/1
219 root 5 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/2
220 root 11 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/3
228 root 15 0 0 0 0 S 0 0.0 0:05.49 kjournald
1067 root 6 -10 2832 464 380 S 0 0.0 0:00.00 udevd
1488 root 19 0 0 0 0 S 0 0.0 0:00.00 kjournald
1644 root 17 0 4248 1360 1124 S 0 0.0 0:00.06 rc
1720 root 16 0 2536 456 396 S 0 0.0 0:13.60 cpuspeed
2171 root 16 0 1704 644 552 S 0 0.0 0:00.16 syslogd
2175 root 15 0 3064 484 420 S 0 0.0 0:00.01 klogd
2185 root 16 0 2980 484 412 S 0 0.0 0:02.19 irqbalance
2229 root 16 0 5912 1052 876 S 0 0.0 0:00.35 rpc.idmapd
2268 rpc 16 0 1924 604 512 S 0 0.0 0:00.00 portmap
2325 ntp 16 0 6324 6324 3436 S 0 0.2 0:02.45 ntpd
2371 root 18 0 1616 552 480 S 0 0.0 0:00.00 acpid
2664 root 16 0 112m 1100 776 S 0 0.0 0:02.10 hpasmd
2694 root 16 0 12572 1332 684 S 0 0.0 0:00.16 cmapeerd
2695 root 16 0 2308 1392 744 S 0 0.0 0:00.73 cmathreshd
2696 root 16 0 12504 1244 864 S 0 0.0 0:35.92 cmahostd
2736 root 16 0 22280 1032 772 S 0 0.0 0:00.35 cmastdeqd
2743 root 15 0 64332 924 692 S 0 0.0 0:05.00 cmahealthd
2745 root 16 0 4016 1236 620 S 0 0.0 0:27.49 cmaperfd
3042 root 15 0 23760 6324 2632 S 0 0.2 0:13.36 snmpd
3051 root 16 0 8804 2124 1692 S 0 0.1 0:00.05 cupsd
3085 root 16 0 4204 1668 1376 S 0 0.0 0:00.00 sshd
3098 root 16 0 3192 860 728 S 0 0.0 0:00.00 xinetd
3116 root 16 0 7880 3124 2084 S 0 0.1 0:01.16 sendmail
3124 smmsp 16 0 6876 2616 1796 S 0 0.1 0:00.00 sendmail
3134 root 16 0 2772 540 468 S 0 0.0 0:00.06 gpm
3143 root 16 0 4708 1116 724 S 0 0.0 0:00.04 crond
3174 xfs 16 0 3932 1688 864 S 0 0.0 0:00.03 xfs
3191 root 16 0 3680 740 624 S 0 0.0 0:00.00 atd
3200 dbus 16 0 3592 1236 1092 S 0 0.0 0:00.01 dbus-daemon-1
3210 root 16 0 4732 552 492 S 0 0.0 0:00.06 rhnsd
3219 root 16 0 7796 5216 1600 S 0 0.1 0:23.22 hald

[rick jones]

The Phantom Process strikes again!-) I think the profile route is the next thing to do to see what if anything is actually consuming CPU cycles. Alas, I know Caliper and q-syscollect, but both of those are Itanium - oprofile exists for x86 but I am unable to suggest command syntax.

Could also compare ps -ef output to the process listing of top I suppose.

[Steve Burt_1]

The Phantom process has been discoverd.

Interogating Oracle produced a piece of memory intensive SQL running 5000 a second..

Apologies for delay sickeness and paternity leave delayed a responce

Thanks for all your help...