top shows only 1 CPU taking the load

[makhs]

Hi all,

I have a rx4640 server with 8 processors. The problem is that only the first CPU is taking the load while the other 7 are effectively idle.

Is there a tunable parameter somewhere to address this problem?

Here's the output from top:
Mon Jul 2 16:00:00 SST 2007
System: sgsapd1 Mon Jul 2 16:00:06 2007
Load averages: 0.55, 0.45, 0.45
459 processes: 416 sleeping, 42 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 4.41 94.6% 0.0% 5.4% 0.0% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
2 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
3 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
4 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
5 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
6 0.00 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
7 0.00 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.55 12.0% 0.0% 0.6% 87.4% 0.0% 0.0% 0.0% 0.0%

Memory: 30007980K (7929152K) real, 46718012K (14238396K) virtual, 6864604K free Page# 1/29

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 26543 qasadm 217 20 5664M 84000K run 10:19 18.82 18.79 dw.sapQAS_DVEBMGS02
0 ? 3904 qasadm 216 20 5652M 55608K run 11:00 18.72 18.69 dw.sapQAS_DVEBMGS02
0 ? 3850 qasadm 219 20 5675M 89080K run 2:11 18.64 18.60 dw.sapQAS_DVEBMGS02
0 ? 3854 qasadm 195 20 3925M 9904K run 0:45 8.31 8.30 oracleQAS
0 ? 3366 qasadm 154 20 3924M 8624K sleep 0:30 7.07 7.05 oracleQAS
0 ? 12882 qbwadm 155 20 5235M 80212K sleep 1:39 5.39 5.38 dw.sapQBW_DVEBMGS04
0 ? 3468 qasadm 155 20 5676M 79800K sleep 1:47 3.13 3.12 dw.sapQAS_DVEBMGS02
0 ? 3437 qasadm 155 20 5669M 84584K sleep 0:52 3.03 3.03 dw.sapQAS_DVEBMGS02
0 ? 26591 qasadm 154 20 3925M 10032K sleep 1:25 2.53 2.52 oracleQAS
0 ? 2162 devadm 155 20 5264M 87540K sleep 0:44 2.11 2.10 dw.sapDEV_DVEBMGS01
0 ? 12977 qbwadm 155 20 5234M 83156K sleep 0:03 1.71 1.71 dw.sapQBW_DVEBMGS04
0 ? 3470 qasadm 154 20 3924M 8616K sleep 0:21 1.22 1.22 oracleQAS
5 ? 57 root 152 20 5328K 4736K run 163:10 1.14 1.13 vxfsd
0 ? 12884 qbwadm 155 20 5227M 76620K sleep 0:21 1.05 1.05 dw.sapQBW_DVEBMGS04
0 ? 6219 qasadm 168 20 40464K 26900K sleep 29:37 0.71 0.71 saposcol
0 pts/0 29598 sysadm7 168 20 10736K 1196K sleep 0:18 0.48 0.48 top


thanks.

[Dennis Handly]

Perhaps the 4 processes have forced themselves to be on the same CPU?

Can you make a shell infinite loop to see which processor it runs on?

[Duncan Edmonstone]

I can think of 3 things that might cause this scenario:

1. The application is doing it - most of the stuff here seems to belong to a SAP instance - have you spoken to your SAP admins about this?

2. Someone has bound all the processors to the first CPU using mpsched. What does mpsched -q show?

3. Someone has set up processor sets on this system - what does psrset -p show?

HTH

Duncan

[makhs]

Thank you all for your replies and contributions.

I have logged a case with hp and they have identified that the server was installed with PRM (Process Resource Manager) and it was configured to assign only 1 CPU per application group.

I have disabled PRM from startup and rebooted the server.
The top output now shows an even distribution of CPU resources.

System: sgsapd1 Wed Jul 4 09:40:18 2007
Load averages: 0.03, 0.05, 0.06
452 processes: 415 sleeping, 36 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.03 1.0% 0.0% 1.0% 98.0% 0.0% 0.0% 0.0% 0.0%
1 0.01 0.0% 0.0% 2.0% 98.0% 0.0% 0.0% 0.0% 0.0%
2 0.04 7.9% 0.0% 1.0% 91.1% 0.0% 0.0% 0.0% 0.0%
3 0.02 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
4 0.02 0.0% 0.0% 1.0% 99.0% 0.0% 0.0% 0.0% 0.0%
5 0.06 5.9% 0.0% 0.0% 94.1% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
7 0.03 1.0% 0.0% 0.0% 99.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 2.0% 0.0% 1.0% 97.0% 0.0% 0.0% 0.0% 0.0%

Memory: 28009552K (9910496K) real, 46091944K (15359776K) virtual, 9333080K free Page# 1/19

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
5 ? 7393 qasadm 155 20 5699M 112M sleep 2:35 8.48 8.47 dw.sapQAS_DVEBMGS02
2 ? 7394 qasadm 154 20 5729M 142M sleep 3:31 4.38 4.37 dw.sapQAS_DVEBMGS02
7 ? 57 root 152 20 5256K 4672K run 0:19 1.40 1.40 vxfsd
0 ? 7397 qasadm 154 20 3926M 10992K sleep 1:14 1.34 1.34 oracleQAS
5 ? 5901 devadm 155 20 5259M 82984K sleep 0:54 0.94 0.94 dw.sapDEV_DVEBMGS01
7 ? 5351 dbwadm 155 20 5260M 71708K sleep 0:07 0.82 0.81 dw.sapDBW_DVEBMGS03
7 ? 4859 dbwadm 168 20 40336K 26788K sleep 0:24 0.72 0.72 saposcol
3 ? 7399 qasadm 155 20 5660M 71724K sleep 0:05 0.67 0.66 dw.sapQAS_DVEBMGS02
4 ? 7401 qasadm 154 20 3926M 10992K sleep 0:25 0.65 0.65 oracleQAS
7 ? 5902 devadm 155 20 5261M 84264K sleep 0:38 0.61 0.61 dw.sapDEV_DVEBMGS01
2 ? 7412 qasadm 154 20 3924M 8880K sleep 0:00 0.57 0.56 oracleQAS
7 ? 7439 qasadm 155 20 5655M 73492K sleep 0:03 0.44 0.44 dw.sapQAS_DVEBMGS02
6 ? 5903 devadm 155 20 5255M 76796K sleep 0:12 0.41 0.41 dw.sapDEV_DVEBMGS01
0 ? 7391 qasadm 154 20 210M 7296K sleep 0:00 0.28 0.28 gwrd
0 ? 37 root 152 20 720K 640K run 0:02 0.25 0.25 schedcpu
7 ? 18 root 191 20 648K 576K run 0:01 0.23 0.22 ksyncer_daemon
0 ? 2 root 128 20 72K 64K sleep 0:04 0.23 0.22 vhand
4 ? 7395 qasadm 155 20 5677M 90564K sleep 1:17 0.16 0.16 dw.sapQAS_DVEBMGS02
2 ? 38 root 191 20 576K 512K run 1:41 0.16 0.16 pagezerod
5 ? 7585 qasadm 152 20 49156K 11576K run 0:00 0.15 0.15 igsmux_mt
6 ? 7459 qasadm 154 20 3925M 9920K sleep 0:01 0.14 0.14 oracleQAS
1 ? 2390 root 154 20 7584K 832K sleep 0:05 0.14 0.14 sendmail:
3 ? 5905 devadm 154 20 3909M 9968K sleep 0:21 0.13 0.13 oracleDEV
5 ? 5446 dbwadm 152 20 49156K 11576K run 0:00 0.13 0.13 igsmux_mt

[Duncan Edmonstone]

Yes, PRM can use PSETs to manage CPUs into different logical groups. It can also use the fair share scheduler to assign CPU resource based on a percentage (which is the one more commonly used).

HTH

Duncan