Proliant Support Pack 8.20 -> Load always at 1

[atqquebec]

Hi,

I've upgraded my PSP to 8.20 to support RHEL 5.3. Ever since this day, the load is constantly 1 on all my servers that are usually idle. I can't see what process that is using CPU or IO resouces in "top".

Kernel 2.6.18-128.1.1.el5 x86_64

Any idea of solution?

[David Altobelli]

I think that is due to a change in the hpilo driver design. The cmasm2d process (from hp-snmp-agents) waits for SNMP traffic and it is constantly polling for traffic, which eats up resources. You can work around this by adjusting the PFLAGS=â -p 30 $MOUSFLAGSâ line in /opt/hp/hp-snmp-agents/server/etc/cmasm2d to read PFLAGS="-p 30 -w 5 $MOUSEFLAGSâ .

[Andrew Stanczak]

I've got exactly the same problem, PSP 8.20 and Scientific Linux 5.3 (which is really RH5) running on bunch of BL480.

OK, in my case changing PFLAGS entry reduced load slightly (from 1 to 0.7) but it is still huge comparing to the load with version 8.15 installed on these host previously (usual load was like 0.1).

To be frank I find this difficult to understand that software which does really the same job may put different load when installed in different version.

I would be very grateful for any hint.

Best,
Andrew


# uname -a
Linux amaranth
2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 06:58:14 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
# uptime
10:16:55 up 13 days, 17:39, 1 user, load average: 1.15, 1.16, 1.11
# uptime
12:45:01 up 13 days, 20:07, 1 user, load average: 0.72, 0.71, 0.70

[OOS Unixbeheer]

We are experiencing the same problem.

We noticed that there are in fact _two_ instances of cmasm2d running. Well, one is quasi permanent in deferred (D in ps) state. This is causing the load, although we think it does not really burdon the system.

# ps aux | grep cmasm2d
root 10886 0.0 0.0 2144 980 pts/0 S 17:40 0:00 cmasm2d -p 30
root 10889 0.0 0.0 2100 336 pts/0 D 17:40 0:00 cmasm2d -p 30
root 21090 0.0 0.0 3912 680 pts/1 S+ 18:45 0:00 grep cmasm2d



Stracing the deferred one reveals a clue:
# strace -f -p 10889 2>&1 | head -n 5
Process 10889 attached - interrupt to quit
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)
nanosleep({0, 1000000}, NULL) = 0
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)

File descriptor 3 is in our case the character device /dev/hpilo/d0ccb4.



Killing the deferred one is one way of dealing with this load issue. We don't know what the impact of killing is, but it'll make the load drop anyways.


The two instances /don't/ seem to be identical, e.g. fd 3 of the 'normal' one id for the character device /dev/hpilo/d0ccb5.


FWIW1: Seem it on a HP ProLiant DL360 G3 box
FWIW2: We have upgraded (from 8.15)

[OOS Unixbeheer]

To disable the RIB agent, simply add cmasm2d to the exclude line in /opt/hp/hp-snmp-agents/cma.conf.

Ours now reads:
exclude cmasm2d cpqriisd cmarackd


Note: first stop the snmp-agents, then edit the file, then start the agents again. Otherwise, it will actually /skip/ the stopping part :-).



What does this agent do anyway?

[David Altobelli]

The process you noticed in D state is waiting for SNMP traffic on the iLO NIC. That means it is reading from hpilo, waiting for a packet.

Basically, cmasm2d handles SNMP information for the management processor (the MP info you see in SMH, SNMP pass through traffic).

If you want to disable cmasm2d, you can modify /opt/hp/hp-snmp-agents/cma.conf, and add cmasm2d to the exclude line, ie:

########################################################################
# exclude is used to exclude individual agents from starting by default.
# Simply add the name of the agent(s) after the exclude directive.
########################################################################
exclude cmasm2d

Or, if you modify the PFLAGS as above, but with slightly different syntax:

PFLAGS="-p 30 -r 0 -w 5 $MOUSEFLAGS"

With that, I see a load ~0.3. The "w" option is a sleep between polling for this SNMP traffic, but only when there is no traffic, so you could also increase this number and probably see your usage go down, with the impact of a longer (first time only) latency if you do shove SNMP traffic through the MP.

[victor leija]

If we disable cmashd2 by adding it to the exclude line in cma.conf file, will we lose and SNMP trap information?

[atqquebec]

I've been told by HP that 8.25 would solve that. I upgraded a few servers to 8.25, nothing changed, even with all firmware updated.

I'm trying to talk to someone at HP about that, but I always get to talk to people who don't even know what is the load.

I saw the "solution", but I want something from HP.

[David Altobelli]

Patrick, the fix will be in 8.2.6 and 8.3.0. I'm sorry that it slipped out of 8.2.5.

[atqquebec]

Finally, I've been told to always update the firmware before updating PSP. So I have updated the firmware, but needed a re-install of the PSP so that the problem disappear. Problem solved.