ProLiant Servers - Netservers
cancel
Showing results for 
Search instead for 
Did you mean: 

Proliant Support Pack 8.20 -> Load always at 1

atqquebec
Advisor

Proliant Support Pack 8.20 -> Load always at 1

Hi,

I've upgraded my PSP to 8.20 to support RHEL 5.3. Ever since this day, the load is constantly 1 on all my servers that are usually idle. I can't see what process that is using CPU or IO resouces in "top".

Kernel 2.6.18-128.1.1.el5 x86_64

Any idea of solution?
21 REPLIES
David Altobelli
Occasional Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

I think that is due to a change in the hpilo driver design. The cmasm2d process (from hp-snmp-agents) waits for SNMP traffic and it is constantly polling for traffic, which eats up resources. You can work around this by adjusting the PFLAGS=â -p 30 $MOUSFLAGSâ line in /opt/hp/hp-snmp-agents/server/etc/cmasm2d to read PFLAGS="-p 30 -w 5 $MOUSEFLAGSâ .
Andrew Stanczak
Occasional Collector

Re: Proliant Support Pack 8.20 -> Load always at 1

I've got exactly the same problem, PSP 8.20 and Scientific Linux 5.3 (which is really RH5) running on bunch of BL480.

OK, in my case changing PFLAGS entry reduced load slightly (from 1 to 0.7) but it is still huge comparing to the load with version 8.15 installed on these host previously (usual load was like 0.1).

To be frank I find this difficult to understand that software which does really the same job may put different load when installed in different version.

I would be very grateful for any hint.

Best,
Andrew


# uname -a
Linux amaranth
2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 06:58:14 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
# uptime
10:16:55 up 13 days, 17:39, 1 user, load average: 1.15, 1.16, 1.11
# uptime
12:45:01 up 13 days, 20:07, 1 user, load average: 0.72, 0.71, 0.70
--
"That is the worst and most contemptible plan since Olaf the Hairy, high chief of all the Vikings, ordered 80,000 battle helmets with the horns on the inside" - Edmund Blackadder
OOS Unixbeheer
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

We are experiencing the same problem.

We noticed that there are in fact _two_ instances of cmasm2d running. Well, one is quasi permanent in deferred (D in ps) state. This is causing the load, although we think it does not really burdon the system.

# ps aux | grep cmasm2d
root 10886 0.0 0.0 2144 980 pts/0 S 17:40 0:00 cmasm2d -p 30
root 10889 0.0 0.0 2100 336 pts/0 D 17:40 0:00 cmasm2d -p 30
root 21090 0.0 0.0 3912 680 pts/1 S+ 18:45 0:00 grep cmasm2d



Stracing the deferred one reveals a clue:
# strace -f -p 10889 2>&1 | head -n 5
Process 10889 attached - interrupt to quit
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)
nanosleep({0, 1000000}, NULL) = 0
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)
read(3, 0x94a9308, 4096) = -1 EAGAIN (Resource temporarily unavailable)

File descriptor 3 is in our case the character device /dev/hpilo/d0ccb4.



Killing the deferred one is one way of dealing with this load issue. We don't know what the impact of killing is, but it'll make the load drop anyways.


The two instances /don't/ seem to be identical, e.g. fd 3 of the 'normal' one id for the character device /dev/hpilo/d0ccb5.


FWIW1: Seem it on a HP ProLiant DL360 G3 box
FWIW2: We have upgraded (from 8.15)
OOS Unixbeheer
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

To disable the RIB agent, simply add cmasm2d to the exclude line in /opt/hp/hp-snmp-agents/cma.conf.

Ours now reads:
exclude cmasm2d cpqriisd cmarackd


Note: first stop the snmp-agents, then edit the file, then start the agents again. Otherwise, it will actually /skip/ the stopping part :-).



What does this agent do anyway?
David Altobelli
Occasional Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

The process you noticed in D state is waiting for SNMP traffic on the iLO NIC. That means it is reading from hpilo, waiting for a packet.

Basically, cmasm2d handles SNMP information for the management processor (the MP info you see in SMH, SNMP pass through traffic).

If you want to disable cmasm2d, you can modify /opt/hp/hp-snmp-agents/cma.conf, and add cmasm2d to the exclude line, ie:

########################################################################
# exclude is used to exclude individual agents from starting by default.
# Simply add the name of the agent(s) after the exclude directive.
########################################################################
exclude cmasm2d

Or, if you modify the PFLAGS as above, but with slightly different syntax:

PFLAGS="-p 30 -r 0 -w 5 $MOUSEFLAGS"

With that, I see a load ~0.3. The "w" option is a sleep between polling for this SNMP traffic, but only when there is no traffic, so you could also increase this number and probably see your usage go down, with the impact of a longer (first time only) latency if you do shove SNMP traffic through the MP.
victor leija
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

If we disable cmashd2 by adding it to the exclude line in cma.conf file, will we lose and SNMP trap information?
atqquebec
Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

I've been told by HP that 8.25 would solve that. I upgraded a few servers to 8.25, nothing changed, even with all firmware updated.

I'm trying to talk to someone at HP about that, but I always get to talk to people who don't even know what is the load.

I saw the "solution", but I want something from HP.
David Altobelli
Occasional Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

Patrick, the fix will be in 8.2.6 and 8.3.0. I'm sorry that it slipped out of 8.2.5.
atqquebec
Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

Finally, I've been told to always update the firmware before updating PSP. So I have updated the firmware, but needed a re-install of the PSP so that the problem disappear. Problem solved.
Ugo Bellavance (ATQ)
Frequent Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

finally, the problem is not solved... as my operations just killed hp-snmp-agents... when I restarded it and the load came back above 1. I'll wait for 8.26.
Rabie Van der Merwe
Occasional Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

Has anyone been given an eta on 8.26 or 8.30?
kheller2
Frequent Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

What is the -r flag for in PFLAGS, it isn't defined in the help flags for the binary.

Also, will the fix be in a new hp-health RPM? Can you even tell us with the fix is if it isn't a binary change but a flag or script change?

The upsetting thing is this was directly reported by us to HP a while ago and we got completely ignored.
João Carlos Mendes Luis
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

Has anybody an open formal support case with HP? Will it help if I open one?
David Altobelli
Occasional Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

830 should be available via the web in September.

The flags are undocumented because it is not normal for them to be configured by hand. Basically, those flags control how often cmasm2d checks for a SNMP packet passed through from the iLO. The fix will change the default wait times in the binary.
kheller2
Frequent Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

Shouldn't there be a notification posted about this? Granted it isn't critical but a lot of folks aren't going to find this little fix.

Suporte SO
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

I really don't think that changing the default wait times is the real fix. Why can't this be fixed by removing the busy wait and use a full wait for the hpilo device?

The process should attempt to read or poll, and sleep waiting for a packet. That's Unix way of doing stuff.

For now, I will simply disable this process in all our servers.
OOS Unixbeheer
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

PSP 8.30 is out. Has anybody tested 8.30 wrt the load problem?
iceman76
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

We have a fresh install of SuSE Linux Enterprise Server 11 64 bit with the latest HP-Management Software

hp-health-8.3.0.43-19
hp-snmp-agents-8.3.0.27-17

Problem ist still there. With cmasm2d running there is a load of 0.6 - 0.7, without it there is a load of 0.05 - 0.10

Andrew Stanczak
Occasional Collector

Re: Proliant Support Pack 8.20 -> Load always at 1

Hiya, just tested this with 64bit CentOS release 5.3 and Proliant Support Pack 8.3.0 running on DL380g6:

- load with default PSP settings: 0.31, 0.33, 0.28

- load with manually modified cma.conf (exclude cmasm2d cpqriisd cmarackd): 0.02, 0.06, 0.08

So yep, I think problem is still there.

Cheers
--
"That is the worst and most contemptible plan since Olaf the Hairy, high chief of all the Vikings, ordered 80,000 battle helmets with the horns on the inside" - Edmund Blackadder
Kåre Hviid
Occasional Visitor

Re: Proliant Support Pack 8.20 -> Load always at 1

8.30 results:

DL360 (various models) w/ RHEL 5.3 64bit:
cmasm2d cause ~0.30 load average

p-Class BL35p G1 w/ RHEL 4.8 32bit:
cmasm2d & cpqriisd cause ~1.30 load average

p-Class BL35p G1 w/ RHEL 4.8 64bit:
cmasm2d & cpqriisd casue ~1.50 load average

atqquebec
Advisor

Re: Proliant Support Pack 8.20 -> Load always at 1

I agree that there has been an improvement in 8.3, my load looks now more like 0.3 instead of 1.03. Will someone at HP eventually take this in charge and get it fixed?