Operating System - Linux
1748104 Members
4449 Online
108758 Solutions
New Discussion юеВ

Re: ICE-Linux mond issues with mdadm

 
Dave McLean
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

The hardware installation is a BL495G5 blade that has two internal SSD 64GB disks. The OS is RHEL 5.4 and mirrored acrossed the two internal drives.

mdadm.conf


# mdadm.conf written out by anaconda
DEVICE partitions
MAILADDR root
ARRAY /dev/md0 level=raid1 num-devices=2 uuid=aa4f5616:1f85a679:04e92872:8cb15fe7
ARRAY /dev/md1 level=raid1 num-devices=2 uuid=6787038e:e6c35d9c:fa5a0916:9729dd5f

dave

ARRAY /dev/md2 level=raid1 num-devices=2 uuid=c90d94d7:2f54ad8e:74248664:92872716
~
William Athanasiou
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

Well, that all looks right. Can you attach the output of "cat /proc/mdstat"?
Dave McLean
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

Output from /proc/mdstat

usorl03p309 ~ -1277> cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
208704 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
12586816 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[0]
49721088 blocks [2/2] [UU]

unused devices:
usorl03p309 ~ -1278>

dave
Mitchell Kulberg
Valued Contributor

Re: ICE-Linux mond issues with mdadm

Hey there Dave,

I'm curious. Are you able to reproduce this error on any other servers other than this one? any chance you've got USB devices on this server?

It's a long shot, but I've had questionable USB devices do that for real.

Thanks,
Mitch
Donna Firkser
Regular Advisor

Re: ICE-Linux mond issues with mdadm

Dave,

After further investigation it looks like this bogus DeviceDisappeared event is occurring because we are running mdadm as the nagios user. This is happening because we changed mond (which calls getMdadmEvents) to run as Nagios instead of root for security purposes. However, when we made this change we forgot to modify mdadm to use sudo so there's a defect in V2.11, in that we should be using "sudo /sbin/mdadm" inside getMdadmEvents.

This defect is fixed in the next IC-Linux release (V6.0) which should be available January 2010.

Do you know if Siemens is planning to move to V6.0 when it becomes available?

In the interim, You could manually work around this issue by making the following changes on every managed system. This is exact same fix that will be available in our V6.0 release.

1) Add the following line to /etc/sudoers on every managed system.
nagios ALL = NOPASSWD: /sbin/mdadm

And

2) Add "sudo" to the following line in /opt/hptc/mdadm/sbin/getMdadmEvents

`/usr/bin/sudo /sbin/mdadm --monitor --scan --program=/opt/hptc/mdadm/sbin/mdadm-handler --oneshot`;

Let me know if this helps.

Thanks,
Donna
Dave McLean
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

Thanks Donna. That's sorta what it was looking like since user root seem to work ok. It's tough doing root level tasks and at the same time maintain security.

I'll give your suggestions a try and report back to you.


dave
Dave McLean
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

Looks like the sudo trick worked.

Ready for another one? something is trying to open /dev/mcelog on 15 minute intervals and getting permission denied.

Nov 10 20:28:27 usorl03p309 mcelog: Cannot open /dev/mcelog
Nov 10 20:43:26 usorl03p309 mcelog: Cannot open /dev/mem for DMI decoding: Permission denied
Nov 10 20:43:26 usorl03p309 mcelog: Cannot open /dev/mcelog
Nov 10 20:58:27 usorl03p309 mcelog: Cannot open /dev/mem for DMI decoding: Permission denied


dave

Donna Firkser
Regular Advisor

Re: ICE-Linux mond issues with mdadm

Glad to hear that did the trick.

The mcelog event is the exact same issue so you need to apply the same work around.

1) Add /usr/sbin/mcelog to /etc/sudoers and
2) Add /usr/bin/sudo to the following line in /opt/hptc/mcelog/sbin/getMcelogEvents.

e.g.
`/usr/bin/sudo /usr/sbin/mcelog --syslog`;

These where the only two sudo issues fixed for V6.0, so you should be all set now.

Donna
Dave McLean
Occasional Advisor

Re: ICE-Linux mond issues with mdadm

Donna,

Applied the changes for mcelog also.

The last issue I'm working so far with Mitch is the wrong system name is being picked up when multiple IP's are plumbed up on the same NIC. Mitch should have all the details but maybe I'll open up a new forum on this one also.

Thanks for your support.

dave
Donna Firkser
Regular Advisor

Re: ICE-Linux mond issues with mdadm

Dave,

Mitch described the NIC/hostname issue to me. I'm going to try and reproduce it and will let you know what I find.

Donna