ICE for Linux 6.3 Nagios problem

[Donna Firkser]

Was thinking we might find the problem if you run nagios in debug mode.  Use these steps to stop nagios and restart nagios in debug mode.   Please forward along any messages you see after starting nagios.

 /etc/init.d/nagios stop_nagios

cd /opt/hptc/nagios/bin

NAGIOS_DEBUG=1 ./nagios ../etc/nagios_local.cfg

Thanks,

Donna

[Allen012]

/opt/hptc/etc/sysconfig/nagios:

NAGIOS_MONITOR_HOST=icelx1

NAGIOS_SUBMIT_HOST=[ip address of sm701]

NRPE_HOST=icelx1

NSCA_HOST=[ip address of sm701]

HTTPD_HOST=icelx1

NAGIOS_MASTER=icelx1

NAGIOS_MASTER_IP=[ip address of sm701]

CPACCESS_HOST=icelx1

NAGIOS_USER=nagios

NAGIOS_GROUP=hpadm

HTTP_GROUP=apache

[Allen012]

Did the debug run. Found that I was getting permission issues. Changed permissions on some directories and files which where owned by root. Started getting connections.

It appears that there is a significant issue when installing ICE into a hardened Linux environment. In that environment the umask for root is set to 0077. That means if the install script does not check permissions, any directory or file created by root during the install is going to have permissions of 700 (rwx --- ---).

Nagios runs as the nagios user, and therefore does not permissions to access directories and files that it needs.

The install procedure and scripts need to be changed to check and/or correct the permissions on all of the directories and files created.

[Donna Firkser]

Glad to hear you found the issue.  At a minimum, for the next IC-Linux release we're working on (i.e. 7.0), we'll document that users with a "hardened Linux environment" may need to change the permissions on the directories used by Nagios. 

Is Nagios now working as expected?

Thanks,

Donna

[Allen012]

DO NOT document as "need to change permissions" There are too many and they are too hard to find - I am still looking. Either emphasize that they MUST set the umask for root to 0022, or else (preferred) in the install script you need to check it and/or change it for the duration of the install.

I am still having some problems - trying to run them down.  There is also an issue in the way that the configuration scripts (try to) modify the sudoers file.  My hardening prevents it, and that causes some of the sensor scripts that rely on sudo to fail.  It would be better if the changes were specified up front, or otherwise made more visible in the documentation.

I am having "unintialized value" errors in the /opt/hptc/supermon/bin/storeMetrics perl script (line 109), and multiple

      'Argument "NRPE" isn't numeric in division (/) at /opt/hptc/nagios/libexec/check_node_config line 128.'

errors, among others.

If I thought that I could do a total "clean" uninstall I would do it and try again, but it has been my experience that the uninstall scripts do not delete all of the affected directories.

Still working the issue.

[Donna Firkser]

There's an uninstall.sh procedure in the Install Guide which outlines which directories you need to manually clean up.  We create a copy of the entries Nagios adds to /etc/sudoers in case you need to update this file on your own.  Does this help.

 # cat /etc/sudoers.icelx.proto
# IC-Linux requires Default requiretty to be disabled # HP-HPTC-defaultrequiretty
Cmnd_Alias CHECKALLSSHKEYS = /opt/hptc/nagios/libexec/check_keys # HP-HPTC-KeySync
Cmnd_Alias CHECKSYSLOGALERTS = /opt/hptc/nagios/libexec/check_syslogalerts # HP-HPTC-SysLog
Cmnd_Alias CHECKSFS = /opt/hptc/nagios/libexec/check_sfs # HP-HPTC-SysLog
Cmnd_Alias CHECKLSF = /opt/hptc/nagios/libexec/check_lsf # HP-HPTC-CheckLSF
Cmnd_Alias CHECKICMP = /opt/hptc/nagios/libexec/check_icmp # HP-HPTC-CheckICMP
Cmnd_Alias CHECKSEL = /opt/hptc/nagios/libexec/check_sel # HP-HPTC-CheckSEL
Cmnd_Alias CHECKSELMON = /opt/hptc/nagios/libexec/check_selmon # HP-HPTC-CheckSELMON
Cmnd_Alias CHECKLVS = /opt/hptc/nagios/libexec/check_lvs # HP-HPTC-CheckLVS
Cmnd_Alias SENSORS = /opt/hptc/supermon/bin/sensors # HP-HPTC-Sensors
Cmnd_Alias CHECKHOSTS = /opt/hptc/nagios/libexec/check_node_status # HP-HPTC-CheckNodeStatus
Cmnd_Alias RRDSWSETUP = /opt/hptc/cacti/rrd_switch_setup # HP-HPTC-RrdSwitchSetup
Cmnd_Alias SWITCHPOLLER = /opt/hptc/nagios/libexec/switch_poller # HP-HPTC-SwitchPoller
Cmnd_Alias SCONTROL = /opt/hptc/bin/scontrol # HP-HPTC-scontrol
Cmnd_Alias POWEROFF = /opt/hptc/sbin/power # HP-HPTC-power
Cmnd_Alias HPASMCLI = /sbin/hpasmcli # HP-HPTC-hpasmcli
Cmnd_Alias MDADM = /sbin/mdadm # HP-HPTC-mdadm
Cmnd_Alias MCELOG = /usr/sbin/mcelog # HP-HPTC-mcelog
nagios ALL = NOPASSWD:  CHECKALLSSHKEYS,CHECKSYSLOGALERTS,CHECKSFS,CHECKLSF,CHECKICMP,CHECKSEL,CHECKSELMON,CHECKLVS,SENSORS,CHECKHOSTS,RRDSWSETUP,SWITCHPOLLER,SCONTROL,POWEROFF,HPASMCLI,MDADM,MCELOG # HP-HPTC-Nagios

[Allen012]

The "Configure Management Services" process does not put any where close to that many entries into the sudoers file, just the following:

Cmnd_Alias HPASMCLI = /sbin/hpasmcli # HP-HPTC-hpasmcli
Cmnd_Alias MDADM = /sbin/mdadm # HP-HPTC-mdadm
Cmnd_Alias MCELOG = /usr/sbin/mcelog # HP-HPTC-mcelog
nagios ALL = NOPASSWD: HPASMCLI,MDADM,MCELOG # HP-ICELX-mond

[Allen012]

I don't find the /etc/sudoers.icelx.proto file in /etc, or anywhere else on my CMS.

[Donna Firkser · Accepted solution]

/etc/sudoers on the managed nodes only has a few entries. i.e.

# HP-ICELX-mond: This is required for monitoring to function when run as the nagios user
Cmnd_Alias HPASMCLI = /sbin/hpasmcli # HP-ICELX-mond
Cmnd_Alias MDADM = /sbin/mdadm # HP-ICELX-mond
Cmnd_Alias MCELOG = /usr/sbin/mcelog # HP-ICELX-mond
nagios ALL = NOPASSWD: HPASMCLI,MDADM,MCELOG # HP-ICELX-mond

/etc/sudoers on the CMS must have the following entries for Nagios to work properly.

# IC-Linux requires Default requiretty to be disabled # HP-HPTC-defaultrequiretty
Cmnd_Alias CHECKALLSSHKEYS = /opt/hptc/nagios/libexec/check_keys # HP-HPTC-KeySync
Cmnd_Alias CHECKSYSLOGALERTS = /opt/hptc/nagios/libexec/check_syslogalerts # HP-HPTC-SysLog
Cmnd_Alias CHECKSFS = /opt/hptc/nagios/libexec/check_sfs # HP-HPTC-SysLog
Cmnd_Alias CHECKLSF = /opt/hptc/nagios/libexec/check_lsf # HP-HPTC-CheckLSF
Cmnd_Alias CHECKICMP = /opt/hptc/nagios/libexec/check_icmp # HP-HPTC-CheckICMP
Cmnd_Alias CHECKSEL = /opt/hptc/nagios/libexec/check_sel # HP-HPTC-CheckSEL
Cmnd_Alias CHECKSELMON = /opt/hptc/nagios/libexec/check_selmon # HP-HPTC-CheckSELMON
Cmnd_Alias CHECKLVS = /opt/hptc/nagios/libexec/check_lvs # HP-HPTC-CheckLVS
Cmnd_Alias SENSORS = /opt/hptc/supermon/bin/sensors # HP-HPTC-Sensors
Cmnd_Alias CHECKHOSTS = /opt/hptc/nagios/libexec/check_node_status # HP-HPTC-CheckNodeStatus
Cmnd_Alias RRDSWSETUP = /opt/hptc/cacti/rrd_switch_setup # HP-HPTC-RrdSwitchSetup
Cmnd_Alias SWITCHPOLLER = /opt/hptc/nagios/libexec/switch_poller # HP-HPTC-SwitchPoller
Cmnd_Alias SCONTROL = /opt/hptc/bin/scontrol # HP-HPTC-scontrol
Cmnd_Alias POWEROFF = /opt/hptc/sbin/power # HP-HPTC-power
Cmnd_Alias HPASMCLI = /sbin/hpasmcli # HP-HPTC-hpasmcli
Cmnd_Alias MDADM = /sbin/mdadm # HP-HPTC-mdadm
Cmnd_Alias MCELOG = /usr/sbin/mcelog # HP-HPTC-mcelog
nagios ALL = NOPASSWD:  CHECKALLSSHKEYS,CHECKSYSLOGALERTS,CHECKSFS,CHECKLSF,CHECKICMP,CHECKSEL,CHECKSELMON,CHECKLVS,SENSORS,CHECKHOSTS,RRDSWSETUP,SWITCHPOLLER,SCONTROL,POWEROFF,HPASMCLI,MDADM,MCELOG # HP-HPTC-Nagios

Donna

[Donna Firkser]

/etc/sudoers.icelx.proto gets created as part of the %post when we install the nagios RPM on the CMS. My guess is that it's not on your system because the "hardening" on your CMS prevented our code from updating /etc/sudoers and /etc/sudoers.icelx.proto is created based on what we put into /etc/sudoers.
i.e.
/bin/grep '# HP-HPTC-' /etc/sudoers > /etc/sudoers.icelx.proto