topic Re: EMS HW monitoring: how to re-enable monitoring? in Operating System - HP-UX

EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Wed, 27 Jul 2005 06:10:24 GMT

Seems to be a simple question.

A failed disk has been removed from the list of devices being monitored. How to enable monitoring again after replacement? I've already tried disable/enable commands in remsmon program.

Thanks and points in advance!

Re: EMS HW monitoring: how to re-enable monitoring?

Mahesh Kumar Malik — Wed, 27 Jul 2005 06:23:04 GMT

Hi

Use following coammand

# /etc/opt/resmon/lbin/monconfig

select and then (E)nable the monitoring for the required device

Regards
Mahesh

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Wed, 27 Jul 2005 06:42:01 GMT

Global enabling does not work (new disk is still out of monitoring). What do you mean by 'selecting'? Thank you.

Re: EMS HW monitoring: how to re-enable monitoring?

Fabio Ettore — Wed, 27 Jul 2005 06:51:42 GMT

Hi,

when you launch the command suggested by Mahesh

/etc/opt/resmon/lbin/monconfig

you should get a list of options.
Interactively you should see

EVENT MONITORING IS CURRENTLY ENABLED.

and then select what you are searching for.

Post if you get other problems.

Best regards,
Fabio

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Thu, 28 Jul 2005 04:18:05 GMT

Hi Fabio,

I can run '(C)heck detailed monitoring status'. I see all the devices except the replaced disk. My question is how to re-enable its monitoring in the most graceful way. Thank you.

Re: EMS HW monitoring: how to re-enable monitoring?

Borislav Perkov — Thu, 28 Jul 2005 04:22:21 GMT

Hi Mihails,

Could you see the disk with ioscan command?

Regards,
Borislav

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Thu, 28 Jul 2005 04:25:59 GMT

Borislav,

The disk is 'CLAIMED' by ioscan, the disk is in use on LVM level. Thank you.

Re: EMS HW monitoring: how to re-enable monitoring?

Borislav Perkov — Thu, 28 Jul 2005 04:39:02 GMT

Can you send the output of

/opt/resmon/bin/resls /storage/events/disks/default

command?

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Thu, 28 Jul 2005 04:58:48 GMT

All disks except the problem one are listed in the file. The entries look like

/storage/events/disks/default/0_6_0_0.1.0

Re: EMS HW monitoring: how to re-enable monitoring?

Borislav Perkov — Thu, 28 Jul 2005 05:16:43 GMT

Strange thing.

You could check configuring request from GUI in SAM->Resource Monitoring->EMS then to add a request to /storage/events/disks/default/ and try to find the disk.

Have you try to restart the EMS?
If you haven't in monconfig main menu enter K and then E. Maybe it will help.

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Wed, 10 Aug 2005 04:45:16 GMT

Hi,

Anyway, the case is still open.

The History

1. A disk failed in a shared storage.
2. EMS HW monitoring removed the failed device from monitoring after 24 hours
3. The failed disk was replaced and returned to normal usage. It is normally shown by ioscan and vgdisplay.
4. The disk is not included in the moncheck command output on _both_ cluster nodes.
5. I tried the following things:
- Restarting EMS from monconfig utility (Kill/Enable)
- Upgarding to the newest available available software
OnlineDiag B.11.11.15.13 HPUX 11.11 Support Tools Bundle, Dec 2004
- Rebootin the system

Note that I don't use SAM for configuring EMS. I see empty list in 'Current Monitoring Requests' in SAM on my HP-UX systems.

Thanks and points in advance for more relevant ideas!

BR,
Mihails

Re: EMS HW monitoring: how to re-enable monitoring?

Ermin Borovac — Wed, 10 Aug 2005 05:23:13 GMT

Does the missing disk appear in the output of set_fixed?

# /opt/resmon/bin/set_fixed -l

Re: EMS HW monitoring: how to re-enable monitoring?

Emanuele_4 — Wed, 10 Aug 2005 05:38:27 GMT

I think you shoul enable the monitoring on ths new disk.

try this:

/etc/opt/resmon/lbin/set_fixed -l

And you can see the new disk NOT monitored

Thank run:

/etc/opt/resmon/lbin/set_fixed -n /storage/status/disks/default/6_0_1_0_0.2.0

where, obviously, you shoud replace the Hw path of the disk.

Hope it can help!

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Wed, 10 Aug 2005 06:47:03 GMT

Hi,

Many thanks for the command 'set_fixed'. Anyway, it does not help yet...

Problem disk does not appear in the output of '/opt/resmon/bin/set_fixed -l'. All other resources are marked 'UP', like this

...
/storage/status/disks/default/0_6_0_0.0.0 UP TRUE
/storage/status/disks/default/0_6_0_0.1.0 UP TRUE
/storage/status/disks/default/0_6_0_0.3.0 UP TRUE
...

'/opt/resmon/bin/set_fixed -L' replies
No resources set to DOWN state.

In this case command set_fixed -n cannot work.

/etc/opt/resmon/lbin/set_fixed -n /storage/status/disks/default/0_6_0_0.2.0
/etc/opt/resmon/lbin/set_fixed: No matching resource names found to set to UP state.

On the other hand, the disk exists and there are several disks with the same part number.

disk 13 0/6/0/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c5t2d0 /dev/rdsk/c5t2d0

Thanks and points for more help!

BR,
Mihails

Re: EMS HW monitoring: how to re-enable monitoring?

Emanuele_4 — Wed, 10 Aug 2005 08:18:01 GMT

Can you look at

/etc/opt/resmon/log

do you see anything different as usual?

the disk is correctly conifgured by lvm?

Is everything ok with pvcreate, vgdisplay, lvdisplay, ioscan, mirror, diskinfo, dd...etc etc...

Re: EMS HW monitoring: how to re-enable monitoring?

Andrew Merritt_2 — Wed, 10 Aug 2005 08:21:01 GMT

Hi Mihails,
I don't have any answers yet, but several questions ;-)

If you run STM, do you see the disk listed there? If not, try running the 'map' command; does this make any difference?

If you can see the disk in STM, run the info tool and see what the firmware level is - does it begin with 'HP'?

Are the other disks with the same part number being monitored OK?

Is there anything in the /var/stm/data/tools/monitor/disabled_instances file?

When you say the disk was removed from the list of devices being monitored, what exactly do you mean? Which list? Do you mean it was removed automatically, or by hand?

Is anything logged by disk_em in the api.log or client.log files in /var/opt/resmon/log?

Is anything shown relating to this disk in the maplog in STM ('Map Log' in the System menu in mstm, or 'ml' command in 'cstm')?

Andrew

Re: EMS HW monitoring: how to re-enable monitoring?

Patrick Wallek — Wed, 10 Aug 2005 08:34:51 GMT

It sounds like you might need to rebuild the /var/stm/data/psm_data file.

1) Stop EMS monitoring -- Go into monconfig and select K to stop EMS.

2) Verify that psmctd is not running. -- 'ps -ef |grep psmctd' -- If it is still there, stop diagnostics -- /sbin/init.d/diagnostic stop -- Check psmctd again, if it is still running kill with 'kill -9'

3) Move the psm_data file somewhere else -- mv /var/stm/data/psm_data /tmp/psm_data

4) If you stopped diagnostics in step 2, restart now with '/sbin/init.d/diagnostic start', if you did not stop them go to step 5.

5) Restart EMS -- Go into monconfig and choose E to start. It may take a while to rebuild the psm_data file and 'set_fixed -l' may return an error. Once the process is finished 'set_fixed -l' should return all hardware.

Re: EMS HW monitoring: how to re-enable monitoring?

Andrew Merritt_2 — Wed, 10 Aug 2005 08:50:42 GMT

Just to tidy up what Patrick has suggested (I don't know if it would work to fix this problem, but it's worth a try), you WILL need to do Step 2, since step one only stops the monitors. psmctd is started by diagmond.

Andrew

Re: EMS HW monitoring: how to re-enable monitoring?

Mihails Nikitins — Fri, 12 Aug 2005 02:27:41 GMT

Patrick, thanks a lot for your procedure, however, it didn't help.

Probably, Andrew found the correct question, i.e. the answer. This is the only disk having firmare revision without 'HP' letters! If this is my problem, I will come back with 10 points.

stm see the disks, 'disabled_instances' are empty.

Logs look OK.

Monitoring of the disk was disabled automatically. From event.log

Summary: Disk at hardware path 0/6/0/0.2.0 : Device removed from monitoring

The disk was removed from monitoring automatically. The device has been removed from the list of devices being monitored by this monitor.

Re: EMS HW monitoring: how to re-enable monitoring?

Andrew Merritt_2 — Fri, 12 Aug 2005 02:41:26 GMT

Hi Mihails,
Yes, that will be the answer; the EMS HW Monitors will only monitor disks with HP firmware ids, because those are the only versions of firmware that are tested and supported by HP.

I would also guess that the "Summary: Disk at hardware path 0/6/0/0.2.0 : Device removed from monitoring" message was logged when this was detected, i.e. after the disk was replaced.

Andrew