ems keeps restarting disk_em every couple of minutes

[Bob Brown_1]

One of the vpar's on my RP8400 (running 11i) suddenly started sending out messages every couple of minutes stating that disk_em is restarting.

This is the only vpar (1 of 4) in the 8400 that is doing this, and only the disk_em monitor is doing this.

Any ideas?

thanks.

-Bob

[Steven E. Protter]

Shalom,

1) Try completely stopping the ems disk configuration and then setting it up anew. This has helped me in the past.

2) Make sure there is a lun on lun0 if fiber attached to the machine. I have seen disk arrays picked up as disk and this causes problems with ems.

SEP

[Bob Brown_1]

How do you stop and restart the ems disk config?

Nothing has changed in the config for this system...it just suddenly started doing this.

This system boots from local disk and has some other filesystems on fibre.

-Bob

[Pupil_1]

You can do all the ems configuration with the command
/etc/opt/resmon/lbin/monconfig

[Andrew Merritt_2]

Hi Bob,

Changing the config with monconfig is unlikely to do anything in this case, don't bother with that.

Have a look in /var/opt/resmon/log/api.log to see if disk_em is logging anything there when it dies.

What version of OnlineDiags do you have installed? If you don't have a recent version, update and install the latest patch for that version.

Did anything change recently on the system?

Andrew

[RAC_1]

With restart message, does it give any disks hardware paths? What are those? Internal or SAN disks? Till the time you figure out, what is happening, you can disable alerts for those disks if you want.

Also, up grading the STM version may help.

[Andrew Merritt_2]

If it mentions any disks, the restart message will list all the disks being monitored, that's not going to help.

Yes, as already mentioned, check you have a recent version of OnlineDiags.
http://www.docs.hp.com/hpux/onlinedocs/diag/stm/stm_upd.htm#table

Do you have ISEE installed?
If so, and it's not the latest version (A.03.90 or later), check the /var/stm/config/tools/monitor/rst_disk_em.clcfg file, and look for an entry over two lines:

DEV_ID:dev_pdev:dev_devclass:dev_inq_vendor:dev_inq_prod:dev_fw_version
DEV_ID:dev_serial_num

Needs to be modified to:

DEV_ID:dev_pdev:dev_devclass:dev_inq_vendor:dev_inq_prod:dev_fw_version:dev_serial_num

(This should be fixed in current versions of ISEE.)

Andrew

[Bob Brown_1]

I looked in the api.log file and I see that disk_em seems to be dying with a segmentation violation.

I am running the same version of hpux and diags on the other vpar's on the RP8400, only 1 vpar is having troubles.

No recent changes to the h/w or s/w environment.

Any ideas? Would restarting diags (how to do this??) help? Would a reboot of hpux help?

thanks.

-Bob

[Joel Girot]

Hi Bob,

Before reboot you can test this procedure. This helped me for a server 11.11 rp3440. Be careful: without guarantee on your system.

- Stop monitoring

# /etc/opt/resmon/lbin/monconfig
Select (K)ill (disable) monitoring

- Stop Diag

/sbin/init.d/diagnostic stop

Kill any monitor processes which are left over.

- Stop p_client

Comment out the p_client entry in /etc/inittab :
# vi /etc/inittab

Change the following line
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
to
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

Reread /etc/inittab :
# init q

- stop emsagent

/sbin/init.d/emsa stop

Check if p_client is still running, if yes kill the process.

- Delete the persistence files and pipes

# cd /etc/opt/resmon/persistence
# rm p*
# rm m*
# cd /etc/opt/resmon/pipe
# rm *

- Have diagnostics remap the system hardware

# cstm
cstm>remap
cstm>quit

Allow enough time for remap to finish before restarting the monitors.

- Clean up monitor flags in the diagnostics directory

# cd /var/stm/data/tools/monitor
# rm *.hwa

- Restart Diag

/sbin/init.d/diagnostic start

- Restart p_client

# vi /etc/inittab
Change the following line
#ems4:3456:respawn:/etc/opt/resmon/lbin/p_client
back to
ems4:3456:respawn:/etc/opt/resmon/lbin/p_client

Re-read /etc/inittab :

# init q

- Restart emsagent
/sbin/init.d/emsa start


- Restart the EMS monitors

# /etc/opt/resmon/lbin/monconfig
Select (E)nable Monitoring

Hope this help you.

[Andrew Merritt_2]

Hi Bob,
Can you post the error message from api.log with the SIGSEGV, and any others from disk_em around the same time?

Did you check the rst_disk_em.clcfg file, as the symptom of that problem is a SIGSEGV?

(As an aside, you should also check you have PHSS_34835 installed, but that doesn't have and fixes for disk_em.)

If all that comes up blank, I'd recommend opening a support call with HP.

Andrew