lvlnboot and other LVM commands hung

[Greg Hall]

Hi,

HPUX 11.11. Somehow I have a production system with some sort of LVM problem. I can run displays ok but lvlnboot -v just hangs; and I did an lvreduce on a LV to remove a mirror. This has removed the mirror, but the command has not ended. Now I want to put the mirror back but can't as I get msg Cannot lock "/etc/lvmconf//lvm_lock" . arrrghhh.

Is there some way out or is a reboot my only option?

thanks, Greg

[Devesh Pant_1]

Did you try to grep for the process ?

I wonder if the command got killed before it completed.

I if none of the LVM commands are working, a reboot is all you can look forward to.

thanks
DP

[DCE]

check your syslog.log file for error messages. It sounds like you have a disk that has gone bad.

Dave

[RAC_1]

You can delete "/etc/lvmconf//lvm_lock"
and command will complate. Onece that is resolved, I think you should not have problems with lvlnboot -v

[saju_2]

Hi

"I get msg Cannot lock /etc/lvmconf//lvm_lock"
comes because another lvm related command is running in the background.
Since lvlnboot -v hangs means that one of the disk in ur system is having some problem. Check the LED status on the disk as well as do ioscan to see everything is claimed. Also check the /var/adm/syslog/syslog.log file

Regards
CS

[Greg Hall]

thankyou all.

I do have an additional known problem. Something wierd happened with a san switch and Secure Path then reported some FAILED redundant paths, a week ago. That's another story, but I seem to have compounded the problem when trying to clean up some unused LUNS. 'spmgr delete' just hung and now 'spmgr display' also hangs. And then the LVM problems started happening. So far the application is running just fine but system is in a sorry state really.

Is it safe to delete the "/etc/lvmconf//lvm_lock" file?

cheers, Greg

[Devender Khatana]

Hi,

You can check for LVM command in ps output. There should is one really running then you should not remove this (Which I suppose should be the case).

If nothing similar is seen in the output you can remove the lock and can try some simple command on some locally visible device like vgdisplay or similar.

Allthough the probem is related to non visibility of the disks properly. Is the system booting through SAN disks ?


HTH,
Devender

[Sameer_Nirmal]

It sounds that you have a bad LVM disk. I suspect the bad disk should belong to vg00 as those commands are hanging.

You need to identify the bad disk. You can do this using
# ioscan -fC disk
# diskinfo -v /dev/rdsk/c?t?d?
# dd if=/dev/rdsk/c?t?d? of=/dev/null bs=256k count=100
Using mstm/cstm utility.

When you tried to reduced the LV mirror ,either the PE/LEs are not de-allocated completly or the automatic vgcfgbackup operation got stucked due to bad disk.

You can check up any LVM command being running using ps -ef command . If you are sure nothing there , then check the date and time of the lvm_lock file which should match the time when you tried to do lvreduce. Then you can safely remove the file.

Check the state of VG00 using vgdisplay -v
lvdisplay -v, pvdisplay -v

[Greg Hall]

Hi again,

thanks for the feedback. I suspect we're chasing a deadend here, and I will have no choice but to reboot, which is scheduled for next Sunday. Hope it works.
I'm not so sure about a bad disk as such, surely EMS would be reporting this also?
- I can't run any ioscans, there are numerous already hung since the initial san switch failure and the subsequent failed paths to the EVA's.
- lvlnboot -v vg00 runs ok, but any other lvlnboot cmd just hangs.
- the lvreduce cmd can still be seen in ps -ef. It has run and I can lvdisplay the lvol and shows one copy only. There is no vgcfgbackup process, but I suspect this is what it's trying to do next.
- the lvm_lock file is dated May 1 2004. From checking other HPUX systems I see this is not unusual. It seems to me this empty file is acquired by an LVM command somehow prior to running, then released after?
- I could possibly run the lvextend to put the copy back again, if I deleted the lvm_lock, but I feel I am just worsening the situation every time I try to do anything.
So will try to leave it alone and hope it keeps running till the weekend. Unless someone's got a better idea? :)

thanks muchly , Greg

[Devender Khatana]

Hi,

The lvm_lock file will be of zero size always. This is necessary to avoid multiple configuration commands running on the same system simultaneously.

Not only in LVM related commands this method is adopted in some other commands as well to avoid double execution of same or similar activity on same or other device. One such example is make_tape_recovery which makes its lock under /var/tmp/ign_configure.

HTH,
Devender

[Greg Hall]

Rebooted system, all Failed Secure Paths restored to standby. LVM ok, all looks good.

Greg