topic Re: LVM Problems in Operating System - HP-UX

LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 15:22:36 GMT

Hello, all!
I had a system crash and got stale partitions in a volume group VG03.

The disk /dev/dsk/c0t1d0 is mirrored on /dev/dsk/c1t1d0.

The VG belongs to cluster.

Follow the sequence of my issues:
1) vgchange -c n /dev/vg03 --- successful
2) vgchange -a y /dev/vg03 --- successful
3) fsck -f VXFS /dev/vg03 --- failed
4) dd if=/dev/dsk/c0t1d0 of=/dev/null bs=1024k
-- failed (I/0 Error)
5) vgchange -a n /dev/vg03
6) vgchange -c y /dev/vg03

Now, I have:

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t1d0": Cross-device link
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c1t1d0": Cross-device link
vgchange: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":The specified path does not correspond to physical volume attached to this volume groupvgchange: Warning: couldn't query physical volume "/dev/dsk/c1t1d0":The specified path does not correspond to physical volume attached to this volume groupvgchange: Warning: couldn't query all of the physical volumes.vgchange: Couldn't activate volume group "/dev/vg03":Quorum not present, or some physical volume(s) are missing.

So, I tried "vgscan" to create a new lvmtab:

Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c0t0d0
/dev/dsk/c1t0d0

Physical Volume "/dev/dsk/c0t1d0" contains no LVM information
Physical Volume "/dev/dsk/c1t1d0" contains no LVM information

Can you help me?

Is it possible to recover this LVM Information without data loss?

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 15:27:16 GMT

Since both disks are gone, I'm more inclined to think you lost a controller (and therefore your data may be just fine).

Have you check the system logs (/var/adm/syslog) for I/O errors?

If you run ioscan do you see the controller?

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 15:29:48 GMT

Hi, John!

Yes, I can see now, after a reboot:

vmunix: SCSI: Attempt to access partially open device - dev: bc022000

Re: LVM Problems

Chan 007 — Mon, 06 Feb 2006 15:41:21 GMT

Check your EMS on syslog.log,
tobstones/ts99

Also check using the crashconf to see why the system failed. If problems/any patches needed can be vital for a production system

also to ensure that system is fine, log a call with HP.

Chan

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 15:46:00 GMT

What's "ioscan" showing for the SCSI controller on that path?

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 15:47:56 GMT

Hi, Chan!

I'm not sure I understood your answer.

Follow the TS99 file.

Regards,
Leila

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 15:54:28 GMT

John,
the disks are in different controllers:

c0t1d0 is 8/0.1.0
c1t1d0 is 8/4.1.0

Attached, the result of ioscan command.

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 15:59:38 GMT

Wow, different controllers, not good.
(Kinda nullifies the controller theory).
"ioscan -fn" please...

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 16:03:46 GMT

John, what does it mean "partially open device"?

Follow the ioscan -fn results.

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 16:10:37 GMT

I really expected to see a "NO_HW" flag in the ioscan. But, it's not there.

I've not seen "partially open" messages like that before.

Can you post the output of
"vgdisplay -v vg03" ?
"pvdisplay /dev/dsk/c0t0d0"
"pvdisplay /dev/dsk/c1t0d0"

I'm wondering if lvmtab got hosed ...

Re: LVM Problems

Steven E. Protter — Mon, 06 Feb 2006 16:12:54 GMT

Shalom,

I am a member of the its a controller and the data is fine club.

Sometimes when disks die totally, ioscan stops showing them even in NO_HW mode.

You need HP hardware to diagnose and correct this and replace parts.

SEP

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 16:14:45 GMT

John, I can't activate VG, so vgdisplay doesn't work.

PVDISPLAY returns:

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":The specified path does not correspond to physical volume attached to this volume group

If I re-create the PV, do I lose data?

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 16:34:00 GMT

Happily, you've got SEP thinking it's hardware too (second opinions make me feel much better, especially from SEP). I wouldn't make a move until they (HP) shows up and give you a diagnosis. If you start pvcreating things you're going to risk losing data. However, b/c it could be a h/w problem, I'm thinking your data is probably OK - mainly b/c I find it unlikely you'd drop two disks mirrored to each other at once. ... Unless you dropped 1/2 of the mirrored area a some time back and didn't notice, and now the second disk is gone too. Hopefully, that's not it... But, just in case, start locating your backup sets.

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 16:35:33 GMT

John, I activated the VG with -q n option (to ignore quorum issues).

Now, I have the results of vgdisplay and lvdisplay.

Attached.

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 16:59:27 GMT

Since you've got no hardware showing in the "PV" columns of the "Logical extents" map of the lvdisplay. I'm back to hardware errors.
You've got hardware missing. Are the two SCSI controller cards plugged into the same I/O expansion bay (cell board I/O bay)? This leaves you waiting on HP for a hardware fix.

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 17:05:35 GMT

John, please take a look at my physical connections attached.

The disks are in blue.

I can't activate the VG in the adoptive node, too.

Where could be the HW problem? On both disks? It's very strange, don't you think?

Do you think it could be a good workaround powering off the whole system, checking physical connections and powering it on again?

Unfortunatelly, I don't have HP HW support at this time :-(

Re: LVM Problems

TwoProc — Mon, 06 Feb 2006 17:11:29 GMT

Are the disks represented in the purple group available and fully working? Please check at the vg, lv, and pv levels.

Re: powering down and checking connections. Not a bad idea at all. Shouldn't hurt a bit. Just make sure you've got your grounding straps on.

Re: LVM Problems

Leila Maria Rebel — Mon, 06 Feb 2006 17:25:32 GMT

John, I'm starting the shutdown.

Will you be there in the next 30 minutes?

Re: LVM Problems

Chan 007 — Mon, 06 Feb 2006 17:25:58 GMT

Can you pull one disk out and put it back, then run ioscan and insf to see the disk is going to UNAVL and back.

Just give a try.

I think you have physical disk problem

Chan

Re: LVM Problems

Chan 007 — Mon, 06 Feb 2006 17:28:41 GMT

I forget to mention, check you FC/SCSI cable sometimes they be be faulty...

Chan