LVM Problems

[Leila Maria Rebel]

Hello, all!
I had a system crash and got stale partitions in a volume group VG03.

The disk /dev/dsk/c0t1d0 is mirrored on /dev/dsk/c1t1d0.

The VG belongs to cluster.

Follow the sequence of my issues:
1) vgchange -c n /dev/vg03 --- successful
2) vgchange -a y /dev/vg03 --- successful
3) fsck -f VXFS /dev/vg03 --- failed
4) dd if=/dev/dsk/c0t1d0 of=/dev/null bs=1024k
-- failed (I/0 Error)
5) vgchange -a n /dev/vg03
6) vgchange -c y /dev/vg03

Now, I have:

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t1d0": Cross-device link
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c1t1d0": Cross-device link
vgchange: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":The specified path does not correspond to physical volume attached to this volume groupvgchange: Warning: couldn't query physical volume "/dev/dsk/c1t1d0":The specified path does not correspond to physical volume attached to this volume groupvgchange: Warning: couldn't query all of the physical volumes.vgchange: Couldn't activate volume group "/dev/vg03":Quorum not present, or some physical volume(s) are missing.

So, I tried "vgscan" to create a new lvmtab:

Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c0t0d0
/dev/dsk/c1t0d0

Physical Volume "/dev/dsk/c0t1d0" contains no LVM information
Physical Volume "/dev/dsk/c1t1d0" contains no LVM information

Can you help me?

Is it possible to recover this LVM Information without data loss?

[TwoProc · Accepted solution]

Since both disks are gone, I'm more inclined to think you lost a controller (and therefore your data may be just fine).

Have you check the system logs (/var/adm/syslog) for I/O errors?

If you run ioscan do you see the controller?

[Leila Maria Rebel]

Hi, John!

Yes, I can see now, after a reboot:

vmunix: SCSI: Attempt to access partially open device - dev: bc022000

[Chan 007]

Check your EMS on syslog.log,
tobstones/ts99

Also check using the crashconf to see why the system failed. If problems/any patches needed can be vital for a production system

also to ensure that system is fine, log a call with HP.

Chan

[TwoProc]

What's "ioscan" showing for the SCSI controller on that path?

[Leila Maria Rebel]

Hi, Chan!

I'm not sure I understood your answer.

Follow the TS99 file.

Regards,
Leila

[Leila Maria Rebel]

John,
the disks are in different controllers:

c0t1d0 is 8/0.1.0
c1t1d0 is 8/4.1.0

Attached, the result of ioscan command.

[TwoProc]

Wow, different controllers, not good.
(Kinda nullifies the controller theory).
"ioscan -fn" please...

[Leila Maria Rebel]

John, what does it mean "partially open device"?

Follow the ioscan -fn results.

[TwoProc]

I really expected to see a "NO_HW" flag in the ioscan. But, it's not there.

I've not seen "partially open" messages like that before.

Can you post the output of
"vgdisplay -v vg03" ?
"pvdisplay /dev/dsk/c0t0d0"
"pvdisplay /dev/dsk/c1t0d0"

I'm wondering if lvmtab got hosed ...

[Steven E. Protter]

Shalom,

I am a member of the its a controller and the data is fine club.

Sometimes when disks die totally, ioscan stops showing them even in NO_HW mode.

You need HP hardware to diagnose and correct this and replace parts.

SEP

[Leila Maria Rebel]

John, I can't activate VG, so vgdisplay doesn't work.

PVDISPLAY returns:

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t1d0":The specified path does not correspond to physical volume attached to this volume group

If I re-create the PV, do I lose data?

[TwoProc]

Happily, you've got SEP thinking it's hardware too (second opinions make me feel much better, especially from SEP). I wouldn't make a move until they (HP) shows up and give you a diagnosis. If you start pvcreating things you're going to risk losing data. However, b/c it could be a h/w problem, I'm thinking your data is probably OK - mainly b/c I find it unlikely you'd drop two disks mirrored to each other at once. ... Unless you dropped 1/2 of the mirrored area a some time back and didn't notice, and now the second disk is gone too. Hopefully, that's not it... But, just in case, start locating your backup sets.

[Leila Maria Rebel]

John, I activated the VG with -q n option (to ignore quorum issues).

Now, I have the results of vgdisplay and lvdisplay.

Attached.

[TwoProc]

Since you've got no hardware showing in the "PV" columns of the "Logical extents" map of the lvdisplay. I'm back to hardware errors.
You've got hardware missing. Are the two SCSI controller cards plugged into the same I/O expansion bay (cell board I/O bay)? This leaves you waiting on HP for a hardware fix.

[Leila Maria Rebel]

John, please take a look at my physical connections attached.

The disks are in blue.

I can't activate the VG in the adoptive node, too.

Where could be the HW problem? On both disks? It's very strange, don't you think?

Do you think it could be a good workaround powering off the whole system, checking physical connections and powering it on again?

Unfortunatelly, I don't have HP HW support at this time :-(

[TwoProc]

Are the disks represented in the purple group available and fully working? Please check at the vg, lv, and pv levels.

Re: powering down and checking connections. Not a bad idea at all. Shouldn't hurt a bit. Just make sure you've got your grounding straps on.

[Leila Maria Rebel]

John, I'm starting the shutdown.

Will you be there in the next 30 minutes?

[Chan 007]

Can you pull one disk out and put it back, then run ioscan and insf to see the disk is going to UNAVL and back.

Just give a try.

I think you have physical disk problem

Chan

[Chan 007]

I forget to mention, check you FC/SCSI cable sometimes they be be faulty...

Chan

[Leila Maria Rebel]

Chan, thanks for your message!

At this moment, I'm trying to power off the system, and I've just shut it down.

After the restart, I'll check the disks again!

Regards,
Leila

[Chan 007]

Hi,

You asked for ems, normally it is Event Monitoring Serv, installe on all system,

Any event (h/w) will be logged in syslog.log, all you have o do is just cut that line from syslog.log to run it,

This is give what is event that has problem, also it gives that Event #, go can browse that site in that output message to know what is wrong...

Hope I confused you much,

Check you have EMS running

Chan

[TwoProc]

Go ahead and reseat the drives and cards while you're down too!

[TwoProc]

Go ahead and reseat the drives and cards while you're down too. But, you can color me surprised if it works!

[Leila Maria Rebel]

Chan, my system is 10.20. I'm afraid I don't have this tool available, do I?

John, I won't color you, unfortunatelly the situation is the same.

The unique difference is that I could activate the VG without disabling quorum...

What about if I break the mirror?

c1t1d0 seems to be accessible, isn't it?