Problem with fs mounting

[Global Unix Team]

Hi,

I have a problem on a server connected to an EMC Symmetrix.

After a reboot, it is unable to mount filesystems in a vg named vg106BEBarch.

If I run a vgdisplay vg106BEBarch, I get this :

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t14d4":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c4t14d4":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg106BEBarch
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 5
Open LV 5
Max PV 64
Cur PV 2
Act PV 2
Max PE per PV 10240
VGDA 4
PE Size (Mbytes) 8
Total PE 2156
Alloc PE 1150
Free PE 1006
Total PVG 2
Total Spare PVs 0
Total Spare PVs in use 0

vgdisplay -v shows the followings disks :
--- Physical volume groups ---
PVG Name 106ARCH
PV Name /dev/dsk/c3t6d4
PV Name /dev/dsk/c3t6d5
PV Name /dev/dsk/c4t6d4
PV Name /dev/dsk/c4t6d5

PVG Name ARCH2
PV Name /dev/dsk/c3t14d4
PV Name /dev/dsk/c4t14d4

I appears that tis vg was moved from another server and was still active on the previous server. I desactivated it on the other server but I can't find how to solve this.

It seems, but I am still not sure, I am checking with storage team, that the two last disks shouldn't belong to this vg.

ioscan -fnC disk shows all the disks claimed.

I daren't move /etc/lvmtab and run vgscan -av, I am not familiar with this and I fear that the solution may be worse that the problem.

I ran vgscan -apv and it doesn't see vg106BEBarch vg.

In addition, these disks are all configured with multipath two by two so when it says that there's only 2 active PV's, it means that there are 4 disks active with multipath. (c3t6d4 multipath with c4t6d4 and so on)

[Alex Lavrov.]

Try:

diskinfo /dev/rdsk/c3t14d4
and
pvdisplay /dev/dsk/c3t14d4

And paste here the output.

[Devender Khatana]

Hi,

Physical disks in your ARCH2 Physical Volume Group are not accessible. As stated earlier see wheather the device files exists.

VG can still be activated by bypassing the quorum check using "vgchange -a y -q n /dev/vg106BEBarch"

But by doing this your Logical volumes which are allocated to these physical volumes fully or partially will not be available.

HTH,
Devender

[Global Unix Team]

Hi,

Here are the results of diskinfo and pvdisplay.

# diskinfo /dev/rdsk/c3t14d4
SCSI describe of /dev/rdsk/c3t14d4:
vendor: EMC
product id: SYMMETRIX
type: direct access
size: 8838720 Kbytes
bytes per sector: 512

root@mou043 [/tmp]
# pvdisplay /dev/dsk/c3t14d4
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t14d4":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c4t14d4":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg106BEBarch".
pvdisplay: Cannot display physical volume "/dev/dsk/c3t14d4".

I have no problem to activate the vg, but if I try to mount the fs, I get such errors :
vxfs mount: /dev/vg106BEBarch/lvol1 is corrupted. needs checking

if I run a fsck, I get :

# fsck /dev/vg106BEBarch/lvol1
file system is larger than device
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...

[Alex Lavrov.]

If you execute the command "insf", do you get any output?

[Global Unix Team]

no, insf does not give any output

[Alex Lavrov.]

And another question, how did you move the VG from the other server? vgexport/vgimport?

If you perform vgscan, do you get any messages?
After vgscan:

strings /etc/lvmtab | more

Can you see this vg/disk there?

[Devender Khatana]

Hi,

It appears the LUN is accessible from the EMC box but not the old contents of the LUN. What actually has happened to the EMC box. Were there some changes done Zoning config etc.

Just see /etc/lvmpvg file all the PVGs should be listed in it.

HTH,
Devender

[Global Unix Team]

I don't know how they were moved, I didn't do it. I assume only a vgchange -a n was done on the original server, which may have caused the problem as the vg was active on both servers (but no filesystem mounted on old server, at least !)
vgscan -apv gives (part of it only):
Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c3t14d4
/dev/dsk/c4t14d4

but nothing on c3t6d4-5 or c4t6d4-5 which should be seen, they are seen by pvdisplay !

[Global Unix Team]

Here is what appears in /etc/lvmpvg

VG /dev/vg106BEBarch
PVG 106ARCH
/dev/dsk/c3t6d4
/dev/dsk/c3t6d5
/dev/dsk/c4t6d4
/dev/dsk/c4t6d5
PVG ARCH2
/dev/dsk/c3t14d4
/dev/dsk/c4t14d4

It seems coherent with vgdisplay, but I don't know if vgdisplay reads this file, in which case it would be normal that it gives the same result :-)

[Devender Khatana]

Hi,

You might have to run vgexport/vgimport now. Just note down the VG minor no. by "ls -l /dev/v06BEBarch/group" Then do a vgexport and then vgimport by specifying the physical Paths of all the Physical disks with vgimport.

Do you have some old bdf/lvdisplay -v output from where you can come to know how many file system were there in this VG and what was the allocation policy ?

HTH,
Devender

[Global Unix Team]

There were 5 lv's distributed on the 4 first disks :
/dev/dsk/c3t6d4
/dev/dsk/c3t6d5
/dev/dsk/c4t6d4
/dev/dsk/c4t6d5
(in fact 2 disks with multipath)

But I don't know what the following disks are :
/dev/dsk/c3t14d4
/dev/dsk/c4t14d4

I can figure some possibilities :

vgexport/vgimport : should I import only the first 4 disks or all ? isn't there a risk to lose all data on the disks ?

mv /etc/lvmtab /etc/lvmtab.old;vgscan -av :
I did that only once, based on a thread in this forum describing exactly my problem. I don't know if this command is dangerous, and besides, I am not sure at all it can solve the problem. I will probably do that, but I'd like to be sure before.

vgreduce 2 unused disks : I will use this at the very end if it appears that nothing else works, I really doubt it would be the solution. And more : if the two last disks :
/dev/dsk/c3t14d4
/dev/dsk/c4t14d4
don't belong to this volume group, to which do they belong ? I don't wan't to solve my problem by erasing someone else's data :-(

[Devender Khatana]

Hi,

You can edit the /etc/lvmpvg file and can make necessary changes to reflect only first four disks in it. (See man lvmpvg)

And if you are sure that this vg was allocated to only four disks then you can do a vgimport with only four devices after changing /etc/lvmpvg. It do not destroy existing contents of the disk.

HTH,
Devender

[Alex Lavrov.]

To continue the previous post:

vgchange -a n vg106BEBarch
vgexport -m /tmp/my.map vg106BEBarch

mkdir /dev/vg106BEBarch
mknod /dev/vg106BEBarch/group c 64 0x

to get the ll /dev/vg106BEBarch/group
and see what it was.

vgimport -m /tmp/my.map vg106BEBarch /dev/dsk/c3t6d4 /dev/dsk/c3t6d5 /dev/dsk/c4t6d4 /dev/dsk/c4t6d5

Check if it works without those two disks. If you wish to bring them back, do the same, but add them to the "vgimport" command.

[Global Unix Team]

Thanks to all of you for you help. Unfortunately, I won't be able to tell you who was right, we found a workaround by using another vg, this one will be removed, which will very likely solve the problem :-)