topic Re: missing PV in Operating System - HP-UX

missing PV

Michael O'brien_1 — Wed, 06 Aug 2008 12:09:24 GMT

Hi,

A user was complain of getting IO write errors
when copying files to a filesystem.

When I did an lvdisplay -v on the logical volume. I notice that the pv was not shown for some LE's/PE'1. Please see below.

All of the disks in the lv are reporting as being available and I'm also not seeing "no hardware" errors when I run ioscan -fnC disk.

My question is how can I identify the missing disk ???, as I guess this is were the problem lies. My next step was to try and umount the filesystem and fsck the filesystem, mount it back up and see if this fixes the problem. Any suggestions as I've never see this issue before.

Thanks
michael

02703 /dev/dsk/c19t2d4 00355 current
02704 ??? 00000 current
02705 ??? 00001 current
02706 ??? 00002 current
02707 ??? 00003 current
02708 ??? 00004 current
02709 ??? 00005 current
02710 ??? 00006 current
02711 ??? 00007 current
02712 ??? 00008 current
02713 ??? 00009 current
02714 ??? 00010 current
02715 ??? 00011 current
02716 ??? 00012 current
02717 ??? 00013 current
02718 ??? 00014 current
02719 ??? 00015 current
02720 ??? 00016 current
02721 ??? 00017 current
02722 ??? 00018 current
02723 ??? 00019 current
02724 ??? 00020 current
02725 ??? 00021 current
02726 ??? 00022 current
02727 ??? 00023 current
02728 ??? 00024 current
02729 ??? 00025 current
02730 ??? 00026 current
02731 ??? 00027 current
02732 ??? 00028 current
02733 ??? 00029 current
02734 ??? 00030 current
02735 ??? 00031 current
02736 ??? 00032 current
02737 ??? 00033 current
02738 ??? 00034 current
02739 ??? 00035 current
02740 ??? 00036 current
02741 ??? 00037 current
02742 ??? 00038 current
02743 ??? 00039 current
02744 ??? 00040 current
02745 ??? 00041 current
02746 ??? 00042 current
02747 ??? 00043 current
02748 ??? 00044 current
02749 ??? 00045 current
02750 ??? 00046 current
02751 ??? 00047 current
02752 ??? 00048 current
02753 ??? 00049 current
02754 ??? 00050 current
02755 ??? 00051 current
02756 ??? 00052 current
02757 ??? 00053 current
02758 ??? 00054 current
02759 ??? 00055 current
02760 ??? 00056 current
02761 ??? 00057 current
02762 ??? 00058 current
02763 ??? 00059 current
02764 ??? 00060 current
02765 ??? 00061 current
02766 ??? 00062 current
02767 ??? 00063 current
02768 ??? 00064 current
02769 ??? 00065 current
02770 ??? 00066 current
02771 ??? 00067 current
02772 ??? 00068 current
02773 ??? 00069 current
02774 ??? 00070 current
02775 ??? 00071 current
02776 ??? 00072 current
02777 ??? 00073 current
02778 ??? 00074 current
02779 ??? 00075 current
02780 ??? 00076 current
02781 ??? 00077 current
02782 ??? 00078 current
02783 /dev/dsk/c15t5d0 00463 current
02784 /dev/dsk/c15t5d0 00464 current
02785 /dev/dsk/c15t5d0 00465 current

Re: missing PV

TTr — Wed, 06 Aug 2008 12:20:13 GMT

Run "vgdisplay -v /dev/vgXX" for the volume group that contains this LV. It should show you the missing PV. You should try fixing the problem at the VG level first by recovering the PV before you get in the LV and filesystem.

Re: missing PV

Michael O'brien_1 — Wed, 06 Aug 2008 12:28:49 GMT

Thanks for you reply,

All of the pv in the vg are showing as available. That's whats so strange about this problem. When I first notice the issue I checked to see if any of the pv's were showing as not available, using vgdisplay, ioscan and diskinfo.

Re: missing PV

TTr — Wed, 06 Aug 2008 13:03:00 GMT

This is really odd. Are your PVS simple JBOD disks or are they LUNs from an array. Maybe the array lost part of a LUN somehow. You have 80 extends missing. Does that correlate to anything? Do you have an old listing that may show the missing PV or do you know that there is really a missing PV or are the missing extends part from an existing PV?

Re: missing PV

Michael O'brien_1 — Wed, 06 Aug 2008 13:19:37 GMT

The makeup of the vg it contains 43 pv's with 9 lv's, each lv is mounted as a sperate filesystem. So the lv's are distributed accross all 43 pv's. The lv with the problem is distributed accross 23 pv's. The pv's are presented from HDS array.
None of the other lv's are displaying any problems but they must be sharing the same pv's as the faulty lv.

I'm checking each pv which contains over 80 pe's for the faulty lv and they all seem okay. I guess I right trying to identify the missing pv looking for extents for the lv starting at 001 on the each pv?

Re: missing PV

Michael O'brien_1 — Wed, 06 Aug 2008 13:34:53 GMT

I've just ran a pvdisplay -v on all the disks in the vg, I'm unable to locate the LE's 02704-02782 for the faulty lv?

Re: missing PV

TTr — Wed, 06 Aug 2008 16:22:50 GMT

Are there any free PEs in the PVs? Is there a block of 80 free PEs somewhere?
> I guess I right trying to identify the missing pv looking for extents for the lv starting at 001 on the each pv?

Not necessarily. Since the PV does not show up in the listing for the missings LEs, the PE numbering may default back to zero. I worked with an HP label HDS 8 years ago and it used the open-8 LUN structure where each LUN had 6 slices corresponding to 6 multipathed disk devices. Usually you assigned all slices of a LUN to a server which does not agree with the number 43 that you have for the number of PVs. But it may be OK. I can't think of anything else to check here. It would help if you had an old listing of (vg/lv)display.

Re: missing PV

psreedhar — Thu, 07 Aug 2008 05:22:18 GMT

There are two possibilities, for LVM to show ??? in lvdisplay.

1) When the PV was available during VG activation.

2) The PV DSF file is removed by some one.

# lvdisplay -v /dev/vgtest/lvol1
--- Logical volumes ---
LV Name /dev/vgtest/lvol1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 40
Current LE 10
Allocated PE 10
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/disk/disk1205 10 10

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/disk/disk1205 00000 current
00001 /dev/disk/disk1205 00001 current
00002 /dev/disk/disk1205 00002 current
00003 /dev/disk/disk1205 00003 current
00004 /dev/disk/disk1205 00004 current
00005 /dev/disk/disk1205 00005 current
00006 /dev/disk/disk1205 00006 current
00007 /dev/disk/disk1205 00007 current
00008 /dev/disk/disk1205 00008 current
00009 /dev/disk/disk1205 00009 current

# rm /dev/disk/disk1205
#
#
# lvdisplay -v /dev/vgtest/lvol1
lvdisplay: Warning: couldn't query physical volume "/dev/disk/disk1205":
The specified path does not correspond to physical volume attached to
this volume group
lvdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vgtest/lvol1
VG Name /dev/vgtest
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 40
Current LE 10
Allocated PE 10
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Logical extents ---
LE PV1 PE1 Status 1
00000 ??? 00000 current
00001 ??? 00001 current
00002 ??? 00002 current
00003 ??? 00003 current
00004 ??? 00004 current
00005 ??? 00005 current
00006 ??? 00006 current
00007 ??? 00007 current
00008 ??? 00008 current
00009 ??? 00009 current

I suspect your problem comes under second category. But even in the second case vgdisplay output through errors to stderr, saying that PV DSF is missing.

Check the stderr of vgdisplay, you must get some messages.

Re: missing PV

Sharma Sanjeev — Thu, 07 Aug 2008 06:05:11 GMT

Hi

@A user was complain of getting IO write errors when copying files to a filesystem.

First Check which disks belongs to that Filesystem with lvdisplay and than check the disk with dd command.
in and out should be same
Regards
Sanjeev

Re: missing PV

Michael O'brien_1 — Thu, 07 Aug 2008 07:26:25 GMT

I think my best option is to backup the data and recreate the logical volume.

Thanks for all your help.

Re: missing PV

Sharma Sanjeev — Thu, 07 Aug 2008 07:43:56 GMT

Hi

But if u are getting i/o error while writing or reading how u can take a backup ?

Re: missing PV

Michael O'brien_1 — Thu, 07 Aug 2008 08:07:05 GMT

Hi,

We are trying to copy any data we can which isn't effected by the missing pv's

Re: missing PV

vjta — Mon, 01 Sep 2008 10:29:39 GMT

hi!

u check that pv with diskinfo & dd also checks in ioscan -fnCdisk
if everything ok

then
u can reduce vg by removing that pv by
vgreduce -l
this will removes entry from lvmtab

now recreate pv on same file by -f

extend the same vg with this pv.

regards
vjta

Re: missing PV

Deepak Kr — Mon, 01 Sep 2008 12:33:55 GMT

you can run fsck with -F switch to check file system error on mounted FS

refer to man fsck for more info.