topic Re: EVA LUNS PVLink Failed! The PV is still accessible in Operating System - HP-UX

EVA LUNS PVLink Failed! The PV is still accessible

cam9269 — Tue, 08 Dec 2009 04:56:12 GMT

Hi Gurus, need advise about this issue. This happened after I added 2 new LUNs to my server. At the moment, it giving me these sets of errors every 2minutes and it's flooding the syslog already.

I've checked the EVA logs and didn't see any errors. Also checked the FC cards and they are both fine.

On the switch side though, I don't know what and how to check it.

But for this particular error, what other things can I do to fix it?

sample log entry
==============================================
Dec 8 15:35:33 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x0c0200 Failed! The PV is still accessible.
Dec 8 15:35:33 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x0d0200 Failed! The PV is still accessible.
Dec 8 15:35:33 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x100200 Failed! The PV is still accessible.
Dec 8 15:35:33 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x110200 Failed! The PV is still accessible.
Dec 8 15:36:04 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x0c0200 Recovered.
Dec 8 15:36:04 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x0d0200 Recovered.
Dec 8 15:36:04 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x100200 Recovered.
Dec 8 15:36:04 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x110200 Recovered.
=============================================

Thanks!

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Tue, 08 Dec 2009 05:18:44 GMT

>>>>Dec 8 15:36:04 syfcispr vmunix: LVM: VG 64 0x010000: PVLink 31 0x110200 Recovered.<<<

take look at

EMS - events,

/var/opt/resmon/log/event.log -> check this file,

vgdisplay -v |more --> look for any stale extents

for me it looks like "PV" has been recovered,but you need check with "dd" command to perfrom through check of the disk

demsg

or if you post the vgdisplay -v output will helpfull to narrow down

Re: EVA LUNS PVLink Failed! The PV is still accessible

cam9269 — Tue, 08 Dec 2009 05:43:22 GMT

Hi Johnson,

I didn't find any EMS events which is connected to this issue.

Nor where there anything on the event.log file

No stale extents from vgdisplay (pls. see o/p below)

Here's the vgdisplay -v o/p:

# vgdisplay -v VGSAP
--- Volume groups ---
VG Name /dev/VGSAP
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 23
Open LV 23
Max PV 16
Cur PV 3
Act PV 3
Max PE per PV 64000
VGDA 6
PE Size (Mbytes) 16
Total PE 92785
Alloc PE 62864
Free PE 29921
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/VGSAP/LVSWAP
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 1250
Allocated PE 1250
Used PV 1

LV Name /dev/VGSAP/sapinst
LV Status available/syncd
LV Size (Mbytes) 35040
Current LE 2190
Allocated PE 2190
Used PV 1

LV Name /dev/VGSAP/tmp_backup
LV Status available/syncd
LV Size (Mbytes) 350000
Current LE 21875
Allocated PE 21875
Used PV 1

LV Name /dev/VGSAP/PDE_TRANS
LV Status available/syncd
LV Size (Mbytes) 30000
Current LE 1875
Allocated PE 1875
Used PV 1

LV Name /dev/VGSAP/oracle
LV Status available/syncd
LV Size (Mbytes) 192
Current LE 12
Allocated PE 12
Used PV 1

LV Name /dev/VGSAP/ora_PDE
LV Status available/syncd
LV Size (Mbytes) 112
Current LE 7
Allocated PE 7
Used PV 1

LV Name /dev/VGSAP/ora_client
LV Status available/syncd
LV Size (Mbytes) 112
Current LE 7
Allocated PE 7
Used PV 1

LV Name /dev/VGSAP/ora_stage
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 375
Allocated PE 375
Used PV 1

LV Name /dev/VGSAP/ora_PRE
LV Status available/syncd
LV Size (Mbytes) 736
Current LE 46
Allocated PE 46
Used PV 1

LV Name /dev/VGSAP/PRE_92064
LV Status available/syncd
LV Size (Mbytes) 5504
Current LE 344
Allocated PE 344
Used PV 1

LV Name /dev/VGSAP/PRE_mirrlogA
LV Status available/syncd
LV Size (Mbytes) 208
Current LE 13
Allocated PE 13
Used PV 1

LV Name /dev/VGSAP/PRE_mirrlogB
LV Status available/syncd
LV Size (Mbytes) 208
Current LE 13
Allocated PE 13
Used PV 1

LV Name /dev/VGSAP/PRE_oraarch
LV Status available/syncd
LV Size (Mbytes) 40000
Current LE 2500
Allocated PE 2500
Used PV 1

LV Name /dev/VGSAP/PRE_origlogA
LV Status available/syncd
LV Size (Mbytes) 208
Current LE 13
Allocated PE 13
Used PV 1

LV Name /dev/VGSAP/PRE_origlogB
LV Status available/syncd
LV Size (Mbytes) 208
Current LE 13
Allocated PE 13
Used PV 1

LV Name /dev/VGSAP/PRE_sapdata1
LV Status available/syncd
LV Size (Mbytes) 120000
Current LE 7500
Allocated PE 7500
Used PV 1

LV Name /dev/VGSAP/PRE_sapdata2
LV Status available/syncd
LV Size (Mbytes) 120000
Current LE 7500
Allocated PE 7500
Used PV 1

LV Name /dev/VGSAP/PRE_sapdata3
LV Status available/syncd
LV Size (Mbytes) 120000
Current LE 7500
Allocated PE 7500
Used PV 1

LV Name /dev/VGSAP/PRE_sapdata4
LV Status available/syncd
LV Size (Mbytes) 120000
Current LE 7500
Allocated PE 7500
Used PV 1

LV Name /dev/VGSAP/PRE_sapreorg
LV Status available/syncd
LV Size (Mbytes) 5008
Current LE 313
Allocated PE 313
Used PV 1

LV Name /dev/VGSAP/PRE_mnt
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/VGSAP/PRE_sap
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/VGSAP/LVSWAP2
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 1250
Allocated PE 1250
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c12t0d1
PV Name /dev/dsk/c13t0d1 Alternate Link
PV Name /dev/dsk/c14t0d1 Alternate Link
PV Name /dev/dsk/c15t0d1 Alternate Link
PV Name /dev/dsk/c16t0d1 Alternate Link
PV Name /dev/dsk/c17t0d1 Alternate Link
PV Name /dev/dsk/c18t0d1 Alternate Link
PV Name /dev/dsk/c19t0d1 Alternate Link
PV Status available
Total PE 63991
Free PE 1127
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c12t0d2
PV Name /dev/dsk/c13t0d2 Alternate Link
PV Name /dev/dsk/c14t0d2 Alternate Link
PV Name /dev/dsk/c15t0d2 Alternate Link
PV Name /dev/dsk/c16t0d2 Alternate Link
PV Name /dev/dsk/c17t0d2 Alternate Link
PV Name /dev/dsk/c18t0d2 Alternate Link
PV Name /dev/dsk/c19t0d2 Alternate Link
PV Status available
Total PE 12797
Free PE 12797
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c12t0d3
PV Name /dev/dsk/c13t0d3 Alternate Link
PV Name /dev/dsk/c14t0d3 Alternate Link
PV Name /dev/dsk/c15t0d3 Alternate Link
PV Name /dev/dsk/c17t0d3 Alternate Link
PV Name /dev/dsk/c16t0d3 Alternate Link
PV Name /dev/dsk/c18t0d3 Alternate Link
PV Status available
Total PE 15997
Free PE 15997
Autoswitch On
Proactive Polling On

Thanks again.

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Tue, 08 Dec 2009 06:40:30 GMT

than you no need to worry .

can you post the below output

ll /dev/vg*/group

in order to confirm, exactly "0x010000" is refering to VGSAP

Re: EVA LUNS PVLink Failed! The PV is still accessible

cam9269 — Tue, 08 Dec 2009 06:56:16 GMT

Here's the o/p:
# ll /dev/*/group
cr--r--r-- 1 root sys 64 0x010000 Jan 25 2006 /dev/VGSAP/group
crw-r----- 1 root sys 64 0x000000 Jan 6 2006 /dev/vg00/group

The trouble is, there is high I/O on these LUNs when I look at them thru the monitoring software we're using. The service time goes to as high as 100000ms! and it worries me when it does that.

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Tue, 08 Dec 2009 07:37:03 GMT

if you are realy worried, about I/O ,

we can check using the "dd" command to verify the disks, still if you see any issue, better to log call to HP - HW call

run below command to check

dd if=/dev/rdsk/c12t0d1 of=/dev/null bs=256k &
dd if=/dev/rdsk/c12t0d2 of=/dev/null bs=256k &
dd if=/dev/rdsk/c12t0d3 of=/dev/null bs=256k &

take note "record in & out " should be same, if not you have "BAD" disk

sample :-

560057+1 records in
560057+1 records out

Re: EVA LUNS PVLink Failed! The PV is still accessible

Prasanth Thomas — Tue, 08 Dec 2009 12:34:26 GMT

Hi,

As i can see one link is missing in the vgdisplay o/p, total should be 8 but here its showing7 and please check the same or recreate the device file by insf -H harware path command.

PV Name /dev/dsk/c12t0d3
PV Name /dev/dsk/c13t0d3 Alternate Link
PV Name /dev/dsk/c14t0d3 Alternate Link
PV Name /dev/dsk/c15t0d3 Alternate Link
PV Name /dev/dsk/c17t0d3 Alternate Link
PV Name /dev/dsk/c16t0d3 Alternate Link
PV Name /dev/dsk/c18t0d3 Alternate Link
PV Status available
Total PE 15997
Free PE 15997
Autoswitch On
Proactive Polling On

Reagards,
Prasanth Thomas.

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Wed, 09 Dec 2009 02:54:15 GMT

insf -e

ioscan -fn

ioscan -fnC disk

vgdisplay -v VGSAP,

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Wed, 09 Dec 2009 03:02:18 GMT

>>As i can see one link is missing in the vgdisplay o/p, total should be 8 but here <<<

# insf -e

-e Reinstall the special files for pseudo-drivers and
existing devices. This is useful for restoring
special files if one or more have been removed.

Once device file has been claimed or shown

# vgextend

Re: EVA LUNS PVLink Failed! The PV is still accessible

Johnson Punniyalingam — Wed, 09 Dec 2009 03:13:59 GMT

Once your issue has been resolved...

D'not forget assign points :)

Re: EVA LUNS PVLink Failed! The PV is still accessible

johnsonpk — Wed, 09 Dec 2009 05:25:16 GMT

Hi,

Is all the three LUN in that vg from same EVA? What is EVA model? can you compare the PV time out value for all the three LUN (pvdisplay )

PV Name /dev/dsk/c12t0d1
PV Name /dev/dsk/c12t0d2
PV Name /dev/dsk/c12t0d3

your sample log entry shows the culprit as disk /dev/dsk/c12t0d2 .. is there any logs for other pvs ?(look for PVLink 31 0x??0100 and 0x??0300 )
if it is only for the newly added PVs , try increasing the pv timeout values to 90 or 180 ( pvchange -t 180 )

Regards
Johnson

Re: EVA LUNS PVLink Failed! The PV is still accessible

cam9269 — Wed, 09 Dec 2009 05:32:10 GMT

Hi Johnson,

Yep, all LUNs came from the same EVA (EVA3000)
pvtimeout = 120 for all LUNs.

There are also other PVs generating the error.

Last night I took out these PVs from the VG alternates, and the error stopped.

The question now is, why am I getting the PVlink errors when I use those alternate links? Any ideas?

Thanks!

Re: EVA LUNS PVLink Failed! The PV is still accessible

johnsonpk — Wed, 09 Dec 2009 06:22:46 GMT

Hi, you can stop logging those time out error in the syslog by disable pro-active polling for pvs (pvchange -p n )for the time being (better than removing all alternate links). However ,try isolating issues by just adding alternative links from same HBA or add one at a time ...

regards
johnson

Re: EVA LUNS PVLink Failed! The PV is still accessible

cam9269 — Wed, 09 Dec 2009 06:30:42 GMT

Hi Johnson,

The problem is, when I add these links, they not only create those error messages, but also create very, very high I/O on the system.

The OS is complaining about excessive I/O errors from the device and I think it will create bottlenecks if ever the new space is allocated to an LV.

Is there some documents which discusses LVM stuff in a deeper sense? I need to find out how these things work so that I can it explain better and perhaps put up a solution for the issue.

Thanks!
Chris

Re: EVA LUNS PVLink Failed! The PV is still accessible

johnsonpk — Wed, 09 Dec 2009 06:44:50 GMT

Hi Chris,

I am sorry to say that i don't have any .Log a call with HP for better analysis..

It may be because of an issue with specific HBA , HBA firmware, or driver .Try update it with latest patches .(alternatively add link from only one hba and see if it with a particular HBA.

Regard!
Johnson

Re: EVA LUNS PVLink Failed! The PV is still accessible

Prasanth Thomas — Wed, 09 Dec 2009 06:57:32 GMT

Hi,

Please let me know the server model,OS and the output of ioscan -fnkC fc

When did u upgrade the pathes (especially HBA) on this server?.
Better you can do a patch assessment on this server and chek. Please fnd the below link to do the same.

http://www13.itrc.hp.com/service/patch/addUpdateSystemsPage.do?BC=main|assess|

Regards,
Prasanth Thomas.