LVM related messages in SYSLOG.

[Karl_]

Hi,

Need help with some LVM messages in our syslog. What does this mean? How can we fix this. (Points are awarded when thread is closed).

syslog
=============================================
May 15 09:02:46 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Failed! The PV is still accessible.
May 15 09:02:46 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x060200 Failed! The PV is still accessible.
May 15 09:03:06 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069b2a000), from raw device 0x1f080
200 (with priority: 2, and current flags: 0x0) to raw device 0x1f040200 (with priority: 0, and current flags: 0x80).
May 15 09:03:06 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Recovered.
May 15 09:03:10 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x060400 Failed! The PV is still accessible.
May 15 09:03:10 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x080400 Failed! The PV is still accessible.
May 15 09:03:21 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x060400 Recovered.
May 15 09:03:36 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x080400 Recovered.
May 15 09:03:36 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x0a0400 Recovered.
May 15 09:03:36 saphost01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x0a0100 Recovered.
May 15 09:03:41 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069b2a000), from raw device 0x1f040
200 (with priority: 0, and current flags: 0x40) to raw device 0x1f080200 (with priority: 2, and current flags: 0x0).
May 15 09:03:41 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Failed! The PV is still accessible.
May 15 09:03:56 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069b2a000), from raw device 0x1f080
200 (with priority: 2, and current flags: 0x0) to raw device 0x1f040200 (with priority: 0, and current flags: 0x80).
May 15 09:03:56 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Recovered.
May 15 09:03:56 saphost01 vmunix: LVM: VG 64 0x020000: PVLink 31 0x060200 Recovered.
May 15 09:04:01 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069af2000), from raw device 0x1f080
100 (with priority: 2, and current flags: 0x0) to raw device 0x1f040100 (with priority: 0, and current flags: 0x80).
May 15 09:04:01 saphost01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x040100 Recovered.
May 15 09:04:36 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069af2000), from raw device 0x1f040
100 (with priority: 0, and current flags: 0x40) to raw device 0x1f080100 (with priority: 2, and current flags: 0x0).
May 15 09:04:36 saphost01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x040100 Failed! The PV is still accessible.
May 15 09:04:36 saphost01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x0a0100 Failed! The PV is still accessible.
May 15 09:04:37 saphost01 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000069bd8000), from raw device 0x1f080
300 (with priority: 2, and current flags: 0x0) to raw device 0x1f040300 (with priority: 0, and current flags: 0x80).
May 15 09:04:37 saphost01 vmunix: LVM: VG 64 0x030000: PVLink 31 0x040300 Recovered.
May 15 09:04:41 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x060400 Failed! The PV is still accessible.
May 15 09:04:41 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x080400 Failed! The PV is still accessible.
May 15 09:04:41 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x0a0400 Failed! The PV is still accessible.
May 15 09:04:41 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x060400 Recovered.
May 15 09:04:41 saphost01 vmunix: LVM: VG 64 0x040000: PVLink 31 0x080400 Recovered.
==============================================

Thanks,

Karl

[freddy_21]

Hello Karl,

Please check ioscan output. I think one of your link have problem. Do you use alternate link?

Please provide : /etc/lvmtab, vgdisplay and ioscan output. fcmustil from both fc card ( if your configuration using fc connection)


thanks
Freddy

[Karl_]

Hi Freddy,

Here are the outputs as requested.
=====================================
/etc/lvmtab
=====================================
# strings lvmtab
/dev/vg00
/dev/dsk/c0t6d0
/dev/dsk/c3t6d0
/dev/DNEevDat
/dev/dsk/c4t0d1
/dev/dsk/c6t0d1
/dev/dsk/c8t0d1
/dev/dsk/c10t0d1
/dev/DNEevApp
/dev/dsk/c4t0d2
/dev/dsk/c6t0d2
/dev/dsk/c8t0d2
/dev/dsk/c10t0d2
/dev/TNEstApp
/dev/dsk/c4t0d3
/dev/dsk/c6t0d3
/dev/dsk/c8t0d3
/dev/dsk/c10t0d3
/dev/TNEstDat
/dev/dsk/c4t0d4
/dev/dsk/c6t0d4
/dev/dsk/c8t0d4
/dev/dsk/c10t0d4

=====================================
vgdisplay
=====================================

# vgdisplay -v
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4350
VGDA 4
PE Size (Mbytes) 8
Total PE 8680
Alloc PE 8642
Free PE 38
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 320
Current LE 40
Allocated PE 80
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 17504
Current LE 2188
Allocated PE 4376
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 232
Current LE 29
Allocated PE 58
Used PV 2

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 56
Current LE 7
Allocated PE 14
Used PV 2

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 5000
Current LE 625
Allocated PE 1250
Used PV 2

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 504
Current LE 63
Allocated PE 126
Used PV 2

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 1912
Current LE 239
Allocated PE 478
Used PV 2

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 6504
Current LE 813
Allocated PE 1626
Used PV 2

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 2536
Current LE 317
Allocated PE 634
Used PV 2


--- Physical volumes ---
PV Name /dev/dsk/c0t6d0
PV Status available
Total PE 4340
Free PE 19
Autoswitch On

PV Name /dev/dsk/c3t6d0
PV Status available
Total PE 4340
Free PE 19
Autoswitch On


VG Name /dev/DNEevDat
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 13
Open LV 13
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 64000
VGDA 2
PE Size (Mbytes) 16
Total PE 15997
Alloc PE 14906
Free PE 1091
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/DNEevDat/oracle
LV Status available/syncd
LV Size (Mbytes) 208
Current LE 13
Allocated PE 13
Used PV 1

LV Name /dev/DNEevDat/ora_client
LV Status available/syncd
LV Size (Mbytes) 128
Current LE 8
Allocated PE 8
Used PV 1

LV Name /dev/DNEevDat/ora_stage
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 375
Allocated PE 375
Used PV 1

LV Name /dev/DNEevDat/ora_DNE
LV Status available/syncd
LV Size (Mbytes) 704
Current LE 44
Allocated PE 44
Used PV 1

LV Name /dev/DNEevDat/DNE_92064
LV Status available/syncd
LV Size (Mbytes) 5008
Current LE 313
Allocated PE 313
Used PV 1

LV Name /dev/DNEevDat/DNE_oraarch
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 1250
Allocated PE 1250
Used PV 1

LV Name /dev/DNEevDat/DNE_origlogA
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/DNEevDat/DNE_origlogB
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/DNEevDat/DNE_sapdata1
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1

LV Name /dev/DNEevDat/DNE_sapdata2
LV Status available/syncd
LV Size (Mbytes) 40000
Current LE 2500
Allocated PE 2500
Used PV 1

LV Name /dev/DNEevDat/DNE_sapdata3
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1

LV Name /dev/DNEevDat/DNE_sapdata4
LV Status available/syncd
LV Size (Mbytes) 60000
Current LE 3750
Allocated PE 3750
Used PV 1

LV Name /dev/DNEevDat/DNE_sapreorg
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 375
Allocated PE 375
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t0d1
PV Name /dev/dsk/c6t0d1 Alternate Link
PV Name /dev/dsk/c8t0d1 Alternate Link
PV Name /dev/dsk/c10t0d1 Alternate Link
PV Status available
Total PE 15997
Free PE 1091
Autoswitch On


VG Name /dev/DNEevApp
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 6
Open LV 6
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 51200
VGDA 2
PE Size (Mbytes) 16
Total PE 12798
Alloc PE 12403
Free PE 395
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/DNEevApp/DNE_mirrlogA
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/DNEevApp/DNE_mirrlogB
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/DNEevApp/DNE_mnt
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Used PV 1

LV Name /dev/DNEevApp/DNE_sap
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Used PV 1

LV Name /dev/DNEevApp/TRANS
LV Status available/syncd
LV Size (Mbytes) 30000
Current LE 1875
Allocated PE 1875
Used PV 1

LV Name /dev/DNEevApp/sapinst
LV Status available/syncd
LV Size (Mbytes) 160000
Current LE 10000
Allocated PE 10000
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t0d2
PV Name /dev/dsk/c6t0d2 Alternate Link
PV Name /dev/dsk/c8t0d2 Alternate Link
PV Name /dev/dsk/c10t0d2 Alternate Link
PV Status available
Total PE 12798
Free PE 395
Autoswitch On


VG Name /dev/TNEstApp
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 4
Open LV 4
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 2560
VGDA 2
PE Size (Mbytes) 16
Total PE 639
Alloc PE 403
Free PE 236
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/TNEstApp/TNE_mirrlogA
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/TNEstApp/TNE_mirrlogB
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/TNEstApp/TNE_mnt
LV Status available/syncd
LV Size (Mbytes) 2000
Current LE 125
Allocated PE 125
Used PV 1

LV Name /dev/TNEstApp/TNE_sap
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t0d3
PV Name /dev/dsk/c6t0d3 Alternate Link
PV Name /dev/dsk/c8t0d3 Alternate Link
PV Name /dev/dsk/c10t0d3 Alternate Link
PV Status available
Total PE 639
Free PE 236
Autoswitch On


VG Name /dev/TNEstDat
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 11
Open LV 11
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 64000
VGDA 2
PE Size (Mbytes) 16
Total PE 15997
Alloc PE 15996
Free PE 1
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/TNEstDat/ora_TNE
LV Status available/syncd
LV Size (Mbytes) 704
Current LE 44
Allocated PE 44
Used PV 1

LV Name /dev/TNEstDat/TNE_92064
LV Status available/syncd
LV Size (Mbytes) 5008
Current LE 313
Allocated PE 313
Used PV 1

LV Name /dev/TNEstDat/TNE_oraarch
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 1250
Allocated PE 1250
Used PV 1

LV Name /dev/TNEstDat/TNE_origlogA
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/TNEstDat/TNE_origlogB
LV Status available/syncd
LV Size (Mbytes) 224
Current LE 14
Allocated PE 14
Used PV 1

LV Name /dev/TNEstDat/TNE_sapdata1
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1

LV Name /dev/TNEstDat/TNE_sapdata2
LV Status available/syncd
LV Size (Mbytes) 40000
Current LE 2500
Allocated PE 2500
Used PV 1

LV Name /dev/TNEstDat/TNE_sapdata3
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1

LV Name /dev/TNEstDat/TNE_sapdata4
LV Status available/syncd
LV Size (Mbytes) 60000
Current LE 3750
Allocated PE 3750
Used PV 1

LV Name /dev/TNEstDat/TNE_sapreorg
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 375
Allocated PE 375
Used PV 1

LV Name /dev/TNEstDat/swap
LV Status available/syncd
LV Size (Mbytes) 23776
Current LE 1486
Allocated PE 1486
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t0d4
PV Name /dev/dsk/c6t0d4 Alternate Link
PV Name /dev/dsk/c8t0d4 Alternate Link
PV Name /dev/dsk/c10t0d4 Alternate Link
PV Status available
Total PE 15997
Free PE 1
Autoswitch On


===========================================
ioscan
===========================================
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 0 1/0/0/3/0.6.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC
/dev/dsk/c0t6d0 /dev/rdsk/c0t6d0
disk 1 1/0/0/3/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 2 1/0/1/0/0/1/1.6.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC
/dev/dsk/c3t6d0 /dev/rdsk/c3t6d0
disk 3 1/0/8/0/0.1.2.0.0.0.1 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
disk 9 1/0/8/0/0.1.2.0.0.0.2 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d2 /dev/rdsk/c4t0d2
disk 13 1/0/8/0/0.1.2.0.0.0.3 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d3 /dev/rdsk/c4t0d3
disk 18 1/0/8/0/0.1.2.0.0.0.4 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d4 /dev/rdsk/c4t0d4
disk 4 1/0/8/0/0.1.3.0.0.0.1 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c6t0d1 /dev/rdsk/c6t0d1
disk 10 1/0/8/0/0.1.3.0.0.0.2 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c6t0d2 /dev/rdsk/c6t0d2
disk 14 1/0/8/0/0.1.3.0.0.0.3 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c6t0d3 /dev/rdsk/c6t0d3
disk 17 1/0/8/0/0.1.3.0.0.0.4 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c6t0d4 /dev/rdsk/c6t0d4
disk 6 1/0/8/0/0.1.8.0.0.0.1 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c8t0d1 /dev/rdsk/c8t0d1
disk 8 1/0/8/0/0.1.8.0.0.0.2 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c8t0d2 /dev/rdsk/c8t0d2
disk 11 1/0/8/0/0.1.8.0.0.0.3 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c8t0d3 /dev/rdsk/c8t0d3
disk 16 1/0/8/0/0.1.8.0.0.0.4 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c8t0d4 /dev/rdsk/c8t0d4
disk 5 1/0/8/0/0.1.9.0.0.0.1 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c10t0d1 /dev/rdsk/c10t0d1
disk 7 1/0/8/0/0.1.9.0.0.0.2 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c10t0d2 /dev/rdsk/c10t0d2
disk 12 1/0/8/0/0.1.9.0.0.0.3 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c10t0d3 /dev/rdsk/c10t0d3
disk 15 1/0/8/0/0.1.9.0.0.0.4 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c10t0d4 /dev/rdsk/c10t0d4
=============================================

I could not run fcmustil. We are using fiber channel connection.

# fcmutil
sh: fcmutil: not found.
# fcmustil
sh: fcmustil: not found.

Thanks,

Karl

[Jov]

Hi Karl,

fcmutil might not be in your path. It should be in a ??/sbin directory or do a 'find / -name fcmutil -print' to search for it.


Jov

[Senthil Kumar .A_1 · Accepted solution]

Hi Karl,

From your vgdisplay -v output, it is clear that, there is no cause for concern. Your VG's are absolutely healthy. Those pvlink switch messages are normal in syslog. It just means that there might be some instances when the system would have lost connections to certain paths on SAN, if there were some problem on SAN fabric or array itself. Your ioscan output is fine and shows all the disk to be claimed. You need to be concerned only if you see "PV Status" as unavailable and some disks in ioscan output is displayed as "UNCLAIMED" or "NO_HW". As far as the last command goes, it is a typo. The command is "fcmsutil" not "fcmutil"

SK

[Asif Sharif]

Hi Karl,

From output sent by you it seems that no issue with paths,you can ignore these syslog error messages,ask your SAN team to check if there were any SAN switch related changes or array problems at May 15 09:04:41. If you find any "Unavailable" status, then you might have to check with the SAN team to check the SAN Switches or Storage.


Regards,
Asif Sharif

[mavrick]

Hi,

There may not be issue with the disks but the
pv path gives the syslog error. you can check for the load balancing policy and see any other link is exluded from this . then run the LB ploicy again and see.

[freddy_21]

the problem happen at May 15 09:03:XX. any problem around that time example: the fc cable unplug at that time? or San Switch off that time?

My suspect is your FC connection have some problem around that time.

How about current condition? have some error at syslog file?


thanks
freddy

[Daniel Anthony]

Hi Karl,

There are several reasons why this could have happened. Most of the reasons were already mentioned by others. But there is still 1 possibility left. As I can see from the output provided, each disks has four paths. Probably the the primary link was reduced from the VG. As there are three alternate links, there was a LUN switch to the first alternate path. To avoid these switches in this case, reduce the VG by first removing the alternate paths.

[Karl_]

Thanks for all your reply.

This problem started on May 11. Our hardware provider have been trying to solve this problem since then but without success.

That's why I made this post. I would like to verify from you guys/experts if what our hardware provider is saying is true. By the looks of it, it seems that our hardware provider are clueless of the problem.

Anyway, i have awarded points to all of you guys.

thanks for your inputs.

rgds,

Karl

[Todd Bowden]

Karl,

Can I make a guess that you are connected to and EVA SAN?

Todd

P.S. Im having some issues now that I have added alternate links to all my VG's that are connected to the SAN.