Urgent : EMC disks not detecting, ?

[rveri]

Hi All,

Initially we got this error in syslog and dmesg:
----------------------------

-----------------------------------------------------------------------------------------


Dec 17 00:51:17 ddsapd02 su: + tty?? root-daqadm
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f4ae700) to PV 4 in VG 41 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x000000007af9c800)
, from raw device 0x1f4ae700 (with priority: 0, and current flags: 0x40) to raw device 0x1f498200
(with priority: 1, and current flags: 0x0).
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f497600) to PV 2 in VG 40 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x000000007b6aa840)
, from raw device 0x1f4ae600 (with priority: 0, and current flags: 0x40) to raw device 0x1f497200
(with priority: 1, and current flags: 0x0).
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f497700) to PV 0 in VG 41 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f498000) to PV 1 in VG 41 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f498200) to PV 4 in VG 41 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f4f1600) to PV 0 in VG 48 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f4f1700) to PV 1 in VG 48 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f498100) to PV 2 in VG 48 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f497200) to PV 4 in VG 48 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f497400) to PV 0 in VG 49 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f4f1400) to PV 1 in VG 49 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Path (device 0x1f496200) to PV 2 in VG 49 Failed!
Dec 17 09:11:07 ddsapd02 vmunix: lv_readvgdats: Could not read VGDA 1 header & trailer from disk
H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
Dec 17 09:11:07 ddsapd02 vmunix: LVM: Failed to restore PV 4 to VG 48!
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[49]: pvnum=0 (dev_t=0x1f4a7400) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[49]: pvnum=1 (dev_t=0x1f4a7500) is POWERFAILED
Dec 17 09:11:07 ddsapd02 vmunix: lv_readvgdats: Could not read VGDA 1 header & trailer from disk
H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
Dec 17 09:11:08 ddsapd02 above message repeats 3 times
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[49]: pvnum=2 (dev_t=0x1f4ae300) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[40]: pvnum=2 (dev_t=0x1f4a7600) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[41]: pvnum=0 (dev_t=0x1f4a7700) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[41]: pvnum=1 (dev_t=0x1f4a8000) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[41]: pvnum=4 (dev_t=0x1f498200) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[48]: pvnum=0 (dev_t=0x1f501600) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[48]: pvnum=1 (dev_t=0x1f501700) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[48]: pvnum=2 (dev_t=0x1f4a8100) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[41]: pvnum=4 (dev_t=0x1f498200) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[48]: pvnum=4 (dev_t=0x1f497200) is POWERFAILED
Dec 17 09:11:08 ddsapd02 vmunix: LVM: vg[48]: pvnum=4 (dev_t=0x1f497200) is POWERFAILED
==============================================


root@ddsapd02 [/]
# dmesg

Dec 18 23:33
...
led!
LVM: Path (device 0x1f497200) to PV 4 in VG 48 Failed!
LVM: Path (device 0x1f497400) to PV 0 in VG 49 Failed!
LVM: Path (device 0x1f4f1400) to PV 1 in VG 49 Failed!
LVM: Path (device 0x1f496200) to PV 2 in VG 49 Failed!
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/0/0.98.32.19.0.7.2 (error = 5)
LVM: Failed to restore PV 4 to VG 48!
LVM: vg[49]: pvnum=0 (dev_t=0x1f4a7400) is POWERFAILED
LVM: vg[49]: pvnum=1 (dev_t=0x1f4a7500) is POWERFAILED
LVM: vg[49]: pvnum=2 (dev_t=0x1f4ae300) is POWERFAILED
LVM: vg[40]: pvnum=2 (dev_t=0x1f4a7600) is POWERFAILED
LVM: vg[41]: pvnum=0 (dev_t=0x1f4a7700) is POWERFAILED
LVM: vg[41]: pvnum=1 (dev_t=0x1f4a8000) is POWERFAILED
LVM: vg[41]: pvnum=4 (dev_t=0x1f498200) is POWERFAILED
LVM: vg[41]: pvnum=4 (dev_t=0x1f498200) is POWERFAILED
LVM: vg[48]: pvnum=0 (dev_t=0x1f501600) is POWERFAILED
LVM: vg[48]: pvnum=1 (dev_t=0x1f501700) is POWERFAILED
LVM: vg[48]: pvnum=2 (dev_t=0x1f4a8100) is POWERFAILED
LVM: vg[48]: pvnum=4 (dev_t=0x1f497200) is POWERFAILED
LVM: vg[48]: pvnum=4 (dev_t=0x1f497200) is POWERFAILED
SCSI TAPE: dev = 0xcd487000 Failed open - st_san_safe enabled

=========================================================================



Also in dmesg :

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 16777216 Kbytes, lockable: 15659592 Kbytes, available: 14951116 Kbytes

msgcnt 1 vxfs: mesg 021: vx_fs_init - /dev/vg10dav/lvol1 file system validation failure
msgcnt 2 vxfs: mesg 021: vx_fs_init - /dev/vg09dav/lvol1 file system validation failure
SCSI TAPE: dev = 0xcd487000 Failed open - st_san_safe enabled
SCSI TAPE: dev = 0xcd479000 Failed open - st_san_safe enabled
SCSI TAPE: dev = 0xcd46a000 Failed open - st_san_safe enabled
SCSI TAPE: dev = 0xcd45b000 Failed open - st_san_safe enabled
=====================================================================


Here are the missing disks: This was taken prior to reboot , and after reboot these are not in ioscan.
------------------------------------------
disk 199 0/2/0/0.97.20.19.0.14.6 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c74t14d6 /dev/rdsk/c74t14d6
disk 200 0/2/0/0.97.20.19.0.14.7 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c74t14d7 /dev/rdsk/c74t14d7

disk 193 1/0/0/0.98.32.19.0.6.2 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t6d2 /dev/rdsk/c73t6d2
disk 197 1/0/0/0.98.32.19.0.7.2 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t7d2 /dev/rdsk/c73t7d2
disk 89 1/0/0/0.98.32.19.0.7.4 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t7d4 /dev/rdsk/c73t7d4
disk 90 1/0/0/0.98.32.19.0.7.6 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t7d6 /dev/rdsk/c73t7d6
disk 91 1/0/0/0.98.32.19.0.7.7 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t7d7 /dev/rdsk/c73t7d7
disk 92 1/0/0/0.98.32.19.0.8.0 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t8d0 /dev/rdsk/c73t8d0
disk 93 1/0/0/0.98.32.19.0.8.1 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t8d1 /dev/rdsk/c73t8d1
disk 198 1/0/0/0.98.32.19.0.8.2 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c73t8d2 /dev/rdsk/c73t8d2

disk 142 1/0/0/0.98.32.19.2.1.4 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c79t1d4 /dev/rdsk/c79t1d4
disk 149 1/0/0/0.98.32.19.2.1.6 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c79t1d6 /dev/rdsk/c79t1d6
disk 150 1/0/0/0.98.32.19.2.1.7 sdisk NO_HW DEVICE EMC SYMMETRIX
/dev/dsk/c79t1d7 /dev/rdsk/c79t1d7
--------------------------------------------------------------------------------------------------------------------
This output was before reboot the server , and after reboot , this devices are not visible to the system.


EMC checked manytimes and said that there is no disk failure and everything is ok.


Also told us to run rmsf , then insf -e , ioscan , reboot and check . we did that but nothing happened.


Because of thiss 2 SAP Filesystem , affected , and belongs to vg10dav , and vg09dav.

# strings lvmtab/ vgdisplay to vg10dav and vg09dav attached.

Please help and suggest asap.
Please suggest how to resove this issue, where is wrong!

Thanks,

[Geoff Wild · Accepted solution]

Looks like someone un-masked those disks from your server....

IE - your server no longer has access to them through the san - EMC is probably right - nothing wrong with the disks at all....

Rgds...Geoff

[Darrel Louis]

Hi,

Are the disks connected through Fibre.
Can you check the status of your Fibre devices:
/opt/fcms/bin/fcmsutil

Missed that in the ioscan:
run the following:
ioscan -fnCfc

GoodLuck

Darrel

[rveri]

Hi ,

Geoff's reply looks impressive ,

Well the error says the date stamp on 17th , and one of our storage team member , extended few file system on this box by doing something , not sure, if it is giving any impact , anything unmapped, .
And hope that could caused some issue.


Here is the iocan -fnC fc , output:
-------
# ioscan -fnC fc
Class I H/W Path Driver S/W State H/W Type Description
=================================================================
fc 1 0/2/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td1
fc 2 1/0/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td2
fc 0 1/8/0/0 td CLAIMED INTERFACE HP Tachyon TL/TS Fibre Channel Mass Storage Adapter
/dev/td0
#

Also fcmsutil shows all ONLINE.

Thanks ,

[Christine Hartman]

do you have access to ECC? if so you need to look and see if the devices are still mapped to your HBAs. Also you need to check your switch for zoning and make sure nothing has happened to your zoning...there should be a zone setup for each HBA. One last thing to check is the ports that each HBA is connected to...I believe it is switchportshow....to make sure nothing is wrong with the port on your switch. You can also reset the gbic (port your HBA is connected to)...just pull it out and put it back in. Make sure that your switch port doesn't have any amber lights and that both lights are lit up.
All of these things could affect connectivity to your EMC disks.

[Darrel Louis]

Hi,

Before doing any changes:
Can you verify with your storage group if the configuration is still in tact.

Try the following:
/usr/symcli/bin/syminq
Do you still see the disks.

GoodLuck

Darrel

[Christine Hartman]

also try running ioscan -fnC disk |more
and see if you see any new entries for EMC disks that maybe don't have their disk device names created yet...if so...run insf on them.....it could also be if your fiber connection was moved on your switch to another port....your disks will still be there but will have all new c#t#d# names...this is a known behavior...and I've seen this done before.

[Darrel Louis]

But that one can be nasty. If the Fibre cables are switched, this means that your device ID's has been changed and I don't think LVM likes that or can handle that.

Get it clear what storage has done.

Darrel

[rveri]

Thanks to all , the problem solved , with vgscan -p -v a & moved lvmtab & rescanned agained it without preview , and activated , and mounted. Even logged call to HP and talked many times but no one insisted us to check with vgscan . Its looks like something magic happened after i posted to itrc. though we were strugling for this near abt20hrs at a streach.

Thanks all ,for the quick response.