HPUX Corrupt Hardware Path

[John Mark Robeck]

hi all, I am a total rookie at this and wondered if anyone could provide me some insight. We recently had a power outage and when our systems were brought back up I found that the hardware paths associated with the LUNS presented from the SAN are not what they should be. there are present in an ioscan -fnCdisk and I beleive the ones from the san are all the ones presented as 255/255. they do not show up when I run bfd. can I repair this or can I run an ignite tape and recover the OS and have them show up again? thanks for any help with this. Again. I am totaly new to HPUX. thanks.


vuhpprod:[/]
# ioscan -fnCdisk
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 0 0/0/3/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GMAS3367NC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 4 0/1/1/1.0.0 sdisk CLAIMED DEVICE HP 36.4GST373455LC
#36
/dev/dsk/c3t0d0 /dev/rdsk/c3t0d0
disk 25 255/255/0/0.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d0 /dev/rdsk/c15t0d0
disk 26 255/255/0/0.1 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d1 /dev/rdsk/c15t0d1
disk 27 255/255/0/0.2 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d2 /dev/rdsk/c15t0d2
disk 28 255/255/0/0.3 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d3 /dev/rdsk/c15t0d3
disk 29 255/255/0/0.4 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d4 /dev/rdsk/c15t0d4
disk 30 255/255/0/0.5 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d5 /dev/rdsk/c15t0d5
disk 31 255/255/0/0.6 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d6 /dev/rdsk/c15t0d6
disk 32 255/255/0/0.7 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d7 /dev/rdsk/c15t0d7
disk 33 255/255/0/1.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d0 /dev/rdsk/c15t1d0
disk 34 255/255/0/1.1 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d1 /dev/rdsk/c15t1d1
disk 35 255/255/0/1.2 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d2 /dev/rdsk/c15t1d2
disk 36 255/255/0/1.3 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d3 /dev/rdsk/c15t1d3
disk 37 255/255/0/1.4 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d4 /dev/rdsk/c15t1d4
disk 38 255/255/0/1.5 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d5 /dev/rdsk/c15t1d5
disk 39 255/255/0/1.6 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d6 /dev/rdsk/c15t1d6
disk 40 255/255/0/1.7 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t1d7 /dev/rdsk/c15t1d7

[SoorajCleris]

Hi,

1. Whats your server model?
2. Do you have ioscan output taken before reboot?
3. "they do not show up when I run bfd." could you please explain this?

4. Is there anything not working in your server after reboot?

Regards,
Sooraj

[Jim Walls]

It looks as if the LUN H/W routes were changed some time in the past... but not noticed because they were all running, mounted and in use. Until the power went off - the "new" LUNS were then discovered during boot-up but the LVM configurations still contain the old information. Hence nothing has been mounted (not displayed by bdf (I'm assuming "bfd" is a typo).

you need to remap the LVM information.

Examine /etc/fstab to establish what is not mounted. You should be able to get a list of "missing" volume groups and logical volumes.

Because this sort of thing happens so infrequently I have not developed a more sophisticated script. See the attached listing.

Basically, together with the information you get from /etc/fstab, the first two lines give you the necessary information to recreate the volume group(s) if you need to. The list of PVs are the newly discovered ones.

The rest of the script creates a mapfile (using vgexport) and then uses that with the list of PVs to recreate the volume group.

[John Mark Robeck]

Thanks. You are correct, bdf was what I was shooting for. I'll give your suggestions a shot. I have a friend who knows HPUX comming in to help me inthe morning.

Thanks for the scripts.

[Torsten.]

What you see here

disk 25 255/255/0/0.0 sdisk CLAIMED DEVICE HSV100 HP
/dev/dsk/c15t0d0 /dev/rdsk/c15t0d0

is the virtual hardware path to the virtual array (EVA3000).

Since this old array is active/passive, your server is using securepath to access it, securepath creates and works with such virtual h/w pathes - normal behavior.

Try

# spmgr display

and please post the output of this and:

# strings /etc/lvmtab

[John Mark Robeck]

# spmgr display
Server: vuhpprod0 Report Created: Thu, Sep 30 11:45:59 2010
Command: spmgr display
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Storage: 5000-1FE1-5003-3390
Load Balance: Off Auto-restore: Off
Path Verify: On Verify Interval: 30
HBAs: td0 td1
Controller: P66C5E2AAQI05R, Operational
P66C5E2AAQI04X, Operational
Devices: c15t0d0 c15t0d1 c15t0d2 c15t0d3 c15t0d4 c15t0d5 c15t0d6
c15t0d7 c15t1d0 c15t1d1 c15t1d2 c15t1d3 c15t1d4 c15t1d5
c15t1d6 c15t1d7

TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 0 c15t0d0 6005-08B4-0010-1552-0000-6000-0339-0000 4
255/255/0/0.0
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t0d1 td0 no Active
c10t0d1 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t0d1 td0 no Standby
c12t0d1 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 1 c15t0d1 6005-08B4-0010-1552-0000-6000-033C-0000 4
255/255/0/0.1
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t0d2 td0 no Active
c10t0d2 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t0d2 td0 no Standby
c12t0d2 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 2 c15t0d2 6005-08B4-0010-1552-0000-6000-033F-0000 4
255/255/0/0.2
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t0d3 td0 no Active
c10t0d3 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t0d3 td0 no Standby
c12t0d3 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 3 c15t0d3 6005-08B4-0010-1552-0000-6000-0342-0000 4
255/255/0/0.3
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t0d4 td0 no Active
c10t0d4 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t0d4 td0 no Standby
c12t0d4 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 4 c15t0d4 6005-08B4-0010-1552-0000-6000-0345-0000 4
255/255/0/0.4
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t0d5 td0 no Active
c10t0d5 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t0d5 td0 no Standby
c12t0d5 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 5 c15t0d5 6005-08B4-0010-1552-0000-6000-0348-0000 4
255/255/0/0.5
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t0d6 td0 no Active
c10t0d6 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t0d6 td0 no Standby
c12t0d6 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 6 c15t0d6 6005-08B4-0010-1552-0000-6000-034B-0000 4
255/255/0/0.6
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t0d7 td0 no Active
c10t0d7 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t0d7 td0 no Standby
c12t0d7 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
0/ 7 c15t0d7 6005-08B4-0010-1552-0000-6000-034E-0000 4
255/255/0/0.7
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t1d0 td0 no Active
c10t1d0 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t1d0 td0 no Standby
c12t1d0 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 0 c15t1d0 6005-08B4-0010-1552-0000-6000-0357-0000 4
255/255/0/1.0
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t1d1 td0 no Active
c10t1d1 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t1d1 td0 no Standby
c12t1d1 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 1 c15t1d1 6005-08B4-0010-1552-0000-6000-035A-0000 4
255/255/0/1.1
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t1d2 td0 no Active
c10t1d2 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t1d2 td0 no Standby
c12t1d2 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 2 c15t1d2 6005-08B4-0010-1552-0000-6000-036B-0000 4
255/255/0/1.2
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t1d3 td0 no Active
c10t1d3 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t1d3 td0 no Standby
c12t1d3 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 3 c15t1d3 6005-08B4-0010-1552-0000-6000-036E-0000 4
255/255/0/1.3
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t1d4 td0 no Active
c10t1d4 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t1d4 td0 no Standby
c12t1d4 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 4 c15t1d4 6005-08B4-0010-1552-0000-6000-0371-0000 4
255/255/0/1.4
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t1d5 td0 no Active
c10t1d5 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t1d5 td0 no Standby
c12t1d5 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 5 c15t1d5 6005-08B4-0010-1552-0000-6000-0374-0000 4
255/255/0/1.5
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t1d6 td0 no Active
c10t1d6 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t1d6 td0 no Standby
c12t1d6 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 6 c15t1d6 6005-08B4-0010-1552-0000-6000-0384-0000 4
255/255/0/1.6
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R YES
c6t1d7 td0 no Active
c10t1d7 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X no
c8t1d7 td0 no Standby
c12t1d7 td1 no Standby


TGT/LUN Device WWLUN_ID H/W_Path #_Paths
1/ 7 c15t1d7 6005-08B4-0010-1552-0000-6000-0387-0000 4
255/255/0/1.7
Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI05R no
c6t2d0 td0 no Active
c10t2d0 td1 no Available

Controller Path_Instance HBA Preferred? Path_Status
P66C5E2AAQI04X YES
c8t2d0 td0 no Standby
c12t2d0 td1 no Standby

# strings /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vg01/lv_arc /archlogs vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg02/lv_bu /backup vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg03/lv_ifas /ifas vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg04/lv_sw /sw vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg05/lv_u01 /u01 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg06/lv_u02 /u02 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg07/lv_u03 /u03 vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg08/lv_ws /workspace vxfs rw,suid,largefiles,delaylog,datainlog 0 2
/dev/vg00/lvol9 ... swap pri=2 0 0

[Jim Walls]

We need to see the output from:

strings /etc/lvmtab

bdf

[John Mark Robeck]

Here are the results - we also found that when I ran ioscan -fnCdisk the devices where clamed under new device names. These names are different than they previously had been.

# strings /etc/lvmtab
/dev/vg00
`@JDJf<
/dev/dsk/c2t1d0
/dev/dsk/c3t0d0
/dev/vg01
`@JDJhu%
/dev/dsk/c14t0d0
/dev/dsk/c14t0d1
/dev/vg02
`@JDJhu4
/dev/dsk/c14t0d2
/dev/dsk/c14t0d3
/dev/vg03
`@JDJhuU
/dev/dsk/c14t0d4
/dev/dsk/c14t0d5
/dev/vg04
`@JDJhuf
/dev/dsk/c14t0d6
/dev/dsk/c14t0d7
/dev/vg05
`@JDJhux
/dev/dsk/c14t1d0
/dev/dsk/c14t1d1
/dev/vg06
`@JDJhu
/dev/dsk/c14t1d2
/dev/dsk/c14t1d3
/dev/vg07
`@JDJhu
/dev/dsk/c14t1d4
/dev/dsk/c14t1d5
/dev/vg08
`@JDJhu
/dev/dsk/c14t1d6
/dev/dsk/c14t1d7

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 204800 120608 83560 59% /
/dev/vg00/lvol1 298928 44160 224872 16% /stand
/dev/vg00/lvol8 4710400 1671368 3015384 36% /var
/dev/vg00/lvol7 2105344 1483768 616784 71% /usr
/dev/vg00/lvol4 1048576 5680 1034808 1% /tmp
/dev/vg00/lvol6 2818048 2101496 711024 75% /opt
/dev/vg00/lvol5 10485760 968232 9443200 9% /home

[John Mark Robeck]

Jim, we cuoldn't run the script out right but we did step by step go through it manually and were able to succesfully restore one of the file systems. There are 8 in all so we will work through each of them.

[John Mark Robeck]

OK, things are back up. we made one modification to your script Jim. Thanks so much for your help on this. Below is the script we ended up using.

vgn=$1
vg=vg0$vgn
minor=0x0"$vgn"0000
pvs=$2

echo "Fixing $vg"
echo "Minor =$minor"
echo "PVS =$pvs"

vgchange -a n $vg
vgexport -m /etc/lvmconf/$vg.mapfile $vg
rm -rf /dev/$vg
mkdir /dev/$vg
mknod /dev/$vg/group c 64 $minor
vgimport -m /etc/lvmconf/$vg.mapfile $vg $pvs
vgchange -a y $vg
vgdisplay -v $vg
vgcfgbackup $vg