Symmetrix storage not available after restart

[Difladermaus_RP]

I am missing c7 and c9 after reboot. Storage team verified disks are still prsented to this host.

I have not been working with HPUX for several years.

May 21 08:26
Found adjacent data tr.  Growing size.  0x3072000 -> 0x7072000.
Pinned PDK malloc pool: base: 0xe000000100f8e000  size=115144K
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.

MFS is defined: base= 0xe000000100f8e000  size= 5672 KB

Unpinned PDK malloc pool: base: 0xe000000108000000  size=180224K
NOTICE: cachefs_link(): File system was registered at index 5.
255/0 ms_fake_drv
NOTICE: nfs3_link(): File system was registered at index 8.
255/2 rdsk_root
NOTICE: mod_fs_reg: Cannot retrieve configured loading phase from KRS for module: cifs. Setting to load at INIT

0 sba
0/0 lba
0/0/1/0 rmp3f01
0/0/1/1 rmp3f01
0/0/1/2 asio0
0/0/2/0 UsbOhci
NOTICE: USB device attached.  Identification String:
Devices/Device/USB/Standard/hp/Unknown/0_1
 <2.1.3.10.1008.4390.1>
0/0/2/0.0 UsbMiniBus
NOTICE: USB device attached.  Identification String:
Devices/Keyboard/USB/Boot/hp/Unknown/0_1
 <2.305.3.100.1008.4390.1>
0/0/2/0.0.0 UsbBootKeyboard
NOTICE: USB device attached.  Identification String:
Devices/Mouse/USB/Standard/hp/Unknown/0_1
 <2.307.3.10.1008.4390.1>
0/0/2/1 UsbOhci
0/0/2/2 UsbEhci
0/0/4/0 gvid_core
0/1 lba
0/1/1/0 igelan
0/1/1/1 igelan
0/2 lba
sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/2/1/0
0/2/1/0 sasd
0/2/2/0 igelan
0/2/2/1 igelan
0/3 gh2p
0/3/0/0 PCItoPCI
fcd: Claimed HP 404986-001 4Gb Fibre Channel port at hardware path 0/3/0/0/0/0 (FC Port 1 on HBA)
0/3/0/0/0/0 fcd
fcd: Claimed HP 404986-001 4Gb Fibre Channel port at hardware path 0/3/0/0/0/1 (FC Port 2 on HBA)
0/3/0/0/0/1 fcd
0/4 gh2p
0/4/0/0 PCItoPCI
fcd: Claimed HP 404986-001 4Gb Fibre Channel port at hardware path 0/4/0/0/0/0 (FC Port 1 on HBA)
0/4/0/0/0/0 fcd
fcd: Claimed HP 404986-001 4Gb Fibre Channel port at hardware path 0/4/0/0/0/1 (FC Port 2 on HBA)
0/4/0/0/0/1 fcd
0/5 gh2p
0/5/0/0 PCItoPCI
120 processor
122 processor
250 pdh
250/0 ipmi
250/1 asio0
250/2 acpi_node
0/2/1/0.0.0 sasd_vbus
0/2/1/0.0.0.0 tgt
0/2/1/0.0.0.0.0 sdisk
0/2/1/0.0.0.1 tgt
0/2/1/0.0.0.1.0 sdisk
255/1 mass_storage
Boot device's HP-UX HW path is: 0/2/1/0.0.0.1.0

    System Console is on the Built-In Serial Interface
igelan2: INITIALIZING HP PCI-X 1000Mbps Dual-port Built-in at hardware path 0/2/2/0
igelan0: INITIALIZING HP PCI-X 1000Mbps Dual-port Built-in at hardware path 0/1/1/0
igelan1: INITIALIZING HP PCI-X 1000Mbps Dual-port Built-in at hardware path 0/1/1/1
igelan3: INITIALIZING HP PCI-X 1000Mbps Dual-port Built-in at hardware path 0/2/2/1
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
    Swap device table:  (start & size given in 512-byte blocks)
        entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
    Dump device table:  (start & size given in 1-Kbyte blocks)
        entry 0000000000000000 - major is 31, minor is 0x11000; start = 2612084, size = 8388604
Starting the STREAMS daemons-phase 1
Create STCP device files
Starting the STREAMS daemons-phase 2
     $Revision: vmunix:    B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
Memory Information:
    physical page size = 4096 bytes, logical page size = 4096 bytes
    Physical: 8357600 Kbytes, lockable: 6082280 Kbytes, available: 7071868 Kbytes

Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576  336144  706880   32% /
/dev/vg00/lvol1    2097152  229992 1852664   11% /stand
/dev/vg00/lvol8    8880128 6883808 1982336   78% /var
/dev/vg00/lvol7    10485760 2928656 7498080   28% /usr
/dev/vg00/lvol6    10485760 6977528 3482304   67% /tmp
/dev/vg00/lvol5    15728640 6349200 9306224   41% /opt
/dev/vg00/lvol4    5242880 1339176 3873232   26% /home

# System /etc/fstab file.  Static information about the file systems
# See fstab(4) and sam(1M) for further details on configuring devices.
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol4 /home vxfs delaylog 0 2
/dev/vg00/lvol5 /opt vxfs delaylog 0 2
/dev/vg00/lvol6 /tmp vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vgonbp/onbasep /onbasep vxfs delaylog 0 2
/dev/vgonbp/onbaseporacle /onbasep/opt/app/oracle/product/10.2.0 vxfs delaylog 0 2
/dev/vgonbp/onbaseporaadmin  /onbasep/opt/app/oracle/admin/ vxfs delaylog 0 2
## IDDR
/dev/vgiddr/iddr01 /iddr vxfs delaylog 0 2
/dev/vgiddr/oracle  /iddr/oracle vxfs delaylog 0 2
/dev/vgtmp/dbatmp  /oraflash vxfs delaylog 0 2

/dev/vg00
/dev/dsk/c1t1d0s2
/dev/dsk/c1t0d0s2
/dev/vgtmp
/dev/dsk/c7t0d1
/dev/dsk/c9t0d1
/dev/vgiddr
/dev/dsk/c7t0d2
/dev/dsk/c9t0d2
/dev/vgonbp
/dev/dsk/c7t8d0
/dev/dsk/c7t8d1
/dev/dsk/c7t8d2
/dev/dsk/c9t0d3
/dev/dsk/c9t0d4
/dev/dsk/c9t0d5

nodens:/etc # symcfg list

S Y M M E T R I X

Mcode Cache Num Phys Num Symm

SymmID Attachment Model Version Size (MB) Devices Devices

000190103742 Local DMX4-24 5772 98304 12 4626

nodens:/etc # syminq

Device Product Device

----------------------- --------------------------- ---------------------

Name Type Vendor ID Rev Ser Num Cap (KB)

----------------------- --------------------------- ---------------------

/dev/rdsk/c1t0d0 HP DH072BB978 HPD9 3PD1PV81 71687369

/dev/rdsk/c1t1d0 HP DH072BB978 HPD9 3PD1PBE1 71687369

[Difladermaus_RP]

/dev

nodens:/dev # ll dsk

total 0

brw-r----- 1 bin sys 31 0x000000 Jul 30 2008 c0t0d0

brw-r----- 1 bin sys 31 0x010000 Jul 30 2008 c1t0d0

brw-r----- 1 bin sys 31 0x010001 Aug 19 2008 c1t0d0s1

brw-r----- 1 bin sys 31 0x010002 Aug 19 2008 c1t0d0s2

brw-r----- 1 bin sys 31 0x010003 Aug 19 2008 c1t0d0s3

brw-r----- 1 bin sys 31 0x011000 Jul 30 2008 c1t1d0

brw-r----- 1 bin sys 31 0x011001 Jul 30 2008 c1t1d0s1

brw-r----- 1 bin sys 31 0x011002 Jul 30 2008 c1t1d0s2

brw-r----- 1 bin sys 31 0x011003 Jul 30 2008 c1t1d0s3

brw-r----- 1 bin sys 31 0x040000 Nov 9 2010 c4t0d0

brw-r----- 1 bin sys 31 0x040100 Jan 16 11:58 c4t0d1

brw-r----- 1 bin sys 31 0x042500 Nov 14 2011 c4t2d5

brw-r----- 1 bin sys 31 0x054000 May 22 2011 c5t4d0

brw-r----- 1 bin sys 31 0x054100 May 22 2011 c5t4d1

brw-r----- 1 bin sys 31 0x054200 Jun 9 2011 c5t4d2

brw-r----- 1 bin sys 31 0x070000 May 30 2011 c7t0d0

brw-r----- 1 bin sys 31 0x070100 Jan 16 11:58 c7t0d1

brw-r----- 1 bin sys 31 0x070200 Sep 25 2012 c7t0d2

brw-r----- 1 bin sys 31 0x078000 May 30 2011 c7t8d0

brw-r----- 1 bin sys 31 0x078100 May 30 2011 c7t8d1

brw-r----- 1 bin sys 31 0x078200 Jun 9 2011 c7t8d2

brw-r----- 1 bin sys 31 0x090000 Feb 23 07:38 c9t0d0

brw-r----- 1 bin sys 31 0x090100 Feb 23 07:49 c9t0d1

brw-r----- 1 bin sys 31 0x090200 Feb 23 07:49 c9t0d2

brw-r----- 1 bin sys 31 0x090300 Feb 23 07:49 c9t0d3

brw-r----- 1 bin sys 31 0x090400 Feb 23 07:49 c9t0d4

brw-r----- 1 bin sys 31 0x090500 Feb 23 07:49 c9t0d5

nodens:/dev # ll rdsk

total 0

crw-r--r-- 1 bin sys 188 0x000000 Jul 30 2008 c0t0d0

crw-r----- 1 bin sys 188 0x010000 Aug 20 2008 c1t0d0

crw-r----- 1 bin sys 188 0x010001 Aug 20 2008 c1t0d0s1

crw-r----- 1 bin sys 188 0x010002 Jan 17 2012 c1t0d0s2

crw-r----- 1 bin sys 188 0x010003 Aug 19 2008 c1t0d0s3

crw-r----- 1 bin sys 188 0x011000 Oct 5 2010 c1t1d0

crw-r----- 1 bin sys 188 0x011001 May 29 2011 c1t1d0s1

crw-r----- 1 bin sys 188 0x011002 Jan 17 2012 c1t1d0s2

crw-r----- 1 bin sys 188 0x011003 Jul 30 2008 c1t1d0s3

crw-r--r-- 1 bin sys 188 0x040000 Nov 9 2010 c4t0d0

crw-r----- 1 bin sys 188 0x040100 Jan 16 12:02 c4t0d1

crw-r--r-- 1 bin sys 188 0x042500 Dec 5 2011 c4t2d5

crw-r--r-- 1 bin sys 188 0x054000 May 22 2011 c5t4d0

crw-r--r-- 1 bin sys 188 0x054100 May 22 2011 c5t4d1

crw-r--r-- 1 bin sys 188 0x054200 Jun 9 2011 c5t4d2

crw-r--r-- 1 bin sys 188 0x070000 May 30 2011 c7t0d0

crw-r----- 1 bin sys 188 0x070100 Jan 16 11:58 c7t0d1

crw-r----- 1 bin sys 188 0x070200 Sep 25 2012 c7t0d2

crw-r--r-- 1 bin sys 188 0x078000 May 30 2011 c7t8d0

crw-r--r-- 1 bin sys 188 0x078100 May 30 2011 c7t8d1

crw-r--r-- 1 bin sys 188 0x078200 Jun 9 2011 c7t8d2

crw-r----- 1 bin sys 188 0x090000 Feb 23 07:38 c9t0d0

crw-r----- 1 bin sys 188 0x090100 Feb 23 07:49 c9t0d1

crw-r----- 1 bin sys 188 0x090200 Feb 23 07:49 c9t0d2

crw-r----- 1 bin sys 188 0x090300 Feb 23 07:49 c9t0d3

crw-r----- 1 bin sys 188 0x090400 Feb 23 07:49 c9t0d4

crw-r----- 1 bin sys 188 0x090500 Feb 23 07:49 c9t0d5

[Difladermaus_RP]

uname -a
HP-UX nodens B.11.23 U ia64 2406227353 unlimited-user license

[Patrick Wallek]

Can you post, or attach, the output from 'ioscan -kfn' please?  

If you run the 'ioscan' does it show the fibre adapter(s) required to see the c7 & c9 disks?

[Difladermaus_RP]

Yes they are listed.

nodens:/var/adm/syslog # ioscan -kfn

Class I H/W Path Driver S/W State H/W Type Description

====================================================================================

root 0 root CLAIMED BUS_NEXUS

ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (4030)

ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)

tty 0 0/0/1/0 rmp3f01 CLAIMED INTERFACE PCI class(255,0) (103c1303)

tty 1 0/0/1/1 rmp3f01 CLAIMED INTERFACE PCI SimpleComm (103c1302)

tty 2 0/0/1/2 asio0 CLAIMED INTERFACE PCI Serial (103c1048)

/dev/diag/mux2 /dev/mux2 /dev/tty2p0

OO 0 0/0/2/0 UsbOhci CLAIMED INTERFACE USB OHCI Interface

OO 1 0/0/2/0.0 UsbMiniBus CLAIMED INTERFACE USB Composite Device

OO 2 0/0/2/0.0.0 UsbBootKeyboard CLAIMED DEVICE Virtual Keyboard

unknown -1 0/0/2/0.0.1 UNCLAIMED UNKNOWN OO Device Driver

OO 3 0/0/2/1 UsbOhci CLAIMED INTERFACE USB OHCI Interface

OO 6 0/0/2/2 UsbEhci CLAIMED INTERFACE USB EHCI Interface

graphics 0 0/0/4/0 gvid_core CLAIMED INTERFACE PCI Display (1002515e)

ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)

lan 0 0/1/1/0 igelan CLAIMED INTERFACE HP PCI-X 1000Mbps Dual-port Built-in

lan 1 0/1/1/1 igelan CLAIMED INTERFACE HP PCI-X 1000Mbps Dual-port Built-in

ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)

escsi_ctlr 0 0/2/1/0 sasd CLAIMED INTERFACE HP PCI/PCI-X SAS MPT Adapter

/dev/sasd0

ext_bus 1 0/2/1/0.0.0 sasd_vbus CLAIMED INTERFACE SAS Device Interface

target 0 0/2/1/0.0.0.0 tgt CLAIMED DEVICE

disk 1 0/2/1/0.0.0.0.0 sdisk CLAIMED DEVICE HP DH072BB978

/dev/dsk/c1t0d0 /dev/dsk/c1t0d0s2 /dev/rdsk/c1t0d0 /dev/rdsk/c1t0d0s2

/dev/dsk/c1t0d0s1 /dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s1 /dev/rdsk/c1t0d0s3

target 1 0/2/1/0.0.0.1 tgt CLAIMED DEVICE

disk 2 0/2/1/0.0.0.1.0 sdisk CLAIMED DEVICE HP DH072BB978

/dev/dsk/c1t1d0 /dev/dsk/c1t1d0s2 /dev/rdsk/c1t1d0 /dev/rdsk/c1t1d0s2

/dev/dsk/c1t1d0s1 /dev/dsk/c1t1d0s3 /dev/rdsk/c1t1d0s1 /dev/rdsk/c1t1d0s3

lan 2 0/2/2/0 igelan CLAIMED INTERFACE HP PCI-X 1000Mbps Dual-port Built-in

lan 3 0/2/2/1 igelan CLAIMED INTERFACE HP PCI-X 1000Mbps Dual-port Built-in

ba 3 0/3 gh2p CLAIMED BUS_NEXUS Local Bus Adapter

ba 4 0/3/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge

fc 0 0/3/0/0/0/0 fcd CLAIMED INTERFACE HP 4Gb Dual Port PCIe Fibre Channel Mezzanine (FC Port 1)

/dev/fcd0

fc 1 0/3/0/0/0/1 fcd CLAIMED INTERFACE HP 4Gb Dual Port PCIe Fibre Channel Mezzanine (FC Port 2)

/dev/fcd1

ba 5 0/4 gh2p CLAIMED BUS_NEXUS Local Bus Adapter

ba 6 0/4/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge

fc 2 0/4/0/0/0/0 fcd CLAIMED INTERFACE HP 4Gb Dual Port PCIe Fibre Channel Mezzanine (FC Port 1)

/dev/fcd2

fc 3 0/4/0/0/0/1 fcd CLAIMED INTERFACE HP 4Gb Dual Port PCIe Fibre Channel Mezzanine (FC Port 2)

/dev/fcd3

ba 7 0/5 gh2p CLAIMED BUS_NEXUS Local Bus Adapter

ba 8 0/5/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge

processor 0 120 processor CLAIMED PROCESSOR Processor

processor 1 122 processor CLAIMED PROCESSOR Processor

ba 9 250 pdh CLAIMED BUS_NEXUS Core I/O Adapter

ipmi 0 250/0 ipmi CLAIMED INTERFACE IPMI Controller

/dev/ipmi

tty 3 250/1 asio0 CLAIMED INTERFACE Built-in RS232C

/dev/diag/mux3 /dev/mux3 /dev/tty3p0

acpi_node 0 250/2 acpi_node CLAIMED INTERFACE Acpi Hardware

usbmsvbus 0 255/1 mass_storage CLAIMED VIRTBUS USB Mass Storage

vroot_rdsk 0 255/2 rdsk_root CLAIMED VIRTBUS

[Difladermaus_RP]

seeing Driver state = AWAITING_LINK_UP from fcmsutil on all hbas

I tried fcmsutil /dev/fcdN -f reset, and I still get same status

Storage team is actively looking, they see storage setup as correct thus far.

nodens:/var/adm/syslog # fcmsutil /dev/fcd0

Vendor ID is = 0x001077

Device ID is = 0x002432

PCI Sub-system Vendor ID is = 0x00103c

PCI Sub-system ID is = 0x001705

PCI Mode = PCI Express x4

ISP Code version = 5.3.2

ISP Chip version = 2

Previous Topology = UNINITIALIZED

Link Speed = UNKNOWN

Local N_Port_id is = None

Previous N_Port_id is = None

N_Port Node World Wide Name = 0x50014380022409bd

N_Port Port World Wide Name = 0x50014380022409bc

Switch Port World Wide Name = 0x0000000000000000

Switch Node World Wide Name = 0x0000000000000000

Driver state = AWAITING_LINK_UP

Hardware Path is = 0/3/0/0/0/0

Maximum Frame Size = 2048

Driver-Firmware Dump Available = NO

Driver-Firmware Dump Timestamp = N/A

Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.1009 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:May 6 2010,19:46:29

[Patrick Wallek · Accepted solution]

AWAITING_LINK_UP sounds like there is a problem with your Fibre switch.  The cards are not negotiating correctly with the switch port to get logged into the fabric and set up the connection.

Your "Link Speed" is also UNKNOWN so there is not any communication to the cards.  I would check the fibre switch and cables.

[Difladermaus_RP]

I disabled one san fabric in virtual connect. After I rebooted, I now do have one path to the disk.

nodens:/ # bdf

Filesystem kbytes used avail %used Mounted on

/dev/vg00/lvol3 1048576 336312 706712 32% /

/dev/vg00/lvol1 2097152 229864 1852784 11% /stand

/dev/vg00/lvol8 8880128 6896048 1970192 78% /var

/dev/vg00/lvol7 10485760 2894440 7532032 28% /usr

/dev/vg00/lvol6 10485760 6977568 3482264 67% /tmp

/dev/vgtmp/dbatmp 107069440 48349791 55049735 47% /oraflash

/dev/vg00/lvol5 15728640 6349112 9306312 41% /opt

/dev/vgonbp/onbasep

108134400 95520076 11826882 89% /onbasep

/dev/vgonbp/onbaseporacle

6291456 6291455 1 100% /onbasep/opt/app/oracle/product/10.2.0

/dev/vgonbp/onbaseporaadmin

5242880 1382728 3618945 28% /onbasep/opt/app/oracle/admin

/dev/vgiddr/iddr01 15728640 3038499 11897137 20% /iddr

/dev/vgiddr/oracle 20971520 14259008 6292988 69% /iddr/oracle

/dev/vg00/lvol4 5242880 1339208 3873200 26% /home

nodens:/ # fcmsutil /dev/fcd0

Vendor ID is = 0x001077

Device ID is = 0x002432

PCI Sub-system Vendor ID is = 0x00103c

PCI Sub-system ID is = 0x001705

PCI Mode = PCI Express x4

ISP Code version = 5.3.2

ISP Chip version = 2

Topology = PTTOPT_FABRIC

Link Speed = 4Gb

Local N_Port_id is = 0xaf004b

Previous N_Port_id is = None

N_Port Node World Wide Name = 0x50014380022409bd

N_Port Port World Wide Name = 0x50014380022409bc

Switch Port World Wide Name = 0x2023000dec21c380

Switch Node World Wide Name = 0x202a000dec21c381

Driver state = ONLINE

Hardware Path is = 0/3/0/0/0/0

Maximum Frame Size = 2048

Driver-Firmware Dump Available = NO

Driver-Firmware Dump Timestamp = N/A

Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.1009 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:May 6 2010,19:46:29

[Difladermaus_RP]

Patrick, thanks for the help pointing me in the right direction. I will work the virtual connect pathing issue from here. For now, system is back online with 1 path to the storage.