Unable to activate my non-root vg

[Bala C]

I have applied the QPKBASE and QPKAPPS patches on to my hpux 11.23 IA system. Soon after the reboot, the server didn't activate the other non-root VG's. These non-root VG has more than a few luns connected to EMC storage.VG00 filesystems reside in internal disk.

output of ioscan -funC disk shows all the LUN's are claimed.

The following is the output,if I try to activate the vg manually.

#/sbin/vgchange -a y vg03
Password:
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c10t0d4":
Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Unknown error
vgchange: Warning: couldn't query physical volume "/dev/dsk/c10t0d4":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c10t0d5":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c10t0d6":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c10t0d7":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c10t1d0":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c12t0d4":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c12t0d5":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c12t0d6":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c12t0d7":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c12t1d0":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "vg03":
Quorum not present, or some physical volume(s) are missing.

[Ganesan R]

Ok. How many luns are part of VG03?

C10 and C12 are alternate links?

[Eric SAUBIGNAC]

Bonjour,

I guess you have to troubleshoot the SAN ...

What about "fcmsutil /dev/fcd0" or "fcmsutil /dev/td0" ? You can also check fcd1, fcd2, ... Same thing if you have /dev/td drivers

Eric

[Bala C]

hereis the lvmtab contents which says about the luns in vg03 :-

/dev/vg00
/dev/dsk/c5t0d0s2
/dev/vg02
/dev/dsk/c10t0d0
/dev/dsk/c10t0d1
/dev/dsk/c10t0d2
/dev/dsk/c10t0d3
/dev/dsk/c12t0d0
/dev/dsk/c12t0d1
/dev/dsk/c12t0d2
/dev/dsk/c12t0d3
/dev/vg03
/dev/dsk/c10t0d4
/dev/dsk/c10t0d5
/dev/dsk/c10t0d6
/dev/dsk/c10t0d7
/dev/dsk/c10t1d0
/dev/dsk/c12t0d4
/dev/dsk/c12t0d5
/dev/dsk/c12t0d6
/dev/dsk/c12t0d7
/dev/dsk/c12t1d0

[Bala C]

All the Luns are connected thru this HBA and it looks good to me.

#fcmsutil /dev/fcd2

Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012df
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.22
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 2Gb
Local N_Port_id is = 0x612d13
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500143800118eb09
N_Port Port World Wide Name = 0x500143800118eb08
Switch Port World Wide Name = 0x202d08008804d4f2
Switch Node World Wide Name = 0x100008008804d4f2
Driver state = ONLINE
Hardware Path is = 0/7/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.06 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Jun 23 2006,14:06:46

[Fabio Ettore]

Hi,

does ioscan see those LUNs as CLAIMED?

Best regards,
Fabio

[john123]

Pls try to read from the suspected disks using dd in the and check whether all the disks can be accessible from the server..



John

[Fabio Ettore]

Sorry, I didn't read you opened the thread with

"output of ioscan -funC disk shows all the LUN's are claimed."

Anyway it really sounds like a SAN issue. Try just dd on one of those disks:

dd if=/dev/rdsk/c10t0d4 of=/dev/null

After a while stop that by CTRL+C, it should report statistics with records read.

HTH.

Best regards,
Fabio

[Bala C]

Yes, ioscan shows all the luns as claimed

[Bala C]

dd doesn't read the luns.

dd if=/dev/rdsk/c12t0d6 of=/dev/null
dd read error: I/O error
0+0 records in
0+0 records out

What could have went wrong?

[john123]

Check with storage team whether they have masked any luns from the server in the storage side.?or for any fabric outage and also check for any HBA hardware changes done in server..

Mean while which model of storage u r using is it Symmetrix or clarion?


regrads
John

[Bala C]

Well, John. This has happened just after patching the server. So, I really dont think storage team would have masked the LUNs.

Is there any configuration need to be checked as the patching would have changed them.

It is EMC clarion.

[Ganesan R]

If dd fails then lvm commands also will not reveal any info.. Does pvdisplay or diskinfo provide any details for /dev/rdsk/c12t0d6 ?

would it be possible to export one vg and import it for testing?

However we should get clarified from storage team why dd is failed to read the disk before we try anything..because dd is the lower disk level read. If this fails lvm also will not be able to read the disks.

[Eric SAUBIGNAC]

You should do nothing at LVM level for the moment. dd is enough : we know there is physical problem.

Cout you post (in attachment please) a full ioscan -fn (that is without -k or -u). The objective is to see if hardware path to EMC storage has changed : maybe SAN team has modified some parameters that deal with HP-UX 11iv2 hardware path ?

Eric

[Bala C]

Thanks Guys. This is resolved now.

I did a powermt display and found some dead paths which are not related with my current configuration. SO, I did a powermt check and removed them.Then, did a powermt config and save.

Now I am able to activate my other vg's.

#powermt display

Symmetrix logical device count=0

CLARiiON logical device count=10

Hitachi logical device count=0

Invista logical device count=0

HP xp logical device count=0

Ess logical device count=0

HP HSx logical device count=0

==============================================================================

----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------

### HW Path Summary Total Dead IO/Sec Q-IOs Errors

==============================================================================

6 UNKNOWN failed 9 9 - 0 0

7 UNKNOWN failed 9 9 - 0 0

8 UNKNOWN failed 9 9 - 0 0

9 UNKNOWN failed 9 9 - 0 0

10 0/7/1/0/4/0.97.36.239.0 optimal 10 0 - 0 0

12 0/7/1/0/4/0.97.37.239.0 optimal 10 0 - 0 0



#powermt check

Warning: license for CLARiiON storage system support is missing or expired.

Warning: CLARiiON device path c6t0d0 is currently dead.

Do you want to remove it (y/n/a/q)? a

Warning: CLARiiON device path c6t0d1 is currently dead.
Warning: CLARiiON device path c6t0d2 is currently dead.
Warning: CLARiiON device path c6t0d3 is currently dead.