vgchange hangs on 11iv3

[Eric SAUBIGNAC]

Hi guys,

need some help ...

HP-UX 11iv3 September 2008, latest patches, on a BL860C, a second one on a BL870C. Both have a double san attachment on 2 independant brocade fabric. There are 2 EVA 4100.

vg00 on internal disks. fine
Others vg, are on the san. Vg have been created with version 2.0 or version 2.1. All of them are distributed and mirrored over the 2 EVA.

We had an electrical outage with the blades and servers have rebooted. No outage with the san.

From this point, I can't activate any vg on the san. Command simply doesn't return ;-(

I have tried -s option to prevent resynchronization, but vgchange still hangs

I can see all my luns with ioscan -m lun. ioscan -P health shows no trouble. Diskinfo or dd on /dev/rdisk/diskXXX are OK.

Any idea ?

Thanks in advance

[Torsten.]

A very first idea - check if the LUNs are write protected for any (unknown) reason.

This may cause very strange results ...

[Torsten.]

Also check for related messages from the diags in syslog and roots mailbox.

[Steven E. Protter]

Shalom,

Can you post command and return codes?

It is also possible that due to the shutdown the file systems on the disks (I know next step) are corrupt and require fixfs prior to mounting.

SEP

[Duncan Edmonstone]

Also check:

1. Output of "lvmadm -l" - does all look as you would expect?

2. Can you run pvdisplay on all the SAN disks?

HTH

Duncan

[Eric SAUBIGNAC]

Thanks torsten,

I have checked from command view eva side : all luns have "write protect" set to "NO", are Read/write access and write cache set to write back

:-(

I continue to check HP-UX's logs files and will tell you.

Eric

[Eric SAUBIGNAC]

Torsten, no mail at all. see syslog.log in attached file.

I see no trouble but :

Oct 2 10:38:31 una00009 vmunix: Target path (class=tgtpath, instance=9) has gone offline. The target path h/w path is 0/4/0/
0/0/1.0x100000062b0afdf5
Oct 2 10:38:31 una00009 vmunix: Target path (class=tgtpath, instance=9) has gone online. The target path h/w path is 0/4/0/0
/0/1.0x100000062b0afdf5


the Path 0/4/0/0/0/1.0x100000062b0afdf5 is a san path but I don't know yet what it is. What i am sure is that it is not an eva.

And second :


Oct 2 10:38:31 una00009 vmunix: WARNING: moduload : module is busy, module id = 5, name = cfsmdr
Oct 2 10:38:31 una00009 vmunix: class : tgtpath, instance 9

Don't know what it means

[Eric SAUBIGNAC]

0/4/0/0/0/1.0x100000062b0afdf5 is simply a backup library with REO drives

[Eric SAUBIGNAC]

Thanks SEP

The command vgchange simply doesn't return. No error code ;-(

And the problem is at LVM level ... for the moment. May be FS are corrupted but I don't think it is the origin of the current problem

[Eric SAUBIGNAC]

Thanks Duncan,

output lvmadm as expected :

root@una00009:/var/adm/syslog>lvmadm -l
--- Version 1.0 volume groups ---
VG Name /dev/vg00
PV Name /dev/disk/disk1_p2

--- Version 2.0 volume groups ---
VG Name /dev/vghpdepot
PV Name /dev/disk/disk56
/dev/disk/disk111

VG Name /dev/vgsapput
PV Name /dev/disk/disk46
/dev/disk/disk55
/dev/disk/disk110
/dev/disk/disk106

VG Name /dev/vgsaptmp
PV Name /dev/disk/disk6
/dev/disk/disk11
/dev/disk/disk16
/dev/disk/disk26
/dev/disk/disk101
/dev/disk/disk107
/dev/disk/disk102
/dev/disk/disk108

VG Name /dev/vgtool
PV Name /dev/disk/disk25
/dev/disk/disk31
/dev/disk/disk36
/dev/disk/disk41
/dev/disk/disk103
/dev/disk/disk109
/dev/disk/disk104
/dev/disk/disk105

VG Name /dev/vgignite
PV Name /dev/disk/disk116
/dev/disk/disk119
/dev/disk/disk124
/dev/disk/disk129

pvdisplay has 2 reactions :

- on disk46 it hangs. I guess it is normal since "vgchange -a y -s vgsapput" is currently hanging

- it complains on disks from other vg. But since those vg are not activated I guess it is normal. Example :

root@una00009:/var/adm/syslog>pvdisplay /dev/disk/disk129
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk116":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk119":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk124":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk129":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vgignite".
pvdisplay: Cannot display physical volume "/dev/disk/disk129".

Eric

[Torsten.]

I'm not yet this familiar with 11.31 - I still struggle with all the new device files and pathes, but anyway, let's try some things:

$ ioscan -fkm lun

$ ioscan -P health -C disk

$ scsimgr -v lun_map

[Eric SAUBIGNAC]

Found on the second system, that has the same problem, the same message in syslog :

Oct 2 10:40:31 una00011 vmunix: WARNING: moduload : module is busy, module id = 6, name = cfsmdr

In OLDsyslog there is no message of this kind.

[Torsten.]

This is the CIFS File System Module (CFSM) - see

http://docs.hp.com/en/B8725-90118/ch11.html

I don't think this is really related.

[Duncan Edmonstone]

Eric,

cfsmdr is the CIFS File System Module Pseudo Driver - can't see that being connected in any way - do you share any of the filesystems out using CIFS/samba??

# kcmodule -v cfsmdr
Module cfsmdr (1.0)
Description CIFS File System Module Pseudo Driver
Timestamp Tue Feb 5 23:54:53 2008 [47A8F74D]
State loaded (via autoload)
State at Next Boot auto (as requested)
Capable auto static loaded unused
Depends On interface HPUX_11_31_PERF:1.0


HTH

Duncan

[Torsten.]

Do you have evainfo installed?

What does it tell?

[Eric SAUBIGNAC]

Torsten,

find in attached file output of command you asked.

I see no problem with those commands, but as for you, I have no big experience with 11iv3.

[Torsten.]

OK, now we now most of the LUNs have status

State = UNOPEN
Last Open or Close state = ACTIVE

but not

State = ACTIVE
Last Open or Close state = ACTIVE

this must have to do with fabric logins, IMHO.

Maybe a timing issue during start up the environment after the power outage?

Are all the eva ports online (from switch point of view)?


Any other (working) connected servers to these evas?

[Eric SAUBIGNAC]

Torsten, Duncan,

I have found that a SAN administrator has given san access to the REO library.

So I have modified zoning to prevent my HP-UX boxes to access this library and have rebooted the boxes.

There is no more message about "cfsmdr" but vgchange still hangs :-(

[Torsten.]

... and how about the good old

# ioscan -fn

all pathes visible, CLAIMED?

[Eric SAUBIGNAC]

Yes, everything seems nice in the fabric : from switch point of view, from command view, even from a PA-RISC 11.11 sap cluster. Autopath shows no defect.

I have not installed evainfo. I will.

Status Unopen, OK, but is it really the sign of a problem since san vg are not activated ?

[Eric SAUBIGNAC]

unfortunatly, the good old ioscan -fn shows everything is CLAIMED :-(

[Torsten.]

The status of

LUN PATH INFORMATION FOR LUN : /dev/rdisk/disk46

is ACTIVE or STANDBY - this looks good.

same for

/dev/rdisk/disk55
/dev/rdisk/disk106
/dev/rdisk/disk110

But that's it ...

Can you pvdisplay them?

[Torsten.]

Let's try this for a working and non-working LUN:

# scsimgr get_info -D /dev/rdisk/disk55

# scsimgr get_info -D /dev/rdisk/disk56

# scsimgr get_info all_lpt -D /dev/rdisk/disk55

# scsimgr get_info all_lpt -D /dev/rdisk/disk56

[Eric SAUBIGNAC]

Sorry for late answer : call back from HP support, nothing for the moment.

Attached output of evainfo -a. Nothing special for me.

As previously said, pvdisplay hangs on disks corresponding to the vg I try to activate and complains on others since vg are not activated.

I take a litle break ...

[Eric SAUBIGNAC]

Here are scsimgr outputs