topic Re: vgchange hangs on 11iv3 in Operating System - HP-UX

vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:05:27 GMT

Hi guys,

need some help ...

HP-UX 11iv3 September 2008, latest patches, on a BL860C, a second one on a BL870C. Both have a double san attachment on 2 independant brocade fabric. There are 2 EVA 4100.

vg00 on internal disks. fine
Others vg, are on the san. Vg have been created with version 2.0 or version 2.1. All of them are distributed and mirrored over the 2 EVA.

We had an electrical outage with the blades and servers have rebooted. No outage with the san.

From this point, I can't activate any vg on the san. Command simply doesn't return ;-(

I have tried -s option to prevent resynchronization, but vgchange still hangs

I can see all my luns with ioscan -m lun. ioscan -P health shows no trouble. Diskinfo or dd on /dev/rdisk/diskXXX are OK.

Any idea ?

Thanks in advance

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 08:08:42 GMT

A very first idea - check if the LUNs are write protected for any (unknown) reason.

This may cause very strange results ...

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 08:13:55 GMT

Also check for related messages from the diags in syslog and roots mailbox.

Re: vgchange hangs on 11iv3

Steven E. Protter — Thu, 02 Oct 2008 08:16:29 GMT

Shalom,

Can you post command and return codes?

It is also possible that due to the shutdown the file systems on the disks (I know next step) are corrupt and require fixfs prior to mounting.

SEP

Re: vgchange hangs on 11iv3

Duncan Edmonstone — Thu, 02 Oct 2008 08:17:02 GMT

Also check:

1. Output of "lvmadm -l" - does all look as you would expect?

2. Can you run pvdisplay on all the SAN disks?

HTH

Duncan

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:24:21 GMT

Thanks torsten,

I have checked from command view eva side : all luns have "write protect" set to "NO", are Read/write access and write cache set to write back

:-(

I continue to check HP-UX's logs files and will tell you.

Eric

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:32:32 GMT

Torsten, no mail at all. see syslog.log in attached file.

I see no trouble but :

Oct 2 10:38:31 una00009 vmunix: Target path (class=tgtpath, instance=9) has gone offline. The target path h/w path is 0/4/0/
0/0/1.0x100000062b0afdf5
Oct 2 10:38:31 una00009 vmunix: Target path (class=tgtpath, instance=9) has gone online. The target path h/w path is 0/4/0/0
/0/1.0x100000062b0afdf5

the Path 0/4/0/0/0/1.0x100000062b0afdf5 is a san path but I don't know yet what it is. What i am sure is that it is not an eva.

And second :

Oct 2 10:38:31 una00009 vmunix: WARNING: moduload : module is busy, module id = 5, name = cfsmdr
Oct 2 10:38:31 una00009 vmunix: class : tgtpath, instance 9

Don't know what it means

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:36:01 GMT

0/4/0/0/0/1.0x100000062b0afdf5 is simply a backup library with REO drives

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:38:29 GMT

Thanks SEP

The command vgchange simply doesn't return. No error code ;-(

And the problem is at LVM level ... for the moment. May be FS are corrupted but I don't think it is the origin of the current problem

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:44:38 GMT

Thanks Duncan,

output lvmadm as expected :

root@una00009:/var/adm/syslog>lvmadm -l
--- Version 1.0 volume groups ---
VG Name /dev/vg00
PV Name /dev/disk/disk1_p2

--- Version 2.0 volume groups ---
VG Name /dev/vghpdepot
PV Name /dev/disk/disk56
/dev/disk/disk111

VG Name /dev/vgsapput
PV Name /dev/disk/disk46
/dev/disk/disk55
/dev/disk/disk110
/dev/disk/disk106

VG Name /dev/vgsaptmp
PV Name /dev/disk/disk6
/dev/disk/disk11
/dev/disk/disk16
/dev/disk/disk26
/dev/disk/disk101
/dev/disk/disk107
/dev/disk/disk102
/dev/disk/disk108

VG Name /dev/vgtool
PV Name /dev/disk/disk25
/dev/disk/disk31
/dev/disk/disk36
/dev/disk/disk41
/dev/disk/disk103
/dev/disk/disk109
/dev/disk/disk104
/dev/disk/disk105

VG Name /dev/vgignite
PV Name /dev/disk/disk116
/dev/disk/disk119
/dev/disk/disk124
/dev/disk/disk129

pvdisplay has 2 reactions :

- on disk46 it hangs. I guess it is normal since "vgchange -a y -s vgsapput" is currently hanging

- it complains on disks from other vg. But since those vg are not activated I guess it is normal. Example :

root@una00009:/var/adm/syslog>pvdisplay /dev/disk/disk129
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk116":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk119":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk124":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/disk/disk129":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vgignite".
pvdisplay: Cannot display physical volume "/dev/disk/disk129".

Eric

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 08:54:13 GMT

I'm not yet this familiar with 11.31 - I still struggle with all the new device files and pathes, but anyway, let's try some things:

$ ioscan -fkm lun

$ ioscan -P health -C disk

$ scsimgr -v lun_map

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 08:59:45 GMT

Found on the second system, that has the same problem, the same message in syslog :

Oct 2 10:40:31 una00011 vmunix: WARNING: moduload : module is busy, module id = 6, name = cfsmdr

In OLDsyslog there is no message of this kind.

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 09:04:13 GMT

This is the CIFS File System Module (CFSM) - see

http://docs.hp.com/en/B8725-90118/ch11.html

I don't think this is really related.

Re: vgchange hangs on 11iv3

Duncan Edmonstone — Thu, 02 Oct 2008 09:08:09 GMT

Eric,

cfsmdr is the CIFS File System Module Pseudo Driver - can't see that being connected in any way - do you share any of the filesystems out using CIFS/samba??

# kcmodule -v cfsmdr
Module cfsmdr (1.0)
Description CIFS File System Module Pseudo Driver
Timestamp Tue Feb 5 23:54:53 2008 [47A8F74D]
State loaded (via autoload)
State at Next Boot auto (as requested)
Capable auto static loaded unused
Depends On interface HPUX_11_31_PERF:1.0

HTH

Duncan

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 09:10:23 GMT

Do you have evainfo installed?

What does it tell?

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 09:15:45 GMT

Torsten,

find in attached file output of command you asked.

I see no problem with those commands, but as for you, I have no big experience with 11iv3.

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 09:21:05 GMT

OK, now we now most of the LUNs have status

State = UNOPEN
Last Open or Close state = ACTIVE

but not

State = ACTIVE
Last Open or Close state = ACTIVE

this must have to do with fabric logins, IMHO.

Maybe a timing issue during start up the environment after the power outage?

Are all the eva ports online (from switch point of view)?

Any other (working) connected servers to these evas?

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 09:21:17 GMT

Torsten, Duncan,

I have found that a SAN administrator has given san access to the REO library.

So I have modified zoning to prevent my HP-UX boxes to access this library and have rebooted the boxes.

There is no more message about "cfsmdr" but vgchange still hangs :-(

Re: vgchange hangs on 11iv3

Torsten. — Thu, 02 Oct 2008 09:23:31 GMT

... and how about the good old

# ioscan -fn

all pathes visible, CLAIMED?

Re: vgchange hangs on 11iv3

Eric SAUBIGNAC — Thu, 02 Oct 2008 09:26:10 GMT

Yes, everything seems nice in the fabric : from switch point of view, from command view, even from a PA-RISC 11.11 sap cluster. Autopath shows no defect.

I have not installed evainfo. I will.

Status Unopen, OK, but is it really the sign of a problem since san vg are not activated ?