URGENT - Mirror disk stale - Unable to form cluster

[Dayanand Naik_2]

Hi All,

Kindly refer to this link which i had earlier posted, I could not reply back but Still 'm unable to bring the cluster back in service.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=605165

Further to the replies by other, sorry i could not reply back, as suggested i tried to reduce the volume group but it gives an error. Attached is the same.

It was a production server, which is moved to another location. So i had to bring this up, while doing the same, Cluster was not comming up.

root @ csora1 # lvreduce -k -m 0 /dev/vgora/lvu01 /dev/dsk/c5t0d0
Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vgora/lvu01".

Do i have to remove this logical volume and recreate the same. Doing this i will loose data, and i can restore the data back as i have all the backups in place. Kindly let know the steps in doing the same.

Any suggestions.

[Duncan Edmonstone · Accepted solution]

Did you follow Stephen's advice in your other thread and change the VGCHANGE parameter in your control file?

VGCHANGE="vgchange -a e"
to
VGCHANGE="vgchange -a e -q n"


This will allow the volume group/logical volume to be activated despite having stale extents on one side of the mirror.

This should at least get you cluster and application up, with a reduced level of availability.

HTH

Duncan

[Dayanand Naik_2]

Hi

I did change the control file as requested, i was able to form the cluster, but unable to start the package CSDB which is configured.

Attached is the log file. for all

Any Suggestion, Welcomed


Dayanand Naik.

[Bharat Katkar]

Daya,

before starting the package try doing this.

# vgchange -c n /dev/vgora

Then try starting the package.

Hope that works.
Regards,

[Johan Lorimier]

Hi,

please coul you also post the logs from the last lines of the file
/etc/cmcluster/CSDB/CSDB.cntl.log.

the solution might be there.

Johan

[Bharat Katkar]

Johan it is there in the attachment to his second reply.

[Johan Lorimier]

OUPS,
Sorry, I missed the last lines.

So, I have nothing more to say than Bharat.

If you really need a quick start, you can bypass the cluster, activate you VG, mount all your FS, assign IP address and perform the NFS export/mount and then start your application without using MCSG. It's worth to have some scripts that perform all these steps (we use them for DRP purpose).

Johan

[Jose Mosquera]

Hi,

I suggest you try to starup manually the volume "vgora" and the file systems involved.

#vgchange -a n vgora (to ensure a disabled vg)
#vgchange -a e vgora
#mount /dev/vgora/lvu01 /

With these steps you can debug the failure point. If these steps working fine you can work without "High Availability" feature.

Pls tell me something about.

Rgds.

[Dayanand Naik_2]

Hi All,

So Far the status,

- Cluster is forming and running
- VG is activated after vgchange -a y /dev/vgora and manually mounted the Filesystem.

after i do a vgdisplay query i get error on /dev/dsk/c5t0d0 disk.

Attached is the output


/home/hp/mount.sh

mount /dev/vgora/lvu01 /u01
mount /dev/vgora/lvu02 /u02
mount /dev/vgora/lvu03 /u03
mount /dev/vgora/lvu04 /u04
mount /dev/vgora/lvu05 /u05


root @ csora1 # /home/hp/mount.sh
/dev/vgora/lvu01: unrecognized file system

root @ csora1 # bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 83776 56180 60% /
/dev/vg00/lvol1 83733 54946 20413 73% /stand
/dev/vg00/lvol8 1142784 496757 609929 45% /var
/dev/vg00/lvol7 1265664 915765 329269 74% /usr
/dev/vg00/lvol4 155648 7997 138523 5% /tmp
/dev/vg00/lvol6 557056 528129 27121 95% /opt
/dev/vg00/lvol5 20480 15017 5145 74% /home
/dev/vgora/lvu02 15204352 2498554 12308742 17% /u02
/dev/vgora/lvu03 15204352 2757188 12058196 19% /u03
/dev/vgora/lvu04 15204352 1867818 12919770 13% /u04
/dev/vgora/lvu05 5005312 2939035 1944573 60% /u05
root @ csora1 #


vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c5t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.

After i did lvdisplay on /dev/vgora/lvu01 logical volume i get one disk missing and other disk as stale


--- Logical volumes ---
LV Name /dev/vgora/lvu01
VG Name /dev/vgora
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 19968
Current LE 2496
Allocated PE 4992
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t0d0 2169 2169
/dev/dsk/c4t1d0 327 327
/dev/dsk/c5t1d0 327 327

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t0d0 00000 stale ??? 00000 current
00001 /dev/dsk/c4t0d0 00001 stale ??? 00001 current
00002 /dev/dsk/c4t0d0 00002 stale ??? 00002 current
00003 /dev/dsk/c4t0d0 00003 stale ??? 00003 current
00004 /dev/dsk/c4t0d0 00004 stale ??? 00004 current
00005 /dev/dsk/c4t0d0 00005 stale ??? 00005 current
00006 /dev/dsk/c4t0d0 00006 stale ??? 00006 current
00007 /dev/dsk/c4t0d0 00007 stale ??? 00007 current

I assume the problem lies with the disk mirroring. Can some 1 help me out in this.

Thanks for your reply, pls do suggest.

Dayanand Naik.

[Johan Lorimier]

Seeing you last log, I believe that both your primary and secondary disk have some pb.
Are they seen in the ioscan output ?

Or there is a LVM bug that put in stale state the PE from the correct disk instead of the failed one.
Well, it seems that you have to restore your data.

A last trick is to try to swap the 2 device c4t0d0 and c5t0d0. The c5t0d0 will come online and as its this disk that are the PE in current state that might work.

May the luck be with you

Johan

[Jose Mosquera]

Hi,

Is this a Jamaica's class disk?
Is a claimed device?
#ioscan -fnC disk

If isn't claimed try to re-plug it. Then disable a enable the VG again. Pls follow the steps to don't interrupt your production!

Try to read disk iformation:
#diskinfo /dev/rdsk/c5t0d0
If you cannot read the disk info, you have a phisycal issue.

Rgds.

[Rita C Workman]

Possibly your disk is bad..or something has happened to it so it doesn't know it's part of that vg.

First...can you see the disk. Trying running a couple things against it like:
pvdisplay -v /dev/dsk/c5t0d0 or
diskinfo -v /dev/rdsk/c5t0d0

If you haven't seen any lbolt errors in your syslog pointing you to this disk, and if you can display the disk with the above, then it's not hardware. So try running:

echo "0x2008?4D" | adb /dev/dsk/c5t0d0

...and run it against one of the other disks in that vg...
Does the information match ? The last column in the output tells you the vgid. If this disk has lost the vgid info, or if it doesn't match the other disks, then it's possible that is why it won't 'rejoin' the group.

You could try doing a vgcfgrestore for that disk. Bring down the mountpoints for this vg.

vgchange -a n /dev/vgname
vgcfgrestore -n /dev/vgname /dev/dsk/c5t0d0
vgchange -a e /dev/vgname

Now try and mount things..you may want to run an fsck on things first, but see if it will mount then.

Just another thing to try and check...remember all problems are the process of elimination.

Rgrds,
Rita

[Stephen Doud]

Hi Dayanand

First, I looked at the log file you provided.
Whenever you see the message:
"Activation mode requested for the volume group conflicts with configured mode."
... this means that the vgchange method used on the disk (in this case, vgchange -a e) does not match the activation mode of the disk. There are 2 vgchange methods available when Serviceguard is installed:
Standard activation = vgchange -a y
Exclusive activation = vgchange -a e
So, when you know that the package control script uses '-a e', and it fails, that's because the VG's activation mode was flipped to standard mode.
To change the mode to "exclusive", insure SG is running on the node, and use "vgchange -c y "
To change the mode to "standard", use "vgchange -c n .

Now, to the specific problem...

Per the lvdisplay, /dev/vgora/lvu01 is on these disks:
/dev/dsk/c4t0d0 2169 2169
/dev/dsk/c4t1d0 327 327
/dev/dsk/c5t1d0 327 327

The extents are on:
--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t0d0 00000 stale ??? 00000 current
00001 /dev/dsk/c4t0d0 00001 stale ??? 00001 current

So which disk is missing?
The messages you provided would seem to suggest it may be c5t1d0, but notice that it's not the same LUN:

vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c5t0d0":

1) Can you restore the function of c5t1d0?
If not, I make no guarantees about the success of the following... but consider this:

2a) Do a vgdisplay vgora. Is there a difference between CUR PV and ACT PV? If so, and restoring the missing disk is out of the question, you may want to get the missing disk out of lvmtab and see if that allows you to proceed if you know the data on the disk is still current.
o cp /etc/lvmtab /etc/lvmtab.old
o strings /etc/lvmtab - take note of all the disks in vgora
o ll /dev/vgora - take note of the minor number, reuse it in the following steps
o vgexport vgora
o mkdir /dev/vgora
o mknod /dev/vgora/group c 64 0xNN0000 (where NN is the minor number from above)
o vgimport /dev/vgora /dev/dsk/____ (list all but c5t0d1
o activate vgora (may require converting the VG to "standard" activation)
If the activation complains about a missing disk, consider using "vgreduce -f" to match the ACT PV to the CUR PV.

2b) Another way to perhaps clear the problem is to "unstale" the extents.
To "unstale" extents, open a case with the HP Response Center and ask for a tool called "lvunstale" - which switches the disk showing the stale extents. It can be found at this HP-internal site
http://wtec.cup.hp.com/~hpux/fs/lvm/tools.htm

lvunstale will "unstale" a mirror copy held on an attached set of PVs which are stale and mark stale the mirror copy held on a missing set of PVs. A missing PV is reported via lvdisplay(1M) with ??? under the PV column, an attached PV is reported with the PV name (/dev/dsk/cXtYdZ). A PV can be made missing by exporting the volume group, importing with the PV missing and then re-activating the VG.

Good luck!
-SD-

[Dayanand Naik_2]

Hi All,

Thanks a ton to everyone who put in their suggestions. The disk are mirrored again and the cluster is forming and the package is starting up too.

1 more qts. when i forcefully stop one node, the other node does not take the package and startup which was working earlier.

What can be the problem.

Suggestion Welcomed.

Dayanand Naik.

[Jose Mosquera]

Hi,

Have you disabled the switching feature?
#cmmodpkg -d -n

You can view the current switcing status with "cmviewcl -v", you must check this area:

NODE_TYPE STATUS SWITCHING NAME
Primary up enabled (current)
Alternate up enabled

To enable switching feature into secondary node:
#cmmodpkg -e -n

Rgds.

[Bharat Katkar]

Daya,

Check you cluster status:

# cmviewcl -v

Check whether other node is up and running

Also check whether package switching is disable. This could be a cause mostly.

# cmmodpkg -e

Incase your node need to started:

# cmrunnode -v -n

Hope that helps.
Regards,

[Stephen Doud]

A node may not run a package for the following reasons:

1) AUTO_RUN or Node_switching may be disabled for the package. As others have suggested, use "cmviewcl -v -p " to view the states of these flags.
AUTO_RUN is a "global" switching flag, meaning, this package is allowed to fail over to ANY other adoptive node.
The Node_switching flags server-specific, meaning, this package is allowed(/not-allowed) to fail over to this specific node.
Use "cmmodpkg -e " to enable AUTO_RUN
Use "cmmodpkg -e -n " to enable Node_switching for that node.
See "man cmmodpkg" and the Managing Serviceguard manual for details.

2) The adoptive node does not have a resource available that the package requires.
The syslog.log and package control log are the major troubleshooting resources for this most SG problems.
Look at the package control log on the adoptive server where you expect the package to start up. Does the log reveal that an ERROR condition occured during package startup attempt? If so, Serviceguard is required to bring the package back down again, and possible toggle the Node_switching flag.
If no error exists in the control script log, check the syslog.log.
Syslog.log may show that a package-dependent SUBNET outage exists. The package is not allowed to start if such a condition exists.
Or syslog may show that a SERVICE the package uses to catch a lost application process exitted, signalling an application process disappeared abnormally.

Also, the cmviewcl report may show that the package depends upon a RESOURCE such as available space in a file system or some other special configuration. syslog will report if there is a problem with the resource.

3) the adoptive node is down or not a currently in the cluster

-SD-