QUORUM NOT PRESENT, OR SOME PHYSICAL VOLUME(S) ARE MISSING

[nsjayasundar]

Dear Gurus,

I have prob in activating in one of the shared vg in cluster , and getting the QUORUM NOT PRESENT, OR SOME PHYSICAL VOLUME(S) ARE MISSING" eroor.

I am giving below the command outputs.

# vgchange -a s /dev/vgredo1_a
vgchange: Couldn't activate volume group "/dev/vgredo1_a":
Quorum not present, or some physical volume(s) are missing.


swlist -l product |grep -i mc/serviceguard
MC/ServiceGuard A.11.15.00
SG-Domain-Tool B.01.09 MC/ServiceGuard HP Domain Script Templates
SG-FasTrack-Tool B.01.09 MC/ServiceGuard Netscape FastTrack Script Templates
SG-Informix-Tool B.01.09 MC/ServiceGuard Informix Script Templates
SG-NFS-Tool A.11.11.02 MC/ServiceGuard NFS Script Templates
SG-NMAPI A.11.15.00 SG-MC/ServiceGuard Oracle Script Templates
SG-Progress-Tool B.01.09 MC/ServiceGuard Progress Script Templates
SG-Sybase-Tool B.01.09 MC/ServiceGuard Sybase Script Templates

# dd if=/dev/rdsk/c8t0d1 of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out
# dd if=/dev/rdsk/c9t0d1 of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out

The same vg I could able to bring it up from another node. Given below the vg details.
## hostname
dbnode2
# vgdisplay -v /dev//vgredo1_a
--- Volume groups ---
VG Name /dev/vgredo1_a
VG Write Access read/write
VG Status available, shared, server
Max LV 255
Cur LV 4
Open LV 4
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 1016
VGDA 2
PE Size (Mbytes) 8
Total PE 127
Alloc PE 32
Free PE 95
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
Diskinfo details from the node1, where I am not able activate the vg.
# diskinfo /dev/rdsk/c8t0d1
SCSI describe of /dev/rdsk/c8t0d1:
vendor: HP
product id: STGAPPS LUN
type: direct access
size: 1048576 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c9t0d1
SCSI describe of /dev/rdsk/c9t0d1:
vendor: HP
product id: STGAPPS LUN
type: direct access
size: 1048576 Kbytes
bytes per sector: 512

# swlist -l product |grep -i lvm
LVM B.11.11 LVM
PHCO_27408 1.0 LVM commands cumulative patch, VA7410 HWE
PHKL_26743 1.0 LVM Cumulative Patch
#

At the same time, If the vg is not activated from node2, I could able to bring up from node1.

Earlist help is reqd

Regards

[Steven E. Protter]

Most common cause of this issue is the disk is activated in exclusive mode on another node.

vgchange -a n /dev/vgredo1_a

take it out of exlusive mode first.

vgchange -c n /dev/vgredo1_a

I'm not an expert yet, but I've done the classes and am stumbling through this for a forgiving client right now.

SEP

[nsjayasundar]

Hi,

Please see the below vgdisplay output from another node. It clearly says that it is activated in shared mode.
vgdisplay -v /dev//vgredo1_a
--- Volume groups ---
VG Name /dev/vgredo1_a
VG Write Access read/write
VG Status available, shared, server
Max LV 255
Cur LV 4
Open LV 4
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 1016
VGDA 2
PE Size (Mbytes) 8
Total PE 127
Alloc PE 32
Free PE 95
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0


And I recreated the vg also..
# vgdisplay /dev/vgredo1_a
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vgredo1_a".
# vgexport -m /tmp/vgredo.map vgredo1_a
# mkdir /dev/vgredo1_a
# mknod /dev/vgredo1_a/group c 64 0x080000
# vgimport -m /tmp/vgredo.map -v vgredo1_a /dev/dsk/c8t0d1 /dev/dsk/c9t0d1
Beginning the import process on Volume Group "vgredo1_a".
vgimport: Warning: Volume Group contains "1" PVs, "2" specified. Continuing.
Logical volume "/dev/vgredo1_a/lvredo1_1" has been successfully created
with lv number 1.
Logical volume "/dev/vgredo1_a/lvredo1_2" has been successfully created
with lv number 2.
Logical volume "/dev/vgredo1_a/lvredo1_3" has been successfully created
with lv number 3.
Logical volume "/dev/vgredo1_a/lvredo1_4" has been successfully created
with lv number 4.
Volume group "/dev/vgredo1_a" has been successfully created.
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating the volume group.
# vgchange -c y vgredo1_a
vgchange: Failed to set the specified attributes for volume group "/dev/vgredo1_a".
# vgchange -c y -S y vgredo1_a
vgchange: Failed to set the specified attributes for volume group "/dev/vgredo1_a".
# vgchange -a s vgredo1_a
vgchange: Couldn't activate volume group "vgredo1_a":
Quorum not present, or some physical volume(s) are missing.


Still I amgetting the same error..


Regards

[Thayanidhi]

How many paths are available to the disk from each node?

I doubt there is device file not created yet, or missing?

run insf -e on both nodes.

Your vg import command says 1 PV found but two specified. That could be reason for quarum!!

From node1, check the VG configured properly (i.e. two PVs), then then repeat vgimport process.

Please post "strings /etc/lvmtab" from both nodes.

Regds
TT

[Devender Khatana]

Hi,

It appears yours node1's /etc/lvmtab could be corrupt.Just see strings /etc/lvmtab on node1 and find out the listings of physical volumes in your VG. Also perform a backup of all LVOL's through other node before running all this.

Then you can try activating VG by bypassing the quorun check i.e. vhchange -a y -q n /dev/vgxx.
If /etc/lvmtab is incosistent try recreating thorugh vgscan.

HTH,
Devender

[Hilary Nicholson]

For ServiceGuard the volume group should be activated in exlusive mode, e.g

VG Name /dev/vgsg01
VG Write Access read/write
VG Status available, exclusive

When you activate the ServiceGuard volume group it is with -a e option e.g.

vgchange -a e /dev/vgsg01

Regards,

Hilary

[nsjayasundar]

Dear TT,

There is two path for everydisk from each node.

Just see my first post , for dd operation on both the path. There is no error.

diskinfo, dd, ioscan everything works fine.

Is there anything related to time stamp?.

Regards

Sundar

[nsjayasundar]

Hi,

Here is the entry in lvmtab for the vg

/dev/vgredo1_a
/dev/dsk/c9t0d1
/dev/dsk/c8t0d1

Regards

[Suraj_2]

Hi

From your second post

# vgchange -c y vgredo1_a
vgchange: Failed to set the specified attributes for volume group "/dev/vgredo1_a".
# vgchange -c y -S y vgredo1_a
vgchange: Failed to set the specified attributes for volume group "/dev/vgredo1_a".
# vgchange -a s vgredo1_a
vgchange: Couldn't activate volume group "vgredo1_a":
Quorum not present, or some physical volume(s) are missing

1)Check whether the cluster processes / daemons are running in the server using cmviewcl or ps command.
2)Generally VGs are activated in exclusive mode in MCSG environemnt and these activations are done thru packages.
From your first post, it seems that the PV( disk or LUN) is not available. Check using the ioscan ,whether the disks are available.
Then activate the VG, first in read only mode
i.e
vgchange -a r vgname

Hope this helps

Rgds
Suraj

[nsjayasundar]

Dear Suraj,

Thanks for the info.
We could able to activate the vg in read only mode. But we need it as a shared and server or client mode.

We have 3 packages in the server, which activtes 7 vg's. This package activates 3 vg's , in that only this vg is not getting activated. So we are not able to start the package.

All other packages are working fine, which means MC service guard is running.


Regarding the physical volume presence, just see the dd command output, it clears that there is no prob in accessing the disk.

Regards

Sundar

[Thayanidhi]

Hi sundar,

From your post, the vgdisplay from other node shows only one PV (active and current).

When vgimported you specified two PV paths?

Is second one is alternate link?

In that case try import using only one PV path.

If you are succuessful in activating the VG, then try to add alternate link. (vgextend)

Revert

Regds
TT

[Thayanidhi]

Totally how many VG's you have?

Please post

#kmtune -q maxvgs
#ll /dev/*/group

from both nodes

Regds
TT

[Devender Khatana]

Hi,

Checking maxvg's is a good options as you have allready 7 Vg's in cluster , One should be root VG, and check that there should not be any unsed VG information on some unused disk. As default maxvg value is 10 & you are allready approached to 8 from the information provided here.

Exporting this with any one of the two link's at a time is also a good option. Try doing it by both links one by one so that you can find the fault here is any.


HTH,
Devender

[nsjayasundar]

Dear TT,

We have two link for the disk, primary and alt link.

The details reqd by you is given below.


Parameter Current Dyn Planned Module Version
===============================================================================
maxvgs 16 - 16



crw-r----- 1 root sys 64 0x000000 Feb 21 2004 /dev/vg00/group
crw-rw-r-- 1 ora9i oinstall 64 0x050000 May 20 2004 /dev/vgab/group
crw-r----- 1 ora9i oinstall 64 0x010000 Apr 12 2004 /dev/vgbh_a/group
crw-rw-r-- 1 root sys 64 0x020000 Apr 12 2004 /dev/vgbh_b/group
crw-rw-rw- 1 root sys 64 0x030000 Feb 26 2004 /dev/vgbharc1/group
crw-rw-rw- 1 root sys 64 0x040000 Feb 26 2004 /dev/vgbharc2/group
crw-rw-rw- 1 root sys 64 0x070000 Feb 27 2004 /dev/vgdpprod/group
crw-rw-r-- 1 root sys 64 0x060000 Apr 29 2004 /dev/vgdwh/group
crw-rw-r-- 1 root sys 64 0x080000 May 16 01:52 /dev/vgredo1_a/group
crw-r--r-- 1 root sys 64 0x0a0000 May 24 2004 /dev/vgredo1_b/group
crw-r--r-- 1 ora9i oinstall 64 0x090000 May 24 2004 /dev/vgredo2_a/group
crw-r--r-- 1 root sys 64 0x0b0000 May 24 2004 /dev/vgredo2_b/group


Regards

Sundar

[Thayanidhi]

maxvgs and group files looks to be OK. Is it same from both nodes?

Did you try my suggestion of importing VG with one PV(without alternate link)?

Revert

Regds
TT

[nsjayasundar]

Hi,

Nothing is worked out, we decided to recreate vg's today as per HP eng's advice.

Before that we want to take the backup of the vg. We hvae 4 raw lv's in that vg used by oracle for redo logs. can anyone suggest, which is the best of doing raw lv backup and recreating vg, export and import to another system. And restoring back the data's in lv's. Command's also will be helpful.

Planned to do it tonight, earliest reply will be awarded.

Regards

[Thayanidhi]

Do you have current backup software installed (such as Dataprotector)? If yes then you back them (raw disk image) to tape.

I can think of only "dd if=/dev/vgxx/lvxx of=/tmp/backup" for raw volumes backup. Make sure the Lv's idle (no process is running).

Regds
TT

[nsjayasundar]

Dear TT,

Thanks for your immediate reply..

Can u please tell me the procedures to create, export and import vg to another system with both the system having alternate link.

Regards

[Devender Khatana]

Hi,

First identify the disks on one system where you want to create VG. Then create Vg on this system using
#pvcreate /dev/rdsk/cxtydz ( Use with -f option if not successful)
#mkdir /dev/vgxx ( Unique VG name )
#mkdir /dev/vgxx/group c 64 0x0?0000 (? is the unique no. on both systems identify by "ll /dev/vg*/group" and use next no.)
#vgcreate /dev/vgxx /dev/dsk/cxtydz ( Target disk/disks)
Create required logical volumes and file systems using
#lvcreate -L (Size in MBs) /dev/vgxx
#newfs -F vxfs /dev/vgxx/lvol1

( Repeat above two steps for multiple file systems)
#vgexport -p -s -m /tmp/vgxx.map
Copy /tmp/vgxx.map to other system.

Now on other system do
#mkdir /dev/vgxx ( Unique VG name same as on hosta )
#mkdir /dev/vgxx/group c 64 0x0?0000 (? same as on hosta)
#vgimport -s -m /tmp/vgxx.map vgxx
#vgchange -a y /dev/vgxx

Like this your vg will be accessible on both hosts. Whatever file systems were there on VG when you exported on hostsa will be there on hostsb after activation.

HTH,
Devender

[Thayanidhi · Accepted solution]

Since you have done several VG configurations under MC/SG I thought it's not required to post here.

1. Backup the current content to appropriate location.

2. Halt the package
3. vgchange -c n -S n /dev/vgxx
4. vgexport
Repeat 3 & 4 for second node.
5. mkdir /dev/vgxx
6. mknod /dev/vgxx/group c 64 0x0x00 ...
7. Repeat 5 & 6 for other node
8. pvcreate
9. vgcreate /dev/vgxx
9b. vgextend /dev/vgxx
10. lvcreate/newfs as needed (use same names)
11. vgchange -a n /dev/vgxx
12. vgchange -c y -S y /dev/vgxx
13. start the package
14. vgexport -p -v -m /tmp/vgxx.map /dev/vgxx
15. rcp /tmp/vgxx.map node2:/tmp
16. vgimport -m /tmp/vgxx /dev/vgxx
17. vgextend /dev/vgxx
18. vgchange -a n /dev/vgxx
19. vgchange -c y -S y /dev/vgxx
20. start the package.

21. After verification halt the package
22. start Vg in standard mode and do the restore.

May be I missed some steps, I can do things, but cannot remember 'in order' how I did!!

Since you are sharing the VG across the node hope you are using only raw LV across the nodes (for RAC) and not using any file systems! Also I assume you are aware that the RAC packages shouldn't be configured for switchover and there is no package IP!


Regds
TT

[nsjayasundar]

Dear TT,

U are correct , But still i need to cross verify myself with experts..

Regards

Sundar

[nsjayasundar]

Dear All,

Issue solved by applying the following patches..

PHKL_30607 and PHKL_31216

Thanks to all

Regards