Operating System - HP-UX
1833108 Members
3022 Online
110051 Solutions
New Discussion

update the LVs created in MC/Service Guard.

 
SOLVED
Go to solution
chung_2
New Member

update the LVs created in MC/Service Guard.

Hi,


I am new to MC/SG and have the following Qns.
We've got MC/SG running on 2 nodes. each node have one oracle instance on it.

If node1 have already created some new LVs( raw device )
How can I update it to the adoptive node ?

I got the advice as below:

i)Halt MC/serviceguard (node1, node 2)
ii)Deactivate VG from cluster control and vary off VG (node 1)
iii)Get LV map file(node1)
iv)Copy LV map file from node1 to node2
v)Export VG and import VG w/ new LV map file(node2)
vi)Vary on VG and verify LV(node2)
vii)Vary off VG (node2)
viii)Start MC service guard (node1 and node2)
ix)Active VG from cluster control(node1)
x)Halt MC/serviceguard (node1 and node2)
xi)Start MC/serviceguard (node1 and node2)
xii)Verify service(node1 & node2)



I have read through the ms/sg manual,
but seems doesn't mention about (vi) and (vii)
any suggestions ?

Below is the version of HP-UX and MC/SG we are using:
HP-UX B.11.00 U 9000/800
B3935DA A.11.12 MC / Service Guard
10 REPLIES 10
Kim Leong
Valued Contributor
Solution

Re: update the LVs created in MC/Service Guard.

Hi Cheung,

There is a detailed document available on this - check out http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065774166


Kim Leong
chung_2
New Member

Re: update the LVs created in MC/Service Guard.

Hi Kim,

Thanks for update.
But my situation is a bit different.
the LVs on node 1 is not newly created.
It already used by oracle ( as raw device) as one of the tablespaces.

Recently , there is hardware failure on node1
The oracle package failover to node2 sucessfully but we cant login to the instance.

after checking the oracle's alert log,
it's found that the instance cant mount those LVs....( as they were created on node1 but without update to node2 )

Thanks
Isralyn Manalac_1
Regular Advisor

Re: update the LVs created in MC/Service Guard.

Kindly post pkg cntl log and syslog and if possible, related cluster files.
Sridhar Bhaskarla
Honored Contributor

Re: update the LVs created in MC/Service Guard.

Hi,

You don't need to halt the package to update the LVs.

On the failover node where you need to update the VG configuration and not running the package, export the volume group(s) that needs to be updated. On the primary node create the map files and use them to import the VG on the failover server.

Say node A has the package running and it has got the updated configuration. node B is to updated with VG changes.

On node A:

#ll /dev/vgxx/group
(note the minor number 0x0?0000)
#vgexport -p -v -s -m /tmp/vgxx.map vgxx

Copy /tmp/vgxx.map onto node B

On node B:

#vgexport vgxx
(You are actually exporting vgxx)
#mkdir /dev/vgxx
#mknod /dev/vgxx/group c 64 0x0?0000
(Replace the 0x0?0000 with the minor number you noted before on nodeA)
#ll /dev/*/group
(ensure the minor number is uniq. If not, delete the group file you created and mknod again with next available minor number)
#vgimport -v -s -m /tmp/vgxx.map vgxx
(Do not activate the VG as it is already activated on the other node)

The above should import the logical volumes exactly like node A.

If these are raw logical volumes used by oracle, you will need to change the permissions on the r* files under /dev/vgxx directory.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
chung_2
New Member

Re: update the LVs created in MC/Service Guard.

Hi all,
Thanks for you guys reply.


Below is the pkg ctl log from Node1. We manually stop the package to force it to failover to Node2


########### Node "Node1": Halting package at Tue Jun 15 11:23:58 EAT 2004 #######
Jun 15 11:23:58 - Node "Node1": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 722 Terminated
Jun 15 11:23:59 - Node "Node1": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 651 Terminated
Jun 15 11:24:00 - Node "Node1": Shutdown mem database...Jun 15 11:24:01 -
Node "Node1": Stop SQL*NET ...Jun 15 11:24:02 - Node "Node1": Remove IP address xxxxxxxx from subnet xxxxxxxxx
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.

########### Node "Node1": Package halt completed at Tue Jun 15 11:24:02 EAT 2004
########### Node "Node1": Starting package at Tue Jun 15 12:10:24 EAT 2004 ######
Jun 15 12:10:24 - "Node1": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:10:25 - "Node1": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 12:10:25 - Node "Node1": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxx
Jun 15 12:10:26 - Node "Node1": Shutdown mem database...Jun 15 12:10:33 -
Node "Node1": Startup mem database...Jun 15 12:12:33 - Node "Node1": Startup SQL*NET ... oracle 138
8 1 0 12:10:37 ? 0:00 ora_pmon_mem
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"
########### Node "Node1": Package start completed at Tue Jun 15 12:12:38 EAT 2004
chung_2
New Member

Re: update the LVs created in MC/Service Guard.

This is the pkg ctl log on Node2 at the time we switch it from Node1 to Node2


########### Node "Node2": Starting package at Tue Jun 15 11:24:07 EAT 2004 ###########
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 11:24:08 - Node "Node2": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxxx
Jun 15 11:24:08 - Node "Node2": Shutdown mem database...Jun 15 11:24:09 -
Node "Node2": Startup mem database...Jun 15 11:24:17 - Node "Node2": Startup
SQL*NET ... oracle 9678 1 0 11:24:10 ? 0:00 ora_pmon_mem
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"

########### Node "Node2": Package start completed at Tue Jun 15 11:24:19 EAT 2004 ###########
########### Node "Node2": Halting package at Tue Jun 15 12:08:28 EAT 2004 ###########
Jun 15 12:08:28 - Node "Node2": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 3122 Terminated
Jun 15 12:08:29 - Node "Node2": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 3092 Terminated
Jun 15 12:08:30 - Node "Node2": Shutdown mem database...Jun 15 12:08:31 -
Node "Node2": Stop SQL*NET ...Jun 15 12:08:31 - Node "Node2": Remove IP address xxxxxxx from subnet xxxxxxxxxxxxx
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
chung_2
New Member

Re: update the LVs created in MC/Service Guard.

And below is part of the alert log on Node2 from oracle after package failover to Node2

**************************************
Completed: alter database mount
Tue Jun 15 11:24:15 2004
alter database open
Tue Jun 15 11:24:15 2004
Errors in file /ora/log/mem/bdump/dbw0_9680_mem.trc:
ORA-01157: cannot identify/lock data file 68 - see DBWR trace file
ORA-01110: data file 68: '/dev/vg02/rmem_tbl01'
ORA-27037: unable to obtain file status
HP-UX Error: 2: No such file or directory
Additional information: 3
******************************************

Actually, these files also got the same errors as aboved:
ORA-01110: data file 69: '/dev/vg02/rmem_idx16'
ORA-01110: data file 70: '/dev/vg02/rmem_idx17'
ORA-01110: data file 71: '/dev/vg02/rmem_idx18'
ORA-01110: data file 72: '/dev/vg02/rmem_idx19'
ORA-01110: data file 73: '/dev/vg02/rmem_idx20'
ORA-01110: data file 74: '/dev/vg02/rmem_idx21'
ORA-01110: data file 75: '/dev/vg02/rmem_idx22'

ORA-1157 signalled during: alter database open ...
Shutting down instance (abort)
License high water mark = 2
Instance terminated by USER, pid = 3155


*******************************************
I found that the following links are only exist on Node1 but not Node2 *******************************************

$pwd
$/dev/vg02
$ls -l

brw-r----- 1 root sys 64 0x020020 Aug 15 2003 mem_tbl01
crw-r----- 1 oracle dba 64 0x020020 Aug 15 2003 rmem_tbl01

brw-r----- 1 root sys 64 0x020021 Aug 22 2003 mem_idx16
crw-rw---- 1 oracle dba 64 0x020021 Aug 22 2003 rmem_idx16

crw-rw---- 1 oracle dba 64 0x020022 Aug 22 2003 rmem_idx17
brw-r----- 1 root sys 64 0x020022 Aug 22 2003 mem_idx17

brw-r----- 1 root sys 64 0x020023 Aug 22 2003 mem_idx18
crw-rw---- 1 oracle dba 64 0x020023 Aug 22 2003 rmem_idx18

brw-r----- 1 root sys 64 0x020024 Aug 22 2003 mem_idx19
crw-rw---- 1 oracle dba 64 0x020024 Aug 22 2003 rmem_idx19

crw-rw---- 1 oracle dba 64 0x020025 Aug 22 2003 rmem_idx20
brw-r----- 1 root sys 64 0x020025 Aug 22 2003 mem_idx20

brw-r----- 1 root sys 64 0x020026 Aug 22 2003 mem_idx21
crw-rw---- 1 oracle dba 64 0x020026 Aug 22 2003 rmem_idx21

brw-r----- 1 root sys 64 0x020027 Aug 22 2003 mem_idx22
crw-rw---- 1 oracle dba 64 0x020027 Aug 22 2003 rmem_idx22

***************************************
they are all RAW devices to oracle and that's why I ask for the standard procedures to update the LVs on the adoptive node .
thanks.


Sridhar Bhaskarla
Honored Contributor

Re: update the LVs created in MC/Service Guard.

Hi Chung,

Doesn't matter if they are raw devices or filesystems. As I said before you don't need to stop the package. Keep running the package on node1 to reduce downtime. Follow the procedure I gave you to refresh the configuration of vg02 on node2. When you get a chance or during the maintenance window, run the package on node2 and see if everything works.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
Simon Hargrave
Honored Contributor

Re: update the LVs created in MC/Service Guard.

Yes, you only need to stop the cluster if you're adding VG's that aren't already defined. If you're only adding LV's, then you can either: -

use vgexport/vgimport as above.

Or, you can do the required mknod's on the other node to create the block and char devices for the new LV.

eg

on node 1 you created /dev/vg10/oracle16lv
and ls -l /dev/vg10/*oracle16lv displays: -

brw-r----- 1 root sys 64 0x00000b May 26 18:12 /dev/vg00/oracle16lv
crw-r----- 1 root sys 64 0x00000b Jan 11 2002 /dev/vg00/roracle16lv

Then on node2, execute

mknod b 64 0x00000b /dev/vg10/oracle16lv
mknod c 64 0x00000b /dev/vg10/roracle16lv

That's all that should be needed.

Regards, Sy
Armin Kunaschik
Esteemed Contributor

Re: update the LVs created in MC/Service Guard.

Just a short note to Simon (and who else is interested :-) ):

You don't even need to stop the cluster if you add new VG's or want to remove VG's.

Simply do a vgchange -c y new_vg to the new VG and import it on the other nodes.

There is one exception: The new/old VG contains a cluster lock disk.

My 2 cents,
Armin
And now for something completely different...