update the LVs created in MC/Service Guard.

[chung_2]

Hi,


I am new to MC/SG and have the following Qns.
We've got MC/SG running on 2 nodes. each node have one oracle instance on it.

If node1 have already created some new LVs( raw device )
How can I update it to the adoptive node ?

I got the advice as below:

i)Halt MC/serviceguard (node1, node 2)
ii)Deactivate VG from cluster control and vary off VG (node 1)
iii)Get LV map file(node1)
iv)Copy LV map file from node1 to node2
v)Export VG and import VG w/ new LV map file(node2)
vi)Vary on VG and verify LV(node2)
vii)Vary off VG (node2)
viii)Start MC service guard (node1 and node2)
ix)Active VG from cluster control(node1)
x)Halt MC/serviceguard (node1 and node2)
xi)Start MC/serviceguard (node1 and node2)
xii)Verify service(node1 & node2)



I have read through the ms/sg manual,
but seems doesn't mention about (vi) and (vii)
any suggestions ?

Below is the version of HP-UX and MC/SG we are using:
HP-UX B.11.00 U 9000/800
B3935DA A.11.12 MC / Service Guard

[Kim Leong · Accepted solution]

Hi Cheung,

There is a detailed document available on this - check out http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065774166


Kim Leong

[chung_2]

Hi Kim,

Thanks for update.
But my situation is a bit different.
the LVs on node 1 is not newly created.
It already used by oracle ( as raw device) as one of the tablespaces.

Recently , there is hardware failure on node1
The oracle package failover to node2 sucessfully but we cant login to the instance.

after checking the oracle's alert log,
it's found that the instance cant mount those LVs....( as they were created on node1 but without update to node2 )

Thanks

[Isralyn Manalac_1]

Kindly post pkg cntl log and syslog and if possible, related cluster files.

[Sridhar Bhaskarla]

Hi,

You don't need to halt the package to update the LVs.

On the failover node where you need to update the VG configuration and not running the package, export the volume group(s) that needs to be updated. On the primary node create the map files and use them to import the VG on the failover server.

Say node A has the package running and it has got the updated configuration. node B is to updated with VG changes.

On node A:

#ll /dev/vgxx/group
(note the minor number 0x0?0000)
#vgexport -p -v -s -m /tmp/vgxx.map vgxx

Copy /tmp/vgxx.map onto node B

On node B:

#vgexport vgxx
(You are actually exporting vgxx)
#mkdir /dev/vgxx
#mknod /dev/vgxx/group c 64 0x0?0000
(Replace the 0x0?0000 with the minor number you noted before on nodeA)
#ll /dev/*/group
(ensure the minor number is uniq. If not, delete the group file you created and mknod again with next available minor number)
#vgimport -v -s -m /tmp/vgxx.map vgxx
(Do not activate the VG as it is already activated on the other node)

The above should import the logical volumes exactly like node A.

If these are raw logical volumes used by oracle, you will need to change the permissions on the r* files under /dev/vgxx directory.

-Sri

[chung_2]

Hi all,
Thanks for you guys reply.


Below is the pkg ctl log from Node1. We manually stop the package to force it to failover to Node2


########### Node "Node1": Halting package at Tue Jun 15 11:23:58 EAT 2004 #######
Jun 15 11:23:58 - Node "Node1": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 722 Terminated
Jun 15 11:23:59 - Node "Node1": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 651 Terminated
Jun 15 11:24:00 - Node "Node1": Shutdown mem database...Jun 15 11:24:01 -
Node "Node1": Stop SQL*NET ...Jun 15 11:24:02 - Node "Node1": Remove IP address xxxxxxxx from subnet xxxxxxxxx
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.

########### Node "Node1": Package halt completed at Tue Jun 15 11:24:02 EAT 2004
########### Node "Node1": Starting package at Tue Jun 15 12:10:24 EAT 2004 ######
Jun 15 12:10:24 - "Node1": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:10:25 - "Node1": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 12:10:25 - Node "Node1": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxx
Jun 15 12:10:26 - Node "Node1": Shutdown mem database...Jun 15 12:10:33 -
Node "Node1": Startup mem database...Jun 15 12:12:33 - Node "Node1": Startup SQL*NET ... oracle 138
8 1 0 12:10:37 ? 0:00 ora_pmon_mem
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"
########### Node "Node1": Package start completed at Tue Jun 15 12:12:38 EAT 2004

[chung_2]

This is the pkg ctl log on Node2 at the time we switch it from Node1 to Node2


########### Node "Node2": Starting package at Tue Jun 15 11:24:07 EAT 2004 ###########
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 11:24:08 - Node "Node2": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxxx
Jun 15 11:24:08 - Node "Node2": Shutdown mem database...Jun 15 11:24:09 -
Node "Node2": Startup mem database...Jun 15 11:24:17 - Node "Node2": Startup
SQL*NET ... oracle 9678 1 0 11:24:10 ? 0:00 ora_pmon_mem
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"

########### Node "Node2": Package start completed at Tue Jun 15 11:24:19 EAT 2004 ###########
########### Node "Node2": Halting package at Tue Jun 15 12:08:28 EAT 2004 ###########
Jun 15 12:08:28 - Node "Node2": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 3122 Terminated
Jun 15 12:08:29 - Node "Node2": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 3092 Terminated
Jun 15 12:08:30 - Node "Node2": Shutdown mem database...Jun 15 12:08:31 -
Node "Node2": Stop SQL*NET ...Jun 15 12:08:31 - Node "Node2": Remove IP address xxxxxxx from subnet xxxxxxxxxxxxx
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.

[chung_2]

And below is part of the alert log on Node2 from oracle after package failover to Node2

**************************************
Completed: alter database mount
Tue Jun 15 11:24:15 2004
alter database open
Tue Jun 15 11:24:15 2004
Errors in file /ora/log/mem/bdump/dbw0_9680_mem.trc:
ORA-01157: cannot identify/lock data file 68 - see DBWR trace file
ORA-01110: data file 68: '/dev/vg02/rmem_tbl01'
ORA-27037: unable to obtain file status
HP-UX Error: 2: No such file or directory
Additional information: 3
******************************************

Actually, these files also got the same errors as aboved:
ORA-01110: data file 69: '/dev/vg02/rmem_idx16'
ORA-01110: data file 70: '/dev/vg02/rmem_idx17'
ORA-01110: data file 71: '/dev/vg02/rmem_idx18'
ORA-01110: data file 72: '/dev/vg02/rmem_idx19'
ORA-01110: data file 73: '/dev/vg02/rmem_idx20'
ORA-01110: data file 74: '/dev/vg02/rmem_idx21'
ORA-01110: data file 75: '/dev/vg02/rmem_idx22'

ORA-1157 signalled during: alter database open ...
Shutting down instance (abort)
License high water mark = 2
Instance terminated by USER, pid = 3155


*******************************************
I found that the following links are only exist on Node1 but not Node2 *******************************************

$pwd
$/dev/vg02
$ls -l

brw-r----- 1 root sys 64 0x020020 Aug 15 2003 mem_tbl01
crw-r----- 1 oracle dba 64 0x020020 Aug 15 2003 rmem_tbl01

brw-r----- 1 root sys 64 0x020021 Aug 22 2003 mem_idx16
crw-rw---- 1 oracle dba 64 0x020021 Aug 22 2003 rmem_idx16

crw-rw---- 1 oracle dba 64 0x020022 Aug 22 2003 rmem_idx17
brw-r----- 1 root sys 64 0x020022 Aug 22 2003 mem_idx17

brw-r----- 1 root sys 64 0x020023 Aug 22 2003 mem_idx18
crw-rw---- 1 oracle dba 64 0x020023 Aug 22 2003 rmem_idx18

brw-r----- 1 root sys 64 0x020024 Aug 22 2003 mem_idx19
crw-rw---- 1 oracle dba 64 0x020024 Aug 22 2003 rmem_idx19

crw-rw---- 1 oracle dba 64 0x020025 Aug 22 2003 rmem_idx20
brw-r----- 1 root sys 64 0x020025 Aug 22 2003 mem_idx20

brw-r----- 1 root sys 64 0x020026 Aug 22 2003 mem_idx21
crw-rw---- 1 oracle dba 64 0x020026 Aug 22 2003 rmem_idx21

brw-r----- 1 root sys 64 0x020027 Aug 22 2003 mem_idx22
crw-rw---- 1 oracle dba 64 0x020027 Aug 22 2003 rmem_idx22

***************************************
they are all RAW devices to oracle and that's why I ask for the standard procedures to update the LVs on the adoptive node .
thanks.

[Sridhar Bhaskarla]

Hi Chung,

Doesn't matter if they are raw devices or filesystems. As I said before you don't need to stop the package. Keep running the package on node1 to reduce downtime. Follow the procedure I gave you to refresh the configuration of vg02 on node2. When you get a chance or during the maintenance window, run the package on node2 and see if everything works.

-Sri

[Simon Hargrave]

Yes, you only need to stop the cluster if you're adding VG's that aren't already defined. If you're only adding LV's, then you can either: -

use vgexport/vgimport as above.

Or, you can do the required mknod's on the other node to create the block and char devices for the new LV.

eg

on node 1 you created /dev/vg10/oracle16lv
and ls -l /dev/vg10/*oracle16lv displays: -

brw-r----- 1 root sys 64 0x00000b May 26 18:12 /dev/vg00/oracle16lv
crw-r----- 1 root sys 64 0x00000b Jan 11 2002 /dev/vg00/roracle16lv

Then on node2, execute

mknod b 64 0x00000b /dev/vg10/oracle16lv
mknod c 64 0x00000b /dev/vg10/roracle16lv

That's all that should be needed.

Regards, Sy

[Armin Kunaschik]

Just a short note to Simon (and who else is interested :-) ):

You don't even need to stop the cluster if you add new VG's or want to remove VG's.

Simply do a vgchange -c y new_vg to the new VG and import it on the other nodes.

There is one exception: The new/old VG contains a cluster lock disk.

My 2 cents,
Armin