topic Re: update the LVs created in MC/Service Guard. in Operating System - HP-UX

update the LVs created in MC/Service Guard.

chung_2 — Thu, 17 Jun 2004 22:01:46 GMT

Hi,

I am new to MC/SG and have the following Qns.
We've got MC/SG running on 2 nodes. each node have one oracle instance on it.

If node1 have already created some new LVs( raw device )
How can I update it to the adoptive node ?

I got the advice as below:

i)Halt MC/serviceguard (node1, node 2)
ii)Deactivate VG from cluster control and vary off VG (node 1)
iii)Get LV map file(node1)
iv)Copy LV map file from node1 to node2
v)Export VG and import VG w/ new LV map file(node2)
vi)Vary on VG and verify LV(node2)
vii)Vary off VG (node2)
viii)Start MC service guard (node1 and node2)
ix)Active VG from cluster control(node1)
x)Halt MC/serviceguard (node1 and node2)
xi)Start MC/serviceguard (node1 and node2)
xii)Verify service(node1 & node2)

I have read through the ms/sg manual,
but seems doesn't mention about (vi) and (vii)
any suggestions ?

Below is the version of HP-UX and MC/SG we are using:
HP-UX B.11.00 U 9000/800
B3935DA A.11.12 MC / Service Guard

Re: update the LVs created in MC/Service Guard.

Kim Leong — Thu, 17 Jun 2004 22:19:59 GMT

Hi Cheung,

There is a detailed document available on this - check out http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065774166

Kim Leong

Re: update the LVs created in MC/Service Guard.

chung_2 — Thu, 17 Jun 2004 23:14:30 GMT

Hi Kim,

Thanks for update.
But my situation is a bit different.
the LVs on node 1 is not newly created.
It already used by oracle ( as raw device) as one of the tablespaces.

Recently , there is hardware failure on node1
The oracle package failover to node2 sucessfully but we cant login to the instance.

after checking the oracle's alert log,
it's found that the instance cant mount those LVs....( as they were created on node1 but without update to node2 )

Thanks

Re: update the LVs created in MC/Service Guard.

Isralyn Manalac_1 — Thu, 17 Jun 2004 23:56:50 GMT

Kindly post pkg cntl log and syslog and if possible, related cluster files.

Re: update the LVs created in MC/Service Guard.

Sridhar Bhaskarla — Fri, 18 Jun 2004 00:18:23 GMT

Hi,

You don't need to halt the package to update the LVs.

On the failover node where you need to update the VG configuration and not running the package, export the volume group(s) that needs to be updated. On the primary node create the map files and use them to import the VG on the failover server.

Say node A has the package running and it has got the updated configuration. node B is to updated with VG changes.

On node A:

#ll /dev/vgxx/group
(note the minor number 0x0?0000)
#vgexport -p -v -s -m /tmp/vgxx.map vgxx

Copy /tmp/vgxx.map onto node B

On node B:

#vgexport vgxx
(You are actually exporting vgxx)
#mkdir /dev/vgxx
#mknod /dev/vgxx/group c 64 0x0?0000
(Replace the 0x0?0000 with the minor number you noted before on nodeA)
#ll /dev/*/group
(ensure the minor number is uniq. If not, delete the group file you created and mknod again with next available minor number)
#vgimport -v -s -m /tmp/vgxx.map vgxx
(Do not activate the VG as it is already activated on the other node)

The above should import the logical volumes exactly like node A.

If these are raw logical volumes used by oracle, you will need to change the permissions on the r* files under /dev/vgxx directory.

-Sri

Re: update the LVs created in MC/Service Guard.

chung_2 — Fri, 18 Jun 2004 02:12:01 GMT

Hi all,
Thanks for you guys reply.

Below is the pkg ctl log from Node1. We manually stop the package to force it to failover to Node2

########### Node "Node1": Halting package at Tue Jun 15 11:23:58 EAT 2004 #######
Jun 15 11:23:58 - Node "Node1": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 722 Terminated
Jun 15 11:23:59 - Node "Node1": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 651 Terminated
Jun 15 11:24:00 - Node "Node1": Shutdown mem database...Jun 15 11:24:01 -
Node "Node1": Stop SQL*NET ...Jun 15 11:24:02 - Node "Node1": Remove IP address xxxxxxxx from subnet xxxxxxxxx
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:02 - Node "Node1": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.

########### Node "Node1": Package halt completed at Tue Jun 15 11:24:02 EAT 2004
########### Node "Node1": Starting package at Tue Jun 15 12:10:24 EAT 2004 ######
Jun 15 12:10:24 - "Node1": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:10:25 - "Node1": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 12:10:25 - Node "Node1": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxx
Jun 15 12:10:26 - Node "Node1": Shutdown mem database...Jun 15 12:10:33 -
Node "Node1": Startup mem database...Jun 15 12:12:33 - Node "Node1": Startup SQL*NET ... oracle 138
8 1 0 12:10:37 ? 0:00 ora_pmon_mem
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 12:12:38 - Node "Node1": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"
########### Node "Node1": Package start completed at Tue Jun 15 12:12:38 EAT 2004

Re: update the LVs created in MC/Service Guard.

chung_2 — Fri, 18 Jun 2004 02:13:33 GMT

This is the pkg ctl log on Node2 at the time we switch it from Node1 to Node2

########### Node "Node2": Starting package at Tue Jun 15 11:24:07 EAT 2004 ###########
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg01 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 11:24:07 - "Node2": Activating volume group /dev/vg02 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Jun 15 11:24:08 - Node "Node2": Adding IP address xxxxxxxxxx to subnet xxxxxxxxxxxxxx
Jun 15 11:24:08 - Node "Node2": Shutdown mem database...Jun 15 11:24:09 -
Node "Node2": Startup mem database...Jun 15 11:24:17 - Node "Node2": Startup
SQL*NET ... oracle 9678 1 0 11:24:10 ? 0:00 ora_pmon_mem
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_mem using
"/etc/cmcluster/pkg1/oracle/mem/oracle.mon /ora/817 mem 10"
Jun 15 11:24:19 - Node "Node2": Starting service ORACLE_sqlnet_mem using
"/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon /ora/817 mem 10"

########### Node "Node2": Package start completed at Tue Jun 15 11:24:19 EAT 2004 ###########
########### Node "Node2": Halting package at Tue Jun 15 12:08:28 EAT 2004 ###########
Jun 15 12:08:28 - Node "Node2": Halting service ORACLE_mem
/etc/cmcluster/pkg1/oracle/mem/oracle.mon[22]: 3122 Terminated
Jun 15 12:08:29 - Node "Node2": Halting service ORACLE_sqlnet_mem
/etc/cmcluster/pkg1/oracle/mem/sqlnet.mon[30]: 3092 Terminated
Jun 15 12:08:30 - Node "Node2": Shutdown mem database...Jun 15 12:08:31 -
Node "Node2": Stop SQL*NET ...Jun 15 12:08:31 - Node "Node2": Remove IP address xxxxxxx from subnet xxxxxxxxxxxxx
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg01
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Jun 15 12:08:31 - Node "Node2": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.

Re: update the LVs created in MC/Service Guard.

chung_2 — Fri, 18 Jun 2004 02:21:16 GMT

And below is part of the alert log on Node2 from oracle after package failover to Node2

**************************************
Completed: alter database mount
Tue Jun 15 11:24:15 2004
alter database open
Tue Jun 15 11:24:15 2004
Errors in file /ora/log/mem/bdump/dbw0_9680_mem.trc:
ORA-01157: cannot identify/lock data file 68 - see DBWR trace file
ORA-01110: data file 68: '/dev/vg02/rmem_tbl01'
ORA-27037: unable to obtain file status
HP-UX Error: 2: No such file or directory
Additional information: 3
******************************************

Actually, these files also got the same errors as aboved:
ORA-01110: data file 69: '/dev/vg02/rmem_idx16'
ORA-01110: data file 70: '/dev/vg02/rmem_idx17'
ORA-01110: data file 71: '/dev/vg02/rmem_idx18'
ORA-01110: data file 72: '/dev/vg02/rmem_idx19'
ORA-01110: data file 73: '/dev/vg02/rmem_idx20'
ORA-01110: data file 74: '/dev/vg02/rmem_idx21'
ORA-01110: data file 75: '/dev/vg02/rmem_idx22'

ORA-1157 signalled during: alter database open ...
Shutting down instance (abort)
License high water mark = 2
Instance terminated by USER, pid = 3155

*******************************************
I found that the following links are only exist on Node1 but not Node2 *******************************************

$pwd
$/dev/vg02
$ls -l

brw-r----- 1 root sys 64 0x020020 Aug 15 2003 mem_tbl01
crw-r----- 1 oracle dba 64 0x020020 Aug 15 2003 rmem_tbl01

brw-r----- 1 root sys 64 0x020021 Aug 22 2003 mem_idx16
crw-rw---- 1 oracle dba 64 0x020021 Aug 22 2003 rmem_idx16

crw-rw---- 1 oracle dba 64 0x020022 Aug 22 2003 rmem_idx17
brw-r----- 1 root sys 64 0x020022 Aug 22 2003 mem_idx17

brw-r----- 1 root sys 64 0x020023 Aug 22 2003 mem_idx18
crw-rw---- 1 oracle dba 64 0x020023 Aug 22 2003 rmem_idx18

brw-r----- 1 root sys 64 0x020024 Aug 22 2003 mem_idx19
crw-rw---- 1 oracle dba 64 0x020024 Aug 22 2003 rmem_idx19

crw-rw---- 1 oracle dba 64 0x020025 Aug 22 2003 rmem_idx20
brw-r----- 1 root sys 64 0x020025 Aug 22 2003 mem_idx20

brw-r----- 1 root sys 64 0x020026 Aug 22 2003 mem_idx21
crw-rw---- 1 oracle dba 64 0x020026 Aug 22 2003 rmem_idx21

brw-r----- 1 root sys 64 0x020027 Aug 22 2003 mem_idx22
crw-rw---- 1 oracle dba 64 0x020027 Aug 22 2003 rmem_idx22

***************************************
they are all RAW devices to oracle and that's why I ask for the standard procedures to update the LVs on the adoptive node .
thanks.

Re: update the LVs created in MC/Service Guard.

Sridhar Bhaskarla — Fri, 18 Jun 2004 08:24:19 GMT

Hi Chung,

Doesn't matter if they are raw devices or filesystems. As I said before you don't need to stop the package. Keep running the package on node1 to reduce downtime. Follow the procedure I gave you to refresh the configuration of vg02 on node2. When you get a chance or during the maintenance window, run the package on node2 and see if everything works.

-Sri

Re: update the LVs created in MC/Service Guard.

Simon Hargrave — Fri, 18 Jun 2004 09:14:27 GMT

Yes, you only need to stop the cluster if you're adding VG's that aren't already defined. If you're only adding LV's, then you can either: -

use vgexport/vgimport as above.

Or, you can do the required mknod's on the other node to create the block and char devices for the new LV.

eg

on node 1 you created /dev/vg10/oracle16lv
and ls -l /dev/vg10/*oracle16lv displays: -

brw-r----- 1 root sys 64 0x00000b May 26 18:12 /dev/vg00/oracle16lv
crw-r----- 1 root sys 64 0x00000b Jan 11 2002 /dev/vg00/roracle16lv

Then on node2, execute

mknod b 64 0x00000b /dev/vg10/oracle16lv
mknod c 64 0x00000b /dev/vg10/roracle16lv

That's all that should be needed.

Regards, Sy

Re: update the LVs created in MC/Service Guard.

Armin Kunaschik — Sat, 19 Jun 2004 09:55:39 GMT

Just a short note to Simon (and who else is interested :-) ):

You don't even need to stop the cluster if you add new VG's or want to remove VG's.

Simply do a vgchange -c y new_vg to the new VG and import it on the other nodes.

There is one exception: The new/old VG contains a cluster lock disk.

My 2 cents,
Armin