Problem with serviceguard and integrity virtual machines

[Roberto Garcia]

Hi all,

I have a two nodes serviceguard cluster with virtual machines inside. I want to package one of them. I've imported configuration in both nodes and hpvmmigrate the VM. The VM is able to start in both nodes

I've used hpvmdevmgmt so you can see this output in both nodes:

# hpvmdevmgmt -l server
esfihp01:CONFIG=server,SERVERADDR=10.120.7.8,SERVERID=1,UUID=b9a30236-c006-11df-b226-e48b86d28647::WWID_NULL
esfihp02:CONFIG=server,SERVERADDR=10.120.7.9,SERVERID=2,UUID=bdd23568-c006-11df-8dec-e2d38fd2eaa5::WWID_NULL
# hpvmdevmgmt -l env
HPVM_MSE_GROUP_ENTRY:CONFIG=env,EXIST=NO,DEVTYPE=UNKNOWN,SHARE=NO,UUID=ab63b998-c188-11df-bd01-0017a4770000,GROUPNAME=HPVM-SG-clvirtual::WWID_NULL
# hpvmstatus -m
[HPVM Multi-server environment]

Server group name : HPVM-SG-clvirtual
Server group uuid : ab63b998-c188-11df-bd01-0017a4770000
This server's identifier: 1

Server_id : 1
Server_state : 1
Server_ipaddr : 10.120.7.8
Server_hostname : esfihp01
Server_uuid : b9a30236-c006-11df-b226-e48b86d28647
Server_physical_uuid : b9a30236-c006-11df-b226-e48b86d28647

Server_id : 2
Server_state : 1
Server_ipaddr : 10.120.7.9
Server_hostname : esfihp02
Server_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5
Server_physical_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5

("Server_uuid" info extracted from "machinfo" command for each node)

However, when i use the packaging script i have an error like this:

# /opt/cmcluster/toolkit/hpvm/hpvmsg_package fsaada01

This is the HP Virtual Machine Serviceguard Toolkit package creation
script.

This script will assist the user to develop and manage a set of
Serviceguard package configuration and associated toolkit template
files.

The templates generated by these scripts will handle many guest
configurations, but they are only templates and may not be
appropriate for your particular configuration needs. You are
encouraged to review and modify these template files, as needed,
for your particular environment.


Do you wish to continue? (y/n):y

[Virtual Machine Details]
Virtual Machine Name VM # OS Type State
==================== ===== ======= ========
fsaada01 2 HPUX Off
[Storage Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
disk scsi 0 1 0 0 0 lv /dev/vg00_fsaada01/rlv_vg00
[Network Interface Details]
Interface Adaptor Name/Num PortNum Bus Dev Ftn Mac Address
========= ========== ========== ======= === === === =================
vswitch lan sw_prod 1 0 0 0 36-39-77-18-72-61

Package the Virtual Machine Summarized above? (y/n):y

Checking the virtual machine and cluster configuration

Analysis of the hpvmstatus -m command on each node indicates that the
HPVM multi-server environment is not configured correctly. Before
hpvmsg_package can assist in creating the package configuration template files
the HPVM multi-server environment must be configured.

Configure Serviceguard nodes into the HPVM multi-server environment now? (y/n):y

ERROR : hpvmsg_package : The multi-server environment is not configured consistently on each node



The cluster is running and no package is configured yet.

Any ideas???

Thanks in advance

[Eric SAUBIGNAC]

Bonjour Roberto,

You have posted output of commands 'hpvmdevmgmt -l server', 'hpvmdevmgmt -l env' and 'hpvmstatus -m' from just one server.

I mean, are you sure that informations are consistent betweeen the 2 hosts ? Particulary UUID of the HPVM_MSE_GROUP_ENTRY and UUID of the hosts ?

Please post an 'hpvmstatus -m' from each node and a simple 'cmviewcl'

Eric

[Roberto Garcia]

Hi Eric,

I've used the hpvmdevmgmt command in order to have exactly the same output from command "hpvmstatus -m" in both nodes.

I attach additional info with the output of some commands made in both nodes.

Thanks,

robe

[Eric SAUBIGNAC]

Not possible to get the attached document. Don't know why. So, just write output in a post.

Which version of HPVM / MCSG ?

Eric

[Roberto Garcia]

These are the versions:

HPUX - 11.31
SG - 11.19
HPVM - 4.20

Output from server 1:

# hostname
esfihp01
# hpvmdevmgmt -l env
HPVM_MSE_GROUP_ENTRY:CONFIG=env,EXIST=NO,DEVTYPE=UNKNOWN,SHARE=NO,UUID=ab63b998-c188-11df-bd01-0017a4770000,GROUPNAME=HPVM-SG-clvirtual::WWID_NULL
# hpvmdevmgmt -l server
esfihp01:CONFIG=server,SERVERADDR=10.120.7.8,SERVERID=1,UUID=b9a30236-c006-11df-b226-e48b86d28647::WWID_NULL
esfihp02:CONFIG=server,SERVERADDR=10.120.7.9,SERVERID=2,UUID=bdd23568-c006-11df-8dec-e2d38fd2eaa5::WWID_NULL
# hpvmstatus -m
[HPVM Multi-server environment]

Server group name : HPVM-SG-clvirtual
Server group uuid : ab63b998-c188-11df-bd01-0017a4770000
This server's identifier: 1

Server_id : 1
Server_state : 1
Server_ipaddr : 10.120.7.8
Server_hostname : esfihp01
Server_uuid : b9a30236-c006-11df-b226-e48b86d28647
Server_physical_uuid : b9a30236-c006-11df-b226-e48b86d28647

Server_id : 2
Server_state : 1
Server_ipaddr : 10.120.7.9
Server_hostname : esfihp02
Server_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5
Server_physical_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5

# hpvmstatus -P fsaada01
[Virtual Machine Details]
Virtual Machine Name VM # OS Type State
==================== ===== ======= ========
fsaada01 23 HPUX On (OS)

[Authorized Administrators]
Oper Groups :
Admin Groups :
Oper Users :
Admin Users :

[Virtual CPU Details]
#vCPUs Entitlement Maximum
====== =========== =======
3 10.0% 100.0%

[Memory Details]
Total Reserved
Memory Memory
======= ========
6 GB 64 MB

[Dynamic Memory Information]
Minimum Target Memory Maximum
Memory Memory Entitlement Memory
=========== =========== =========== ===========
512 MB 2106 MB 0 MB 6144 MB

[Storage Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
disk scsi 0 1 0 0 0 lv /dev/vg00_fsaada01/rlv_vg00

[Network Interface Details]
Interface Adaptor Name/Num PortNum Bus Dev Ftn Mac Address
========= ========== ========== ======= === === === =================
vswitch lan sw_prod 1 0 0 0 36-39-77-18-72-61

[Misc Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
serial com1 tty console

# cmviewcl -v

CLUSTER STATUS
clvirtual up

NODE STATUS STATE
esfihp01 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg00_fsaada01 /dev/disk/disk27 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/3/0/0/0 lan0

NODE STATUS STATE
esfihp02 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg00_fsaada01 /dev/disk/disk24 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/3/0/0/0 lan0




Output from server 2:

# hostname
esfihp02
# hpvmdevmgmt -l env
HPVM_MSE_GROUP_ENTRY:CONFIG=env,EXIST=NO,DEVTYPE=UNKNOWN,SHARE=NO,UUID=ab63b998-c188-11df-bd01-0017a4770000,GROUPNAME=HPVM-SG-clvirtual::WWID_NULL
# hpvmdevmgmt -l server
esfihp01:CONFIG=server,SERVERADDR=10.120.7.8,SERVERID=1,UUID=b9a30236-c006-11df-b226-e48b86d28647::WWID_NULL
esfihp02:CONFIG=server,SERVERADDR=10.120.7.9,SERVERID=2,UUID=bdd23568-c006-11df-8dec-e2d38fd2eaa5::WWID_NULL
# hpvmstatus -m
[HPVM Multi-server environment]

Server group name : HPVM-SG-clvirtual
Server group uuid : ab63b998-c188-11df-bd01-0017a4770000
This server's identifier: 2

Server_id : 1
Server_state : 1
Server_ipaddr : 10.120.7.8
Server_hostname : esfihp01
Server_uuid : b9a30236-c006-11df-b226-e48b86d28647
Server_physical_uuid : b9a30236-c006-11df-b226-e48b86d28647

Server_id : 2
Server_state : 1
Server_ipaddr : 10.120.7.9
Server_hostname : esfihp02
Server_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5
Server_physical_uuid : bdd23568-c006-11df-8dec-e2d38fd2eaa5

# hpvmstatus -P fsaada01
[Virtual Machine Details]
Virtual Machine Name VM # OS Type State
==================== ===== ======= ========
fsaada01 2 HPUX Off

[Authorized Administrators]
Oper Groups :
Admin Groups :
Oper Users :
Admin Users :

[Virtual CPU Details]
#vCPUs Entitlement Maximum
====== =========== =======
3 10.0% 100.0%

[Memory Details]
Total Reserved
Memory Memory
======= ========
6 GB 64 MB

[Dynamic Memory Information]
NOTE: Dynamic data unavailable, configured values only
Minimum Target Memory Maximum
Memory Memory Entitlement Memory
=========== =========== =========== ===========
512 MB 2048 MB 0 MB 6144 MB

[Storage Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
disk scsi 0 1 0 0 0 lv /dev/vg00_fsaada01/rlv_vg00

[Network Interface Details]
Interface Adaptor Name/Num PortNum Bus Dev Ftn Mac Address
========= ========== ========== ======= === === === =================
vswitch lan sw_prod 1 0 0 0 36-39-77-18-72-61

[Misc Interface Details]
Guest Physical
Device Adaptor Bus Dev Ftn Tgt Lun Storage Device
======= ========== === === === === === ========= =========================
serial com1 tty console






# cmviewcl -v

CLUSTER STATUS
clvirtual up

NODE STATUS STATE
esfihp01 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg00_fsaada01 /dev/disk/disk27 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/3/0/0/0 lan0

NODE STATUS STATE
esfihp02 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg00_fsaada01 /dev/disk/disk24 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/3/0/0/0 lan0


Thanks again

[Eric SAUBIGNAC]

Well. After checking I can't see any missconfigured parameter :-((( Everything seems to be OK. So in a strict point of view of HPVM Multiserver environment it should be OK.

In the past I had some problems with previous version of hpvmsg_package. Beginning with HPVM 4.2 I thought it was more robust.


In fact I am a bit stuck


So, let us have some imagination ;-)

1)
In your 1st post you said "/opt/cmcluster/toolkit/hpvm/hpvmsg_package fsaada01"

The correct syntax is ".../hpvmsg_package -P fsaada01". You missed the -P. I know it's just a detail, but test. Maybe there is a bug when you omit the -P

2)
Try also the '-s' option (simulate) : ".../hpvmsg_package -s -P fsaada01" . With previous version of HPVM it as solved some problems.

3)
As far as I remember, I have always ran the command to package a VM from the node where the VM where running. Did you ?

4)
I have never tested online VM migration. Yes, shame on me :-(((

Anyway, maybe the problem could come from this state.

Once hpvmsg_package has done its job, you can notice that some special parameters of the VM have been modified. And one parameter is : online_migration=disable

So try to modify the VM, on both node and offline with :

hpvmmodify -P fsaada01 -x online_migration=disable


No more idea for the moment ...

Eric

[Eric SAUBIGNAC]

Maybe just found something :

PHYSUUID is missing in the database.

Here is an example from one of my client's hosts :

#hpvmdevmgmt -l env -V; hpvmdevmgmt -l server
HPVM_MSE_GROUP_ENTRY:CONFIG=env,EXIST=NO,DEVTYPE=UNKNOWN,SHARE=NO,UUID=c22dd704-68d6-11df-8b02-00226403810e,GROUPNAME=HPVM-SG-lukua::WWID_NULL
haran:CONFIG=server,SERVERID=1,SERVERADDR=10.64.8.70,UUID=fd8549d4-64a2-11df-8783-999ac6256892,PHYSUUID=fd8549d4-64a2-11df-8783-999ac6256892::WWID_NULL
uharte:CONFIG=server,SERVERID=2,SERVERADDR=10.64.8.72,UUID=024dea3d-64a3-11df-82dd-89a72cdf0383,PHYSUUID=024dea3d-64a3-11df-82dd-89a72cdf0383::WWID_NULL

[Eric SAUBIGNAC]

Back home ? Lucky guy ;-)

If I did no mistake with UUIDs, try on both nodes these 2 commands :


hpvmdevmgmt -m server:esfihp01:attr:PHYSUUID=b9a30236-c006-11df-b226-e48b86d28647
hpvmdevmgmt -m server:esfihp02:attr:PHYSUUID=bdd23568-c006-11df-8dec-e2d38fd2eaa5

Then try again hpvmsg_package

Eric

[Roberto Garcia]

Hi Eric,

Sorry but i was back at home...

Tomorrow i'll try your fix.

Thanks,

robe

[Dennis Handly]

>Eric: Not possible to get the attached document. Don't know why.

This is a known problem with the forum and IE. Use the following USL:
http://forums.itrc.hp.com//service/forums/getattachment.do?attachmentId=362121&ext=.ZIP

[Eric SAUBIGNAC]

@Dennis. If I understand, the tip is to use capital letters in the filename extension. Yesterday I tried this link and it did not worked :

http://forums.itrc.hp.com/service/forums/getattachment.do?attachmentId=362121&ext=.zip

With ".ZIP" instead of ".zip", effectively, it works much better.

Thanks a lot Dennis

[Roberto Garcia]

Hi Eric,

I tryed your commands:

hpvmdevmgmt -m server:esfihp01:attr:PHYSUUID=b9a30236-c006-11df-b226-e48b86d28647
hpvmdevmgmt -m server:esfihp02:attr:PHYSUUID=bdd23568-c006-11df-8dec-e2d38fd2eaa5

And all worked OK!!!!!

Thank you very much. Please, re-post the message and i assign you 10 points!!! (I assigned 7 before trying the fix)

[Dennis Handly]

>Eric: the tip is to use capital letters in the filename extension.

Or anything that's not ".zip".

>re-post the message and I assign you 10 points! (I assigned 7 before trying the fix)

You can assign 10 to any of Eric's other messages.

[Eric SAUBIGNAC · Accepted solution]

Happy that it worked :-)

Dont't know how you evaluate hpvm / mcsg integration, but I found that some features are poor :

- shutdown of a VM by the package is not smart : no execution of shutdown scripts

- monitoring of the health of the VM could be a problem if you just want to stop the VM for maintenance purposes.

So you will find in the attached file some adaptations I made to the scripts hpvmsg_mon and hpvmsg_stop. It is limited to legacy packages, some explanations are in french, one can probably do it better ... Take them ASIS

Eric

[Roberto Garcia]

Hi Eric,

I agree with the limitations of the config/control scripts generated by hpvmsg_package.

Thank you very much for your modified scripts, i think i'm going to use them.

[Roberto Garcia]

You can find the solution and more data within the discussion

[Eric SAUBIGNAC]

- WARNING -

Hi everybody. Be aware that the scripts I have posted in my latest post are not compatible with the script hpvmsg_move that manages online migration in the cluster. I have to modify them to be compliant with hpvmsg_move behaviour.

Also found a funny way to start a VM on 2 nodes, simultaneously in a 3 nodes cluster ... ;-)

- have a package able to run in this order on node1, node2 and node3
- configure hpmvsg_mon with NUM_OF_RETRYS=150
(greater than 100)
- start the package on node3
- move the package to node2 with hpvmsg_move
- wait and see what happens on node1 and node2
- reinstall the guest !!!

Eric