HP Virtual machine online migration fails in frozen phaze with SOME vm-guests

[Gemeente Gouda]

Does someone have an idea concerning the following error: we have 4 HP VM hosts with several vm guests. All can be migrated online without problems. The two biggest ones (more memory) fail however in the last phase:
hpvmmigrate: Starting guest 'tokyo' on target host 'africa'
Opening minor device and creating guest machine container
Creation of VM, minor device 3
Initialize guest memory mapping tables
Starting event polling thread

Online migration initiated by source 'asia.gouda.lok' (89.150.25.30)
Target: online migration started with encryption algorithm AES-128-CBC.

hpvmmigrate: Init phase completed successfully.
hpvmmigrate: Copy phase completed successfully.
hpvmmigrate: I/O quiesce phase completed successfully.
hpvmmigrate: Frozen phase (step 1) - progress 0%
Target: transfer aborted by source
hpvmmigrate: ERROR (tokyo): Remote message: Target guest exited. Status 2.
hpvmmigrate: ERROR (tokyo): Migration was aborted by timeout in Frozen phase step 2.

[Bhadresh]

Hi,

Which HPVM Version you are using? Is it your VM Host participating in multiple migrations at a time(as a source and as a target)?

Regards,
Bhadresh

[Gemeente Gouda]

The HPVM version is B.0.4.20.00 So the latest version. It is only 1 migration at a time.

The VM-guests concerning about however are quite busy (about 20 Oracle databases are running on it). Do you know if there is some limititation on resources or how busy a guest is ?

Thanks Robert.

[butti]

Hi,

is it an ServiceGuard Cluster ?
Could you post the logfile under "/var/adm/cmcluster/log"

by,
butti

[Gemeente Gouda]

No sorry this is not a HP Serviceguard cluster.
We have just 4 RX6000 HP Integrity servers, connected by ethernet in a 2x 1 Gb. HP APA network.

But, I tried to analyse it more:
I just tried to bring the vmquest in init-mode 2 (in stead of 3 - so without oracle databases) and now the migration succeeded !
So in some way this points to the fact that HP VM cannot migrate servers that are too busy unfortunately. Is this a know issue ?

[Eric SAUBIGNAC]

Bonjour,

Just passing through the forum ...

Regarding your latest post, some questions to investigate. But since I have not so much feedback today with online migration, maybe they are silly questions ;-)

- "Remote message: Target guest exited" --> are you sure that you have enough memory on the target host to start the VM ? I mean, if you use dynamic memory, you probably uses less when you work at runlevel2.

- "aborted by timeout in Frozen phase " --> how huge is the memory of the VM at run-level3 ? Depending on the network you use, maybe the transfert is too long.

- what about log files on both hosts : syslog, /var/opt/hpvm/common/hpvm_mon_log, /var/opt/hpvm/guests/tokyo/log ... ?


HTH

Eric

[Gemeente Gouda]

According to the logfile in the frozen phase there is insufficient time to copy the memory:


Source: online migration started with encryption algorithm AES-128-CBC.
Source: (protocol low) header read timeout, 30 seconds
Source: could not copy pages
Source: online migration abort - severity 3, status 9


Source: online migration started at Fri Sep 24 18:38:46 2010

Source: online migration started with encryption algorithm AES-128-CBC.
Source: frozen phase timeout abort - insufficient time available to transfer memory
Source: estimated 64 seconds required - 60 available
Source: need to transfer 5291 MB - Copy phase rate 82.1643 MB/s
Source: online migration abort - severity 3, status 1012
Watchdog Timer flags saved: expiration(0x0) use(0x4)

[Earl_Crowder]

Gemeente,

Couple of things you could try:

Disable encryption with -Y flag to hpvmmigrate, unless your environment requires it. In my experience it speeds up migration.

Use hpvmmodify -x migrate_frozen_phase_timeout=90 on the VM.

Earl

[Gemeente Gouda]

We changed the time-out to 90 (default is 30) and it worked !
Except that it may have negative consequenses for client-connections. !
I will close the discussion, thanks for your input !

Robert/Municipality of Gouda/Holland.

[Gemeente Gouda]

Closed: the timeout session can be changed but should be used with care.