Quorum server relocation + cluster nodes reconfiguration

[Noorisyam Hamid]

Hi Guys,

FYI, i have a quorum server to be relocated to another building and a new IP will be assigned to the quorum server. Currently the quorum server only uses two network interfaces for MP and OS.

My questions:
1. I have two clusters (with 2 nodes eachly) which are currently pointing to the quorum server. Do I need to have a complete downtime for both clusters??
2. According to the cmcluster.ascii configuration file, the QS HOST is currently set to hostname instead of IP address. Do relocation still requires those clusters to have a downtime?
3. WOuld appreciate if you can share the steps from relocating the quorum server plus the cluster nodes configuration.

Thanks in advance.

[Matti_Kurkela · Accepted solution]

Which version(s) of Serviceguard are you running in your clusters?

Knowing the Serviceguard version is quite important in this case: with the newest version (A.11.19), you can change the cluster lock configuration while the cluster is up & running, which means you can deconfigure the old quorum server hostname/IP and configure a new one any time you want.

With older versions, cluster downtime is required to change the cluster lock configuration, including the quorum server hostname/address.

But the required downtime can be quite short: if your clusters are otherwise OK, they will keep running even if the quorum server is not reachable. The only problem is, if the cluster nodes lose all heartbeat connections with each other *while* the quorum server is not available, both nodes will execute a TOC reboot.

If you have followed the cluster design recommendations in the "Managing Serviceguard" manual, you should have multiple heartbeat connections in both your clusters, and there should be no SPoFs, i.e. there should be no single component whose failure would stop all the heartbeat connections. If your situation is like this, you could easily keep your cluster running without a quorum server for a short while (maybe a day or two).

Then, once your quorum server is in the new location and you've verified all the cluster nodes can access it, you could have a short downtime for each cluster to update the cluster configuration:

1.) use cmgetconf to get the current cluster configuration in ASCII form
2.) modify the ASCII configuration file to point to the new quorum server
3.) use cmcheckconf to verify the new ASCII configuration file is good
4.) run cmhaltcl to stop the cluster (*downtime begins*)
5.) use cmapplyconf to update the cluster configuration
6.) run cmruncl to restart the cluster (*downtime ends*)

MK

[Noorisyam Hamid]

Hi MT,

Thanks for your reply.

FYI, below are the serviceguard versions:
Cluster 1: A.11.18.00
Cluster 2: A.11.19.00

As per your explanation, I only require a downtime for Cluster 1 due to older version.

Refering to the given steps, the cmcluster.ascii file for our both clusters are currently configured as follows:

QS_HOST ux-pen01.xxxx.xxxx.com
QS_POLLING_INTERVAL 120000000
QS_TIMEOUT_EXTENSION 2000000

Since the QS_HOST parameter is configured as hostname instead of IP address, do I still need to reapply the cluster configuration file/require downtime (Cluster 1)?

Appreciate of your advice.

Thanks.

[melvyn burnard]

If your DNS or hosts files point to the correct ip address, this should be fine to do without a restart of the cluster.
At worst you will see a message logged to syslog.log every so often saying it cannot communicate to the QS.

[Noorisyam Hamid]

Hi Melvyn,

Thanks for the advice.

[Noorisyam Hamid]

Hi Guys,

FYI, i have completed the relocation activity of the quorum server to another building.

Till today, our cluster nodes still have problem to communicate with the quorum server.

Initially, the qs IP address was: x.x.171.52
New IP address is : x.x.128.175

I did run # cmviewconf command on the cluster nodes and found that they are still pointing to the old IP:

.
.
qs host: ux-pen01.x.x.com
qs ip address 1: 172.17.171.52
.
..

When refering to the cmcluster configuration file, the qs host is actually defined by hostname instead of IP address:

.
.
QS_HOST ux-pen01.x.x.com
.
..

Can someone please advice, which configuration file should I modify to update the cluster nodes to point to the qs new IP address??

Thanks in advance.

[SoorajCleris]

Hi,

whats your nslookup output says from each nodes?

Is this new IP address in different subnet from the currecnt config? If yes please refere the "quorum server product release notes" ., what about the version of the quorum server product = try to use the latest.

I have seen cases, cluster need to be restarted even if you use aliases instaed if hostname/ipaddress in cluster configuration file.

==> your dns/host file should resolve the the host/ip properly.

So ready to restart your cluster.
Regards,
Sooraj

[Noorisyam Hamid]

Hi Sooraj,

Thanks for the reply.
FYI, all cluster nodes managed to lookup the quorum server via dns (after new IP changed).

According to my previous email, I wonder which cluster configuration file (other than cmcluster.ascii) should I look into to update the quorum server IP address since the cluster still shown/pointing the old IP address (via cmviewconf command).

Appreciate of your advice.
Thanks in advance.

[Mike Chisholm]

The QS IP address is resolved at cmapplyconf time and stored in the cluster binary file /etc/cmcluster/cmclconfig. You cannot edit this file directly. If you change the quorum server IP but not the hostname(assuming you use the hostname in your cluster ascii file), then you should be able to run a cmapplyconf command to update the stored IP address. Depending on your version of Serviceguard you may be able to do this as an online change.

You should refer to the Managing Serviceguard manual for the version you are running and look at Table 7-1 "Types of Changes to the Cluster Configuration" which will detail which operations are online or offline changes you can make. Chapter 7 in general is a good reference for these kinds of questions.

All the recent manuals can be found online here:
http://www.hp.com/go/hpux-serviceguard-docs

[Noorisyam Hamid]

Hi Mike,

Thanks to those replied.
FYI, I have completed the activity yesterday and went through the steps shared by all of you!

1. Online cluster configuration
I managed to re-apply the cmcluster.ascii binary file (while the cluster service running) on Cluster 2 (serviceguard version A.11.19.00) but not for Cluster 1.

Cluster 1: A.11.18.00 - impossible
Cluster 2: A.11.19.00 - possible

2. QS host - Nslookup@DNS server
The QS host in cmcluster.ascii has been defined as hostname instead of IP address and even though the new IP has been resolved after re-apply the cmcluster.ascii, the QS server still inaccessible by clusters (e.g. QS server status still shown error/down in syslog). The only possible solution was to restart the cluster service (cmhaltcl & cmruncl) to complete the QS update on cluster!

Thanks again to all!