Advice on setting up big clusters

[Olivier Masse]

I'm planning ServiceGuard clusters for a project that will be installed next spring. The architects have decided to use a "one cluster per server" approach, as we are currently doing with our other clusters. So for this project this means a total of seven 2-node clusters, with the second nodes being used for development and failover.

I was instead thinking of regrouping these nodes in one big 14 node cluster, so that in the future it could be possible to move the packages from one node to the other and have more flexibility.

Of course I'm nearing the 16 node limit so I might consider building two smaller clusters instead, but yet 7 clusters is a lot.

Is there anybody who have had bad experiences with big clusters, and are there any reasons to stick to smaller ones?

Thanks

[Bernhard Mueller]

Basically I like the idea of 2 seven-node clusters. You would not need a lock disk then.

Having more nodes on which a single package can run adds flexiblity but also complexity plus more effort for testing/verifying changes.

I think with two big clusters you should be able to distribute the load in a way so that you could potentially save one or two machines, which can sum up to a few $.

Having a two node cluster of identical machines for each HA application has the charm of simplicity on the other hand.

Regards,
Bernhard

[TOMAS BERNABEU]

He is not worse. After all, they are applications that are executed in machines. You must think about the load balance. It thinks about the money that you save in maintenance costs. It thinks about the comfort that you must now to migrate, to patch, etc. You must put this in a balance and value it. My case is similar to yours. I am integrating machines. Integrating storage, etc.

Tom

[A. Clay Stephenson]

Without knowing more of the specifics, it's very difficult to say but this seems like a large number of servers so the sales people should love this model. I would never tolerate having a test/development box in the same cluster as production. That is truly asking for disaster. About the only valid reason for this many clusters is that for some reason you have chosen copper SCSI and cable lengths and connection limits restrict your choices.

Again, I lack data but I would go for three clusters with one dedicated to test/development unless I were firmly convinced that the applications would be CPU bound -- which is rather rare in typical UNIX applications.

Your approach is more typical of PC guys who favor the "one big application per server" model.

[Olivier Masse]

I've done more reading this morning and can supply more info.

Test environments are already planned in a smaller external cluster. The 14 nodes I'm talking about are all production, this is part of an immense SAP project. And more nodes will come at a later phase so I'm considering these as well for a total of 20 nodes so at least two clusters are needed anyway. These nodes can be partitions in rp8400s or superdomes, we don't know yet what will be the least expensive.

Since we don't have a third data center and the nodes will be split even between two sites, there is a risk of a split brain if the connection is severed between both sites. We don't have a third site so a metrocluster or external quorum server can't be used.

The solution I'm currently documenting is building 5 medium-sized clusters with these 20 nodes, each covering a range of applications. Lock disks can't be used when there are over 4 nodes, so quorum servers will be used everywhere except in one two-node cluster which will still use a lock disk and run the quorum servers for everyone else. So basically, this cluster will be the tie breaker for the other ones should the link between the sites go down.

Any thoughts on this?

I know I could hire an HP consultant to help me on this, but we did so with our first SAP project and have succesfully built our own clusters afterwards, so I don't think we need one this time.

Thanks

[Stephen Doud]

Ongoing administration complexity and effort should be considered:

- Rolling upgrades inhibit cluster or package configuration changes until ALL nodes have been updated. Larger clusters take longer to complete the upgrade, hence, preventing cluster changes for a longer time.

- 2 10-node clusters require 2 cluster ASCII files, whereas 10 2-node clusters require 10.

- Automatic cluster formation requires ALL nodes be running and on the network in order to form a cluster. It is more likely that a dead node will prevent a 10-node cluster from forming than a 2-node cluster (more points of failure).

- No quorum server or lock disk needed for 5 or greater node clusters.

About extended clusters
- MetroCluster has a distance limitation and requires storage array data copying facility such as HP's XP-array Business Copy, or EMC's SRDF utility. (More redundant but more costly too)

StephenD

[Rita C Workman]

Some great info above already.
You mention two sites, but not a third site yet.
My question would be do you know if you will every need to fail from site to site?

As was mentioned above Metro Cluster may be something to look into. Anything above a certain distance requires a Continental Cluster.
Based on what you've said 'so-far' I would lean towards doing 2 clusters with 7 nodes in each...you can then extend these up to the node limit as you increase. That way you only have to local clusters to care for.
If you must have failover between sites...then you need to look into Continuace Access (HP Product) or SRDF (EMC Product).
The question might be - Do you have disk hardware and technology to copy from site to site??
And get ready to loose alot of sleep no matter which you choose....

Been there,
Rgrds,
Rita