Newbie Cluster Configuration

[Francisco Moore]

First I am a biologist so please keep that in mind! I am a long time but "periodic" VMS user. In addition I have managed my own stand alone VMS workstations on and off for a number of years. So I am sure that my questions may not be as clear as you might like but here goes . . .

I currently have a couple students that want to run a large number of simulations. Distributing jobs across a cluster is the way to go from a user standpoint. So we have acquired 9 es45s and we are trying to get a lightly coupled cluster together. About all we need is to be able to share a batch queue across the cluster. A LAN connection exists but I was not planning on any other connection between nodes.
My Question is essentially "what is the absolute easiest way to form a cluster that can share a queue for distribution of jobs across nodes?"

I think I may have created a cluster like this 10 or so years ago with Cluster_config, but have failed to get a workable cluster with Cluster_config_LAN this time. I only have 3 users and I am unlikely to have many changes in users. Can I, for instance, just form a cluster without automated sharing of any system files and instead manually distribute the authorization files across the nodes?

I have tried to form a cluster of two of our machines using cluster_config_LAN two ways. First I tried using multiple independent boot server heads, and this was fine in terms up cluster communications but unstable based on boot order and I could not get the license database to pass correctly.

Second I tried a head satellite configuration, but just after the cluster formed I would get a crash on the satellite (pretty cool since I had never seen a VMS system crash hard) with a lock up of the system account on the head. I suspect that again I am not correctly configuring/passing my common files for authorizations etc.

I am betting that I am not getting the proper alterations to logical names but the templates are a little rough to figure out for a self taught 'system manager' like my self.

I'm having fun learning but I am getting pragmatic at this point. I don't have any VMS people in my University to help me out and am looking for the simple solution.
Thanks for your thoughts!

Paco Moore
Integrated Bioscience Program
University of Akron

[Hoff]

"Absolutely the easiest" starts by reading the manuals.

http://www.hp.com/go/openvms/doc

There are two clustering manuals, and two system management manuals that are the obvious candidates here.

Why manuals? Configuration mistakes here can and variously do lead to severe data corruptions.

Your initial problems here almost certainly transpire from incorrectly shared database files; you can't be just slightly clustered here.

You're also going to have some issues here with the scale of servers and with the available bandwidth for your cluster; you're going to want gigabit Ethernet, and you're going to have to be careful with your switches.

You're also going to have one (and potentially busy) voting node here, or you're going to want or need to add a shared storage interconnect.

If you're on a budget, SCSI would be an obvious choice here. That can be combined with shadowing (for instance) to get three hosts sharing and serving three disks providing the cluster system disk.

Fibre Channel SAN would be another option.

Here are the files that need to be shared:

http://labs.hoffmanlabs.com/node/169

Here are cluster votes and expected votes:

http://labs.hoffmanlabs.com/node/153
http://labs.hoffmanlabs.com/node/569

Integrity servers are faster and smaller and cheaper, too.

And for a simple and brute-force batch-based grid, you can dispense with the OpenVMS cluster here and use the DQS package. Run ten nodes off local disks. (This has problems around username management and installing and maintaining packages in parallel, in the absence of single-signon or other technologies.)

If you're centrally setting up a compute grid here (and don't have specific requirements for OpenVMS software), then OpenVMS itself isn't a particularly good choice in terms of the available software. There's comparatively little software available for OpenVMS for what most folks consider compute clusters, for instance. (And yes, you can use OpenVMS Cluster capabilities for this task.)

http://labs.hoffmanlabs.com/node/320

[Robert Gezelter · Accepted solution]

Paco,

First, Let me welcome you to the HP ITRC OpenVMS Forum.

I concur with much of Hoff's comments, but will, with full respect, in some areas.

Having worked on heavy scientific computation in the past (some using OpenVMS clusters, some using other platforms), it is quite possible that lock traffic would not be a problem. Many scientific tasks are compute bound in the extreme, with little IO (one project that I supported had IO requirements in the tens of bytes/hour; even a serial Teletype was overkill).

Without details, it is difficult to be precise. I would certainly agree with Hoff's suggestion to get a shared interconnect, although I note that it need not span all of the machines, two or three would be sufficient to allow a single node to restart without creating problems for the other nodes.

I would configure the other nodes to use local system disks, but served common files (e.g., SYSUAF, RIGHTSLIST, and the queues). Personally, I would "clone" the system disks from a common master, and change the parameters that are specific to individual nodes (e.g., SCSNODE, SCSID, node names, IP addresses, DECnet node numbers).

I would also make sure that my computational jobs have regular checkpoints.

I would also start by creating a "cluster" of a single node, and then adding the other nodes, initially as diskless nodes, then using a disk served over MSCP as a source for BACKUP to copy the base system disk to each satellite node. This strategy reduces the amount of installs and hardware manipulations, at some one-time (per node) use of LAN bandwidth, a reasonable trade IMHO (I started doing this as an alternative to using TK50-based installations; it was easier to image the disk over the LAN, and then reboot and change parameters than it was to wait for the tape to spin and blink).

This approach also makes it a straightforward process to add nodes to the cluster.

I hope that the above is helpful. If I have been unclear, please let me know.

- Bob Gezelter, http://www.rlgsc.com

[Francisco Moore]

Thanks to both to both of you.

Hoff you clarified much of what I was picking up from the manuals and actually from the Hoffman labs site. You have added the context for me to see why I should do things certain ways. Sometimes I find it hard to find what I should be reading in the manuals. I also appreciate the eye towards economy and appropriateness of operating system. I am working with alphas because I'm doing this out of pocket and could get a dozen used alphas (I found a great source) for the cost of one low end integrity box, but I know that these will be my last alphas for many reasons including economics. I tried running linux clusters, but frankly I find the compilers are not up to the standards of the VMS compilers, and I can bring an inexperienced programmer up to speed faster on a VMS system.

Bob you are spot on about what I need. I will have essentially almost no Disk IO (a job may use nearly all the CPU on a node for a week, but not write to disk more than a few seconds once a day). I will think over the clone the first cluster member strategy.

I'm going to sit down again with the manuals today see if I can figure it all out and will hopefully post a message with my solution rather than further questions soon.
thanks
Paco

[Jan van den Ende]

Paco,

If you need another vote, I have done essentially what Bob is suggesting (quite often in fact, but it WAS over 15 years ago, and on VAXen mainly, although I _DID_ do some mixed Vax-Alpha clusters that way as well).
Cloning the system disk, mounting it as a data disk, adapting Node-specific parameters (node name, network specs -- remember they are now in DATA files, NOT in system files, they become only that after the move & boot-up of the new node).
At your number of systems, DO get the first clone right (which might include starting anew one or two times), and once you have the entire procedure, just do it exactly the same way as often as needed.

Do not hesitate to ask any unclear specifics!

SUCCESS!!

Proost.

Have one on me.

jpe

[Hoff]

Current price for Integrity rx2600 on the used market is somewhere around US$350 or so, each. Plus shipping.

Jan's had success with renaming hosts. I've had awful experiences with that in recent times. It really depends on what you're doing, and what's installed. I have a topic posted on the various "stuff" that can need to be changed when you're renaming a host.

And personally, I am increasingly viewing system disks and disk images as drop-in single-use installations for specific tasks, and not as the classic long-term and evolving installations that have been typical with OpenVMS in years past. This whether you're rolling the image in from InfoServer or from distro or otherwise. I'd like to see wider distribution of the FIS stuff to this end, too, but that's another discussion.

[Jan van den Ende]

Paco,

one (very important) thing I forgot to include in my previous entry.

In your first node, you need to define QMAN$MASTER as the directory where your queue control files will reside.
Better make that on a NON-system disk.
Before cloning, make sure taht disk gets mounted early in the bootstrap (SYLOGICALS is a suitable location) AND define QMAN$MASTER pointing there (also before queue manager gets started, again SYLOGICALS is a suitable point).

Failing that, the cluster NOT get operational (well, unless you go to multiple queue managers, but that is another can of worms).

Proost.

Have one on me.

jpe

[Jon Pinkley]

Paco,

RE:"what is the absolute easiest way to form a cluster that can share a queue for distribution of jobs across nodes?"

I thought DQS like Hoff, but when I looked at the SPD (software product description) it appears to be a print queue only solution.

http://h18000.www1.hp.com/info/SP2880/SP2880PF.PDF

Given your low I/O requirements and an infrequent need to reboot, the simplest way is to have a single system disk and boot 8 of the 9 nodes as satellite nodes. Boot performance will be bad, and disk access to disks served by the boot node from the satellite nodes will be slower (the system disk and any other shared data disks that the boot node is serving), the boot node will be a single point of failure, and it will have a larger load having to serve disks to the other nodes. But these may be acceptable tradeoffs for the relative ease of setup and maintenance, and what I haven't seen others mention, the homogeneity of the solution. You have a common system disk with common DCL tables, etc. What works on one node will in general work on another (except for hardware differences).

When you said your satellite crashed right after the cluster formed, was this satellite booting from the common system disk across the network? Did it have unique cluster root and SCSNODE and SCSSYSTEMID? When SYS$MANAGER:CLUSTER_CONFIG_LAN asks for the node's SCS node name and SCSSYSTEMID number, it is asking for what you want the new node to use, not what the SCSSYSTEMID of the current system. I just tried this on a test system and put in the SCS node name and SCSSYSTEMID of the system I was running on, and it didn't complain. I aborted before it asked for the system root. Allowing a user to enter the SCSNODE name or SCSSYSTEMID of a system that is currently a member of the cluster, without any warning, seems to be accident-prone. At any rate, it you specified the name or id of the current system, I would expect a crash of the satellite node on the first boot.

This it getting too long, so see attachment for some stuff about satellite nodes.

Jon