OVMS 7.2.1 cluster with RA3000

[Markus Waldorf_1]

Hi,

I'm going to receive some hardware soon and need to setup a 2-node OpenVMS cluster using OVMS 7.2.1.

The hardware will be 2 DS20E with 3X-KZPDC and 6-slot drive cage. The cluster storage will be KZPBA differential SCSI in each Alpha and Storage Works RA3000 with dual HSZ22 controllers. For Cluster Interconnect I'm going to use CCMAB memory channel.

I'm not a beginner, but unfortunately I have not done this for a long time and don't remember a lot of details anymore. I will have to setup everything from scratch including the cabling.

If I remember correctly I will use cluster_config, which also creates a local modparams.dat in each system root. I should set alloclass=1, votes=1. The location for "sysdump.dmp" is defined at the srm console, which I can direct to a local drive. I should not forget to remove the termination resistors on each HSZ22. What about alloclass and device naming on the RA3000 console?

Do you have some more tips to setup the system?

I have installed a PC with RA3000 SW 2.1 and will use serial connections and Windows GUI to configure the storage. I have found a good document "Storage Works RA3000 for OpenVMS SCSI Clusters Application Note", Version 1.0, on the CD, but many pages of the document (cluster.pdf) are truncated, in particular the tables.

Does anyone have a better version of this document?


Best Regards,
Markus

[marsh_1 · Accepted solution]

hi,

no docs, but wondering why v7.2-1, this is an old version now out of support, v7.3-2 or higher are currently supported.

fwiw

[Markus Waldorf_1]

We use a special application, which is 14 years old, and according to the vendor 7.3 has not been tested - users reported problems. 7.2 is supposed to work fine. Unfortunately I do not have enough time left to give it a try. Maybe I will do that on a test system later.

Memory Channel:

I have reviewed some more documentation meanwhile and I need to set jumpers on each of the memory channel cards in the system accordingly. J1 - Hub mode, node 0 jumper 2-3, node 1 no jumper. I will use BN39B-10 cable to connect the two cards. J3 needs to be on pin 1-2 for OVMS. Once I installed and connected the cards I can run "mc_diag" and "mc_cable" on each system srm console to verify the installation.

Question: J5 Alphaserver 8x00 mode - should I set this on DS20E?

[marsh_1]

AFAIK that is set off by default, i have never changed it for es45's when i have used memory channel in the past.

fwiw

[cnb]

Hi Markus,

J5:

J5 - AlphaServer 8X00 Mode
Increases the maximum sustainable bandwidth of 8X00 platforms by 10MB/s. If this jumper is inadvertently set in any other platform, the maximum sustainable
bandwidth will decrease by 10MB/s. This jumper may be overridden by the Module Configuration Register (MODCFG) in case it is not installed properly.

hth,

[Jeremy Begg]

Hi Markus,

I have a collection of RA3000 stuff on our public server.

http://ftp.vsm.com.au/ra3000/
or
ftp://ftp.vsm.com.au/ra3000/

The OpenVMS clustering documentation has notes on setting port-specfic allocation classes for shared SCSI storage.
http://h71000.www7.hp.com/doc/82final/6318/6318pro.html
http://h71000.www7.hp.com/doc/731final/4477/4477pro.html (esp. Chapter 6)

Regards,
Jeremy Begg

[Uwe Zessin]

The RA3000 does not provide allocation classes - I suggest to use different host allocation classes to create unique names for the CD-ROM devices and port allocation classes to create common names for the shared bus and, again, unique names for the disks in the embedded drive cage. That is some work, but gives a clean system.

When you use the same value for the host allocation classes you will have to deal with duplicate device names. If I remember correctly V7.2-1 was the first release to implement PACs completely working, so a cluster with all unique names and too many surprises was possible.

Be careful with the RA3000, though! It is a low-level OEMed box that does not even have embedded cache batteries and requires an external UPS. Good luck with it.

[Markus Waldorf_1]

Please correct me if I'm wrong but I think I have to use the same alloclass modparam on each host of the shared scsi bus. I'm planning to use alloclass=1, and the shared devices should show up like $2$DKxxx on each node.

Regarding local drives, should they not show up as nodename$dkxxx, or do I need to set this somewhere?

Where do I specify that I want to use Memory Channel for cluster interconnect. Is there just the change option in the cluster config utility, or can I use it right from the beginning?

Thanks.

[Jeremy Begg]

Hi Markus,

It's a while since I set up the shared SCSI cluster here, but I recall that it certainly wasn't necessary to set ALLOCLASS the same on both nodes. In fact it's recommended that ALLOCLASS *not* be the same on all nodes if they have local (non-shared) storage.

The shared SCSI bus must be set up using "port" allocation classes as explained in the OpenVMS documentation. Paraphrasing from Appendix A in the "Guidelines for OpenVMS Cluster Configurations"...

"All host adapters attached to a shared SCSI bus must have the same OpenVMS device name (e.g. PKA0) unless port allocation classes are used. Each system attached to a shared SCSI bus must have a non-zero disk allocation class value (ALLOCLASS)."

Port allocation classes are enabled by the SYSGEN parameter DEVICE_NAMING (set it to 1) and then use CLUSTER_CONFIG to set the port allocation class value. It creates the file SYS$SYSTEM:SYS$DEVICES.DAT which you can edit if you wish (but probably shouldn't).

Note that if ALLOCLASS is non-zero, local (non-shared) drives will be $ccc$Ddcuuu: where 'ccc' is the ALLOCLASS value. You can still refer to them as node$Ddcuuu: but SHOW DEVICE will show the ALLOCLASS form of the name.

I'm not sure about Memory Channel, I've never used it. I notice there are a whole load of MC_SERVICES_Pn SYSGEN parameters, some of which may be relevant. Note that SCS will use whatever paths are available for inter-node communication so it's quite possible you don't have to do anything.

Regards,
Jeremy Begg

[Markus Waldorf_1]

Hi Jeremy,

thanks a lot for your message. Meanwhile I have also done some reading in the OpenVMS Cluster Systems manual. Together with your info I'm concluding the following so far:

Port allocation is enabled by DEVICE_NAMING=1 and SCSSYSTEMIDH=0 sysgen parameters. Port allocation is a designation for all ports attached to a single interconnect. It replaces the node allocation class (ALLOCLASS) in the device names.

Port allocation class 0 does not become part of the device name. Instead, the name of the node to which the device is attached becomes the first part of the device name. The controller letter PKA, PKB, remains in the device name. The result will be e.g. node$DKA100, node$DKB100.

Port allocation class -1 disables port allocation and uses node ALLOCLASS for the bus interconnect.

A Port allocation class 1-32767 is used as the first part of device names for the attached devices. e.g. Port allocation class = 2 or 3, devices names result in $2$DKA100 or $3$DKA100

All nodes connecting to a shared SCSI bus must use port allocation class. All nodes of a cluster must use the same device names for devices attached to the cluster storage. Port allocation class can therefore not be 0, or -1.

My plan is:
- port allocation class 0 for all local controllers
- port allocation class 2 for the shared scsi bus.

Port allocation class can be assigned when using cluster_config.com to setup the cluster, or using SET/CLASS PKB 2 at the SYSBOOT> prompt (boot -fl 0,1). Or edit sys$system:sys$devices.dat manually.

Result:
Local devices will be nodeX$DKA100 on nodeX, and nodeY$DKA100 on nodeY. For the shared bus I will use port allocation class=2 and devices will show up like $2$DKA0, $2$DKA1 on nodes X and Y of the cluster.

Questions:

What could be the reason for assigning -1 to a bus and disable port allocation? Anything to do with SRM console variables? e.g. dumpfile?

I plan to use DUMPSTYLE=14 sysgen parameter and specify a local directory on a local drive in SRM sysdump variable.

I should disable SCSI reset on the adapters, in case of KZPBA, I read that I need to set the SRM OS_type variable to windows_nt to boot into the ARC utility, and than change it back after modifying card parameters. Is this correct?

Thanks,
Markus

[Markus Waldorf_1]

Something I found in the cluster_config batch file: If you do not define port allocation classes later in this procedure for shared SCSI buses, all nodes sharing a SCSI bus must have the same non-zero ALLOCLASS value. If multiple nodes connect to a shared SCSI bus without the same allocation class for the bus, system booting will halt due to the error or IO AUTOCONFIGURE after boot will keep the bus offline. OpenVMS 7.1-2.

[Markus Waldorf_1]

Finally I received the hardware.

To configure a drive I put 2 x 18 GB drives into the 6-drive bay and then used
P00>> run bios pya0
to start the SmartArray 5300 bios in order to create a logical drive RAID-1. Everything ok.

But when I do "sho dev" at the console I don't see any disks. Only the controllers
dva0, dqa0, pka0, pkb0, pkc0, pky0.

Shouldn't I see some dk* devices?

There is a 1 GB NIC in addition to a dual port 100 MB Ethernet card. I can see eia0 and eib0, which correspond to the 100 MB card, but I cannot see the 1 GB Ethernet card. It also shows all LED's on at the back. Is that normal?

Unlike the DS20, the DS20E guide does not outline anything about using specific PCI slots for specific cards.

The following is installed.

Slot 1 - PBXGK-BB VGA (Elsa Gloria Synergy)
Slot 2 - DEGPA-TA Gigabit Ethernet
Slot 3 - 3X-DE602-BB Dual Ethernet 10/100
Slot 4 - KZPBA-CB Scsi Wide/diff
slot 5 - CCMAB-AA Memory Channel
Slot 6 - 3X-KZPDC-BE (SmartArray 5302/128)

Is there something wrong?

Best regards,

[marsh_1]

hi,

similar has cropped up before here, see if any thing in the following thread applies :-

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1244830315822+28353475&threadId=1004733

hth

[Markus Waldorf_1]

Although the card was in hose1, I did not set the BOOTBIOS pya0 SRM variable. >>> set BOOTBIOS pya0 >>> INIT did the trick. No I see a dy* drive. Thanks!

What about the 1 GB NIC? Will this work after the system is installed - initialized by a driver, etc?

[cnb]

See here for driver info:

http://h18000.www1.hp.com/alphaserver/options/asds20e/asds20e___-degpa-ta.html

hth,

[cnb]

Sorry I forgot to also add this:

recommendation:

For VMS:
For systems using Gigabit Ethernet, increase system parameter NPAGEVIR by 1500K for each Gigabit Ethernet NIC in system.

Should work fine with the required SRM version, but I would update to the latest firmware all across the system and options.

hth,

[Markus Waldorf_1]

The SRM console sees the 1 logical drive from the SmartArray 5300, named "dy0", but the 7.2-1 installation does not see it. The SmartArray has firmware 3.56 which is newer than the one i have on the 6.8 firmaware cd (3.40) Any ideas?

[marsh_1]

hi,

did you set heap_expand 2mb ?

[marsh_1]

BTW

OpenVMS support for the Smart Array 5300A was introduced in the OpenVMS Alpha V7.3-1 release. The following TIMA kit is required:

DEC-AXPVMS-VMS731_FIBRE_SCSI-H0200â 4.PCSI

Full boot support from SA5300A logical volumes is provided in the supported Tru64 V5.1A and OpenVMS V7.3-1 operating system environments.

fwiw

[Markus Waldorf_1]

I cannot use 7.3-1 this time. So I will remove the SmartArray 5300 and put in some other normal Scsi controller. It should be good enough for its purpose. But I have another problem:

For RA3000 Pedestal (2 controller) I have the following configuration (working live cluster).

Node 1 - BN21W-0B (Scsi Y-splitter)
Node 2 - BN21W-0B (Scsi Y-splitter)
The inner ends of the Y split are connected together using BN21K-05 cable.
The outer ends of the Y split connect to each of the Controller's Host 0 in and Host 1 in using 2 BN38C-05 cables.

As far as I understand the above shows that the Nodes are in the middle of the SCSI bus and the HSZ22 controllers terminate each SCSI end. The KZPBA controllers inside the cluster nodes have termination resisters removed.


Unfortunately I'm missing the BN21W-0B Y-splitter cables/adapters. But I have rackmount RA3000 (2 controller) this time, which unlike the Pedestal version has Host 0 and 1 in and OUT.

Is it possible to configure a cluster storage with redundant controller by installing the RA3000 in the middle of the Scsi chain? I was wondering if it would work to terminate each adapter in the Nodes, connect Node 1 to controller Host 1 in, and Node 2 to controller Host 0 out. Then connect controller Host 1 out and Host 0 in together. Will this work?

[Markus Waldorf_1]

I found some interesting document regarding this the cabling of the RA3000.

So I guess I could just connect Node 1 and 2 of the cluster to the controller host 0 and 1 in, as outlined on page 8. With the Adpaters terminating the ends of the scsi chain.

I am a bit confused why our current cluster uses a Scsi Y-cable to interconnect each node. Apart from moving the nodes in the middle of the SCSI chain, what is the advantage?

What is the difference on page 9?

[Markus Waldorf_1]

Think I got all the answers about RA3000 Scsi cabling from a document I found on the net "Trucluster hardware configuraton technical update for RA3000". It shows several ways of possible scsi cabling and explains them. I have attached it if someone's interested. Too bad I did not have this info earlier.

[Markus Waldorf_1]

Unfortunatly I don't have a BN21W-0B Y-scsi adapter cable, so I use termination at the KZPBA adapters and have node 1 connected to controller host 0 in, node 2 to controller host 1 out, and a cable between controller host 0 out to host 1 in. I was able to use The StorageWorks software v1.1 and created 3 virtual RAID-1 disks.

The KZPBA adapters are listed as Qlogic ISP pka0 in "sho config" and I set pka0_host_id on node 1 to 7 and node 2 to 6, in order to avoid a scsi conflict. On each of the alpha's SRM console I can see the devices listed as dka0, dka1 and dka2.

Problem: At the beginning of the OVMS installation it asks for target disk and entering ? shows available disks. It shows dka0 status online. But, when I choose dka0 it reports "medium is offline" when trying to mount. I works fine on node 1, but fails on node 2. What am I missing here?

Another issue: I would like to enter the KZPBA bios to verify its settings. I enter "run bios pka0" and it starts the bios Utility, but the keyboard does not work to select an option or to move the cursor. Any ideas?

Thanks in advance,
Markus

[Markus Waldorf_1]

Today I decided to try a different cabling approach. I found two H8861AA tri-link adapters and they match to the adapter cable coming from each KZPBA controller. From there I connected each node to controller host 1 and 2, and use another wire to link the two nodes together. Is there a difference between a tri-link adapter and Y-cable? Obviously there is because I don't see any disks anymore?!

While removing the termination resistors on the KZPBA cards I noticed that one of them had 1 of the 8 resistors missing. Could this be the reason for the failing of the HD mounting on that node when using the RA3000 in mid-bus configuration?


Anyone there? Thanks,
Markus

[cnb]

Markus,

Do you have the KZPBA-CB User Guide?

http://vt100.net/mirror/mds-199909/cd3/storage/kzpbcugc.pdf

It should answer most of your termination and adapter questions.

You'll need to run eeromcfg to check/set the adapter's settings and starting the utility depends upon your console set-up configuration.

hth,