topic Re: OpenVMS Blade lost boot entry after shutdown in Operating System - OpenVMS

OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 10:17:13 GMT

Hello community, I'm in trouble with on blade BL860C with OpenVMS 8.4.

This blade is connected on EVA storage and OpenVMS is on storage. EVA storage was in fault so I had to shutdown the Blade with OpenVMS. Now, with HP support, EVA is ok, but OpenVMS doesn't boot anymore.

I tried to enter in EFI shell and boot manually, it seems I identified the correct OS disk:

Problem is that it hangs to the following boot part and there is no way to continue boot process:

Previously it works perfect. Ant clue?

Thank you

Lucas

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 06:38:34 GMT

Lucas,

I'd guess, that it can't find/mount the system disk.

Boot from the Openrating System DVD and see, if you can find and mount the system disk.

Then use @sys$MANAGER:BOOT_OPTIONS to re-configure the boot settings.

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 06:53:01 GMT

Thank you for reply Volker, I have an internal blade disk where I can boot and mount the system disk without any issue, plus I can see the other disks presented by the storage, is there anything I can do from there?

EDIT - Here is the output of the command you gave me

$$$ @sys$MANAGER:BOOT_OPTIONS OpenVMS I64 Boot Manager Boot Options List Management Utility (1) ADD an entry to the Boot Options list (2) DISPLAY the Boot Options list (3) REMOVE an entry from the Boot Options list (4) MOVE the position of an entry in the Boot Options list (5) VALIDATE boot options and fix them as necessary (6) Modify Boot Options TIMEOUT setting (B) Set to operate on the Boot Device Options list (D) Set to operate on the Dump Device Options list (G) Set to operate on the Debug Device Options list (E) EXIT from Boot Manager utility You can also enter Ctrl-Y at any time to abort this utility. Enter your choice: 2 To display all entries in the Boot Options list, press Return. To display specific entries, enter the entry number or device name. (Enter "?" for a list of devices): EFI Boot Options list: Timeout = 10 secs. Entry Description Options ----- ---------------------------------------------------------- ------------- 1 OpenVMS IA64 FGB0.5001-4380-04c6-6559 -l00 $1$DGA100 PCI(0|4|0|0) Fibre(5001438004C66558,Lun64000000000000) 2 EFI Shell [Built-in] VenHw(d65a6b8c-71e5-4df0-d2f009a9) 3 OpenVMS IA64 FGB0.5001-4380-04c6-655D -l000 $1$DGA100 PCI(0|4|0|0) Fibre(5001438004C6655C,Lun64000000000000) -------------------------------------------------------------------------------- 3 entries found.

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 06:51:29 GMT

Lucas,

try $ ANALYZE/DISK $1$DGA100:

Then @sys$MANAGER:BOOT_OPTIONS to define a new boot for this disk

Then try a full verbose boot with VMS_LOADER -fl 0,30000

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 07:00:08 GMT

This is what I did, I deleted the old boot entries and then add a new one from DGA100:

Before try to reboot, may I have to do something else?

OpenVMS I64 Boot Manager Boot Options List Management Utility (1) ADD an entry to the Boot Options list (2) DISPLAY the Boot Options list (3) REMOVE an entry from the Boot Options list (4) MOVE the position of an entry in the Boot Options list (5) VALIDATE boot options and fix them as necessary (6) Modify Boot Options TIMEOUT setting (B) Set to operate on the Boot Device Options list (D) Set to operate on the Dump Device Options list (G) Set to operate on the Debug Device Options list (E) EXIT from Boot Manager utility You can also enter Ctrl-Y at any time to abort this utility. Enter your choice: 1 EFI Built-in EFI Shell Enter the device name (Enter "?" for a list of devices): $1$DGA100: Enter the desired position number (1,2,3,,,) of the entry. To display the Boot Options list, enter "?" and press Return. Position [1]: ? EFI Boot Options list: Timeout = 10 secs. Entry Description Options ----- ---------------------------------------------------------- ------------- 1 EFI Shell [Built-in] VenHw(d65a6b8c-71e5-4df0-d2f009a9) -------------------------------------------------------------------------------- 1 entries found. Enter the desired position number (1,2,3,,,) of the entry. To display the Boot Options list, enter "?" and press Return. Position [1]: 1 Enter the value for VMS_FLAGS in the form n,n. VMS_FLAGS [NONE]: 0,0 Enter a short description (do not include quotation marks). Description ["$1$DGA100:"]: OpenVMS IA64 FGB0.5001-4380-04c6-6559 efi$bcfg: $1$dga100: (Boot0002) Option successfully added efi$bcfg: $1$dga100: (Boot0003) Option successfully added efi$bcfg: $1$dga100: (Boot0004) Option successfully added efi$bcfg: $1$dga100: (Boot0005) Option successfully added

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 07:15:51 GMT

Here is the output of: VMS_LOADER -fl 0,30000

https://pastebin.com/aTGfLjht

Honestly I'm stuck like the blade!!!

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 09:17:50 GMT

Lucas,

I see:

%SYSINIT-I- waiting to form or join an OpenVMS Cluster

This Blade is supposed to become a cluster member ? Did you wait long enough for possible 'Have connection to' messages ?

After the '%STACONFIG-I-CANGO, enabling process deletion' message you would expect messages like:

%STACONFIG-I-MSCP, configuring MSCP/HSC disks/tapes
%CNXMAN,  Sending VMScluster membership request to system I64VMS
%CNXMAN,  Now a VMScluster member -- system VSIAXP

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 09:34:19 GMT

Hi Volker, yes, this is a cluster member, but the second node is powered down for hardware issue (system board in fault). Today regarding the hardware is a disaster!!!

Now I did a manual boot from fs1: VMS_LOADER -fl 0,0 and I'm waiting as per your advice, something is changed since I can see now link up/down, I believe this is the cluster. I'll keep the node in this state, waiting for hardware replacement in other node

%LLB0, Logical LAN failset device connected to physical device EIA0
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down

I hope this will solve the issue.

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 09:41:38 GMT

Lucas,

so this node is unable to form a cluster, if the 2nd node is missing. Is this intentional ?

You can always boot this node conversationally ( -fl 0,1 ) and set EXPCTED_VOTES to the same value as VOTES. This will at least allow this node to boot - if it can do any useful work without the other node being present.

If EWA0 is the cluster interconnect - directly cabled to the other node - these messages can be expected, if the other node is down.

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 10:03:57 GMT

So, you suggest to do:

fs1: VMS_LOADER -fl 0,1

Currently I have this config:

SYSBOOT> show CLUSTER/EXPECTED_VOTES Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- CLUSTER_CREDITS 128 32 10 128 Credits

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 09:48:07 GMT

Lucas,

a conversational boot ( -fl 0,1 ) will get you into SYSBOOT

SYSBOOT> SHOW VOTES

SYSBOOT> SHOW EXPECTED

SYSBOOT> SET EXPECTED_VOTES 1 ! if VOTES=1

SYSBOOT> CONT

Good luck,

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 10:11:33 GMT

THank you for your time Volker, FINALLY THE NODE IS UP !!!!!!!!!! This is what I did:

fs1:\EFI\VMS> VMS_LOADER -fl 0,1 PGQBT-I-INIT-UNIT, IPB, PCI device ID 0x2432, FW 4.02.02 PGQBT-I-BUILT, version X-33, built on Apr 17 2012 @ 18:29:57 PGQBT-S-SERDES, status 0x0001, mbxsts 0x4000, 1g 0x0400, 2g 0x1D00, 4g 0x2500 PGQBT-I-LINK_WAIT, waiting for link to come up PGQBT-I-TOPO_WAIT, waiting for topology ID SYSBOOT> show CLUSTER/EXPECTED_VOTES Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- CLUSTER_CREDITS 128 32 10 128 Credits SYSBOOT> SHOW VOTES Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- VOTES 1 1 0 127 Votes SYSBOOT> SHOW EXPECTED Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- EXPECTED_VOTES 3 1 1 127 Votes SYSBOOT> SET EXPECTED_VOTES 1 SYSBOOT> SHOW EXPECTED Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- EXPECTED_VOTES 1 1 1 127 Votes SYSBOOT> cont

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 10:16:55 GMT

Lucas,

why EXPECTED_VOTES = 3 ? Do you have a QUORUM disk (parameter DISK_QUORUM and QDSKVOTES) ?

If so, maybe that QUORUM disk is not working ?

A 2-node cluster with a quorum disk should have VOTES=1 for both nodes and QDSKVOTES=1 for the quorum disk, this results in EXPECTED_VOTES=3 and a quorum of 2. So just one node AND the qorum disk should allow a boot !

Please check your quorum disk

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Tue, 04 Oct 2022 14:11:26 GMT

Hi Volker,

during boot I can see:

Quorum: 1 (of 1 vote); this node contributes 1 vote
Cluster has no voting quorum disk.

So maybe it was a wrong settings. These servers had an uptime of years, so I don't know, maybe the wrong settings are there from years.

Just a question, do you see any downside with the configuration I did also for the other node?

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Tue, 04 Oct 2022 14:39:11 GMT

Lucas,

from an availability and redudancy perspective, a 2-node cluster without a quorum disk is not optimal - as you've learned

If you do not want to configure a quorum disk, both nodes should have VOTES=1 and EXPECTED_VOTES=2, so either both of them are running or none of them. This prevents cluster partitioning - assuming you are using shared disks, which is most likely for SAN storage config.

If you want higher redundancy, you should configure a quorum disk using system parameters DISK_QUORUM="$1$DGAx:" and QDSKVOTES=1, then increase EXPECTED_VOTES from 2 to 3. This will allow either 2 nodes without access to the quorum disk or one node AND the quorum disk to be able to boot or continue to run, if one of the nodes or the quorum disk fails.

The addition of a quorum disk will make cluster state transitions take a little bit longer, but wil provide higher availability of the cluster.

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Wed, 05 Oct 2022 05:30:18 GMT

Thank you very much for explanation. I have other two clusters and I checked the quorum/votes config. Indeed there are 3 votes expected, but there is also a QUORUM disk configured who corresponds to the system disk $1$DGA100:

SYSGEN> SHOW DISK_QUORUM Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- DISK_QUORUM "$1$DGA100 " " " " " "ZZZZ" Ascii SYSGEN> SHOW EXPECTED Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- EXPECTED_VOTES 3 1 1 127 Votes SYSGEN> SHO QDSKVOTES Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- QDSKVOTES 1 1 0 127 Votes

While on the system that did not want to boot, there is no quorum.

SYSGEN> SHOW DISK_QUORUM Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- DISK_QUORUM " " " " " " "ZZZZ" Ascii SYSGEN> SHO QDSKVOTES Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- QDSKVOTES 1 1 0 127 Votes

I don't know if this setting was lost after shutdown or never configured, these clusters were installed 15 years ago, and in their life probably have been turned off one or two times and most probably never together.

So, what I have to do is configure the quorum disk as per other cluster:

SYSGEN> set disk_quorum "$1$DGA100:"

And then restore the votes:

SYSGEN> SET CLUSTER/EXPECTED_VOTES=3

Are there any other settings to check? Can I do this mod on already booted system and the changes will be permanent?

As always, thank you Volker, in the last 15 years you saved me a lot of time

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Wed, 05 Oct 2022 05:03:12 GMT

Lucas,

the command to set EXPECTED_VOTES in SYSGEN is SET EXPECTED_VOTES 3, not SET CLUSTER/EXPECTED_VOTES 3 !

One would normally add these commands to MODPARAMS.DAT and then run AUTOGEN, but this may be too risky on a 15-year old cluster, because other parameters may have been manually changed in the past as well, without adding the changes to MODPARAMS.DAT

Do not forget SYSGEN> WRITE CURRENT after changing the parameters.

You can change these parameters in the running system. But as there is no quorum disk yet, the quorum file ( $1$DGA100:[000000]QUORUM.DAT ) needs to be created and this does only happen in a running cluster, so you need to reboot one of the nodes, while the other one is up, the 2 nodes will then form a cluster and create the quorum disk file.

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Wed, 05 Oct 2022 05:31:40 GMT

**bleep**! I don't know why it seems it's not permanent

SYSGEN> SET DISK_QUORUM "$1$DGA100" SYSGEN> WRITE CURRENT SYSGEN> SHO DISK_QUORUM Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- DISK_QUORUM "$1$DGA100 " " " " " "ZZZZ" Ascii SYSGEN> EXIT > mc sysgen SYSGEN> SHOW DISK_QUORUM Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- DISK_QUORUM " " " " " " "ZZZZ" Ascii

Re: OpenVMS Blade lost boot entry after shutdown

Volker Halle — Wed, 05 Oct 2022 05:42:02 GMT

Lucas,

when invoking SYSGEN, the parameters shown are from the running system (implied USE ACTIVE).

When you want to see (and modify) the on-disk parameters, which will be used during the next boot, you'll have to specify:

SYSGEN> USE CURRENT

then modify the parameters and then use SYSGEN> WRITE CURRENT

Volker.

Re: OpenVMS Blade lost boot entry after shutdown

smsc_1 — Wed, 05 Oct 2022 06:06:19 GMT

Ok good, I'll try it, but first one question. I currently have the second node down. The system board sent by HP was DOA, so need to be replaced again.

What happen if I'll change the QUORUM_DISK and EXPECTED_VOTES on the only running system? Considering that there is no $1$DGA100:[000000]QUORUM.DAT file on it? I'm worried about it, if for example I reboot again the current node, maybe it doesnt start anymore due to missing QUORUM.DAT?

Is it better to wait the second node is up and then change parameters (probably on both nodes) and then try to reboot the second node to generate the quorum file, then try again to reboot the whole cluster?