OpenVMS Blade lost boot entry after shutdown

[smsc_1]

Hello community, I'm in trouble with on blade BL860C with OpenVMS 8.4.

This blade is connected on EVA storage and OpenVMS is on storage. EVA storage was in fault so I had to shutdown the Blade with OpenVMS. Now, with HP support, EVA is ok, but OpenVMS doesn't boot anymore.

I tried to enter in EFI shell and boot manually, it seems I identified the correct OS disk:

Problem is that it hangs to the following boot part and there is no way to continue boot process:

Previously it works perfect. Ant clue?

Thank you

Lucas

[Volker Halle]

Lucas,

I'd guess, that it can't find/mount the system disk.

Boot from the Openrating System DVD and see, if you can find and mount the system disk.

Then use @sys$MANAGER:BOOT_OPTIONS to re-configure the boot settings.

Volker.

[smsc_1]

Thank you for reply Volker, I have an internal blade disk where I can boot and mount the system disk without any issue, plus I can see the other disks presented by the storage, is there anything I can do from there?

EDIT - Here is the output of the command you gave me

$$$ @sys$MANAGER:BOOT_OPTIONS


    OpenVMS I64 Boot Manager Boot Options List Management Utility

    (1) ADD an entry to the Boot Options list
    (2) DISPLAY the Boot Options list
    (3) REMOVE an entry from the Boot Options list
    (4) MOVE the position of an entry in the Boot Options list
    (5) VALIDATE boot options and fix them as necessary
    (6) Modify Boot Options TIMEOUT setting


    (B)  Set to operate on the Boot Device Options list
    (D)  Set to operate on the Dump Device Options list
    (G)  Set to operate on the Debug Device Options list

    (E) EXIT from Boot Manager utility 

    You can also enter Ctrl-Y at any time to abort this utility.

Enter your choice: 2

To display all entries in the Boot Options list, press Return.
To display specific entries, enter the entry number or device name.
(Enter "?" for a list of devices): 


EFI Boot Options list:   Timeout = 10 secs.

Entry  Description                                                Options
-----  ---------------------------------------------------------- -------------
   1   OpenVMS IA64  FGB0.5001-4380-04c6-6559                     -l00 
         $1$DGA100 PCI(0|4|0|0) Fibre(5001438004C66558,Lun64000000000000) 
   2   EFI Shell [Built-in]                                       
         VenHw(d65a6b8c-71e5-4df0-d2f009a9) 
   3   OpenVMS IA64 FGB0.5001-4380-04c6-655D                      -l000 
         $1$DGA100 PCI(0|4|0|0) Fibre(5001438004C6655C,Lun64000000000000) 
--------------------------------------------------------------------------------
3 entries found. 

[Volker Halle]

Lucas,

try $ ANALYZE/DISK $1$DGA100:

Then @sys$MANAGER:BOOT_OPTIONS to define a new boot for this disk

Then try a full verbose boot with VMS_LOADER -fl 0,30000

Volker.

[smsc_1]

This is what I did, I deleted the old boot entries and then add a new one from DGA100:

Before try to reboot, may I have to do something else?

    OpenVMS I64 Boot Manager Boot Options List Management Utility

    (1) ADD an entry to the Boot Options list
    (2) DISPLAY the Boot Options list
    (3) REMOVE an entry from the Boot Options list
    (4) MOVE the position of an entry in the Boot Options list
    (5) VALIDATE boot options and fix them as necessary
    (6) Modify Boot Options TIMEOUT setting


    (B)  Set to operate on the Boot Device Options list
    (D)  Set to operate on the Dump Device Options list
    (G)  Set to operate on the Debug Device Options list

    (E) EXIT from Boot Manager utility 

    You can also enter Ctrl-Y at any time to abort this utility.

Enter your choice: 1

EFI     Built-in EFI Shell

Enter the device name (Enter "?" for a list of devices): $1$DGA100:

Enter the desired position number (1,2,3,,,) of the entry.
To display the Boot Options list, enter "?" and press Return.
Position [1]: ?


EFI Boot Options list:   Timeout = 10 secs.

Entry  Description                                                Options
-----  ---------------------------------------------------------- -------------
   1   EFI Shell [Built-in]                                       
         VenHw(d65a6b8c-71e5-4df0-d2f009a9) 
--------------------------------------------------------------------------------
1 entries found. 


Enter the desired position number (1,2,3,,,) of the entry.
To display the Boot Options list, enter "?" and press Return.
Position [1]: 1

Enter the value for VMS_FLAGS in the form n,n. 
VMS_FLAGS [NONE]: 0,0

Enter a short description (do not include quotation marks).
Description ["$1$DGA100:"]: OpenVMS IA64  FGB0.5001-4380-04c6-6559

efi$bcfg: $1$dga100: (Boot0002) Option successfully added 

efi$bcfg: $1$dga100: (Boot0003) Option successfully added 

efi$bcfg: $1$dga100: (Boot0004) Option successfully added 

efi$bcfg: $1$dga100: (Boot0005) Option successfully added 

[smsc_1]

Here is the output of: VMS_LOADER -fl 0,30000

https://pastebin.com/aTGfLjht

Honestly I'm stuck like the blade!!!

[Volker Halle]

Lucas,

I see:

%SYSINIT-I- waiting to form or join an OpenVMS Cluster

This Blade is supposed to become a cluster member ? Did you wait long enough for possible 'Have connection to' messages ? 

After the '%STACONFIG-I-CANGO, enabling process deletion'  message you would expect messages like:

%STACONFIG-I-MSCP, configuring MSCP/HSC disks/tapes
%CNXMAN,  Sending VMScluster membership request to system I64VMS
%CNXMAN,  Now a VMScluster member -- system VSIAXP

Volker.

[smsc_1]

Hi Volker, yes, this is a cluster member, but the second node is powered down for hardware issue (system board in fault). Today regarding the hardware is a disaster!!!

Now I did a manual boot from fs1:   VMS_LOADER -fl 0,0 and I'm waiting as per your advice, something is changed since I can see now link up/down, I believe this is the cluster. I'll keep the node in this state, waiting for hardware replacement in other node

%LLB0, Logical LAN failset device connected to physical device EIA0
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down
%EWA0, Link up: 1000 mbit, full duplex, flow control (recv)
%EWA0, Link down

I hope this will solve the issue.

[Volker Halle]

Lucas,

so this node is unable to form a cluster, if the 2nd node is missing. Is this intentional ?

You can always boot this node conversationally ( -fl 0,1 ) and set EXPCTED_VOTES to the same value as VOTES. This will at least allow this node to boot - if it can do any useful work without the other node being present.

If EWA0 is the cluster interconnect - directly cabled to the other node - these messages can be expected, if the other node is down.

Volker.

[smsc_1]

So, you suggest to do:

fs1: VMS_LOADER -fl 0,1

Currently I have this config:

SYSBOOT> show CLUSTER/EXPECTED_VOTES
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
CLUSTER_CREDITS 128 32 10 128 Credits 

[Volker Halle · Accepted solution]

Lucas,

a conversational boot ( -fl 0,1 ) will get you into SYSBOOT

SYSBOOT> SHOW VOTES

SYSBOOT> SHOW EXPECTED

SYSBOOT> SET EXPECTED_VOTES 1       ! if VOTES=1

SYSBOOT> CONT

Good luck,

Volker.

[smsc_1]

THank you for your time Volker, FINALLY THE NODE IS UP !!!!!!!!!!  This is what I did:

fs1:\EFI\VMS> VMS_LOADER -fl 0,1

PGQBT-I-INIT-UNIT, IPB, PCI device ID 0x2432, FW 4.02.02
PGQBT-I-BUILT, version X-33, built on Apr 17 2012 @ 18:29:57
PGQBT-S-SERDES, status 0x0001, mbxsts 0x4000, 1g 0x0400, 2g 0x1D00, 4g 0x2500
PGQBT-I-LINK_WAIT, waiting for link to come up
PGQBT-I-TOPO_WAIT, waiting for topology ID

SYSBOOT> show CLUSTER/EXPECTED_VOTES
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
CLUSTER_CREDITS               128         32        10        128 Credits    

SYSBOOT> SHOW VOTES
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
VOTES                           1          1         0        127 Votes      

SYSBOOT> SHOW EXPECTED
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
EXPECTED_VOTES                  3          1         1        127 Votes      

SYSBOOT> SET EXPECTED_VOTES 1

SYSBOOT> SHOW EXPECTED
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
EXPECTED_VOTES                  1          1         1        127 Votes      

SYSBOOT> cont

[Volker Halle]

Lucas,

why EXPECTED_VOTES = 3 ? Do you have a QUORUM disk (parameter DISK_QUORUM and QDSKVOTES) ? 

If so, maybe that QUORUM disk is not working ?

A 2-node cluster with a quorum disk should have VOTES=1 for both nodes and QDSKVOTES=1 for the quorum disk, this results in EXPECTED_VOTES=3 and a quorum of 2. So just one node AND the qorum disk should allow a boot !

Please check your quorum disk

Volker.

[smsc_1]

Hi Volker,

during boot I can see:

Quorum: 1 (of 1 vote); this node contributes 1 vote
Cluster has no voting quorum disk.

So maybe it was a wrong settings. These servers had an uptime of years, so I don't know, maybe the wrong settings are there from years.

Just a question, do you see any downside with the configuration I did also for the other node?

[Volker Halle]

Lucas,

from an availability and redudancy perspective, a 2-node cluster without a quorum disk is not optimal - as you've learned

If you do not want to configure a quorum disk, both nodes should have VOTES=1 and EXPECTED_VOTES=2, so either both of them are running or none of them. This prevents cluster partitioning - assuming you are using shared disks, which is most likely for  SAN storage config.

If you want higher redundancy, you should configure a quorum disk using system parameters DISK_QUORUM="$1$DGAx:" and QDSKVOTES=1, then increase EXPECTED_VOTES from 2 to 3. This will allow either 2 nodes without access to the quorum disk or one node AND the quorum disk to be able to boot or continue to run, if one of the nodes or the quorum disk fails.

The addition of a quorum disk will make cluster state transitions take a little bit longer, but wil provide higher availability of the cluster.

Volker.

[smsc_1]

Thank you very much for explanation. I have other two clusters and I checked the quorum/votes config. Indeed there are 3 votes expected, but there is also a QUORUM disk configured who corresponds to the system disk $1$DGA100:

SYSGEN> SHOW DISK_QUORUM
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
DISK_QUORUM "$1$DGA100 " " " " " "ZZZZ" Ascii 

SYSGEN>  SHOW EXPECTED
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
EXPECTED_VOTES                  3          1         1        127 Votes      

SYSGEN>  SHO QDSKVOTES
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
QDSKVOTES                       1          1         0        127 Votes      

While on the system that did not want to boot, there is no quorum.

SYSGEN>  SHOW DISK_QUORUM
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "                "    "    "    "    "     "ZZZZ" Ascii    

SYSGEN>  SHO QDSKVOTES
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
QDSKVOTES                       1          1         0        127 Votes

I don't know if this setting was lost after shutdown or never configured, these clusters were installed 15 years ago, and in their life probably have been turned off one or two times and most probably never together.

So, what I have to do is configure the quorum disk as per other cluster:

SYSGEN> set disk_quorum "$1$DGA100:"

And then restore the votes:

SYSGEN>  SET CLUSTER/EXPECTED_VOTES=3

Are there any other settings to check? Can I do this mod on already booted system and the changes will be permanent?

As always, thank you Volker, in the last 15 years you saved me a lot of time

[Volker Halle]

Lucas,

the command to set EXPECTED_VOTES in SYSGEN is SET EXPECTED_VOTES 3, not SET CLUSTER/EXPECTED_VOTES 3 !

One would normally add these commands to MODPARAMS.DAT and then run AUTOGEN, but this may be too risky on a 15-year old cluster, because other parameters may have been manually changed in the past as well, without adding the changes to MODPARAMS.DAT

Do not forget SYSGEN> WRITE CURRENT after changing the parameters.

You can change these parameters in the running system. But as there is no quorum disk yet, the quorum file ( $1$DGA100:[000000]QUORUM.DAT ) needs to be created and this does only happen in a running cluster, so you need to reboot one of the nodes, while the other one is up, the 2 nodes will then form a cluster and create the quorum disk file.

Volker.

[smsc_1]

**bleep**! I don't know why it seems it's not permanent

SYSGEN>  SET DISK_QUORUM "$1$DGA100"
SYSGEN>  WRITE CURRENT
SYSGEN>  SHO DISK_QUORUM
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "$1$DGA100       "    "    "    "    "     "ZZZZ" Ascii       
SYSGEN>  EXIT

> mc sysgen 
SYSGEN>  SHOW DISK_QUORUM
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "                "    "    "    "    "     "ZZZZ" Ascii      

[Volker Halle]

Lucas,

when invoking SYSGEN, the parameters shown are from the running system (implied USE ACTIVE).

When you want to see (and modify) the on-disk parameters, which will be used during the next boot, you'll have to specify:

SYSGEN> USE CURRENT

then modify the parameters and then use SYSGEN> WRITE CURRENT

Volker.

[smsc_1]

Ok good, I'll try it, but first one question. I currently have the second node down. The system board sent by HP was DOA, so need to be replaced again.

What happen if I'll change the QUORUM_DISK and EXPECTED_VOTES on the only running system? Considering that there is no $1$DGA100:[000000]QUORUM.DAT file on it? I'm worried about it, if for example I reboot again the current node, maybe it doesnt start anymore due to missing QUORUM.DAT?

Is it better to wait the second node is up and then change parameters (probably on both nodes) and then try to reboot the second node to generate the quorum file, then try again to reboot the whole cluster?

[Volker Halle]

Lucas,

your question shows me, that you not only read my responses but also try to understand them

Yes, you're right. Adding the quorum disk while only one node is up and running is risky. It won't reboot, if the QUORUM.DAT file has not yet been created. One could overcome this problem by temporarily dropping EXPECTED_VOTES to 1 during conversational boot. Better just wait until the 2nd node works again and then add the quorum disk on both nodes.

Volker.

[smsc_1]

No way...

SYSGEN>  USE CURRENT
SYSGEN>  SET DISK_QUORUM "$1$DGA100"
SYSGEN>  WRITE CURRENT
SYSGEN>  SHO DISK_QUORUM
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "$1$DGA100       "    "    "    "    "     "ZZZZ" Ascii       

SYSGEN>  EXIT

> MC SYSGEN
SYSGEN>  SHO DISK_QUORUM
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "                "    "    "    "    "     "ZZZZ" Ascii       

Maybe I can try VMS_LOADER -fl 0,1 from SYSBOOT prompt? It seems that in this way I'll be able to keep configuration since I changed the EXPECTED_VOTES from 3 to 1 and after reboot I always have 1.

Second node is still down, so I have to wait for it anyway

[Volker Halle]

Lucas,

if you invoke SYSGEN, it's using an implied 'USE ACTIVE' command, i.e. it shows you the ACTIVE system parameters. If you would issue a USE CURRENT, it will show you the system parameters stored in the system parameter file on disk, which would be used during the next boot.

Volker.

[smsc_1]

Sorry Volker, I don't understand.

What are the correct commands I should use to setup QUORUM_DISK and EXPECTED_VOTES directly on already started OS and keep them also after reboot? Thank you.

EDIT: aaaah wait understood now:

SYSGEN>  USE CURRENT
SYSGEN>  SHO DISK 
Parameter Name            Current    Default     Min.       Max.   Unit  Dynamic
--------------            -------    -------   -------    -------  ----  -------
DISK_QUORUM     "$1$DGA100       "    "    "    "    "     "ZZZZ" Ascii    

[Volker Halle]

Lucas,

DISK_QUORUM and EXPECTED_VOTES are NOT dynamic system parameters - no 'D' in last column (Dynamic) shown for these parameters in SYSGEN SHOW xxx

So you can only write the new parameters to the system parameter file SYS$SYSTEM:IA64VMSSYS.PAR using SYSGEN WRITE CURRENT.

So the correct commands to be used are:

$ MC SYSGEN

SYSGEN> USE CURRENT

SYSGEN> SET DISK_QUORUM "$1$DGA100:"

SYSGEN> SET EXPECTED_VOTES 3

SYSGEN> WRITE CURRENT

SYSGEN> EXIT

If you want to boot a single node with these new parameters, while the other cluster-node is not up and running, you must use a conversational boot "-fl 0,1" and SYSBOOT> SET EXPECTED_VOTES 1. This will cause the single node to form a cluster without the votes of the quorum disk. Once running as a cluster node, it will then create the QUORUM file (the quorum disk must be mounted !) and undergo a cluster state transition to add the quorum disk (see messages on console). Once that has happened, raise EXPECTED_VOTES to 3 again (on the disk) using above commands.

Volker.