OpenVMS network down after reboot

[smsc_1]

Dear community I'm in trouble with OpenVMS V8.4 server.

Due to memory fault I needed to switch off the server so that can HP can replace tha faulty DIMM, now the server bootup correctly with 32GB but the problem is that it's now completly isolated from the newtork. I can reach it only via iLO/MP. No configuration changes was applied before and after DIMM replacement and the server was rebooted 4 month ago and comeup without issue.

HP says that there is no network card fault, but I'm not properly sure, since, as I wrote, no configuration changes are applied and after reboot everything are there (like routes, IP assignment, etc.).

From the HP switch I can see that related port and VLAN are correctly UP:

Vlan4 UP UP 10.97.13.140 MESSAGING
Vlan7 UP UP 10.97.13.154 OAM

GE1/0/6 UP 1G F A 4 NODE82-S6-P3
GE1/0/7 UP 1G F A 7 NODE82-S6-P4

GE2/0/6 UP 1G F A 4 NODE82-S5-P3
GE2/0/7 UP 1G F A 7 NODE82-S5-P4

Network interfaces on OpenVMS shown no traffic but the strange thing is that there is no mac address shown while on other node of the cluster, currently working, I can see the mac address

Interface: LE0
IP_Addr: 10.97.13.146 NETWRK: 255.255.255.240 BRDCST: 10.97.13.159
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0


Interface: LE1
IP_Addr: 10.97.13.130 NETWRK: 255.255.255.240 BRDCST: 10.97.13.143
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0

Could someone help me to understand what's wrong? And if there are any other commands I can do on OpenVMS to show me better what's the interfaces situation? I'm not OpenVMS expert and I'm stuck.

Thank you very much,

Lucas

[Volker Halle]

Lucas,

the IP interface flags should be: UP BRDCST RUN MCAST SMPX

So the RUN bit is missing in your case.

Check SYS$MANAGER:OPERATOR.LOG for error messages from TCPIP startup.

Volker.

[smsc_1]

Thank you very much Volker, what does it mean or what could be the problem that can cause flag missing? Hardware? Software? What do you think?

Anyway, here is the operator.log, thanks:

https://pastebin.com/sELZCLbt

EDIT: Adding more information about interfaces:

From TPCIP$CONFIG I can see:

1 - LE0 Menu (LLA0: Unspecified 10mbps)
2 - 10.97.13.146/28 NODE82 Configured,Active

3 - LE1 Menu (LLB0: Unspecified 10mbps)
4 - 10.97.13.130/28 NODE82 Configured,Active

The related devices LLA0 and LLB0 are currently DOWN:

LANCP> SHO DEV LLA0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CD Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CD MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EID0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state

LANCP> SHO DEV LLB0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CC Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CC MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EIC0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state

[Volker Halle]

Lucas,

you're running some IP interfaces on LAN Failover Devices (LE0 and LE1, these match the OpenVMS devices LLA0 and LLB0). With LANCP> SHOW DEV/FULL LLx, you should be able to find out, which physical LAN interfaces are used by these LAN Failover Devices.

You might need to look at SYS$MANAGER:LAN$ACP.LOG and LANCP> SHOW DEVICE/INTERNAL LLAx to see the LAN Driver messages from early startup.

Volker.

[smsc_1]

Hi Volker,

I did the command you suggest me, but I cannot fin anything interesting, or at least something not clear for me

I can see: Logical LAN failover device added to failset, EIC0 for LLA0 and EID0 for LLB0.

But these devices (EIC0 and EID0) are both down.

What does it mean? That LE0 have 2 interfaces (EIC0 and LLA0) like bond in linux?

What is not clear for me is that the configuration works perfect before RAM failure, and noone changes it before or after RAM issue.

SYSTEM> SHOW DEV/FULL LLA0
Device LLA0:, device type LLAN, is online, network device, device is a template
only.

Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512

Operating characteristics: Link down.

Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CD Current MAC addr AA-00-00-07-F4-CD

SYSTEM> SHOW DEV/FULL LLB0

Device LLB0:, device type LLAN, is online, network device, device is a template
only.

Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512

Operating characteristics: Link down.

Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CC Current MAC addr AA-00-00-07-F4-CC

LANCP> SHOW DEVICE/INTERNAL LLA0

SMWI82 Device Internal Counters LLA0 (9-SEP-2021 08:50:22.03):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLA" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EID LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:22.86 Logical LAN failset device created

LANCP> SHOW DEVICE/INTERNAL LLB0

SMWI82 Device Internal Counters LLB0 (9-SEP-2021 08:50:32.53):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLB" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EIC LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failset device created

SYSTEM> ty SYS$MANAGER:LAN$ACP.LOG
9-SEP-2021 07:49:48.35 Defined LAN$DLL to be SYS$SYSROOT:[MOM$SYSTEM]
9-SEP-2021 07:49:48.36 Found LAN device EIA0, hardware address 00-1E-0B-88-16-76
9-SEP-2021 07:49:48.36 Found LAN device EIB0, hardware address 00-1E-0B-88-16-77
9-SEP-2021 07:49:48.36 Found LAN device EIC0, hardware address 00-1A-4B-07-F4-CC
9-SEP-2021 07:49:48.36 Found LAN device EID0, hardware address 00-1A-4B-07-F4-CD
9-SEP-2021 07:49:48.36 Found LAN device EIE0, hardware address 00-1E-0B-88-21-2C
9-SEP-2021 07:49:48.36 Found LAN device EIF0, hardware address 00-1E-0B-88-21-2D
9-SEP-2021 07:49:48.37 Found LAN device EIG0, hardware address 00-1E-0B-88-21-2E
9-SEP-2021 07:49:48.37 Found LAN device EIH0, hardware address 00-1E-0B-88-21-2F
9-SEP-2021 07:49:48.37 Found LAN device LLA0, hardware address AA-00-00-07-F4-CD
9-SEP-2021 07:49:48.37 Found LAN device LLB0, hardware address AA-00-00-07-F4-CC
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failset device created
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failset device created
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %EIA0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIB0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIE0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIF0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIG0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 %EIH0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 Node database file, LAN$NODE_DATABASE, not found
9-SEP-2021 07:49:48.40 LANACP initialization complete

[Volker Halle]

Lucas,

for LAN Failover, you configure (typically) more than ONE physical interface into a LAN Failover set. If one of those physical interfaces is up, so is the LAN Failover interface. 

EIC0 and EID0 did NOT report "Link up" so they are either defect or the cable is unplugged or the switch port has a problem !

And if those LAN interfaces are the ONLY LAN interface configured into LLA0 resp. LLB0, it's clear why the LAN Failover devices are also down.

Did the LAN cables get removed during the repair action ? If so, have they been correctly plugged in again ? Your system has 8 LAN interfaces, are the cables and interfaces correctly documented ?

Volker.

[smsc_1]

Cables should be ok, I ask HP engineer to double check again.

What I understand from your explanation is that currently I have 2 LOGICAL interfaces LE0 and LE1, each interfaces point to 2 physical devices, eg:

LE0 = LLA0 + EID0

LE1 = LLB0 + EIC0

Is this correct?

These 4 physical devices anyway are all down, that's why LE0 and LE1 are down too.

mc lancp show dev LLA0/char
Link Down Link state
mc lancp show dev LLB0/char
Link Down Link state
mc lancp show dev EIC0/char
Link Down Link state
mc lancp show dev EID0/char
Link Down Link state

---

---

So, if the cables is ok, switch is also ok, what is not clear for me is if the problem is on OpenVMS or I have to check some server hardware part like network interfaces.

Thank you for your time Volker

[Volker Halle]

Lucas,

the LAN Failover Devices LE0 = LLA0 (consisting of just EID0) and LE1 = LLB0 (just EIC0) are down, because the only PHYSICAL LAN interfaces in both of these LAN failover sets did not report 'Link up'. You would normally configure LAN Failover sets with more than ONE LAN interface...

You have another cluster member up and running ? Same type of system ? Check the LAN interfaces and cabling on that one and compare to non-working system. Also look at the LEDs on those 8 physical LAN interfaces on your system.

Volker.

[smsc_1]

Unfortunately I'm not onsite (the server is on other city), anyway, sorry, still not clear the actual configuration and how can I check it.

Taking as example LE0 and basing on log, as I understood:

LE0 have 2 PHYSICAL device (means two cables): LLA0 + EID0

Am I wrong?

[Volker Halle]

Lucas,

a LAN Failover Device (e.g. LLA0 in OpenVMS, named LE0 in TCPIP) is a 'virtual LAN device'. It consists of one (or more) PHYSICAL LAN interfaces. If any of those PHYSICAL LAN interfaces is up (Link Up), the LAN traffic is sent/recevied via that PHYSICAL LAN interface. If that interface/link fails, the 'other' LAN interface will be selected for send/receive. If you only have ONE PHYSICAL interface in a LAN Failover set and that interface is 'Link down', so is you LAN Failover device.

Volker.

[smsc_1]

a LAN Failover Device (e.g. LLA0 in OpenVMS, named LE0 in TCPIP) is a 'virtual LAN device'. It consists of one (or more) PHYSICAL LAN interfaces. If any of those PHYSICAL LAN interfaces is up (Link Up), the LAN traffic is sent/recevied via that PHYSICAL LAN interface. If that interface/link fails, the 'other' LAN interface will be selected for send/receive. If you only have ONE PHYSICAL interface in a LAN Failover set and that interface is 'Link down', so is you LAN Failover device.

Ok, so:

LE0/LLA0 ==> EID0 (PHYSICAL LAN interfaces)

So currently I have only one PHYSICAL device associated to logical device.

Because EID0 is down, also LE0 is down and the node is not reachable.

Am I right now?

Still don't understand if the issue is related to OpenVMS or hardware like network card. I bet hardware because, as I wrote, no one change the confgiuration before or after HP intervention.

[Volker Halle]

Lucas,

you've got it !

If you have another working cluster member with the same LAN config, compare LANCP LIST DEV/CHAR from both systems.

How many network cables are plugged into those systems ? 6 or 8 ? You have 8 LAN interfaces, 6 have reported 'Link up'. Are 2 LAN interface without a cable ? If not, you have to check and follow those cables, for the 2 interfaces, which do NOT have the 'Link' LED on.

If you're not onsite, have someone send you pictures from the back of those systems...

Volker.

[smsc_1]

Hi Volker, thank you again for explanation.

On both nodes I have the following failover (the first node is working perfect and have exactly the same output shown).

I don't understand what it means. As I understand LLA0 have the following faolover devices:

"EID0" Failover device
"EIJ0" Failover device

But EIJ0 doesn't exists, so it means that the network on OpenVMS is misconfigured, because, as per in this case, if EID0 fails, the whole connection goes down since there is no other device active.

NODE82 Device Characteristics, permanent database, EIA0 (9-SEP-2021 12:58:51.28):

NODE82 Device Characteristics, permanent database, EIB0 (9-SEP-2021 12:58:51.28):

NODE82 Device Characteristics, permanent database, EIC0 (9-SEP-2021 12:58:51.28):
Value Characteristic
----- --------------
20 Failover priority

NODE82 Device Characteristics, permanent database, EID0 (9-SEP-2021 12:58:51.29):
Value Characteristic
----- --------------
20 Failover priority

NODE82 Device Characteristics, permanent database, EIE0 (9-SEP-2021 12:58:51.29):

NODE82 Device Characteristics, permanent database, EIF0 (9-SEP-2021 12:58:51.29):

NODE82 Device Characteristics, permanent database, EIG0 (9-SEP-2021 12:58:51.29):

NODE82 Device Characteristics, permanent database, EIH0 (9-SEP-2021 12:58:51.30):

NODE82 Device Characteristics, permanent database, EII0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
10 Failover priority

NODE82 Device Characteristics, permanent database, EIJ0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
10 Failover priority

NODE82 Device Characteristics, permanent database, LLA0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
Yes Logical LAN enabled for startup
"EID0" Failover device
"EIJ0" Failover device

NODE82 Device Characteristics, permanent database, LLB0 (9-SEP-2021 12:58:51.31):
Value Characteristic
----- --------------
Yes Logical LAN enabled for startup
"EIC0" Failover device
"EII0" Failover device

[Volker Halle]

Lucas,

check the system and network documentation !!!

If you believe the stored LAN Failover config is correct, you may also be missing/have lost a complete LAN card (with 2 ports, i.e. EII and EIJ) !

Count the LAN cables and look at the pictures of the LAN interface ports of BOTH systems.

Volker.

[smsc_1]

You're right Volker, EII and EIJ are missing from second node (compared with the first one) due to hardware fault. I miss that information while I compared both cluster nodes information.

And finally HP was convinced  that there was HW issue. After changing the faulty ethernet adapter, everything work as expected now. The node is currently reachable.

Anyway, thank you Volker for your time, thank to the log I sent to HP, using your comands, we are able to address the issue.

Now I have another issue to address on this system

On both cluster nodes I have the following:

---

SYSTEM> SHOW DEVICES/REBUILD_STATUS
Device Name Rebuild needed?
NODE82$DKA100: Yes
$1$DGA100: Yes
$1$DGA200: Yes
$1$DGA300: Yes
$1$DGA301: Yes
$1$DGA302: Yes
$1$DGA303: Yes
$1$DGA400: Yes
$1$DGA401: Yes
$1$DGA500: Yes
$1$DGA501: Yes
$1$DGA502: Yes
$1$DGA503: Yes
$1$DGA600: Yes

Rebuild needed on all volumes (they are volumes connected via 3par). Honestly I don't know what kind of rebuil it needs to do. e.g. this is an internal server disk: NODE82$DKA100, moreover not used in OpenVMS startup operation (we used it just as application log repository)

During boot, OpenVMS gives me an error like "cannot rebuild due to missing space", but there is sufficient space on all volumes.

**bleep**! Never ending story!

[Volker Halle]

Lucas,

please open a new topic for the 'rebuild needed' problem and then we'll see...

Please report the exact error message seen during OpenVMS boot when mounting the disks.

Volker.

[smsc_1]

Thank you Volker, solved using this documentation without wasting other forum space

https://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/6015/6017p018.htm

[Volker Halle]

Lucas,

now that you've learned something about LAN Failover, it's time to review your system network configuration - as a first step, compare the setup with the other cluster member.

You have 2 LAN Failover devices (LLA consisting of EID,EIJ and LLB consisting of EIC,EII). By loosing EIJ and EII through a LAN card failure, both of your LAN Failover devices failed. And that's because EID and EIC are not correctly connected to your network switches ! With your 10 LAN ports, you should have 10 network cables connected to this system.

LAN Failover is used to increase redundancy of the LAN connection against network component failures. So it is correct, to distribute the members of a LAN Failover set across different LAN interface cards - as you've now learned. But you also need to make sure, that the 'other' members of each LAN Failover set correctly works. After you've made sure, that the 'other' LAN interfaces (EID and EIC) have a network 'Link up', you can use LANCP> SET DEVICE/SWITCH LLAx to test, whether the 'other' member in the LAN Failover are functioning correctly.

And for the future: regularily monitor all members of LAN Failover sets, that they all have 'Link up'. 

Volker.

[smsc_1]

Sure, this will be my next step, and I'm already working on it.

According to HP engineer, who was onsite, there are no cables on EID and EIC at all. That's why, with other failover member down, due to hardware failure, the connectivity gone down too. This is clearly a design error, even because services on these servers are critical.

Maybe it's time to travel for me, even if I don't known if my company allow me to travel for the COVID situation.

[Volker Halle]

Lucas,

regarding REBUILD: you've probably received - and misinterpreted - the following messages during MOUNT:

%MOUNT-I-REBLDREQD, rebuild not performed; some free space unavailable; diskquota usage stale

When a disk is not cleanly dismounted before a node shuts down (e.g. through a system crash), some blocks may still be marked as used in the volume bitmap and therefore not available for allocation. A SET VOLUME/REBUILD resolves this.

Volker.

[smsc_1]

Yes, this is the error I got.

For now I tested the rebuild with internal disk DKA100:. I did the rebuild command, unomunt and mount again and now the flag "Need Rebuild" for this disk is set as NO.

Still not clear for me if I have to do SET VOLUME/REBUILD on each other volumes and then reboot the server.

[Volker Halle]

Lucas,

$ SET VOL/REBUILD - on every disk needing rebuild - is enough, no re-mount, no reboot required.

Volker.

[smsc_1]

Got it, REBUILD NEEDED is now set as NO:

$1$DGA100: No
$1$DGA200: No
$1$DGA300: No
$1$DGA301: No
$1$DGA302: No
$1$DGA303: No
$1$DGA400: No
$1$DGA401: No
$1$DGA500: No
$1$DGA501: No
$1$DGA502: No
$1$DGA503: No
$1$DGA600: No

Thank you

[smsc_1]

In the mean time I got other CRC errors on this server... As already wrote, the neverending story!!!!

Errors are correctly handled by ECC DIMMs, but I opened a new call to HP to investigate. These servers are like tanks, no one can stop them, but when they got one hardware issue, you'll get into an infinite loop!!!

92 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
91 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
90 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
89 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
88 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
87 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
86 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR
85 SFW 0 1 0x24001C7D00E00000 FFFFFFFF002BFF74 MEM_SBE_CORR_ERR

[Volker Halle]

Lucas,

now this IS a HARDWARE error - right ?

Maybe HP or HPE or OpenVMS does not have/include the correct tools (or knowledge ?) to diagnose these SBEs and pin-point the correct DIMM(s) to be replaced ?

Sorry, but I can't help you with this...

Volker.