topic Re: OpenVMS network down after reboot in Operating System - OpenVMS

OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 04:43:43 GMT

Dear community I'm in trouble with OpenVMS V8.4 server.

Due to memory fault I needed to switch off the server so that can HP can replace tha faulty DIMM, now the server bootup correctly with 32GB but the problem is that it's now completly isolated from the newtork. I can reach it only via iLO/MP. No configuration changes was applied before and after DIMM replacement and the server was rebooted 4 month ago and comeup without issue.

HP says that there is no network card fault, but I'm not properly sure, since, as I wrote, no configuration changes are applied and after reboot everything are there (like routes, IP assignment, etc.).

From the HP switch I can see that related port and VLAN are correctly UP:

Vlan4 UP UP 10.97.13.140 MESSAGING
Vlan7 UP UP 10.97.13.154 OAM
GE1/0/6 UP 1G F A 4 NODE82-S6-P3
GE1/0/7 UP 1G F A 7 NODE82-S6-P4
GE2/0/6 UP 1G F A 4 NODE82-S5-P3
GE2/0/7 UP 1G F A 7 NODE82-S5-P4

Network interfaces on OpenVMS shown no traffic but the strange thing is that there is no mac address shown while on other node of the cluster, currently working, I can see the mac address

Interface: LE0
IP_Addr: 10.97.13.146 NETWRK: 255.255.255.240 BRDCST: 10.97.13.159
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0

Interface: LE1
IP_Addr: 10.97.13.130 NETWRK: 255.255.255.240 BRDCST: 10.97.13.143
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0

Could someone help me to understand what's wrong? And if there are any other commands I can do on OpenVMS to show me better what's the interfaces situation? I'm not OpenVMS expert and I'm stuck.

Thank you very much,

Lucas

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 05:41:41 GMT

Lucas,

the IP interface flags should be: UP BRDCST RUN MCAST SMPX

So the RUN bit is missing in your case.

Check SYS$MANAGER:OPERATOR.LOG for error messages from TCPIP startup.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 06:26:14 GMT

Thank you very much Volker, what does it mean or what could be the problem that can cause flag missing? Hardware? Software? What do you think?

Anyway, here is the operator.log, thanks:

https://pastebin.com/sELZCLbt

EDIT: Adding more information about interfaces:

From TPCIP$CONFIG I can see:

1 - LE0 Menu (LLA0: Unspecified 10mbps)
2 - 10.97.13.146/28 NODE82 Configured,Active
3 - LE1 Menu (LLB0: Unspecified 10mbps)
4 - 10.97.13.130/28 NODE82 Configured,Active

The related devices LLA0 and LLB0 are currently DOWN:

LANCP> SHO DEV LLA0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CD Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CD MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EID0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state
LANCP> SHO DEV LLB0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CC Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CC MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EIC0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 06:41:39 GMT

Lucas,

you're running some IP interfaces on LAN Failover Devices (LE0 and LE1, these match the OpenVMS devices LLA0 and LLB0). With LANCP> SHOW DEV/FULL LLx, you should be able to find out, which physical LAN interfaces are used by these LAN Failover Devices.

You might need to look at SYS$MANAGER:LAN$ACP.LOG and LANCP> SHOW DEVICE/INTERNAL LLAx to see the LAN Driver messages from early startup.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 07:00:00 GMT

Hi Volker,

I did the command you suggest me, but I cannot fin anything interesting, or at least something not clear for me

I can see: Logical LAN failover device added to failset, EIC0 for LLA0 and EID0 for LLB0.

But these devices (EIC0 and EID0) are both down.

What does it mean? That LE0 have 2 interfaces (EIC0 and LLA0) like bond in linux?

What is not clear for me is that the configuration works perfect before RAM failure, and noone changes it before or after RAM issue.

SYSTEM> SHOW DEV/FULL LLA0
Device LLA0:, device type LLAN, is online, network device, device is a template
only.
Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512
Operating characteristics: Link down.
Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CD Current MAC addr AA-00-00-07-F4-CD

SYSTEM> SHOW DEV/FULL LLB0
Device LLB0:, device type LLAN, is online, network device, device is a template
only.
Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512
Operating characteristics: Link down.
Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CC Current MAC addr AA-00-00-07-F4-CC

LANCP> SHOW DEVICE/INTERNAL LLA0
SMWI82 Device Internal Counters LLA0 (9-SEP-2021 08:50:22.03):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLA" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EID LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:22.86 Logical LAN failset device created
LANCP> SHOW DEVICE/INTERNAL LLB0
SMWI82 Device Internal Counters LLB0 (9-SEP-2021 08:50:32.53):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLB" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EIC LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failset device created
SYSTEM> ty SYS$MANAGER:LAN$ACP.LOG
9-SEP-2021 07:49:48.35 Defined LAN$DLL to be SYS$SYSROOT:[MOM$SYSTEM]
9-SEP-2021 07:49:48.36 Found LAN device EIA0, hardware address 00-1E-0B-88-16-76
9-SEP-2021 07:49:48.36 Found LAN device EIB0, hardware address 00-1E-0B-88-16-77
9-SEP-2021 07:49:48.36 Found LAN device EIC0, hardware address 00-1A-4B-07-F4-CC
9-SEP-2021 07:49:48.36 Found LAN device EID0, hardware address 00-1A-4B-07-F4-CD
9-SEP-2021 07:49:48.36 Found LAN device EIE0, hardware address 00-1E-0B-88-21-2C
9-SEP-2021 07:49:48.36 Found LAN device EIF0, hardware address 00-1E-0B-88-21-2D
9-SEP-2021 07:49:48.37 Found LAN device EIG0, hardware address 00-1E-0B-88-21-2E
9-SEP-2021 07:49:48.37 Found LAN device EIH0, hardware address 00-1E-0B-88-21-2F
9-SEP-2021 07:49:48.37 Found LAN device LLA0, hardware address AA-00-00-07-F4-CD
9-SEP-2021 07:49:48.37 Found LAN device LLB0, hardware address AA-00-00-07-F4-CC
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failset device created
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failset device created
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %EIA0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIB0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIE0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIF0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIG0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 %EIH0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 Node database file, LAN$NODE_DATABASE, not found
9-SEP-2021 07:49:48.40 LANACP initialization complete

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 07:06:26 GMT

Lucas,

for LAN Failover, you configure (typically) more than ONE physical interface into a LAN Failover set. If one of those physical interfaces is up, so is the LAN Failover interface.

EIC0 and EID0 did NOT report "Link up" so they are either defect or the cable is unplugged or the switch port has a problem !

And if those LAN interfaces are the ONLY LAN interface configured into LLA0 resp. LLB0, it's clear why the LAN Failover devices are also down.

Did the LAN cables get removed during the repair action ? If so, have they been correctly plugged in again ? Your system has 8 LAN interfaces, are the cables and interfaces correctly documented ?

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 07:18:34 GMT

Cables should be ok, I ask HP engineer to double check again.

What I understand from your explanation is that currently I have 2 LOGICAL interfaces LE0 and LE1, each interfaces point to 2 physical devices, eg:

LE0 = LLA0 + EID0

LE1 = LLB0 + EIC0

Is this correct?

These 4 physical devices anyway are all down, that's why LE0 and LE1 are down too.

mc lancp show dev LLA0/char
Link Down Link state
mc lancp show dev LLB0/char
Link Down Link state
mc lancp show dev EIC0/char
Link Down Link state
mc lancp show dev EID0/char
Link Down Link state

So, if the cables is ok, switch is also ok, what is not clear for me is if the problem is on OpenVMS or I have to check some server hardware part like network interfaces.

Thank you for your time Volker

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 07:22:59 GMT

Lucas,

the LAN Failover Devices LE0 = LLA0 (consisting of just EID0) and LE1 = LLB0 (just EIC0) are down, because the only PHYSICAL LAN interfaces in both of these LAN failover sets did not report 'Link up'. You would normally configure LAN Failover sets with more than ONE LAN interface...

You have another cluster member up and running ? Same type of system ? Check the LAN interfaces and cabling on that one and compare to non-working system. Also look at the LEDs on those 8 physical LAN interfaces on your system.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 07:33:31 GMT

Unfortunately I'm not onsite (the server is on other city), anyway, sorry, still not clear the actual configuration and how can I check it.

Taking as example LE0 and basing on log, as I understood:

LE0 have 2 PHYSICAL device (means two cables): LLA0 + EID0

Am I wrong?

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 07:39:24 GMT

Lucas,

a LAN Failover Device (e.g. LLA0 in OpenVMS, named LE0 in TCPIP) is a 'virtual LAN device'. It consists of one (or more) PHYSICAL LAN interfaces. If any of those PHYSICAL LAN interfaces is up (Link Up), the LAN traffic is sent/recevied via that PHYSICAL LAN interface. If that interface/link fails, the 'other' LAN interface will be selected for send/receive. If you only have ONE PHYSICAL interface in a LAN Failover set and that interface is 'Link down', so is you LAN Failover device.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 07:58:31 GMT

Ok, so:

LE0/LLA0 ==> EID0 (PHYSICAL LAN interfaces)

So currently I have only one PHYSICAL device associated to logical device.

Because EID0 is down, also LE0 is down and the node is not reachable.

Am I right now?

Still don't understand if the issue is related to OpenVMS or hardware like network card. I bet hardware because, as I wrote, no one change the confgiuration before or after HP intervention.

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 09:28:20 GMT

Lucas,

you've got it !

If you have another working cluster member with the same LAN config, compare LANCP LIST DEV/CHAR from both systems.

How many network cables are plugged into those systems ? 6 or 8 ? You have 8 LAN interfaces, 6 have reported 'Link up'. Are 2 LAN interface without a cable ? If not, you have to check and follow those cables, for the 2 interfaces, which do NOT have the 'Link' LED on.

If you're not onsite, have someone send you pictures from the back of those systems...

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Thu, 09 Sep 2021 13:21:53 GMT

Hi Volker, thank you again for explanation.

On both nodes I have the following failover (the first node is working perfect and have exactly the same output shown).

I don't understand what it means. As I understand LLA0 have the following faolover devices:

"EID0" Failover device
"EIJ0" Failover device

But EIJ0 doesn't exists, so it means that the network on OpenVMS is misconfigured, because, as per in this case, if EID0 fails, the whole connection goes down since there is no other device active.

NODE82 Device Characteristics, permanent database, EIA0 (9-SEP-2021 12:58:51.28):
NODE82 Device Characteristics, permanent database, EIB0 (9-SEP-2021 12:58:51.28):
NODE82 Device Characteristics, permanent database, EIC0 (9-SEP-2021 12:58:51.28):
Value Characteristic
----- --------------
20 Failover priority
NODE82 Device Characteristics, permanent database, EID0 (9-SEP-2021 12:58:51.29):
Value Characteristic
----- --------------
20 Failover priority
NODE82 Device Characteristics, permanent database, EIE0 (9-SEP-2021 12:58:51.29):
NODE82 Device Characteristics, permanent database, EIF0 (9-SEP-2021 12:58:51.29):
NODE82 Device Characteristics, permanent database, EIG0 (9-SEP-2021 12:58:51.29):
NODE82 Device Characteristics, permanent database, EIH0 (9-SEP-2021 12:58:51.30):
NODE82 Device Characteristics, permanent database, EII0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
10 Failover priority
NODE82 Device Characteristics, permanent database, EIJ0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
10 Failover priority
NODE82 Device Characteristics, permanent database, LLA0 (9-SEP-2021 12:58:51.30):
Value Characteristic
----- --------------
Yes Logical LAN enabled for startup
"EID0" Failover device
"EIJ0" Failover device
NODE82 Device Characteristics, permanent database, LLB0 (9-SEP-2021 12:58:51.31):
Value Characteristic
----- --------------
Yes Logical LAN enabled for startup
"EIC0" Failover device
"EII0" Failover device

Re: OpenVMS network down after reboot

Volker Halle — Thu, 09 Sep 2021 14:02:30 GMT

Lucas,

check the system and network documentation !!!

If you believe the stored LAN Failover config is correct, you may also be missing/have lost a complete LAN card (with 2 ports, i.e. EII and EIJ) !

Count the LAN cables and look at the pictures of the LAN interface ports of BOTH systems.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Fri, 10 Sep 2021 04:24:52 GMT

You're right Volker, EII and EIJ are missing from second node (compared with the first one) due to hardware fault. I miss that information while I compared both cluster nodes information.

And finally HP was convinced that there was HW issue. After changing the faulty ethernet adapter, everything work as expected now. The node is currently reachable.

Anyway, thank you Volker for your time, thank to the log I sent to HP, using your comands, we are able to address the issue.

Now I have another issue to address on this system

On both cluster nodes I have the following:

SYSTEM> SHOW DEVICES/REBUILD_STATUS
Device Name Rebuild needed?
NODE82$DKA100: Yes
$1$DGA100: Yes
$1$DGA200: Yes
$1$DGA300: Yes
$1$DGA301: Yes
$1$DGA302: Yes
$1$DGA303: Yes
$1$DGA400: Yes
$1$DGA401: Yes
$1$DGA500: Yes
$1$DGA501: Yes
$1$DGA502: Yes
$1$DGA503: Yes
$1$DGA600: Yes

Rebuild needed on all volumes (they are volumes connected via 3par). Honestly I don't know what kind of rebuil it needs to do. e.g. this is an internal server disk: NODE82$DKA100, moreover not used in OpenVMS startup operation (we used it just as application log repository)

During boot, OpenVMS gives me an error like "cannot rebuild due to missing space", but there is sufficient space on all volumes.

**bleep**! Never ending story!

Re: OpenVMS network down after reboot

Volker Halle — Fri, 10 Sep 2021 04:50:01 GMT

Lucas,

please open a new topic for the 'rebuild needed' problem and then we'll see...

Please report the exact error message seen during OpenVMS boot when mounting the disks.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Fri, 10 Sep 2021 05:19:27 GMT

Thank you Volker, solved using this documentation without wasting other forum space

https://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/6015/6017p018.htm

Re: OpenVMS network down after reboot

Volker Halle — Fri, 10 Sep 2021 05:46:13 GMT

Lucas,

now that you've learned something about LAN Failover, it's time to review your system network configuration - as a first step, compare the setup with the other cluster member.

You have 2 LAN Failover devices (LLA consisting of EID,EIJ and LLB consisting of EIC,EII). By loosing EIJ and EII through a LAN card failure, both of your LAN Failover devices failed. And that's because EID and EIC are not correctly connected to your network switches ! With your 10 LAN ports, you should have 10 network cables connected to this system.

LAN Failover is used to increase redundancy of the LAN connection against network component failures. So it is correct, to distribute the members of a LAN Failover set across different LAN interface cards - as you've now learned. But you also need to make sure, that the 'other' members of each LAN Failover set correctly works. After you've made sure, that the 'other' LAN interfaces (EID and EIC) have a network 'Link up', you can use LANCP> SET DEVICE/SWITCH LLAx to test, whether the 'other' member in the LAN Failover are functioning correctly.

And for the future: regularily monitor all members of LAN Failover sets, that they all have 'Link up'.

Volker.

Re: OpenVMS network down after reboot

smsc_1 — Fri, 10 Sep 2021 06:04:16 GMT

Sure, this will be my next step, and I'm already working on it.

According to HP engineer, who was onsite, there are no cables on EID and EIC at all. That's why, with other failover member down, due to hardware failure, the connectivity gone down too. This is clearly a design error, even because services on these servers are critical.

Maybe it's time to travel for me, even if I don't known if my company allow me to travel for the COVID situation.

Re: OpenVMS network down after reboot

Volker Halle — Fri, 10 Sep 2021 06:34:36 GMT

Lucas,

regarding REBUILD: you've probably received - and misinterpreted - the following messages during MOUNT:

%MOUNT-I-REBLDREQD, rebuild not performed; some free space unavailable; diskquota usage stale

When a disk is not cleanly dismounted before a node shuts down (e.g. through a system crash), some blocks may still be marked as used in the volume bitmap and therefore not available for allocation. A SET VOLUME/REBUILD resolves this.

Volker.