OpenVMS network down after reboot

[smsc_1]

Dear community I'm in trouble with OpenVMS V8.4 server.

Due to memory fault I needed to switch off the server so that can HP can replace tha faulty DIMM, now the server bootup correctly with 32GB but the problem is that it's now completly isolated from the newtork. I can reach it only via iLO/MP. No configuration changes was applied before and after DIMM replacement and the server was rebooted 4 month ago and comeup without issue.

HP says that there is no network card fault, but I'm not properly sure, since, as I wrote, no configuration changes are applied and after reboot everything are there (like routes, IP assignment, etc.).

From the HP switch I can see that related port and VLAN are correctly UP:

Vlan4 UP UP 10.97.13.140 MESSAGING
Vlan7 UP UP 10.97.13.154 OAM

GE1/0/6 UP 1G F A 4 NODE82-S6-P3
GE1/0/7 UP 1G F A 7 NODE82-S6-P4

GE2/0/6 UP 1G F A 4 NODE82-S5-P3
GE2/0/7 UP 1G F A 7 NODE82-S5-P4

Network interfaces on OpenVMS shown no traffic but the strange thing is that there is no mac address shown while on other node of the cluster, currently working, I can see the mac address

Interface: LE0
IP_Addr: 10.97.13.146 NETWRK: 255.255.255.240 BRDCST: 10.97.13.159
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0


Interface: LE1
IP_Addr: 10.97.13.130 NETWRK: 255.255.255.240 BRDCST: 10.97.13.143
Ethernet_Addr: 00-00-00-00-00-00 MTU: 1500
Flags: UP BRDCST MCAST SMPX
RECEIVE SEND
Packets 0 0
Errors 0 0
Collisions: 0

Could someone help me to understand what's wrong? And if there are any other commands I can do on OpenVMS to show me better what's the interfaces situation? I'm not OpenVMS expert and I'm stuck.

Thank you very much,

Lucas

[Volker Halle]

Lucas,

the IP interface flags should be: UP BRDCST RUN MCAST SMPX

So the RUN bit is missing in your case.

Check SYS$MANAGER:OPERATOR.LOG for error messages from TCPIP startup.

Volker.

[smsc_1]

Thank you very much Volker, what does it mean or what could be the problem that can cause flag missing? Hardware? Software? What do you think?

Anyway, here is the operator.log, thanks:

https://pastebin.com/sELZCLbt

EDIT: Adding more information about interfaces:

From TPCIP$CONFIG I can see:

1 - LE0 Menu (LLA0: Unspecified 10mbps)
2 - 10.97.13.146/28 NODE82 Configured,Active

3 - LE1 Menu (LLB0: Unspecified 10mbps)
4 - 10.97.13.130/28 NODE82 Configured,Active

The related devices LLA0 and LLB0 are currently DOWN:

LANCP> SHO DEV LLA0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CD Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CD MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EID0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state

LANCP> SHO DEV LLB0/CHA
Value Characteristic
----- --------------
1500 Device buffer size
Normal Controller mode
External Internal loopback mode
AA-00-00-07-F4-CC Default MAC address (Hardware LAN address)
Multicast address list
Ethernet Communication medium
FF-FF-FF-FF-FF-FF MAC address (Current LAN address)
128 Minimum receive buffers
256 Maximum receive buffers
No Full duplex enable
No Full duplex operational
AA-00-00-07-F4-CC MAC address (Current LAN address)
Unspecified Line media type
10 Line speed (mbps)
Disabled Auto-negotiation
Disabled Flow control
Disabled Jumbo frames
"EIC0" Failover device
Enabled/Failset disconn Logical LAN state
0 Failover priority
Link Down Link state

[Volker Halle]

Lucas,

you're running some IP interfaces on LAN Failover Devices (LE0 and LE1, these match the OpenVMS devices LLA0 and LLB0). With LANCP> SHOW DEV/FULL LLx, you should be able to find out, which physical LAN interfaces are used by these LAN Failover Devices.

You might need to look at SYS$MANAGER:LAN$ACP.LOG and LANCP> SHOW DEVICE/INTERNAL LLAx to see the LAN Driver messages from early startup.

Volker.

[smsc_1]

Hi Volker,

I did the command you suggest me, but I cannot fin anything interesting, or at least something not clear for me

I can see: Logical LAN failover device added to failset, EIC0 for LLA0 and EID0 for LLB0.

But these devices (EIC0 and EID0) are both down.

What does it mean? That LE0 have 2 interfaces (EIC0 and LLA0) like bond in linux?

What is not clear for me is that the configuration works perfect before RAM failure, and noone changes it before or after RAM issue.

SYSTEM> SHOW DEV/FULL LLA0
Device LLA0:, device type LLAN, is online, network device, device is a template
only.

Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512

Operating characteristics: Link down.

Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CD Current MAC addr AA-00-00-07-F4-CD

SYSTEM> SHOW DEV/FULL LLB0

Device LLB0:, device type LLAN, is online, network device, device is a template
only.

Error count 0 Operations completed 0
Owner process "" Owner UIC [SYSTEM]
Owner process ID 00000000 Dev Prot S:RWPL,O:RWPL,G,W
Reference count 0 Default buffer size 512

Operating characteristics: Link down.

Speed (Mbits/sec) 10
Def. MAC addr AA-00-00-07-F4-CC Current MAC addr AA-00-00-07-F4-CC

LANCP> SHOW DEVICE/INTERNAL LLA0

SMWI82 Device Internal Counters LLA0 (9-SEP-2021 08:50:22.03):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLA" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EID LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:22.86 Logical LAN failset device created

LANCP> SHOW DEVICE/INTERNAL LLB0

SMWI82 Device Internal Counters LLB0 (9-SEP-2021 08:50:32.53):
Value Counter
----- -------
--- Internal Driver Counters ---
" LLB" Device name
"Jul 19 2011 16:17:51" Driver timestamp
36 Driver version (X-n)
--- Failover set ---
"EIC LinkDn 00000014" Device 1 (Name State Priority)
1 Number of devices in failover set
Yes All devices unavailable
No All devices offline
Yes Device inited
Normal Saved CON parameter
Internal Saved ILP parameter
--- Failover counters ---
--- Driver Messages ---
9-SEP-2021 07:49:22.86 Logical LAN failover devices all unavailable
9-SEP-2021 07:49:22.86 Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:22.86 Logical LAN failset device created

SYSTEM> ty SYS$MANAGER:LAN$ACP.LOG
9-SEP-2021 07:49:48.35 Defined LAN$DLL to be SYS$SYSROOT:[MOM$SYSTEM]
9-SEP-2021 07:49:48.36 Found LAN device EIA0, hardware address 00-1E-0B-88-16-76
9-SEP-2021 07:49:48.36 Found LAN device EIB0, hardware address 00-1E-0B-88-16-77
9-SEP-2021 07:49:48.36 Found LAN device EIC0, hardware address 00-1A-4B-07-F4-CC
9-SEP-2021 07:49:48.36 Found LAN device EID0, hardware address 00-1A-4B-07-F4-CD
9-SEP-2021 07:49:48.36 Found LAN device EIE0, hardware address 00-1E-0B-88-21-2C
9-SEP-2021 07:49:48.36 Found LAN device EIF0, hardware address 00-1E-0B-88-21-2D
9-SEP-2021 07:49:48.37 Found LAN device EIG0, hardware address 00-1E-0B-88-21-2E
9-SEP-2021 07:49:48.37 Found LAN device EIH0, hardware address 00-1E-0B-88-21-2F
9-SEP-2021 07:49:48.37 Found LAN device LLA0, hardware address AA-00-00-07-F4-CD
9-SEP-2021 07:49:48.37 Found LAN device LLB0, hardware address AA-00-00-07-F4-CC
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failset device created
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device unavailable, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover device added to failset, EID0
9-SEP-2021 07:49:48.37 %LLA0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failset device created
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device unavailable, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover device added to failset, EIC0
9-SEP-2021 07:49:48.38 %LLB0, Logical LAN failover devices all unavailable
9-SEP-2021 07:49:48.38 %EIA0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIB0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIE0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIF0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.38 %EIG0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 %EIH0, Link up: 1000 mbit, full duplex, flow control (receive only)
9-SEP-2021 07:49:48.39 Node database file, LAN$NODE_DATABASE, not found
9-SEP-2021 07:49:48.40 LANACP initialization complete

[Volker Halle]

Lucas,

for LAN Failover, you configure (typically) more than ONE physical interface into a LAN Failover set. If one of those physical interfaces is up, so is the LAN Failover interface. 

EIC0 and EID0 did NOT report "Link up" so they are either defect or the cable is unplugged or the switch port has a problem !

And if those LAN interfaces are the ONLY LAN interface configured into LLA0 resp. LLB0, it's clear why the LAN Failover devices are also down.

Did the LAN cables get removed during the repair action ? If so, have they been correctly plugged in again ? Your system has 8 LAN interfaces, are the cables and interfaces correctly documented ?

Volker.

[smsc_1]

Cables should be ok, I ask HP engineer to double check again.

What I understand from your explanation is that currently I have 2 LOGICAL interfaces LE0 and LE1, each interfaces point to 2 physical devices, eg:

LE0 = LLA0 + EID0

LE1 = LLB0 + EIC0

Is this correct?

These 4 physical devices anyway are all down, that's why LE0 and LE1 are down too.

mc lancp show dev LLA0/char
Link Down Link state
mc lancp show dev LLB0/char
Link Down Link state
mc lancp show dev EIC0/char
Link Down Link state
mc lancp show dev EID0/char
Link Down Link state

---

---

So, if the cables is ok, switch is also ok, what is not clear for me is if the problem is on OpenVMS or I have to check some server hardware part like network interfaces.

Thank you for your time Volker

[Volker Halle]

Lucas,

the LAN Failover Devices LE0 = LLA0 (consisting of just EID0) and LE1 = LLB0 (just EIC0) are down, because the only PHYSICAL LAN interfaces in both of these LAN failover sets did not report 'Link up'. You would normally configure LAN Failover sets with more than ONE LAN interface...

You have another cluster member up and running ? Same type of system ? Check the LAN interfaces and cabling on that one and compare to non-working system. Also look at the LEDs on those 8 physical LAN interfaces on your system.

Volker.

[smsc_1]

Unfortunately I'm not onsite (the server is on other city), anyway, sorry, still not clear the actual configuration and how can I check it.

Taking as example LE0 and basing on log, as I understood:

LE0 have 2 PHYSICAL device (means two cables): LLA0 + EID0

Am I wrong?

[Volker Halle]

Lucas,

a LAN Failover Device (e.g. LLA0 in OpenVMS, named LE0 in TCPIP) is a 'virtual LAN device'. It consists of one (or more) PHYSICAL LAN interfaces. If any of those PHYSICAL LAN interfaces is up (Link Up), the LAN traffic is sent/recevied via that PHYSICAL LAN interface. If that interface/link fails, the 'other' LAN interface will be selected for send/receive. If you only have ONE PHYSICAL interface in a LAN Failover set and that interface is 'Link down', so is you LAN Failover device.

Volker.