Procurve 2900 24G - Startup failure

[Francisco Bermúdez]

We have a cluster with 8 nodes, 1 front-end, 1 NAS, 1 SAI and 2 switches (2900-24G, 2610). The attached file is a picture with the network structure.

Configuration front-end:

2 NIC's:

# /etc/sysconfig/network-scripts/ifcfg-eth0
# Interfaz de datos
DEVICE=eth0
ONBOOT=yes
HWADDR=00:30:48:55:01:B8
BOOTPROTO=static
MTU=9000
IPADDR=10.4.1.2
NETMASK=255.255.255.0

# /etc/sysconfig/network-scripts/ifcfg-eth0.2
# Interfaz de gestion
DEVICE=eth0.2
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.2.1.3
NETMASK=255.255.255.0
VLAN=yes

# /etc/sysconfig/network-scripts/ifcfg-eth0.3
# Interfaz de paso de mensajes
DEVICE=eth0.3
ONBOOT=yes
BOOTPROTO=static
MTU=9000
IPADDR=10.3.1.2
NETMASK=255.255.255.0
VLAN=yes

# /etc/sysconfig/network-scripts/ifcfg-eth1
# Interfaz publica
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=static
MTU=9000
IPADDR=193.147.33.209
NETMASK=255.255.255.192

# /etc/sysconfig/network-scripts/ifcfg-eth1:0
# Interfaz publica
DEVICE=eth1:0
ONBOOT=yes
BOOTPROTO=static
MTU=9000
IPADDR=193.147.33.208
NETMASK=255.255.255.192


Configuration nodes (example 0):

4 NIC's:

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:16:36:e0:87:94
IPADDR=10.4.1.254
NETMASK=255.255.255.0
BOOTPROTO=none
ONBOOT=yes

# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:16:36:E0:87:95
ONBOOT=no
TYPE=Ethernet

NOTE: eth1 is management device and it's configured with the management software included with hardware

# /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
HWADDR=00:16:36:e0:87:96
IPADDR=10.3.1.5
NETMASK=255.255.255.0
BOOTPROTO=none
ONBOOT=yes
MTU=9000
VLAN=yes

# /etc/sysconfig/network-scripts/ifcfg-eth3
DEVICE=eth3
HWADDR=00:16:36:E0:87:97
ONBOOT=no
TYPE=Ethernet

NOTE: eth3 is not used

Configuration NAS:

3 NIC's:

# /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=10.4.1.253
NETMASK=255.255.255.0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MTU=9000

# /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
MTU=9000

# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
MTU=9000

Configuration SAI:

1 NIC:

Management device configured with software integrated in hardware.

Configuration Switch 2900-24G:

Software revision : T.13.63
ROM Version : K.12.12

# show vlans
Status and Counters - VLAN Information

Maximum VLANs to support : 256
Primary VLAN : publica
Management VLAN :

VLAN ID Name | Status Voice Jumbo
------- -------------------- + ---------- ----- -----
1 publica | Port-based No Yes
2 gestion | Port-based No No
3 mensajes | Port-based No Yes
4 datos | Port-based No Yes

# show ip

Internet (IP) Service

IP Routing : Disabled

Default Gateway : 10.2.1.2
Default TTL : 64
Arp Age : 20
Domain Suffix :
DNS server :

VLAN | IP Config IP Address Subnet Mask Proxy ARP
------- + ---------- --------------- --------------- ---------
publica | Manual 10.1.1.1 255.255.255.0 No
gestion | Manual 10.2.1.1 255.255.255.0 No
mensajes| Manual 10.3.1.1 255.255.255.0 No
datos | Manual 10.4.1.1 255.255.255.0 No

# show vlan 1
Status and Counters - VLAN Information - VLAN 1

VLAN ID : 1
Name : publica
Status : Port-based
Voice : No
Jumbo : Yes

Port Information Mode Unknown VLAN Status
---------------- -------- ------------ ----------
1 Untagged Learn Up
2 Untagged Learn Up
23 Untagged Learn Down
24 Untagged Learn Down
A1 Untagged Learn Down
A2 Untagged Learn Down
A3 Untagged Learn Down
A4 Untagged Learn Down

# show vlan 2

Status and Counters - VLAN Information - VLAN 2

VLAN ID : 2
Name : gestion
Status : Port-based
Voice : No
Jumbo : No

Port Information Mode Unknown VLAN Status
---------------- -------- ------------ ----------
3 Tagged Learn Up
4 Untagged Learn Up

# show vlan 3
Status and Counters - VLAN Information - VLAN 3

VLAN ID : 3
Name : paso_mensajes
Status : Port-based
Voice : No
Jumbo : Yes

Port Information Mode Unknown VLAN Status
---------------- -------- ------------ ----------
3 Tagged Learn Up
8 Untagged Learn Up
10 Untagged Learn Up
12 Untagged Learn Up
14 Untagged Learn Up
16 Untagged Learn Up
18 Untagged Learn Up
20 Untagged Learn Up
22 Untagged Learn Up

# show vlan 4
Status and Counters - VLAN Information - VLAN 4

VLAN ID : 4
Name : datos
Status : Port-based
Voice : No
Jumbo : Yes

Port Information Mode Unknown VLAN Status
---------------- -------- ------------ ----------
3 Untagged Learn Up
7 Untagged Learn Up
9 Untagged Learn Up
11 Untagged Learn Up
13 Untagged Learn Up
15 Untagged Learn Up
17 Untagged Learn Up
19 Untagged Learn Up
21 Untagged Learn Up
Trk1 Untagged Learn Up

# show interfaces config

Port Type | Enabled Mode Flow Ctrl MDI
------ --------- + ------- ------------ --------- ----
1 100/1000T | Yes Auto Disable Auto
2 100/1000T | Yes Auto Disable Auto
3 100/1000T | Yes Auto Disable Auto
4 100/1000T | Yes Auto Disable Auto
5-Trk1 100/1000T | Yes Auto Disable Auto
6-Trk1 100/1000T | Yes Auto Disable Auto
7 100/1000T | Yes Auto Disable Auto
8 100/1000T | Yes Auto Disable Auto
9 100/1000T | Yes Auto Disable Auto
10 100/1000T | Yes Auto Disable Auto
11 100/1000T | Yes Auto Disable Auto
12 100/1000T | Yes Auto Disable Auto
13 100/1000T | Yes Auto Disable Auto
14 100/1000T | Yes Auto Disable Auto
15 100/1000T | Yes Auto Disable Auto
16 100/1000T | Yes Auto Disable Auto
17 100/1000T | Yes Auto Disable Auto
18 100/1000T | Yes Auto Disable Auto
19 100/1000T | Yes Auto Disable Auto
20 100/1000T | Yes Auto Disable Auto
21 100/1000T | Yes Auto Disable Auto
22 100/1000T | Yes Auto Disable Auto
23 100/1000T | Yes Auto Disable Auto
24 100/1000T | Yes Auto Disable Auto
A1 | Yes Disable 0
A2 10GbE-CX4 | Yes Auto Disable 0
A3 10GbE-CX4 | Yes Auto Disable 0
A4 | Yes Disable 0

# show trunks
Load Balancing

Port | Name Type | Group Type
---- + -------------------------------- --------- + ----- -----
5 | nas_0_0_eth0_datos 100/1000T | Trk1 Trunk
6 | nas_0_0_eth1_datos 100/1000T | Trk1 Trunk

# show lacp

LACP

no LACP ports found.

# show config

time timezone 60
time daylight-time-rule Western-Europe
no telnet-server
module 1 type J86xxA
module 3 type J90XXA
interface 1
name "roseta_publica"
exit
interface 2
name "frontend_eth1_publica"
exit
interface 3
name "frontend_eth0_gestion_datos_paso_mensajes"
exit
interface 4
name "switch_gestion"
exit
interface 5
name "nas_0_0_eth0_datos"
exit
interface 6
name "nas_0_0_eth1_datos"
exit
interface 7
name "compute_0_0_eth0_datos"
exit
interface 8
name "compute_0_0_eth3_paso_mensajes"
exit
interface 9
name "compute_0_1_eth0_datos"
exit
interface 10
name "compute_0_1_eth3_paso_mensajes"
exit
interface 11
name "compute_0_2_eth0_datos"
exit
interface 12
name "compute_0_2_eth3_paso_mensajes"
exit
interface 13
name "compute_0_3_eth0_datos"
exit
interface 14
name "compute_0_3_eth3_paso_mensajes"
exit
interface 15
name "compute_0_4_eth0_datos"
exit
interface 16
name "compute_0_4_eth3_paso_mensajes"
exit
interface 17
name "compute_0_5_eth0_datos"
exit
interface 18
name "compute_0_5_eth3_paso_mensajes"
exit
interface 19
name "compute_0_6_eth0_datos"
exit
interface 20
name "compute_0_6_eth3_paso_mensajes"
exit
interface 21
name "compute_0_7_eth0_datos"
exit
interface 22
name "compute_0_7_eth3_paso_mensajes"
exit
interface 24
name "frontend_hydra_publica"
exit
trunk 5-6 Trk1 Trunk
timesync sntp
sntp unicast
snmp-server community "public" Unrestricted
vlan 1
name "publica"
untagged 1-2,23-24,A1-A4
ip address 10.1.1.1 255.255.255.0
no untagged 3-4,7-22,Trk1
jumbo
exit
vlan 2
name "gestion"
untagged 4
ip address 10.2.1.1 255.255.255.0
tagged 3
exit
vlan 3
name "paso_mensajes"
untagged 8,10,12,14,16,18,20,22
ip address 10.3.1.1 255.255.255.0
tagged 3
jumbo
exit
vlan 4
name "datos"
untagged 3,7,9,11,13,15,17,19,21,Trk1
ip address 10.4.1.1 255.255.255.0
jumbo
exit
sntp server priority 1 10.2.1.2
sntp server priority 2 10.2.1.3
sntp server priority 3 193.147.33.193
ip ssh filetransfer
ip timep manual 193.147.33.193
ip route 0.0.0.0 0.0.0.0 10.2.1.2
spanning-tree Trk1 priority 4
no tftp client
password manager
password operator


Configuration Switch 2610:

# show config
Startup configuration:

; J9085A Configuration Editor; Created on release #R.11.04

hostname "ProCurve Switch 2610-24"
time timezone 60
time daylight-time-rule Western-Europe
no telnet-server
interface 1
name "sai_gestion"
exit
interface 2
name "nas_0_0_gestion"
exit
interface 3
name "compute_0_0_eth1_gestion"
exit
interface 4
name "compute_0_1_eth1_gestion"
exit
interface 5
name "compute_0_2_eth1_gestion"
exit
interface 6
name "compute_0_3_eth1_gestion"
exit
interface 7
name "compute_0_4_eth1_gestion"
exit
interface 8
name "compute_0_5_eth1_gestion"
exit
interface 9
name "compute_0_6_eth1_gestion"
exit
interface 10
name "compute_0_7_eth1_gestion"
exit
interface 25
name "switch_principal"
exit
sntp server 10.2.1.3
timesync sntp
sntp unicast
snmp-server community "public" Unrestricted
vlan 1
name "gestion"
untagged 1-28
ip address 10.2.1.2 255.255.255.0
exit
ip ssh
ip ssh filetransfer
no tftp client
no tftp server
password manager
password operator

PROBLEM: LOGS from Switch 2900-24G

Log after reboot due to power failure:

M 07/24/09 12:07:58 sys: 'System reboot due to Power Failure'
I 07/24/09 12:07:58 00061 system: ----------------------------------------------

I 07/24/09 12:07:58 00062 system: System went down without saving crash
information
I 07/24/09 12:07:59 00690 udpf: DHCP relay agent feature enabled
I 07/24/09 12:07:59 00118 timep: timep client enabled
I 07/24/09 12:07:59 00433 ssh: Ssh server enabled
I 07/24/09 12:07:59 00399 stack: Stack Protocol enabled
I 07/24/09 12:07:59 00128 tftp: Enable succeeded
I 07/24/09 12:07:59 00417 cdp: CDP enabled
I 07/24/09 12:07:59 00688 lldp: LLDP - enabled
I 07/24/09 12:07:59 00066 system: System Booted
I 07/24/09 12:07:59 00068 chassis: Ports 1-24 Inserted
I 07/24/09 12:08:05 00375 chassis: Ports 1-24 Downloading
I 07/24/09 12:08:06 00376 chassis: Ports 1-24 Download Complete
I 07/24/09 12:08:12 00179 mgr: SME CONSOLE Session - MANAGER Mode
I 07/24/09 12:08:20 00422 chassis: Ports 1-24 Ready
I 07/24/09 12:08:20 00068 chassis: Slot A Inserted
I 07/24/09 12:08:23 00001 vlan: publica virtual LAN enabled
I 07/24/09 12:08:23 00076 ports: port 1 is now on-line
I 07/24/09 12:08:24 00001 vlan: gestion virtual LAN enabled
I 07/24/09 12:08:24 00078 ports: trunk Trk1 is now active
I 07/24/09 12:08:24 00001 vlan: datos virtual LAN enabled
I 07/24/09 12:08:24 00076 ports: port 2 is now on-line
I 07/24/09 12:08:24 00076 ports: port 4 is now on-line
I 07/24/09 12:08:24 00076 ports: port 5 in Trk1 is now on-line
I 07/24/09 12:08:24 00076 ports: port 7 is now on-line
I 07/24/09 12:08:24 00076 ports: port 6 in Trk1 is now on-line
I 07/24/09 12:08:24 00001 vlan: paso_mensajes virtual LAN enabled
I 07/24/09 12:08:24 00076 ports: port 8 is now on-line
I 07/24/09 12:08:24 00076 ports: port 9 is now on-line
I 07/24/09 12:08:24 00076 ports: port 10 is now on-line
I 07/24/09 12:08:24 00076 ports: port 11 is now on-line
I 07/24/09 12:08:24 00076 ports: port 12 is now on-line
I 07/24/09 12:08:24 00076 ports: port 13 is now on-line
I 07/24/09 12:08:24 00076 ports: port 14 is now on-line
I 07/24/09 12:08:24 00076 ports: port 15 is now on-line
I 07/24/09 12:08:24 00076 ports: port 17 is now on-line
I 07/24/09 12:08:25 00076 ports: port 16 is now on-line
I 07/24/09 12:08:25 00076 ports: port 18 is now on-line
I 07/24/09 12:08:25 00076 ports: port 19 is now on-line
I 07/24/09 12:08:25 00076 ports: port 20 is now on-line
I 07/24/09 12:08:25 00076 ports: port 21 is now on-line
I 07/24/09 12:08:25 00076 ports: port 22 is now on-line
I 07/24/09 12:08:26 00375 chassis: Slot A Downloading
I 07/24/09 12:08:27 00376 chassis: Slot A Download Complete
I 07/24/09 12:08:41 00422 chassis: Slot A Ready


Log after # boot

I 07/24/09 12:12:43 00061 system: ----------------------------------------------
I 07/24/09 12:12:43 00063 system: System went down: 07/24/09 12:11:00
I 07/24/09 12:12:43 00064 system: Operator cold reboot from CONSOLE session.
I 07/24/09 12:12:43 00690 udpf: DHCP relay agent feature enabled
I 07/24/09 12:12:43 00118 timep: timep client enabled
I 07/24/09 12:12:44 00433 ssh: Ssh server enabled
I 07/24/09 12:12:44 00399 stack: Stack Protocol enabled
I 07/24/09 12:12:44 00128 tftp: Enable succeeded
I 07/24/09 12:12:44 00417 cdp: CDP enabled
I 07/24/09 12:12:44 00688 lldp: LLDP - enabled
I 07/24/09 12:12:44 00066 system: System Booted
I 07/24/09 12:12:44 00068 chassis: Ports 1-24 Inserted
I 07/24/09 12:12:50 00375 chassis: Ports 1-24 Downloading
I 07/24/09 12:12:51 00376 chassis: Ports 1-24 Download Complete
I 07/24/09 12:13:05 00422 chassis: Ports 1-24 Ready
I 07/24/09 12:13:05 00068 chassis: Slot A Inserted
I 07/24/09 12:13:08 00001 vlan: publica virtual LAN enabled
I 07/24/09 12:13:08 00076 ports: port 1 is now on-line
I 07/24/09 12:13:09 00001 vlan: gestion virtual LAN enabled
I 07/24/09 12:13:09 00001 vlan: paso_mensajes virtual LAN enabled
I 07/24/09 12:13:09 00001 vlan: datos virtual LAN enabled
I 07/24/09 12:13:09 00078 ports: trunk Trk1 is now active
I 07/24/09 12:13:09 00076 ports: port 3 is now on-line
I 07/24/09 12:13:09 00076 ports: port 4 is now on-line
I 07/24/09 12:13:09 00076 ports: port 6 in Trk1 is now on-line
I 07/24/09 12:13:09 00076 ports: port 5 in Trk1 is now on-line
I 07/24/09 12:13:09 00076 ports: port 2 is now on-line
I 07/24/09 12:13:09 00076 ports: port 7 is now on-line
I 07/24/09 12:13:09 00076 ports: port 8 is now on-line
I 07/24/09 12:13:09 00076 ports: port 9 is now on-line
I 07/24/09 12:13:09 00076 ports: port 10 is now on-line
I 07/24/09 12:13:09 00076 ports: port 11 is now on-line
I 07/24/09 12:13:09 00076 ports: port 12 is now on-line
I 07/24/09 12:13:09 00076 ports: port 13 is now on-line
I 07/24/09 12:13:09 00076 ports: port 14 is now on-line
I 07/24/09 12:13:10 00076 ports: port 15 is now on-line
I 07/24/09 12:13:10 00076 ports: port 16 is now on-line
I 07/24/09 12:13:10 00076 ports: port 17 is now on-line
I 07/24/09 12:13:10 00076 ports: port 18 is now on-line
I 07/24/09 12:13:10 00076 ports: port 19 is now on-line
I 07/24/09 12:13:10 00076 ports: port 20 is now on-line
I 07/24/09 12:13:10 00076 ports: port 21 is now on-line
I 07/24/09 12:13:10 00076 ports: port 22 is now on-line
I 07/24/09 12:13:11 00375 chassis: Slot A Downloading
I 07/24/09 12:13:12 00376 chassis: Slot A Download Complete
I 07/24/09 12:13:15 00179 mgr: SME SSH from 10.2.1.3 - MANAGER Mode
I 07/24/09 12:13:25 00422 chassis: Slot A Ready

The problem is port 3 is not online until boot switch from console. Anyone help me?

Excuse for my english and very large message.

[Pieter 't Hart]

suggestion : next time also post the configs in an attachment.

in case of operator requested cold-reboot
all vlan's assigned to the port are up, so the port comes up
>>>
I 07/24/09 12:13:09 00001 vlan: gestion virtual LAN enabled
I 07/24/09 12:13:09 00001 vlan: paso_mensajes virtual LAN enabled
I 07/24/09 12:13:09 00001 vlan: datos virtual LAN enabled
I 07/24/09 12:13:09 00078 ports: trunk Trk1 is now active
I 07/24/09 12:13:09 00076 ports: port 3 is now on-line
<<<

when booting after power failure
gestion and datos are up but paso_mensajes vlan is not activated yet.
so the port 3 is not up yet.
>>>
vlan: publica virtual LAN enabled
I 07/24/09 12:08:23 00076 ports: port 1 is now on-line
I 07/24/09 12:08:24 00001 vlan: gestion virtual LAN enabled
I 07/24/09 12:08:24 00078 ports: trunk Trk1 is now active
I 07/24/09 12:08:24 00001 vlan: datos virtual LAN enabled
I 07/24/09 12:08:24 00076 ports: port 2 is now on-line
I 07/24/09 12:08:24 00076 ports: port 4 is now on-line
I 07/24/09 12:08:24 00076 ports: port 5 in Trk1 is now on-line
I 07/24/09 12:08:24 00076 ports: port 7 is now on-line
I 07/24/09 12:08:24 00076 ports: port 6 in Trk1 is now on-line
I 07/24/09 12:08:24 00001 vlan: paso_mensajes virtual LAN enabled
I 07/24/09 12:08:24 00076
<<<

Is the front-end also restarted when this power-failure occured?

as it's the only vlan-trunk in your network, you may need to dig into manuals of your front-end.

[Francisco Bermúdez]

First, thanks for your fast reply.

<<<
suggestion : next time also post the configs in an attachment.
>>>

Sorry for that. I won't forget next time.

<<<
Is the front-end also restarted when this power-failure occured?
>>>

Yes. Both are restarted at the same time due to return input power to the cluster.

By other hand, i think that vlan 'paso_mensajes' is not necessary on port 3 because only nodes will use it. I'll probe to change configuration and test it.

Thanks for your atention.