Switch loses layer 3 connectivity for 20 minutes, nothing logged, layer 2 still working.

[rprague]

I'm stumped.  I have a medium sized all HP network covering a 70k sqft office/manufacturing building.  Core is a pair of E5412zl's and the edge switches are a mixture (J9089A, J4903A etc).

Starting last week, randomly some of the switches will just completely lose layer 3 services for 20 minutes.  I can't ping, telnet or snmp query the switches.  I'm unable to ping/telnet directly from the core the switch is attached to.  While that's happening the switch is still happily passing traffic layer 2 without issue or performance hit.

There's nothing in the logs of either the switch that goes offline or the core switch it connects to.   Of the 37 switches in the network, 9 of them are sporadically displaying this behavior.  Its almost completely limted to the J9089As, but one of the J4903As has done this more than any of the rest.

There's no difference in the configurations between the ones that are 'failing' and the ones that aren't other than different IP addresses.  All of the switches are monitored by Nagios for health and Cacti (using spine) for bandwidth data.

Configuration files between the switches having issues and those not are pretty much identical other than the vlan 1 ip address.  (config file attached).

I'm open to ideas at this point, I'm completely stumped.

[UpsideDown]

Could it be something as simple as duplicate IP addressing on your network? Have you excluded your network IP addresses from any DHCP scopes?

[rprague]

There's no DHCP server on the management network, and the MAC?ARP tables don't show another IP trying to take that device's traffic.