Very Strange 5412zl Firmware Bug

[CCNP87]

Hi Folks,

Lately we have noticed upon issuing "show config" command our Core Switch is not displaying the full running configuration, rather it shows partial configurtaion.

I know it very well that the problem that occurred is not because of the recent configuration changes or the older ones, which in itself are harmless (As it does not require downtime).

Rather it is because of the bug in the device's code or the flash and it could have happened at any point of time with any configuration changes.

But everything perfectly works fine, as we have not received any complains from any users. We have around 7000 end nodes.

Another problem that wenoticed is, we are unable login into the switch via web browser which worked perfectly fine previously. It displays the credentials page and upon entering credentials it loads for a bit and remains blank.

And yes we were login in with the manager privileged account. And I feel folks at HPE support aren't a best option with whom I did not had much luck in the past so I am not even planning to raise a ticket this time around. Did any of you ever faced similar issues?

Rebooting might probably fix the problem but I would like to keep that as the last option since the site runs 24x7 365 days. The network is flat layer 2 with some static routes and the secondary CoreSW running VRRP, 5 distribution switches and around 150 access switches. On top of that the wireless controller is a zl moduled plugged into the main coreSW controlling around 200 APs. And there is no redundant controller!

And currently the SW is running firmware k15.04

I am afraid that it might even reboot if I issue few commands in order to troubleshoot as it happened with others in the past (I've read on bolgs and forums).
Awaiting your precious response, thanks a million in advance.

[Ian Vaughan]

Hello,

Do you have other switches on the same software version that are working properly and don't exhibit the same behavior?

That software image could be up to 5 years old so a lot of bugs could have been fixed in the meantime. I would treat it to an upgrade to the latest 15.x image at the next point you get downtime or you can switch traffic across to its VRRP buddy (assuming you have some HA in your configuration).

HPE support will probably ask you to upgrade to the latest version as a matter of course if you raise a support ticket.

Thanks

Ian

[Stephen A Swain]

15.04 is very old. Amazing you had that switch running for so long.

I think I remember a problem from a few years ago that was similar, and was due to free space loss (flash filling up) on that platform due to a buggy image. If it is a similar problem, you'll need to repair the switch with a firmware upgrade and possibly rebuild the config, since a 100% full filesystem means config changes aren't committed. I only noticed the problem during some changes, wr mem, and then later on a power failure caused a further outage as the commands were missing from the startup config. I remember the fixed image at the time did a cleanup of flash at boot time to clear the space again, but I had to redo the missing commands.

As you can no longer display the config, then you'll need to rely on backups or reconstruction (use show commands to map out vlans, etc).

I don't think you can avoid some downtime at this point, but it's best to aim to do that when you have some time to rebuild the missing config items. I have seen an alternative cleanup method since then, getting access to the filesystem at low level, but that might have been using a reboot anyway.

[CCNP87]

Yes, the rest of the switches(The other core sw and Dist sw(s) are running the same code and they are behaving normal.

Yes, I hope VRRP gets the job done. What do mean by HA ?

Thanks,

[CCNP87]

Yes I know its very old, actually the old folks over here prefer to keep things as they are as long as they work fine but I think its time that we must upgrade to the latest and the most stable next available firmware.

I do have the recent configuration backup from a point where things were normal,  I will use the same.

The irony is, our wireless controller (no redundancy) is plugged in to the Core1 SW and we have around 200APs, those will be down too.

Thank you so much.

[Stephen A Swain]

I've also seen failures of flash cards, sometimes they need replacing too. That can cause config file issues. You may need to lodge a case anyway, so they might send out a replacement flash, just in case you need it during recovery. I've had this happen a few times in the last several years on the 8200/5400's.

The other post (HA) is referring to High Availability designs, i.e. you probably won't have VRRP unless you already have a HA design in place. Converting from normal IP interface to VRRP isn't complicated, but there is a small amount of interruption in the change. You can speed things up by clearing MAC and ARP tables. However, how many changes can you do to the crippled switch before it crashes completely?

In either case, be prepared to repair the switch with an upgrade and some downtime, unless you can utilise a spare chassis, and move all services from the old switch to the new switch, with each service getting a small downtime. If I had the choice, I'd migrate to a spare switch myself, it takes longer but means less overall downtime. It's just VRRP needs to be considered if the original switch is to be maintained, or you could avoid VRRP totally, by joining vlans and moving the gateway IP's and any ip helpers.

[parnassus]

---

CCNP87 wrote: And currently its running firmware k15.04

Wow...really old (2011/2012?).

What's about Flash LED status (Green On|Off|Orange Blinking|Organge Blinking at a fast rate)?

For the sake of curiosity and if you can, what show flash, dir and show version commands exactly report?

The show version command is used to check the exact software version the Switch is currently running and what Boot Image the Switch is configured to boot from (Primary/Secondary).

The show flash and dir commands are used to check the content of the Compact Flash and, eventually, USB Flash Memory stick (if inserted).

[CCNP87 · Accepted solution]

Sorry for the late response,

The problem has been fixed by upgrading the switch to K15.18.007,  its been couple of months since the upgrade and running good.

Thank you all for you amazing support.

[parnassus]

Glad you solved.