Lost connection to all my AP's after upgrade to V5.5.2.1 from V5.3.1 MSM765

[SFFKPS]

Just upgraded my MSM765zl controller to V5.5.2.1 from V5.3.1, and all my AP's (422) are offline.

They change between all the states:

Waiting for acceptance
Validating Firmware
Installing firmware
Validating capabilities

Validating configuration

etc,

cycling and rebooting all the time.

What is wrong? All my AP's are down as a result!

In some short periods one of the AP's are online for some munites, then starts cycling again

During the few minutes they have an IP, and partially connected to the controller, I see the firmware is correct, and setup also, but it never gets in the syncronized state.

Have even tried a brand new AP, with the same result.

[Patrick R]

I think it is a config compactibility issue, you are coming from a very low firmware to the latest, A few features have been introduced in 5.4.x and 5.5.x. the config must be some how corrupt or not well adopted by the new FM

I would reset and reconfigure the controller with the new firmware and rediscover the AP's

[SFFKPS]

I'm waiting for an new FW version from HP (have opened a support case), but I fear I have to reconfigure/reload. Here is a log entry from one of the AP's:

Jul 12 13:44:26 err websoap      MX950ZV1PP Detail: <error><errorcode>1009</errorcode><errorinfo>The primary server shared secret is invalid.</errorinfo></error>
Jul 12 13:44:26 err websoap      MX950ZV1PP SOAP FAULT: SOAP-ENV:Client "Invalid configuration"
Jul 12 13:44:26 err websoap      MX950ZV1PP Failed to commit transaction (ID=<35>, error=<The primary server shared secret is invalid.>).

And here islogs from the controller, many errors there, any clues to what I can change in the config? Maybe I can get it running if I change/update some parameters?

Jul 13 07:41:14 debug monitord     Starting /ram/tmp/strace-MAESTRO_SC.10483
Jul 13 07:41:25 warning monitord     KEEP ALIVE failure on <MAESTRO_SC>, restart process if running
Jul 13 07:41:25 debug monitord     Stopping [1,10]: 'maestro_sc' [pid 10483, up for 315 sec(s)]
Jul 13 07:42:14 debug monitord     SIGTERM timeout expired after 45 seconds, SIGKILL used on PID=-10483
Jul 13 07:42:15 debug monitord     Starting [1,3]: 'maestro_sc' (pid='10514')
Jul 13 07:42:15 debug maestro_sc   Heap memory limits changed to 5000 KB
Jul 13 07:42:15 debug busd         Error while calling sendto on socket 3: Connection refused (111)
Jul 13 07:42:15 debug busd         SendMessageToListeners: Error sending packet: Connection refused(111) to listener #14(/tmp/busclient.maestro_sc.10483.1)
Jul 13 07:42:15 debug openvpn      event_wait : Interrupted system call (code=4)
Jul 13 07:42:15 debug openvpn      TCP/UDP: Closing socket
Jul 13 07:42:15 debug openvpn      SIGUSR1[hard,] received, process restarting
Jul 13 07:42:17 debug openvpn      WARNING: --keepalive option is missing from server config
Jul 13 07:42:17 debug openvpn      Re-using SSL/TLS context
Jul 13 07:42:17 debug openvpn      WARNING: normally if you use --mssfix and/or --fragment, you should also set --tun-mtu 1500 (currently it is 1431)
Jul 13 07:42:17 debug openvpn      TLS-Auth MTU parms [ L:1472 D:166 EF:66 EB:0 ET:0 EL:0 ]
Jul 13 07:42:17 debug openvpn      Preserving previous TUN/TAP instance: tun0
Jul 13 07:42:17 debug openvpn      Data Channel MTU parms [ L:1472 D:1450 EF:41 EB:4 ET:0 EL:0 ]
Jul 13 07:42:17 debug openvpn      UDPv4 link local (bound): [undef]:1194
Jul 13 07:42:17 debug openvpn      UDPv4 link remote: [undef]
Jul 13 07:42:17 debug openvpn      Initialization Sequence Completed

[DougB-CCCP]

I have seen this before - before you completely wipe and rebuild the controller, re-apply the config that you backed up while still running 5.3.1 to the 5.5.2.1 system.  In my case it allows the AP's to reconnect.  I haven't heard any post-issues from the customers after doing it this way.

[SFFKPS]

Sorry, tried to load the old 5-3-x config to version 5.4.x, but no luck, the AP's cycle through all the boot and config stages, so no luck there.

Tried a downgrade from 5.5.x to 5.4 x today, the config was completely gone, had to enter IP manually to get web management on the controller again. Then a downgrade to my old 5.3.x-version, lost config/IP again, set new IP, and restored my old 5.3.x-config, and soon all the AP's started to downgrade and got back online. (between these stages I used an ACL on the switch to prevent the AP's access to the controller, in case they would start loading an incorrect config out of control - a good safety mesure). All my AP's are routed, none are directly connected with VLAN, so an ACL was possible.

So now I'm online with all AP's, but no further toward my 5.5.x-version. HP is working on the case.

I hope we fint a way to do it, so I don't have to enter all config from scratch on verison 5.5.x... 

[Patrick R]

This is not a hardware issue. Depending on the config you have, you might have a problem after updating to a version with a major fix (5.x). I am not sure a new firmware release can correct a corrupt config, in those cases you will need to f reset and update, and reconfigure the controller. 

If there is a new firmware, it will probebly help to avoid this issues after the update. 

[Patrick R]

And the right update path is from 5.3.x to 5.4.x then 5.5.x 

[SFFKPS]

After succesfully downgrading to 5.3, and reloading my old backup config, and verifying that all my AP's where back online, I tried another upgrade to a 5.4.x version received from HP, no luck there. Same error.

Also tried on v 5.4 to reload the old 5.3 config, not any better.

So, the real problem is that the normal upgrade from 5.3.x to 5.4.x does not work either....

[Patrick R]

You should have your case with the support update with this information so that a fix can be implimented for the problem. or get the right update path, by this i mean 5.4.x.xxxx to 5.5.x.xxxx

[SFFKPS]

HP support L2 is on the case and they have all thsi info and much more, including a lot of tech/config files etc., and the 5.4.x was from the support team. So for the time beeing we'll let them work out a plan and solution. I tried Doug's trick of using old config on new firmware, no luck as i described above.

So now I'll wait for HP.