Lost connection to all my AP's after upgrade to V5.5.2.1 from V5.3.1 MSM765

[SFFKPS]

Just upgraded my MSM765zl controller to V5.5.2.1 from V5.3.1, and all my AP's (422) are offline.

They change between all the states:

Waiting for acceptance
Validating Firmware
Installing firmware
Validating capabilities

Validating configuration

etc,

cycling and rebooting all the time.

What is wrong? All my AP's are down as a result!

In some short periods one of the AP's are online for some munites, then starts cycling again

During the few minutes they have an IP, and partially connected to the controller, I see the firmware is correct, and setup also, but it never gets in the syncronized state.

Have even tried a brand new AP, with the same result.

[Patrick R]

I think it is a config compactibility issue, you are coming from a very low firmware to the latest, A few features have been introduced in 5.4.x and 5.5.x. the config must be some how corrupt or not well adopted by the new FM

I would reset and reconfigure the controller with the new firmware and rediscover the AP's

[SFFKPS]

I'm waiting for an new FW version from HP (have opened a support case), but I fear I have to reconfigure/reload. Here is a log entry from one of the AP's:

Jul 12 13:44:26 err websoap      MX950ZV1PP Detail: <error><errorcode>1009</errorcode><errorinfo>The primary server shared secret is invalid.</errorinfo></error>
Jul 12 13:44:26 err websoap      MX950ZV1PP SOAP FAULT: SOAP-ENV:Client "Invalid configuration"
Jul 12 13:44:26 err websoap      MX950ZV1PP Failed to commit transaction (ID=<35>, error=<The primary server shared secret is invalid.>).

And here islogs from the controller, many errors there, any clues to what I can change in the config? Maybe I can get it running if I change/update some parameters?

Jul 13 07:41:14 debug monitord     Starting /ram/tmp/strace-MAESTRO_SC.10483
Jul 13 07:41:25 warning monitord     KEEP ALIVE failure on <MAESTRO_SC>, restart process if running
Jul 13 07:41:25 debug monitord     Stopping [1,10]: 'maestro_sc' [pid 10483, up for 315 sec(s)]
Jul 13 07:42:14 debug monitord     SIGTERM timeout expired after 45 seconds, SIGKILL used on PID=-10483
Jul 13 07:42:15 debug monitord     Starting [1,3]: 'maestro_sc' (pid='10514')
Jul 13 07:42:15 debug maestro_sc   Heap memory limits changed to 5000 KB
Jul 13 07:42:15 debug busd         Error while calling sendto on socket 3: Connection refused (111)
Jul 13 07:42:15 debug busd         SendMessageToListeners: Error sending packet: Connection refused(111) to listener #14(/tmp/busclient.maestro_sc.10483.1)
Jul 13 07:42:15 debug openvpn      event_wait : Interrupted system call (code=4)
Jul 13 07:42:15 debug openvpn      TCP/UDP: Closing socket
Jul 13 07:42:15 debug openvpn      SIGUSR1[hard,] received, process restarting
Jul 13 07:42:17 debug openvpn      WARNING: --keepalive option is missing from server config
Jul 13 07:42:17 debug openvpn      Re-using SSL/TLS context
Jul 13 07:42:17 debug openvpn      WARNING: normally if you use --mssfix and/or --fragment, you should also set --tun-mtu 1500 (currently it is 1431)
Jul 13 07:42:17 debug openvpn      TLS-Auth MTU parms [ L:1472 D:166 EF:66 EB:0 ET:0 EL:0 ]
Jul 13 07:42:17 debug openvpn      Preserving previous TUN/TAP instance: tun0
Jul 13 07:42:17 debug openvpn      Data Channel MTU parms [ L:1472 D:1450 EF:41 EB:4 ET:0 EL:0 ]
Jul 13 07:42:17 debug openvpn      UDPv4 link local (bound): [undef]:1194
Jul 13 07:42:17 debug openvpn      UDPv4 link remote: [undef]
Jul 13 07:42:17 debug openvpn      Initialization Sequence Completed

[DougB-CCCP]

I have seen this before - before you completely wipe and rebuild the controller, re-apply the config that you backed up while still running 5.3.1 to the 5.5.2.1 system.  In my case it allows the AP's to reconnect.  I haven't heard any post-issues from the customers after doing it this way.

[SFFKPS]

Sorry, tried to load the old 5-3-x config to version 5.4.x, but no luck, the AP's cycle through all the boot and config stages, so no luck there.

Tried a downgrade from 5.5.x to 5.4 x today, the config was completely gone, had to enter IP manually to get web management on the controller again. Then a downgrade to my old 5.3.x-version, lost config/IP again, set new IP, and restored my old 5.3.x-config, and soon all the AP's started to downgrade and got back online. (between these stages I used an ACL on the switch to prevent the AP's access to the controller, in case they would start loading an incorrect config out of control - a good safety mesure). All my AP's are routed, none are directly connected with VLAN, so an ACL was possible.

So now I'm online with all AP's, but no further toward my 5.5.x-version. HP is working on the case.

I hope we fint a way to do it, so I don't have to enter all config from scratch on verison 5.5.x... 

[Patrick R]

This is not a hardware issue. Depending on the config you have, you might have a problem after updating to a version with a major fix (5.x). I am not sure a new firmware release can correct a corrupt config, in those cases you will need to f reset and update, and reconfigure the controller. 

If there is a new firmware, it will probebly help to avoid this issues after the update. 

[Patrick R]

And the right update path is from 5.3.x to 5.4.x then 5.5.x 

[SFFKPS]

After succesfully downgrading to 5.3, and reloading my old backup config, and verifying that all my AP's where back online, I tried another upgrade to a 5.4.x version received from HP, no luck there. Same error.

Also tried on v 5.4 to reload the old 5.3 config, not any better.

So, the real problem is that the normal upgrade from 5.3.x to 5.4.x does not work either....

[Patrick R]

You should have your case with the support update with this information so that a fix can be implimented for the problem. or get the right update path, by this i mean 5.4.x.xxxx to 5.5.x.xxxx

[SFFKPS]

HP support L2 is on the case and they have all thsi info and much more, including a lot of tech/config files etc., and the 5.4.x was from the support team. So for the time beeing we'll let them work out a plan and solution. I tried Doug's trick of using old config on new firmware, no luck as i described above.

So now I'll wait for HP.

[channel]

Hi,

i have the same Problem (MSM760). Is there any solution (HP Support)?

[Arimo]

Hi there

The first thing you need to remember is that when you purchase an MSM the you are eligible to have the latest available software at the time of purchase, and maintenance releases for one year after this. This means that for one year you would be eligible to have updates for 5.3 -branch (i.e. you are eligible to have 5.3.2, 5.3.3, 5.3.4 etc) but upgrade to 5.4, 5.5 and so on requires purchasing a software Care Pack. This is detailed in Warranty Summary, http://h17007.www1.hp.com/docs/support/warranty/HPNetworkingProductWarrantySummary.pdf

5.5 was published 1 Feb 2011, and if your controller is running 5.3.1 it's most likely purchased before that date. This means that you might be eligible for 5.4 -branch, but not 5.5.

The second thing is that updating directly from 5.3 to 5.5 is like trying to update directly from Win95 to Win7. There are significant changes in the software and the configuration, which will cause your existing implementation to break. Your first step is to get the controller back to 5.3.1.

If you updated your controller to resolve a specific issue, I'd advise you to discuss with support about it. They should be able to help you resolve it.

[PRay]

I had the similar error message in mine. During 5.3 I had played with Radius Server, and then decommissioned the Radius, this did not cause errors in 5.3 but upon upgrade to 5.5 was causing all sorts of Soap type errors.

Deleting any invalid Radius Profile(s), (I deleted all) resolved the issue for me.

Hope That Helps Someone,

Regards,

PRay

[gtrsteve]

From the events, it seems like there's an authentication failure. Are you using RADIUS authentication for your APs in Controller\Controlled APs\Authentication? If so, have you tried deleting and recreating the RADIUS profile selected there and then doing a remove/rediscover of the APs? OR, try deselecting authentication and then remove/rediscover the APs.

Just some ideas. Hope this helps!

Steve

[runner26]

any solution to this issue as I'm having the same issue and support doesn't seem to have an answer.

[SFFKPS · Accepted solution]

I had to go the hard way, factory reset after upgrading to latest  version 5.5.2.1, and entering all config manually.

I suggested for HP that they could use some resources in configuring the missing parameters I Web GUI on either v5.3 or v5.5, since the error log noted some missing parameters, but they where not willing to do that.

So I had to do the following:

1. Printscreen of all config from webgui from my old  V5.3.x, pasted with comments into word.
2. Saved old config in case of rollback.
3. Made an ACL on my nearest router to prevent AP's connecting to controller while working - important if you have routed connection, since configuring manually could cause AP's reading wrong config before you are ready to deploy, possibly loosing all AP's!!! 
4. Upgrade to 5.5.2.1-01-10012
5. Then reset to factory default
6. Then entered all config manually from my documentation, only a few small differences in the versions.
7. Removed ACL for one AP first, this upgraded and became stable, then the others, and all booted correctly and remained stable after this. Have been running for a week now.

Note that we have no Radius or firewall/access control setup on the controller, and no locally connected AP's (all are routed in the WAN). So my config was "relatively" simple to recreate.

[Stephen Swain]

I've had similar trouble with msm765zl whilst upgrading from 5.3 to 5.4. Ended up corrupting the flash/hdd. Had to have replacement shipped out. And it happened twice!

The fix would have been "reset to default", but that option failed post upgrade, so it could not be fixed.

And a "reset to defaults" with upgrade and a restore of old version config also failed.

Now, as a rule of thumb, I "reset to defaults" before any major upgrade and reconfigure manually. I'm getting so used to it, I know the config in my head now. :P

The pain, oh the pain.

Btw, now running on 5.5.2.14 and the upgrade from 5.5.2.4 was error free.

Regards,

Steve.