Second team member is in loop(Resetting, Uploading Config.) after 6.6.5 firmware upgrade, [MSM 720]

[Enthusia]

Hi, 

I upgraded MSM720 team wifi controller 6.5.1 to 6.6.5. Then all syncronizing completed on AP and manager controller. But second team member not syncronized at all. It's in loop. (uploading and resetting configuration)

I attached the logs. 

Firmwares is 6.6.5 all of them. 

Manager: 74:46:A0:DE:04:88
Other member: 74:46:A0:DE:04:B8

Thanks for your help.

Ctrl-1 (Manager):

Jan 10 14:25:59.629 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANPort()
Jan 10 14:25:59.649 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__DeleteAllTaggedPortsFromVlan()
Jan 10 14:25:59.656 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__DeleteAllUnTaggedPortsFromVlan()
Jan 10 14:25:59.676 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANMapping()
Jan 10 14:25:59.707 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLAN()
Jan 10 14:25:59.720 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANPort()
Jan 10 14:25:59.741 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__DeleteAllTaggedPortsFromVlan()
Jan 10 14:25:59.757 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__DeleteAllUnTaggedPortsFromVlan()
Jan 10 14:25:59.770 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANMapping()
Jan 10 14:25:59.807 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__AddVLAN()
Jan 10 14:25:59.820 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANMapping()
Jan 10 14:25:59.859 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__AddVLAN()
Jan 10 14:25:59.872 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANMapping()
Jan 10 14:25:59.892 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__AddVLAN()
Jan 10 14:25:59.905 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__UpdateVLANMapping()
Jan 10 14:25:59.929 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__ExecuteCloseGlobalTransaction()
Jan 10 14:26:00.288 debug 74:46:A0:DE:04:88 collective: New connection toward 169.254.248.4:1056 returned: Connection refused (111).
Jan 10 14:26:00.970 err 74:46:A0:DE:04:88 mscconf: Could not configure group (0.11)
Jan 10 14:26:00.970 debug 74:46:A0:DE:04:88 mscconf: Called soap_call_PROCURVE_MOBILITY_MSM__ExecuteCloseGlobalTransaction()
Jan 10 14:26:00.977 err 74:46:A0:DE:04:88 mscconf: [22221] Could not actually configure the device.
Jan 10 14:26:00.978 err 74:46:A0:DE:04:88 mscconf: [74:46:A0:DE:04:B8] Error configuring slave msc
Jan 10 14:26:01.054 debug 74:46:A0:DE:04:88 mscconf: process <22221> terminated.
Jan 10 14:26:01.063 debug 74:46:A0:DE:04:88 maestro_master: [74:46:A0:DE:04:B8] report: error doing full configuration (event)
Jan 10 14:26:01.063 debug 74:46:A0:DE:04:88 mscconf: [74:46:A0:DE:04:B8] CFG_SYNC_FAILURE.
Jan 10 14:26:01.064 debug 74:46:A0:DE:04:88 collective: New connection toward 169.254.248.4:1056 returned: Connection refused (111).

Ctrl-2:

Jan 10 14:25:22.940 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:25:22.943 err pbserver: Failed to commit transaction (ID=<160>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:25:22.951 err pbserver: No global transaction opened
Jan 10 14:25:41.431 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:25:41.434 err pbserver: Failed to commit transaction (ID=<295>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:25:41.441 err pbserver: No global transaction opened
Jan 10 14:25:44.524 err cfg2openvpn-server: Could not get SC team member info from MAC address.
Jan 10 14:26:00.297 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:26:00.300 err pbserver: Failed to commit transaction (ID=<430>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:26:00.306 err pbserver: No global transaction opened
Jan 10 14:26:06.384 err cfg2openvpn-server: Could not get SC team member info from MAC address.
Jan 10 14:26:18.609 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:26:18.611 err pbserver: Failed to commit transaction (ID=<565>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:26:18.618 err pbserver: No global transaction opened
Jan 10 14:26:24.545 err cfg2openvpn-server: Could not get SC team member info from MAC address.
Jan 10 14:26:37.738 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:26:37.741 err pbserver: Failed to commit transaction (ID=<700>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:26:37.746 err pbserver: No global transaction opened
Jan 10 14:26:42.645 err cfg2openvpn-server: Could not get SC team member info from MAC address.
Jan 10 14:26:56.530 err confighandler: Validation error occurred in confighandler VLAN: Port orphaned. A Port must have at least one VLAN mapped to it..
Jan 10 14:26:56.532 err pbserver: Failed to commit transaction (ID=<835>, error=<Port orphaned. A Port must have at least one VLAN mapped to it.>).
Jan 10 14:26:56.544 err pbserver: No global transaction opened
Jan 10 14:27:01.876 err cfg2openvpn-server: Could not get SC team member info from MAC address.

GUI Alarms on manager Controller:

Severity: Major Major
ID: 1
Device: CN38F2F3C9
Category: Teamed Controller
Type: Controller is silent and presumably down
Latest associated event: ID 6569: Controller (name='74:46:A0:DE:04:B8') is silent and has been taken out of the team. Reason: (value='Secure connection to controller went down')
First occurrence: 2016-12-30 17:48:41,908
Last change: N/A
Cleared: N/A
Description: Controller (name='74:46:A0:DE:04:B8') is silent and has been taken out of the team. Reason: (value='Secure connection to controller went down').
Probable cause: The controller is not connected to the team anymore.
Recommended action: Make sure the device is powered and connected to the network.

[Arimo]

I don't see the logs, so it's a bit difficult to say. I'd suggest that you do the following:

1. Ensure that the Access and Internet interfaces are not members of the same VLANs in the connecting switch
2. Ensure that the teaming configuration is correct as per the Config and Deployment guides
3. Delete all AP groups not in use (you also might try re-creating the groups and deleting the original ones)
4. Save the config
5. Restore the member to factory default settings
6. Configure the appropriate IP address and subnet mask
7. Configure the appropriate Default Route
   1. Put the Default Route to Untagged VLAN of Access or Internet interface
   2. If the Default Route is pointing to another VLAN than those, remove it from the manager and let the team form - you can then re-configure the Default Route. I recommend keeping it in the Untagged VLAN tho', less headache.
8. Enable teaming

If that doesn't do the trick, you might have a corrupt config. The fact that the manager fails to configure group 0.11 for the member suggests this. You can then try:

1. Save current config
2. Restor both devices to defaults
3. Set the IP addressing and Def Route on both
4. Enable teaming and let them team form
5. Upload saved config

The team should form when both are on defaults - provided the IP addressing and port VLAN Membership is correctly done. If it does but fails after you upload your saved config, the best way is just to re-configure the wireless from scratch after the team is formed with default settings. With MSM720s I'd be inclined to skip the second option above and do this as if there's a corruption in the config it's not really possible to find and fix it.

[Enthusia]

Thank you for your help,

I checked all my settings again. And I changed Vlans config and then manager synced with member succesfully.

I think in my system, a group error of 0.11 is a vlan error. Member is not synchronized due to Vlan error.

I was not using "Ekbina-Misafir" and "Ekbina-Personel" vlans. And I disconnected the port 3. Then I untagged port 3 with "Access network". With these settings, the team is synchronized with each other. Only the "default" vlan was left to look wrong.

Lastly, I changed default vlans to "Access Network" with CLI commands.
(Ex.:
CLI(config)# network-profile "Access network"
CLI(network-profile)# default
CLI(network-profile)# end
)

Have a nice day.

Invalid ConfigChanged Config. Sync worked with this.Last config, with true default vlan setting.