Replication partners fail after upgrades

[Keith Martin_3]

I just upgraded two systems, one from 2.1.4 to 2.2.9 and the other from 2.1.4 to 2.2.8.

The first was configured to replicate to the second.

The first one wouldn't connect to the second one after it was upgraded and the second system was still running 2.1.4.

So I figured I would upgraded the second system, that didn't help.

Has anyone seen issues between replication partners after upgrades like this?

Not sure how to troubleshoot this one.  I can ping between them, but can't get the replication partnership connected.

I get the message "Failed to contact replication partner.  Unable to reach replication service on..."

[jliu79]

Are you using Management IP to replicate or the data IP? I would contact Nimble support to take a look.

[Keith Martin_3 · Accepted solution]

Jason, thanks for replying.  The systems are configured to use the Management network.

We have been in touch with Nimble support and here's the situation.  Not sure if this was caused by the upgrade or just by the controller fail-over during the upgrade.

The systems are configured to replication across VPN tunnels and have NAT statements pointing to the management VIP from the outside.  For whatever reason after the fail-over instead of the system using the management VIP to transmit the data it started using the active controller IP to transmit the data, thus nullifying our NAT statement and breaking replication.

We've since made the necessary changes to NAT statements on the firewalls to reestablish communications and replication, but this leaves us in a state where in the event of another controller fail-over the system will no longer be able to communicate.

The Nimble engineer is looking into this issue and at this point the only workaround we have is to change the firewall NAT statements.  Prior to the upgrades the systems were using the VIP without issue.

I'll update again as I get more information.

[justinakajuice]

This gave me crap the past day so I wanted to post online what it was in my scenario:

When adding a replication partner, Name of the array is not the name of the array, but is actually the Group name. (Assuming leftover since 1.x versions where the actual name was used. Nimble has since changed to Group names.)

I tried a bunch of different configurations w/iSCSI and mgmt networks and it kept failing. I saw errors saying that the arrays were unreachable and that the arrays were in solo mode. 

Hope it helps someone, or even me in the future!

[Marc_BE]

Thanks for the post, we had the same issue and it saved us !