Backups to remote clusters no longer working after 4.0.1 upgrade.

[huie83]

Still working through this with support but progress is slow. 

I upgraded from 3.7.7 to 4.0.1 over the last few months. After I upgraded to 4.0.1 I discovered two oddities, wondering if anyone else has seen this? 

I have 2 clusters in different sites in the same federation connected with a PTP VPN. Full 10GB for all connections, MTU on all my internal networks are 9000. 

1) My clusters can no longer see if the remote clusters VM's are in storage HA. 

2) Remote backups no longer work. They start and then go to "queued" after a few minutes. They never finish. 

Nothing besides upgrade to 4.0.1 has changed that I can find. My networking engineer verified all connectivity looks fine to him. 

Support balked at the fact their tests from OVC management ports with a MTU of 9000 failed, which is correct. MTU on the PTP VPN is only 1500, it's always been this way. If you ping from local to remote OVC with MTU of 1500, it goes through. 

I posed the question if something had changed networkig wise in 4.0.1 but the engineer said not much besides Simplivity using fastpath for network routing. I don't believe that would cause this. 

[dhooley]

@huie83  can you confirm if the MTU is configured on the Management network to 1500 or to 9000? Using ifconfig to check the eth0 configuration.

While we only require an MTU of 1500 for management, it is critical that it is set to 1500 at every OVC in the federation. Otherwise you will have the nodes trying to communicate using 9000 cluster to cluster which can cause backups to Queue.

If you have a support case number, I can check on the case in more depth if you like.

[huie83]

Thank you for the info. I believe they are indeed set at 9000, when we run Zeus network testing for the federation they try and talk over 9000MTU and fail between clusters. Ping between the OVC's at opposite sites with an MTU of 1500 suceeds. Oddly, it's always been this way and it has been working the past 2 years fine until 4.0.1

So if I'm understanding correctly, there is a way on the OVC's to set the MTU to 1500? All of our NIC's on the hosts are 10GB so all the MTU's on the Vswitches and switch ports are set to 9000. 

The case number is 5356303032 if you would like to take a look. 

[dhooley]

Yeah exactly so regardless of whether your switches/Nic's are using 10GB uplinks etc, if we can only pass MTU 1500, then the OVC Management network (across the entire federation) should be configured to 1500 MTU. I think this will resolve your problems. To achieve this you will need to run the dsv-address-change command on each OVC, and I recommend you reboot the OVC once complete. Simply run:

dsv-address-change --interface Management --address <same Management IP currently in use> --mtu 1500

Rinse & repeat for all nodes. Highlight this to the engineer and they should be more than capable to assist. If not, have them reach out to me internally (David Hooley is my name), and I can guide them.

As for the fact that this occurred after an upgrade and worked previously, not sure I can provide an answer to this with the info we have, but it does seem odd. I would have expected this to not work in previous versions also.

Thanks.

[huie83]

Awesome, thank you very much for the info. I'll mention this info to my engineer.