MAC Address changes during SG Migration

[Al McKay]

I have an unusual issue. Whenever I migrate a HPVM using SG (Online Migration) The MAC address for the network interface changes. Any ideas/

[Steven E. Protter]

Shalom,

This is normal.

The floating IP address is a second virtual IP address that runs on the the same hardware (network card) as another IP on the system.

When the package is running on node A it gets a MAC address from node A. If its running on node B it gets the MAC address from node B.

SEP

[Al McKay]

OK Now here is the real issue, I am forced to manually add the MAC Address to the arp table to communicate with other HPVM's on Blades servers (860i2) on the same enclosure. I can communicate with HPVM's on the same blade as well as other servers (physical and virtual) outside the C7000 enclosure without any problems. Just having issues talking between HPVM's on different blades on the same enclosure. This does not seem correct.

[Bill Hassell]

It doesn't really matter where the machine is (real, virtual, same enclosure, etc). The MAC address is the lowest level of the communication protocol and the arp table's job is to remember where an IP address belongs. It is a cache and will eventually get refreshed and pickup the new MAC, but for immediate usage, you'll need to use arp -d to clear the cache or arp -s to change the MAC for this host on machines that need to talk to this new MAC address immediately.

[Al McKay]

I understand that but the underlying issue of no communication between HPVM's on different blades in the same enclosure is an issue. No ping, telnet, ssh, ftp, sftp. No communications whatwoever. That is why we are forced to manually add the MAC to the Arp Cache. I know this is a band-aid but what else can we do?

[Mel Burslan]

Without knowing your exact network topology and how your routers are configured, it is very hard to make a suggestion, but if you are having disconnects when the MAC address changes, this usually points to something on the arp table refresh not happening as fas as you expect it to be. Your network gear may be configured to issue a pre-emptive arp refresh command, every so many seconds or minutes, which will eventually take care of this problem, but at the same time, issuing a lot of unnecessary arp refreshes will take its toll on the network load. But till you figure out the reason why this is not taking place, you might consider the pre-emptive refresh option.

[Bill Hassell]

arp -s to all the blades and VMs that need an immediate update should be a trivial network event. The rest of the participants will see the new MAC after arp cache timeouts. I guess I would just add this to the packages.

[Emil Velez]

Did you specify a mac address with the virtual machine when you created the NIc. If you did not it will not have the same mac address when it starts on another node. You need to have specified a MAC address when you did the hpvmmodify to create the mach card.

[Bob_Vance]

Something doesn't seem right, here.

What is your network interconnect module?

When the MAC address changes, the network stack should send out a Gratuitous ARP, which lets the other devices on the network see the new MAC/IP combo and update or clear their cache entries.

In normal SG environment, when you move a Service Guard package, the VIP arp cache entry is immediately cleared on other servers on the network (not just other nodes in the cluster).


You say that other VMs in the other local blades have:

>No ping, telnet, ssh, ftp, sftp. No communications whatwoever.


When a 'ping', for example fails, is it the case that the arp cache has NO entry for the IP, or that it still has the OLD entry with OLD MAC address?

When you add the arp entry manually, are you using the "temp" argument?

If not, then the entry will be permanent.
So, perhaps you've built up a bunch of permanent entries that are now not changing normally?


bv

[Bob_Vance]

The HPVMs cannot communicate, but what about the main O/S hosting the VMs?


bv

[Bob_Vance]

BTW,
The arp behavior can be seen easily by following steps,
if you happen to have 2 servers to play with :


1) add a TEMP BOGUS arp cache entry for temp, unused IP

Lilly1 ## arp -s 172.16.99.99 0:1:2:3:4:5 temp


2) ping it on another server and dump arp cache

Lilly1 ## ping 172.16.99.99 -n 1 -m 3 | grep loss \
; arp -a | grep 99.99
1 packets transmitted, 0 packets received, 100% packet loss
172.16.99.99 (172.16.99.99) at 0:1:2:3:4:5 ether

Still BAD arp entry exists;
ping failed because arp entry wrong


3) add the temp IP as an alias on a second server

Pine4 ## ifconfig lan0:9 172.16.99.99 netmask 255.255.0.0


4) dump arp cache on first server again (NO ping)

Lilly1 ## arp -a | grep 99.99
<>

arp cache cleared because of GARP issued.


5) ping and check arp cache again

Lilly1 ## ping 172.16.99.99 -n 1 -m 3 | grep loss \
; arp -a | grep 99.99
1 packets transmitted, 1 packets received, 0% packet loss
172.16.99.99 (172.16.99.99) at 0:30:6e:f4:87:f8 ether

GOOD MAC now in cache because of normal arp,
and ping works


6) clear IP; check arp cache

Pine4 ## ifconfig lan0:9 0
Lilly1 ## arp -a | grep 99.99
172.16.99.99 (172.16.99.99) at 0:30:6e:f4:87:f8 ether

Still in cache, manually delete

Lilly1 ## arp -d 172.16.99.99
172.16.99.99 (172.16.99.99) deleted



NOTE: the default arp cache cleanup interval is 5 minutes:

Pine4 ## ndd -get /dev/arp arp_cleanup_interval
300000



bv