How to setup HP/UX to support direct routing load balancing?

[Wayne Yu_1]

We have need to setup HP/UX servers to work in the direct routing load balancing environment. The incoming traffic come to load balancer, load balancer modify the TCP packet header and send to each HP/UX server, and expect the HP/UX server to reply directly back to the client. We have been able to configure the Windows and Linux working properly with the load balancer, but just can not make HP/UX server working with it. They key thing is to have the loopback adapter answering the incoming connection request and process the request, but forward the outgoing packet through the real network interface. Thanks for any help!

[Steven E. Protter]

You can do port agregation.

It may be possible to have multiple lan interfaces with the same IP and with add in software you get load balancing.

It is possible to purchase a 4 port PCI LAN card from HP and Port Aggregation software and do load balancing that way.

4 Ports 1 IP address, four times the bandwidth.

You'll need a decent fast switch otherwise this might be a bottleneck.

SEP

[monasingh_1]

You could try
#ndd -h supported
and see if some of the parameters help you. I think there is one parameter which is related to route and how mahcine responds for outgoing trafic.

[Wayne Yu_1]

I had done
#ndd -set ip_forwarding
but no IP packets was forwarded from lo0:1 to the network card for outgoing. I would think anything Windows and Linux can do, HP/UX should be able to do it also, but just do not know if that assumption is true.

[monasingh_1]

check the doc ID A5093598

[Wayne Yu_1]

I did a search for this document on itrc site and got this:
"Your search for A5093598 produced no results!"

Where to find this document?

[rick jones · Accepted solution]

minor nit - it is HP-UX, not HP/UX :)

I'm surprised that a direct routing load balancer (also known as triangle routing, IIRC) would acually modifie the TCP header. I thought that it simply encapsulated the existing TCP segment in an Ethernet frame with the MAC address of the desired back-end server.

Anyhow...

Indeed, if you are on HP-UX 11.X (and perhaps sufficiently patched), it should be possible to assign the VIP to an lo0 alias (lo0:1, lo0:2 etc). You may need to make a rather restrictive netmask on that ifconfig to make sure that the routing code does not think that the remote clients are on that VIP (sub)network.

[Wayne Yu_1]

We are able to assign IP address to lo0:1 or lo0:2. But it seems the under layer code would not forward the packet to the real network card. The requirement is the box has only one network card still have to work with this schem. Why the ndd set ip_forwarding does not forward from loopback to the real network card? Maybe the problem is not forwarding the IP packets, but somewhere something stop this from working :-< and no way to tell where, since the network code is buried inside the kernel.

[Wayne Yu_1]

loopback adapter by its nature should only loopback inside box. If the loopback adapter could talk to the outside, we can also do a "route delete" to get rid the route, so it has to forward the packet to the real network interface to get out. However, the IP packets just would not leave the box...:-< TCPDUMP on the network showed no traffic comes out the HP-UX box (hope this time I typed right for the OS name :-)

[rick jones]

now we need to start getting into details - like the ip address and netmask you used on the ifconfig lo0:1, and the revision and patch level (ARPA transport patch) you are running.

[Wayne Yu_1]

We have a lot of patches applied to this 11.0 box. I just copy few lines here:
PHNE_19754 1.0 LAN product cumulative patch
PHNE_19754.LAN-RUN 1.0 LAN product commands fileset
PHNE_19754.LAN2-KRN 1.0 LAN product kernel fileset
PHNE_19754.NW-ENG-A-MAN 1.0 LAN product manpage fileset
# PHNE_19826 1.0 Built-in PCI 100BASE-T patch
PHNE_19826.LAN-RUN 1.0 LAN-RUN
PHNE_19826.LAN2-KRN 1.0 LAN2-KRN
# PHNE_19899 1.0 cumulative ARPA Transport patch


Then, the IP addresses are all internal test IP address. the real IP address on the ethernet card is 192.168.12.84. The virtual IP address on lo0:1 is 192.168.12.100, without specify the netmask. We also tried use netmask 255.255.255.255. Both ways do not work.

[rick jones]

There have been something like 20 Cumulative ARPA Transport patches since PHNE_19899 was released.

I tried a quick experiment on my 11.11 system, and adding a 192.168.1.123 lo0:1 alias to my interface at 192.168.1.46, and then manually setting an ARP entry in another system would have pings to the 192.168.1.123 address generate responses.

So, it is known to work on 11.11, and I believe it was supposed to work on 11.0, but you may need to get a bit more up-to-date on patches.

...time passes... I also just tried the same experiment on a pair of 11.0 systems installed off the December 2002 media and that test also worked. Try more up-to-date patches.

[Wayne Yu_1]

Because the TCP header must be changed, plain ping can not prove it will work. Since the reply from the server must send back to the client through HP-UX's real network interface -- to increase the hop count for the replay packet, I am not sure there is any way to test it with out load balancer. But I will try to get those patches and install them. We have the patch updated last October. It would be nice to know which patch is really needed, than download all the patches and install them all. Thanks.

[rick jones]

ok, lets backup a step - why does the _TCP_ header need to be changed? I've not heard of any direct routing load balancing requiring a change in the TCP header. my understanding is that with direct routing the TCP and IP headers remain unchanged from what the client first sent - since the client is talking to the VIP, and replies are going directly to the client from the back-end server with the VIP as the source I don't see what could possibly be changable in the TCP (or IP) header.

Now, if the LB were rewriting IP headers, that could change the pseudo-header and so the TCP checksum would have to change, but if the LB were rewriting the IP header, that wouldn't be direct routing and aliases on the loopback interface would not be involved - unless I've missed something.

can you try ping from the "far" side of the load balancer? it might be useful to verify that things at the IP level are/aren't working.

since you've repeated that the TCP header is changed, it might be good to check the TCP stats on the system (netstat -p tcp) to make sure that things like checksum failures and the like aren't being generated.

[Wayne Yu_1]

I just installed PHNE_26771 and from far side, I can ping the HP server to the VIP address 192.168.12.100 I added to the lo0:1.

However, to get direct routing work, it needs the reply from the real network address on the server, in my test, that is 192.168.12.84, not the 192.168.12.100 VIP. The reason for this is because the IP forwarding cause the hop number increased, so that the router in front of the local network will not think the VIP is direct connected, so it will continue talk to the LB not the server(s). We will conduct tests to see after those patches installed, would make any difference, I do hope it will work.

[rick jones]

ok, I think we have a non-trivial disconnect in how we each think that direct routing works.


here is how I think things like this are supposed to work :)

the client connects to a VIP. he does this by sending a TCP SYN segment to the VIP. a TCP connection is named by the four-tuple of local/remote IP local/remote port. this means that when a client sends to a VIP, he expects and requires the response to come from that VIP.


the TCP SYN is intercepted by the load balancer. with direct routing, the load balancer remembers the four-tuple, and passes along the SYN to one of the back-end servers - he does not alter the TCP or IP header since he will not see the reply (it is routed directly to the client). all he does is slap a fresh ethernet header onto the TCP SYN segment.

the SYN segment is received by the back-end server, and he processes it like it had some to him directly - sending a SYN|ACK with the VIP as the source IP address (since using his own source IP would not match what the client is expecting).


now, if direct routing were _not_ being used, the load balancer would indeed modify the destination IP address and then forward the TCP SYN segment. this works only when the back-end server is configured to use the load balancer as the router to reach the client - when the back-end server sendds the SYN|ACK with his "real" IP address, the load balancer re-writes the source IP address to match what the client is expecting. (and in this case, there is no VIP configured onto the back-end server since he never needs to see that address)

at least that is how I understand load balancers to work.

so, your saying that when using a VIP and direct routing requires the reply to come from the _real_ IP address of the server it makes me wonder if either you or I are confised as to how things are supposed to work :) there is no way I can see a TCP connection could be established successfully if the client sent to the VIP and the server replied from his real address. most non-TCP applications, if they care, expect replies to come from the IP address to which the request was sent (aka the VIP).

[Wayne Yu_1]

Thanks so much for your help. First of all, after apply the patch, it works! So there is some change in the patch that does not exist before. Hope whatever the change will remain in the HP-UX kernel never get drop out in the future version.

Secondly, I think we are seeing things from different angle, so we talk about different things. What you said is obsolutely correct. What I kept talking is that the direct routing need to modify the MAC address of the data frame to be able to re-route the data packet to the real server, and that MAC address can route packets to the real server can not be discovered by other equipment on the network.

It is great to have someone knowledgeable to help out on this! Thanks again!