topic Re: LB_MAC vs. LB_PORT APA balancing with Oracle RAC in Operating System - HP-UX

LB_MAC vs. LB_PORT APA balancing with Oracle RAC

Jose M. del Rio — Mon, 02 Nov 2009 14:21:42 GMT

Hi,
we have a 2-node Oracle RAC cluster.
Each node uses a 2-NIC APA aggregate as Oracle RAC heartbeat.
We have noticed that only one NIC in the aggregate is being used:

node 1:
ensnada5:/root/JMR#lanadmin -g mibstats 2 | egrep -e 'Description|Outbound Oc>
Description = lan2 HP PCI 1000Base-T Release B.11.11.27
Outbound Octets = 22940
ensnada5:/root/JMR#lanadmin -g mibstats 4 | egrep -e 'Description|Outbound Oc>
Description = lan4 HP PCI 1000Base-T Release B.11.11.27
Outbound Octets = 3896665757

node 2:
ensnada6:/#lanadmin -g mibstats 3 | egrep -e 'Description|Outbound Octets'
Description = lan3 HP PCI 1000Base-T Release B.11.11.27
Outbound Octets = 2966334394
ensnada6:/#lanadmin -g mibstats 4 | egrep -e 'Description|Outbound Octets'
Description = lan4 HP PCI 1000Base-T Release B.11.11.27
Outbound Octets = 19592

The aggregates are configured with the default LB_MAC algorithm in /etc/rc.config.d/hp_apaconf:
HP_APA_INTERFACE_NAME[1]=lan901
HP_APA_LOAD_BALANCE_MODE[1]=LB_MAC
HP_APA_HOT_STANDBY[1]=off
HP_APA_GROUP_CAPABILITY[1]=4
HP_APA_KEY[1]=1

According to the 'HP Auto Port Aggregation Administrator's Guide', LB_MAC uses the
"least significant byte of the link level destination MAC address" as the hash key to choose the port. Therefore, in a 2-node RAC cluster we suspect all the traffic to the other node's heartbeat MAC address is being sent over the same port.
We wonder if this is really the case and if the LB_PORT algorithm is the recommended option for a 2-node cluster, as we gather from the Guide.

PS:
We have another 4-node RAC cluster, and the load is being distributed between the heartbeat ports.

Thanks.

Re: LB_MAC vs. LB_PORT APA balancing with Oracle RAC

DogBytes — Mon, 02 Nov 2009 15:19:44 GMT

I'm not an APA "expert" so others will no doubt step in to assist, but the following commands may help you:

#lanadmin -x -i 901
Link Aggregate PPA # : 901
Link Aggregation Mode : MANUAL
Load Balance Mode : MAC Address based (LB_MAC)
Active Ports PPA # : 0 2
Port(s) not ready : NONE

#lanadmin -s -m 901
MTU Size = 1500
Speed = 2000000000

the first shows both ports (0 and 2 in my case) are active, none are not ready.
The second shows the expected speed of 2 Gb/s. If only one port was active, it would be half.

Also, shouldn't your hp_apaconf have a line similar to the following?:
HP_APA_MANUAL_LA[1]="1,2"

Re: LB_MAC vs. LB_PORT APA balancing with Oracle RAC

Jose M. del Rio — Tue, 03 Nov 2009 11:26:16 GMT

>>>> Also, shouldn't your hp_apaconf have a line similar to...
I don't think so: we are using LACP_AUTO.

Re: LB_MAC vs. LB_PORT APA balancing with Oracle RAC

Jim Keeble — Tue, 03 Nov 2009 12:31:25 GMT

If all the outbound traffic from the aggregate is to the same destination (distinguished according to the LB algorythm), then it will all be sent on the same physical interface.

Since the destination aggregate is only using one shared mac address, there is only one "data flow" to use.

Will using LB_PORT result in more than one "data flow" to balance across multiple links ? Maybe, but only if the packets sent to the destination use more than one destination TCP or UDP port number.

On one system, run "netstat -in |grep X.X.X.X", where X.X.X.X is the other system's HB IP address. Under the foreign address column you will see X.X.X.X.Y . If there are different Y values, then the load can be balanced using LB_PORT. If only one, then it can't be.

Why do you see load balancing on your 4 node cluster ? Because you have 3 remote mac addresses to send HB to, not one.

Is this a real concern? Probably not, unless the outbound send rate aproaches one Gbit/sec at times. If it does, and there is only one data flow the only solution is to go to a higher speed device, perhaps 10Gbit Lan.