Jumbo Frame on Oracle RAC - HP-UX

[Shabahang]

Hello,
I am about increasing Jumbo Frame on cisco 3560G catalist switch (rac inter connect).
is there any consideration about HP-UX 11.23 or the NIC cards?

[Bill Hassell]

Changing to jumbo frames (9000 byte payloads) requires support by the NIC cards as well as ALL the switches and routers in the path. Any location between the 2 computers (including the remote system) that does not handle jumbo frames will force the packets back to standard sizes.

http://en.wikipedia.org/wiki/Jumbo_frame

[rick jones]

If the NICs in your system running 11.23 support Jumbo Frames then you should be able to enable it with the lanadmin command. The -m option reports current MTU, the -M option is used to set the MTU.

As Bill points-out, the MTU should (must) be the same on all nodes in a given "broadcast domain" - that is to say all the systems connected to the same set of switches. Or put another way, all the systems on the "same side" of a router.

Now, whether and how much performance benefit you will see is another question. Are these 1 GbE NICs or 10 GbE NICs?

[Shabahang]

Thanks Rick,
I want to use the Jumbo Frame on Oracle RAC Interconnect , which is recommended by Oracle .
Now I have high Net cluster wait (cache fusion) .

[rick jones]

Unless either the sender or the receiver is CPU bound, enabling JumboFrame does not increase throughput, it only decreases CPU utilization.

In some cases, it can also increase latency as measured by a netperf TCP_RR test, unless one is running on a system that has bad interrupt coalescing heuristics (I've seen very poor "out of the box" latency under Solaris which may hide what follows).

Most NICs these days are "store and forward" - they bring the entire frame into the NIC before transmitting it on the wire. For say an 8KB message, if one is using a 1500 byte MTU, that will be 6 frames, and if they will have overlap between DMA into the NIC and transmission onto the network. If though one uses a 9000 byte MTU, there is no longer overlap (parallelism) between DMA'ing the frame (now 8+KB) into the NIC and transmitting it onto the wire.

As the saying goes, there is no such thing as a free lunch. And there could be something else entirely leading to the latency you are seeing. I merely point-out the potential issue with JumboFrames for completeness.

[John Jimenez]

Shabahang, did you implement this?  Did it help?  Sounds like you go some good info, but I see you did not assign any kudos.        We are replacing a switch this week and am thinking if making this change too.   We are running UX 11.23 on PA-RISC RP7420.       I am not sure if these cards can run Jumbo  Frames.