topic Re: Possible Network Bottleneck in Operating System - HP-UX

Possible Network Bottleneck

Bobby Chamness_1 — Tue, 29 Oct 2002 05:19:39 GMT

We have a superdome 32-way all procs active Running SAP/PS there have been network performance problems. I have been using glance and the alarms, there are allot of network bottleneck alarms. Also netstat -a shows send queues as high as 8660. These stats are from lan10 which is 100mbit full duplex. Is there anything else to check or is this enough? Also I am recommending gigabit fibre or Auto Port Aggregation.

Re: Possible Network Bottleneck

U.SivaKumar_2 — Tue, 29 Oct 2002 05:23:14 GMT

Hi,
What application in running on your server ?.
Do you have enough Free memory ?

For TCP tuning for performance
Have a look at this document,

http://www.psc.edu/networking/perf_tune.html

regards,
U.SivaKumar

Re: Possible Network Bottleneck

U.SivaKumar_2 — Tue, 29 Oct 2002 05:25:19 GMT

Are you using fixed settings 100Mbit full duplex at swith end also ?.

Have you tried changing the switch port or switch and observed whether the problem exists ?.

Have tried changing the Network Interface of the server for isolating the problem ?

regards,
U.SivaKumar

Re: Possible Network Bottleneck

Bobby Chamness_1 — Tue, 29 Oct 2002 05:29:53 GMT

There is 32Gb Total and 15.2Gb free. SAP people soft with Oracle database.

Re: Possible Network Bottleneck

U.SivaKumar_2 — Tue, 29 Oct 2002 05:31:09 GMT

What is the output of this command ?
#netstat -in

regards,

U.SivaKumar

Re: Possible Network Bottleneck

Bobby Chamness_1 — Tue, 29 Oct 2002 05:32:16 GMT

lan10 1500 10.64.7.32 10.64.7.37 586935543 70 181391250 0 0

This is when there is no usage I'll put one on tommarow during peak usage.

Re: Possible Network Bottleneck

U.SivaKumar_2 — Tue, 29 Oct 2002 05:35:03 GMT

Sir,
You have not answered the questions in my second posting

regards,
U.SivaKumar

Re: Possible Network Bottleneck

Michael Tully — Tue, 29 Oct 2002 05:39:34 GMT

The 'FD' = Full Duplex.

Re: Possible Network Bottleneck

U.SivaKumar_2 — Tue, 29 Oct 2002 05:42:07 GMT

Hi,
You have given me details of your Server's network card.

I want speed details of the switch port whereyour server is connected ?.

By the by , FD stands for Full-Duplex

regards,
U.SivaKumar

Re: Possible Network Bottleneck

Bobby Chamness_1 — Tue, 29 Oct 2002 05:42:43 GMT

lan10 HP A5506B PCI 10/100Base-TX 4 Port [100BASE-TX,FD,MANUAL,

I beleive the FD stands for fixed correct?

Thanks,
Bobby :)

Re: Possible Network Bottleneck

Bobby Chamness_1 — Tue, 29 Oct 2002 05:53:10 GMT

I am fairly new at my present company and will have to get the info about the switch tommarow sorry about that.

Re: Possible Network Bottleneck

rick jones — Wed, 30 Oct 2002 18:35:53 GMT

Depending on the revision of glance, the network bottleneck alarms have a tendency to give false positives. They are better than they used to be.

The send queue in netstat -a output is simply how much data is queued to the socket buffer for which the local system is awaiting an ACKnowledgement from the remote. That could indeed mean there is a network bottleneck, but it does not say where in the network between the machines the bottleneck might happen to be.

The default SO_SNDBUF setting for HP-UX 11 is 32768 bytes, so an 8Kish value does not mean that the send socket buffers are full.

If a NIC is bottlenecked, it should show a consistently non-zero outbound queue length in its lanadmin statistics.

If a NIC is _really_ bottlenecked, you would see an increasing count of outbound discards. (Or inbound for the inbound path).

You might also check the netstat -p tcp statistics and see what the TCP retransmission and retransmisstion timeout rates are relative to the number of TCP segments sent.

Finally, I think the latest revs of glance can show the KB/s rates going through the interface(s) - so you might want to make sure you are up on the latest rev and see if those KB/s rates are close to the max for the NIC. It would also be nice to know what Glance is reporting as the packet per second rates.

Finally, while it would be unlikely with just a 100BT interface, is there by any chance a single CPU that is running at 100% utilization when these alarms are going-off?