Performance issue: network card bottleneck?

[Dave Pinsker]

NOTE: I'm aware the sofware is unsupported - there will be an upgrade plan in place shortly.

Summary:
Question: How likely is a 10Mb server NIC to cause a bottleneck in an environment with clients with 100Mb NIC's and switched hubs?

Background:
My clients are reporting intermittent slowdowns in server response, when running a custom application. I haven't been called in during a big hit, so I can't be sure of exactly where the problem lies.

The Win2K clients populate their screens with forms data from Oracle Forms, which talks to an Oracle database on a server on the local subnet. While the server does appear to be busy during a reported slowdown, responsiveness at the command-line appears to be OK. There also looks to be plenty of memory.

Looking at the problem via a sniffer (EtherPeek NX), shows that there are "Slow Server Response" and "Slow Server->Client Throughput" problems approx. 8% of the time (averaged across all packets received), during average slowdowns. The network utilization as a whole is idling at <1%. A sampling taken during a hit, shows ping times are <10ms. We have no data transfer sample for the period.

So far, I think I'm looking at:

1. The server NIC. It's a 10Mb NIC in a 100Mb world. The "weakest link" rule may be kicking in... Although, according to lanadmin, "Deferred Transmissions" are <1% of Inbound Octets, if this is a fair measure.

2. The server CPU. Yes it's underpowered, but the box does appear responsive from the command-line during high-CPU activities.

3. Kernel Tunables. Shared memory and semaphores have been adjusted for Oracle, but the dynamic buffer cache % has not been throttled down. nfile is also fairly low.

4. The application. It's sending lots of little packets to get its job done (also represented by EtherPeek showing a couple of "Inefficient Client" problems).

The environment is:

- HP K220: 1 processor, 1.5Gb phys mem, 10Mb NIC
- HP-UX 10.10 (Y2K, and ARPA & LAN cumlative patched to June, 2000)
- Oracle 7.3.2.2
- Oracle Forms (used to populate *every* field in the Windows 2000 client's window) -- this is an older rev, which is not supported with Win2k
- switched hub w/100Mb ports

Thanks for your input,

_Dave_

[Vincent Fleming]

Your network sounds OK from your description of it...

But that K220... it and it's software (OS *and* database) are just absolutely ancient!

I would suggest taking a good long hard look at the server and what it's doing during those slowdowns... look for I/O bottlenecks and paging activity. These would probably be your most likely culprits.

Just because it appears responsive doen't mean it's not slow - I/O bottlenecks are like that; much of the time the CPU is loafing around waiting for the I/Os to complete (during a I/O bottleneck), so it may appear very responsive.

Good luck!

[A. Clay Stephenson]

There's really no way to answer this because you don't have enough data. I can tell you that if you left the dynamic buffer cache at the default max_pct of 50% then your buffer cache for a 10.x system is much too large with 1.5GB of memory. 10.x systems actually perform more poorly in most situation when buffer cache exceeds about 300MB or so. I think that you will find that a fixed buffer cache of about 300MB will be a good option (bufpages=76800). Because this is a database server, it is possible that the terrible template that set timeslice=1 rather than 10 might have been applied. That will cause all sorts of very unusual behaviors. You've really got to catch these beast when it's loaded and at least get some sar values before anything of value can be gleaned.

[Steffen Jaiser]

I think it's not the network, though I would really check if by chance the server is not connected to a hub (and not a switched hub or switch).
What you also should check - just to be sure - is the network traffic for the server link, because remember if 10 clients try to talk to the server a the same time each one would ideally only get 1 Mbps to the server. What can happen now is that if too many computers want to talk to the server a once and the switch on the server link runs out of buffer space it starts discarding frames that eventually have to be resent (well the loss has to be detected first).
Hope that helped.

[John Dvorchak]

A classic example of performance tuning. To determine where the bottle neck is you will have to follow the prescribed method of:

1. Make a baseline of the sytem.
2. Measure during peak times and off peak.
3. Examine the data and make recomendations

To address wether you need a new network card, I say it is like chicken soup, It couldn't hurt. The system WILL respond faster on a 100 mb link than it will on a 10. You will more fully utilize the available bandwith. A mistake admins and network people often make is to look at a 10 mb link in a 100 mb world and say "Look, you aren't using the bandwidth you have" because they will often see low utilzation percentages on the LAN. The fact of the matter is that 100 goes faster than 10.

[rick jones]

some of these may be helpful when diagnosing networking things:

annotated_netstat.txt
annotated_nettune.txt

from

ftp.cup.hp.com under dist/networking/briefs/

basically, you work your way down the layers looking at stats and then work your way across. checking the lanadmin stats on the server is goodness - checking the stats on the corresponding switch port would also be goodness

i would also reiterate the comments about the age of the system and the software. you could replace that thing with an rp2430 and comparatively fly

btw, "switched hub" is something of a strange term. something is either a hub - no traffic isolation, only half-duplex, or it is a switch - offers traffic isolation, may do full-duplex. some salescritters may have invented the term "switched hub" to sound interesting

[Dave Pinsker]

Thanks to all for your input.

To summarize my responses:

- Yes, I have taken a baseline of the machine under typical load, with another sampling to be taken on notice of a slowdown. Point taken on why the command-line would be responsive, while still encountering a slowdown.

- The SAM database server template was not applied, so timeslice remains at a more reasonable 10

- Wow. Was planning on recommending reduction of the dbc, but wasn't aware that too high a buffer cache % could affect 10.x response negatively - thanks.

_Dave_