L2000 slow to answer

[Mariani Alberto]

Hi, All!

I'm having a problem with an L2000 regarding response times for networking services.
The situation is as follow: on this l2000 (hp-ux 11.11) we have our main database (oracle 8.1.7.4.0) and the web server (apache) used to interact with the users through some forms6i programs.

The problem is that, sometimes, the delays we experience during work sessions (for example, when calling a new form or the program has to show an error message) are awfully long, of the order of tens of seconds.

We checked with the Oracle guys the performance of the db and there is no problem with that (except when you ask for a very large dataset, all queries are processed within milliseconds).

After using other network services (telnet, ftp) for some time, i found the same problem: on connection there are sometimes when the machine answers after a lot of time.

This is not the case with ssh, however (could it be a problem with inetd? but that wouldn't explain apache's behaviour, since it's a separate service and is not run from inetd), as the connection is immediate.

As i said, our hp-ux is 11.11 with all the patches recommended by HP.

Any ideas on how to solve the problem?

Thanks,

Alberto

P.S. If needed, i can attach the complete list of complete patches.

[Steven E. Protter · Accepted solution]

If its really a network problem, this could be it:

lanadmin -x #

put in the number of the nic card.

If its supposed to be 100 BaseT Full Duplex and its not, check your switch settings.

Autonegotiate on Cisco and other switches makees HP NIC's unhappy. Have the sysadmin check it anyway, and hard code it to the settings you want explicitly without any auto negotiate.

Then boot your box and run lanadmin again.

You may find you are at half duplex.

If so, hard code your settings in /etc/rc.config.d/hpbtlanconf

Again, best idea is to boot and run lanadmin again.

About half the time people report these symptons, these steps fix it, more than half if Cisco built the switch.

Does Glance show network bottlenecks?

Are there entries in /etc/var/adm/syslog/syslog.log

The oracle alert log.

The apache error_log

I'm atttaching background performance data collection scripts. Crank them next time the problem comes up, you'll get great data and narrow things down.

SEP

[Michael Tully]

You can almost be certain that the problem lies with a mis-match for your lan card duplexing to your switch.

You can also do the change on-line before changing your appropriate /etc/rc.config.d file.

# lanadmin -x 0 (gives details)

# lanadmin -X 0 100FD (makes changes, allow 11 seconds for the change.

Regards
Michael
"When I have trouble spelling, it's called fat finger syndrome."

[George_Dodds]

I'm currently having the same issue with informix on an L2000 running 11.0.

From what i've found the cause of the problem is the local Cisco switch.

Just trying to harras the network admin at the moment to take a look.

If not i'll do it for him ;)

Cheers

George

[Mariani Alberto]

Thanks guys, i'm planning to try the more permanent solution asap (namely, at the end of the lunch break), as i've already hard coded the 100FD on the switch on the port of our l2000 (it was set to auto-neg), although the switch is a HP procurve 4104gl, not a cisco model.

For Michael, i tried to use the commands you specified, but this is wat i get:

l2000:/tmp>lanadmin -x 0
Current Config = 100 Half-Duplex AUTONEG
l2000:/tmp>lanadmin -X 0 100FD
Invalid PPA Number entry

(I'll set the parameter in the appropriate file, i was just interested in the output here).

Thanks,

Alberto

[Mariani Alberto]

For Steve:

you were exactly right: i hard coded the switch, rebooted the l2000 and poff! 100 HD :-(
Now i'm rebooting again after setting 100FD in the config file.

I hope it works.

Thanks

Alberto

[Pete Randall]

Mariani,

Your difficulty with the lanadmin command (bad PPA number) can be fixed by examing lanscan output to determine the proper instance (or PPA) number of your card and substituting that number for the 0 you used in the lanadmin command.


Pete

[Bill Hassell]

The LAN issue can be verified by just running lanadmin and looking at the stats. FCS, framing errors and collisions are all signs that the LAN is mismatched. Note that you also must set the switch to not autonegotiate too.

However, unless you are transmitting or receiving thousands of packets during these transactions, I would not expect many seconds of delays. Instead, it sounds like your DNS server is having problems (overloaded? erratic response time?). You can verify this by running nslookup when the system seems slow. An unresponsive DNS server will delay the lookup by about 20 seconds before going to the next server (or falling back to /etc/hosts). If this is the case, change /etc/nsswitch.conf to resolve using /etc/hosts first:

hosts: files [NOTFOUND=continue UNAVAIL=continue TRYAGAIN=continue] dns

Now put all the important IP addresses in the /etc/host file and see if the problem goes away. If it does, the current DNS server needs replacing or some serious maintenance. DNS is a critical service for most networks and you would be surprised at the volume of requests that take place. For instance, commercial network backup programs will use gethostbyname for EVERY file that is copied! Other programs (like telnet and rlogin daemons) ask for IP-to-name reverse lookup as a security measure. SSH probably work quickly because it uses keys to validate the connection rather than a DNS server.

[Michael Steele_2]

Regarding "...except when you ask for a very large dataset, all queries are processed within milliseconds..."

You want to monitor the DBWR process within top and note its utilization during these periods. In general, DBWR writes occur only when more data needs to be read into the SGA and there is too little free space in the database buffers.

The SGA supports communication between server and background processes and tracks resources shared by users (such as cache buffers, database and log buffers, locks, and dictionary caches. For each instance there is an SGA which can be defined within the INIT.ORA file.

Note memory consumption and increase the SGA. If you can't, add more RAM so you can. There is also the UGA or User Global Area which is apart of either the large pool or the shared pool. Look at v$sesstat and v$statname for these metrics and increase the large pool or shared pool to obtain better user performance.

From HP, use 'sar -b' in conjunction with sizing kernel parameters 'dbc_min' and 'dbc_max.' You want 100% hits within the cache.

[Mariani Alberto]

A few answer for everyone:

First of all, i managed to get the nic on 100 FD manual.

Secondly, for Pete Randall, i figured it out by looking at the comments in the /etc/rc.config.d/hpbtlanconf file. (by the way, "Mariani" is my family name)

For Bill, the fact is we are sending thousands of packets per second, since we (we as in 3 programmers) use the forms6i tools on remote X servers (Reflection X) on our PCs, not as client-server, and that generates a lot of traffic.
Our DNS works well, too (although i have to put up with a w2kAS machine for that).

For Michael Steele, while not the case at the moment, the information you provided is most interesting and i'll make sure to look into that asap.

Thank you all, guys.

Alberto

[Pete Randall]

My apologies, Alberto!


Pete