throughput problem (FTP and oracle database application)

[Freddie Valdez]

Help pls!

Recently we built an database application on two HPUX 11 (N4000) servers, one has the apps and the other has the database Oracle 8.1.6.
the network cards and LANs setting are confirmed to be 100M bps, and users are happy with ftp rates between 8M bytes - 14M bytes per second.

Now we moved the database serve over VLAN with
inter ATM 90M bps and we found out that typical
Oracle batch job which normally run for one (1) hour now runs for 23 hours. Nothing has changed on the servers except the VLAN setup now.

We're looking at the TCP windows, MTU sizes if they need changing to adapt and to be optimized
for the VLAN setup. This is not production but clients are wary that their test apps are running way below expectations. I understand running SQL-net over VLAN or WAN for that matter may have some drawbacks but we need some specific actions to resolve these issues.

would greatly appreciate helps from anyone.

[harry d brown jr]

Freddie,

I'd look at the network way before I'd look at the server in too much detail. Do a traceroute from your server to determine how the packets are being routed. Is DNS still functioning correctly? A bad DNS setup or availability will hammer your server.

live free or die
harry

[Wilfred Chau_1]

lanadmin -x 0 (for lan0)

does lan0(or whichever lan) set at 100FD?

Also check your DNS, ftp uses reverse lookup
to authenticate connection.

[Wilfred Chau_1]

lanadmin -x 0 (for lan0)

does lan0(or whichever lan) set at 100FD?

Also check your DNS, ftp uses reverse lookup
to authenticate connection.

[Wilfred Chau_1]

lanadmin -x 0 (for lan0)

does lan0(or whichever lan) set at 100FD?

Also check your DNS, ftp uses reverse lookup
to authenticate connection.

[sven verhaegen]

reading you post there are 2 things that immediatly spring to mind

1/ I've seen it several times before on these kind of performance issues but the most likely problem is a duplex issue , there where changes to the VLAN settings so there is a slight posibility that the port settings where reset , if you are having a duplex issue that could result in a drop in performance up to 80% sometimes even worse then a 10Mb card ,there will be no visible errors but greath influence on th esystem , check the settings of you lancard with lanadmin then check the exact port settings on the switch , if it is duplex related than this is a uni-directional problem meaning from one side only will it be slow , but both machines involved could have bad settings so check sending host , switch ports for sender and receiver and receiving host

a second issue only exists in 11.x machines and is related to the lancard load , since 11.x hp implements icmp source quenching on the system , this is a mechanism to prevent buffer overflow on the lancards input buffer effectively stopping transmissions from thje other side waiting to clear the buffers , in some rare cases there have been notices of this functionality causing performance problems , as icmp source quenching isn't 100% needed for functionality , it was not implemented prior to 11 and 10.20 works fine , one could decide to disactivate it , if source quenching plays you should however see some messages in your systems syslofg or the receiving systems syslog on

"icmp source quench"


The Source Quench error is generated by a system (router or host) when it receives datagrams at a rate that is too fast to be processed.
A system is not required to send a source quench, even if it runs out of buffers and throws datagrams away.
If UDP (User Datagram Protocol) is used, then the datagrams are just dropped. This could result in data loss, because the application has no knowledge of whether the data it received was complete. Any application that runs under UDP will need to handle this condition.
Data transfer for TCP (Transmission Control Protocol) is slowed down for the connection, but data loss is not a problem because this
protocol takes care of the situation.
For HP-UX 11.0 and 11i, this is tunable and can only be enabled (1) or disabled (0). The default is enabled.In most cases, the source quench ICMP messages are just a nuisance so
disabling them does not harm the work of the system at all. Here is how to disable the ip_send_source_quench:

1. Check the current value:

# ndd -get /dev/ip ip_send_source_quench

2. Disable the sending of source quench ICMP messages:

# ndd -set /dev/ip ip_send_source_quench 0


-To ensure after reboot the parameter stays deactivated:

3. nddconf example entry:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_send_source_quench NDD_VALUE[0]=0

The Source Quench messages should stop after the ip_send_source_quench is disabled.

[Freddie Valdez]

Much thanks to Harry, Wilfred and Sven.
I have addressed each of the suggestions you guys gave one after another.
So far:
1. We don't use DNS for this particular config
2. lanadmin confirmed all the related NICs are
100M bps, FD, autonegotiate=off and this applies to switches and routers, ethernet/ATM which is part of the whole WAN.
3. the ICMP source quench errors are not in syslog.log and it's still enabled. do I disable it.

another interesting note: our ethernet sniffer and analyzer tool reported a MINOR issue about the MSS. it says:

Small Negotiated TCP Maximum Segment Size
Connection: TCP: 205.207.7.219:49959 <-> 205.207.7.221:20 (FTP Data)
The negotiated TCP Maximum Segment Size of 0 is less than the configured "Minimum Maximum Segment Size for TCP Connections", 1460.
Possible Causes and Actions to Take
The TCP/IP stack is an older protocol stack that assumes that when you cross a router the MSS should be 576 bytes.
Determine if an upgrade to the TCP/IP stack is available. This limitation causes performance limitations in all-LAN environments.
The TCP/IP stack has a hard-coded MSS.
Determine if an upgrade to the TCP/IP stack is available. The stack is a poor implementation if it uses a hard coded MSS.
One or both of the stations are configured to use the smaller MSS. This may be done on purpose to be more polite to other users and other protocols across WAN links.
If performance is acceptable, no action is required.
If performance is not acceptable, you may need to tune the TCP/IP stack in one or both of the stations to achieve better performance.

I did check my values, they are:
tcp_mss_def = 536
tcp_mss_min = 108
tcp_mss_max = 65495 and they are
identical on both servers.

Does these make any sense at all?

server a
ip_pmtu_strategy 1 (Strategy 1)
tcp_recv_hiwater_def 32768
tcp_recv_hiwater_lfp 65536
tcp_recv_hiwater_lnp 8192
tcp_xmit_hiwater_def 32768
tcp_xmit_hiwater_lfp 65536
tcp_xmit_hiwater_lnp 8192

server b
ip_pmtu_strategy 1 (Strategy 1)
tcp_recv_hiwater_def 32768
tcp_recv_hiwater_lfp 65536
tcp_recv_hiwater_lnp 8192
tcp_xmit_hiwater_def 32768
tcp_xmit_hiwater_lfp 65536
tcp_xmit_hiwater_lnp 8192

appreciate further advice :>