netstat -nr <ip> question & understanding network slowness issue

[rveri]

Hi Gurus,

As oracle seeing some performance issue while sql query are connecting but saying waiting to get data from the 11.31 server where the db isrunning. The sql connections connect from another server to this server using port 1525

Somone told to check with "netstst -nr server_ip" and the 2nd column "output" supposed to be ZERO, not sure what that means ,

# netstat -nr
(lan1)-> input output (Total)-> input output
packets packets packets packets
4498521 273729592 134463343 273856025
15 10 29 24
14 8 32 26
26 20 40 34
8 6 22 20
17 11 31 25
31 19 45 33
#


- netstat -in # doesn't shows any error or collusion.
- ping response 0 ms.

Any idea, what the above netstat -nr ip , means and what else to check,

Thanks,

[rveri]

Attached the details in good format,
Please provide your suggessions.,

[Steven E. Protter]

Shalom,

Current Config = 100 Full-Duplex MANUAL

This means either the network switch infrastructure is set to 100 BaseT or the port configuration is wrong.

In the unlikely event that you are actually using a 100 BaseT card, you should consider using a 1000 BaseT card.

I recommend you first check with your network team on network infrastructure.

Your system configuration and equipment can be obtained by seeing documentation at http://docs.hp.com or:

ioscan

SEP

[Steven Schweda]

> [...] sql query are connecting but saying
> waiting to get data [...]

Is that a network problem, or is the server
slow in finding the data to return?

> Somone told [...], not sure what that
> means ,

Ask the fellow who told you to do it?

Does any other program show any of this
"network slowness"? Why did you decide that
the problem is in the network?

[rick jones · Accepted solution]

netstat -rn makes little sense. The -r option is to show the routing table. The -n option is to not try to convert IPs to names.

I suspect what is happening is the IP address at the end of the command line is overriding the -rn. You will I suspect see the same sort of output if you simply use:

netstat

I think it means the person giving you suggestions is at least slightly confused.

I've never been all that fond of hardcoding duplex settings - all it takes is one "helpful" person cleaning-up the switch to move the cable from one set hardcoded to one set autoneg. I like to keep everything autoneg. If there is indeed a duplex mismatch with your interface you should see an increasing error count in the output of:

lanadmin -g mibstats 1

You would probably also see increasing numbers of tcp retransmissions in the output of:

netstat -s -p tcp

One can take "snapshots" of those statistics, and then run the snapshots through beforeafter:

ftp://ftp.cup.hp.com/dist/networking/tools/

as in

lanadmin -g mibstats 1 > before
sleep 60
lanadmin -g mibstats 1 > after
beforeafter before after > delta

and then you will see how many of what happened in those 60 seconds.

[rveri]

Steven (SEP),
Thanks,

In the unlikely event that you are actually using a 100 BaseT card, you should consider using a 1000 BaseT card.

- I too agree to upgrade to 1000 BaseT cards. But this is old SD and no plan for upgrade yet.
- This is old system, and all the cards are 100Mbps , and it works well, Untill after the 11.11 to 11.31 upgrade done and having issues from apps side. But there is nothing wrong in the system or dmesg or syslog or other bottleneck.



nwmgr output:
Name/ Interface Station Sub- Interface Related
ClassInstance State Address system Type Interface
============== ========= ============== ======== ============== =========
lan0 DOWN 0x00557F4B3AA3 igelan 1000Base-T
lan1 UP 0x00557F5GA266 btlan 100Base-TX
lan2 DOWN 0x00557F5GA265 btlan 100Base-TX
lan3 DOWN 0x00557F5GA2A2 btlan 100Base-TX
lan4 DOWN 0x00557F5GAD41 btlan 100Base-TX
lan5 UP 0x00557F5GA2BA btlan 100Base-TX
lan6 DOWN 0x00557F5GA289 btlan 100Base-TX
lan7 DOWN 0x00557F5GA2A3 btlan 100Base-TX
lan8 DOWN 0x00557F5GA277 btlan 100Base-TX
lan9 UP 0x00557F5GA261 btlan 100Base-TX
lan10 DOWN 0x00557F5GA290 btlan 100Base-TX
lan11 DOWN 0x00557F5GA273 btlan 100Base-TX
lan12 DOWN 0x00557F5GA27A btlan 100Base-TX



-----------



Steven S,


Thanks,
Is that a network problem, or is the server
slow in finding the data to return? -- Doesnot looks to me like a server problem, but apps teams are saying oracle/sql sessions points to network wait, However they are working on it with enabling trace etc..

> Somone told [...], not sure what that
> means , -- that means one of the application team member.




Does any other program show any of this
"network slowness"? Why did you decide that
the problem is in the network?

- Well, no other program run slow, or there is no sign of slowness from os point of view, earlier when the netwrok duplex setting was wrongly set i.e Half duplex, that time I saw issues ( like ls -l used to hang ) , (kctune display used to hang ) etc, after making full-duplex it works perfect, I took several ignite backup over the network. So I have not decided it is a network problem, but oracle application point of view they are seeing network wait issue.



------------







Rick,
Thanks for the valuable inputs, seems to be these are really very good steps,

The switch side is configured as full duplex, so there is no harm keeping full duplex, and it is working fine,
Checked with lanadmin and found no error:

$ lanadmin -g mibstats 1 | grep -i err
Inbound Errors = 0
Outbound Errors = 0
Alignment Errors = 0
FCS Errors = 0
Deferred Transmissions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Internal MAC Receive Errors = 0

--



Also netstat -s -p tcp output shows it is not increasing very rapidly :


# netstat -s -p tcp | grep -i ret
19475 data packets (5900803 bytes) retransmitted
1295 retransmit timeouts

After 3/4 minutes: (not much load at this point)
# netstat -s -p tcp | grep -i ret
19476 data packets (5900832 bytes) retransmitted
1296 retransmit timeouts







$ netstat -s -p tcp
tcp:
275819988 packets sent
274834532 data packets (392872340182 bytes)
19475 data packets (5900803 bytes) retransmitted
907547 ack-only packets (94956 delayed)
0 URG only packets
286 window probe packets
180 window update packets
77918 control packets
135485743 packets received
133166866 acks (for 392873611425 bytes)
10965 duplicate acks
0 acks for unsent data
2418195 packets (2598897472 bytes) received in-sequence
113 completely duplicate packets (67717 bytes)
7 packets with some dup data (7059 bytes duped)
115439 out of order packets (117272183 bytes)
0 packets (0 bytes) of data after window
2 window probes
10739 window update packets
793 packets received after close
0 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
31676 connection requests
6561 connection accepts
38237 connections established (including accepts)
61520 connections closed (including 23349 drops)
22953 embryonic connections dropped
133146775 segments updated rtt (of 133146775 attempts)
1295 retransmit timeouts
6 connections dropped by rexmit timeout
286 persist timeouts
968 keepalive timeouts
644 keepalive probes sent
14 connections dropped by keepalive
0 connect requests dropped due to full queue
23264 connect requests dropped due to no listener
0 suspect connect requests dropped due to aging
0 suspect connect requests dropped due to rate

Also the beforeafter tools looks good, to get the netstat stats,
---------


Thanks everyone, now it seems apps are getting good speed. will update further.,

[rveri]

Also to add with Ricks reply,

I think it means the person giving you suggestions is at least slightly confused.

Rick, Thats seems true, on asking him what the $ netstat -nr means , he said not sure and told to ignore it.
Good catch, Thanks for the reply.