Network Adapter "NIC" Bottleneck??

[Jorge Cocomess]

Is there a way to find out if your network adapter is your bottleneck? I am running OpenVMS 7.3-2 TCPIP ECO 2 - Connected through a Gigabit HP 2800's series Switch. I can see that the server detected the adapter as 1000Mbps. On some day our system just flies through all the updates and some day just run a few hours longer. Btw, these are queries running against Oracle 10g DB.

I also have Polycenter runnning which I can pull some stats as well.

But please, direct me to where and what I can do do measure the suspected or potential network bottleneck.

Thank you in advance.

J

[Michael Yu_3 · Accepted solution]

Hi Jorge,

I do not think that it is related to bottleneck at the NIC.

Have you installed the patch VMS732_LAN-V0300? If yes, you can use LANCP> show device/int to check if there has been any duplex mismatch.

Thanks and regards.

Michael

[Hein van den Heuvel]

Agreed, unlikely to be a network bottleneck @1gb/s

Maybe you can share measured mb/sec and pckt/sec in busy windows, to confirm?

I would recommend T4 (google: +t4 +site:hp.com) to measure and graph system usage details in slow vs normal time windows.
It can give nice timelines mapping network as well as CPU and other data. And it can correlate those numbers.

Are you coming to the bootcamp? (again, google) T4 and many other performance plays will be discussed.

If the main load is Oracle, then I would recommend to also add statspack reports.
Those will tell you how much 'sql*net' action there is, how much waiting for client data and more data, or whether the execution itself was slow.

Hope this helps,
Hein.

[Email me if you think I can help beyond the scope of this Forum.]

[Jorge Cocomess]

Gentlemen,

Per Micheal request, here is a status on these adapters; Looks like my EWA0 is down - Btw, EWA0 is a 10/100 copper NIC.

Device Internal Counters EWA0:
Value Counter
----- -------
--- Internal Driver Counters ---
111 Driver version (X-n)
00000001 Driver flags
00001100 Device type
LinkDown Link status
2 Tulip reset count
5 Setup buffers issued
4 CSR6 changes
39 Transmit multiple addresses
309504 Device interrupts
1 Inits (not using map registers)
1458 Transmits issued (using map registers)
Off Auto-negotiation state
Off Autosense state
0:00:03.00 Transmit time limit
0:00:01.00 Timer routine interval
F0660004 Most recent CSR5 contents <31, 30, 29, TS2, TS1, RS1,
RS0, TU>
02ACE002 Most recent CSR6 contents TR0, ST, SR>
000000C6 Most recent CSR12 error contents <7, 6, LS10, LS100>
00050008 Most recent CSR15 contents
230998751 Current time (EXE$GL_ABSTIM_TICS)
--- Driver Messages ---
22-FEB-2006 05:29:55.92 FastFD mode set by console

Device Internal Counters EWB0:
Value Counter
----- -------
--- Internal Driver Counters ---
"DEGXA-TB" Device name
"Feb 9 2005 13:25:57" Driver timestamp
39 Driver version (X-n)
11000000 Device revision (Broadcom 5701,5703,5704 chip)
-740748270 Device interrupts
9 Link transitions
15 Link transitions avoided
214 Status block link state changes
2308303 Link checks
1 Device resets
1 Device initializations
10 User start/change/stop requests
58 Transmits queued
732355745 Receives issued (using map registers)
18 Rescheduled forks (too long in fork)
320 Standard receive buffers
8 Jumbo receive buffers (current)
8 Jumbo receive buffers (minimum)
8 Jumbo receive buffer allocations
2158 Standard buffer size (bytes)
1518 Standard packet size (bytes)(device standard ring)
9658 Jumbo buffer size (bytes)
9018 Jumbo packet size (bytes)(device jumbo ring))
000002A4 Requested link state Auto-negotiation>
000000A5 Current link state Link up>
00008090 Driver flags
00000008 Driver state
64 DMA width (bits)
66 BUS speed (mhz)
PCI BUS type
00000086 MSI control (Alloc<6:4>,Req<3:1>,Enable<0>)
16 Transmit coalesce value
16 Receive coalesce value
50 Transmit interrupt delay (usec)
10 Receive interrupt delay (usec)
5888 Map registers allocated
0:00:03.00 Transmit time limit
0:00:01.00 Timer routine interval
--- Registers (wrote/read) ---
110002B8 110000BA Misc Host Control
00E04F08 00E04F08 MAC Mode
0F40041C 00000003 MAC Status
00000001 00000001 MI Status
00001002 00001002 RX Mode
FFFFFFFF 00000008 TX Status
--- Time Stamps ---
641:39:47.52 Current uptime
0:00:46.97 Last reset
420:33:30.15 Last link up
420:31:30.45 Last link down
641:36:44.40 Total link uptime
0:03:03.12 Total link downtime
--- Driver Auto-Negotiation Context (fiber) ---
Not_Autoneg Current state
--- Status Block ---
00000017 Status tag value
00000001 Status
93 Receive Standard Consumer index
605 Receive Ring 0 Producer index
130 Send Ring 0 Consumer index
--- Statistics Block ---
----- Statistics - Receive MAC ---
3570002538716 Bytes received
9282758603 Unicast packets received
401052 Multicast packets received
16626677 Broadcast packets received
103115611 Packets (64 bytes)
199371071 Packets (65-127 bytes)
2538687096 Packets (128-255 bytes)
4202376048 Packets (256-511 bytes)
2025774969 Packets (512-1023 bytes)
230461540 Packets (1024-1522 bytes)
----- Statistics - Transmit MAC ---
1065773633315 Bytes sent
9104928700 Unicast packets sent
376088 Multicast packets sent
7849 Broadcast packets sent
----- Statistics - Receive List Placement State Machine
---
9299785655 Frames received onto return ring 1
9339 DMA write queue full
841 Inbound discards
1288 Receive threshold hit
----- Statistics - Send Data Initiator State Machine --
-
9105312637 Frames sent from send ring 1
160657 DMA Read Queue full
----- Statistics - Host Coalescing State Machine ---
9181025330 Send producer index updates
16439070607 Ring status updates
16438747125 Interrupts generated
323482 Interrupts avoided
1892456 Send threshold hit
--- Driver Messages ---
11-MAR-2006 18:04:17.66 Link up: 1000 mbit, full duplex, flow control disabled
11-MAR-2006 18:02:21.46 Link down
6-MAR-2006 21:35:33.36 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:35:32.32 Link down
6-MAR-2006 21:29:48.74 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:29:46.02 Link down
22-FEB-2006 05:30:02.93 Link up: 1000 mbit, full duplex, flow control disabled
22-FEB-2006 05:29:59.78 Device type is BCM5703C (UTP) Rev A0 (11000000)
22-FEB-2006 05:29:59.77 DEGXA-TB located in 64-bit, 66-mhz PCI slot
22-FEB-2006 05:29:59.77 Auto-negotiation mode set by console (EGA0_MODE)
LANCP>

Thanks,
J

[Michael Yu_3]

Hi Jorge,

Can we also see the counters?

LANCP> show dev/count

Thanks and regards.

Michael

[Volker Halle]

Jorge,

yes, your EWA0 is down and has never been up after boot. Is there a cable connected at all ?

T4 is probably the best tool to start with collecting data. It can collect data on your LAN interfaces, which - as far as I remember - Polycenter can't. It's also very easy to set up and it's free !

Start collecting data NOW and then have a look at the data and compare 'good' days and 'bad' days.

Note that performance analysis can be a complex job and may need needs lots of data, questions and answers exchanged, which is not always possible in a forum like this.

Volker.

[Ian Miller.]

T4 is in
http://h71000.www7.hp.com/openvms/products/t4/

or in SYS$ETC on newer VMS.

[Jorge Cocomess]

Good day, gentlemen!

Here's the stats after I ran the command sho dev/count on my Alpha server;

LANCP> sho dev/count

Device Counters EWA0:
Value Counter
----- -------
2365353 Seconds since last zeroed
0 Bytes received
0 Bytes sent
0 Packets received
0 Packets sent
0 Multicast bytes received
0 Multicast bytes sent
0 Multicast packets received
0 Multicast packets sent
0 Unrecognized unicast destination packets
0 Unrecognized multicast destination packets
0 Unavailable station buffers
0 Unavailable user buffers
0 Alignment errors
0 Frame check errors
0 Frame size errors
0 Frame status errors
0 Frame length errors
0 Frame too long errors
0 Data overruns
0 Send data length errors
0 Receive data length errors
0 Transmit underrun errors
0 Transmit failures
321770 Carrier check failures
0 Station failures
0 Initially deferred packets sent
0 Single collision packets sent
0 Multiple collision packets sent
0 Excessive collisions
0 Late collisions
0 Collision detect check failures
Characteristic #0x00C9, Value = 0000
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
Characteristic #0x0BF6, Value = 00A53081
Characteristic #0x00CF, Value = 00000000
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
Characteristic #0x0081, Value = 00000000
0 Seconds since last zeroed
0 Seconds since last zeroed

Device Counters EWB0:
Value Counter
----- -------
2365353 Seconds since last zeroed
3497553439267 Bytes received
1058834861029 Bytes sent
9544407751 Packets received
9344816014 Packets sent
1319983733 Multicast bytes received
28923907 Multicast bytes sent
17525066 Multicast packets received
393784 Multicast packets sent
1 Unrecognized unicast destination packets
1885300 Unrecognized multicast destination packets
0 Unavailable station buffers
0 Unavailable user buffers
0 Alignment errors
0 Frame check errors
0 Frame size errors
0 Frame status errors
0 Frame length errors
0 Frame too long errors
0 Data overruns
0 Send data length errors
0 Receive data length errors
0 Transmit underrun errors
0 Transmit failures
58 Carrier check failures
0 Station failures
0 Initially deferred packets sent
0 Single collision packets sent
0 Multiple collision packets sent
0 Excessive collisions
0 Late collisions
0 Collision detect check failures
Characteristic #0x00C9, Value = 0005
5 Seconds since last zeroed
5 Seconds since last zeroed
Characteristic #0x0D03, Value = 28BE
2935930974950842400 Abort delimiters sent
57549 Abort delimiters sent
10430 Seconds since last zeroed
57551 Abort delimiters sent
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
57557 Abort delimiters sent
10430 Seconds since last zeroed
10430 Seconds since last zeroed
LANCP>


I will try to install T4 sometime later this week to try it out.

Thanks so much!

Regards,
J

[Jim_McKinney]

YOu should try and determine why EWB0 has

> 58 Carrier check failures

this indicates a total loss of electrical connectivity. Either hardware is failing or the cable is being unplugged at one end or the other. That is not normal. I don't know the significance of "Abort delimiters sent".

(The carrier check failures that you see on EWA0 are likely a result of there not being a physical connection to the network and your system enabling protocols on the NIC - you might consider modifying your network configurations to disable whichever protocols are attempting to use the interface).

[Michael Yu_3]

Hi Jorge,

The carrier check failures should be worth investigating. In fact, the link downs should be caused by these.

--- Driver Messages ---
11-MAR-2006 18:04:17.66 Link up: 1000 mbit, full duplex, flow control disabled
11-MAR-2006 18:02:21.46 Link down
6-MAR-2006 21:35:33.36 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:35:32.32 Link down
6-MAR-2006 21:29:48.74 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:29:46.02 Link down
22-FEB-2006 05:30:02.93 Link up: 1000 mbit, full duplex, flow control disabled

Also it might be better if flow control is enabled. Currently it is disabled. You might need to set it up on the switch. The following inbound discards were resulted.

841 Inbound discards
1288 Receive threshold hit

These inbound discards probably happened during peak hours. They would cause TCP retransmissions and would slow things down.

Thanks and regards.

Michael