Network Adapter "NIC" Bottleneck??

[Jorge Cocomess]

Is there a way to find out if your network adapter is your bottleneck? I am running OpenVMS 7.3-2 TCPIP ECO 2 - Connected through a Gigabit HP 2800's series Switch. I can see that the server detected the adapter as 1000Mbps. On some day our system just flies through all the updates and some day just run a few hours longer. Btw, these are queries running against Oracle 10g DB.

I also have Polycenter runnning which I can pull some stats as well.

But please, direct me to where and what I can do do measure the suspected or potential network bottleneck.

Thank you in advance.

J

[Michael Yu_3 · Accepted solution]

Hi Jorge,

I do not think that it is related to bottleneck at the NIC.

Have you installed the patch VMS732_LAN-V0300? If yes, you can use LANCP> show device/int to check if there has been any duplex mismatch.

Thanks and regards.

Michael

[Hein van den Heuvel]

Agreed, unlikely to be a network bottleneck @1gb/s

Maybe you can share measured mb/sec and pckt/sec in busy windows, to confirm?

I would recommend T4 (google: +t4 +site:hp.com) to measure and graph system usage details in slow vs normal time windows.
It can give nice timelines mapping network as well as CPU and other data. And it can correlate those numbers.

Are you coming to the bootcamp? (again, google) T4 and many other performance plays will be discussed.

If the main load is Oracle, then I would recommend to also add statspack reports.
Those will tell you how much 'sql*net' action there is, how much waiting for client data and more data, or whether the execution itself was slow.

Hope this helps,
Hein.

[Email me if you think I can help beyond the scope of this Forum.]

[Jorge Cocomess]

Gentlemen,

Per Micheal request, here is a status on these adapters; Looks like my EWA0 is down - Btw, EWA0 is a 10/100 copper NIC.

Device Internal Counters EWA0:
Value Counter
----- -------
--- Internal Driver Counters ---
111 Driver version (X-n)
00000001 Driver flags
00001100 Device type
LinkDown Link status
2 Tulip reset count
5 Setup buffers issued
4 CSR6 changes
39 Transmit multiple addresses
309504 Device interrupts
1 Inits (not using map registers)
1458 Transmits issued (using map registers)
Off Auto-negotiation state
Off Autosense state
0:00:03.00 Transmit time limit
0:00:01.00 Timer routine interval
F0660004 Most recent CSR5 contents <31, 30, 29, TS2, TS1, RS1,
RS0, TU>
02ACE002 Most recent CSR6 contents TR0, ST, SR>
000000C6 Most recent CSR12 error contents <7, 6, LS10, LS100>
00050008 Most recent CSR15 contents
230998751 Current time (EXE$GL_ABSTIM_TICS)
--- Driver Messages ---
22-FEB-2006 05:29:55.92 FastFD mode set by console

Device Internal Counters EWB0:
Value Counter
----- -------
--- Internal Driver Counters ---
"DEGXA-TB" Device name
"Feb 9 2005 13:25:57" Driver timestamp
39 Driver version (X-n)
11000000 Device revision (Broadcom 5701,5703,5704 chip)
-740748270 Device interrupts
9 Link transitions
15 Link transitions avoided
214 Status block link state changes
2308303 Link checks
1 Device resets
1 Device initializations
10 User start/change/stop requests
58 Transmits queued
732355745 Receives issued (using map registers)
18 Rescheduled forks (too long in fork)
320 Standard receive buffers
8 Jumbo receive buffers (current)
8 Jumbo receive buffers (minimum)
8 Jumbo receive buffer allocations
2158 Standard buffer size (bytes)
1518 Standard packet size (bytes)(device standard ring)
9658 Jumbo buffer size (bytes)
9018 Jumbo packet size (bytes)(device jumbo ring))
000002A4 Requested link state Auto-negotiation>
000000A5 Current link state Link up>
00008090 Driver flags
00000008 Driver state
64 DMA width (bits)
66 BUS speed (mhz)
PCI BUS type
00000086 MSI control (Alloc<6:4>,Req<3:1>,Enable<0>)
16 Transmit coalesce value
16 Receive coalesce value
50 Transmit interrupt delay (usec)
10 Receive interrupt delay (usec)
5888 Map registers allocated
0:00:03.00 Transmit time limit
0:00:01.00 Timer routine interval
--- Registers (wrote/read) ---
110002B8 110000BA Misc Host Control
00E04F08 00E04F08 MAC Mode
0F40041C 00000003 MAC Status
00000001 00000001 MI Status
00001002 00001002 RX Mode
FFFFFFFF 00000008 TX Status
--- Time Stamps ---
641:39:47.52 Current uptime
0:00:46.97 Last reset
420:33:30.15 Last link up
420:31:30.45 Last link down
641:36:44.40 Total link uptime
0:03:03.12 Total link downtime
--- Driver Auto-Negotiation Context (fiber) ---
Not_Autoneg Current state
--- Status Block ---
00000017 Status tag value
00000001 Status
93 Receive Standard Consumer index
605 Receive Ring 0 Producer index
130 Send Ring 0 Consumer index
--- Statistics Block ---
----- Statistics - Receive MAC ---
3570002538716 Bytes received
9282758603 Unicast packets received
401052 Multicast packets received
16626677 Broadcast packets received
103115611 Packets (64 bytes)
199371071 Packets (65-127 bytes)
2538687096 Packets (128-255 bytes)
4202376048 Packets (256-511 bytes)
2025774969 Packets (512-1023 bytes)
230461540 Packets (1024-1522 bytes)
----- Statistics - Transmit MAC ---
1065773633315 Bytes sent
9104928700 Unicast packets sent
376088 Multicast packets sent
7849 Broadcast packets sent
----- Statistics - Receive List Placement State Machine
---
9299785655 Frames received onto return ring 1
9339 DMA write queue full
841 Inbound discards
1288 Receive threshold hit
----- Statistics - Send Data Initiator State Machine --
-
9105312637 Frames sent from send ring 1
160657 DMA Read Queue full
----- Statistics - Host Coalescing State Machine ---
9181025330 Send producer index updates
16439070607 Ring status updates
16438747125 Interrupts generated
323482 Interrupts avoided
1892456 Send threshold hit
--- Driver Messages ---
11-MAR-2006 18:04:17.66 Link up: 1000 mbit, full duplex, flow control disabled
11-MAR-2006 18:02:21.46 Link down
6-MAR-2006 21:35:33.36 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:35:32.32 Link down
6-MAR-2006 21:29:48.74 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:29:46.02 Link down
22-FEB-2006 05:30:02.93 Link up: 1000 mbit, full duplex, flow control disabled
22-FEB-2006 05:29:59.78 Device type is BCM5703C (UTP) Rev A0 (11000000)
22-FEB-2006 05:29:59.77 DEGXA-TB located in 64-bit, 66-mhz PCI slot
22-FEB-2006 05:29:59.77 Auto-negotiation mode set by console (EGA0_MODE)
LANCP>

Thanks,
J

[Michael Yu_3]

Hi Jorge,

Can we also see the counters?

LANCP> show dev/count

Thanks and regards.

Michael

[Volker Halle]

Jorge,

yes, your EWA0 is down and has never been up after boot. Is there a cable connected at all ?

T4 is probably the best tool to start with collecting data. It can collect data on your LAN interfaces, which - as far as I remember - Polycenter can't. It's also very easy to set up and it's free !

Start collecting data NOW and then have a look at the data and compare 'good' days and 'bad' days.

Note that performance analysis can be a complex job and may need needs lots of data, questions and answers exchanged, which is not always possible in a forum like this.

Volker.

[Ian Miller.]

T4 is in
http://h71000.www7.hp.com/openvms/products/t4/

or in SYS$ETC on newer VMS.

[Jorge Cocomess]

Good day, gentlemen!

Here's the stats after I ran the command sho dev/count on my Alpha server;

LANCP> sho dev/count

Device Counters EWA0:
Value Counter
----- -------
2365353 Seconds since last zeroed
0 Bytes received
0 Bytes sent
0 Packets received
0 Packets sent
0 Multicast bytes received
0 Multicast bytes sent
0 Multicast packets received
0 Multicast packets sent
0 Unrecognized unicast destination packets
0 Unrecognized multicast destination packets
0 Unavailable station buffers
0 Unavailable user buffers
0 Alignment errors
0 Frame check errors
0 Frame size errors
0 Frame status errors
0 Frame length errors
0 Frame too long errors
0 Data overruns
0 Send data length errors
0 Receive data length errors
0 Transmit underrun errors
0 Transmit failures
321770 Carrier check failures
0 Station failures
0 Initially deferred packets sent
0 Single collision packets sent
0 Multiple collision packets sent
0 Excessive collisions
0 Late collisions
0 Collision detect check failures
Characteristic #0x00C9, Value = 0000
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
Characteristic #0x0BF6, Value = 00A53081
Characteristic #0x00CF, Value = 00000000
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
0 Seconds since last zeroed
Characteristic #0x0081, Value = 00000000
0 Seconds since last zeroed
0 Seconds since last zeroed

Device Counters EWB0:
Value Counter
----- -------
2365353 Seconds since last zeroed
3497553439267 Bytes received
1058834861029 Bytes sent
9544407751 Packets received
9344816014 Packets sent
1319983733 Multicast bytes received
28923907 Multicast bytes sent
17525066 Multicast packets received
393784 Multicast packets sent
1 Unrecognized unicast destination packets
1885300 Unrecognized multicast destination packets
0 Unavailable station buffers
0 Unavailable user buffers
0 Alignment errors
0 Frame check errors
0 Frame size errors
0 Frame status errors
0 Frame length errors
0 Frame too long errors
0 Data overruns
0 Send data length errors
0 Receive data length errors
0 Transmit underrun errors
0 Transmit failures
58 Carrier check failures
0 Station failures
0 Initially deferred packets sent
0 Single collision packets sent
0 Multiple collision packets sent
0 Excessive collisions
0 Late collisions
0 Collision detect check failures
Characteristic #0x00C9, Value = 0005
5 Seconds since last zeroed
5 Seconds since last zeroed
Characteristic #0x0D03, Value = 28BE
2935930974950842400 Abort delimiters sent
57549 Abort delimiters sent
10430 Seconds since last zeroed
57551 Abort delimiters sent
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
10430 Seconds since last zeroed
57557 Abort delimiters sent
10430 Seconds since last zeroed
10430 Seconds since last zeroed
LANCP>


I will try to install T4 sometime later this week to try it out.

Thanks so much!

Regards,
J

[Jim_McKinney]

YOu should try and determine why EWB0 has

> 58 Carrier check failures

this indicates a total loss of electrical connectivity. Either hardware is failing or the cable is being unplugged at one end or the other. That is not normal. I don't know the significance of "Abort delimiters sent".

(The carrier check failures that you see on EWA0 are likely a result of there not being a physical connection to the network and your system enabling protocols on the NIC - you might consider modifying your network configurations to disable whichever protocols are attempting to use the interface).

[Michael Yu_3]

Hi Jorge,

The carrier check failures should be worth investigating. In fact, the link downs should be caused by these.

--- Driver Messages ---
11-MAR-2006 18:04:17.66 Link up: 1000 mbit, full duplex, flow control disabled
11-MAR-2006 18:02:21.46 Link down
6-MAR-2006 21:35:33.36 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:35:32.32 Link down
6-MAR-2006 21:29:48.74 Link up: 1000 mbit, full duplex, flow control disabled
6-MAR-2006 21:29:46.02 Link down
22-FEB-2006 05:30:02.93 Link up: 1000 mbit, full duplex, flow control disabled

Also it might be better if flow control is enabled. Currently it is disabled. You might need to set it up on the switch. The following inbound discards were resulted.

841 Inbound discards
1288 Receive threshold hit

These inbound discards probably happened during peak hours. They would cause TCP retransmissions and would slow things down.

Thanks and regards.

Michael

[Jorge Cocomess]

Gentlemen,

I do remember now....We had problems with one of our HP Procurve switch about 2-3 weeks ago with a data storm, which shut down one of the port on the switch. I had to reset the switch and since then, I have updated to the lastest and greatest patches. Well, so far so good. The kink up and link down you're seeing is that when I was updating the switch firmware couple of weeks ago. I don't think these errors are registering any longer. I can look at it again tomorrow to see if I still gettinh these errors again.

Is there away for me to reset the stats on the NIC without rebooting the server?

Thanks much!!

J.

[Jorge Cocomess]

Michael - How do I enabled the flow control? Also, where or how did you came up with 841 Inbounds, and 1288 Received threshold hit?

With the 1288 Received threshold hit the limit, shouldn't that be a major concern? Where do I increase or push the threshold back?


Thanks for your time.

J

[Michael Yu_3]

Hi Jorge,

The requested link state of the controller was set to

The current link state is

So flow control has been disabled. Since we requested to have flow control and the negotiated result was no flow control, it should be the switch port which was set to no flow control. Hence to enable flow control, you have to configure the switch port to have flow control.

The 841 inbound discards and 1288 receive threshold hit came from the output of LANCP> show device/int.

I am not too concerned about the receive threshold hit, but the inbound discards have to be dealt with.

If flow control is enabled and the receive threshold is hit, a pause frame will be sent to the switch port. But flow control has been disabled.

Thanks and regards.

Michael

[Jim_McKinney]

> Is there away for me to reset the stats
> on the NIC without rebooting the server?


LANCP> SET DEVICE/DEVICE_SPECIFIC=FUNCTION="CCOU" devname

[Wim Van den Wyngaert]

I would first investigate what the system is doing : who is eating the cpu / IO and is the consumption normal. Check what the top images are.

Wim

[Jorge Cocomess]

Hi,

I thought turn on control flow at the switch was bad. I am wrong or I just missed read or something?

Thanks,
J

[Karl Rohwedder]

Jorge,

you know the manual SYS$HELP:LAN_COUNTERS_AND_FUNCTIONS.TXT?

It explains the device specific internal counters and commands such like CCOU.

regards Kalle

[Richard Brodie_1]

"I thought turn on control flow at the switch was bad. I am wrong or I just missed read or something?"

There is certainly a common view that nesting another flow control algorithm inside TCP is generally a bad idea.

[Michael Yu_3]

Hi,

The flow control between the switch port and the controller is at the datalink level while the TCP flow control is at the transport layer.

Flow control can be enabled on a per port basis at the switch.

Pause is a mechanism for full duplex flow control which is discussed in IEEE 802.3 Annex 31B.

Thanks and regards.

Michael

[Richard Brodie_1]

Michael,

I am aware what port based flow control is. I said it was a common view that it is a bad idea. This blog article is a well-argued example of a proponent of that view:

http://virtualthreads.blogspot.com/2006/02/beware-ethernet-flow-control.html

I am not saying I believe you were wrong in suggesting that it should be enabled, merely noting that (in the general case) it is a contentious area and a topic of active research.

I'm thus not surprised that Jorge heard it was "bad". Now he's heard another view.

[Jorge Cocomess]

Gentlemen,

Would it be possible to have NIC teaming on Alpha servers? If not, would it possible to have multiple network connections to the GB switch for one single Alpha server? I believe the bottleneck that I'm seeing could possibly be the single network conecction between our back-end Oracle DB servers. Here's a little background of our network topology:

1) Application Server (Alpha - OpenVMS 16GB memory 4x1.0Ghz)
4 paths to the HP SAN storage with Shadow Copy. One (1) single connection to the Gbit HP Network Switch.

2) Oracle DB servers running Linux RHAS 3.0 w/Oracle RAC - One Single connection to 3 HP MSA1500 storage and through a 2GB SAN switch. The configurations on the MSA1500 SANs are all RAID10 (8x72) 15K RPM across 3 MSA1500s.

3) The front-end is a Windows application interface between the client desktop and the application (VMS) - The queries are being called from the VMS server and the results are being stored on the VMS server. The files will then be retreived by Pathworks.

My suspicion is in the network because the only way for these servers communicate to each other is by the network connection.

Any other ideas, I would love to hear more.

Thanks,
J

[Volker Halle]

Jorge,

as I've already said, performance analysis may be a complicated process.

AFAIK, T4 is the simplest tool to use, which will at least collect the most basic performance data on an OpenVMS system. And it includes statistics on the LAN interfaces, which most other tools don't. It provides simple ZIPped .CSV files as output, which you could even post here and provide REAL data for the experts to look at.

Everything else than real data only leads to suspicions, speculations and may be a waste of time.

You said yourself, that the system sometimes works very well and sometimes it works slowly. Compare the T4 data from both such days or time frames and it should become much more obvious, what the system is doing.

Volker.

[Jorge Cocomess]

Volker - How difficult is it do install T4 on the Alpha server?

Thanks,
J

[Volker Halle]

Jorge,

installing and setting up T4 is really easy. Getting the performance data and looking at it is also easy using the TLVIZ tool. But analyzing the data, understanding
the underlying problems and drawing the right conclusions is where you may need the help of an expert...

On OpenVMS V7.3-2, you'll find a T4 V3.4 kit and a SYS$ETC:T4*.TXT file, which contain the installation and setup instructions.

You might want to download the most recent T4 kit (V4.0) and TLVIZ from the hp T4 home page:

http://h71000.www7.hp.com/OpenVMS/products/t4/index.html

This page also includes a FAQ with additional information. Just install and configure T4 and start collecting data now...

Volker.

[Hein van den Heuvel]

If you have a chance to get to the OpenVMS bootcamp... Steve Lieman and friends will do several sessions on T4. (I'll help out with some).

Like Volker suggests, get collecting and looking now. Get to understand the 'normal' picture, and then learn to interpret the signs for a problem.

Hein.