Receive buffer overrun on NIC (OpenVms Alpha 7.3.2)

[Swee Eng, Yeong]

Due to high traffic rate (TCP/IP), and possibly resource contention, receive buffer overrun was observed on the NIC (network device). Following was shown when running the command:
$mc lancp show dev /internal_counter eia0
--> "28 Receive overrun errors"

Current device min/max receive buffers are as below:
128 Minimum receive buffers
256 Maximum receive buffers

I expect tuning of above should help to reduce the overrun problem, but I couldn't find any recommendation on the tuning i.e. if there's any quota/limitation of what can be set and what's the impact of the increase of the buffer setting.

In addition, other possibility is resource contention. Any suggestion on tuning in this area?

Would appreciate if anyone can enlighten me on above.

Thanks in advance.
- Swee Eng

[Wim Van den Wyngaert]

28 is not much. In how much time ?

Wim

[Andy Bustamante]

Swee Eng,

As Wim points out, 28 is a trivial figure. I wouldn't be concerned unless this figure was increasing constantly. Increasing the receive buffers with LANCP uses a minimal amount of memory so there's no issue with doubling these if desired.

Do you have a performance issue you're attempting to troubleshoot? If TCPIP is the bulk of your network traffic, are you using the scalable kernel (on an SMP system).

Andy

[Swee Eng, Yeong]

Hi Wim and Andy,

Thanks for the prompt replies.

The overrun errors do increase daily, though not at alarming rate. Unfortunately, the application we developed is using UDP multicast as the data transport, hence, we can't afford even a single overrun. Until we upgrade the application to TCP, we're finding ways to eliminate any possible causes to the UDP message loss.

I need to check if we're using SMP model, but I believe should be with Scalable Kernel (AlphaServer ES45, TCPIP V5.4-15).

Best regards,
Swee Eng

[Duncan Morris]

Swee Eng,

note that the scalable kernel is not the default with TCPIP V5.4 - you have to enable it by setting

DEFINE/SYSTEM/EXECUTIVE TCPIP$STARTUP_CPU_IMAGES "PERF=ALL"

See the reference:-

http://h71000.www7.hp.com/DOC/732FINAL/TCP_RN/TCP_RNPRO.HTML


Duncan

[Volker Halle]

Swee Eng,

with TCPIP V5.4, you had to set a logical

$ DEFINE/SYSTEM/EXECUTIVE TCPIP$STARTUP_CPU_IMAGES "PERF=ALL"

for the scaleable kernel to be used. To find out, if you're running a SMP system, just try:

$ SHOW CPU

While you can try to increase the no. of receive buffers with LANCP> SET DEV/MIN_BUFFER=xxx EIA0 and see, if the problem goes away, to really understand this, one needs to find out, what this counter really indicates.

If there is a chance during this week, I'll ask the LAN engineers during the OpenVMS bootcamp...

Greetings from Nashua,

Volker.

[Swee Eng, Yeong]

Hi Duncan, and Volker,

Thanks a lot for the add-on information.

Just confirmed we didn't turn on the Scalable Kernel. I'll turn it on and see any improvement.

As for the SMP, following displayed using 'show cpu':
AlphaServer ES45 Model 2B
(there're 4 CPUs).

Any idea if model 2B == SMP?

Best Regards,
SweeEng

[Oberwagner]

Swee Eng,

Model 2B does not specify SMP

There are ES45 Models 1,2 and 3
Model 1 supports maximum of 16GB memory, Model 2,3 maximum of 32GB,
Model 1 has less PCI slots.
see http://h18002.www1.hp.com/alphaserver/download/html/alphaserver_brochure_0704.html

Check for SMP:
$ show cpu/full

and look for
SMP execlet = 3 : Enabled : Streamlined.

which is default for SMP capable systems (2 or more CPUs present).

Regards,
Gerhard

[Swee Eng, Yeong]

Hi Gerhard,

Cool! Confirmed the systems are SMP enabled.

However, just recall that we're using TCPWARE instead of UCX. Is TCPWARE using the same OpenVMS TCPIP Services as the underlying? I.e. the Scalable Kernel still applicable? If not, any idea any performance differences in using TCPWARE instead of UCX?

Thanks in advance!

Best regards,
SweeEng

[Andy Bustamante]

The performance kernel doesn't apply to TCPware. You can start by upping the buffers in LANCP. Another point to check is

$show mem/pool/full

Look for increases, a change from original to current size. If you're seeing growth, you might consider using autogen to tune the system. This may or may not have an impact.

Andy

[Volker Halle]

Swee Eng,

OpenVMS LAN engineering seems to be aware of similar issues and some work has been done in EIDRIVER for high load situations. Please consider to raise a call with HP.

I will discuss this problem in tomorrows network troubleshooting session at the OpenVMS bootcamp and report the results here.

Stay tuned,

Volker.

[Swee Eng, Yeong]

Hi Andy, and Volker,

Thanks for the add-on information.

I tried tuning up the buffer. So far so good, though notice that tuning a buffer on one NIC may improve the performance on the other one. Probably they share the same controller or buffer ... will check.

Meanwhile, looking forward for additional info that Volker will get.

Cheers,
Swee Eng

[Volker Halle]

Swee Eng,

could you please issue the command

$ MC LANCP SHOW DEV/INT EIA0

and attach the full output of this command as a .TXT file in your next reply ?

Volker.

[Swee Eng, Yeong]

Hi Volker,

Please find the attached.

FYI, I clear the counter daily in order to monitor any new increment. The attached reported 1 receive overrun. When this counter increment, I'll notice occurence of burst of message loss (UDP) in the application. Some other counters look suspicious as well "Number of rus no resources" and "Number of rus fork instead of restart", but have not able to find documentation explaining the counter.

Thanks a lot for helping.

Best regards,
Swee Eng

[Wim Van den Wyngaert]

On a GS160 server I have
- number of rus fork instead of restart at 30
- number of rus no resources 61

Uptime : 1 year, 3 other servers have their counters on 0.

fwiw

Wim

[Volker Halle · Accepted solution]

Swee Eng,

the suggestion from OpenVMS engineering is to increase the receive buffers to 512 using

LANCP> SET DEV/MIN=512/MAX=512 EIA0

You are most likely already running the driver with the fix for the problem, where the receiver was off until the next one-second timer. Make sure you're running the most recent SYS$EIDRIVER available for V7.3-2. For a definite answer, you need to raise a call with HP.

Volker.

[YEONG SWEE ENG]

Wim,
Thanks for sharing the stats.

Volker,
Thanks for getting the recommendtations from your local VMS LAN expert. I did forward my query to my local HP support here .... but apparently, replies that I get from this forum here is faster and more helpful :)

Though I already made the change in production 2 days ago (after some testing), your statement here further affirm I'm doing the right thing.

I'll close this thread here.
Before that, many thanks to all that has been so helpful.

Best Regards,
SweeEng

[Swee Eng, Yeong]

Increasing the NIC buffer seems improve the buffer overrun issue.