topic Re: Suddenly sourrce quench problems ... in Operating System - HP-UX

Suddenly sourrce quench problems ...

Dr. Peer-Joachim Koch — Wed, 20 Aug 2003 14:53:50 GMT

Hi,

a few days ago our hpux file server started
to responed a ping with a source quench message.
Even when the load is very low (using glance
and netstat to monitor the network traffic).
Also the switch shows a very low load on the
port.

What is normaly the reason for this kind of
responce ?
How can I check it ?
I used
ndd -set /dev/tcp ip_send_source_quench 0
to deactivated the messages, otherwise our
monitoring system will fail.

Hardware L-class, 2 CP,1 GB memory, HPUX 11.0
running nfs and samba.

Thanks for your help.

bye, Peer

Re: Suddenly sourrce quench problems ...

Ken Hubnik_2 — Wed, 20 Aug 2003 14:57:34 GMT

This sounds like the classic case of your
switch and your NIC card out of sync with
the full Vs half duplexing.

Run this to check depending on your card ID

# lanadmin -x lan0
or
# lanadmin -x 0
Current Speed = 100 Full-Duplex Auto-Negotiation-OFF

If there is a mis-match between the two, you can fix it by doing this.

# lanadmin -X 100FD 0 (100 full duplex)

Re: Suddenly sourrce quench problems ...

Stefan Farrelly — Wed, 20 Aug 2003 15:01:55 GMT

There was a patch for this;

PHSS_27962 (for 11.0)

This patch lists one of the problems it fixes as;
JAGaa35855 : rpcd at 11.0 produces unwanted ICMP source quench messages.

Re: Suddenly sourrce quench problems ...

Ron Kinner — Wed, 20 Aug 2003 19:08:13 GMT

Well known problem with 11.0. As Stefan said there is a patch for it but you can also just turn it off in NDD.

ndd -set /dev/ip ip_send_source_quench 0

It's obsolete and serves no real purpose any more. Also so it will stay after a reboot, edit /etc/rc.config.d/nddconf to add:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_send_source_quench
NDD_VALUE[0]=0

Use the next higher integer in the bracket if you already have entries.

Ron

Re: Suddenly sourrce quench problems ...

Dr. Peer-Joachim Koch — Thu, 21 Aug 2003 11:01:43 GMT

Hi,

it look's funny:
bash-2.03# lanadmin -x lan1
Current Speed = 100 Full-Duplex Auto-Negotiation-ON

But it is a gigabit interface !

However, I disabled the source quench and will
apply the patch at the next maintance day.
We are currently trouble shooting our network
diveces, and suddenly this problem occured,too.
So I'm still not sure, if it is somehow related
to our network hardware...

Many thanks for the quick help!

Bye, Peer

Re: Suddenly sourrce quench problems ...

Rita C Workman — Thu, 21 Aug 2003 11:12:14 GMT

Dr Koch,

Auto negotiation has never, in my experience, worked as it should. We have found it better to turn it off.......

Rgrds,
Rita

Re: Suddenly sourrce quench problems ...

Ron Kinner — Thu, 21 Aug 2003 13:55:10 GMT

I think

lanadmin -x lan1

should be

lanadmin -x 1

since the man calls for the PPA number and not the name.

You are probably missing a patch

"PHNE_22353:
1. When the PPA number given as a lanadmin argument
is wrong, lanadmin displays the information about the
interface with PPA number 0."

Included in PHNE_28143

Patch Description: s700_800 11.00 LAN product cumulative patch

You really ought to have the cumulative patch. There are a lot of changes which effect the gigabit LAN.

To be sure what you are looking at run
lanadmin
lan
ppa 1
display

On the source quench issue I had a link to an article from HP, Doc id# 200000049809940, which explained what the problem was but unfortunately it is no longer good and their search engine is hopeless but I kept a copy in an email archive. This should make you feel better about turning it off:

"PROBLEM
Upon pinging an 11.0 system, I am seeing a packet loss and Internet
Control Message Protocol (ICMP) source quench messages.

Why am I getting these messages?

CONFIGURATION
Operating System - HP-UX
Version - 11.0
Subsystem - Internet Control Message Protocol (ICMP)
RESOLUTION
ICMP source quench messages are generated when an IP packet is
received by the 11.0 system that can't be delivered to the socket
buffer of the receiving application. The intent is to inform the
sender of the full buffer condition so the rate of the transmission
is slowed down until the buffer can be read by the receiving
application.

Setting the ndd parameter ip_send_source_quench to 0 can be an
effective way to deal with the messages.

Programs that use icmp protocol, such as ping, use a type of
socket called SOCK_RAW. The nature of using raw IP sockets is
that ALL packets received that match the protocol type of the raw
socket are delivered to ALL the sockets using that protocol. It is
up to the application to read all the data in it's socket buffer
and discard the data it's not interested in. If any of these
sockets are full, the icmp source quench message will be generated.

One process that uses one of these sockets is part of DCE, and it
is 'rpcd'. This program opens a raw socket in order to listen for
icmp messages, which it uses to monitor the health of other systems
on the network running DCE. In this case 'rpcd' used a 32K buffer,
and processed the messages received every 5 minutes, which led to
the buffer full condition.

PHSS_17810 addresses the problem by increasing
the buffer size to 128K and processing the messages every 2 minutes."

No one uses source quench these days anyway. It is not a problem - it just covers up the real problem. Have your router guy (assuming the local router is a Cisco) run an extended ping on each interface and let it sweep a range of sizes. The result should be all !'s. If you get any .'s mixed in with the !'s then there is a problem - probably with the NIC, tho it could be a switch port or bad cable. On ours it was that the builtin NICs were sensitive to EM interference.

Check your lanadmin's second page of the display for errors which could indicate a duplex mismatch tho I think that is unlikely on a gigabit.

Ron

Re: Suddenly sourrce quench problems ...

Dr. Peer-Joachim Koch — Thu, 28 Aug 2003 13:39:15 GMT

Hi,

I've tried to intsall the patch bundle for the source quench problem. I installed it first on our compute server (N-class 4 cpu; only compilers + memory, the rest is the same as described before).
Installation was fine (some patches were already there) -> new kernel -> reboot, booting the system, but when the two FC-HBA's were activated hte machine goes back to 0.
On the console a few error messages were shown(see below) and everything started again.
I could boot into single user mode, but can't
see the problem.

While I was doing some other work the system
booted finaly after 3h and endless tries.
The onyl difference was the full file system under /var - it could write the dump.

Any idea how to debug the machine ?
************* SYSTEM ALERT **************
SYSTEM NAME: c_bonita
DATE: 10/18/2002 TIME: 08:36:11
ALERT LEVEL: 3 = System blocked waiting for operator input

REASON FOR ALERT
SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

LEDs: RUN ATTENTION FAULT REMOTE POWER
ON FLASH OFF ON ON

0xF8E038301100E000 00000000 0000E000 - type 31 = legacy PA HEX chassis-code
0x58E038301100E000 00006609 1208240B - type 11 = Timestamp 10/18/2002 08:36:11
A: ack read of this entry - X: Disable all future alert messages
Anything else skip redisplay the log entry
->Choice:
*****************************************

************* SYSTEM ALERT **************
SYSTEM NAME: c_bonita
DATE: 10/18/2002 TIME: 08:36:11
ALERT LEVEL: 3 = System blocked waiting for operator input

REASON FOR ALERT
SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

LEDs: RUN ATTENTION FAULT REMOTE POWER
ON FLASH OFF ON ON

0xF8E038301100E000 00000000 0000E000 - type 31 = legacy PA HEX chassis-code
0x58E038301100E000 00006609 1208240B - type 11 = Timestamp 10/18/2002 08:36:11
A: ack read of this entry - X: Disable all future alert messages
Anything else skip redisplay the log entry

____________