topic Re: ndd command failing in Operating System - HP-UX

ndd command failing

Terrence — Wed, 13 Sep 2006 11:19:40 GMT

Gah! I'm getting this error when trying to run this routine command.

#ndd -get /dev/tcp tcp_status
operation failed, Invalid argument

This command works fine on other servers.

Help!

Re: ndd command failing

IT_2007 — Wed, 13 Sep 2006 11:25:12 GMT

find whether you are using correct ndd or not.

# ll /usr/bin/ndd
-r-xr-xr-x 1 bin bin 114688 May 31 2005 /usr/bin/ndd
# which ndd
/usr/bin/ndd

Re: ndd command failing

Patrick Wallek — Wed, 13 Sep 2006 11:25:37 GMT

I just tried this and it worked on an 11.11 and 11.23 box, but NOT on an 11.0 box.

On the 11.0 box, a 'ndd -get /dev/tcp ?' shows that tcp_status is available, but it apparently does not work.

Re: ndd command failing

Mel Burslan — Wed, 13 Sep 2006 11:26:32 GMT

is the output of

ll /dev/tcp

still looking something like this ?

# ll /dev/tcp
crw-rw-rw- 1 root root 72 0x000072 Aug 30 07:38 /dev/tcp

Re: ndd command failing

Jonathan Fife — Wed, 13 Sep 2006 11:28:51 GMT

ndd -get /dev/tcp tcp_status works fine on my 11.0 machine. I have the following patch installed, but the patch description doesn't mention tcp_status.

$ swlist -l fileset | grep -i ndd
# PHNE_20125 1.0 ndd general patch

Re: ndd command failing

Patrick Wallek — Wed, 13 Sep 2006 11:29:27 GMT

If this is 11.0, you may need to try the latest ARPA transport commands and libnm patch - PHNE_31965.

http://www1.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHNE_31965

Re: ndd command failing

Mel Burslan — Wed, 13 Sep 2006 11:30:31 GMT

also, check this one:

swlist -l product | grep "ARPA transport"

what is your patch level ? PHNE_31965 is the latest and I think you need to have a pretty recent one if not the latest for ndd to function properly.

Re: ndd command failing

A. Clay Stephenson — Wed, 13 Sep 2006 11:31:01 GMT

I would adopt a very radical approach and compare my working boxes to my non-working boxes looking for differences in OS versions and patches (especially PHNE_ patches).

Plan B. Grab the network cable, close your eyes, and use "The Force" to diagnose the problem.

Here is a possible cause:
http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000080027900

Re: ndd command failing

Jonathan Fife — Wed, 13 Sep 2006 11:31:05 GMT

Looks like the following patch might be relevant: PHNE_31965

Re: ndd command failing

Patrick Wallek — Wed, 13 Sep 2006 11:34:59 GMT

Hmmm.....Well, I just check for the PHNE_31965 on the 11.0 machine that is failing for me and the patch exists.

I tried this on 3 other 11.0 machines and it works on them. It is possible that this is failing due to a large number of network connections on the machine.

The 11.0 machine it fails on for me is an Oracle DB server with almost 500 connections to various DBs.

The 11.0 machines it worked on do not have anywhere close to that number of network conenctions (2 are deve servers and 1 an internal web server).

Re: ndd command failing

Terrence — Wed, 13 Sep 2006 11:40:34 GMT

This is an 11.11 box in a service guard configuration. The servers are identical.

This command has worked in the past and it works right now on the other node.

The server matches all the examples above.

Re: ndd command failing

Dave Olker — Thu, 14 Sep 2006 10:55:23 GMT

Hi Terrence,

I seem to recall a problem similar to this when the number of TCP connections needing to be displayed is too large to fit in a memory buffer.

Can you verify the following:

1) How many TCP connections are currently on both servers - the working one and the failing one? Use the command:

# netstat -an | grep tcp | wc -l

to see how many TCP connections are present on each box.

2) Use the following command on both systems to print the current value of the STREAMS buffer size:

# echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem

I will be curious if the failing system either has a smaller STREAMS buffer size or if there are significantly more TCP connections on the failing system than the working system.

Regards,

Dave

Re: ndd command failing

Terrence — Thu, 14 Sep 2006 11:14:27 GMT

Thanks for the follow up.

Here's the results on the server where the command is failing

#netstat -an | grep tcp | wc -l
440
impact2:/
#echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem
strmsgsz:
strmsgsz: 65535

Here's the results on the server where the command works

impact1:/
#netstat -an | grep tcp | wc -l
118
impact1:/
#echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem
strmsgsz:
strmsgsz: 65535

Re: ndd command failing

Dave Olker — Thu, 14 Sep 2006 11:23:22 GMT

Ok, so a significantly higher number of TCP connections on the failing system. This is what I was expecting. Let me see if I can find a solution for this.

In the mean time, can you please post what ARPA Transport and STREAMS patches you have on both systems:

# swlist -l product | grep PHNE

Regards,

Dave

Re: ndd command failing

Terrence — Thu, 14 Sep 2006 11:28:44 GMT

Both have the same patches.

impact2:/
#swlist -l product | grep PHNE
PHNE_23289 1.0 mux4.h header file patch
PHNE_24384 1.0 gated (1M) patch.
PHNE_24512 1.0 NTP timeservices upgrade plus utilities
PHNE_24829 1.0 telnet kernel, telnetd(1M), telnet(1) pa
PHNE_25083 1.0 Streams Pty cumulative patch
PHNE_27765 1.0 ftpd(1M) patch
PHNE_27777 1.0 r-commands cumulative mega-patch
PHNE_27796 1.0 libnss_dns DNS backend patch
PHNE_28328 1.0 inetd(1M) cumulative patch
PHNE_28444 1.0 nettl(1M), netfmt(1M) and nettladm(1M) p
PHNE_28476 1.0 Cumulative STREAMS Patch
PHNE_28799 1.0 100BT unified driver cumulative patch
PHNE_28923 1.0 LAN product cumulative patch
PHNE_29577 1.0 Cumulative Mux and Pty Patch
PHNE_29774 1.0 sendmail(1m) 8.9.3 patch
PHNE_29887 1.0 cumulative ARPA Transport patch
PHNE_30068 1.0 Bind 8.1.2 Patch
PHNE_30378 1.0 ONC/NFS General Release/Performance Patc

Re: ndd command failing

Dave Olker — Thu, 14 Sep 2006 11:45:28 GMT

Ok, you have:

PHNE_29887 1.0 cumulative ARPA Transport patch

installed, which means you already have the fix for the known problem of ndd not returning all connections from tcp_status. My guess is the only thing left to do is to increase the size of STRMSGSZ on the failing system to something larger than 64K.

The STRMSGSZ tunable defines the maximum size of the data portion of a Streams message. If the number of TCP connections to display is greater than what will fit in a STRMSGSZ buffer then the command will fail.

Unfortunately this is not a dynamic tunable parameter where you could make the change on the fly. You will need to modify the STRMSGSZ parameter using whatever method you normally use on 11.11 (SAM, /stand/system, etc.), build a new kernel and reboot the system with the new kernel.

Rather than make a specific recommendation on the appropriate size of this tunable for your environment (since I don't know your environment and every environment is different), I'd suggest you read about this tunable before selecting a new value. This parameter is described in detail on page 203 of the "Tunable Kernel Parameters, HP-UX Release 11i" manual found here:

http://docs.hp.com/en/TKP-90202/TKP-90202.pdf

After reviewing the information on that page, select a higher value for STRMSGSZ to hold the tcp_status information for your connections.

I hope this helps,

Dave

Re: ndd command failing

Dave Olker — Thu, 14 Sep 2006 11:50:28 GMT

Looking at detailed description of STRMSGSZ, it looks like the default is 0 but on your systems it looks like it is being set to 64K. Can you check the /stand/system file on your servers and see if you see:

STRMSGSZ=65535

If you do, this is the line that's limiting STRMSGSZ to 64K. You might want to either remove this line or set the value to 0, as that is the recommended value.

Regards,

Dave