ndd command failing

[Terrence]

Gah! I'm getting this error when trying to run this routine command.

#ndd -get /dev/tcp tcp_status
operation failed, Invalid argument

This command works fine on other servers.

Help!

[IT_2007]

find whether you are using correct ndd or not.

# ll /usr/bin/ndd
-r-xr-xr-x 1 bin bin 114688 May 31 2005 /usr/bin/ndd
# which ndd
/usr/bin/ndd

[Patrick Wallek]

I just tried this and it worked on an 11.11 and 11.23 box, but NOT on an 11.0 box.

On the 11.0 box, a 'ndd -get /dev/tcp ?' shows that tcp_status is available, but it apparently does not work.

[Mel Burslan]

is the output of

ll /dev/tcp

still looking something like this ?

# ll /dev/tcp
crw-rw-rw- 1 root root 72 0x000072 Aug 30 07:38 /dev/tcp

[Jonathan Fife]

ndd -get /dev/tcp tcp_status works fine on my 11.0 machine. I have the following patch installed, but the patch description doesn't mention tcp_status.

$ swlist -l fileset | grep -i ndd
# PHNE_20125 1.0 ndd general patch

[Patrick Wallek]

If this is 11.0, you may need to try the latest ARPA transport commands and libnm patch - PHNE_31965.

http://www1.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHNE_31965

[Mel Burslan]

also, check this one:

swlist -l product | grep "ARPA transport"

what is your patch level ? PHNE_31965 is the latest and I think you need to have a pretty recent one if not the latest for ndd to function properly.

[A. Clay Stephenson · Accepted solution]

I would adopt a very radical approach and compare my working boxes to my non-working boxes looking for differences in OS versions and patches (especially PHNE_ patches).

Plan B. Grab the network cable, close your eyes, and use "The Force" to diagnose the problem.


Here is a possible cause:
http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000080027900

[Jonathan Fife]

Looks like the following patch might be relevant: PHNE_31965

[Patrick Wallek]

Hmmm.....Well, I just check for the PHNE_31965 on the 11.0 machine that is failing for me and the patch exists.

I tried this on 3 other 11.0 machines and it works on them. It is possible that this is failing due to a large number of network connections on the machine.

The 11.0 machine it fails on for me is an Oracle DB server with almost 500 connections to various DBs.

The 11.0 machines it worked on do not have anywhere close to that number of network conenctions (2 are deve servers and 1 an internal web server).

[Terrence]

This is an 11.11 box in a service guard configuration. The servers are identical.

This command has worked in the past and it works right now on the other node.

The server matches all the examples above.

[Dave Olker]

Hi Terrence,

I seem to recall a problem similar to this when the number of TCP connections needing to be displayed is too large to fit in a memory buffer.

Can you verify the following:

1) How many TCP connections are currently on both servers - the working one and the failing one? Use the command:

# netstat -an | grep tcp | wc -l

to see how many TCP connections are present on each box.


2) Use the following command on both systems to print the current value of the STREAMS buffer size:

# echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem


I will be curious if the failing system either has a smaller STREAMS buffer size or if there are significantly more TCP connections on the failing system than the working system.

Regards,

Dave

[Terrence]

Thanks for the follow up.

Here's the results on the server where the command is failing

#netstat -an | grep tcp | wc -l
440
impact2:/
#echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem
strmsgsz:
strmsgsz: 65535


Here's the results on the server where the command works

impact1:/
#netstat -an | grep tcp | wc -l
118
impact1:/
#echo 'strmsgsz/D' | adb -k /stand/vmunix /dev/kmem
strmsgsz:
strmsgsz: 65535

[Dave Olker]

Ok, so a significantly higher number of TCP connections on the failing system. This is what I was expecting. Let me see if I can find a solution for this.

In the mean time, can you please post what ARPA Transport and STREAMS patches you have on both systems:

# swlist -l product | grep PHNE

Regards,

Dave

[Terrence]

Both have the same patches.

impact2:/
#swlist -l product | grep PHNE
PHNE_23289 1.0 mux4.h header file patch
PHNE_24384 1.0 gated (1M) patch.
PHNE_24512 1.0 NTP timeservices upgrade plus utilities
PHNE_24829 1.0 telnet kernel, telnetd(1M), telnet(1) pa
PHNE_25083 1.0 Streams Pty cumulative patch
PHNE_27765 1.0 ftpd(1M) patch
PHNE_27777 1.0 r-commands cumulative mega-patch
PHNE_27796 1.0 libnss_dns DNS backend patch
PHNE_28328 1.0 inetd(1M) cumulative patch
PHNE_28444 1.0 nettl(1M), netfmt(1M) and nettladm(1M) p
PHNE_28476 1.0 Cumulative STREAMS Patch
PHNE_28799 1.0 100BT unified driver cumulative patch
PHNE_28923 1.0 LAN product cumulative patch
PHNE_29577 1.0 Cumulative Mux and Pty Patch
PHNE_29774 1.0 sendmail(1m) 8.9.3 patch
PHNE_29887 1.0 cumulative ARPA Transport patch
PHNE_30068 1.0 Bind 8.1.2 Patch
PHNE_30378 1.0 ONC/NFS General Release/Performance Patc

[Dave Olker]

Ok, you have:

PHNE_29887 1.0 cumulative ARPA Transport patch

installed, which means you already have the fix for the known problem of ndd not returning all connections from tcp_status. My guess is the only thing left to do is to increase the size of STRMSGSZ on the failing system to something larger than 64K.

The STRMSGSZ tunable defines the maximum size of the data portion of a Streams message. If the number of TCP connections to display is greater than what will fit in a STRMSGSZ buffer then the command will fail.

Unfortunately this is not a dynamic tunable parameter where you could make the change on the fly. You will need to modify the STRMSGSZ parameter using whatever method you normally use on 11.11 (SAM, /stand/system, etc.), build a new kernel and reboot the system with the new kernel.

Rather than make a specific recommendation on the appropriate size of this tunable for your environment (since I don't know your environment and every environment is different), I'd suggest you read about this tunable before selecting a new value. This parameter is described in detail on page 203 of the "Tunable Kernel Parameters, HP-UX Release 11i" manual found here:

http://docs.hp.com/en/TKP-90202/TKP-90202.pdf

After reviewing the information on that page, select a higher value for STRMSGSZ to hold the tcp_status information for your connections.

I hope this helps,

Dave

[Dave Olker]

Looking at detailed description of STRMSGSZ, it looks like the default is 0 but on your systems it looks like it is being set to 64K. Can you check the /stand/system file on your servers and see if you see:

STRMSGSZ=65535

If you do, this is the line that's limiting STRMSGSZ to 64K. You might want to either remove this line or set the value to 0, as that is the recommended value.

Regards,

Dave