HP-UX 11.11: rpcbind crashes on RPCB_GETVERSADDR

[Rudolf Polzer (RBI)]

HP-UX kronos B.11.11 U 9000/800 1510700538 unlimited-user license

I am sorry for sending this to the forum, but I found no other way to report this and ask about it because I do not have further access to the HP web site - I am just a part time administrator, the problem is urgent and my boss says that the support contract has expired.

rpcbind often crashes with the following messages in "rpcbind -d":

rpcbind: check_rmtcalls: rpcb_rmtcalls == 0
rpcbind: check_rmtcalls: rpcb_rmtcalls == 0
rpcbind: rpcbs_procinfo: RPCBVERS4 case
rpcbind: rpcb_service_4: RPCBPROC_GETVERSADDR case
rpcbind: rpcbproc_getversaddr_4: RPCB_GETVERSADDR rqst for (100021, 4, udp) from x.x.x.143.128.11 (32779) :
rpcbind: mergeaddr: mergeaddr: contact uaddr = x.x.x.25
Bus error(coredump)
kronos:/root#

x.x.x.143 is a Linux client, by the way, using the current kernel of Fedora Core 6.

I was able to reproduce the crash with a C program that creates this request. It seems to not crash when putting for example x.x.x.25.0.0 as r_addr in the struct rpcb instead, so maybe this request is in fact invalid and caused by a buggy RPC client. But even then, it should never lead to a crash of rpcbind on the server.

How to fix this? Is there maybe a patch that fixes it and it just hasn't been installed yet, in that case, which is the ID of the patch? At the moments our servers are practically unusable because of rpcbind crashing so often and so soon.

[Steven E. Protter]

Shalom,

I would say that there is a problem with DNS lookup in the network. Somewhere inconsistent answers are coming up with regards to this Linux host.

if /etc/nsswitch.conf uses files first, you may be able to fix this with an /etc/hosts entry for the Linux host.

I would do the following from the patch standpoint.

1) Install a bi-annual patch update that is less than a year old.
2) Search the patch database for rpcbind and nfs and read the notes to find a patch with this or a similar result and error condition.

SEP

[Rudolf Polzer (RBI)]

Thanks, this sounds promising - my boss is doing the HP patching, and he always said it's so complicated and thus he never did it in the last years. He did not even know there are such bundles... and neither does he know that one can evade the multiple reboots by combining all patches to one depot (shell script for that is included in the shell archive I can download after selecting patches).

Now looking into this bundle: PHNE_35418 looks promising, it contains "rpcbind crashes with signal 10, Bus error (SIGBUS)" in the list of fixes. However, it looks like this patch is already installed - the existing rpcbind has a checksum of

564530978 73728 /usr/sbin/rpcbind

which matches the one from PHNE_36168, supersedes PHNE_35418.

As for possible DNS setup problems - as far as I can see it, the DNS setup for this host is sane, forward and reverse lookup on it work quite well. Also, I wonder what DNS lookup can have to do with rpcbind... and I read a bit further and saw that this r_addr is supposed to contain ip1.ip2.ip3.ip4.portHi.portLo for IPv4, and in the requests that cause rpcbind to crash, the elements portHi and portLo of this "universal address" were missing. When I send the very same request with values - ANY values - of portHi and portLo, rpcbind won't crash but returns a valid universal address of the service I asked for. Also, I get a crash when I just pass a random string like XXXXXXXXXX there.

Still, it looks like the patch pack or at least the result of the "restricted" patch assessment should get installed and may very well fix the issue. This will be tried soon.

Thanks!

[Dave Olker]

Hi Rudolf,

Could you please collect an Ethereal/Wireshark network trace of your C program causing the rpcbind core dump? I'd like a copy of the raw (unformatted) trace file so I can see exactly what the packet looks like that is causing rpcbind to fail.

Also, if you wouldn't mind sharing your C program with me I'd appreciate a copy so I can attempt to duplicate the problem on my system. If you don't mind posting them here on ITRC, fine. If you prefer to email them to me please do so at: dave.olker@hp.com.

Thanks,

Dave

[skt_skt]

these are the one i used during the recent patch upgrade

grep -i nfs 2006.11i.patch.list
PHKL_25993 1.0 thread nostop for NFS, rlimit, Ufalloc fix
PHKL_34595 1.0 VM NFS umount fix
PHNE_34662 1.0 ONC/NFS General Release/Performance Patch

[Rudolf Polzer (RBI)]

I got some further information using gdb: the crash happens inside libnsl.1 in the function uaddr2taddr:

(gdb) bt
#0 0xc02d0bf0 in __inet_uaddr2taddr () from /usr/lib/libnsl.1
#1 0xc02d0864 in __inet_netdir_options () from /usr/lib/libnsl.1
#2 0xc02afb30 in netdir_options () from /usr/lib/libnsl.1
#3 0x4e60 in ?? ()
#4 0xc02d0bd4 in __inet_uaddr2taddr () from /usr/lib/libnsl.1
Cannot access memory at address 0x40000bf4.

Too bad I have no clue about PA-RISC assembler, otherwise I would have
tried to at least evade the bug by disabling the offending function
call.

Patch PHNE_34662 is installed and the libnsl.1 matches the checksum information mentioned in the patch description ("338405851 753664").

I recorded an offending packet using tcpdump to http://hagger.rbi.informatik.uni-frankfurt.de/~polzer/temp/rpcb4query.pcap, but ignore the broken UDP checksum - that's okay for outgoing packets when tcpdumping on a network card that does UDP checksum offloading.

[Dave Olker · Accepted solution]

Hi Rudolf,

Thanks for the additional data. This is what I was hoping to find out by reproducing the problem in house.

This is a reported problem known in the lab as JAGag46146. The lab has created a pre-release binary with the fix for this problem and given it to other customers reporting the issue.

My suggestion to you is to immediately open a support case with HP Support (if you haven't done so already) and tell them you need to have your case escalated to the WTEC team. Then let the WTEC folks know you need a copy of the binary containing the solution for JAGag46146. Hopefully they'll be able to provide one to you quickly and get your systems back up and running.

Let me know if you have any questions about this.

Regards,

Dave

[Rudolf Polzer (RBI)]

Looks like we can't do that. After checking the contracts that were made back then, we found out that we only had ordered HW support and SW updates, but no SW support. So apart from waiting till the patch is public, there is nothing we can do? Is there maybe an alternative to rpcbind one can install? I found one on the net that LOOKED like one, and it did compile, but the services won't register to it and in netstat, it looked like it was listening to UDP only.

[Dave Olker]

I'm not aware of any alternative to rpcbind on HP-UX. I've never heard of anyone running an open-source portmapper that works on HP-UX.

I guess you'll either have to wait for a patch or upgrade your contract to include software support.

Dave

[Jakmani]

Hi All,

Please share the core file path for rpcbind process crash. 

Thanks in Advance.

           Believe the Hope!!!

[Dennis Handly]

>the core file path for rpcbind process crash.

You could look in /.  Otherwise you would need to search for it.

Or look at the init.d scripts that start it.:

/sbin/init.d/nfs.client
/sbin/init.d/nfs.core
/sbin/init.d/nfs.server
/sbin/init.d/nis.client
/sbin/init.d/nis.server
/sbin/init.d/nisplus.client
/sbin/init.d/nisplus.server

(I don't see any cd commands there.)