rpcbind: server not responding, timed out (Linux client, HP-UX server)

[rayvd]

I have a few HP-UX 11.11 servers and have noticed that Linux clients tend to have problems with talking to them via RPC.

This appears to be related to NFS, although these error messages continue to show up long after the directories in question have been unmounted by the Linux client.

On the Linux client:

Sep 13 08:51:49 leoray kernel: rpcbind: server tusk not responding, timed out
Sep 13 08:51:49 leoray kernel: lockd: couldn't create RPC handle for tusk

Over and over and over --- neverending really in approximately 45-60 second intervals.

A tcpdump shows lots of portmap V4 GETVERSADDR calls (from Linux) and replies, and then a random UDP packet from Linux typically with source oprt 32769 and a destination port number of 54224 or thereabouts followed by an ICMP Port Unreachable response from the HP-UX server.

So what should be listening on that destination port? Which side is to blame? I just want to make these error messages stop. :)

Output of rpcinfo on the HP-UX server in question:

bash-3.2# rpcinfo
program version netid address service owner
100000 4 ticots nike.esri.com.rpc rpcbind superuser
100000 3 ticots nike.esri.com.rpc rpcbind superuser
100000 4 ticotsord nike.esri.com.rpc rpcbind superuser
100000 3 ticotsord nike.esri.com.rpc rpcbind superuser
100000 4 ticlts nike.esri.com.rpc rpcbind superuser
100000 3 ticlts nike.esri.com.rpc rpcbind superuser
100000 4 tcp 0.0.0.0.0.111 rpcbind superuser
100000 3 tcp 0.0.0.0.0.111 rpcbind superuser
100000 2 tcp 0.0.0.0.0.111 rpcbind superuser
100000 4 udp 0.0.0.0.0.111 rpcbind superuser
100000 3 udp 0.0.0.0.0.111 rpcbind superuser
100000 2 udp 0.0.0.0.0.111 rpcbind superuser

Thanks in advance...

[skt_skt]

try some of this ...

Debug logging can be toggled on and off by sending the SIGUSR2 signal to the running lockd or statd process(kill -17 pid).By default , the dubug information is logged to the /var/adm/rpc/lockd.log and /var/adm/rpc.statd.log files respectively.

Collect a debug logfile from rpc.statd and rpc.lockd

# ps -ef | grep rpc
# kill -17
# kill -17

wait 30 seconds

# kill -17
# kill -17

[Dave Olker]

Hi,

A couple questions. Is that the full rpcinfo output on the server? I don't see anything but rpcbind registration here. No nfsd, no rpc.lockd, etc. Are these programs running on the HP-UX system?

Can you cut/paste a full output from "rpcinfo -p" on the HP-UX system?

As for the packets you see in tcpdump, I have to imagine some program on the Linux box cached an old port number on the HP-UX system (lockd? statd? who knows) and then keeps sending the packet expecting a specific response. All it gets is an ICMP port unreachable, which means the program it was trying to send to is not listening on that port any longer.

If you would like to collect a Wireshark trace of the offending packets and send them to me I can tell you what type of request they are and what the Linux client is hoping to get in response. However, I have to blame the Linux box in this case if it is continuing to send packets to a port that no longer is being listened to by the HP-UX system.

Regards,

Dave