RPC problem, statd & lockd not availble!

[Andrew_4]

Hi guys,

Got a RPC problem, which is eventuates in a SG package being brough down.

The SG package is using the monitor script to monitor NFS availability. A
part of the NFS monitoring script, is that it 'pings' and checks the status of
various RPC daemons, including the statd & nlockd.

These two daemons seem to be running:

root-hostname:/> ps -ef | grep rpc
root 1520 1 0 Apr 27 ? 15:11 /usr/sbin/rpcbind
root 1945 1 0 Apr 27 ? 4:24 /opt/dce/sbin/rpcd
root 2300 1 0 Apr 27 ? 0:57 /usr/sbin/rpc.mountd
root 12190 1606 0 Jun 11 ? 0:00 /usr/dt/bin/rpc.ttdbserver
root 3941 1 0 Jul 1 ? 0:21 /usr/sbin/rpc.lockd
root 3935 1 0 Jul 1 ? 1:30 /usr/sbin/rpc.statd

root-hostname:/> rpcinfo | grep -e 100021 -e 100024
100024 1 tcp 0.0.0.0.241.202 status superuser
100024 1 udp 0.0.0.0.238.129 status superuser
100021 1 tcp 0.0.0.0.241.203 nlockmgr superuser
100021 1 udp 0.0.0.0.238.130 nlockmgr superuser
100021 3 tcp 0.0.0.0.241.204 nlockmgr superuser
100021 3 udp 0.0.0.0.238.131 nlockmgr superuser
100021 4 tcp 0.0.0.0.241.205 nlockmgr superuser
100021 4 udp 0.0.0.0.238.132 nlockmgr superuser
100021 2 tcp 0.0.0.0.241.206 nlockmgr superuser

But when the SG monitor script comes to check on them, it gets:

root-hostname:/> rpcinfo -u 127.0.0.1 100024 1
rpcinfo: RPC: Timed out
program 100024 version 1 is not available
root-hostname:/> rpcinfo -u 127.0.0.1 100021 3
rpcinfo: RPC: Timed out
program 100021 version 3 is not available
root-hostname:/>

As you can see, the rpcinfo summary reckons that status & nlockmgr are
running... but when you try to 'ping' them... it times out.

Not sure why this would be occurring. I've already stopped & started
core/server/client sbin scripts... but that still didn't fix the problem.

Any ideas ?

[Tom Geudens]

Hi Andrew,
Sorry if I'm stating the obvious, but RPC-problems are usually very tricky (I'm fighting a couple myself at the moment).
Did you check (and install) the latest DCE/RPC patches ? HP will almost always recommend to do that first.

Not much help, I know ...
Regards,
Tom

[Suresh Patoria]

Hi,

First you check the your NFS version

if you using NFS version 3 than it uses TCP protocol

above command use for only if your NFS server have the old version otherwise it wan't work

because you command rpcinfo -u 127.0.0.1 10024 1 uses the UDP protocol

If you have NFS verson 2 or 3
it is uses the TCP protocol and you use the command rpcinfo -t 127.0.0.1 10024 1 command

and also check the all the coinfiguration file and services running or not

[Andrew_4]

Thanks for suggestion!... We are on mostly the latest NFS patches already (just did a patch review)...

But I've fixed the problem now anyway. It was very obscure though.

I enabled logging on statd & lockd via /etc/rc.config.d/nfsconf

LOCKD_OPTIONS="-l /tmp/rpc.statd.log"
STATD_OPTIONS="-l /tmp/rpc.lockd.log"

Stopped and started nfs.server & nfs.core

Checked the log files, found that lockd couldn't bind onto statd... So upon checking statd.log, found a bunch of these messages:

"statd cannot talk to statd at sm0017"
(sm0017 is a client system)

man 1m rpc.statd

suggested that files exist in /var/statmon

Checked those files... found a whole heap for sm00xx machines...

Then found Document id: 2200211723, which suggested I delete all the 'sm' files, and restart nfs...

which fixed the problem !

Hope this might help for you too.

Regards,

Andrew