RPCD Daemon fails some hours after starting it!

[Gian Antonio D'Addetta_1]

Hi all,

lately, all of our HP=UX 10.20 and 11.00 Machines, mostly C3600 show the following error:

The rpcd damon is started when booting (see output from rc.log) and works some hours.After a while the daemon stops working and must be restarted by hand. Unfortunately this lasts again only some hours.I checked two log files in /var/opt/dce/svc (see below), but couldn't find the problem.

***************************************
output from /etc/rc.log:

Start NCS broker daemons
Output from "/sbin/rc2.d/S590ncs start":
----------------------------
starting the DCE RPC Daemon
/opt/dce/sbin/rpcd

*************************************

in /var/opt/dce/svc:
output of fatal.log
2003-08-06-12:55:52.125+00:00I----- dced FATAL dhd general main.c 699 0x7afb9e00 Cannot use '*all*' protocol sequence, Address already in use

output of error.log
2003-08-06-12:55:52.200+00:00I----- dced ERROR dhd general main.c 1273 0x7afb9e00 Process (pid 5446) exited with status 01


Does anyone have an idea home to fix this problem?
Thanks in advance for your help!

Gian Antonio

[Alex Glennie · Accepted solution]

Is a core file produced anywhere ? perhaps /opt/dce/bin or /opt/dce/sbin, if yes file core yields ?

Patch levels wrt networking OK/recent ? also important are dce / libC patches and can cause odd problems if they are not patched to a similar level.

fatal log : suggests somethings got hold of port 135 .... is the timestamp consistent with rpc dying or you restarting it .... you say rpcd died I presume you checked with ps -ef | grep rpcd right ?

Try running netstat -n -a|grep 135|grep -i listen ? in case something else is using port 135 at the time of the problem ?

Are you using glance or measureware products at all at the time of the problem ?

nslookup works as expected at the time of the problem ?

No routers/switches etc involved here ?

Try setting this :

RPC_SUPPORTED_NETADDRS=protocol:ip_address[,protocol:ip_address]

ie:

RPC_SUPPORTED_NETADDRS= ncadg_ip_udp:15.1.1.1
export

RPC_SUPPORTED_NETADDRS can be set in /etc/opt/dce/dce_com_env

[Steven Gillard_2]

Make sure you've got the latest DCE runtime patches installed:

11.0: PHSS_27962
10.20: PHSS_19739

Are there any core files being left behind?

Regards,
Steve

[Mark Greene_1]

Run this:

UNIX95= ps -efH|head -1; UNIX95= ps -efH|egrep "inetd|rpc|mount|nfs"|grep -v grep; rpcinfo -p|egrep "service|nfs"; showmount -e; mount -p


This will tell you if you have all of the required rpc daemon processes running, or not. It will also show you tcp port allocations and nfs mounted file systems, if any.

Be sure that the rpc.lockd and rpc.statd processes have the same (or very nearly so) start times. If not, kill them and restart them. If these two processes are not syncronous, rpc behaves weirdly.

The rpcinfo -p will show you what protocols are in use, if any.

mark

[Gian Antonio D'Addetta_1]

Thanks for the quick response!

I started to check some of the above points.

No core file is produced in /opt/dce/bin or /opt/dce/sbin.

I don't think that our patch levels wrt networking have been recently updated. The same applies to dce / libC patches.

I run netstat -n -a|grep 135|grep -i listen and found that the only device that is on is "tcp".

This is where i stand right now, but found out that we were checked by the central computer center of our university as they routinelly tried to find out security defects of our network. However, i don't know if they checked port 135, too. But definitely, since we were checked we have this problems. I'm in contact with the corresponding persons right now, to find out, what the problem is.

If no solution will be found, i will continue with your proposals, especially to bring the machine on a recent patch level.

By the way DCE runtime patches PHSS_27962 (11.0) & PHSS_19739 (10.20) are not installed on the machines. So, i will have to update them, too.

Thanks again for your help! I will post more, when i can proceed!

Regards
Gian Antonio

[Gian Antonio D'Addetta_1]

Mark:

I run the command line you gave me, and to me everything seems quite normal. rpc.lockd and rpc.statd daemons have both the same strat time.
Uniquely /opt/dce/sbin/rpcd has another one, as i restarted the daemon later again, after it failed. The output of rpcinfo -p is given below, but doesn't tell me anything, as i'm rather an amateur concerning HP-UX.

program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 814 status
100024 1 tcp 816 status
100021 1 tcp 820 nlockmgr
100021 1 udp 1030 nlockmgr
100021 3 tcp 824 nlockmgr
100021 3 udp 1031 nlockmgr
100021 4 tcp 828 nlockmgr
100021 4 udp 1032 nlockmgr
100020 1 udp 4045 llockmgr
100020 1 tcp 4045 llockmgr
100021 2 tcp 835 nlockmgr
100068 2 udp 1033 cmsd
100068 3 udp 1033 cmsd
100068 4 udp 1033 cmsd
100068 5 udp 1033 cmsd
100083 1 tcp 1036 ttdbserver
100005 1 udp 896 mountd
100005 1 tcp 898 mountd
100003 2 udp 2049 nfs
1342177279 4 tcp 1110
1342177279 1 tcp 1110

Thanks again!

Gian Antonio

[Gian Antonio D'Addetta_1]

Hello again,

now we found the problem: In order to prevent damages to the internal network of our campus net a scan of all IP's within the net and there especially of port 135 have been done. When checking the HP-UX machines the problem described above appears. The scan was due to the distribution of the W32/Blaster or Lovsan worm. However, now our HP machines are no longer scanned, as they originally are not supposed to show the same problems along with this worm.

The vulnerability has been investigated by HP (Hewlett-Packard Company Information for VU#377804). For information see http://www.kb.cert.org/vuls/id/IAFY-5Q6GES.

Nevertheless, thank you all for the help.

Regards
Gian Antonio

[James Saffer_1]

Are you positive you are seeing this problem on 11.00?

I have a lot of 10.01 and 10.20 servers that MSblaster is causing rpcd to die on.

Do you have PHSS_19739 (circa 1999) applied? It has no dependancies, and a reboot is not required.

[Rick Krul]

The bottom of the patch install file notes that Oracle 7.3.3 and above will have to be relinked. Has anyone installed the PHSS_19739 and relinked Oracle? I know no reboot is required, but I do have a concern in relinking all my Oracle instances. How is this done?

[leyearn]

my system have the similar symptom !
but my os is hpux10.10
which patch can i use?