DCE daemon dying following patch installs

[Mel Haylaz]

Hi

I've got a single DCE server on a HPUX 11.00 box and 5 DCE clients, 3 on HPUX 10.20 and 2 on HPUX 11.00.

We recently installed some security patches on the 2 HPUX 11.00 DCE clients. After the installs were complete when we attempt to restart DCE the dced process dies after a minute or so. Thinking the patches were the cause we rolled back the patches on one of the hosts but the problem persists. Being unable to keep the daemon running on either of these boxes via the normal restart, either 'dce_config' or '/sbin/init.d/dce start' we have found that if you start the dced process manually 2 times in a row it stays up. However, on the host that still has the patches it stays up for approx 10hrs each of every time. On the host that had the patches rolled back the regularity of the daemon dying is not consistent. To make it even more interesting approx 2 months after the installs the other 3 HPUX 10.20 clients have started having the same problem.

We have setup cron jobs to restart the dced process manually each time it goes down, but obviously this is not ideal and in the long run can cause other undesirable results.

The problem is DCE restarts OK, no errors in any of the obvious logs /etc/rc.log, /var/adm/syslog/syslog.log /var/opt/dce/dced.log, /var/opt/dce/config/dce_config.log etc.
The dced process just drops out and we haven't been able to find any output.

We have checked the server and no errors reported there either and no corruption present that we could find.

I have attached a list of the patches. Any ideas?

Note: we tried it out in a test environment and had no problems, however in our test environment the server is HPUX 10.20 and the clients are HPUX 11.00.

[Tim Adamson_1]

Mel,

Is a core file produced by any chance?

Thats about the only suggestion I can make.


Tim.

[Steven Gillard_2]

All DCE processing log to files in the /var/opt/dce/svc directory - usually they write to the fatal.log file here before they exit.

Something we used to have to do with older versions sometimes was move aside the /var/opt/dce/dced/Ep.db and Lib.db files before restarting. You might want to try that as well - but remember to restart all your RPC server processes afterwards.

Regards,
Steve

[Steven E. Protter]

Read this thread:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x1380392d7e097d4882194df255578d96,00.html

You may have a security problem on your network(blaster et al) that is triggering the DCD vulnerability.

There may not be patches, but you might want to disable icmp, or take other steps to secure your network.

SEP

[Mel Haylaz]

Hi Tim

I've attached a copy of the core file along with some logs, I previously thought there weren't any, but it appears someone was emptying the log files :->

Hi Steven G

The log files are attached, I don't know enough about DCE and the way in which our application accesses it to move the *.db files around. Our application doesn't work without DCE and we can't have outages without schedules, approval and alloted change windows so we want to make sure we have a clear understanding of the problem and the proposed solution because we won't get too many shots at it.

Hi Steve P

The patches on the URL you mention are for HPUX 11.11, we are only at 11.00 and I've checked for the specified patches and we don't have any of them installed.

[Tim Adamson_1]

Mel,

I tried to save the attachment but it appears to be corrupt and I can't view the contents.

Maybe you could try the following:

# cd
# file core

You could also install gdb or wdb and try to debug the core file.

Alternatively, if you have a valid HP software support contract, log a call to your local Response Centre and get them working on the problem too.


Tim

[Steven Gillard_2]

I can't read the attachment either... the main log to look at is fatal.log, dced should write a message there when it dies. Have a look at the most recent messages from dced after it dies, they're going to be your best clue.

Also I notice you're not running the latest DCE patches. For 11.0 these are:

PHSS_27962 HP DCE/9000 1.7 Runtime cumulative patch
PHSS_27963 HP DCE/9000 1.7 Integrated Login cum. patch
PHSS_27964 HP DCE/9000 1.7 Server/DevTools cum. patch

You might want to try installing them.

Regards,
Steve

[Mel Haylaz]

Results of "what core"
vuh273 # what core
core:
HP DCE/9000 1.7-1.7.1 PHSS_20444-47 Module: dced (Export) Date: Feb 29 2000 18:08:09
$RCSfile: environment.c,v $ $Revision: /main/HPDCE02/HPDCE_64/1 $ (OSF) $Date: 1997/11/06 19:44 UTC $
$ PATCH/11.00:PHCO_23189 Mar 6 2001 21:58:04 $
HP DCE/9000 1.7-1.7.1 PHSS_20444-47 Module: libdcedpvt.1 (Export) Date: Feb 29 2000 17:46:58
HP DCE/9000 1.7-1.7.1 PHSS_20444-47 Module: libcma.2 (Export) Date: Feb 24 2000 23:11:56
HP DCE/9000 1.7-1.7.1 PHSS_20444-47 Module: libdce.2 (Export) Date: Feb 29 2000 17:30:00
rec_seq.c 8.2 (Berkeley) 9/7/93
libXOM 1.9 (BULL S.A) 7/1/92
$RCSfile: environment.c,v $ $Revision: /main/HPDCE02/HPDCE_64/1 $ (OSF) $Date: 1997/11/06 19:44 UTC $
$ PATCH/11.00:PHCO_27731 Oct 11 2002 12:55:05 $
SMART_BIND
92453-07 dld dld dld.sl B.11.30 011005

Also, attached fatal.log

[Tim Adamson_1]

Try moving the existing hostdata.db file aside (never delete it unless you are absolutely sure you don't want it). Then run dced -i to reinitialise the databases. If they already exist, it wont create a new one. That is according to the dced(1M) man page. maybe the existing database is corrupt.

The database should reside in /var/opt/dce/dced.

Then see if you can progress.


Tim.

[Mel Haylaz]

How do I know if it's referring to the local client database or the server database?

[Mel Haylaz]

The DCE cell was rebuilt from scratch, problem went away. The security patches must have corrupted the cell somehow. The Sys Admins may have installed the patches without properly shutting down DCE.