bdf not responding system hangs

[Doug_85]

We are running hp-ux 11.0 on L class server. Recently we've been experiencing lockups when we use bdf to display disk usage. When I try to create an ignite tape it displays "NFS server not responding still trying" Recent backups look odd also. Here's the output:

Full backup started on akl : Mon May 31 22:30:01 YDT 2004 (Scheduled Backup)
{ fbackup -0 -u -g /etc/sam/br/graphBDAa16639 -I /var/sam/log/br_index.full -c /etc/sam/br/fbackup_config -d /var/adm/fbackupfiles/dates -f /dev/rmt/0m } 2>&1
Full backup completed with warnings on akl : Mon May 31 23:59:49 YDT 2004 . Exit code = 4 Full backup started on akl : Tue Jun 1 22:30:00 YDT 2004 (Scheduled Backup)
{ fbackup -0 -u -g /etc/sam/br/graphBDAa16639 -I /var/sam/log/br_index.full -c /etc/sam/br/fbackup_config -d /var/adm/fbackupfiles/dates -f /dev/rmt/0m } 2>&1
Full backup started on akl : Wed Jun 2 22:30:01 YDT 2004 (Scheduled Backup)
{ fbackup -0 -u -g /etc/sam/br/graphBDAa16639 -I /var/sam/log/br_index.full -c /etc/sam/br/fbackup_config -d /var/adm/fbackupfiles/dates -f /dev/rmt/0m } 2>&1



Thanks,
Doug

[KapilRaj · Accepted solution]

One of ur NFS server is hung / switched-off / IP_ADDRESS is changed.

unmount and re-mount it

Regds,

Kaps

[Anupam Anshu_1]

Hi Doug,

Looks like you have some stale nfs mount on the system.

Even I faced a similar problem recently.

Remeady is to unmount and then mount those stale nfs mounts.

Hope that helps,

Anshu

[Doug_85]

Would rebooting the system refresh the NFS mounts? I also suspect that new mounts to our system were added sometime last week. Does the NFS mount error occur even if only one mount is bad? If it doesn't affect other mounts currently running I'll reboot the system when users are off the system.

Thanks,
Doug

[KapilRaj]

If ur NFS server is stilll down your system will hang while booting. So it's better to remove them from /etc/fstab... Also try having soft mounts rather than hard mounts

Regds,

Kaps

[Doug_85]

Here are the NFS daemons that are currently running. They have the same date as when the system started having problems. Would it be advisable to do a "/sbin/init.d/nfs.server start" to refresh these daemons?

Thanks,
Doug
root 1432 1429 0 May 25 ? 0:00 /usr/sbin/nfsd 4
root 1429 1 0 May 25 ? 0:00 /usr/sbin/nfsd 4
root 1431 1430 0 May 25 ? 0:00 /usr/sbin/nfsd 4
root 1430 1429 0 May 25 ? 0:00 /usr/sbin/nfsd 4
root 18589 16815 0 08:21:52 pts/to 0:00 grep nfsd

[KapilRaj]

Doug,

cat /etc/fstab |grep nfs|awk -F ":" '{print $1}'

the above command would give u a list of NFS servers from which you have nfs mounted some filesystems.

ping to every system and make sure that all are up. if not investigate and fix that.

What u r trying to restart is the NFS Server daemon.. there would be no luck anyway if u try that. u could try nfs.client stop & start

Regds,

Kaps

[Doug_85]

Just found out "super master agent" for EMC was installed on our L server. This was the same time bdf started to hang and backups erroring.

Any suggestions,

Doug

[KapilRaj]

Doug,

No idea what this product is but as you have the following error message as you mentiuoned in the request "NFS server not responding still trying" ... It is something to do with the NFS mounts !

Regds,

Kaps

[John Carr_2]

Doug

With a stale nfs mount sometimes you can make it live using mount -O on the filesystem -O means overlay the existing one.
you can also try to unmount it using the -f force option. if not reboot is the only way I know to solve a stale mount point.

John.

[Anupam Anshu_1]

With stale nfs mount, its better to identify the machine on which the problem is existing (ie the nfs server - system on which the actual files and directories exists). Bring-up the system, check if the mnttab entries are correct.

Once you verify and correct the nfs server, then re-mount the nfs directory.

This should solve your problem.

Hope that helps,

Anshu

[Doug_85]

The problem was with NFS on the SD_CDROM. Trying to kill the process was impossible even with kill -9. We tried stopping and restarting the client and changing the nfsconf file from 1 to 0 on both the client and server but was still unsuccessful. After rebooting the server everything is working fine. Thanks for all your help.

Doug

[Bill Hassell]

Just a note about NFS and production servers: I would not use NFS unless the servers and the network are very stable and no rogue sysadmins try mounting something from an unapporved source. NFS will cause hangs on bdf (and other filesystem query tools like df) but will also hang all new logins and will affect some applications. Running the old 11.00 version of HP-UX, expecially without the latest networking and NFS patches is very risky with NFS.

And since you mantioned SD_CDROM, are you installing software from HP CDs with NFS? DON'T! swinstall has the ability to install without any NFS running. Just use swreg on the server with the CDROM to register the CD. Then every server on your network can see the CDROM with swinstall. (and no NFS hangs)