AdvFS vdump process hangs on Tru64 V4.0F

[Geert Van Pamel]

Since recently, vdump hangs on an incremental vdump backup of a simple AdvFS file system /users.

The filesystem does not seem to have errors or other visible problems.

The tape drive works normally for all the other file systems, so I would exclude tape problems.

ps -elf |grep vdump
F S UID PID PPID %CPU PRI NI RSS WCHAN STARTED TIME COMMAND
80008001 U N 0 25205 1 0.0 54 10 144K * Dec 24 0:26.50 /sbin/vdump -u -f /dev/nrmt0h -9 /users

kill or kill -9 does not work since the process is in Uninterruptle state.

I cannot upgrade the OS since the AlphaStation 500 only supports up to Tru64 V4.0F

df -k /users
Filesystem 1024-blocks Used Available Capacity Mounted on
users_domain#user 26658816 11257193 2978840 80% /users

ls -ld /etc/fdmns/users_domain/*
lrwxr-xr-x 1 root system 9 Oct 23 2005 /etc/fdmns/users_domain/re0c -> /dev/re0c

The disk is since 9 years connected via xcr dual channel RAID controller. We did not change the hardware recently.

Model name: KZPSC Number of channels: 3 Installed in: PCI slot 20 Firmware version: 2.42 Cache size( in MB ): 4

Logical Drive
Drive RAID Size Cache Group Current
Number: Level: (in MB): Policy: Spanned: Status:
---------------------------------------------------------
0 5 26034 WRITE THRU 0 OPTIMAL

Group 0 : <0,0><0,2><1,1><1,3>

<0,0> DEC RZ1DB-CA (C) DEC LYJ0 W OPT
<0,2> DEC RZ1DB-CS (C) DEC 0307 W OPT
<1,1> DEC RZ1DB-CA (C) DEC LYJ0 W OPT
<1,3> DEC RZ1DB-CA (C) DEC LYJ0 W OPT

Any obvious ideas? How to troubleshoot hung processes on Tru64?

[Ivan Ferreira]

This problem occurs only with incremental backups or also full backups?

You said tape works, but have you tried running the backup to /dev/null?

I would run a "/sbin/advfs/verify" on the domain.

[Neelmani Pandey]

verify will not work. as OS version is 4.0,

what i suggest u check fsck

The following command checks all the file systems in the /etc/fstab file
but makes no corrections:

# fsck -n

check if any error accurs

[Geert Van Pamel]

Thanks for your suggestions!

"Verify" on Tru64 V4 can only run on *unmounted* domains. Because /usr was mounted to the same domain as /users and we could not shutdown our (production)machine we could not verify the AdvFS disk.

But in the end, the problem was *not* AdvFS but an NFS server that was not responding.

One of my users created an NFS mount point on the /users disk; when the NFS server did not respond, the vdump backup could not continue beyond the mount point...

Easy solution:

- create a dedicated small AdvFS file system e.g. /mnt to contain all of the remote mount points. The /mnt partition does not need to be backed up then, since it does *not* contain local data.
- create a symbolic link to the /mnt/xxxxxxxx mount point. Since the target of a symbolic link is not verified by vdump, the backup can still continue even when the NFS server does not (temporarily) respond.

df -kt nfs
Filesystem 1024-blocks Used Available Capacity Mounted on
...
NFS3 server xxxxxxxx not responding still trying
(df or vdump hanging on NFS disk)
^c

Thanks anyway... and we should think beyound Tru64 AdvFS when troubleshoting assumed disk errors.