all commands hang on file

[David Child_1]

I am having a problem with a file on our systems;

Environment:
- several L3000 servers running HP-UX 11.11
- Veritas VCS (for sharing file system)

Problem: one of the files for a custom application was updated recently. It is some input file to a process engine. Any time we try and run a command against this file it hangs and the terminal you are on locks up. We have tried 'cp, cat, file, rm, strings, mv, etc' and they all exhibit this problem. This file is one of many in the VCS shared file system. This problem happens with this particular file on all servers in the cluster.

My last attempt I used tusc and it ran along for a second until it tried a 'stat64' on the file. At that point it locked up again.

Any ideas?

Note: the file isn't very large (maybe 4k). ('ls -l' does work on it).

Thanks,
David

[A. Clay Stephenson · Accepted solution]

Well, this has all the signs of a corrupt filesystem eventhough this isn't supposed to happen with VCS. I would look through the documentation for whatever the equivalent of fsck is and attempt to fix the filesystem.

[A. Clay Stephenson]

One other thought is that there is some sort of mandatory (as opposed to the more common advisory lock) on this file and the process that set this lock is hung or waiting for some resource to become available. Check you VCS documentation for locking policy. If it isn't obvious this is a WAG on my part but it does explain your symptoms.

[RAC_1]

along with AC's suggestions, run lsof/fuser on that file and see what process is holding the file.
Any messages in vcs log file?? in dmesg, in syslog.log?

Anil

[Robert-Jan Goossens_1]

Hi David,

Try to exclude a disk problem.

# dd if=/dev/rdsk/cxtydz of=/dev/null bs=1024k

Robert-Jan

[David Child_1]

Thanks all,

I believe A. Clay had it. Unfortunately the file system completely hung before we could do much. All the servers in the cluster were bounced and after they came up we had to fsck the file system. Once that was done we were back in business.

RAC, we had actually tried fuser and that hung. We never got a chance to try the lsof.

Robert-Jan, all the disks were okay.

Thanks again,
David