topic Re: Very Stange Problem - HP-UX 11.11 in Operating System - HP-UX

Very Stange Problem - HP-UX 11.11

Lee Harris_5 — Wed, 25 Jun 2008 10:57:09 GMT

Hi,

I hope someone can help me because this is a very strange problem. I have noticed that it seems that I am not able to run commands from within certain filesystems without them hanging.

It seems that the problems only occur when my $PWD is a filesystem that resides on external disk. I cannot run commands such as...

rm (with the -rf options)
which
man

...these just hang, BUT, these commands...

ls
cd
rm (with no options)
top

...all run fine.

However, the problems do not appear to be present if $PWD is a filesystem on internal disk.

There are no disk related errors in syslog, or dmesg, ioscan looks fine, the load on the box is very low, iowait is very low (mostly 0 with the occasional 2-5% WIO).

I'm totally confused by this behaviour. Does anyone have any clues for me?

Thanks - Lee

Re: Very Stange Problem - HP-UX 11.11

OFC_EDM — Wed, 25 Jun 2008 11:03:29 GMT

Last 2 times I saw this had similar load symptoms....but the disk utilization was high.

There was no consistency in which commands were slow to respond.

Came down to someone accidentally kicking off a full backup during production hours. Killed the backup, not the admin ;) , and all returned to normal.

A 2nd time it was a process spawning hundreds of smaller processes. Load was low but number of processes running were insane. Killed jobs and system returned to normal.

Re: Very Stange Problem - HP-UX 11.11

James R. Ferguson — Wed, 25 Jun 2008 11:21:32 GMT

Hi Lee:

Any NFS mounts involved?

Is '.' (the dot) in your PATH and where? If so, the beginning? the end?

When did yuo first observe this? What changed just before you noticed this behavior?

Regards!

...JRF...

Re: Very Stange Problem - HP-UX 11.11

Lee Harris_5 — Wed, 25 Jun 2008 11:32:30 GMT

Thanks for the replies guys... as far as I know nothing significant changed before the behaviour began.

The filesystems I'm working from within, e.g. the ones I'm in when I issue the commands that hang are local (on external SCSI disks) and not NFS mountpoints. However we are running an NFS server on this machine that is exporting some of the affected filesystems, but not all.

dot (.) is not in my $PATH anywhere.

Re: Very Stange Problem - HP-UX 11.11

Steven E. Protter — Wed, 25 Jun 2008 12:06:29 GMT

Shalom,

JRF correctly points out that this behavior is typical of stale NFS mounts. Might there be a softlink to an NFS mount buried in there.

The alternative is that there is a bad second of disk the file system is sitting on or an open file handle with an i/o lock.

fuser -cu /filesystem

SEP

Re: Very Stange Problem - HP-UX 11.11

Lee Harris_5 — Thu, 26 Jun 2008 15:11:33 GMT

Thanks for your help... I never did get to the exact cause of the problem, and we had to reboot the server anyway, at which point once it came back up it was behaving itself again. I'm guessing the problem was to do with the NFS services running on the box.