cannot unmount filesystem lsof and fuser do NOT show it as busy

[Steve Post]

I have looked in the forums and found a lot of people that have had the problem where they cannot unmount a filesystem.

They all say to install lsof and run "lsof /dev/vgbck/lv01  "  to find the process that is hanging it.    NOPE.  No process.

One says to run  "UNIX=95 fuser /dev/vgbck/lv01"  nope.

Reboot?  Nope

UNIX=95 lsof?  Nope

cd /  before unmounting?   Nope

NFS in use?  Nope

my only GUESS is a process called postmaster used by sfmdb.  But I can find nothing on it.

It appears that if I remove the filesystem from /etc/fstab and reboot, the filesystem is no longer mounted.  However, if I add it back into fstab and remount, it hangs. 

And .....it hangs on whatever mount point I gave it.  I mounted it to a different spot.  That seemed to work ok.  Then I tried unmounted it.  Still ok.... Then I mount the unmounted filesystem to /entbck.  BOMB.  It saysit is already mounted....but it is NOT.

"UX:vxfs mount:ERROR: V0-3-21264: /dev/vgbck/lv01 is already mounted, /entbck is busy, allowable number of mount points exceeded."

itanium rx2200-i2  hpux11.3.    onlineJfs is running. I also have glance plus. 

I have tried this on TWO identical itanium boxes.  I have tried this on an EMC filesystem, and an unused vg00 filesystem.

My only dumb solution is to remove the mount point from  /etc/fstab, rebooot, wait 10 minutes, then add it back into /etc/fstab.

I find this error a bit crazy.   Any ideas where the problem lies?

[Kris_Knigga]

Well, sfmdb is part of System Fault Manager, but I don't think that should be touching anything other than maybe /opt, /usr, /var, and /tmp.

Try running lsof and fuser on the mountpoint, not the LV's device file.  Examples:

# fuser -cu /tmp
/tmp:     2797o(root)    2787o(root)    2779o(root)    2771o(root)

# /usr/local/bin/lsof /tmp
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
postmaste 2745  sfmdb    6u  unix 64,0x4      0t0 1326 /tmp/.s.PGSQL.10864 (0x58122380)
snmpdm    2757   root    5u  unix 64,0x4      0t0 1353 /tmp/.AgentSockets/A (0x58122c80)
hp_unixag 2771   root    1r   DIR 64,0x4     8192    2 /tmp
hp_unixag 2771   root    2r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    1r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    2r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    3u   REG 64,0x4        0    9 /tmp/ipv6agt.crashlog
mib2agt   2787   root    1r   DIR 64,0x4     8192    2 /tmp
mib2agt   2787   root    2r   DIR 64,0x4     8192    2 /tmp
trapdesta 2797   root    1r   DIR 64,0x4     8192    2 /tmp
trapdesta 2797   root    2r   DIR 64,0x4     8192    2 /tmp

Post the output here so we can take a gander.

[Steve Post]

lsof /entbck  

NOTHING

fuser /entbck

NOTHING

UNIX=95  lsof /entbck

NOTHING

UNIX=95 fuser /entbck

NOTHING

lsof /dev/vgbck/lv01

NOTHING

UNIX=95   lsof /dev/vgbck/lv01

NOTHING

fuser /dev/vgbck/lv01

NOTHING

UNIX=95 fuser /dev/vgbck/lv01

NOTHING

umount /entbck

umount:  cannot unmount /dev/vgbck/lv01  : Device busy

umount:  return error 1.

When I say "NOTHING", I really am getting a null response.  I get the device name or mount point, followed by a colon, then the unix prompt.

And I tried fuser -c and fuser -cu.

[Kris_Knigga]

I know you said you checked NFS, but I would still give this a try:

grep entbck /etc/dfs/sharetab

[Steve Post]

Yeah I tried that too.   I had it in there, but I removed it.  It had no affect.   And it was not nfs mounted anywhere. 

Strange enough, a REBOOT will not fix it.  But a reboot with it flat out removed from /etc/fstab works for a FEW minutes until I add it back into fstab.  It's crazy. I really wish I could find some clue as to why it says it is busy, yet is not.

I ran "what /sbin/umount" it is from PHCO_38752

I should look up that patch.

[Kris_Knigga]

Yeah, that'd be my next step.

[Dennis Handly]

>And it was not NFS mounted anywhere.

It doesn't have to be mounted, just exported, to hose you over.

[Steve Post]

Thanks for the NFS advice.  I know it can't be NFS though. I have a number of other vg00 filesystems that are not part of the standard hpux. One of them has this problem too.  It is /dev/vg00/lvol10 mounted to /dumplog.  This one never used NFS.

But it seems like the /etc/fstab file is involved a bit too much.  Listen to THIS craziness.  I removed /entbck from the fstab file.  I rebooted.  I made a new mount point /entbckL.   I could unmount and mount to it no problem.  I added /entbckL to /etc/fstab.  Seems ok.  Now I unmounted it, commented out the /entbckL from the fstab file and tried to mount the unmounted filesystem to /entbck......WAIT A MINUTE......

HA!

I found that if I removed the /entbck mount point and recreated it, things begin to work again.  I still do not know the cause.  But I have a really DUMB workaround.

1. remove the filesystem from fstab

2. reboot

3. delete the mount point, and recreate it (aka rmdir/mkdir).

4. wait 120 seconds for the stealth process of evil to get done taking over stuff.

(oh this step is a little wishy-washy).

5. mount the disk

6. edit /etc/fstab.

This is NOT really the solution because I have to reboot. 

[whiz_kid911]

u can forcefully dismount the filesystem

vxumount -o force /mountpoint

[Steve Post]

yep.  My choices are:   possibly corrupt a filesystem, crash it, or reboot.  I was hoping to find the process that is hanging on, and staying hidden.  The forceful unmount should kill it.  But I would rather identify it.

edit:  GOSH... This is terribly inconvenient.  forcing an unmount fails too.   So my ONE AND ONLY OPTION is to comment out the filesystem in /etc/fstab, then reboot.   

Very, very wrong....correction....WEIRD.

I have had a support call into HP for the past 2 days.  I have not gotten an answer.  I guess they think it is weird too.

edit...past 5 days and counting.

I wouldn't be whining about it, except that these commands are BROKEN?  fuser, umount, lsof?