cannot unmount filesystem lsof and fuser do NOT show it as busy

[Steve Post]

I have looked in the forums and found a lot of people that have had the problem where they cannot unmount a filesystem.

They all say to install lsof and run "lsof /dev/vgbck/lv01  "  to find the process that is hanging it.    NOPE.  No process.

One says to run  "UNIX=95 fuser /dev/vgbck/lv01"  nope.

Reboot?  Nope

UNIX=95 lsof?  Nope

cd /  before unmounting?   Nope

NFS in use?  Nope

my only GUESS is a process called postmaster used by sfmdb.  But I can find nothing on it.

It appears that if I remove the filesystem from /etc/fstab and reboot, the filesystem is no longer mounted.  However, if I add it back into fstab and remount, it hangs. 

And .....it hangs on whatever mount point I gave it.  I mounted it to a different spot.  That seemed to work ok.  Then I tried unmounted it.  Still ok.... Then I mount the unmounted filesystem to /entbck.  BOMB.  It saysit is already mounted....but it is NOT.

"UX:vxfs mount:ERROR: V0-3-21264: /dev/vgbck/lv01 is already mounted, /entbck is busy, allowable number of mount points exceeded."

itanium rx2200-i2  hpux11.3.    onlineJfs is running. I also have glance plus. 

I have tried this on TWO identical itanium boxes.  I have tried this on an EMC filesystem, and an unused vg00 filesystem.

My only dumb solution is to remove the mount point from  /etc/fstab, rebooot, wait 10 minutes, then add it back into /etc/fstab.

I find this error a bit crazy.   Any ideas where the problem lies?

[Kris_Knigga]

Well, sfmdb is part of System Fault Manager, but I don't think that should be touching anything other than maybe /opt, /usr, /var, and /tmp.

Try running lsof and fuser on the mountpoint, not the LV's device file.  Examples:

# fuser -cu /tmp
/tmp:     2797o(root)    2787o(root)    2779o(root)    2771o(root)

# /usr/local/bin/lsof /tmp
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
postmaste 2745  sfmdb    6u  unix 64,0x4      0t0 1326 /tmp/.s.PGSQL.10864 (0x58122380)
snmpdm    2757   root    5u  unix 64,0x4      0t0 1353 /tmp/.AgentSockets/A (0x58122c80)
hp_unixag 2771   root    1r   DIR 64,0x4     8192    2 /tmp
hp_unixag 2771   root    2r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    1r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    2r   DIR 64,0x4     8192    2 /tmp
ipv6agt   2779   root    3u   REG 64,0x4        0    9 /tmp/ipv6agt.crashlog
mib2agt   2787   root    1r   DIR 64,0x4     8192    2 /tmp
mib2agt   2787   root    2r   DIR 64,0x4     8192    2 /tmp
trapdesta 2797   root    1r   DIR 64,0x4     8192    2 /tmp
trapdesta 2797   root    2r   DIR 64,0x4     8192    2 /tmp

Post the output here so we can take a gander.

[Steve Post]

lsof /entbck  

NOTHING

fuser /entbck

NOTHING

UNIX=95  lsof /entbck

NOTHING

UNIX=95 fuser /entbck

NOTHING

lsof /dev/vgbck/lv01

NOTHING

UNIX=95   lsof /dev/vgbck/lv01

NOTHING

fuser /dev/vgbck/lv01

NOTHING

UNIX=95 fuser /dev/vgbck/lv01

NOTHING

umount /entbck

umount:  cannot unmount /dev/vgbck/lv01  : Device busy

umount:  return error 1.

When I say "NOTHING", I really am getting a null response.  I get the device name or mount point, followed by a colon, then the unix prompt.

And I tried fuser -c and fuser -cu.

[Kris_Knigga]

I know you said you checked NFS, but I would still give this a try:

grep entbck /etc/dfs/sharetab

[Steve Post]

Yeah I tried that too.   I had it in there, but I removed it.  It had no affect.   And it was not nfs mounted anywhere. 

Strange enough, a REBOOT will not fix it.  But a reboot with it flat out removed from /etc/fstab works for a FEW minutes until I add it back into fstab.  It's crazy. I really wish I could find some clue as to why it says it is busy, yet is not.

I ran "what /sbin/umount" it is from PHCO_38752

I should look up that patch.

[Kris_Knigga]

Yeah, that'd be my next step.

[Dennis Handly]

>And it was not NFS mounted anywhere.

It doesn't have to be mounted, just exported, to hose you over.

[Steve Post]

Thanks for the NFS advice.  I know it can't be NFS though. I have a number of other vg00 filesystems that are not part of the standard hpux. One of them has this problem too.  It is /dev/vg00/lvol10 mounted to /dumplog.  This one never used NFS.

But it seems like the /etc/fstab file is involved a bit too much.  Listen to THIS craziness.  I removed /entbck from the fstab file.  I rebooted.  I made a new mount point /entbckL.   I could unmount and mount to it no problem.  I added /entbckL to /etc/fstab.  Seems ok.  Now I unmounted it, commented out the /entbckL from the fstab file and tried to mount the unmounted filesystem to /entbck......WAIT A MINUTE......

HA!

I found that if I removed the /entbck mount point and recreated it, things begin to work again.  I still do not know the cause.  But I have a really DUMB workaround.

1. remove the filesystem from fstab

2. reboot

3. delete the mount point, and recreate it (aka rmdir/mkdir).

4. wait 120 seconds for the stealth process of evil to get done taking over stuff.

(oh this step is a little wishy-washy).

5. mount the disk

6. edit /etc/fstab.

This is NOT really the solution because I have to reboot. 

[whiz_kid911]

u can forcefully dismount the filesystem

vxumount -o force /mountpoint

[Steve Post]

yep.  My choices are:   possibly corrupt a filesystem, crash it, or reboot.  I was hoping to find the process that is hanging on, and staying hidden.  The forceful unmount should kill it.  But I would rather identify it.

edit:  GOSH... This is terribly inconvenient.  forcing an unmount fails too.   So my ONE AND ONLY OPTION is to comment out the filesystem in /etc/fstab, then reboot.   

Very, very wrong....correction....WEIRD.

I have had a support call into HP for the past 2 days.  I have not gotten an answer.  I guess they think it is weird too.

edit...past 5 days and counting.

I wouldn't be whining about it, except that these commands are BROKEN?  fuser, umount, lsof? 

[Steve Post]

I'm am certain that eventually I will find my error.   I just have not found it yet.

There was an update to VXFS as of 9 March 2012.  I thought that would fix my problem.  It did not.

Getting the filesystem to properly unmount is a bit like pulling a bone away from a really BIG junkyard dog.

For your information, here is my hand to hand combat with the filesystem.

....oh... cut and paste does not work?   Ok... at least I can TYPE.  

I will call the filesystem  /dev/vgbck/lv01  /back

I put in patches.  And reboot.  I still cannot unmount  /back

I remove  /back from the /fstab and reboot.  Now I cannot MOUNT /back because something says that /dev/vgbck/lv01 IS IN USE ALREADY! <chomp goes that dog>

What's using it? 

lsof -V  /dev/vgbck/lv01   <nothing>

fuser /dev/vgbck/lv01  <nothing>

lsof -V /back  <nothing>

....yadda yadda yadda ....<nothing>

OK ....chomp goes that GHOST dog

I flat out remove the mount point.   (THERE.  Try to use THAT)  And I reboot.

I can't mount it because the mount point is gone.  I deleted it. 

So I mkdir /back.  And I run  mount  /dev/vgbck/lv01  /back

It mounts.

I run  umount  /back

It UN mounts fine.

mount/unmount/mount/unmount......yep.  No problems anymore.

So I update /etc/fstab to have it mount.

I unmount /back

I run mount -a

It is fine.

mount/unmount/mount/unmount.....STILL...NO problems anymore.

I reboot?   NOT YET....  I need to work on other stuff.  But I bet it will start to fail again after a reboot.

the force options do not work.

the vxumount  -oforce  /back   says it is UNlicensed.  This is silly since I have onlineJFS installed, and the vxfs type (via ftyp  -V /dev/vgbck/lv01) says it is type SEVEN. 

Whatever.... I guess this will look more like a blog, than a solution.  

So for at least the next day or two, the umount command will work for me.

PS: Yyou just KNOW it has to be something stupid.  I always hope the cause of the error is something I did.  At least that gives me something I can solve.

[Steve Post]

03May2012

I tried to set up an rc file to mount the UNMOUNTABLE filesystem after everything else is up.
I kept the filesystem out of fstab.
After everything is up I ran (via rc script):
/usr/sbin/vgchange -a y /dev/vgbck
/usr/sbin/fsck -F vxfs -y /dev/vgbck/lv01
/usr/sbin/mount -F vxfs -olargefiles,delaylog,datainlog /dev/vgbck/lv01
/back

IT WORKED!   IT DID NOT WORK!   It died after I finished writing up my solution.

Oh.  Back to square one.

Well the good news is that I can unmount file filesystem with this in a mere 2 reboots.

It seems a bit harsh.

I also looked into fuser and lsof.  the file system is "in use."  Yet lsof and fuser say it is NOT in use.  I don't even care if it is in use or not.  I just want to be able to unmount it.   (And FORCING the mount fails too by the way). 

I have a big database running.  I know logical volume /dev/vgtst1/lv01 is is use.   I run lsof /dev/vgtst1/lv01.Lsof says that nothing is running on it.  I run lsof -V /dev/vgtst1/lv01.  It says the filesystem is not in use.   I repeat this with fuser.   Nope.  Both commands say this is not in use.  

ok...FINE.   I KNOW this is in use.   I wonder if the mount point for the ROOT filesystem is in use?  YES.  Yes it is.  I get a lot of stuff.  I run   lsof -V   /   | grep vgtst'\/lv01  and Ah HA!   There is is.  So I can find a process for a logical volume that I know is in use.  All I have to do is search for the filesystem I want to unmount.

Ok.... So now I know that lvof -V / will show all the hooks.  I cannot find anything that is use /back and or /dev/vgbck/lv01.  Back to square one....again.

So for this entry: 

rc script and removal from /etc/fstab did not solve my problem.  It fixed it only for a few minutes.

And the lsof -V command can't find any processes unless I run it on ROOT, and grep for stuff.

For earlier entries:

forcing the umount fails.

Putting in patches for Vxfs did not fix it.

[Steve Post]

Ok.  thanks to HP, I found the problem.

In the first second of the error (so many months ago), I figured it was NFS.  I immediately made sure I didn't have anything NFS mounted.  That didn't even take a second.  I saw I had nothing NFS mounted.

Why I could not unmount?   IT WAS NFS!   But in a weird way.

You see, I have /etc/dfs/dfstab setup so I could mount this one, certain filesystem.  But it was NOT umounted up.   Well?   I guess that's close enough.  Just the small desire to someday NFS mount this one file system was good enough for the system to decide to never allow it to unmount, and never tell me why.

How HP found it?  I told them that I thought is was something in the startup.  They had me shutdown to single user mode and run each startup command one at a time.   They are a LOT of those.  But I did it anyway.  And WHILE I was running every rc/S#### link, I would try to unmount that filesytem /back.   Well I got to the NFS server and it decided to not allow me to unmount filesystem /back!    A HA!

Solution:  Maybe there is a patch on this.   But here is MY solution.  It will take about 5 seconds to implement.

1.  unmount filesytem.  Error?

2.  make sure the filesystem is not in use.  via lsof -V /back   and lsof -V /dev/vgbck/lv01  and fsuser.

3.  Still error?   Stop the NFS server via  /etc/init.d/nfs.server stop

4.  There is no step 4.  THAT is it.

BIG PS:  

No wait a second.   Forum person Mkdlxk told me I should grep for the filesystem in /etc/dfs.  but he didn't tell me anything else.  

Even if I:   "removed the /back from that /etcdfs/dfstab, and ran exportfs -a" it would have no affect.  I still would have a unmountable filesystem.

I would have to also stop and start the nfs server.

THE REAL SOLUTION is below.   This was MY solution. I put in way too much effort on this.

[Kris_Knigga · Accepted solution]

You said that you removed it from /etc/dfs/sharetab, so I didn't go any farther down that route.  I take it you put it back in?

An 'unshareall' should do the trick.  You shouldn't have to completely stop NFS.

[Steve Post]

Yep.  This is how the forums work for me.  I ask a question.  I get answers that don't help or no response for weeks.

I dig and dig and finally get a solution.

I post the solution.

15 minutes later I get the answer I was looking for when I first entered the question.

Thanks.  It is too bad the timing of that is always off.   Perhaps the next time I have a problem, I should pretend I solved it.  Then I will get the solution immediately.

unshareall is the solution.

I knew it would be a simple solution.  I knew it was a bonehead problem.  I just could not read minds.

[Kris_Knigga]

You insisted that you checked NFS and when I asked about /etc/dfs/share you said, "I had it in there, but I removed it. It had no affect.". Why would I push the issue?

[Steve Post]

OH......It is just big communications problem here.  You did fine.  You gave me the true solution.  I actually HOPED that since I had my GOOFY solution, some one would immediate give the REAL solution.  And you did.  THANK YOU.

What I know as absolute trues are now not so absolute. 

Some things that are not true anymore.

fuser will tell you if something is in use. 

Lsof will tell you if something is in use.

NFS will not get in the way of a filesystem because the filesystem is not nfs-mounted anywhere.  

And there is a new command called unshareall that has probably been in this universe for years.

[Dennis Handly]

>I immediately made sure I didn't have anything NFS mounted.

I also mentioned that you must not have it NFS exported.  That won't show up by lsof.

>And there is a new command called unshareall that has probably been in this universe for years.

Only for 11.31.

[Kris_Knigga]

If the NFS daemons truely do not hold open filesystems that contain NFS exports (suggested by the lack of help from fuser or lsof), then I'm guessing a list of NFS exports is consulted when an unmount is attempted.  If this is the case, maybe a request for enhancement should be created to give a more useful error message?  I'm sure this bites most people new to 11.31 at some point.

[Steve Post]

I did not THINK I had it nfs exported or nfs mounted.
I ran bdf on the client box. It was not there.
I removed had it NFS exported. BUT....
I emptied /etc/dfs/dfstab
I ran "exportfs -a"
And the 11.3 apparently decided that wasn't good enough release the NFS export. Too bad 11.3 did NOT tell me so.

[Kris_Knigga]

I don't think 'exportfs -a' unexports/unshares things that have gone missing from dfstab. I believe you've got to 'exportfs -u <exported path>', 'exportfs -ua', 'unshare <exported path>', or 'unshareall'.

[Steve Post]

"If the NFS daemons truely do not hold open filesystems that contain NFS exports (suggested by the lack of help from fuser or lsof), then I'm guessing a list of NFS exports is consulted when an unmount is attempted.  If this is the case, maybe a request for enhancement should be created to give a more useful error message?  I'm sure this bites most people new to 11.31 at some point."

Kris, you are SO RIGHT.  I got bit!

I know where to look now.  I'll view the contents of /etc/dfs/sharetab to make sure the filesystem to unmount is not listed there.

I see I can also run:   "unshare   /back" in the event I need to unhook an NFS exported filesystem yet keep others still NFS mounted to other client computers.

[Matti_Kurkela]

Like /etc/mnttab is used by the mount/umount commands to track currently mounted filesystems, /etc/dfs/sharetab is used by the share/unshare commands to track currently shared filesystems.

Neither file should be edited by hand in any normal situation.

If you edit them, it will cause confusion, unless you really know what you're doing.

Tools like fuser and lsof will only tell you if something is being used by any userspace process: the kernel NFS server is not an userspace process, although some of its components may appear as processes in the ps -ef listing.

[Dennis Handly]

>I'm sure this bites most people new to 11.31 at some point.

I don't think this has changed for 11.31, except you need to look at /etc/dfs/sharetab instead of /etc/xtab.

>I don't think "exportfs -a" unexports/unshares things that have gone missing from dfstab

Right, exportfs -a only adds filesystems, unless -u is used.  I assume that -au uses the info in /etc/dfs/sharetab.

>Neither file should be edited by hand in any normal situation.

It probably doesn't matter since mnttab is a device file (on 11.31) and can't be edited.