cannot unmount (exported) filesystem lsof and fuser do NOT show it as busy

[Steve Post]

I'm am certain that eventually I will find my error.   I just have not found it yet.

There was an update to VXFS as of 9 March 2012.  I thought that would fix my problem.  It did not.

Getting the filesystem to properly unmount is a bit like pulling a bone away from a really BIG junkyard dog.

For your information, here is my hand to hand combat with the filesystem.

....oh... cut and paste does not work?   Ok... at least I can TYPE.  

I will call the filesystem  /dev/vgbck/lv01  /back

I put in patches.  And reboot.  I still cannot unmount  /back

I remove  /back from the /fstab and reboot.  Now I cannot MOUNT /back because something says that /dev/vgbck/lv01 IS IN USE ALREADY! <chomp goes that dog>

What's using it? 

lsof -V  /dev/vgbck/lv01   <nothing>

fuser /dev/vgbck/lv01  <nothing>

lsof -V /back  <nothing>

....yadda yadda yadda ....<nothing>

OK ....chomp goes that GHOST dog

I flat out remove the mount point.   (THERE.  Try to use THAT)  And I reboot.

I can't mount it because the mount point is gone.  I deleted it. 

So I mkdir /back.  And I run  mount  /dev/vgbck/lv01  /back

It mounts.

I run  umount  /back

It UN mounts fine.

mount/unmount/mount/unmount......yep.  No problems anymore.

So I update /etc/fstab to have it mount.

I unmount /back

I run mount -a

It is fine.

mount/unmount/mount/unmount.....STILL...NO problems anymore.

I reboot?   NOT YET....  I need to work on other stuff.  But I bet it will start to fail again after a reboot.

the force options do not work.

the vxumount  -oforce  /back   says it is UNlicensed.  This is silly since I have onlineJFS installed, and the vxfs type (via ftyp  -V /dev/vgbck/lv01) says it is type SEVEN. 

Whatever.... I guess this will look more like a blog, than a solution.  

So for at least the next day or two, the umount command will work for me.

PS: Yyou just KNOW it has to be something stupid.  I always hope the cause of the error is something I did.  At least that gives me something I can solve.

[Steve Post]

03May2012

I tried to set up an rc file to mount the UNMOUNTABLE filesystem after everything else is up.
I kept the filesystem out of fstab.
After everything is up I ran (via rc script):
/usr/sbin/vgchange -a y /dev/vgbck
/usr/sbin/fsck -F vxfs -y /dev/vgbck/lv01
/usr/sbin/mount -F vxfs -olargefiles,delaylog,datainlog /dev/vgbck/lv01
/back

IT WORKED!   IT DID NOT WORK!   It died after I finished writing up my solution.

Oh.  Back to square one.

Well the good news is that I can unmount file filesystem with this in a mere 2 reboots.

It seems a bit harsh.

I also looked into fuser and lsof.  the file system is "in use."  Yet lsof and fuser say it is NOT in use.  I don't even care if it is in use or not.  I just want to be able to unmount it.   (And FORCING the mount fails too by the way). 

I have a big database running.  I know logical volume /dev/vgtst1/lv01 is is use.   I run lsof /dev/vgtst1/lv01.Lsof says that nothing is running on it.  I run lsof -V /dev/vgtst1/lv01.  It says the filesystem is not in use.   I repeat this with fuser.   Nope.  Both commands say this is not in use.  

ok...FINE.   I KNOW this is in use.   I wonder if the mount point for the ROOT filesystem is in use?  YES.  Yes it is.  I get a lot of stuff.  I run   lsof -V   /   | grep vgtst'\/lv01  and Ah HA!   There is is.  So I can find a process for a logical volume that I know is in use.  All I have to do is search for the filesystem I want to unmount.

Ok.... So now I know that lvof -V / will show all the hooks.  I cannot find anything that is use /back and or /dev/vgbck/lv01.  Back to square one....again.

So for this entry: 

rc script and removal from /etc/fstab did not solve my problem.  It fixed it only for a few minutes.

And the lsof -V command can't find any processes unless I run it on ROOT, and grep for stuff.

For earlier entries:

forcing the umount fails.

Putting in patches for Vxfs did not fix it.

[Steve Post]

Ok.  thanks to HP, I found the problem.

In the first second of the error (so many months ago), I figured it was NFS.  I immediately made sure I didn't have anything NFS mounted.  That didn't even take a second.  I saw I had nothing NFS mounted.

Why I could not unmount?   IT WAS NFS!   But in a weird way.

You see, I have /etc/dfs/dfstab setup so I could mount this one, certain filesystem.  But it was NOT umounted up.   Well?   I guess that's close enough.  Just the small desire to someday NFS mount this one file system was good enough for the system to decide to never allow it to unmount, and never tell me why.

How HP found it?  I told them that I thought is was something in the startup.  They had me shutdown to single user mode and run each startup command one at a time.   They are a LOT of those.  But I did it anyway.  And WHILE I was running every rc/S#### link, I would try to unmount that filesytem /back.   Well I got to the NFS server and it decided to not allow me to unmount filesystem /back!    A HA!

Solution:  Maybe there is a patch on this.   But here is MY solution.  It will take about 5 seconds to implement.

1.  unmount filesytem.  Error?

2.  make sure the filesystem is not in use.  via lsof -V /back   and lsof -V /dev/vgbck/lv01  and fsuser.

3.  Still error?   Stop the NFS server via  /etc/init.d/nfs.server stop

4.  There is no step 4.  THAT is it.

BIG PS:  

No wait a second.   Forum person Mkdlxk told me I should grep for the filesystem in /etc/dfs.  but he didn't tell me anything else.  

Even if I:   "removed the /back from that /etcdfs/dfstab, and ran exportfs -a" it would have no affect.  I still would have a unmountable filesystem.

I would have to also stop and start the nfs server.

THE REAL SOLUTION is below.   This was MY solution. I put in way too much effort on this.

[Kris_Knigga · Accepted solution]

You said that you removed it from /etc/dfs/sharetab, so I didn't go any farther down that route.  I take it you put it back in?

An 'unshareall' should do the trick.  You shouldn't have to completely stop NFS.

[Steve Post]

Yep.  This is how the forums work for me.  I ask a question.  I get answers that don't help or no response for weeks.

I dig and dig and finally get a solution.

I post the solution.

15 minutes later I get the answer I was looking for when I first entered the question.

Thanks.  It is too bad the timing of that is always off.   Perhaps the next time I have a problem, I should pretend I solved it.  Then I will get the solution immediately.

unshareall is the solution.

I knew it would be a simple solution.  I knew it was a bonehead problem.  I just could not read minds.

[Kris_Knigga]

You insisted that you checked NFS and when I asked about /etc/dfs/share you said, "I had it in there, but I removed it. It had no affect.". Why would I push the issue?

[Steve Post]

OH......It is just big communications problem here.  You did fine.  You gave me the true solution.  I actually HOPED that since I had my GOOFY solution, some one would immediate give the REAL solution.  And you did.  THANK YOU.

What I know as absolute trues are now not so absolute. 

Some things that are not true anymore.

fuser will tell you if something is in use. 

Lsof will tell you if something is in use.

NFS will not get in the way of a filesystem because the filesystem is not nfs-mounted anywhere.  

And there is a new command called unshareall that has probably been in this universe for years.

[Dennis Handly]

>I immediately made sure I didn't have anything NFS mounted.

I also mentioned that you must not have it NFS exported.  That won't show up by lsof.

>And there is a new command called unshareall that has probably been in this universe for years.

Only for 11.31.

[Kris_Knigga]

If the NFS daemons truely do not hold open filesystems that contain NFS exports (suggested by the lack of help from fuser or lsof), then I'm guessing a list of NFS exports is consulted when an unmount is attempted.  If this is the case, maybe a request for enhancement should be created to give a more useful error message?  I'm sure this bites most people new to 11.31 at some point.

[Steve Post]

I did not THINK I had it nfs exported or nfs mounted.
I ran bdf on the client box. It was not there.
I removed had it NFS exported. BUT....
I emptied /etc/dfs/dfstab
I ran "exportfs -a"
And the 11.3 apparently decided that wasn't good enough release the NFS export. Too bad 11.3 did NOT tell me so.