Why MC/sg package does not mount again ?

[Frank de Vries]

The day before yesterday I switched a Mc/sg package to the primary node. (We did not touch
the secondary node).
Today I am switching back and it gives an error:
Mar 17 08:04:46 - Node "orasrv2": Mounting /dev/vgtst/lvbackup at /tstnew/backup
vxfs mount: /dev/vgtst/lvbackup is already mounted, /tstnew/backup is busy,
or allowable number of mount points exceeded
ERROR: Function check_and_mount
ERROR: Failed to mount /dev/vgtst/lvbackup to /tstnew/backup
Mar 17 08:04:47 - Node "orasrv2": Deactivating volume group vgtst
Deactivated volume group in Exclusive Mode.
Volume group "vgtst" has been successfully changed.

I am very surprised by this:
I can't see why it could be busy as it was halted and bdf shows none of the FS.
Furthermore I can still start the package on the primary node.
And it has always run on the secondary node,
which is its default location. So it was allowed before why not now. It does not make sense.

What am I missing here ?
Cheers

pac

[RAC_1]

What does package log say?? What does cmviewcl say for that package?? Are you sure nothing is mounted??

[R Berwald · Accepted solution]

Check if there are any processes in the directory busy, eg. a user-shell with curent path /tstnew/backup.
You can use fuser to do so. lsof can help too.

If you can't find any processes, just rename /tstnew/backup to /tstnew/backup_old, create a new /tstnew/backup and mount your file-system.

[Frank de Vries]

Hi Rac
Sorry I went straight to it.
The package was halted.
It is that what the logs says,
I copied and pasted exactly from the log.

As you know the package control script
runs a loop for all the file mounts,
and it failed on the first one in the list,
and halted. Quite simple realy, only the
reason why appears not to be simple.

And yes I am very SURE, bdf showed up
nothing. (Anyway to my experience if a package is halted NOTHING SHOULD BE MOUNTED)

so I gather from your comment that I am not the only one who finds it bizar.

[Devender Khatana]

Hi,

The only reason seems that either the mount point is busy or it lies under some other mounted file system.

fuser will help you to find out the first option and check the directory availablility abd physically mounting the file system on mount point to know a bit more about the error.


HTH,
Devender

[huruilin]

do you have tryed "fuser -cku /dev/vgtst/lvbackup "? Then umount/mount it.
I think there maybe some process make it busy

[Muthukumar_5]

Mar 17 08:04:46 - Node "orasrv2": Mounting /dev/vgtst/lvbackup at /tstnew/backup
vxfs mount: /dev/vgtst/lvbackup is already mounted, /tstnew/backup is busy,
or allowable number of mount points exceeded

>>>>>>>

Busy means,

1) Some user is still accessing the mounted directory of files.

2) You are trying to mount it under a directory which is also mounted one.

Check points:

1) fuser -cuk will remove user who are using that directory

2) mount -p

will give mounted directory list. Are you seeing your directory as a child directory in any of them which are using NFS.

--
Muthu

[RAC_1]

Also, run lsof on fs/mount_points under this package. I feel lsof gives better results then fuser.

[Frank de Vries]

Hmmm,

Majority of you lean towards the idea
that a process is running on it.

Al the FS from the Mc/sg package are
mounted straight on top of the root /

And yes / root is always busy, UNIX IS RUNNING ON IT.

and as to fuser:
[root@orasrv2:]/etc/cmcluster/tstnew<>>> fuser -cku /dev/vgtst/lvbackup
/dev/vgtst/lvbackup: fuser: could not find file system mounted at /dev/vgtst/lvbackup.

So , I still can't see it how it is done:
Before the package is started, nothing is
is yet mounted except / and /tmp
(it isn't even in fstab)

The tip for moving _backup is a interesting one, I will try that anyway

Thanks for the mount -p command !!

I also found out that Mc/sg has a build in
fuser -cu command which can be activated
with FS_MOUNT_RETRY_COUNT > 0,
so I will try that as well.

Unfortunately I cannot experiment right
now, will try that later, and will
allocate generous points for your input.

More ideas always welcome :)
Thanks

[Devender Khatana]

Hi Frank,

run fuser on the mount point and not the file system device file so

#fuser -cu /tstnew/backup

After finding the processes run it with -k option to kill the processes.

HTH,
Devender

[Frank de Vries]

Hi Devender

How would you
fuser -cu on /tstnew/backup
when it is not mounted and it cannot be
mounted ?

[root@orasrv2:]/<>>> fuser -cu on /tstnew/backup
on: fuser: could not find file system mounted at on.
[root@orasrv2:]/<>>>cmrunpkg -v tstnew
will not mount it either (see message above)

[Devender Khatana]

Hi,

Yes, it will not do that but will do it without -c option.

#fuser -u /tstnew/backup

will give you the PID of processes using mount point and not allowing the mount.

Also can you try mounting some other local file system here if possible to eliminate this probability.

HTH,
Devender

[Frank de Vries]

Devender ,
You were right

fuser -u /tstnew/backup
/tstnew/backup: 26240c(root) 28105c(root) 26239c(root)

These are rlogins, just sitting on
the directory.

Plain simple.
So , you were all right, just didn't see it.

Thanks a million to all of you !!!!!!