System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Filesystem not getting unmounted

 
SOLVED
Go to solution
Highlighted
Regular Advisor

Filesystem not getting unmounted

Let me explain you the background of this problem .

We have /cbsora filesystem integrated in Cluster package xxx. This FileSystem /cbsora is the home for oracle.

Now the problem is whenever we halt the package xxx this Filesystem doesnt unmount, it says the cannot unmount /cbsora. Device busy.

We have also found the exact cause. This happens when any database administrator has sudo to oracle ID ( home of oracle : /cbsora/ora10g) at the moment when the package is halted the FS /cbsora doesnt get unmounts.

Now the problem to be solved is that why this /cbsora is not getting unmounted. When the package halt script is running fuser -ku this should get unmount but its not getting unmount.

The oracle Database shutdowns properly during package halt so oracle has told that its problem from HP-UX side.

Guys how do we resolve this Issue because this is creating a huge problem because the package failover is not happening in case of any disaster on our primary node because unless the package halt is successful on primary node the package will not start on secondary node. Plzz help
18 REPLIES 18
Highlighted
Respected Contributor

Re: Filesystem not getting unmounted

Hi,

1)Check package control log and syslog,what its saying??

2)Is manual unmounting successfull??? What its output??

3)Provide some more details about cluster,
#cmviewcl -v, etc..

Regards,
LIJEESH N G
Highlighted
Advisor

Re: Filesystem not getting unmounted

Hi,

Pls run the following command and see any process is still using the FS and kill/stop them as necessaary.

fuser -cu /cbsora
Highlighted
Honored Contributor

Re: Filesystem not getting unmounted

Raiden,
-You need to check in the control file , /etc/cmcluster/pkg.cntl if the oracle shutdown and kill commands are given.

-Also while package shutdown keep an eye on the log file , pkg.cntl.log

-What was the last error before it says unable to unmount the /cbsora filesystem.

-Also keep another session with # fuser -cu /cbsora/ora10g , you will what processes are holding it from geting it unmounted.

- Can you unmount the /cbsora manually , after package halt. If not then to check what process attached to this fs and to troubleshoot further.

Cheers,
Raj.

" If u think u can , If u think u cannot , - You are always Right . "
Highlighted
Valued Contributor

Re: Filesystem not getting unmounted

Being home directory of oracle user, can it be unmounted when the oracle user logged in.????
Logout the oracle user and try a manual switching of package.It should success.
Highlighted
Regular Advisor

Re: Filesystem not getting unmounted

@ Lijeesh , Raj

Package control log says that the Filesystem /cbsora cannot be unmounted. FileSystem busy
Manual unmounting is also not possible after package halt unless i tell the oracle user to logout.

In package control file I see some some root and oracle named process using that fileSystem and I guess these are process created from user oracle login.

We and Database team are sure that the oracle is getting shutdown properly before the FS unmounting starts during package halt.

@ sen

The FS canot be unmounted when any oracle user is logged in .The only solution we are trying till now during package failover is to tell the oracle user to logout from their sessions.


But what in case if any disaster happens???? . Thats the only concern for us and my management because last week by manual mistake the primary node was shutdown and due to these existing oracle login the package did not halted successfully on primary and hence failed to start on secondary.


How do we take care of this .. please suggest guys.... because all blames are on Unix..... Why the fuser -ku command is not working for that Filesystem /cbsora during package halt..... ... . Please help ...
Highlighted
Honored Contributor

Re: Filesystem not getting unmounted

Raiden,

"Manual unmounting is also not possible after package halt unless i tell the oracle user to logout. "

I am wondering how oracle shutsdown itself during pkg halt,& without killing /closing the 'oracle users' session !!

You mentioned that , manual unmount also not possible unless you tell the oracle user to logout. The oracle shutdown step shuould kill/close any open oracle session , and then it should shutdown/dismount the database. Oracle should take care of this . Make sure you have dbhalt script properly in place in the pkg.cntl file under customer defined function.


hth,
Raj.
" If u think u can , If u think u cannot , - You are always Right . "
Highlighted
Valued Contributor

Re: Filesystem not getting unmounted

Can you paste the fuser command section of the package script.?
Highlighted
Regular Advisor

Re: Filesystem not getting unmounted

@ Raj

Please understand that these are not database level logins ,, I am talking about system level logins wherein a normal database team users logins with his ID....then does sudo to oracle to connect to SQL prompt... I hope now you understand I am talking about system level logins..

Oracle is doing its job properly .... It is properly disconecting all users who are connected through Application before shutting it down .. The problem is only with this stupid HPUX box who is not able to kill the user level login who are using the /cbsora Filesystem ( the home of oracle ID)
Highlighted
Honored Contributor

Re: Filesystem not getting unmounted

Raiden,

So here you go ,
To put an extra line in the customer_defined_start script like below :

--> After oracle db halt commands in
pkg.cntl file:
#---------
kill -9 `who -u | grep oracle | awk '{print $7}' | xargs`
fuser -cu /cbsora
fuser -ku /cbsora
echo "Checking /cbsora active processes again .."
fuser -cu /cbsora >> fuser.cbsora.final.txt
#---------


Hope this will help to kill the left over prcesses to /cbsora and it should umount fine. Also fuser.cbsora.final.txt file should be blank if all killed properly with the extra script.


Cheers,
Raj.

" If u think u can , If u think u cannot , - You are always Right . "