Filesystem not getting unmounted

[raiden]

Let me explain you the background of this problem .

We have /cbsora filesystem integrated in Cluster package xxx. This FileSystem /cbsora is the home for oracle.

Now the problem is whenever we halt the package xxx this Filesystem doesnt unmount, it says the cannot unmount /cbsora. Device busy.

We have also found the exact cause. This happens when any database administrator has sudo to oracle ID ( home of oracle : /cbsora/ora10g) at the moment when the package is halted the FS /cbsora doesnt get unmounts.

Now the problem to be solved is that why this /cbsora is not getting unmounted. When the package halt script is running fuser -ku this should get unmount but its not getting unmount.

The oracle Database shutdowns properly during package halt so oracle has told that its problem from HP-UX side.

Guys how do we resolve this Issue because this is creating a huge problem because the package failover is not happening in case of any disaster on our primary node because unless the package halt is successful on primary node the package will not start on secondary node. Plzz help

[Lijeesh N G_1]

Hi,

1)Check package control log and syslog,what its saying??

2)Is manual unmounting successfull??? What its output??

3)Provide some more details about cluster,
#cmviewcl -v, etc..

Regards,
LIJEESH N G

[Ganesh.A]

Hi,

Pls run the following command and see any process is still using the FS and kill/stop them as necessaary.

fuser -cu /cbsora

[Raj D.]

Raiden,
-You need to check in the control file , /etc/cmcluster/pkg.cntl if the oracle shutdown and kill commands are given.

-Also while package shutdown keep an eye on the log file , pkg.cntl.log

-What was the last error before it says unable to unmount the /cbsora filesystem.

-Also keep another session with # fuser -cu /cbsora/ora10g , you will what processes are holding it from geting it unmounted.

- Can you unmount the /cbsora manually , after package halt. If not then to check what process attached to this fs and to troubleshoot further.

Cheers,
Raj.

[sen_ux]

Being home directory of oracle user, can it be unmounted when the oracle user logged in.????
Logout the oracle user and try a manual switching of package.It should success.

[raiden]

@ Lijeesh , Raj

Package control log says that the Filesystem /cbsora cannot be unmounted. FileSystem busy
Manual unmounting is also not possible after package halt unless i tell the oracle user to logout.

In package control file I see some some root and oracle named process using that fileSystem and I guess these are process created from user oracle login.

We and Database team are sure that the oracle is getting shutdown properly before the FS unmounting starts during package halt.

@ sen

The FS canot be unmounted when any oracle user is logged in .The only solution we are trying till now during package failover is to tell the oracle user to logout from their sessions.


But what in case if any disaster happens???? . Thats the only concern for us and my management because last week by manual mistake the primary node was shutdown and due to these existing oracle login the package did not halted successfully on primary and hence failed to start on secondary.


How do we take care of this .. please suggest guys.... because all blames are on Unix..... Why the fuser -ku command is not working for that Filesystem /cbsora during package halt..... ... . Please help ...

[Raj D.]

Raiden,

"Manual unmounting is also not possible after package halt unless i tell the oracle user to logout. "

I am wondering how oracle shutsdown itself during pkg halt,& without killing /closing the 'oracle users' session !!

You mentioned that , manual unmount also not possible unless you tell the oracle user to logout. The oracle shutdown step shuould kill/close any open oracle session , and then it should shutdown/dismount the database. Oracle should take care of this . Make sure you have dbhalt script properly in place in the pkg.cntl file under customer defined function.


hth,
Raj.

[sen_ux]

Can you paste the fuser command section of the package script.?

[raiden]

@ Raj

Please understand that these are not database level logins ,, I am talking about system level logins wherein a normal database team users logins with his ID....then does sudo to oracle to connect to SQL prompt... I hope now you understand I am talking about system level logins..

Oracle is doing its job properly .... It is properly disconecting all users who are connected through Application before shutting it down .. The problem is only with this stupid HPUX box who is not able to kill the user level login who are using the /cbsora Filesystem ( the home of oracle ID)

[Raj D.]

Raiden,

So here you go ,
To put an extra line in the customer_defined_start script like below :

--> After oracle db halt commands in
pkg.cntl file:
#---------
kill -9 `who -u | grep oracle | awk '{print $7}' | xargs`
fuser -cu /cbsora
fuser -ku /cbsora
echo "Checking /cbsora active processes again .."
fuser -cu /cbsora >> fuser.cbsora.final.txt
#---------


Hope this will help to kill the left over prcesses to /cbsora and it should umount fine. Also fuser.cbsora.final.txt file should be blank if all killed properly with the extra script.


Cheers,
Raj.