Filesystem not getting unmounted

[Raj D.]

Correction to the above:

To put an extra line in the "customer_defined_halt " script .

Cheers,
Raj.

[raiden]

Raj,

The solution which you have given is already implemented in cluster functionality by default.

Whenever the package is halted the fuser -ku is executed by default to kill any process using the Cluster Package Filesystem.

In my Issue also the command fuser -ku is being executed but still the processes using /cbsora are not killed and thts the only concern.

Please guys suggest some alternative.

[BUPA IS · Accepted solution]

Hello,

I have found that fuser does not always report all the processes using a file system. If you have lsof installed try it using the -t option and using that as input to kill -9 you can use this instead of the fuser -k command .

lsof /cbsora to list them

kill -9 `lsof -t /cbsora`
sleep 30
lsof /cbsora

lsof for hp can be downloaded from the porting and archiving center if you do not have it. Choose a mirror near you .

http://hpux.connect.org.uk/
I hope this miy be of some use
Mike

[R.K. #]

Hello Raiden,

You can also check:
# bdf /cbsora
# du -sk /cbsora

There will be a good difference, as bdf gives usage counting the unkilled processes while du does not count these processes.

This will give the output of processes which are using deleted files.
# lsof +aL1 /ora_temp

List open files that are using file system /ora_temp
# lsof /ora_temp

As said you need to install lsof.

Regds..

[Bill Hassell]

> fuser -ku

This is a very dangerous command to use without first running fuser -u. It will kill processes that currently have the filesystem open and without knowing which processes are being killed, you could cause filesystem corruption.

Secondly, fuser is pitifiully inadequate to discover the reason that a moutpoint is busy. It only works about 50% of the time. Your only choice is to download lsof (the right version for your OS). Then check the output carefully. You'll want to script a realiable method to terminate processes that have the mountpoint open. NOTE: a big problem involves users that login and then cd to the mountpoint directory -- not necessary or desirable.

And if the oracle login uses /cbsora/ora10g as the HOME directory, change this immediately. There is no reason that the Oracle DBA has to be in this directory. I am sure that the DBA learned to cd to the directory to avoid typing long pathnames like /cbsora/ora10g/some-Oracle-command, but is is trivial top fix (set the DBA's $PATH correctly).

[raiden]

@ BUPA IS

Thanks for the solution . Its working perfectly as we wanted. Indeed it killed all the processes. Way better that fuser -ku

I will implement this in our cluster halt script.
kill -9 `lsof -t /cbsora`
sleep 30
lsof /cbsora

Hv now a Reason for cheering. Thanks again

[Bill Hassell]

> kill -9 `lsof -t /cbsora`

Please, never use kill -9, especially on complicated applications and database programs. This is the worst possible way to terminate a program and almost guarantees corrupted data files. Although beginner Unix courses teach kill -9, they always leave out the serious consequences.

There are many, many kill signals (man kill) and most can be handled within the program to perform an orderly shutdown. kill with no value (which is actually kill -15) is always the correct signal to stop a program. If the program needs another signal (and you have no documentation on what is proper), use kill -1 and then kill -2. Each of these signals can be sensed by a program or script to perform a proper close of open files and release shared memory.

kill -9 is very dangerous in that it gives the program no chance at all to perform a proper shutdown. It should only be used manually (never in a script) when you can identify the program and know that a kill -9 will not damage files. kill -9 can leave shared memory badly fragmented and will require a reboot to fix. Your script needs to write the process names that are running into a log, then issue kill -15 for each PID, sleep for 10-20 seconds, then check again. Log any PIDs that still exist and issue kill -1 against those processes. When the script fails to stop all the processes, log the process names with ps -fpPID,PID.. so you can fix the broken code at a later time, then issue the kill -9.

Complicated programs and databases must respond to kill -15, -1 or -2 or they must be repaired for proper operation.

[raiden]

@ Bill

Thanks for the Information Bill. It was valuable.

But the problem which I am facing is on a database Server. During package halt the script is properly shutting down the database.

The only processes utilising the /cbsora FileSystem are from the system level user logins ( which I guess must have been generated when database team users do sudo to oracle).

SO i think i can safely execute kill -9 on these processes . Your advice please.

[Bill Hassell]

Yes, it is safe to kill these shell processes, but use kill -15 rather than kill -9. Then change the HOME location (/cbsora/ora10g) for the sudo logins. The ability to switch the package is much more important than the convenience of a DBA login. The key to automatically killing these processes is to first identify what they are and then use kill -15.