Operating System - Linux
1748271 Members
3923 Online
108760 Solutions
New Discussion юеВ

Re: shutdown is taking more than about 15 minutes

 
SOLVED
Go to solution

shutdown is taking more than about 15 minutes

Hi.
Linux Node1 2.6.10-telco-1.46-mckinley-smp #1 SMP Fri May 30 18:29:43 UTC 2008 ia64 GNU/Linux

I have above linux platform. When I issue shutdown, sometimes I find system is taking about 15 minutes after it had broadcasted the message about shutdown. I issue shutdown -r now
17 REPLIES 17
Oviwan
Honored Contributor

Re: shutdown is taking more than about 15 minutes

is there any cluster software running or any other software that has a long time to finish?

Re: shutdown is taking more than about 15 minutes

we have dual node software systems. By the way. shutdown should be taking care of it I believe. When system is issued shutdown it sends SIGTERM to all for graceful exit, if processes are still stuck up , then brutal SIGKILL is sent and reboot is forced. Why do you think in any case , shutdown should hang for 15 minutes ?
Wouter Jagers
Honored Contributor

Re: shutdown is taking more than about 15 minutes

Some things just won't die ;-)

Processes can be in an 'uninterruptable sleep', for example when performing I/O operations. They will then only receive the KILL when coming out of that state. Maybe you have a process blocked in such a state for a while. (your fencing mechanism, maybe?)

Can you see where it's hanging, exactly ? The console should be telling you what the machine is currently trying to do. It could be useful to provide the last few lines of console output you see when it hangs.

You can also try to pinpoint this by stopping services manually. Since the cluster software would be our prime suspect, see what happens when you manually shut down these services (using the init scripts in /etc/init.d) while the machine is still up.

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Wouter Jagers
Honored Contributor

Re: shutdown is taking more than about 15 minutes

Sorry, I forgot to ask an important question.

Are you using HP ServiceGuard for linux or another clustering solution ?

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.

Re: shutdown is taking more than about 15 minutes

Its linux system.
I did not observe any console message flashed after shutdown issued. No console messages for 15 minutes. Do you suggest any degug I should add? Like
1. Adding "top" output to see , if any process is hung

2. Adding ps output and check out the status of processes after shutdown command is issued.

If you have any such specific debug command that I can insert in my perl code to find out which processes goes in uninterruptable sleep, that would be great help

Another question , why would a process goes for uninterruptable sleep ? KILL -9 also can not kill them ? though I have seen many a times, kill -9 too does not help in killing out the processes

Thanks

Re: shutdown is taking more than about 15 minutes

Hey...
I have one important question. Will shutdown hang , if any of the process does not exit , or hung. I thought "shutdown -r now" would anyway proceed for rebooting, if its not able to kill any of the process that are hung.

Pls clerify , if my concept is right or wrong. Describe how shutdown works in steps.
skt_skt
Honored Contributor

Re: shutdown is taking more than about 15 minutes

"I did not observe any console message flashed after shutdown issued"

r u really wathcing the console (not the terminal you issued the shutdown). Or try watching the /var/log/boot.log or /var/log/messages from another terminal parallay.
Wouter Jagers
Honored Contributor

Re: shutdown is taking more than about 15 minutes

Why would a process go into such an uninterruptable state: it mustn't get killed in the middle of an I/O operation, because that could cause corruption. However, a process is supposed to be in such a state for only very small amounts of time, unless if it's blocked for some reason. It will not ignore the kill -9, it will just only see it when it returns from the operation.

Thay only illustrates why a process would not react to a kill -9 immediately, though.

The man page for shutdown is quite short and describes the shutdown process well.

Are you working on the system console, or through telnet/ssh ?

We still need to know which cluster software you are using.

What's the output of these commands:

# whereis cmviewcl
# service cman status

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.

Re: shutdown is taking more than about 15 minutes

service cman status
-su: service: command not found

service tcp status
-su: service: command not found

whereis cmviewcl
cmviewcl:

I have console connection to platform. Its not telnet / ssh sessions. thats for sure.

I did not find boot.log file under /var/log

kern.log
Jul 7 02:29:05 Node1 kernel: EXT2-fs warning: maximal mount count reached, running e2fsck is recommended
Jul 7 02:45:18 Node1 kernel: nfsd: last server has exited
Jul 7 02:45:18 Node1 kernel: nfsd: unexporting all filesystems

Shutdown was issued around 02:29:46

There is no messages in syslog and kern.log around 15 mins after shutdown was invoked.

Thanks