Periodical reboot of the servers

[Shivkumar]

Dear Sirs,

I was told by someone that HPUX unix servers needs to be rebooted periodically. I also read that IBM AIX needs reboot periodically for better performance.

I am not sure why and when it should be rebooted. Is there any way to find out when we need to reboot the servers ?

Thanks,
Shiv

[Sudeesh]

It depends on the application running. If lot of memory leak is there I would suggest you reboot the machine periodically. Otherwise reboot during the quarterly pathching will also serve the purpose.


Sudeesh

[Devender Khatana · Accepted solution]

Hi Shiv,

This formula does not apply on HPUx or even on hardware supporting HPUx. I have around 30 odd HPUx PA-RISC servers and the uptime for these is listed here

*************************************
COMMAND : uptime
*************************************

10:23am up 77 days, 10:23, 6 users, load average: 0.07, 0.06, 0.05
10:23am up 49 days, 10:10, 0 users, load average: 0.11, 0.08, 0.04
10:23am up 9:26, 4 users, load average: 0.11, 0.11, 0.10
10:23am up 84 days, 23:52, 6 users, load average: 0.01, 0.01, 0.01
10:23am up 104 days, 15:35, 3 users, load average: 0.12, 0.06, 0.05
10:23am up 104 days, 15:38, 0 users, load average: 0.03, 0.04, 0.06
10:23am up 104 days, 15:38, 0 users, load average: 0.02, 0.01, 0.01
10:23am up 104 days, 15:38, 0 users, load average: 0.01, 0.02, 0.02
10:23am up 104 days, 15:38, 0 users, load average: 0.03, 0.04, 0.04
10:23am up 104 days, 15:38, 0 users, load average: 0.02, 0.03, 0.03
10:23am up 104 days, 15:38, 0 users, load average: 0.16, 0.08, 0.05
10:23am up 104 days, 15:38, 0 users, load average: 0.01, 0.01, 0.02
10:23am up 64 days, 11:23, 0 users, load average: 0.00, 0.01, 0.02
10:23am up 119 days, 10:23, 0 users, load average: 0.01, 0.03, 0.03
10:23am up 119 days, 10:16, 0 users, load average: 0.02, 0.01, 0.02
10:23am up 119 days, 10:14, 0 users, load average: 0.05, 0.03, 0.03
10:23am up 119 days, 10:14, 0 users, load average: 0.04, 0.05, 0.04
10:23am up 119 days, 10:11, 0 users, load average: 0.03, 0.04, 0.06
10:23am up 119 days, 10:10, 0 users, load average: 0.01, 0.02, 0.02
10:23am up 119 days, 10:09, 0 users, load average: 0.02, 0.03, 0.02
10:23am up 119 days, 10:09, 0 users, load average: 0.01, 0.02, 0.01
10:23am up 119 days, 10:11, 0 users, load average: 0.02, 0.02, 0.02
10:23am up 100 days, 19:01, 0 users, load average: 0.00, 0.00, 0.00
10:23am up 119 days, 9:56, 0 users, load average: 0.03, 0.03, 0.02

There has been only one or two instances during this time that application was shut that too was done manually for changes.

HTH,
Devender

[Mahesh Kumar Malik]

Hi Shiv

There is no set rule for hp-ux servers to be rebooted. It depends on application environment. If there are memory leaks and number of sleeping processes are on rise, reboot is recommended.

Regards
Mahesh

[Bill Hassell]

You were misinformed. Properly patched HP-UX is extremely stable and can run for years without a reboot. As mentioned, the problems are with bad application programs but even these can be fixed by restarting the apps, not rebooting. The ONLY reason to reboot is to perform kernel patches, perhaps 2-3 times each year.

[Steven E. Protter]

The only thing I ever rebooted my servers for was patches and security updates.

Nothing else.

No need to do it. Since I patched once a quarter, thats how often I booted them.

SEP

[Babu A]

Hi Shiv,

There is no hard rule for rebooting. The info was not true. I have servers running for more than 400 days without a reboot. All we need to do is keep monitoring it.

Example of server which is being running for the past 401 days.

4:38pm up 401 days, 7:49, 1 user, load average: 0.00, 0.01, 0.01

Regards,

Babu

[Shivkumar]

Are more number of sleeping processes harmful ?

So is reboot only option left to resolve this ?

Thanks,
Shiv

[Babu A]

Hi Shiv,

You may kill the sleeping processes, but may not be able to kill all the Zombie processes. To kill all zombie process we need to reboot the server.

Below is the explanation of Zombie process.

Zombie processes are created when the parent process either does not
wait on the child process, or does not ignore the child's termination
by setting the 'SIGCLD, SIG_IGN' signal within their program. Zombies
remain until the parent exits. Although zombies do not use CPU time,
they do occupy a slot in the process table.

Zombie processes are also created when a child process completes, and
sends a termination to the parent, before the parent receives the
'SIG_IGN, SIGCLD' signal to ignore the termination.

These processes cannot be killed, as they are already dead. If the
parent is killed before it exits properly, and cleans up any zombie
child processes, only a reboot will clear the zombies from the system

Regards,

Babu

[A. Clay Stephenson]

Zombies consume no resources (other than a slot in the process table) so as long as you are well below nproc there is absolutely no harm in them. Rebooting to clear zombies is really a Mickey Mouse solution; the real answer is to find out why you have then in the first place and thus fix the cause rather than treating the symptoms.

HP-UX boxes are typically up for months without reboot and the only time they should be rebooted is for periodic patchset installation.

[Shivkumar]

Under what circumstances parent processes gets killed prior to children processes ?

Some folks told me that on some very higher end hpux unix reboot may not be possible even after patching. Are these higher end hpux like superdome has different architecture than normal hpux servers ?

Thanks,
Shiv

[Shivkumar]

A Clay Stephenson;

You mentioned that "Zombies consume no resources (other than a slot in the process table) so as long as you are well below nproc there is absolutely no harm in them."

Could you please eloborate on relationship between nproc and zombies ?

Thanks and regards,
Shiv

[Bill Hassell]

You asked:

"Some folks told me that on some very higher end hpux unix reboot may not be possible even after patching. Are these higher end hpux like superdome has different architecture than normal hpux servers ?"

Your information is incorrect. The largest HP 9000 computers run the same HP-UX code that a small workstation runs. If a reboot is not possible, then the multi-million dollar machine would be useless. Perhaps what was incorectly stated is that high end machines may not need to be rebooted because some patches do not require a reboot. The same is true for a small HP-UX workstation. Virtually all kernel and driver patches will require a reboot. There are features in high end machines that allow independent virtual computers to be created.

"Could you please eloborate on relationship between nproc and zombies ?"

nproc sets the size of the process table. A zombie was a program that was improperly terminated so it occupies one slot in the process table. If you have some very bad programs that create lots of zombies, it's possible that after a few weeks, the number of zombies will be significant. But as Clay said, you fix the broken code, not reboot the server to correct extraneous zombie programs.

[Yogeeraj_1]

hi shiv,

As expert have said above, there is indeed no good reasons for rebooting a server which is performing well.

if you suspect that there is performance problem, better check it at the application level.

In most cases, The applications can be restarted without a server restart!

We have a system which has been running about 10 years! and we have restarted it only in some cases of extended power-cut, preventive maintenace and ... i don't remember...

The last figures are:
K250: home/yd>uptime
8:33am up 201 days, 1:25, 100 users, load average: 0.00, 0.01, 0.01


hope this helps too!
regards
yogeeraj

[Rajesh SB]

Hi Shiv,

As you are interested to know about relation between zombie and nproc.
Here I would like to share on realtime example facing here. One of my HPUX 11.00 running with Samba server. When keep it running long time, generating many "smbd" zombie process. These process consuming system parameter "nproc". One day nproc parameter overflow and system wouldn't respond. Only way is to reboot.

Example here:
See the "top" command output: Obeserve for 340 zombies generated by smbd.

System: shimsha Mon Sep 5 10:10:45 2005
Load averages: 0.03, 0.04, 0.04
458 processes: 114 sleeping, 4 running, 340 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.07 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
1 0.00 0.2% 0.0% 0.8% 99.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 0.0% 0.0% 0.4% 99.6% 0.0% 0.0% 0.0% 0.0%

Memory: 110344K (60352K) real, 99812K (58844K) virtual, 184816K free Page# 1/42


CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 27945 root 149 20 0K 0K zomb 0:01 1.98 1.98 smbd
0 ? 4708 root 149 20 0K 0K zomb 0:00 0.44 0.44 smbd


# sar -v 2 3

HP-UX shimsha B.11.00 A 9000/800 09/05/05

10:12:09 text-sz ov proc-sz ov inod-sz ov file-sz ov
10:12:11 N/A N/A 459/4116 0 0/5700 0 634/9785 0

# ps -ef|grep defunc|wc -l
341

I am living with this test setup by increasing the nproc parameters to high value and periodically restarting samba service or server zombies are clearing.



Thanks & Regards,
RAjesh

[kenny chia]

I have a few systems that suffer from memory leaks.
But it seems that restarting the applications will not release the memory, only a reboot will

[Cem Tugrul]

Shiv,
Thers is no rule for all unix systems to be periodically rebooted.The main point is to monitoring your systems and detect zombie processes if you have many zombies then if possible to kill parent processes if not
of course rebooting is the only way...
Good Luck,

[Alessandro Pilati]

Shiv,
there is a way to release memory resources held by dead processes or idle clients.

If semaphores or shared memory, use ipcs to view and ipcrm to remove.

If network resources, use
ndd -get /dev/tcp tcp_status
to view the network resources status

and
ndd -set /dev/tcp tcp_discon
to release the resource ( is a value of the first column of the "ndd -get ..." command )


Have a look also to this past forum :
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=204695
Zombie process many times are not a problem.

Anyway, I agree with the idea that reboot must be the last solution.

Best regards,
Alex