Regular PM - reboots?

[Reston Tech Support]

Since I'm one for one today, I thought I'd throw this out. Hopefully I'll be two for two!

The next question in the mix regards regular preventative mx (PM) and regular reboots. One theory is that some machines run into routine problems and we should reboot them on a regular basis. The thought was once a month, or once every two months at the least.

My theory goes along with what I have seen other threads in other areas - that should not be needed. Here's why:

1. You may introduce more problems than you seek to fix.
2. You may in fact not fix the problems you seek to fix, because the problems may be more sporadic, and thus not affected by the amount of uptime on a machine.
3. Regular rebooting is a synonym for "I don't know what the problem is, but a reboot always fixes it". Find the source of any problems, and fix them.
4. The problems of two or three machines are isolated, and should not set the policy for dozens of machines which run fine for hundreds of days at a time before a power outage makes them boot.
5. Regular PMs during times to upgrade patches, and the like should be sufficient.

Again, opinions are welcomed!

TIA,
FD

Any

[Thierry Poels_1]

Hi,

my opinion is to keep the server running as long as possible.
Even if you never "plan" reboots, you'll have to reboot a few times a year for system maintenance (patches etc).
But next to the server we "never" reboot, we also have a server which is rebooted weekly due to many hanging processes; result of a third-party package :(
So your choice should depend on your situation.

regards,
Thierry

[A. Clay Stephenson]

Hi:

I am very much in the camp of 'if it ain't broke' don't fix it. I have had machines up for years without reboot. My preferred method of operation is to only take the machine down for scheduled maintenance.

One frequent reason cited for the need of periodic reboots is to get rid of zombies. That is actually fixing the symptom rather than fixing the problem. In that case, zombies really do no harm and the cure is to beat the programmers and the careless SA's who caused the zombies in the first place over the head.

The other excuse for rebooting is to deal with memory leaks. Same answer - fix the code.


My 2 cents, Clay

[James R. Ferguson]

Hi:

My systems run 24x7 with downtime pre-announced and "contracted" to be 0200-0400 once a week.

I basically reboot as/if needed for patch application.

I don't subscribe to the idea of periodic reboots in the absence of any particular reason. That's simply *not* logical in my opinion. It's the same statement you make in your third point, and I couldn't agree more.

Obviously, if one suspects, or finds memory leaks, then one reboots. Obviously, if one finds hung ports or zombies, then one reboots. Interestingly, *all* of these cases are really symptoms of application problems that need to be fixed in and of themselves in the first place -- ergo.

I *do* believe that a reboot, when necessary, is an opportunity to review startup and shutdown scripts and logs (/etc/rc.log) and to generally "health-check" your server. You might find that some daemons that *were* running are not, and need to be added to startup scripts.

Regards!

...JRF...

[Sridhar Bhaskarla]

I don't reboot the servers regularly unless some maintenance like patches is required. However, it is better to regularly check the systems for jombies, ipcs etc. Some of the applications are not well designed that may result jombies and memory leaks. If your system is up and running for longtime and if you want to reboot it, then run "/sbin/init.d/clean_ex start" before rebooting the server. Otherwise it will take long time coming up.

-Sri

[harry d brown jr]

I personally abhor the thought of scheduled reboots, unless of course it's an NT, in which case we have it reboot daily (or more often).

I agree with everyone else, that if it ain't broke, don't fix it!

[Santosh Nair_1]

I thought NT takes care of rebooting itself on a regular basis ... or is that crashing on a regular basis :-)

That said, unless you have a real need to reboot the machine, you really don't gain much by having regularly scheduled reboots. We go for months before we have to reboot a machine (and that's only because we're a development house and those wacky developers wreak all sorts of havock on the machines).

Just my 2 cents.

-Santosh

[Stefan Farrelly]

Its a real tossup if youre in the 'dont touch it because its working fine' camp or if youre in the 'keep it updated' camp. Its a difficult choice. I find it depends on a few things;

1. Does your application need tight security ? and thus you will need to install regular HP security patches- most of which will require a reboot.

2. Do you add new hardware often ? If so you will need to keep uptodate with SCSI and LVM patches which will require a reboot (to prevent problems with the new hardware not working properly or causing crashes).

3. Are the applications memory hoggers ? Eg. Oracle. If memory continues to grow over time and you cant ID why (many other apps have the same problem) then you may need regular reboots just to cleanup.

4. Do musterious user problems ocurr which you cant easily explain or put a finger on ? regular reboots may be needed to see if they resolve the problem.

If none of these apply then you can simply leave your HP servers alone. They will stay up for years without a reboot and be super reliable. Ive been on several sites where this is the case, but Im finding them less frequent now as with the age of the internet and almost every server connected to it some way you need to keep some patching uptodate and with the advent of larger and larger servers the potential for more application conflicts and problems means regular reboots seem to become more and more the norm. We usually do ours quarterly and take this time to update patches and anything else. Our troublesome servers get rebooted more frequently.

[Scott_14]

Hello:

I have to agree, I am not a fan of re-booting just to boot, but some sites I have worked at require us to no matter what the reason they offer. I think a lot also depends on ones situation, sometimes our clients have more control.
Unfortuantly sometimes a re-boot, cause you to find some problems, in a start up script, that a programmer changed and didnt test or bother to tell you, and this is the only way it is caught, or if a system crashes due to a disk or something that you can identify and correct, and when you think you have it corrected, you find some other "bug" to aggravated your system on comming up, at a bad time.

[Brian Markus]

One thing that I am definately for is a routine time for maintance. If you set a time each week for scheduled down time, who say's you have to use it. You don't have to reboot. At least this way your in control of your maintance schedule. I hate begging for downtime. If you get the chance, Set something up on a regular basis. The client will generally not mind once a week. But if your just doing it as needed they always seem to stick you with crappy times. I've noticed I'm much more pro-active on the systems with routine scheduled down time. The systems that I have to request down time on get delayed. 1. request 2. they think about it. 3 they think their doing you a favor. 4. then they give you a junk time. YUCK..

Well, I got my 2 cents in.

Hope it helps..

Brian.

[Michael Tully]

Hi,

I'm also in the camp of "if no need don't"
except clearly for maintenance purposes.
The fix it approach is all well and good if
you have the source code to able to do it.
As far as beating up developers where's the
line and I'll stand in it. As already stated
there are two distinct problems in that is
zombies and of course memory leaks. On my
site we run applications that are dependant
on a number of factors that simply cannot
be fixed, we don't have the source code and the vendor cannot (or will not) reproduce problems so the solution is of course
weekly reboots. So I guess it comes down to

If necessary DO
If not necessary DON'T

My $0.03 cents worth
-Michael

[someone_4]

Hi
I have always felt good in the camp of 'if it ain't broke' don't fix it. But listening to the more seasoned admins here you cant go wrong with that train of thought.

Richard

[Bill Hassell]

SA's that are used to other flavors of Unix (ie, non-HP-UX) find that this is not only a common practice but in some cases, an un-fixable requirement. However, a properly patched and sysadmin'ed HP-UX box NEVER needs a reboot. Now you should reboot about 2-3 times a year to add patches, or possibly add some more RAM or disk, but otherwise, there is no need for PM reboots on HP-UX.

My primary HP-UX email server for the Response Center started in 1991 (an HP 9000/870) and ran 24x7 until it was replaced with a much faster D370 in 1999 (8 years). I added patches about once a year and RAM and processors about 3 times during the 8 years. When I shut it down, it had 1,800 users in the password file.

Now true, I ran scripts to notify me about swap space and disk space problems, killed runaway processes automatically, and made phone calls to (ab)users that consumed too many resources, and this machine did had no compilers or other development tools (which should always be true for production servers).

So if you fix the underlying problems (those pesky users -- 8-)) automatically, you won't have to reboot at all. Naturally, you must determine how to backup locked or time-critical files (ie, databases) in a 24x7 system.