rebooting UNIX bad for the system?

[Ken Penland_1]

I have always been told that rebooting a UNIX system is bad for the system, and shouldn't be done if you can help it... well, now they are talking about rebooting every morning automatically through cron...does anyone know of a white paper or any documentation discussing the effects of a reboot on the system? is it really bad or is that just a myth?

Thanks, Ken

[Sridhar Bhaskarla]

Hi Ken,

Rebooting a UNIX system is not bad for maintenance purposes like applying patches, pdc etc., Rebooting it because it has bad code on it (applications with memory leaks etc.,) is bad.

We realized that we had a system that was running for more than 1000 days (> 3 years!!). These systems are supposed to be robust when used properly. If you keep track of the changes and configuration of the system, rebooting shouldn't be a headache. There are times when sys admins forget to update the configuration files which may create problems. For ex., add a filesystem but forget to update /etc/fstab, add an IP manually but not in /etc/rc.config.d/netconf etc.,. The chances of these human errors are more on the systems that have been running since a long time.

-Sri

[Jean-Luc Oudart]

Well I guess the guy who told you that must refer to NT box or something like that.

You don't need to reboot your server unless you 've just rebuild a new kernel (to chnage parameters or after patches were installed).

I would not say it's bad, I'd say you're loosing your time and time is ...

Regards,
Jean-Luc

[A. Clay Stephenson]

Rebooting, of itself, is neither bad nor good. If the plan is to cold boot every day then I would say that that is bad because power-on (unless temperature limits are exceeded) is the most likely time for an electronic component to fail.

The first question to ask is "Why are we doing this?". To clear hung processees? To free shared memory? Because we have memory leaks? or just because? The answer to all of these is the same --- fix the real problem rather than dealing with the symptoms. Production UNIX boxes run for months and often years without a reboot. My boxes come down for only one reason : scheduled maintenance windows to apply upgrades and/or patches.

[G. Vrijhoeven]

Hi Ken,

I don not have a white paper but about how bad it is, but.... I does not "ever" solve a problem, it can be a work around at best!

It can be bad for database caches becouse they a flushed, uptime command output. Unix admin motivation.
What is the reason they want to setup the "cron of death".
Is is a application that does not give its memory back?

Gideon

[David Burgess]

Ken,

We reboot our servers twice every night as part of our daily processing. It doesn't cause an issue. I blame the application for not cleaning up after itself properly when it shuts down. Ideally it would tidy up shared memory and semaphores, but hey what can you do!? Assuming all your start up files are intact then you shouldn't have an issue.

We found out about a corrupt /stand one night when one server failed to reboot. Not the best way of testing, but it made people sit up and listen that old hardware does eventually wear out!

HTH

Regards,

Dave.

[Cheryl Griffin]

The unix standard measure of greatness, is how long the system can stay up. The goal is high availability.
# uptime

If you have a lot of defunct processes that require you to reboot. That's the sign of a badly coded application.

[Robert-Jan Goossens]

Hi,

No nonsense, a reboot does not affect the health of your server. About rebooting every morning, normaly a unix server does not need to be rebooted so often <--> is there a problem with an application ? consuming to mutch memory ?

Regards,
Robert-Jan

[Ken Penland_1]

The case with us is this...we have an application on the system that has a memory leak SOMEWHERE....each of our systems with this application on it only lasts about 3 or 4 days depending on how much it is used before vhand goes nuts and the system slows down to the point where just performing a shutdown takes an hour to do....currently to get around this, since none of the people that wrote the code can find a problem, is we get notified whenever the memory gets above 80% once it does, we reboot to clear it out....this is our workaround, but the higher ups dont like the "randomness" of the reboots, they want it scheduled before users come in..hence the daily reboots...

[Lorenzo Facello]

Hi,
I think that you can reboot your wks every day, but it can makes some trouble.
you surely stress the wks... it has been created to stay alive more then one day.
You can met some trouble with the configs, logs, managing files system ecc...

It's only a my opinion.
Hope this helps
L.

[Sridhar Bhaskarla]

Ken,

It's not that difficult to find memory leaks on the system. Run the following small script once in 6 hrs for couple of days. Put it in cron or on your schedulers if you have any.

DATE=$(date +%y%m%d%H%M)
UNIX95= ps -e -o "vsz pid args"|sort > /tmp/memleak.$DATE

Compare the first file with the last file that got generated. Identify the process whose 'vsz' has increased substantially. That process may be having memory leak and your developers should get it fixed.

-Sri

[Patrick Wallek]

It sounds like the folks that wrote this application need to go back and look at their code a little harder.

They obviously have a problem somewhere.

As said above, reboots generally just fix the symptom, not the real underlying problem.

[Robert-Jan Goossens]

Hi Ken,

There is only one answer, send the code back to the developers. rebooting a server every day because of memory leak is the limit. And before you ask, YES it happened to me too, I kept giving a negative advice to go to production until they fixed the problem.

Kind regards,
Robert-Jan

[Jean-Luc Oudart]

A Memory leak ?
Well reboot will never solve your problem. It will just hide it.
Ask the developer to find and correct the pb.

regards,
Jean-Luc

[Duncan Edmonstone]

Fix the problem, not the workaround - this will only come back and bite you again later...

Have your developers looked at 3rd party products for detecting memory leaks:

Rational Purify comes to mind, ( http://www-306.ibm.com/software/awdtools/purifyplus/ )

And of course I'm sure there are freeware products available that do similar things

HTH

Duncan

[A. Clay Stephenson]

In that case, the answer is to fix your programmers. I find that baseball bats adjust them quite well. The rule is quite simple, if you allocate the memory then it is your responsibility to free it. The leak may be occurring in a 3rd-Party library routine; there the answer is to notify the software vendor. If your programmers are having trouble finding the leak (and it really usually is easy to find -- especially if you have someone who did not write the code examine the code) the you should look into a product (e.g. Rational's Purify) that is made to track down these leaks. In this cause, daily reboots are "M I C K E Y M O U S E" solutions.

[Umapathy S]

Ken,

Rebooting is never going to solve your problem. You need to say to those people (developers) to fix the appication first. You can find out which one eats the memory via Sri's script.

A good profiler is needed to find the memory leaks. Try using Rational Purify (it is costly also). It will clearly say where and when the memory is consumed.

HTH,
Umapathy

[Ken Penland_1]

Dont have to tell me twice, I know it does not fix the problem...actually they said that they have monitored it and there isn't a memory leak at all, they suggest that the problem is the D 390's that the application is running on are too small for the application, and it is kind of like pouring water into a funnel, the water is coming in faster than it is going out, so eventually it overflows...which could be true...I dont know...their solution is to buy bigger machines...but until that happens....

Anyways, I understand that the problem should be fixed instead of a workaround, but from where I sit, all I can provide is the workaround... so back to my orig question, does anyone know pro's / con's of rebooting a system daily? I am gathering from the first few posts that it does not really hurt any thing, but if something IS going to break, during a reboot is the time it will happen...

[Geoff Wild · Accepted solution]

Rebooting shouldn't hurt a system.

It's not bad - even a good thing - if you have a cpu that is about to die - on a reboot - it may fail the hardware pre-test - and be deactivated - much better to happen in a "outage" window then to have the cpu fail during prime time.

I agree with others - the problem you have (memory leak) should be fixed...

If they say it's the hardware - ask for more ram - when it still "leaks" you'll see have fast management gets on your side.

We have an issue with a CICS app right now on an AIX box - with a memory leak - turns out that it is not the application, but the revision of AIX - waiting for a patch from IBM - we have to reboot box dailey....

Rgds...Geoff

[Sridhar Bhaskarla]

Ken,

//... so back to my orig question, does anyone know pro's / con's of rebooting a system daily? //


The probability of any hardware component to fail is *greater* with frequent reboots than when the system is continuously up and running. I don't believe in their pouring water into funnel theory but looks like there isn't much you yourself can do.

-Sri

[David Burgess]

Ken,

If rebooting is the only option I would go with it, but make sure you have a good Ignite recovery tape and full system backup just incase. If you do run into trouble at least you will have a way out.

Perhaps let the users know that it's not the best option, but if the code doesn't get fixed then it's the only option!

Regards,

Dave.

[Patrick Wallek]

As long as you are just rebooting (shutdown -ry 0) then everything SHOULD be OK. If you were to power the server off and back on every morning, that would be another story. Things most often fail when power is reapplied after haveing been turned off.

[Jeff Schussele]

Hi Ken,

I agree w/Clay. Nothing will impart a sense of urgency to these so-called developers better than the sight of a Louisville Slugger in your hands.
You don't have to use it, just routinely march by their cubes grumbling menacingly every time you have to go reboot the systems.

Cheers,
Jeff

[A. Clay Stephenson]

Now I know that you need to adjust the programmers. The "water pouring into a funnel" example is a very poor analogy. Any well-written application should block when it can't accept any more data. This is exactly analogous to handshaking between a host and a slow output device such as a printer. That "handskake" must be part of the protocol designed into the system on the front end. The fact that process memory is growing is a clear sign that your real problem is a memory leak. A faster box with more resources will merely mask the problem longer. Believe it or not, I make developers work on small, resource limited, slow (more or less) boxes. They hate it BUT they are then forced to develop good, tight, efficient code rather than relying upon the hardware to get them out of a jam. In most cases good software on slow boxes will outperform poorly written software on blazingly fast boxes. When you do move the well-written code to fast production boxes then it truly flies and there are no surprises.

[Jannik]

This is a classic one, teknician are not very mush into firmpolitics (I for one). What i would do is:
1. tell them the facts.
which is:
- reboot the system dail no problemo.
- the problem of memory leak will not dissapere it will grow, in a year we can reboot every hour (K-Class fulli loaded ½ an hour - uptime 12 hours a day).

- make a ignite tape if you want... really if the bad code...(BOFH