How often do you reboot....and why?

[Ted Flanders]

I run a single HP9000 K220 w/HPUX 10.20. I reboot my box bi-weekly. I have heard to sides to doing this. HP says absolutely do not do this, your hardware might fail on boot up. Frankly, thats fine with me, I pay 16K per year for a service contract and all parts and labor is covered. We have used this contract once in the last 2 years and got charged an extra $1,800 for a "special" charge.
My software people tell me to reboot weekly. The other sys admin here tells me bi-weekly and my boss tells me to "find out".
What does everyone here do? And why?
Thanks.

[Ajitkumar Rane]

We do not reboot the servers unless there is a reason for doing it, like patches, upgrade etc etc, so the servers do not get rebooted for months together.
I remember reading in the sys admin manuals, it is recomended that you should unmount your filesystems and run file system checks to keep your filesystem intact and know of any errors before they suddenly shoot up at production hours.
Sometime back I was in a shop where the server were rebooted monthly, as they had a application which caused memory leaks.
Now there is a difference in robooting the server and shutdown with power cycle.Power cycle can cause hardware failures.

[Rita C Workman]

I do not reboot that often. Whenever patches (or new bundles) are installed, and those quarterly bundles I only do that maybe twice a year. I reboot whenever there is a serious problem with the system and the only resolution is to try a reboot. I used to try and just do a reboot maybe once a month. But even then folks complained about the downtime. So I keep reboots to a scheduled minimum and only when absolutely no other alternative.

/rcw

[Vikas Khator]

Hi,

I believe in rebooting servers once a quarter and usually coincide with patch bundle install.

Only other reason I would reboot more often is to fix memory leaks . ( a very common problem in devlopment environment) . However right thing is to get the devlopers to fix the memory leak.

[Patrick Wallek]

I have one weekend per month where I get the servers to myself with no users. During this time I reboot the machines, run fsck on all filesystem (while in single user mode), make sure I have a good backup of the system with the database down, resize the database files (this is a Unidata database), install patches, and whatever else needs to be done.

This schedule has been working for about 3 years and I generally don't get too many complaints from users.

[Patrick Wallek]

Just had one of the other responses sink in.....

I do agree with one of the previous responses that said rebooting and powering off are distinctly different. I don't turn power off to the machines unless I absolutely have too. You do run the risk of having hardware failures then.

[Rick Garland]

You are right, different sides to the issue and I find it depends on your environment. If you do have troubles, like memory leaks or something else, you will need to reboot more often. The software dev folks like to see reboots more often than what is typically necessary.

If you have a system in the prod environment, chances are you will need to reboot less frequently.

And if doing patches, some patches require reboot after installing - can't get around this.

You may be able to run a system for a year or more without rebooting or you may decide to get the SW dev folks to be happy by rebooting more often.

The systems are designed to run without having to reboot and for some apps the systems will do just that. But developers have a habit of introducing issues in their code that only a reboot will clear up. Then get the developers to fix their code.

[Bill Hassell]

I never reboot except to apply patches or to add new hardware. I ran a 9000/870 as an email server from 1991 to 1999. I had one unscheduled reboot to fix a runaway NFS problem that was later patched. For the first 4-5 years, it ran 24x7 for at least 400-500 days.

Now as mentioned, poorly written programs can use up RAM but that's easily fixed with the kill command. Same with runaway programs that consume CPU and/or LAN bandwidth. There is no need to reboot HP-UX as long as you keep up on patches.

[Philip Chan_1]

Yes, rebooting the server could cause damage to the hardware. This is hard to believe but really happened to my site a few weeks ago. We cold booted 5 HP servers and 2 of them resulted in hardware failure (CPU in one server and disk drive in another). I think this is due to voltage fluctuation during cold boot (warm boot should be fine).

Having said that reboot could damage hardware, I still think that rebooting servers periodically are good practice simply because:

1. every restart will force the fsck to run against your disk drive.

2. will clear any zombie processes and ill behaved processes that holding up memory.

3. will ensure your server be able to start after any system change.

4. it doesn't do any harm providing that
- you got a hardware support contract with HP
- your disk drives are mirrored or RAID'ed (or else you'll have to restore data from tape)

Regarding the time, I guess once every month should be fine.

Rgds,
Philip

[Ted Flanders]

I warm boot, and do not cycle the power. I mainly do this because my uniplex email starts misbehaving. It doesnt notify users when they have mail, etc.

[Jitendra_1]

I used to work in a development shop on my previous job. There were number of BEA developers and we used to reboot the boxes every sunday . This used to clear up a lot of memory otherwise held by some stray user programs .
However occasional reboots for patches and maintenance also were normal.

[Gram Ludlow]

We reboot our servers only when we install patches that specifically require reboots. Hardware failures can and will happen, especially with hard drives. Once a place I worked for lost power and cold-booted a RAID5 disk array attached to a server, and two drives died. We had to restore over 100gig of data from backups.

If your software recommends "rebooting" every week, then I suggest you run a cron job to get the pid, kill it and re-start you software. This could fix memory leaks without killing anything else on the machine, and would be automated enough not to be a hassle. I would leave the servers on as long as possible, however.

[Victor BERRIDGE]

Hi Ted,
I had a HP 827s running SAS that went on reboot every friday evening supported by HP (...) it worked like that during 7 years without any trouble.
I have a an AIX machine that is rebooted every night since 8 months, I decided this because I couldnt deal with the machine rebooting any time in the day, and am still looking for the bug (This machine controles all our working time and so has about 70 badging modems behind...) so its a workaround...
And the others, it depends if there has been some patchs install that require, or because I have to reboot because some machines are on a same disk subsystem and Ive just done some maintenace or disk upgrade...
so time will vary from once every 3 months to once a year
I had a 822S that wasnt rebooted for 950 days (until HP came and took the machine away...)

All the best
Victor

[Mark Vollmers]

We normally do not reboot, and did not do so for many months. Recently, we have had to many times, due to a corrupt file (or files?) that have caused the server to crash repeatedly. I was told by our old admin. that the server is stable, and did not need to be rebooted. It might not be a bad idea to do it every now and then(I wonder if out files might not have gotten screwed up if I had rebooted every few months to clear it out).

[John Bolene]

We reboot the application servers about once a month because Hyperhelp has a memory leak that has never been fixed. The other servers are only rebooted when patches are applied, and the DNS server has been up almost 2 years (it was taken down at that time when we were adding more power islands and took that opportunity to power the building down and replace the main power switch with a larger one.

[Karthik_2]

we normally dont reboot production servers until and unless there is a need like patch install or some hardware failure .No proactive reboot is done.But we do reboot (warm boot ) our developement servers on a regular basis due to the memory leaks and frequent patch installs and application installs.

In few situations I have seens the hardware especially hard disks fail after a cold reboot.
It is also a practice that we reboot systems before handing them over to other sys admins or other teams /departments as a part of handover.

[Subhas R. Reddy]

I work in a large datacenter which runs 24/7/52. We have a Mainframe, a bunch of SUN and HP midranges and a bunch of Intel based servers (NT and NetWare). The only boxes that get rebooted are NT boxes which are rebooted on a regular schedule either automatically or manually. SUN and HP systems never get rebooted except for hardware and /or software upgrades. Mainframe folks do IPL every weekend. I have not seen any problems on SUN and HP systems that can be attributed to systems being up for a long time. Not even on NetWare servers. The ones that need rebooting on a regular basis are NT Boxes only and none else.

[Peggy Fong]

I worked in a large scale shop with 300+ 9000 series (D,N,T500,T600,H,I,J,...) Like many of the responses we did not reboot the systems since we were 7/24 on most applications. Some were automatically rebooted monthly through cron and some weekly - these were application users. The only reboots I supported were for system changes (software, firmware, hardware, etc). And I found that if you had a primary server or servers with backup servers waiting to take over it would be wise to use the backup servers occasionally. In the old days we had Switchover/UX which required a reboot to get to the backup server. (I actually had a primary fail - the backup rebooted as the primary and it had a hardware problem as well - so we were down for 4+hours). With ServiceGuard we really don't reboot we just move the packages from host to host to make sure all hardware is working with the applications. Funny, but all the reboot requests came from our applications, never sysadmin or operations.
Peggy

[Steven Sim Kok Leong]

Hi,

In my environment, I usually reboot to:
1) remove hung processes ie. kill -KILL has no effect
2) clean up the harmless processes
3) clear memory leaks

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[David Mabo_1]

I have a number of servers running HP-UX, plus 2 SCO 5.05 and 5.04, Linix and a number of NT servers. We run databases, web servers, mail servers and development machines. On the Unix boxes, we never reboot unless:

1. Change Hardware.
2. Patch or upgrade OS.
3. Something will not kill, or we have a problem.

This means 1 or 2 times a year, sometimes quarterly. I once looked at a D200 and found it had been up over a year w/o a reboot.

Now Nt is another matter, we seem to reboot some server often, and most monthly.

I also support 2 24x7 customers, and they only reboot when we patch.

[Manuel Plaza]

Hi,

I only reboot when a new software that require it is installed or when there is a "big problem" in the system that only is resolved with a reboot.

Regards,
Manuel

[Dieter Degrendele_1]

Hi,

The unix systems I manage are up for some weeks. This is due to our patch management. Although I have an old server here with no critical info that is up for three years without problems.

[Shannon Petry]

I agree with Mr Hassel of course that most systems do not need rebooting. My HP J2210 and 715/125 mail servers, J2210 FTP Servers, C360 Proxy Servers, J2210 WWW servers and K460 oracle servers has been running for 9 months now with no reboot. Sendmail, IMAPd, POPd, FTPd, HTTPd, Squid and most other applications are upgraded without rebooting.
>Service stop
>Service start

I have had 1 of my K Oracle servers panic and boot itself after the developers started playing the "Just How much memory will this thing allocate?" game. For the most part though even oracle can be stopped and started on the fly.

When you get into shakey ground are CAD/CAM FEA and CAE applications. Most of those I have reboot monthly via cron. This is because those applications tend to be buggy, have memory leaks, etc..but they have to run those apps to support the Big3's CAD Design.

The best thing to do is perform your normal maintenance. I.E.
1. At least weekly if not daily clean /tmp and /var/tmp.
2. use glance, vmstat and sar to monitor system performance. Look at your daily memory useage history, disk history, etc.....
3. Look at least daily through /var/adm/syslog/syslog.log for errors. This is where you will see SCSI disk errors as well as other key information.
4. Weekly stop and start the system logger to keep logs at a reasonable size (unless your system is very slow!)

You will probably find more custom things to do based on what that system is doing.

Regards,
Shannon