Planned UX reboots? Timing between them?

[Peter A. Berger Jr.]

I have a ? concerning planned reboots of a HPUX 11.11 (RP7410 machine; 10GB RAM; Oracle 8; BaaN v4c4 ERP). My new boss seems to think that all machines (Wintel & UX) should run 24x7x365 and NEVER ever require a reboot. In the practical world we SysAdmins know that sometimes a reboot for the heck of it improves performance and solves goofy problems caused by application memory leaks, etc. I was wondering if there is any documentation out there, or best practices to determine when to schedule a reboot of UX boxes. Our boxes have been running for 4months w/ zero problems. We're just now experiencing some slowness and are planning a simple reboot. I know that this 4months is much longer than most of our Wintel boxes and we're proud of 4months. Any help/hints/best practices/white papers would be great to show that yes UX machines do need to be rebooted sometimes and that other people in the world cannot run their machines forever. Thanks in advance.

[Simon Hargrave · Accepted solution]

Define a patching strategy for your servers, eg every 3 months when the patch bundles come out. You then have a "required" shutdown window, for security and stability maintenance purposes, in which you can shut down ;)

[Sanjay_6]

I don't think you are going to find any best practice doc saying you should reboot your unix boxes every x number of days / weeks /months.

We hardly reboot any box, unless we have a scheduled patch / app upgrade which needs a reboot. We have servers which haven't been rebooted in two years. you still have 20 more months to go before you reach that state. 4 months not booted / no problem.

Hope this helps.

Regds

[Jeff Schussele]

Hi Peter,

The only planned reboots we do are for kernel patch installs that require such. Typically these are just twice a year when we apply the "standard" bundles. There might be an occasion when a security alert will demand the install of a patch that requires a reboot - but not very often.
If you're experiencing memory leaks then the normal procedure would be to require the development team - or vendor - to fix them.
I find a good maple Louisville Slugger to be quite an incentive for them. But I'm old-fashioned that way because the aluminium one "ring" too much. ;~))

Rgds,
Jeff

[Pete Randall]

Quarterly when we do patch updates. That's all we do, though we used to think it was necessary to reboot monthly - it turns out that isn't the case at all.


Pete

[John Poff]

Hi,

I haven't seen any documentation about suggested reboots for HP-UX machines but the topic has been raised here before.

I agree with your boss. My philosophy is that a HP-UX/Unix/Linux system should just run right and run all the time. If you are experiencing problems with memory leaks, application bugs/errors, or O/S bugs, you should track them down and fix them. Rebooting the system just to clear up some vague slowness doesn't solve anything, and some day the problem might get worse and then you'll really have to stop and solve it.

Our Windows admins here reboot their systems every month, but that is their choice and they don't know what it fixes or why it helps. Rebooting systems 'just because' falls under the topic of Voodoo System Administration, and I'm no good at Voodoo. :)

JP

[Geoff Wild]

I agree with Simon and Sanjay.

I never reboot for the "heck" of it.

I only reboot because of a kernel parameter change (that's not dynamic) or when applying patches that require a reboot.

I follow a conservative patching strategy - only patch every 6-9 months.

Instead of reboot, just restart app....

I currently have a box that's been up for over 500 days - which is good and bad - good that's it's been up - bad on my part cause I havn't patched it in 500 days...

Rgds...Geoff

[Carlo Corthouts]

Hi,

Planned reboots? Unix boxes when administred right don't need to be cycled every so many time.

But a good strategy would be to setup a few maintenance windows during a year for general interventions. I would always try to go for about 4 to 5 maintenance windows where we can have planned activities on those boxes.

These would include :

- Patching
- Installing software upgrades
- reconfiguring applications
- taking offline backups (when running db's)
- Tuning your kernel
- General hardware check/some cleaning
(if necessary, not everybody puts their boxes in sofisticated datacenters)
- restore tests.
- Powercycle your box
(If something is wrong with your hardware
it might show up after powercycling)

But if you don't need your maintenance window, don't use it to just reboot. But do
include these other activities.

If you can, keep the box up and running as long as you can, else you never will see if you have other problems with unix or applications like memory leaks.

Regards,

Carlo

[Jeff_Traigle]

I have to agree with your boss too... at least on the *nix side of the shop. Planned power outages and system patches are the only time I reboot systems.

I had a request come through from current client's customer to schedule a cron to reboot regularly because "some support person on the help desk said it needed to be done" per some supposed ancient HP suggestion. Despite my statements of experience that this wasn't necessary, they felt better doing it so I set it up for them. Doesn't hurt anything... not a battle worth fighting to make their customer happy... so forth and so on.

On the other hand, Windows servers need a lot of hand-holding and babysitting. There are a lot more poorly written applications (as well as the OS not being so great in the first place) and rebooting regularly does help keep them somewhat better behaved. With all of the patches MS sends out to plug security holes, you're probably rebooting monthly as a matter of maintenance anyway... if not more often. (And if you aren't applying them, start or you'll get bitten in the behind at some point.)

Basically, if the boss says don't do it "just because", don't sweat it... enjoy one less task that you need to worry about and get on with some productive work. :) Now, if he's saying don't even install patches because they require a reboot, he needs to have his head examined.

[Carlo Corthouts]

Agreed, a good patching strategy is always needed for any unix flavor.

But it is possible to keep a machine running for some years without patching once you are up to a good patchlevel.

But do keep in mind the next time some hardware is getting added like another storagebox or a fiber needs replacement, the first thing that most likely will need to be done is patching.

[Rita C Workman]

There is no rule. Many folks have had boxes up for years without ever rebooting. (In other words, no patching that would have forced a reboot done either).

You reboot when you patch a system and the patch forces a reboot. How often you patch, and when, is up to you.

In house here, got it approved to that the UNIX Admin 'owns' the box 1 Saturday a month to do with it whatever needs done...and they may include rebooting for whatever reason. -or not- That is our scheduled downtime. And since we run a failover shop...we can move packages around, so it is really of little to no impact to production during this time. The only full downtime comes when I have to reconfigure clusters.

Regards,
Rita

[Bill Hassell]

My experience is that HP-UX (when kept patched) never needs to be rebooted. I ran a mail server for 2,000 users that was rebooted once in 5 years (not related to kernel patching). That reboot was scheduled after we discovered that a power supply was dead (possibly for months). This was on an 870/400 which had no diagnostics available for power supplies (unlike modern HP-UX servers). We found it by accident when taking off the skins to marvel at the massively large memory cards.

As mentioned, applications can get very trashy with memory leaks and orphaned shared memory segments, and may need to be restarted but reboots are seldom necessary on HP-UX.

On the Wintel side, reboots are built into the opsystem. You can tell when it is time to reboot because the screen turns blue. To be fair, a properly patched *and* restricted Wintel server running Win2k or WinXP can run for months. Just don't allow IE to run on those machines with external access.

[Steven E. Protter]

In todays security environment to NEVER reboot a server is not practical.

There is a constant stream of critical security updates that need to be installed occaisionally to maintain security and sytstem supportability.

That being said, I've never HAD to reboot a production HP-9000 server just for the sake of it.

If You really want 365 x 24 x 7 reliability, I would recommend some shared disk, a second server and ServiceGuard.

Then with stable systems, you can actually update and boot the two nodes without the users being effected at all.

Personally, I've run systems up to 95 days without booting. The only reason those "frozen" systems were not able to run longer was someone needed to change building power.

I've seen posts of systems running reliably for two or three years. That in my opinion is a bad but possible practice.

SEP

[Peter A. Berger Jr.]

THANKS to all of you who submitted your valuable input! Rock On!

[Peter A. Berger Jr.]

HPUX rules