General
cancel
Showing results for 
Search instead for 
Did you mean: 

Reboot Schedule for HP-UX 11iv3 system.

SOLVED
Go to solution

Reboot Schedule for HP-UX 11iv3 system.

Hello everyone. I have a somewhat objective question and I wanted to see if I could get some good opinions.

We recently upgraded to 11.31 from an 11.0 system. We had a reboot scheduled every day except Sunday at 1:45am.

We're looking into expanding business from an 8 to 5 schedule in some areas to 24 hours and we've been looking anew at the rebooting schedule.

Is it still a good idea to reboot our 11iv3 system on a daily basis? Is it possible to extend that to rebooting once a week?


Thank you.
19 REPLIES
Patrick Wallek
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

>>We had a reboot scheduled every day except Sunday at 1:45am.

Why??? Was there a problem?

>>Is it still a good idea to reboot our 11iv3 system on a daily basis?

Was it ever a good idea to reboot the systems daily? In my opinion, and experience, a well tuned, well behaved, well running system can run for YEARS without being rebooted. I have lots of systems that I reboot twice yearly when I patch them and that is all.

>>Is it possible to extend that to rebooting once a week?

Absolutely! But you REALLY REALLY REALLY need to revisit the reason behind rebooting these systems so often. These are NOT Windows servers! They don't have to be rebooted like a Windows server does.

If you are running some application that causes you to have to reboot due to its inefficiencies, talk with the application vendor and get it fixed.
Steven Schweda
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

> [...] Is it still a good idea to reboot our
> 11iv3 system on a daily basis? [...]

Was it _ever_ a good idea? Why???

Re: Reboot Schedule for HP-UX 11iv3 system.

Thank you for your replies so far. Let me explain a little.

I joined this IT team a little over two years ago. It was mainly our CIO and one or two others who were maintaining the system.

The hardware we were previously on was 17 years old, and these reboots had been put into place at that time on the HP-UX 9 system.

In that time, it was never a necessity to look into the reboot schedule, plus the manpower wasn't available to "fix what wasn't broken", so to speak.

Now our IT department is larger and we can look at these type of things. So one of my duties since I came on is to bring our HP-UX system management more in line with the times and how other admins are operating.

I gather the general consensus is that we can do much longer reboots. Could I also ask what logfiles you find it important to trim on these longer reboot schedules (for those that don't go years) and what sort of scripts you might have in place?

Basically what we want to determine is what maintenance/admin sort of tasks that we're not doing now that we should be.

Thank you again for your assistance.

Re: Reboot Schedule for HP-UX 11iv3 system.

I apologize. We are now off our much older system onto an rx2660 system.
Bill Hassell
Honored Contributor
Solution

Re: Reboot Schedule for HP-UX 11iv3 system.

As Patrick mentioned, you don't need to reboot HP-UX for years -- as long as your patches are up to date. But many kernel and network patches will require rebooting your server, so your reboot will be handled by your semi-annual or quarterly patch schedule. (you are keeping up to date on patches, correct?). A cold install of 11.31 needs several dozen patches.


Bill Hassell, sysadmin

Re: Reboot Schedule for HP-UX 11iv3 system.

Yep, those who were maintaining the system always kept up on patches and updates for the 11.0 system.

As my boss explained it, the system they started on was incredibly small in size, much much smaller than we have now. So a nightly reboot made more sense. Then it was just a habit that was kept in place since the possibilty of extension had not been researched.

Does someone know of a reference somewhere of good tips on recommended routine tasks? I want to make sure that we're trimming the necessary logfiles, doing the recommended hardware checks and other things of that nature.

I'm not an expert admin by any means. I've taken the two classes and am preparing for my certification exam. So any advice from veterans of this sort of thing would be most appreciated.

Re: Reboot Schedule for HP-UX 11iv3 system.

Sorry for the double post again. We are also completely up-to-date on 11.31 patches.
Steven Schweda
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

> [...] So a nightly reboot made more sense.
> [...]

Why?
Bob E Campbell
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

You should not need to reboot the system to trim the log files, and on your new system you should have plenty of space under /var. If you find you are still cramped we can help you with that.

If you have logs that are growing at a rapid rate, take a look inside. If you have not been keeping the data, you might not need the same level of information.

Based on your past experience why not schedule a monthly patch window. You do not *have* to install anything, you just give yourself the opportunity. If you work with DRD (http://www.hp.com/go/DRD) it will be similar to a sluggish reboot.
Pete Randall
Outstanding Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

Tip: Bob's DRD link won't work because of the parentheses. If you want to put parentheses around a link, use spaces to separate the two. In this case, just click the link then delete out the trailing parenthesis in the resulting 404 URL.


Pete

Pete

Re: Reboot Schedule for HP-UX 11iv3 system.

Steven, I do not have all the details on the why. My boss has explained that it was 17 years ago when systems were much smaller, did not have as efficient of internal hardware monitoring as we have now and did not have as efficient of operating systems or application programs. Their IT professional at that time recommended nightly reboots and gave sufficient reasoning for them that they felt it was the best choice.

And yes, Bob, our /var has plenty of space now. We have not had any issues relating to running out of room on any of our filesystems. We've only been on our new system since April at this time.

Re: Reboot Schedule for HP-UX 11iv3 system.

And Bob, one of the things I was looking for were suggestions of log files to keep an eye on. I can write shell scripts to do any necessary trimming. I just need some guidelines on what to look at there and just regular "cleanup" that is normal for sys admins to schedule or perform.
Bob E Campbell
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

There are many answers to that and many have already been discussed in detail by folks that know more in other threads. My short answer is:

1. /var/adm/syslog/*log
2. Make sure root email is seen by someone
3. Depends on what you are using the system for

Spend some time looking for "HP-UX log files" in google and I bet several threads in these forums will show up.
James R. Ferguson
Acclaimed Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

Hi Candace:

My bet is that your boss worked on systems with poorly coded, poorly debugged application code that leaked memory.

Once a process allocates memory (as with the 'malloc()' system call), memory is added to the process's "heap". A subsequent 'free()' deallocates the memory *but* only returns it to the heap, not to the operating system. This is designed to be a performance tweak when a subsequent 'malloc()' wants memory it has the chance to take it from the heap.

Should a program have been poorly written such that it fails to 'free()' memory from its heap, eventually the program will terminate when it can't get any more memory or cause *other* programs to swap as the available pool of *system* memory reaches critically low levels. Programs that misbehave by failing to appropriately allocate and deallocate memory are said to have "memory leaks".

Since a leaking program returns all of the memory it acquisitioned to the operating system at large when the program terminates, a short-running program isn't usually a problem. Long-lived processes, however, may slowly consume more an more memory and as your boss may have found, can be "fixed" by restarting them --- a reboot being one blind choice.

Regards!

...JRF...
Steven Schweda
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

> [...] a reboot being one blind choice.

Blind, because even in that case, restarting
the resource-hungry application should be all
that was needed, not restarting the whole
system.

> [...] Their IT professional at that time
> recommended nightly reboots and gave
> sufficient reasoning for them that they
> felt it was the best choice.

Next time, get it in writing.

> Based on your past experience why not
> schedule a monthly patch window. [...]

Or try running for two days. Then, if that
seems to work, try it for four days. Then
eight. And so on. Quit extending the
interval when you can't stand the strain, or
when you find some actual reason for doing a
reboot.

Re: Reboot Schedule for HP-UX 11iv3 system.

The recommendation I'm going to make to my boss is that we schedule it for once a week and monitor our resource usage for a week or two.

If there are no adverse affects, then we'll try a month. Matching it to the patch schedule is a good idea and if we don't experience any adverse affects I think that's a good option for us.

Thanks for all your input.
doug hosking
Esteemed Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

It's hard to make absolute statements about what's best without knowing much more about your environment, but I'll make a few general comments.

I generally agree with the idea of extending the reboot frequency to far longer than the daily schedule you have now. HP-UX itself is easily capable of going many months or years between reboots. Although occasionally a defect like a memory leak will slip through the testing processes, those get serious attention by HP (and most major application vendors) when they are noticed, so if you're keeping current with patches, that should rarely be a problem.

I worked for many years at a company that was a major user of HP-UX systems and was fortunate enough to have very stable power. We had a number of systems that had not been rebooted in over 3 years and were still happily running (though they were obviously not current on all patches).

I can only think of a few reasons that you might benefit from a more frequent reboot schedule.

1) I have heard of cases where some HP-UX systems were so reliable that nobody on site remembered the proper processes for rebooting. This is more likely to be a problem as sites migrate to different generations of hardware, such as only having a few PA-RISC systems left after a migration to Integrity serves, or when there is staff turnover. Going through the process manually frequently enough to stay comfortable with it seems like good practice, though that doesn't mean it has to be done on all production machines.

2) I have run across cases where an admin did something stupid (like deleting /stand/vmunix or some other critical file) and it wasn't noticed for so long that it became a problem later. Problems of this type tend to be noticed during reboots. Learning about them before the old backups get overwritten can sometimes make the problems easier to deal with.

3) There is marginal benefit in running fsck and similar integrity checkers occasionally, to help keep small problems from becoming big ones. Given the reliability of modern hardware and software, "extra" fsck runs are probably of less value than in years past.

Many other factors, such as security needs, redundancy, tolerance for down time, maturity of key applications, stability of power, availability of staff, etc. can all influence the best practices for any given environment. Generally speaking, I think you could likely easily get away with reducing your daily reboot schedule for HP-UX to a monthly or quarterly schedule, depending on your patching needs, if you can't identify specific reasons to do more frequent reboots.
Don't let (mis)behavior of other operating systems cloud your judgment about HP-UX.

Viktor Balogh
Honored Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

Hi,

We have some systems with a year of uptime, (we patch regularly, every year). At this large uptime you want to watch out one thing: the majority of the hardware problems come at the next reboot after a long enough uptime. (With "long enough uptime" I mean more than 500 days, even around 1000 or more days.)

I know about a bug in the scsi driver of 11.11, where around ~480 days an overflow occurs and the syslog gets fulled with error messages. So maybe some bugs are uptime related, but I think that the latest 11.31 should run stable for years.
****
Unix operates with beer.
doug hosking
Esteemed Contributor

Re: Reboot Schedule for HP-UX 11iv3 system.

Good point, Viktor. I was thinking about mentioning that in my reply, though that gets into disaster recovery and other areas not well suited to short replies. That's a "pay me now or pay me later" situation. Hitting those type of failures on a staggered basis may be much less painful than hitting them simultaneously on multiple machines, such as after a site-wide power failure. Some of those types of failures show up on simple reboots, while others are more likely after power cycling when devices may go through more exhaustive self-tests, more thermal changes, etc.