Routine maintenance

[clearcase]

I am a hpux sys admin maintaining over 80 HP9000 servers runing mixed 10.20 and 11.11. This is a critical production environment.

We do monthly maintenance for all the servers - rebooting and installing latest patches.

I am curious how often do you do maintenance in your environment. Please share your experience.

Thank you,
Lixin Qin

[Michael Tully]

We have reboot two production servers weekly. Unfortuate but true, these have a memory leak that we are still waiting for the application vendor to fix. The rest are rebooted only when necessary (unscheduled outage ... not often) and regular patching routines.

Here is a recent posting on a similar subject.
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x136128c64656d71190080090279cd0f9,00.html

[John Poff]

Hi,

We take one Saturday each month for maintenance. We only patch quarterly, test and development systems one month and production systems the next month. We can also get hardware maintenance done on these days.

JP

[Steven E. Protter]

Second and forth saturday nights.

Patches and stuff go on development machines first and are then installed on production machines after 2 two four weeks of actual testing.

After maintenance there is a testing procedure to see that major user applications are still working.

We do not boot servers that do not require it.

SEP

[A. Clay Stephenson · Accepted solution]

Normally SCHEDULED maintenance is not a problem; however, unplanned ourtages are a big problem. I am now at about 3.75 yrs with zero unplanned downtime.

1) Install patches; new untried hardware on a Sandbox.

2) After verifying the sandbox environment; install on a test/development environment.

3) After verifying on test box(es), request a maintenance window. You should now know rather precisely how much time will be required; allow a little leeway for error but NEVER EXCEED your planned outage period.

I typically only shutdown once a quarter to apply maintenance patches. You will go a long way towards near 100% planned uptime by using onlyhotswap/hotplug components. I have enough disk drives running that one or two replacements a week is not uncommon but that requires no downtime. I also keep common spare on hand (disks, cables, terminators, controllers, LAN cards, and memory).

If you always keep your planned outages within your requested timeframe, you should have little difficulty getting them.

The exceptions to quarterly scheduled maintenance are those patch release notes which mention things like "possible data corruption". Read those carefully and apply them ASAP. The very last thing that you want is little bits of data corruption here and there that you might not find out about for months.