Precious Downtime - What to do?

[Greg Stark_1]

I am about to schedule downtime for each of our servers to upgrade to the latest OS patches. During this time, I'd like to update as many other things that needed to be updated as possible such as STM, PDC code, etc...

Can anyone think of other maintenance type things that should be performed during a scheduled outage?

Thanks,
Greg

[Jannik]

veritas if you use it!

[A. Clay Stephenson · Accepted solution]

It's a good time to review any static kernel tunables and see if they need tweaking.

Disk Array / Tape Library / Drive Firmware updates.

You aleady mention the other firmware updates.

I also take the time while the boxes are down to check each and every SCSI/Fibre/Network connector and power cord to make sure they are still tight.

It's also a most excellent time to test UPS's -- especially those that simply have a Test/Exercise button rather than better diagnostics. If any of your UPS batteries are nearing the 3 year mark, now is a good time to replace them as well. The same goes for any batteries in disk array controllers.

Finally, I trust that you have actually applied all the intended patches on a Sandbox so that you know they are safe --- and how long they take to apply. I find that scheduled downtime is not terribly difficult to get, iff you always keep your promises about when the systems will again be available.

Just before starting the patches is a very good time to do a "lifeboat" backup to another set of boot disks --- not the mirrors, so that no matter how bad the patches are, you can restore the boxes to their "before" state quicker than you can say "Ignite".

[SS_6]

I understand your concern to take advantage of scheduled downtime window but it is not good idea to put everything at once. You may encounter some problems and it will be very hard to trouble shoot. A.Clay has given good points but I would suggest you to do the tasks in steps. Like STM etc.are Online changes. You may further do the cluster failover test(if have cluster) or clean up root file systems.Check the status power supplies from GSP, speed of LAN cards after reboot.

[Steve Post]

Write up a plan to do the stuff. Make a check list. You don't have to make the plans look official. You do this for yourself, so that you don't miss anything.

Make printouts of vgdisplay, "lvlnboot -v", ioscan, bdf, /etc/hosts, /etc/fstab, etc.

Have an ignite tape of your box.

And I agree about not making too many changes. Let's say to make 5 changes to the system, and it is now acting weird. Which one of the 5 changes caused it?