Red Hat Upgrade checklist suggestions.

[Steven E. Protter]

I have tested Red Hat Enterprise 3.0 release 1 for a few weeks now on two test servers.

I've worked out all the kinks and every bit of my configuration works right.

So I'm planning an upgrade: Here is my current plan. I'd like comment and suggestions.

The production server is 7.3 and there is no upgrade path. I have to overwrite. I am satisfied with the disk setup which is RAID 1 and won't be needing to change that. I have adequate space on /var and /usr

1) Take the machine off the public Internet and have one of my backup servers handle web traffic. This will be done by adding the ip address via the ifconfig command. This has been tested. Test that websites are functional on the backup server.

2) Back up the /home filesystem(all the websites) to three other machines and tape. In the intall I don't plan on formatting /home

3) Copied /etc/passwd /etc/group /etc/shadow I plan to edit these copies on another box and leave my add in users alone. I may need to reset the passwords, but the user and group status will be perserved.

4) Copy the /var/mail data off so its there waiting for the users when the machine comes back online.

5) Copy the entire /etc configuration to another machine via tar. This is to have a reference and be able to copy back in components as I need them when I bring production back online. This includes all /sendmail customization.

6) Copy /usr/contrib/bin scripts to another location. This has been practiced and if I keep my hot little fingers off I don't have to do this again.

7) Verify all items are present on all three backup servers visually, while checking off a printed checklist with these items on it.

8) Copy and test all virtual host information from the apache 1.3 isntallation and update it to work with 2.0.x syntax. I've already done thsi and tested it on an HP-9000 server.

9) Run the Red Hat ES 3.0 installation, not formatting the /home filesystem.

10)Copy in appropriate configuration files that were tested and I know work.

11) Turn on ipfowarding on the firewall so that my wife can get on the Internet


12) Copy in user /var/mail from backup.

13) Copy in and test custom configuration of sendmail.

14) Copy virtual hosts from the working HP-9000 apache 2 server into the new httpd.conf Make other changes required to make cgi-configuration and ssi work as it is supposed to.

15) Enable pop2 and pop3 in xinetd.conf

16) Bring down the IP addresses added to the failover server running the web sites.

17) Delete keys to the old production server from all known_hosts files.

18) Rebuild and exchange public keys from new production server to all other Linux/Unix hosts

19) Reinstall awstats programs in new production server.

20) Physically reconnect the production server to the public Internet. Restart the server and run a services checklist:


http
ftp
ssh
sendmail
popmail
smtp
formscript
elm
custom scripts test on the commandline
website stats testing
A. Clay Stephenson date scripts testing
firewall

Make changes as necessary to make the server work properly.

All suggestions will receive some points.

If you catch something major that I missed, you get a bunny.

SEP

[Olivier Drouin]

Connect to the net
Run an up2date right after.
Disconnect from the net.

Run tripwire to build a known sane system database. Reconnect to the net.

[Paul Cross_1]

Of course there are probably a lot of site-specific things that we can't help you with, but the only things that popped into my head were:

- Wouldn't it be easier to just copy the old ssh keys to the new server instead of regenerating new keys and distributing the new key? This is a VERY minor point, but it avoids that annoying "someone is doing something nasty" message, which tends to freak out end users and annoy sysadmins.

- Make sure that any non-standard perl modules get installed that may be necessary for cgi scripts (mod_perl too! this one bit me in the ass once during an upgrade).

ps: what a great idea, vetting your test plan with the community... I like it!

[Steven E. Protter]

pcross

A key mismatch against known hosts occurs. I think its a good idea to generate new keys once in a while. An upgrade is a good time to do so.

tripwire is not distributed on the RH ES 3 disks.

I need to figure out how to correctly compile the open source version because the rpm version generated errors on my test servers.

SEP

[Paul Cross_1]

SEP,
You are probably right about generating new keys every once in a while just for security's sake, but I've never had a problem moving a host key from machine to machine in terms of key mismatches.

-P

[Bruce Copeland]

SEP:

You're probably not running any postgreSQL databases. But just in case, remember to run pgdumpall on the database(s) BEFORE upgrading since most upgrades entail a version change. I suggest this because I once forgot to do it before a distribution upgrade, and it caused me no end of grief .

Bruce

[Stuart Browne]

Just root's home directory (/root/). I know a number of little bits and pieces get dumped there on some of my boxes.

[Mark Travis]

Quickly rolling back to the starting point is a very nice thing. Especially when the number of maintenance steps exceeds the number of fingers one has.

I would think about peeling off one of the mirrors prior to the upgrade. Then mount its partitions read only for reference.

This would allow you to perhaps save a few manual steps. It also gives you a very fast way to roll back if worse comes to worse and your secondary(ies) goes offline for some reason.

[Stuart Browne]

Oh, side note.. make sure you get the contents of /var/spool/mail/, and not just the symbolic link /var/mail ;)

[Martin P.J. Zinser]

Hello SEP,

minor stuff:

- Check /usr/local for any non-distro software you might have installed. If != 0 get current
RPMs or source distributions
- Make sure your customers/users know about the upgrade
- Get some of the more cooperative customers/users to pound on the system as soon as it is back again to find any problems asap.

Greetings, Martin

[Steven E. Protter]

As expected, Stuart catches the first thing /var/spool/mail that I totally missed.

Bunny alert.

My plans are partially disrupted due to a NIC failure on one the HP-9000 server thats serving as one of the three backups.

SEP

[Steven E. Protter]

I was wondering if I can get away with NOT formatting the /var filesystem.

SEP

[Steven E. Protter]

Getting a warning from the httpd server some kind of virutal hosts issue.

Non-critical

Starting httpd: [Sat Feb 21 23:15:45 2004] [error] VirtualHost _default_:443 -- mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results


A bunny for a fix.

attaching httpd.conf

SEP

[Steven E. Protter]

Okay i fixed that.

but the virutal hosts in an include file in /etc/httpd/conf.d and added it as an inlucde in httpd.conf

Is it possible to get ssl working on a server doing virtual names hosting.

Please look at my config file virtual hosts section and let me know.

I was planning on enabling it and doing some authentication for part of one of my sites.

Also wondering if the iframe directive has been obsoleted from apache 2.0

SEP

[Stuart Browne · Accepted solution]

By default, the SecureServer is set up as a VirtualHost.

On a RH8/9, there'll be an /etc/httpd/conf.d/ssl.conf, of which inclues stuff like:



...

SSLEngine on
SSLCipherSuite ...
SSLCertificateFile ...
SSLCertificateKeyFile ...


You can use the SSL configuration directives on ANY VirtualHost section.

Now, as for the 'iframe' directive, I thought that was a HTML thing, not a server directive. It isn't listed in the Apache configuration documentation.

[Stuart Browne]

Oh, just re-read the post above :P Mixing _default_: and *: dont mix.. but you can use a with a 'servername' directive just like any others with the SSL directives..

[Steven E. Protter]

I'll be working with this on the secondary and tertiary ES 3.0 Servers through the week. I'll report back on progress or start a new thread as needed.

A good ssl cookbook covering everything including customizing the certificates would be nice but there might be enough on apache.org to proceed.

SEP

[Steven E. Protter]

I found vsftp refreshing. It has it's own service. Its user control status seems much better than ws-ftpd.

Not sure I'd call it secure, but its a major improvement.

SEP.

[Steven E. Protter]

I just got squirrelmail working.

Quite a nice little featue.


The only portion of my original plan that was not in place Monday morning was the popmail server. pop2 was working pop3 was disable=yes

Not a bad transition if I say so myself.

Thanks a lot for all your wonderful help.

SEP