Live system upgrade tips

[Sasha Pachev]

Hello, everyone:

Recently I have had an opportunity to upgrade a number of live services. In 
some cases the upgrade was not trivial, not successful at the first attempt, 
and required a temporary downgrade to keep the service running while I was 
trying to figure out what exactly had gone wrong. Here are some tips I have 
gathered from the experience, mostly common sense, but easy to forget or 
neglect especially under pressure:

  * Make sure you know every file that the upgrade is going to replace and 
back it up
  * Write a script that will restore the backed up files and start the old 
service. You will to run it at once the momemnt you find out your upgrade 
attempt failed.
  * Have a script that can test if the service is running and at least 
somewhat functional ( eg. try to wget a web page on your server if you are 
upgrading Apache, and diff it against the master copy to make sure it was 
served correctly)
  * Script as many commands as possible, even relatively simple ones if you 
have to type the name of a long file or a directory. For example:

Instead of manually typing:

mkdir /var/lib/prog-data
chown proguser /var/lib/prog-data

put it in a script:

#! /bin/sh

DATA=/var/lib/prog-data

mkdir $DATA
chown proguser $DATA

This way, you have to type /var/lib/prog-data right only once, and if you 
type it wrong, it will be at least consistent and one mv command will fix 
your error. The additional benefit will also be realized if you have to do it 
on another system the next day.

 * If possible, practice the upgrade on a non-live system that is similar to 
the one you will be upgrading. If you have several, it may be a good idea to 
do it on more than one so your fingers will be thinking for you when you get 
to the live one.

I hope some will find this helpful.
 
-- 
Sasha Pachev
Create online surveys at http://www.surveypro.com/