==Professionally upgrading FreeBSD one version up==
By Nathan Butcher 2006.12.6
In these notes, I detail a method of systematically updating FreeBSD with a
minimal amount of system downtime. These notes have been tested on FreeBSD
4.10->4.11, but they should work equally well on FreeBSD 5.x as well. Also note
that you can upgrade from FreeBSD 5.4 to 6.0 with this method. However, doing
this requires that new /etc/passwd and /etc/group files be merged, so BE CAREFUL.
In fact, go check out the /usr/src/UPDATING file because that will usually tell you of any pitfall which may befall you before upgrading.
Note that it always makes good sense to test an upgrade before you do the real
thing. You will discover a plentitude of possible problems by doing this - most
of which you will be able to solve before you get into downtime.
(1) Firstly, if you have cvsup as a cron job running in the background, (as is
common practice) you should comment it out or disable it.
The reason is because if cvsup runs and updates the kernel source code in
/usr/src between buildworld and installworld, your installworld is going to
crash, and crash hard. Take it from me, fixing a crashed intallworld really
sucks. Learn from my mistake.
For exmaple, in your crontab if you have something like this, comment the lines
out:
#0 1 * * * root /usr/local/bin/cvsup -g -L 2 /usr/local/etc/cvsup/stable-supfile
(2) Adjust your cvsup stable-supfile settings to reflect the version you will be
upgrading to. You may have these in their respective cvsup directories or
somewhere (Possibly in your make.conf file if you use the "make update" feature
from /usr/src.). When upgrading frm 4.10 to 4.11 I made the change listed below.
# cd /usr/local/etc/cvsup
# less stable-supfile
check this -> * default release=cvs tag=RELENG_4_11
(3) Start cvsup
Hopefully you've already your automated cvsup settings. Now do a manual update of
cvsup with your stable supfile. This will update your kernel sources to the next
version.
# /usr/local/bin/cvsup -g -L 2 /usr/local/etc/cvsup/stable-supfile
(4) Recheck your /etc/make.conf file, and ensure your KERNCONF variable has the
name of the kernel config you will be using. Most people (if they are using the
kernel ipfiler or SMP kernel configurations) will list their custom kernel
config in KERNCONF.
(5) Remove the contents of /usr/obj. It will be used for buildworld, so it's best
to clear it out before work really commences.
# cd /usr/obj
# chflags -R noschg usr
# rm -rf *
(6) Start buildworld. This will process away on your running server. It will not
affect anything currently running on the server.
# cd /usr/src
# make buildworld
(7) Owing to a successful compile (which should happen), it's time to rebuild the
kernel for the updated system. With KERNCONF listed in your make.conf file, you
don't need to mention KERNCONF here.
# cd /usr/src
# make buildkernel
(8) Pre-Reboot system checking.
When you are updating a server you don't have much knowledge of, it's a risky
proposition to reboot it. Some sloppy admin (and hopefully not you) may have come
before you may have made some undocumented changes which will affect the system
on reboot, or at least they didn't put these changes into a start script which
gets executed at reboot. As a result, it makes good sense to check the system out
for any oddities before you reboot.
Check IP address settings, and other network settings.
# less /etc/rc.conf
Investigate the start script directory, to make sure that services will
start up upon reboot.
# cd /usr/local/etc/rc.d
# ls -al
If the server has ipfilter rules, check them.
for example:
# less /etc/ipf.rules
and make sure there is actually a kernel to reboot.
# cd /
(9) Make a copy of how all the net sockets should look. We can check against this
when we reboot the machine.
# netstat -an > /home/user/socket_all.txt
(10) Also, it may be a good idea to back up /etc should a mergemaster do irreperable damage later on before work commences. Sometimes the /etc/passwd and /etc/group files get changed (at least they do from FreeBSD5.4 -> FreeBSD6.0)
# cp -Rp /etc /etc_bak
---Serivce Time---
At this point, I assume that you have made an announcement at least one week
prior that service downtime will occur on your server. If not, then you should.
This is especially important in a large institute or Service Provider when users
use your network 24-7. You will also have to let your customer/boss know what
you will be doing exactly.
You should also schedule this work at a time when you are less likely to disturb
users (such as their lunch time, or after hours). You should also have a
redundant server ready as backup when one of them goes down for service (it is
standard practice to have a redundant server anyway) but you want to make sure
it's capable to take over for a while.
If that's not enough to consider, remember that it's not a good idea to do
maintenance work right before you go on holiday or take the weekend off (so
Fridays are off-limits). Otherwise if your update fails or develops a few quirks,
you won't be around to fix them.
Lastly, make sure you actually do the work at the time you planned. It is easy to
forget that you have to do certain work at a certain time if you don't have a
large calendar, or personal organizer with alarm, or something to keep you
organized. Maybe somebody will call you out to lunch and distract you from the
work you planned on doing. It happens, and it shouldn't. Keep to your schedule.
With all these precautions done, you should be able to upgrade the server without
anybody noticing that it was ever down - which is how it should be. Yes, this is
the joy of being a sysadmin. If you do a good job, nobody ever knows about it.
When you screw up EVERYONE finds out. You just can't win!
(1) Install the new kernel
# cd /usr/src
# make installkernel
(2) Reboot the machine. Now your work really begins because we are working during
our scheduled downtime period.
# shutdown -r now
*REBOOT*
Boot into single user mode:
ok boot -s
When prompted, use /bin/sh
The following series of commands guarantees that your file systems will come up
without any problems. I don't think you need to fsck the filesystem (especially
in FreeBSD 5). You can probably skip it, but if you don't have to, maybe you
should run it anyway.
# fsck
# mount -u /
# mount -a -t ufs
# swapon -a
# adjkerntz -i
(3) Start a script log
This log will be helpful in debugging should installworld die an ugly death. Pray
that you never have to use it!
# script /root/rebuild.out
(4) For installworld to run smoothly, you need to run mergemaster in
pre-buildworld mode (yes, I know we've already built world)
# mergemaster -p
Do you wish to delete what is left of /var/tmp/temproot ? [no]
(5) Start installworld
After you have finished prayers to your deity of choice, proceed to installworld.
# cd /usr/src
# make installworld
(6) Assuming your install didn't bork itelf, it's time to do mergemaster for real.
# mergemaster
Use existing /var/tmp/temproot ? yes
This is where it gets a little tricky. Mergemaster compares the /etc directory
for configurations that may conflict with the new version. It's usually a good
idea before going through with an update like this to do a test first to see
what's going to get overwritten. If you don't, things won't go very smoothly
during your maintenance downtime.
(For example: the FreeBSD4.10->4.11 /etc file changes and what I did for
most systems):-
/etc/defaults/rc.conf (install)
/etc/MAKEDEV (install)
/etc/rc.conf (merge)
usr/local/examples/make.conf (install)
/etc/defaults/pccard.conf (install)
/etc/defaults/periodic.conf (install)
/etc/gnats/freefall (install)
/etc/mail/freebsd.cf (install)
/etc/mail/freebsd.submit.cf (install)
/etc/mail/sendmail.cf (delete)
/etc/mail/submit.cf (delete)
src/etc/mail/Makefile (install)
src/etc/mtree/BSD.usr.dist (install)
/etc/mtree/BSD.x11-4.dist (install)
/etc/periodic/weekly/400.status-pkg (install)
/etc/hosts (delete)
/etc/hosts.allow (merge)
/etc/motd (install)
/etc/rc (install)
/etc/rc.firewall6 (install)
src/etc/remote (install)
/etc/services (install)
/COPYRIGHT (install)
Delete /var/tmp/temproot ? no
Run MAKEDEV ? yes
This bit works a bit differently on FreeBSD 5.x and up to 6.x (no need to run MAKEDEV on FreeBSD 6.x if I recall correctly).
(7) Reboot the server
Now to see the results of what's been done, and time to cross your fingers.
# shutdown -r now
*REBOOT*
Upon (hopefully successful) reboot, you can now check what version you are running at.
# uname -a
(look for the "FreeBSD x.x-px KERNELNAME)
(8) Service operation check
ps will check the status of processes running on your server. Make sure your services are all up.
# ps -ax
(9) Reinstating automatic cvsup
Congratulations for getting this far.
# vi /etc/crontab
UNCOMMENT
---------------------
0 1 * * * root /usr/local/bin/cvsup -g -L 2 /usr/local/etc/cvsup/stable-supfile
---Damage Control---
If something has gone wrong, try going back over what you did that may have
caused the problem. Definitely any changes to /usr/src can cause an update to
crash. Other reasons may include missing a step, or misconfiguring a file
somewhere. As always, it pays to check and double check your work.
If the worst happens and you want to "roll-back" here's what to do:
(1) swapping old kernel and modules back.
You need to unlock the kernel to swap it out for your old one.
# chflags noschg kernel
# mv /modules /modules.new ; mv /kernel /kernel.new
# mv /modules.old /modules ; mv /kernel.old /kernel
(2) Double check everything
IP address, network, system settings.
# less /etc/rc.conf
Start scripts
# cd /usr/local/etc/rc.d
# ls -al
IPfilter rules
# less /etc/ipf.rules
Kernel's existance.
# cd /
(3) Finally, reboot into your old kernel.
# shutdown -r now
