Adding RAID-1 to FreeBSD with GEOM
(or how to avoid using expensive, dedicated, hardware RAID cards)
By Nathan Butcher 2006.12.6
----------------------------------------------------------------

FreeBSD has it's own software RAID solution in the form of GEOM.
There have been older implementations of FreeBSD software RAID (old vinum, ccd, etc),
 but GEOM seems to be the way forward for FreeBSD 6.x

People I have talked with have said that, in most cases, the software RAID performance 
with GEOM in FreeBSD is equal to, if not better than, hardware RAID. 
Even if they are hoplessly wrong, they assert that GEOM software RAID does not tax the 
server very much anyway to be noticeable to performance, and it's cheaper too.
They have GEOM ruinning on production servers.

Most of this information explains how to set up a simple GEOM RAID-1. I borrowed a lot 
of info from http://people.freebsd.org/~rse/mirror/ , which also has more information 
on a more flexible solution.

Also note that it is possible with GEOM, do create RAID devices from physical disks, 
and then use those new RAID devices to create bigger RAID disks in a heirarchical way. 
By implementing both RAID 1 and RAID 0 amongst volumes you the best of both worlds! 
(and more redundancy than you may need!)

The details are fairly straightforward and basic to follow. For testing I suggest using 
hard disks of exactly the same size for mirroring ( not doing this
results in wasted disk space). The MBR gets copied as well to both drives, which is nice
to know.

This is what you would normally do when building a new server. You will see later how to
implement disk spares with this kind of setup.

*************

Assuming that you have FreeBSD on /dev/ad0, you would like to mirror everything onto /dev/ad1 as well

Add a GEOM mirror label onto /dev/ad1 (your second disk).
It will be called gm0. The command options specify that we will be
verbose (-v), turn off autosyncing (-n), and use the round-robin algorithm when choosing what 
component to read from.

	# gmirror label -v -n -b round-robin gm0 /dev/ad1

Now active the kernel layer (/dev/mirror/gm0 should appear)

	# gmirror load

This command set creates a FreeBSD slice over the whole disk. 
The Bootcode is created with (-B) and (-I) initializes the bootcode.

	# fdisk -v -B -I /dev/mirror/gm0

Next step is to create a disk labels for the second drive. (-w) writes a standard label
and (-B) specifies writing the bootcode.

	# bsdlabel -w -B /dev/mirror/gm0s1

Now, you can create custom partitions on the /dev/mirror/gm0s1 device. However, this is
quite unweildy, because you're doing this from the inside of a text editor without no safety-net.
The best thing to do is to cut and paste the contents of

	# bsdlabel /dev/ad0

and paste the contents into
	
	# bsdlabel -e /dev/mirror/gm0s1

********

From here, we have to generate new filesystems on the mirror. How this is done will 
depend entirely on your labelling layout from your first drive (/dev/ad0). 
The filesystems are all newly created with the "newfs" command. The specifics of 
which devices you will use depends entirely on your labelling layout.

	# newfs -U /dev/mirror/gm0s1a
	# newfs -U /dev/mirror/gm0s1d 
	(etc.)

for each newly created filesystem, you can now mount them onto the system. I usually create
a directory for this under /mnt

	# mkdir -p /mnt/mirror

Then add all the newly created filesystems

	# mount /dev/mirror/gm0s1a /mnt/mirror     (this one for the root partition)
	# mount /dev/mirror/gm0s1d /mnt/mirror/var
	(etc.)

Now we have to copy the contents from our first drive to our second drive. I'll use the dump
command (normally used for backing up tapes) to copy over entire filesystems.
The dump (-L) option specifies that we will be dumping a "live" filesystem (i.e. the file system
we are copying is mounted and currently in use). 
Full backup is specified by (-0) where 0 is the lowest backup level 
(and higher numbers mean incremental backups). 
The (-f) option specifies that the content of your dump will go to a file, 
but in this case, the hyphen (-) after that option tells dump to send the dump to standard output. 
The output is then piped into the target directory, and restored with the restore command.
The restore (-r) option specifies recovery of the filesystem, (-v) speicifies verbose as 
usual, and much like the dump command but in reverse, (-f) asks restore to get it's data for 
recovery, again from standard input.

	# dump -L -0 -f- / | (cd /mnt; restore -r -v -f-)

After this, the system has to be adjusted to prepare for when the mirror is finally put into use.
backup the /mnt/mirror/etc/fstab

	# cp -p /mnt/mirror/etc/fstab /mnt/etc/fstab.orig

then covert all references from /dev/ad0 into /dev/mirror/gm0

	# sed -e 's/dev/\ad0/dev\/mirror\/gm0/g' < /mnt/mirror/etc/fstab_ORG > /mnt/mirror/etc/fstab

And then add the following command to make sure that the GEOM kernel module gets activated.

	# echo 'geom_mirror_load="YES"' >> /mnt/mirror/boot/loader.conf

What this next adjustment does is to force the boot loader to start the
computer up from the newly created second disk, (remember, the new fstab is
on the second disk, and we cannot change the fstab of the current disk or
the computer will die a nasty death!). We need to boot into the system from
the second disk before flipping the mirror on.

	# echo "1:ad(1,a)/boot/loader" > /boot.config

And now to reboot into the mirrored drive

	# shutdown -r now

*************


Now that we can see that everything is working perfectly (it is, isn't it?),
 we can kick start the mirror and enable auto-synchronisation.

	# gmirror configure -a gm0

Now we add our first disk into the mirror, after which, it will be synchronised
with our second disk.

	# gmirror insert gm0 /dev/ad0

You can check out the status of the mirror now with the following command. The 
drives will take a while to synchronize, but eventually they will both
be in ACTIVE state.

	# gmirror list

However, you will need to wait until the disks synchronise before starting up 
properly into your fully working mirror!

Using the gmirror commands, you should now be able to see how to make 3 disk RAID 
mirrors and prepare hot spares should you ever require them. Read the gmirror man 
pages.