Preparing for disaster is an essential part of system
administration. You must plan a recovery strategy for any potential disaster,
from the loss of a single disk to the total loss of an entire data center.
Off-site data storage is an essential part of a disaster
recovery plan. In the event of a catastrophic disaster, the only safe data
repository might be at an off-site location. You can routinely send one archive
copy of important files to a single VSN or set of VSNs. Those VSNs can be
exported on a regular schedule and taken to off-site storage. Data from
exported VSNs can be made available by returning the VSN to the library and
importing it.
This paper provides detailed information on preparing for
disaster recovery for SAM-FS and SAM-QFS software file systems. It covers
planning for backup, performing backups and recovering from file and file
system loss.
The first section covers backup planning and the creation
of metadata backup for the archiving file systems SAM-FS and SAM-QFS. The
second section covers restoration of file system metadata for an entire
archiving file system and for a single file. The final section of this paper
discusses two methods for reading individual files off tapes when there is no
metadata backup.
Sun StorageTek SAM and QFS software must be installed on
servers running the Solaris Operating System. Some existing Solaris OS
configuration files are created or edited as part of the installation process.
In addition, all the log files, executables, control files, man pages and
configuration files installed by SAM-FS/QFS reside on UFS file systems.
Only user and application data is placed in the SAM-FS,
SAM-QFS and QFS file systems. Regular files on those file systems are archived
and need no other backup, but metadata is not archived in any usable way and
must be backed up. There are therefore two types of data that must be backed up
on your SAM-FS, or SAM-QFS server: administrative files contained in
UFS file systems and the SAM-FS and SAM-QFS file system metadata.
Backing Up
Administrative Files
Administrative files on the UFS file systems require no
special backup strategies other than those your site currently uses to back up
the root, /usr and /var partitions.
The locations of administrative files used with SAM-FS/QFS
are listed below along with suggested backup frequency.
|
File |
Back Up Frequency |
Location |
|
Configuration, log and control files |
Regular |
• /etc/opt/SUNWsamfs • /var/opt/SUNWsamfs |
|
Executables and man pages |
At installation and when modified |
• /opt/SUNWsamfs |
|
Solaris OS files created or modified at installation time |
At installation and when modified |
• /etc/syslog.conf • /etc/system • /etc/inittab • /etc/name_to_sysnum • Files in /kernel/drv • Files in /kernel/strmod • Any other files configured as part of installation |
|
Dump files for SAM-FS, SAM-QFS or QFS file systems |
Daily |
Location chosen by administrator |
|
SUNWqfs and SUNWsamfs software package and patch |
Once, shortly after downloading |
Location chosen by administrator |
Page 2
The files in /var/opt/SUNWsamfs, /etc/opt/SUNWsamfs, /kernel/strmod
and /kernel/drv requiring backup will vary with the hardware and software
configuration of your server.
The Sun StorageTek SAM software archives data placed in
SAM-FS and SAM-QFS file systems, and thereby provides an automatic backup of
individual files at whatever archiving intervals you have configured. These
files can be recovered from archive media directly, but it is a slow and
laborious process. If you want to recover more than a few files, you must have
a backup of the file system metadata, which is made with the command samfsdump. This backup can be used in
case of disaster to quickly restore directories, symbolic links and the .inodes
file to disk. It takes about five minutes per 1 million files to back up with samfsdump and about 15 minutes per
million files to restore that metadata. Files are then staged back to disk as
they are accessed without any special recovery process. Run samfsdump at least three times a day.
QFS standalone file systems are not archived, and therefore
must be backed up on a regular schedule like any other non-archiving file
system. The command qfsdump backs up
data and metadata on a QFS standalone file system, much the way ufsdump backs up a UFS file system. The
qfsdump command may not be adequate
to the needs of your file system, in which case you will need a separate backup
solution. Since QFS standalone file
systems have all data and metadata on disk at all times, backup software such as NetBackup or Legato
can be used to back them up. Such software will only back up the first 128
bytes of the inode, so if some QFS specific attributes have been set, they will
be lost. SAM-FS and SAM-QFS file systems
can also be backed up with backup software, but all files will have to be
staged first, so this generally does not work. SAM-FS and SAM-QFS file systems
also need the part of the inode not backed up by a backup solution, so proper
archiving and metadata dumps are generally the only possible way to back up
archived file systems.
Files in SAM-FS and SAM-QFS file
systems that have not been archived cannot be recovered with any method if they
are lost. You must set your archive interval and archive age so that important
files are archived frequently enough that they can be recovered. Inform users
of backup strategies and make sure they understand their role in protecting
their own data files.
Page 3
The samfsdump
command creates a dump file containing metadata for each specified file or
directory. The syntax of the command is:
samfsdump [-u] [-U] [-P][-other
options] –f dump_file [file...]
The required -f dump_file
option specifies the absolute path name of the dump file. This should not
be located in a SAM-FS or SAM-QFS file system.
The samfsdump
command creates a dump file, as follows:
1. If
the argument to samfsdump is one or
more individual files, the command creates a dump file containing the specified
file or files.
2. If
the argument to samfsdump is a
directory the command creates a dump file containing the metadata for every
file in that directory and its subdirectories.
3. If
the files to be dumped are not specified, the samfsdump command creates a dump file containing the metadata for
every file in the current directory and its subdirectories.
The simplest method of performing a metadata dump is as
follows:
1. Change to the mount point of the file system or to the
directory that is being dumped:
# cd /sam1
2. Create a dump file with the samfsdump command:
# samfsdump -f dump_file
You can also specify a directory name under the mount point
as the argument to the samfsdump command.
The samfsrestore command will always
restore to the path name used to create the dump file and it will balk at
restoring any UFS directory, including the mount point, so you must change
directory into the mount point to perform a dump of metadata, even if you wish
to specify a particular directory to dump. For example, to back up the the
directory “memos” in the SAM-FS file
system mounted on /sam1, you could issue the samfsdump command while /sam1 is the current working directory, or
you could issue the following commands:
1. Change to the mount point of the file system or to the
directory that is being dumped:
# cd /sam1
2. Create a dump file with the samfsdump command:
# samfsdump -f /var/tmp/sam1dump memos
By default samfsdump
does not dump any data, so files without a current archive copy can not be
restored from the dump. The -u
option to samfsrestore includes any
such files in the dump. You can also include all data current on disk in the
dump with the –U option, and all
stubs left by partial releasing with the –P
option. The –U and –P options will considerably increase
the size of the dump, but –U allows
you to restore the file system exactly as it was directly from the dump, while –P allows you to restore the file
system and any file stubs without having to stage and re-release files that may
be very large, and which you may not even be able to stage.
When you run samfsdump
without the –u option on a directory
containing files without a current archive copy, you will be warned that they
are unrecoverable:
Page 4
/pathname/filename: Warning! File data will
not be recoverable
(file will be marked damaged).
If the disk is lost, you can put back that file's inode,
but the file will be "damaged." Damaged files cannot be recovered using the dump file, even
if they archived after the dump occurs and before the disk is lost, because the
backup contains no information about the location of their archive copies. Such
files can be recovered using the request
and star commands and the archiver
logs as long as you have backups of the archiver logs available. The request and star commands are discussed later in this module. Similarly files
created, archived and subsequently lost between dumps are recoverable the same
way. Files created but not archived will be lost in the event of a disk
disaster.
Dumps are performed with the file system mounted. Send your
dump files to a UFS file system that will be backed up at least daily or is on
another host. If the dump is
stored in a SAM-FS/QFS file system it can be automatically archived
immediately. That would be convenient were it not that the metadata for the
dump itself will then have to be included in another dump stored somewhere
else. In the event of multiple system loss, you will find it difficult to
locate your dumps. They should go to UFS file systems.
You must have enough space on the disk to which the dump is
sent to hold the .inodes file, directories, and any other metadata on the disk.
Each inode is 512 bytes and so a file system with many files can produce a very
large dump even without file data. If
you use the -u option to samfsdump, the size of the dump will
increase by the number of bytes contained in the backed up files.
If you routinely run the recycler as a cron job, you must
synchronize it with your samfsdump
metadata dumps. Once the recycler has completed a run, some archive copies will
have moved and the dump file will be obsolete. Rearchiving of files is logged
in the archiver log file, so files moved as part of recycling can also be
recovered using the request and star commands.
The samfsrestore
command allows you to recover metadata for a single file or for an entire file
system from a metadata dump. The file’s data can then be staged from an archive
copy. This section covers recovery of a single file or an entire file system
using samfsrestore. The following
and final section discusses the recovery of files without using samfsrestore.
The samfsrestore
command uses the contents of the samfsdump
dump file to restore metadata. It can be used to restore one or more specified
files or all the files in the dump file.
The syntax of samfsrestore
is:
samfsrestore [-t]
[-g log_file] -f dump_file [file…]
The samfsrestore
command takes these important options:
Page 5
The required -f dump_file
option specifies the absolute path name to a dump file created with samfsdump.
-t writes a table of contents to standard output. No
restoration is performed.
-g log_file specifies a log file
to which all restored files are written. This log file is used with the script
/opt/SUNWsamfs/examples/restore.sh to automatically stage all files that were
in disk cache at the time of the dump.
The samfsrestore
command returns metadata to disk. Unless a file is specified, metadata for all
files in the dump file are restored. If a filename is specified, only that file’s
metadata is restored. Its path and file name must exactly match the path and
file name in the dump file. By default, all files are restored to the location
specified in the dump file.
The samfsrestore
command can be used to restore a single file from output created with the samfsdump command. To restore using a samfsdump output file:
1. Create a directory in which to restore the files.
# cd /sam1
# mkdir restore
2. Use the archive
command with the -n and -r options to prevent the archiver from
making archive copies of file data placed in this directory:
# archive -n -r restore
The -n option
prevents the archiver from archiving files or directory information in the
restore directory. The -r option
ensures that new files added to this directory inherit the no archive
attribute.
3. Change to the restore directory:
# cd restore
4. Use the samfsrestore
command with the -t and -f options to verify the file’s name
and path:
# samfsrestore -t -f dump_file | grep file_name
Identify the syntax of the file name to restore. The file_name
used with samfsrestore must
exactly match the name of the file being restored.
5. Use the samfsrestore
command with the -f option to
restore the file relative to the current directory, as follows:
# samfsrestore -f dump_file file_name
To restore a SAM-FS or SAM-QFS software file system by
using samfsdump output determine
whether you will put the contents of the file system in a directory in an
existing file system or will reconstruct the entire file system. In the latter case, follow the procedures
outlined in the paper Configuring the
SAM-FS and QFS File Systems to construct and
initialize the file system.
Use the following procedure to restore files to the
reconstructed file system or to the destination directory:
a. Change to the mount point for the file system or to the
directory where the file system will be restored:
# cd /sam1
Page 6
b. Use the samfsrestore
command to restore the entire file system, relative to the current directory,
from an existing dump file:
# samfsrestore -f dump_file
The restore.sh executable shell script takes the samfsrestore log file created by the
following command as input:
# samfsrestore -f dump_file -g log_file
It then generates a stage request for every file that was
on disk at the time the dump_file was created. The script is executed as
follows:
# restore.sh log_file mount_point
By default, the restore.sh script restores the entire
contents of the log file. If the site has tens of thousands or hundreds of
thousands of files to be staged, segment the log file into manageable chunks
and run restore.sh on each of those chunks so that the staging process does not
overwhelm the system.
Single files or the contents of an entire tape may be
recovered even if there is no samfsdump
output file available. This section of the module will discuss recovery
directly from tape using the request
and star commands with the output of
the archiver log file or by using the mt
and star commands.
The request
command takes the VSN and position on tape of a a tar file containing an
archive copy of a file as input. This information is available in the inodes or
in the archiver log file. It then creates a removable-media file on a SAM-FS or
SAM-QFS file system that acts like a logical device name for the tar file at
that position on the tape. The removable-media file created with request can then be used with the star command to restore any file in
that tar file to the current working directory.
Archives of removable media files can cause problems with
recycling, so they should always be placed in unarchived directories, and they
should be deleted as soon as restoration is done.
The request
command takes the following syntax:
request -m media_type
-v VSN -p position file
Where:
-m media_type specifies the
Equipment Type identifier of the tape from field four of the archiver log file.
-v VSN specifies the VSN of
the tape on which the archive is located from field five of the archiver log
file.
-p position specifies the
position on tape of the tar file containing the archive copy of the file. This
position must be preceded with “0x” to indicate that it is a hexadecimal value.
The position can be found in the first part of the seventh field of the
archiver log file. Use only the value in front of the dot (.).
Page 7
file is the name of the
removable-media file. The name of this file is up to the administrator, but it
must not be the same as the name of the file to be restored if they will go in
the same directory.
The star command
is used by the sam-arcopy daemon to write archive requests to tape or disk. It
uses the standard UNIX syntax in which options are preceded by a dash and
arguments immediately follow options. The star
command also supports the familiar tar
format for options.
The star command
is used by the administrator to read directly from tape. Like tar, it can read the table of contents
of a tar file or extract one or more files from a tar file. To extract a file
or files from a tape using star:
#star -xvn -f device_file|removable_media_file -b
blocksize [file...]
Where:
-b blocksize is the block size in
512 byte blocks in which the tape drive writes. This information can be
obtained by issuing the samu command
and then typing r to get the
removable media menu. That menu displays the blocksize in bytes in which the
tape is written. This must be divided by 512 to get the block size in 512 byte
blocks. For example, if the blocksize displayed in the output of samu r is 16384, the block size used
with star will be 32. Any integral
multiple of the blocksize will also work. DAT tape drives and Exabyte tape
drives write in blocks of 32 - 512 byte blocks by default.
-f device_file|removable_media_file
is the logical device name of the
drive containing the tape, or the name of a removable-media file created with
the request command.
-n allows star to
overwrite a file with another file of the same name only if it is newer than
the file already on disk.
-x restores a file or files to disk
-v indicates verbose output
file... is the name of the
file or files to be restored.
To read the table of contents of a tar file, use the
following syntax:
#star -tv -f device_file|removable_media_file -b blocksize
[file...]
Where -t sends
the table of contents of the dump file to standard output.
The following procedure demonstrates the restoration of the
metadata and data of a file. This example uses a file called test which was
archived on a tape with the VSN TAPE03.
1. Create a working directory for restoring the file, and
set it recursively to no archive:
# mkdir /sam1/restore
# archive -n -r /sam1/restore
2. Examine the archiver log file and locate the archive
copy you want to restore. Find the VSN, the equipment type identifier of the
tape and the position of the tar file containing that archive copy.
One line from an archiver log file entry is shown here. The
format of this entry is discussed in detail in the paper Archiving Concepts. In this example, the VSN of the tape on which this
Page 8
archive copy is located is TAPE03. The location of the tar
file containing the archive copy is 426, and the Equipment Type Identifier for
the drive type on which the file was archived is lt.
A 2003/11/26 15:01:54 lt TAPE03 all.1
426.1 samfs1 1149.1 1121 test f 0 100
3. Create a removable-media file in the dedicated directory
with the request command:
# request -m lt -v TAPE03 -p 0x426 /sam1/restore/mediafile
4. Verify the contents of the tar file by displaying its
table of contents with the star
command using the -t option:
# star -tv -b 32 -f /sam1/restore/mediafile test
-rw-r--r-- root/other 2242 2003-11-26 15:03 test
5. Change to the directory where you want the file
restored.
# cd /sam1/restore
6. Extract the contents of the tar file with the star command using the -x option:
# star -xv -b 32 -f /sam1/restore/mediafile test
test
7. Check that the file has been restored:
# sls -D test
8. Move the file to its usual location:
# mv test /sam1/test
9. Remove the removable media file:
# rm /sam1/restore/mediafile
It is possible to use the star utility to read files directly from tape. It is difficult to
recover a single file this way, since there is no simple way of positioning the
tape to the file desired. However, if a tape label is damaged, and the tape
contained the only archive copy of a file, reading the tape with star may be the only way to recover the
file. In this section, the basic procedure for reading from a tape is
discussed. The use of the file tarback.sh to automate the process of using star to read in the entire contents of
a tape is then covered.
The Solaris mt
utility can be used to skip past the label, which is the first record on the
tape. The star utility can then be
used to read files from the tape. This process will only work if the tape label
was damaged - not if the tape was relabeled. The process of relabeling a tape
places an End Of Tape mark directly after the label. The star command will not read past this mark. Do not relabel a tape
unless you are certain you do not want anything on it.
The following process demonstrates the use of star to read from a tape with a damaged
label:
1. Change to the directory where you want the files
restored:
# cd /sam1/restore
2. Make one drive unavailable to the SAM software:
# samcmd unavail eq
Where eq is the equipment type ordinal of the tape.
3. Unload the drive just made unavailable if it contains a
tape:
# unload -w eq
4. Load the tape you want to read into the drive:
# load -w -vsn VSN
eq
Page 9
Where VSN is the VSN of the tape.
If you are using DAT tapes or Exabyte tapes,
the command samcmd unavail will
cause the tape to be ejected from the tape drive. Simply push the tape back
into the drive before proceeding.
5. Rewind the tape:
# mt -f /dev/rmt/X rewind
Where X is the logical device name of the drive,
i.e. /dev/rmt/1.
6. Skip past the tape label:
# mt -f /dev/rmt/X fsf
1
7. Read the first file off the tape using the no rewind
logical device:
# star -xv -b 32 -f
/dev/rmt/Xn
8. Repeat step 7 until the desired files have been read.
The tarback.sh script automates the process of recovering
data written to a tape. This shell script loops through the process of using star with the -n option to read every archive file written on a tape volume. This
file data is read back into disk cache if it is newer than the file already on
disk. The end result is a directory containing the most recent version of every
file on the tape, including files that were deleted prior to the loss of the
file system.
There have been problems with star refusing to read files of more than 8 Gbytes. The -E option, used with the tar (not star) command can handle such large files.
Depending on the use and growth needs of your file system, you may be able to use samfsrestore to simplify administrative tasks. One of these is moving file systems. Although samgrowfs allows you to grow a file system on the fly, at times you may need to move an entire file system to a different device or set of devices. Normally backing up and restoring the file system is time consuming, and users cannot write to the file system while the process is occurring.
Because samfsrestore only restores the metadata backed up with samfsdump, there is far less to restore to the new disks than there would be with conventional backup strategies. After the mcf file is updated, samfsrestore can quickly place metadata on the new devices, at which time users can begin using the new file system, staging files as they use them. If desired, the –U option to samfsdump can be used to restore the file system in its previous state.
The samfsrestore command also allows users to recover their own files. If your file system usage involves large numbers of users each creating many small files, you might run samfsdump very frequently, and use it immediately to create restore directories for specific times and dates. Every time you run samfsdump, you would create a restore directory with the output. A user who then finds she has inadvertently deleted a file she needed can simply change directory to the correct directory, and copy the file she finds back to its original location - without requiring administrator action. So if a user deletes a file on 19 Dec that he discovers he needs on 20 Dec, he can cd to the restore directory /samfs1/19dec and get his file back.
This example of the uses of samfsrestore is specific to one type of file system, of course. It works on a file system that has a modest number of files and with ample backup media. If space on archive media is scarce, you will be recycling and obsolete archive copies will be lost. If you have a very large number of files, the metadata backup may be too large to keep backups online.