This chapter contains information for the administrative maintenance of DMF:
The daemon generates the SPOOL_DIR/ daemon_name/dmdlog.yyyymmdd log file, which contains a record of DMF activity and can be useful for problem solving for several months after creation. All MSPs and LSs generate a SPOOL_DIR/msp_or_ls_name /msplog.yyyymmdd log file, which also contains useful information about its activity. These log files should be retained for a period of some months. Log files more than a year old are probably not very useful.
Do not use DMF to manage the SPOOL_DIR filesystem.
The dmfsmon(8) automated space management daemon generates a log file in SPOOL_DIR/ daemon_name/autolog.yyyymmdd, which is useful for analyzing problems related to space management.
To manage the log files, configure the run_remove_logs.sh task, which automatically deletes old log files according to a policy you set. See “Configuring Daemon Maintenance Tasks” in Chapter 2, for more information.
The daemon and the LS generate journal files that are needed to recover databases in the event of filesystem damage or loss. You also configure DMF to generate backup copies of those databases on a periodic basis. You need only retain those journal files that contain records created since the oldest database backup that you keep. In theory, you should need only one database backup copy, but most sites probably feel safer with more than one generation of database backups.
For example, if you configure DMF to generate daily database backups and retain the three most recent backup copies, then at the end of 18 July there would be backups from the 18th, 17th, and 16th. Only the journal files for those dates need be kept for recovery purposes.
To manage the journal files and the backups, configure the run_remove_journals.sh and run_copy_databases.sh tasks. These tasks automatically delete old journal files and generate backups of the databases according to a policy you set. See “Configuring Daemon Maintenance Tasks” in Chapter 2, for more information.
When a file is first migrated, a bit-file identifier (BFID) is placed in the inode; this is the key into the daemon database. When a migrated file is removed, its BFID is no longer needed in the daemon database.
Initially, it would seem that you could delete daemon database entries when their files are modified or removed. However, if you actually delete the daemon database entries and then the associated filesystem is damaged, the files will be irretrievable after you restore the filesystem.
For example, assume that migrated files were located in the /x filesystem, and you configured DMF to generate a full backup of /x on Sunday as part of your site's weekly administrative procedures (the run_full_dump.sh task). Next, suppose that you removed the migrated files in /x on Monday morning and removed the corresponding daemon database entries. If a disk hardware failure occurs on Monday afternoon, you must restore the /x filesystem to as recent a state as possible. If you restore the filesystem to its state as of Sunday, the migrated files are also returned to their state as of Sunday. As migrated files, they contain the old BFID from Sunday in their inodes, and, because you removed their BFIDs from the daemon database, you cannot recall these files.
Because of the nature of the filesystem, a daemon database entry is not removed when a migrated file is modified or removed. Instead, a deleted date and time field is set in the database. This field indicates when you were finished with the database entry, except for recovery purposes; it does not prohibit the daemon from using the database entry to recall a file. When the /x filesystem is restored in the preceding example, the migrated files have BFIDs in their inodes that point to valid database entries. If the files are later modified or removed again, the delete field is updated with this later date and time.
The term soft-deleted refers to a database entry that has the delete date and time set. The term hard-deleted refers to a file that is removed completely from the daemon database and the MSPs/LSs. You should hard-delete the older soft-deleted entries periodically; otherwise, the daemon database continues to grow in size without limit as old, unnecessary entries accumulate. Configure the run_hard_deletes.sh task to perform hard-deletes automatically. See “Configuring Daemon Maintenance Tasks” in Chapter 2, for more information.
If you look at all of the tapes before and after a hard-delete operation, you will see that the amount of space used on some (or all) of the tapes has been reduced.
 | Note: Because hard-deletions normally use the same expiry times as backups, the run_hard_deletes.sh is normally run from the same task group. |
This section discusses the interrelationships between DMF and backup products:
 | Caution: The fact that DMF maintains
copies of data on another medium does not mean that it is a backup system.
The copies made by DMF may become inaccessible if there is a failure and
proper backups have not been made.
In addition, although using RAID may protect you against the failure of one disk spindle, data can still be endangered by software problems, human error, or hardware failure. Therefore, backups are essential. |
Many backup and recovery software packages make backup copies of files by opening and reading them using the standard UNIX system calls. In a user filesystem managed by DMF, this causes files that are offline to be recalled back to disk before they can be backed up. If you have a DMF-managed filesystem in which a high percentage of the files are offline, you may see a large amount of tape or other activity caused by the backup package when it initially does its backups. You should take this behavior into account when deciding whether or not to use such backup packages with filesystems managed by DMF.
The xfsdump(1M) and xfsrestore(1M) commands back up filesystems. These utilities are designed to perform the backup function quickly and with minimal system overhead. They operate with DMF in two ways:
When xfsdump encounters an offline file, it does not cause the associated data to be recalled. This distinguishes the utility from tar(1) and cpio(1), both of which cause the file to be recalled when they reference an offline file.
The dmmigrate(8) command lets you implement a 100% migration policy that does not interfere with customary management of space thresholds.
The xfsdump command supports the -a option specifically for DMF. If you specify -a, xfsdump will dump DMF dual-state (DUL) files as if they were offline (OFL) files. That is, when xfsdump detects a file that is backed up by DMF, it retains only the inode for that file because DMF already has a copy of the data itself. This dramatically reduces the amount of tape space needed to back up a filesystem and it also reduces the time taken to complete the dump, thereby minimizing the chances of it being inaccurate due to activity elsewhere in the system. An added advantage of using -a is that files that are actively being recalled will still be backed up correctly by xfsdump because it does not need to copy the file's data bytes to tape.
You can also use dmmigrate to force data copies held only in a DCM cache to be copied to tapes in the underlying volume groups. This removes the need to back up the cache filesystem. However, if you do wish to back up the cache instead of of flushing it to tape, you can use any backup utility. As the cache is not a DMF-managed filesystem, you are not restricted to using xfsdump.
Most installations periodically do a full (level 0) dump of filesystems. Incremental dumps (levels 1 through 9) are done between full dumps; these may happen once per day or several times per day. You can continue this practice after DMF is enabled. When a file is migrated (or recalled), the inode change time is updated. The inode change time ensures that the file gets dumped at the time of the next incremental dump.
To automatically manage dump tapes, DMF includes configurable administrative scripts called run_full_dump.sh and run_partial_dump.sh , which employ xfsdump. Both of these tasks are simple wrappers around a script called do_xfsdump.sh, which performs the following actions:
(optional) Migrates all eligible files to dual-state
(optional) Copies all eligible DCM files on a DCM system to dual-residency state
Performs a database snapshot using dmsnap
Backs up the directory containing that snapshot
Backs up other filesystems
After a successful full backup, frees up old backup tapes for future reuse
DMF also supports a matching wrapper around xfsrestore named dmxfsrestore to be used when restoring files that were dumped by these backup scripts. See the dmxfsrestore(8) man page for more information on running the command.
You can configure tasks in the dump_tasks object to automatically do full and incremental dumps of the DMF-managed filesystems. See “Configuring Daemon Maintenance Tasks” in Chapter 2, for more information.
A typical dump_tasks stanza might look like the following:
define dump_tasks
TYPE taskgroup
RUN_TASK $ADMINDIR/run_full_dump.sh on sunday at 03:00
RUN_TASK $ADMINDIR/run_partial_dump.sh \
on monday tuesday wednesday \
thursday friday saturday at 03:00
RUN_TASK $ADMINDIR/run_hard_deletes.sh at 23:00
DUMP_TAPES HOME_DIR/tapes
DUMP_RETENTION 4w
DUMP_DEVICE dg1
DUMP_MIGRATE_FIRST yes
DUMP_FLUSH_DCM_FIRST yes # Only if you run a DCM
DUMP_INVENTORY_COPY /save/dump_inventory
enddef |
For more information about parameters, see Chapter 2, “Configuring DMF”.
Sites using OpenVault can add new backup tapes by using dmov_makecarts and/or dmov_loadtapes by providing the name of the task group as a parameter. Sites using TMF do not need any special steps to add new tapes, as TMF does not record details of which tapes are available to it.
Recycling old backup tapes is performed automatically after the successful completion of a full dump. In certain situations, such as running out of dump tapes, this pruning must be done manually by running dmxfsprune.
The xfsdump program is written such that it assumes dumps will only be taken within filesystems that are not actively changing. xfsdump cannot detect that a file has changed while it is being dumped, so if a user should modify a file while it is being read by xfsdump, it is possible for the backup copy of the file to be inaccurate.
To ensure that all file backup copies are accurate, perform the following steps when using xfsdump to dump files within a DMF filesystem:
Make sure that there is no user activity within the filesystem.
Ensure that DMF is not actively migrating files within the filesystem.
Run xfsdump, preferably with the -a option.
If you choose to dump and restore DMF filesystems without using the provided DMF scripts, there are several items that you must remember:
The DMF scripts use xfsdump with the -a option to dump only data not backed up by DMF. You may also wish to consider using the -a option on xfsdump when dumping DMF filesystems manually.
Do not use the -A option on either xfsdump or xfsrestore . The -A option avoids dumping or restoring extended attribute information. DMF information is stored within files as extended attributes, so if you do use -A, migrated files restored from those dump tapes will not be recallable by DMF.
When restoring migrated files using xfsrestore , you must specify the -D option in order to guarantee that all DMF-related information is correctly restored.
If you use the Tape Management Facility (TMF) to mount tapes for use by xfsdump, be aware that xfsdump will not detect the fact that the device is a tape, and will behave as if the dump is instead being written to a regular disk file. This means that xfsdump will not be able to append new dumps to the end of an existing tape. It also means that if xfsdump encounters end-of-tape, it will abort the backup rather than prompting for additional volumes. You must ensure that you specify enough volumes using the tmmnt -v option before beginning the dump in order to guarantee that xfsdump will not encounter end-of-tape.
When you restore files, you might be restoring some inodes containing BFIDs that were soft-deleted since the time the dump was taken. (For information about soft-deletes, see “Soft- and Hard-Deletes”.) dmaudit(8) will report this as an inconsistency between the filesystem and the database, indicating that the database entry should not be soft-deleted.
Another form of inconsistency occurs if you happen to duplicate offline or dual-state files by restoring all or part of an existing directory into another directory. In this case, dmaudit will report as an inconsistency that two files share the same BFID. If one of the files is subsequently deleted causing the database entry to be soft-deleted, the dmaudit-reported inconsistency will change to the type described in the previous paragraph.
While these dmaudit-reported inconsistencies may seem serious, there is no risk of losing user data. The dmhdelete(8) program responsible for removing unused database entries always first scans all DMF-managed filesystems to make sure that there are no remaining files which reference the database entries it is about to remove. It is able to detect either of these inconsistencies and will not remove the database entries if inconsistencies are found.
Sites should be aware that inconsistencies between a filesystem and the DMF database can occur as a result of restoring migrated files. It is good practice to run dmaudit after every restore to correct those inconsistencies.
Some third-party backup packages can use a DMF library to perform backups in a DMF-aware manner. When the DMF-aware feature is enabled, these packages will not cause offline (OFL) files to be recalled during a backup. Dual-state (DUL) files will be dumped as if they were offline, which will reduce the time and space needed for a backup.
To use a DMF-aware third-party backup package to back up DMF filesystems, do the following:
Configure the backup package to include the DMF filesystems in the backups.
Enable the DMF-aware feature on those filesystems.
For more information about third-party backup packages, see Appendix F, “Third-Party Backup Package Configuration”.
DMF provides a script called do_predump.sh that is meant to be run just prior to a backup of the DMF filesystems using a third-party backup package. The do_predump.sh script does the following:
(Optional) Migrates all eligible files to dual-state
(Optional on a DCM system) Copies all eligible DCM files to dual-residency state
(Optional) Performs a database snapshot using dmsnap
To use do_predump.sh, do the following:
Configure the backup package to run do_predump.sh as the pre-backup command. For details, see the application-specific information in Appendix F, “Third-Party Backup Package Configuration”.
Define a task group in the dmf.conf file that is referred to by the dmdaemon object. In the supplied configurations, this task group is called dump_tasks.
The parameters do_predump.sh uses are as follows:
Because hard-deletions normally use the same expiry time as backups, run_hard_deletes.sh is normally run from the same task group. The DUMP_RETENTION parameter should match the retention policy of the backup package.
When using a third-party backup package, a typical dump_tasks stanza might look like the following:
define dump_tasks
TYPE taskgroup
RUN_TASK $ADMINDIR/run_hard_deletes.sh at 23:00
DUMP_RETENTION 4w # match backup package's policy
DUMP_MIGRATE_FIRST yes
DUMP_FLUSH_DCM_FIRST yes # only if running a DCM
DUMP_DATABASE_COPY /path/to/db_snapshot
enddef |
| Note: Backups and restores must be run from the DMF server.
Only root can perform backups and restores. Although some third-party backup packages normally allow unprivileged users to restore their own files, unprivileged users cannot restore their own files from a DMF filesystem because doing so requires root privilege to set the DMF attribute. Files backed up from a DMF filesystem should only be restored to a DMF filesystem. Otherwise, files that are offline (or treated as such) will not be recallable. |
You can use the xfsdump facility to backup XVM snapshots of a DMF-managed user filesystem. Note the following:
XVM snapshots of DMF-managed filesystems should not be added to the DMF config file.
You should not attempt to migrate or recall files from an XVM snapshot of a DMF filesystem.
For more information about XVM snapshots, see the XVM Volume Manager Administrator's Guide.
If you never want to wait for DMF to recall small files (say 4K or 64K), you should configure DMF to migrate all files, but to give a negative priority weight to small files when freeing them. For example:
AGE_WEIGHT -1 0 when space < 64k |
| Note: You could also configure DMF to never
migrate files smaller than a certain size:
However, although you would not have to wait for these files to come back online, backups may take a long time because of the amount of time needed to find these files on the disk and copy them to the backup image. This can become a problem for sites with high file counts. Therefore this method is not recommended. |
Be aware of the following:
For extremely small files (under a few hundred bytes), the disk space required for DMF database entries may exceed the size of the original file. For extremely large numbers of such files, this issue should be considered.
The space value in a when clause, as used above, refers to the space the file occupies on disk, which for sparse files may actually be smaller than the size of the file as shown by ls -l. The space value will be rounded upward to a multiple of the disk blocksize defined by mkfs(8); the default is 4096 bytes. For example, attempting to discriminate between files above or below 1000 bytes based on their space value is futile because all non-empty files will have a space value that is a multiple of (typically) 4096 bytes.
The result is that when the filesystem is filled up, DMF will never free the blocks for the small files. The files therefore are always dual-state, ready to be used. When the filesystem is backed up, the backup facility will recognize that they are dual-state and therefore back them up as offline. The net effect is that there is no file data in the backup at all for these small files, just their inodes, while keeping the small files always available.
If you use AGE_WEIGHT or SPACE_WEIGHT , DMF automatic migration will never free the space for these files but a user can still do a dmput -r on them to manually free the space.
However, if you do not want small files to migrate for any reason, then you must continue to use the SELECT_VG method despite the slower and larger backups.
If you are depending on an FTP MSP or a standard disk MSP to provide copies of your offline files in order to safeguard your data, then they should also be backed up.
If you use them just to hold extra copies for convenience or to speed data access, they need not be backed up. But you should consider how you would handle their loss. You would probably need to remove references to lost copies from the DMF daemon's database, using dmdadm, which can only be done when the daemon is not running.
A DCM differs from a conventional disk MSP in that it uses DMAPI to manage the files. It will not operate properly if the files are reloaded by a package that cannot also restore the DMAPI information associated with each file.
| Note: For simplicity, this discussion assumes that the site wishes to keep two copies of migrated files at all times to guard against media problems. (Keeping only one copy is considered risky, and keeping more than two copies is frequently impractical.) |
The DCM can have one of the following configurations:
A DCM may be holding an extra copy of files in addition to the normal number of tape-based copies. That is, after the initial migration has completed, there will be two tape copies and a third in the cache. The DCM may easily remove this third copy from the cache after some period of time, just leaving two tape-based copies. With this configuration, there is normally no need to back up the cache filesystem.
The initial migration could result in one cache copy and one on tape. Later on, when the cache has to be flushed, a second tape copy is written by the DCM before the cache-resident one is deleted. If the file is hard-deleted before the cache flushes, the second tape copy will never be made, thereby saving time and tape. The tradeoff is that cache-flushing is slower and the cache filesystem should be backed up; otherwise a tape media problem in conjunction with a disk failure would result in data loss. With this configuration, the cache filesystem should be backed up. Otherwise, the loss of the cache disk could leave you with just one copy of data on tape. This is considered to be risky.
For both configurations, any backups require the use of a DMF-aware backup package (as listed in Appendix F, “Third-Party Backup Package Configuration”) to back up the cache.
To use do_xfsdump.sh to backup any of these filesystems, simply include the pathname of its mountpoint in the DUMP_FILE_SYSTEMS parameter.
The following DMF private filesystems do not require a DMF-aware backup package:
HOME_DIR
JOURNAL_DIR
SPOOL_DIR
TMP_DIR
CACHE_DIR
MOVE_FS
Care should be taken when backing up the databases in HOME_DIR if there is any DMF activity going on while the backup is underway, due to the risk of making the copy of the database while it is being updated. A safe technique is to take a snapshot of the databases with dmsnap and back up the snapshot. The do_xfsdump.sh script does this automatically.
The journal files in JOURNAL_DIR should also be backed up if you keep older snapshots of the databases that may have to be reloaded and brought up-to-date with dmdbrecover. Preferably, journals should be backed up when DMF activity (apart from recalls) is minimal. The do_xfsdump parameters DUMP_MIGRATE_FIRST and DUMP_FLUSH_DCM_FIRST help achieve this by processing any queued up migration requests immediately before starting the backup.
SPOOL_DIR contains log files that may be of use for problem diagnosis, as well as history files controlling things like tape error recovery and reporting scripts. The loss of these files will not endanger user data, although DMF may act a little differently for a while until it reestablishes them. Back up SPOOL_DIR if you can.
The TMP_DIR, CACHE_DIR, and MOVE_FS filesystems do not require backup.
To use do_xfsdump.sh to backup any of these filesystems, simply include the pathnames of their mountpoints in the DUMP_FILE_SYSTEMS parameter.
The dmfill(8) command allows you to fill a restored filesystem to a specified capacity by recalling offline files. When you execute xfsdump -a, only inodes are dumped for all files that have been migrated (including dual-state files). Therefore, when the filesystem is restored, only the inodes are restored, not the data. You can use dmfill in conjunction with xfsrestore to restore a corrupted filesystem to a previously valid state. dmfill recalls migrated files in the reverse order of migration until the requested fill percentage is reached or until there are no more migrated files left to recall on this filesystem.
The basic strategy for recovering a lost or damaged DMF database is to recreate it by applying journal records to a backup copy of the database. For this reason it is essential that the database backup copies and journal files reside on a different physical device from the production databases; it is also highly desirable that these devices have different controllers and channels. The following sections discuss the database recovery strategy in more detail:
You configure tasks in the run_copy_databases.sh task in the dump_tasks object to automatically generate DMF database backups. See “Configuring Daemon Maintenance Tasks” in Chapter 2, for more information.
There are several databases in the DMF package. The daemon database consists of the following files:
HOME_DIR/ daemon_name/dbrec.dat
HOME_DIR/ daemon_name/dbrec.keys
HOME_DIR/ daemon_name/pathseg.dat
HOME_DIR/ daemon_name/pathseg.keys
The database definition file (in the same directory) that describes these files and their record structure is named dmd_db.dbd.
Each LS has two databases in the HOME_DIR/ ls_name directory:
The CAT database (files tpcrdm.dat, tpcrdm.key1.keys, and tpcrdm.key2.keys)
The VOL database (files tpvrdm.dat and tpvrdm.vsn.keys)
The database definition file (in the same directory) that describes these files and their record structure is named libsrv_db.dbd .
The DMF daemon and LS write journal file records for every database transaction. These files contain binary records that cannot be edited by normal methods and that must be applied to an existing database with the dmdbrecover(8) command. The following procedure explains how to recover the daemon database.
 | Warning: If you are running on multiple LSs, always ensure that you have the correct journals restored in the correct directories. Recovering a database with incorrect journals can cause irrecoverable problems. |
Procedure 7-1. Recovering the Databases
If you lose a database through disk spindle failure or through some form of external corruption, use the following procedure to recover it:
Stop DMF:
# /etc/init.d/dmf stop
If you have configured the run_copy_databases task, restore the files from the directory with the most recent copy of the databases that were in HOME_DIR to HOME_DIR/daemon or HOME_DIR/LS_NAME .
If you have not configured the run_copy_databases task, reload an old version of the daemon or LS database. Typically, these will be from the most recent dump tapes of your filesystem.
Ensure that the default JOURNAL_DIR/daemon_name (or JOURNAL_DIR/ls_name) directory contains all of the time-ordered journal files since the last update of the older database.
For the daemon, the files are named dmd_db. yyyymmdd[.hhmmss].
For the LS, the journal files are named libsrv_db. yyyymmdd[.hhmmss].
Use dmdbrecover to update the old database with the journal entries from journal files identified in step 4.
Example 7-1. Database Recovery Example
Suppose that the filesystem containing HOME_DIR was destroyed on February 1, 2004, and that your most recent backup copy of the daemon and LS databases is from January 28, 2004. To recover the database, you would do the following:
Stop DMF:
# /etc/init.d/dmfstop
Ensure that JOURNAL_DIR/daemon_name (or JOURNAL_DIR/ls_name ) contains the following journal files (one or more for each day):
JOURNAL_DIR/daemon_name
dmd_db.20040128.235959 dmd_db.20040129.235959 dmd_db.20040130.235959 dmd_db.20040131.235959 dmd_db.20040201
JOURNAL_DIR/ls_name
libsrv_db.20040128.235959 libsrv_db.20040129.235959 libsrv_db.20040130.235959 libsrv_db.20040131.235959 libsrv_db.20040201
Restore databases from January 28, to HOME_DIR /daemon_name and/or HOME_DIR/ls_name. The following files should be present:
HOME_DIR/daemon_name
dbrec.dat dbrec.keys pathseg.dat pathseg.keys
HOME_DIR/ls_name
tpcrdm.dat tpcrdm.key1.keys tpcrdm.key2.keys tpvrdm.dat tpcrdm.vsn.keys
Update the database files created in step 3 by using the following commands:
dmdbrecover -n daemon_name dmd_db dmdbrecover -n ls_name libsrv_db