This chapter provides an overview of the Data Migration Facility (DMF) and its administration. It discusses the following:
DMF is a hierarchical storage management system for SGI environments. DMF allows you to oversubscribe your online disk in a manner that is transparent to users; a user cannot determine, by using POSIX-compliant commands for filesystem enquiry, whether a file is online or offline. Only when special commands or command options are used can a file's actual residence be determined. This transparent migration is possible because DMF leaves inodes and directories intact within the native filesystem.
DMF automatically detects a drop below the filesystem free-space threshold and migrates selected data from expensive online disk to cheaper secondary storage, such as tapes. DMF automatically recalls the file data from offline media when the user accesses the file with normal operating system commands.
Figure 1-1 provides a conceptual overview of the data flow between applications and storage media.
DMF supports a range of storage management applications. In some environments, DMF is used strictly to manage highly stressed online disk resources. In other environments, it is also used as an organizational tool for safely managing large volumes of offline data. In all environments, DMF scales to the storage application and to the characteristics of the available storage devices.
DMF interoperates with the following:
Standard data export services such as Network File System (NFS) and File Transfer Protocol (FTP)
XFS filesystems
CXFS (clustered XFS) filesystems
Microsoft's Server Message Block (SMB), which is also known as the Common Internet File System (CIFS), as used by Samba when fileserving to Windows systems
By combining these services with DMF you can configure an SGI system as a high-performance file server.
DMF transports large volumes of data on behalf of many users. Because system interrupts and occasional storage device failures cannot be avoided, it is essential that the safety and integrity of data be verifiable. Therefore, DMF also provides tools necessary to validate your storage environment.
DMF has evolved around these customer requirements for scalability and the safety of data. As a filesystem migrator, DMF manages the capacity of online disk resources by transparently moving file data from disk to offline media. Most commonly, the secondary storage is tape, managed by OpenVault or the Tape Management Facility (TMF). However, the secondary storage can be any bulk-storage device accessible locally through NFS or FTP.
DMF includes the following software subsystems:
Server, which provides the full set of DMF functionality, including the DMF daemon, infrastructure, user and administrator commands, online manuals, and all man pages. This applies to IRIX systems and SGI ProPack for Linux 64-bit on SGI Altix 3000 systems.
Client, which provides the limited set of user commands, libraries, and a subset of the man pages. This applies to all supported operating systems (see “Hardware and Software Requirements”).
Only one of these subsystems can be installed on a given machine.
The DMF server subsystem runs on the following:
The DMF client subsystem supports nodes running the following operating systems:
SGI ProPack 2.2.1 or later for Linux
IRIX 6.5.2 or later
IBM AIX 5L version 5.1 ML 4 (64-bit kernel mode) APAR number IY42428
Red Hat Linux on supported IA32 platforms:
Red Hat Linux 7.3
Red Hat Linux 8.0
Red Hat Linux 9
Solaris 8 plus appropriate patch (see the release notes)
Solaris 9 plus appropriate patch
Windows 2000 Service Pack 3 or Service Pack 4
Windows XP Service Pack 1
The client-only user commands are as follows:
dmattr dmcopy dmget dmfind dmls dmput |
The DMF libdmfusr.so user library lets you write your own custom DMF user commands that use the same application program interface (API) as the above DMF user commands.
The most commonly used devices on IRIX and Linux systems are DLT 4000/7000, SCSI versions of IBM 3590, and STK TimberLine and RedWood drives. All STK robots, Grau, and IBM 3494 are supported.
For filesystems to be managed by DMF, they must be mounted on the DMF server in order to enable the Data Management (DMAPI) interface. Do one of the following for each platform:
IRIX:
Use the following command:
mount -o dmi
Declare parameter 4 in the fstab entry to be dmi
Linux:
Use the following command:
mount -o dmapi -o mtpt = mountpoint
Add dmapi, mtpt = mountpoint to the fourth field in the fstab entry
For more information, see the mount and fstab man pages.
The software licensing used by DMF servers is based on the FLEXlm product from Macrovision Corporation. You must have a separate license for each DMF server. (No licensing is required on the client host.)
Software keys are used to enforce licensing. DMF licenses apply to a single specific system. DMF license fees vary depending on the amount of data being managed.
When you order DMF, you will receive an entitlement ID and the URL to a key generation webpage. You must submit the system host ID, hostname, and entitlement ID when requesting your permanent DMF license. To determine the hostname and host ID number: launch dmmaint and select License Info.
To obtain your permanent DMF license, follow the instructions on the key generation page. After the required information is provided, a key will be generated and displayed on the webpage along with installation instructions. You can use the Update License button on the dmmaint GUI to install the license.
For more information about licensing, see the following webpage:
As a DMF administrator, you determine how disk space capacity is handled by selecting which filesystems DMF will manage and by specifying the volume of free space that will be maintained on each filesystem. Space management begins with a list of user files that are ranked according to criteria you define. File size and file age are among the most common ranking criteria.
File migration occurs in two stages:
Stage One: A file is copied (migrated) to secondary storage.
Stage Two: After the copy is secure, the file is eligible to have its data blocks released (this usually occurs only after a minimum space threshold is reached).
A file with all offline copies completed is called fully migrated. A file that is fully migrated but whose data blocks have not yet been released is called a dual-state file; its data exists both online and offline, simultaneously. After a file's data blocks have been released, the file is called an offline file.
You choose both the percentage of filesystem volume to migrate and the volume of free space. You can trigger file migration, or file owners can issue manual migration requests.
Offline media is the destination of all migrated data and is managed by daemon-like DMF components called the media-specific process (MSP) and the library server (LS).
 | Note: Linux systems do not support the tape MSP. Tape support on Linux is available only via the LS. |
The following types of MSPs are supported:
Disk
FTP
MSP
In addition, you can configure the disk MSP to run in disk cache manager (DCM) mode for n-tier capability. DMF can manage the disk MSP's storage filesystem and further migrate it to tape, thereby using a slower and less-expensive dedicated filesystem as a cache to improve the performance when recalling files.
The FTP MSP (dmftpmsp) uses the FTP protocol to transfer to and from disks of another system on the network. The disk MSP (dmdskmsp) is similar, but uses a filesystem mounted on the DMF server itself. This can be a local filesystem or a remote one mounted through NFS or similar filesharing protocol. If the disk MSP is configured as a DCM, the filesystem used by the DCM must be a local XFS filesystem.
Most commonly, the offline media is magnetic tape, usually in a tape library (also known as a robotic library or silo). DMF has two tape components: the tape MSP (dmatmsp) and the library server ( dmatls).
| Note: The tape MSP has limitations in some environments and has been superseded by the newer LS. |
Figure 1-2 summarizes the various MSP types.
Figure 1-3 shows the architecture of the LS and tape MSP.
There is one LS process (dmatls) per tape library, which maintains a pair of databases that all of its components share. The entities in the shaded boxes in Figure 1-3 are internal components of the dmatls process. Their functions are as follows:
In contrast to the LS process, each tape MSP has its own database of tape volumes it controls and the user files (at most one copy of each) that they contain. It is somewhat similar to the volume group previously described. Tape MSPs refer to a "device object," which controls a group of tape drives in a similar, but less flexible, way as the drive group previously described. A site can use any combination of the various MSPs or LSs; they are not mutually exclusive. Exception: The tape MSP is not supported on Linux systems.
The dmatrc and dmatwc. These processes are called the read- and write-children, and are created by MSPs and volume groups to perform the actual reading and writing of tapes. Unlike most of the other DMF processes that run indefinitely, these processes are created as needed, and are terminated when their specific work has been completed.
Media transports and robotic automounters are also key components of all DMF installations. Generally, DMF can be used with any transport and automounter that is supported by either OpenVault or TMF. Additionally, DMF supports absolute block positioning, a media transport capability that allows rapid positioning to an absolute block address on the tape volume. When this capability is provided by the transport, positioning speed is often three times faster than that obtained when reading the volume to the specified position. For details, see “Hardware and Software Requirements”.
DMF provides capabilities ensure the integrity of offline data. For example, you can have multiple MSPs or volume groups with each managing its own pool of media volumes. Therefore, you can configure DMF to copy filesystem data to multiple offline locations.
DMF stores data that originates in a CXFS or XFS filesystem. (You can also convert other file servers to IRIX or Linux file servers running DMF.) Each object stored corresponds to a file in the native filesystem. When a user deletes a file, the inode for that file is removed from the filesystem. Deleting a file that has been migrated begins the process of invalidating the offline image of that file. In the tape MSP or LS, this eventually creates a gap in the migration medium. To ensure effective use of media, the LS provides a mechanism for reclaiming space lost to invalid data. This process is called volume merging.
Much of the work done by DMF involves transaction processing that is recorded in databases. The DMF database provides for full transaction journaling and employs two-phase commit technology. The combination of these two features ensures that DMF applies only whole transactions to its database. Additionally, in the event of an unscheduled system interrupt, it is always possible to replay the database journals in order to restore consistency between the DMF databases and the filesystem. DMF utilities also allow you to verify the general integrity of the DMF databases themselves.
DMF consists of the DMF daemon and one or more MSPs or LSs. The DMF daemon accepts requests from the DMF administrator or from users to migrate filesystem data, and communicates with the operating system kernel to maintain a file's migration state in that file's inode.
The DMF daemon is responsible for dispensing a unique bit file identifier (BFID) for each file that is migrated. The daemon also determines the destination of migration data and forms requests to the appropriate MSP/LS to make offline copies.
The MSP/LS accepts requests from the DMF daemon. For outbound data, the MSP/LS accrues requests until the volume of data justifies a volume mount. Requests for data retrieval are satisfied as they arrive. When multiple retrieval requests involve the same volume, all file data is retrieved in a single pass across the volume.
DMF uses the kernel interface defined by the Data Management Interface Group (DMIG). DMAPI is also supported by X/Open, where it is evolving as the XDSM standard.
Figure 1-4 illustrates the DMF architecture.
DMF has evolved in production-oriented, customer environments. It is designed to make full use of parallel and asynchronous operations, and to consume minimal system overhead while it executes, even in busy environments in which files are constantly moving online or offline. Exceptions to this rule will occasionally occur during infrequent maintenance operations when a full scan of filesystems or databases is performed.
The capacity of DMF is measured in several ways, as follows:
Total number of files. The DMF daemon database addressing limits the size of the daemon database to approximately 4 billion entries. There is one database entry for each copy of a file that DMF manages. Therefore, if a site makes two copies of each DMF-managed file, DMF can manage approximately 2 billion files.
Total volume of data. Capacity in data volume is limited only by the physical environment and the density of media.
Total volume of data moved between online and offline media. The number of tape drives configured for DMF, the number of tape channels, and the number of disk channels all figure highly in the effective bandwidth. In general, DMF provides full-channel performance to both tape and disk.
Storage capacity. DMF can support any file that can be created on the CXFS or XFS filesystem being managed.
DMF can be configured for a variety of environments including the following:
Support of batch and interactive processing in a general-purpose environment with limited disk space
Dedicated file servers
Lights-out operations
DMF manages two primary resources: pools of offline media and free space on native filesystems.
As a DMF administrator, you must characterize and determine the size of the environment in which DMF will run. You should plan for a certain capacity, both in the number of files and in the volume of data. You should also estimate the rate at which you will be moving data between the DMF store and the native filesystem. You should select autoloaders and media transports that are suitable for the data volume and delivery rates you anticipate.
Beyond initial planning and setup, DMF requires that you perform recurring administrative duties. DMF allows you to configure tasks that automate these duties. A task is a cron-like process initiated on a time schedule you determine. Configuration tasks are defined with configuration file parameters. The tasks are described in detail in “Configuring Daemon Maintenance Tasks” in Chapter 2, and “Configuring Maintenance Tasks for Tape MSP and LS” in Chapter 2.
DMF requires administrative duties to be performed in the following areas:
File ranking. You must decide which files are most important as migration candidates. When DMF migrates and frees files, it chooses files based on criteria you chose. The ordered list of files is called the DMF candidate list. Whenever DMF responds to a critical space threshold, it builds a new migration candidate list for the filesystem that reached the threshold. See “Generating the Candidate List” in Chapter 3.
Automated space management. You must decide how much free space to maintain on each managed filesystem. DMF has the ability to monitor filesystem capacity and to initiate file migration and the freeing of space when free space falls below the prescribed thresholds. See Chapter 3, “Automated Space Management”.
Offline data management. DMF offers the ability to migrate data to multiple offline locations. Each location is managed by a separate MSP or volume group and is usually constrained to a specific type of medium.
Complex strategies are possible when using multiple MSPs, LSs, or volume groups. For example, short files can be migrated to a device with rapid mount times, while long files can be routed to a device with extremely high density.
You can describe criteria for MSP or volume group selection. When setting up a tape MSP or volume group, you assign a pool of tapes for use by that MSP/volume group. The dmvoladm(8) utility provides management of the tape MSP/LS media pools.
You can configure DMF to automatically merge tapes that are becoming sparse--that is, full of data that has been deleted by the owner. With this configuration (using the run_merge_tapes.sh task), the media pool is merged on a regular basis in order to reclaim unusable space.
Recording media eventually becomes unreliable. Sometimes, media transports become misaligned so that a volume written on one cannot be read from another. Two utilities are provided that support management of failing media. The dmatsnf(8) utility is used to scan a DMF volume for flaws, and dmatread (8) is used for recovering data. Additionally, the volume merge process built into the MSP/LS is capable of effectively recovering data from failed media.
Chapter 6, “Media-Specific Processes and Library Servers”, provides more information on administration.
Integrity and reliability. Integrity of data is a central concern to the DMF administrator. You must understand and monitor processes in order to achieve the highest levels of data integrity, as follows:
Even though you are running DMF, you must still run backups because DMF moves only the data associated with files, not the file inodes or directories. You can configure DMF to automatically run backups of your DMF-managed filesystems.
The xfsdump and xfsrestore utilities understand when a file is fully migrated. The xfsdump utility has an option that allows for dumping only files that are not fully migrated. Files that are dual-state or offline have only their inodes backed up.
You can establish a policy of migrating 100% of DMF-managed filesystems, thereby leaving only a small volume of data that the dump utility must record. This practice can greatly increase the availability of the machine on which DMF is running because, generally, dump commands must be executed in a quiet environment.
You can configure the run_full_dump.sh and run_partial_dump.sh tasks to ensure that all files have been migrated. These tasks can be configured to run when the environment is quiet.
DMF databases record all information about stored data. The DMF databases must be synchronized with the filesystems DMF manages. Much of the work done by DMF ensures that the DMF databases remain aligned with the filesystems.
You can configure DMF to automatically examine the consistency and integrity of the DMF daemon and MSP/LS databases. You can configure DMF to periodically copy the databases to other devices on the system to protect them from loss (using the run_copy_databases.sh task). This task also uses the the dmdbcheck utility to ensure the integrity of the databases before saving them.
DMF uses journal files to record database transactions. Journals can be replayed in the event of an unscheduled system interrupt. You must ensure that journals are retained in a safe place until a full backup of the DMF databases can be performed.
You can configure the run_remove_logs.sh and run_remove_journals.sh tasks to automatically remove old logs and journals, which will prevent the DMF SPOOL_DIR directory from overflowing.
You can configure the run_hard_delete.sh task to automatically perform hard-deletes, which are described in “Recalling a Migrated File”.
While the administrator has access to a wide variety of commands for controlling DMF, the end user sees very little. Migrated files remain cataloged in their original directories and are accessed as if they were still on online disk. The only difference users might notice is a delay in access time.
However, commands are provided for file owners to affect the manual storing and retrieval of data. Users can do the following:
DMF regards files as being one of the following:
DMF does not migrate pipes, directories, or UNIX special files.
Like a regular file, a migrated file has an inode. Only an offline file requires the intervention of the DMF daemon to access its data; a dual-state file is accessed directly from the online disk copy.
The operating system informs the DMF daemon when a migrated file is modified. If anything is written to a migrated file, the offline copy is no longer valid, and the file becomes a regular file until it is migrated again.
A file is migrated when the automated space management controller dmfsmon(8) selects the file or when an owner requests that the file be migrated by using the dmput(1) command.
The DMF daemon keeps a record of all migrated files in its database. The key to each file is its bit file identifier (BFID). For each migrated file, the daemon assigns a BFID that is stored in the file's inode.
When the daemon receives a request to migrate a file, it adjusts the state of the file, ensures that the necessary MSPs or volume groups are active, and sends a request to the MSPs or volume groups. MSPs or volume groups then copy data to the offline storage media.
When the MSPs or volume groups have completed the offline copies, the daemon marks the file as fully migrated in its database and changes the file to dual-state. If the user specified the dmput -r option, or if dmfsmon requested that the file's space be released, the daemon releases the data blocks and changes the user file state to offline.
When a migrated file must be recalled, a request is made to the DMF daemon. The daemon selects an MSP or volume group from its internal list and sends that MSP/volume group a request to recall a copy of the file. If more than one MSP or volume group has a copy, the first one in the list is used. (The list is created from the configuration file.)
After a user has modified or removed a migrated file, its bit file identifier (BFID) is soft-deleted. A file is soft-deleted when it is logically deleted from the daemon database. This is accomplished by setting the delete date field in the database to the current date and time for each entry referring to the modified or removed file.
A file is hard-deleted when its BFID is physically removed from the DMF database. You can configure DMF to automatically perform hard-deletes. This is done using the run_hard_delete.sh task, which uses the dmhdelete(8) utility.
The soft-delete state allows for the possibility that the filesystem might be restored after the user has removed a file. When a filesystem is reloaded from a dump image, it is restored to a state at an earlier point in time. A file that had been migrated and then removed might become migrated again due to the restore operation. This can create serious problems if the database entries for the file have been physically deleted (hard-deleted). In this case, the user would receive an error when trying to open the file because the file cannot be retrieved.
Do not hard-delete a database entry until after you are sure that the corresponding files will never be restored. Hard-delete requests are sent to the relevant MSPs and volume groups so that copies of the file can be removed from media. For a tape MSP/volume group, this involves compression (or merging).
The following section provides definitions for administrator commands grouped by function.
The configuration file, /etc/dmf/dmf.conf , contains configuration objects and associated configuration parameters that control the way DMF operates. By changing the values associated with these objects and parameters, you can modify the behavior of DMF.
For information about editing the configuration file, see Chapter 2, “Configuring DMF”. The following man pages are related to the configuration file:
The DMF daemon, dmfdaemon(8), communicates with the kernel through a device driver and receives backup and recall requests from users through a socket. The daemon activates the appropriate MSPs and LSs for file migration and recall, maintaining communication with them through unnamed pipes. It also changes the state of inodes as they pass through each phase of the migration and recall process. In addition, dmfdaemon maintains a database containing entries for every migrated file on the system. Updates to database entries are logged in a journal file for recovery. See Chapter 4, “The DMF Daemon”, for a detailed description of the DMF daemon.
 | Caution: If used improperly, commands that make changes to the DMF database can cause data to be lost. |
The following administrator commands are related to dmfdaemon and the daemon database:
The following commands are associated with automated space management, which allows DMF to maintain a specified level of free space on a filesystem through automatic file migration:
See Chapter 3, “Automated Space Management”, for details.
The DMF tape MSP and LS maintain a database that contains volume (VOL) records and catalog (CAT) records. VOL records contain information about tape volumes, and CAT records contain information about offline copies of migrated files.
The disk and FTP MSPs allow the use of local or remote disk storage for storing migrated data. They use no special commands, utilities, or databases. For more information, see “Disk MSP” in Chapter 6, and “FTP MSP” in Chapter 6.
The following commands manage the CAT and VOL records for the tape MSP/LS:
Most data transfers to and from tape media are performed by components internal to the MSP/LS. However, there are also two utilities that can read tape MSP/LS volumes directly:
| Command | Description |
| dmatread(8) | Copies data directly from MSP/LS volumes to disk. |
| dmatsnf(8) | Audits and verifies the format of MSP/LS volumes. |
There are also tools that check for MSP database inconsistencies:
The following commands support the DCM:
The following utilities are also available: