Media-specfic processes (MSPs) and library servers (LSs) migrate files from one media to another. There are the following types of MSPs:
Tape MSP, which copies files from a disk to tape, or copies files from tape to disk.
File transfer protocol (FTP) MSP, which allows the DMF daemon to manage data by moving it to a remote machine.
Disk MSP, which migrates data to a directory that is accessible on the current systems and would be a cache disk if used in disk cache manager (DCM) mode.
LSs, like tape MSPs, copy files from a disk to a tape or from a tape to a disk. However, although the tape MSP and the LS have many characteristics in common, one of the primary differences is that while the tape MSP can manage at most one active copy of a migrated file, the LS can manage more than one copy. An LS is comprised of one or more volume groups. When a file is migrated from disk to tape, the selection policy can specify that it be copied to more than one volume group. Each volume group can manage at most one copy of a migrated file. Each volume group has an associated pool of tapes. Data from more than one volume group is never mixed on a tape.
This chapter discusses the following:
The tape MSP consists of the following programs:
dmatmsp
dmatwc
dmatrc
The DMF daemon executes dmatmsp as a child process. The MSP communicates with the daemon through a pair of unnamed pipes. In turn, dmatmsp executes dmatwc (the write child) to write data to tape and dmatrc (the read child) to read data from tape.
The LS consists of the following programs:
dmatls
dmatwc
dmatrc
The DMF daemon executes dmatls as a child process. In turn, dmatls executes dmatwc (the write child) to write data to tape and dmatrc (the read child) to read data from tape.
The dmatmsp or the dmatls program maintains the following types of records in its database:
The database is not a text file and cannot be updated by standard utility programs. Detailed information about the database and its associated utilities is provided in “CAT Database Records”, and “VOL Database Records”.
The tape MSP/LS provides a mechanism for copying active data from volumes that contain largely obsolete data to volumes that contain mostly active data. This process is referred to as volume merging or compression. Data on MSP/LS volumes becomes obsolete when users delete or modify their files. Volume merging can be configured to occur automatically (see “Configuring Maintenance Tasks for Tape MSP and LS” in Chapter 2). It can also be triggered by marking MSP/LS volumes as sparse with the dmvoladm(8) command.
The tape MSP/LS provides two utilities that read MSP/LS volumes directly:
Each instance of the tape MSP/LS needs three types of directories, one for each of the following:
Databases
Database journal files
Log files
Sites define the location of these directories by editing the base object configuration file parameters HOME_DIR, JOURNAL_DIR, and SPOOL_DIR, whose values are referred to as HOME_DIR, JOURNAL_DIR, and SPOOL_DIR in this document. A given instance of the tape MSP/LS creates a subdirectory named after itself in each of these three directories.
For example, if an instance of the tape MSP is called cart1, its database files reside in directory HOME_DIR/cart1. If another instance of the tape MSP is called cart2, its database files reside in HOME_DIR/cart2. If an instance of the LS is called cart3, its database files reside in HOME_DIR/cart3.
Similarly, MSP cart1 stores its journal files in directory JOURNAL_DIR/cart1 and its log files and other working files in SPOOL_DIR /cart1.
The tape MSP/LS takes full advantage of the capabilities of modern tape devices, including data compression and fast media positioning. To accommodate these capabilities and to provide recovery from surface or other media defects, dmatmsp and dmatls use a number of structural concepts built on top of traditional tape structure.
The components are as follows:
The block is the basic structural component of most tape technologies. It is the physical unit of I/O to and from the media. The optimal block size varies with the device type. For example, the default block size for a 3480/3490 device is 65,536 bytes.
A chunk is as much or as little of a user file as fits on the remainder of the tape (see Figure 6-1). Thus, every migrated file has at least one, and sometimes many, chunks. Such a concept is necessary because the capacity of a volume is unknown until written, both because of natural variation in the medium itself and because the effect of data compression varies with the data contents.
A zone is a logical block containing several physical blocks ending with a tape mark. A zone has a target size that is configurable by media type. The default zone target size is 50 MB.
The MSP or volume group writes chunks into the zone until one of three conditions occurs:
The zone size is exceeded
The MSP or volume group exhausts chunks to write
The end of tape is encountered
Thus, the actual zone size can vary from well below the target size to the entire tape volume. A zone never spans physical volumes.
The zone plays several roles:
The zone size is the amount of data that triggers dmatmsp/ dmatls to start a process to write files to tape.
The MSP/LS records the position of the beginning of each zone in its database so that it can use fast hardware positioning functions to return there to restore the chunks in that zone.
When a tape volume develops a defect, the data loss usually will be restricted to the zone.
Because getting the tape position and writing a tape mark can be very costly, the concept of a zone and the target size provides a way to control the trade offs between write performance, safety, and recall speed.
Figure 6-1 illustrates the way files are distributed over chunks, zones, and volumes, depending upon the file size. The tape with volume serial number (VSN) VOL001 has two zones and contains six files and part of a seventh. The tapes with VSNs VOL002 and VOL003 contain the rest of file g. Notice that on VOL001 file g is associated with chunk 7, while on the other two tapes it is associated with chunk 1. File g has three VSNs associated with it, and each tape associates the file with a chunk and zone unique to that tape.
Records in the tape catalog (CAT), tpcrdm, store the location of each file chunk in terms of its volume, zone, and chunk number. The key for these records is the file's bit file identifier (BFID).
You do not explicitly create CAT records in the database. They are created when files migrate.
The CAT portion of the MSP/LS database consists of the following files:
For MSPs, the database definition file (in the same directory) that describes these files and their record structure is named atmsp_db.dbd . For LSs, it is named libsrv_db.dbd.
All files are non-ASCII and cannot be maintained by standard utility programs. The dmcatadm command provides facilities to create, query, and modify CAT database records (see “dmcatadm Command”).
 | Note:: The ability to create or modify CAT database records with dmcatadm is provided primarily for testing purposes. In the normal course of operations, you would never use this capability. |
Records in the tape volume (VOL) portion of the MSP/LS database, tpvrdm, contain information about each volume that exists in the pool of tapes to be used by dmatmsp or dmatls. These records are indexed by the volume serial number (VSN) of each volume and contain such information as the volume's type, estimated capacity, label type, and a number of flags indicating the state of the volume. For LSs, the record also contains the volume's volume group or allocation group. Unlike the CAT records, you must create the VOL records in the database before using dmatmsp or dmatls for the first time.
The VOL portion of the MSP/LS database consists of two files:
For MSPs, the database definition file (in the same directory) that describes these files and their record structure is named atmsp_db.dbd . For LSs, it is named libsrv_db.dbd.
Both files contain binary data and require special maintenance utilities. The dmvoladm command, described in more detail in “dmvoladm Command”, provides facilities to create, query, and modify VOL records in the database. Additional database maintenance utilities are described in “Database Recovery” in Chapter 7.
| Note:: If you have more than one instance of the tape MSP or volume group, you must ensure that the volume sets for each are mutually exclusive. |
Each instance of dmatmsp or dmatls protects its database by recording every transaction in a journal file. For MSPs, journal file pathnames have the following format:
JOURNAL_DIR/msp_name/atmsp_db.yyyymmdd[.hhmmss] |
For LSs, journal file pathnames have the following format:
JOURNAL_DIR/ls_name/libsrv_db.yyyymmdd[.hhmmss] |
The MSP/LS creates journal files automatically.
Existing journal files are closed and new ones created in two circumstances:
When the first transaction after midnight occurs
When the journal file reaches the size defined by the JOURNAL_SIZE configuration parameter
When the first transaction after midnight occurs, the existing open journal file is closed and the suffix .235959 is appended to the current file name no matter what the time (or date) of closing. The closed file represents the last (or only) transaction log of the date yyyymmdd. A new journal file with the current date is then created.
When the journal file reaches JOURNAL_SIZE, the file is closed and the suffix .hhmmss is added to the name; hh, mm, and ss represent the hour, minute, and second of file closing. A new journal file with the same date but no time is then created.
For example, the following shows the contents of a JOURNAL_DIR/msp_name directory on 15 June 1998:
atmsp_db.19980527.235959 atmsp_db.19980606.235959 atmsp_db.19980528.235959 atmsp_db.19980607.235959 atmsp_db.19980529.235959 atmsp_db.19980608.235959 atmsp_db.19980530.235959 atmsp_db.19980609.235959 atmsp_db.19980531.235959 atmsp_db.19980610.235959 atmsp_db.19980601.235959 atmsp_db.19980611.235959 atmsp_db.19980602.235959 atmsp_db.19980612.235959 atmsp_db.19980603.235959 atmsp_db.19980613.235959 atmsp_db.19980604.235959 atmsp_db.19980614.235959 atmsp_db.19980605.235959 atmsp_db.19980615 |
For every date on which database transactions occurred, there will exist a file with that date and the suffix .235959, with the exception of an existing open journal file. Some dates may have additional files because the transaction log reached JOURNAL_SIZE at a specified time and the file was closed.
You can configure daemon_tasks parameters to remove old journal files (using the run_remove_journals.sh task and the JOURNAL_RETENTION parameter. For more information, see “Configuring Daemon Maintenance Tasks” in Chapter 2.
If an MSP/LS database becomes corrupt, recovery consists of applying the journal files to a backup copy of the database.
All DMF MSPs and LSs maintain log files named msplog. yyyymmdd in the MSP/LS spool directory which, by default, is SPOOL_DIR/ mspname. SPOOL_DIR is configured in the base object of the configuration file; mspname is the name of the MSP/LS in the daemon object of the configuration file; yyyymmdd is the current year, month, and day.
These log files are distinct from the logs maintained by the DMF daemon; however, some of the messages that occur in the daemon log are responses that the tape MSP/LS generates. The content of the log is controlled by the MESSAGE_LEVEL configuration parameter. For a description of the levels of logging available, see the dmf.conf(5) man page.
The msplog.yyyymmdd file is the primary log for the tape MSP/LS and contains most of the messages. This file is written by dmatmsp, dmatls, dmatrc, and dmatwc. A new msplog. yyyymmdd is created for each day.
This section describes informational statistics provided by the tape log files. These messages appear in the SPOOL_DIR /msp_name/msplog. yymmdd files. Timing information provided (such as MB transferred per second) should not be used as an accurate benchmark of actual data transfer rates. This information is provided for monitoring DMF and should only be used in comparison to similar data provided by DMF. Text in all uppercase references a parameter defined in the DMF configuration file. You can reference the comments in the sample configuration file or in the dmf.conf(5) man page for a more detailed definition of these parameters.
| Note: Because the MSP/LS will continue to create log files and journal files without limit, you must remove obsolete files periodically by configuring the run_remove_logs.sh and run_remove_journals.sh tasks in the configuration file, as described in “Configuring Daemon Maintenance Tasks” in Chapter 2. |
Example 6-1. Tape MSP Statistics Messages
The following is an example of tape MSP statistics messages taken from an msplog.yyyymmdd file. These messages are automatically and periodically issued by the MSP.
09:06:00-I 18979429-dmatmsp stats: children=3/3/0/4, btp=92274688/130740224/250609687, wc=2/4, cwc=0 09:06:00-I 18979429-dmatmsp stats: data put=722.469 mb, data recalled= 45.089 mb 09:06:00-I 18979429-dmatmsp stats: Put_File - 6 33 0 6 09:06:00-I 18979429-dmatmsp stats: Get_File - 1 13 0 0 09:06:00-I 18979429-dmatmsp stats: Delete_File - 0 1 0 0 09:06:00-I 18979429-dmatmsp stats: Cancel_Req - 0 6 0 0 09:06:00-I 18979429-dmatmsp stats: Flushall - 0 8 3 0 09:06:00-I 18979429-dmatmsp stats: Merge - 20 14 0 0 09:06:00-I 18979429-dmatmsp stats: mc=4, ms=0, mu=0, sm=1 |
The information provided by these entries is defined as follows:
children=3/3/0/4 represents the total child processes (3), the active child processes (3), the clean processes running (0), and the configured value of CHILD_MAXIMUM (4). Clean children are used when a dmatrc or dmatwc process dies without cleaning up.
btp=92274688/130740224/250609687 represents the bytes queued for putting (92274688), the threshold at which to start the next put child (130740224), and the bytes assigned to socket I/O (250609687)
wc=2/4 represents the active write child processes (2) and the configured value of MAX_PUT_CHILDREN (4)
cwc=0 represents the process ID of the current write child (that is, the write child that is accepting data to write). 0 represents none.
The next line gives the total amount of data put (722.469 megabytes) and recalled (45.089 megabytes).
The next six lines provide statistics for each type of MSP request. Statistics information is provided only for requests that have been issued since the MSP was started. These lines have the following format:
request_name active successful errors canceled |
active represents the number of requests not yet completed; successful represents the number of successfully completed requests; error represents the number of requests that completed with errors; canceled represents the number of canceled requests.
The last line provides the following information:
mc is the configured value for MERGE_CUTOFF, the cutoff to stop scheduling tapes for merging (4)
ms is the configured value for CACHE_SPACE, the merge cache space available (0 bytes)
mu is the merge cache space used (0 bytes)
sm is the number of socket merge children (1)
Example 6-2. LS Statistics Messages
The following is an example of LS statistics messages taken from an msplog.yyyymmdd file. These messages are automatically and periodically issued by the LS.
00:02:00-I 13902144-dmatls vg9a16.stats: children=1/0/0/7, btp=28098297/0/0, wc=0/7, cwc=0 00:02:00-I 13902144-dmatls vg9a17.stats: children=1/0/0/7, btp=59032803/0/0, wc=0/7, cwc=0 00:02:00-I 13902144-dmatls vg9a16.stats: data put=608.607 mb, data recalled=114.270 mb 00:02:00-I 13902144-dmatls vg9a17.stats: data put=1068.423 mb, data recalled=210.575 mb 00:02:01-I 13902144-dmatls vg9a16.stats: Put_File - 10 172 0 12 00:02:01-I 13902144-dmatls vg9a16.stats: Delete_File - 0 130 0 0 00:02:01-I 13902144-dmatls vg9a16.stats: Cancel_Req - 0 12 0 0 00:02:01-I 13902144-dmatls vg9a16.stats: Flushall - 0 2 0 0 00:02:01-I 13902144-dmatls vg9a16.stats: Merge - 45 25 0 16 00:02:01-I 13902144-dmatls vg9a17.stats: Put_File - 14 210 0 8 00:02:01-I 13902144-dmatls vg9a17.stats: Get_File - 0 1 0 0 00:02:01-I 13902144-dmatls vg9a17.stats: Delete_File - 0 178 0 0 00:02:01-I 13902144-dmatls vg9a17.stats: Cancel_Req - 0 8 0 0 00:02:01-I 13902144-dmatls vg9a17.stats: Flushall - 0 2 0 0 00:02:01-I 13902144-dmatls vg9a17.stats: Merge - 18 28 0 22 00:02:01-I 13902144-dmatls vg9a16.stats: mc=7, ms=500000000, mu=133107712, sm=0 00:02:01-I 13902144-dmatls vg9a17.stats: mc=7, ms=500000000, mu=73105408, sm=0 |
The fields in each message have the same meanings as in the MSP statistics messages (see Example 6-1), except that they are on a volume group basis.
The tape MSP/LS write child (dmatwc) and read child (dmatrc) also produce statistics messages in the MSP/LS log file. These messages contain timing statistics whose format changes from release to release, and they are not documented in this manual.
When users delete or modify their migrated files, the copy on tape becomes obsolete. Over time, some volumes will become entirely empty and can be reused. However, most volumes experience a gradual increase in the ratio of obsolete data to active data; such volumes are said to be sparsely populated or simply sparse. To reclaim the unused space on these volumes, DMF provides a volume merge facility, which copies the active data from several sparse volumes to a new volume, thus freeing the sparse volumes for reuse. Volume merging can be configured to occur automatically by using the run_merge_tapes.sh task (see “Configuring Maintenance Tasks for Tape MSP and LS” in Chapter 2).
Volume merging can also be done manually. dmatmsp/dmatls perform merge operations whenever sparse volumes and the necessary resources exist at the same time. Use the dmvoladm select directive to mark MSP or volume group volumes as sparse. (The select directive is described in “dmvoladm Command”.) Because the merge processing occurs simultaneously with other DMF activities, it is easiest to configure DMF to automatically perform merges at night or during other periods of relatively low activity.
The dmatmsp/dmatls utilities can perform volume-to-volume merging. Volume-to-volume merging is accomplished by moving data across a socket connection between the MSP/LS tape read-child and the MSP/LS tape write-child. The benefit of using a socket to transfer data between volumes is that you do not have to reserve disk space. The drawback to using a socket for data transfer is the cost of linking the process that performs the read with the process that performs the write.
In busy environments that have heavy contention for tape drives, the close coupling between the socket's tape reader and tape writer can be costly, especially when short files are being transferred. For large files, the overhead and possible delays in waiting for both tapes to be mounted is small compared to the benefit of rapid transfer and zero impact on free disk space. For this reason, you can move small files through a disk cache and big files through a socket. This process is mediated by the following configuration parameters:
Using a small amount of disk space to hold small chunks can have a significant impact on the total time required to perform merges. The default configuration options are set to move 100% of merge data across sockets.
| Note: It is important to avoid volume merging on more than one MSP
or volume group simultaneously if they share a tape device. If you initiate
a merge process on more than one MSP or volume group on the same device
at the same time (either by entering the same time in the DMF configuration
file or by triggering the process manually), both processes will compete
for tape transports. When a limited number of tape transports are available,
a deadlock can occur. If you chose not to configure DMF to perform merges
automatically by configuring the run_tape_merge.sh
task, ensure that your cron jobs that automatically
initiate volume merging refrain from initiating a second merge process
until after all previously initiated merges are complete. You can accomplish
this by using the dmvoladm command within the
cron job to check for tapes that have the hsparse
flag, as shown in the following example for MSPs:
|
The dmcatadm(8) command provides maintenance services for CAT records in the MSP/LS database.
When you are inside the dmcatadm interface (that is, when you see the adm command_number > prompt), the command has a 30-minute timeout associated with it. If you do not enter a response within 30 minutes of the prompt having been displayed, the dmcatadm session terminates with a descriptive message. This behavior on all the database administrative commands limits the amount of time that an administrator can lock the daemon and MSP/LS databases from updates.
| Note:: Most of these facilities, especially the ability to create and modify CAT database records, are intended primarily for testing purposes. |
The dmcatadm command executes directives from stdin or from the command line when you use the -c option. All directives start with a directive name followed by one or more parameters. Parameters may be positional or keyword-value pairs, depending on the command. White space separates the directive name, keywords, and values.
The dmcatadm directives are as follows:
| Directive | Description |
| count | Displays the number of records that match the expression provided. |
| create | Creates a CAT record. |
| delete | Deletes an existing CAT record. |
| dump | Prints the specified CAT records to standard out in ASCII; each database field is separated by the pipe character (|). |
| help | Displays help. |
| list | Shows the fields of selected CAT records. You may specify which fields are shown. |
| load | Applies records to the MSP/LS database obtained from running the dump directive. |
| quit | Stops program execution after flushing any changed database records to disk. The abbreviation q and the string exit produce the same effect. |
| set | Specifies the fields to be displayed in subsequent list directives. |
| update | Modifies an existing CAT record. |
| verify | Verifies the MSP/LS database against the dmfdaemon database. |
The first parameter of most directives specifies the database records to manipulate, and the remaining parameters are keyword-value pairs.
The syntax for the dmcatadm directives is summarized as follows:
count selection [limit] create key field... delete selection [limit] dump selection [limit] help list selection [limit] [format] load filename quit (or q or exit) set [format] update selection [limit] to fields...For MSPs:verify selection [entries] [mspname] [limit]For LSs:verify selection [entries] [vgnames] [limit] |
The value for key may be a bit file identifier (BFID) designator in the form of a hexadecimal number.
The value for selection can be one of the following:
A key or range of keys in the form key [ -] [key]. key- specifies all records starting with key, and -key specifies all records up to key.
The keyword all
A period (.), which recalls the previous selection
An expression involving any of the above, field value comparisons, and, or, or parentheses.
A field value comparison may use < (less than), > (greater than), = (equal to), <= (less than or equal to), or >= (greater than or equal to) to compare a field keyword to an appropriate value.
The syntax for selection is as follows:
selection ::= or-expr or-expr ::= and-expr [ or or-expr ] and-expr ::= nested-expr [ and and-expr ] nested-expr ::= comparison | ( expression ) comparison ::= key-range | field-keyword op field-value op ::= < | > | = | >= | <= key-range ::= key [ - key] | [key - [key]] | key-macro key-macro ::= all field-keyword ::= name or abbreviation of the record field field-value ::= appropriate value for the field key ::= character representation of the record key |
Thus valid selections could be any of the following:
1510-1514 10000000000- -15138 all chunkoffset>0 chunknumber>0 and writeage<5d . recordorder data vsn=S07638 |
The field keywords listed below specify new values for fields. Some of the keywords are valid only if you also specify the -u option (unsafe mode), which allows changes to most database fields and supports the use of dmcatadm to repair database damage but must be used with care.
| Keyword | Description | |
| chunkdata (cd) | Specifies the actual number of bytes written to tape by the MSP or volume group for the chunk. In the case of sparse files, this field will be smaller than chunklength. This is valid only in unsafe (-u) mode. | |
| chunklength (cl) | The size of the chunk in bytes; an integer. This is valid only in unsafe (-u) mode. | |
| chunknumber (cn) | The ordinal of the chunk on its volume. For example, 1 if the chunk is the first chunk on the volume, 2 if it is the second, and so on. Valid only as part of selection. | |
| chunkoffset (co) | The byte offset within the file where the chunk begins; an integer. For example, the first chunk of a file has chunkoffset 0. If that first chunk is 1,000,000 bytes long, the second chunk would have chunkoffset 1000000. This is valid only in unsafe (-u) mode. | |
| chunkpos (cp) | The block offset within the zone where the chunk begins -- a hexadecimal integer. For example, the first chunk in a zone has chunkpos 1. A value of 0 means unknown. Valid only in unsafe (-u) mode in LS databases. | |
| filesize (fs) | The original file size in bytes, an integer. This is valid only in unsafe (-u) mode. | |
| flags (fl) | For future use. | |
| readage (ra) | The date and time when the chunk was last read; the same as readdate, except specified as age. | |
| readcount (rc) | The number of times the chunk has been recalled to disk; an integer. | |
| readdate (rd) | The date and time when the chunk was last read, an integer that reflects raw UNIX time. | |
| volgrp (vg) | The volume group name. This keyword is valid for LSs only. | |
| vsn (v) | The volume serial numbers; a list of one or more 6-character alphanumeric volume serial numbers separated by colons (:). | |
| writeage (wa) | The date and time when the chunk was written; the same as writedate, except specified as age . This is valid only in unsafe (-u) mode. | |
| writedate(wd) | The date and time when the chunk was written, an integer that reflects raw UNIX time. This is valid only in unsafe (-u ) mode. | |
| zoneblockid (zb) | Allows just the block ID portion of the zonepos to be displayed, returned, or changed. This is valid only in unsafe (-u) mode. | |
| zonenumber (zn) | Allows just the zone number portion of the zonepos to be displayed, returned, or changed. This is valid only in unsafe (-u) mode. | |
| zonepos (zp) | The physical address of the zone on the volume, expressed in the form integer/ hexinteger, designating a zone number and block ID. A value of zero is used for hexinteger if no block ID is known. integer is the same as zonenumber, and hexinteger is the same as zoneblockid. This is valid only in unsafe (-u) mode. |
The date field keywords (readdate and writedate) have a value of either now or raw UNIX time (seconds since January 1, 1970). These keywords display their value as raw UNIX time. The value comparison > used with the date keywords means newer than the value given. For example, >36000 is newer than 10AM on January 1, 1970, and >852081200 is newer than 10AM on January 1, 1997.
The age field keywords (readage and writeage) let you express time as age, a string in a form such as 8w12d7h16m20s (meaning 8 weeks, 12 days, 7 hours, 16 minutes, and 20 seconds old). The age keywords display their value as an integer followed by w, d, h, m, or s (weeks, days, hours, minutes, and seconds, respectively). The comparison > used with the age keywords means older than the value given (that is, >5d is older than 5 days).
The limit keywords limit the records acted upon:
| Keyword | Description | |
| recordlimit (rl) | Limits the number of records acted upon to the value that you specify; an integer. | |
| recordorder (ro) | Specifies the order that records are scanned; may be key, vsn, or data. key specifies that records are scanned in ascending order of the chunk key. vsn specifies that records are scanned in ascending order of the chunk VSN. data specifies that records are scanned in the order in which they are stored in the database, which is fastest but essentially unordered. |
The following keywords specify files of daemon database entries:
| Keyword | Description | |
| entries (e) | Specifies a file of daemon database entries; a string. | |
| mspname (mn) | Specifies the name of the MSP associated with the record; a string. | |
| vgnames (vn) | Specifies the names of the volume groups associated with the record; a quoted, space-separated list of names. |
The format keyword selects a format to use for the display. If, for example, you want to display fields in a different order than the default or want to include fields that are not included in the default display, you specify them with the format keyword. Values for format can be default, keyword, or a list of field keywords enclosed in quotation marks.
For any field that takes a byte count, you may append the letter k, m, or g (in either uppercase or lowercase) to the integer to indicate that the value is to be multiplied by one thousand, one million, or one billion, respectively.
For information about the role of the dmcatadm (8) command in database recovery, see “Database Recovery” in Chapter 7.
Example 6-3. dmcatadm list directive
The following is sample output from the dmcatadm list directive. The file with key 3273d5420001e244 has two chunks because it spans two physical tape volumes; the first chunk contains bytes 0 through 24821759, and the second chunk bytes 24821760 (the CHUNK OFFSET) to the end of the file.
adm 3>list 3273d5420001e242- recordlimit 10
WRITE CHUNK CHUNK CHUNK
KEY AGE OFFSET LENGTH NUM VSN
-----------------------------------------------------------------------
3273d5420001e242 61d 0 77863935 13 S12940
3273d5420001e244 61d 0 24821760 168 S12936
3273d5420001e244 61d 24821760 23543808 1 S12945
3273d5420001e245 61d 0 51019776 2 S12945
3273d5420001e246 61d 0 45629440 59 S12938
3273d5420001e247 61d 0 35586048 60 S12938
3273d5420001e248 61d 0 9568256 3 S12944
3273d5420001e249 61d 0 14221312 4 S12944
3273d5420001e24a 61d 0 458752 5 S12944
3273d5420001e24b 61d 0 14155776 6 S12944 |
The following is sample output from the dmcatadm list directive for an LS. The file with key 3b4b28f2000000000000ae80 has 2 chunks because it was migrated to two different volume groups within this LS. The output from the dmvoladm list directive that follows shows that VSN 000700 is assigned to the volume group named vg8a15, and VSN 00727 is assigned to the volume group named vg8a05 .
# dmcatadm -m ls1
adm 1>list 3b4b28f2000000000000ae80- recordlimit 4
WRITE CHUNK CHUNK CHUNK
KEY AGE OFFSET LENGTH NUM VSN
--------------------------------------------------------------------
3b4b28f2000000000000ae80 1d 0 2305938 120 000700
3b4b28f2000000000000ae80 4d 0 2305938 32 000727
3b4b28f2000000000000ae82 1d 0 234277 247 003171
3b4b28f2000000000000ae82 1d 0 234277 186 003176
adm 2> quit
# dmvoladm -m ls1
adm 1>list vsn=000700
DATA EOT EOT WR/FR
VSN VOLGRP LB DATA LEFT WRITTEN CHUNK ZONE HFLAGS AGE
-------------------------------------------------------------------------------
000700 vg8a15 al 150.280473 233.786093 123 9 ------u-- 1d
adm 2>list vsn=000727
DATA EOT EOT WR/FR
VSN VOLGRP LB DATA LEFT WRITTEN CHUNK ZONE HFLAGS AGE
-------------------------------------------------------------------------------
000727 vg8a05 al 159.107337 200.443980 102 6 --------- 1d
|
The dmvoladm(8) command provides maintenance services for VOL records in the MSP/LS database. In addition to the creation and modification of volume records, dmvoladm has an important role in the recovery of VOL records from a database checkpoint and is the mechanism that triggers volume merge activity.
When you are inside the dmvoladm interface (that is, when you see the adm command_number > prompt), the command has a 30-minute timeout associated with it. If you do not enter a response within 30 minutes of the prompt having been displayed, the dmvoladm session terminates with a descriptive message. This behavior on all the database administrative commands limits the amount of time that an administrator can lock the daemon and MSP/LS databases from updates.
The dmvoladm command executes directives from stdin or from the command line when you use the -c option. The syntax is the same as for dmcatadm: a directive name followed by parameters or paired keywords and values, all separated by white space.
The syntax for the dmvoladm directives is summarized as follows:
count selection create vsnlist [field...] delete selection [limit...] dump selection [limit...] help list selection [limit...] [format] load filename quit (or q, or exit) repair selection select selection [limit...] set [format] update selection [limit...] to field verify selection |
The value for vsnlist may be a single 6-character volume serial number (VSN) or a range of VSNs separated by the hyphen (-) character. A VSN string may consists entirely of letters, entirely of digits, or may be a series of letters followed by digits. In a range of VSNs, the first must be lexically less than the second.
The value for selection may be one of the following:
A vsnlist or range of VSNs in the form vsn[-vsn ]. vsn- specifies all records starting with vsn, and -vsn specifies all records up to vsn.
A period (.), which recalls the previous selection
The name of one of the flags in the keyword list that follows in this section.
One of the words all, used , empty, or partial or any of the hflags, whose meanings are as follows:
Flag Description
all Specifies all volumes in the database
empty Specifies all volumes in which data written is 0
partial Specifies used volumes in which hfull is off
used Specifies all volumes in which data written is not 0
The syntax for selection is as follows:
selection ::= or-expr or-expr ::= and-expr [ or or-expr ] and-expr ::= nested-expr [ and and-expr ] nested-expr ::= comparison | ( expression ) comparison ::= vsnlist | field-keyword op field-value op ::= < | > | = | >= | <= key-range ::= vsn [ - vsn] | [vsn - [vsn]] | key-macro key-macro ::= all | empty | used | partial | flag(s) field-keyword ::= name or abbreviation of the record field field-value ::= appropriate value for the field vsnlist ::= character representation of the volume serial number |
Thus valid selections could be any of the following:
tape01-tape02 tape50- -vsn900 all herr or hbadmnt used and hfull=off datawritten>0 and hfull=off . and eotchunk>3000 and (eotchunk<3500 or hfree=on) hfull and threshold<30 |
The field keywords specify new values for fields:
| Keyword | Description | |
| blocksize (bs) | Specifies the data block size in bytes when the tape was first written; an integer. The default is 65,536. This keyword is used only when mounting tapes with existing valid data. When an empty tape is first written, the MSP or volume group uses the default value for the tape type, unless it is overridden by a value in the BLOCK_SIZE parameter for the tape device in the DMF configuration file. This is valid only in unsafe (-u) mode. | |
| chunksleft (cl) | Specifies the number of active chunks on the volume; an integer. This is valid only in unsafe (-u) mode. | |
| dataleft (dl) | Specifies the number of bytes of active data on the volume. You specify this number as an integer, but for readability purposes it is displayed in megabytes (MB). This is valid only in unsafe ( -u) mode. | |
| datawritten (dw) | Specifies the maximum number of bytes ever written to the volume. You specify this number as an integer, but for readability purposes it is displayed in MB. This is valid only in unsafe ( -u) mode. | |
| eotblockid (eb) | Specifies the blockid of the chunk containing the end-of-tape marker; a hexinteger. This is valid only in unsafe (-u) mode. | |
| eotchunk (ec) | Specifies the number of the chunk containing the end-of-tape marker; an integer. This is valid only in unsafe (-u) mode. | |
| eotpos (ep) | Specifies the absolute position of the end-of-tape marker zone in the form integer/ hexinteger, designating a zone number and block ID. A value of zero is used for hexinteger if no block ID is known. integer the same as eotzone, and hexinteger is the same as eotblockid. This is valid only in unsafe ( -u) mode. | |
| eotzone (ez) | Specifies the number of the zone containing the end-of-tape marker; an integer. This is valid only in unsafe (-u) mode. | |
| label (lb) | Specifies the label type: al for ANSI standard labels; sl for IBM standard labels; or nl for nonlabeled volumes. The default is al. | |
| tapesize (ts) | Specifies the estimated capacity in bytes; an integer. The default is 215 MB. | |
| threshold (th) | Specifies the ratio of dataleft to datawritten as a percentage. This field cannot be displayed or updated. | |
| upage (ua) | (Display only.) Specifies the date and time of the last update to the volume's database record. The same as for update , except that it is expressed as age. This is valid only in unsafe (-u) mode. | |
| update (ud) | (Display only.) Specifies the date and time of the last update to the volume's database record, expressed as an integer that reflects raw UNIX time. This is valid only in unsafe (-u) mode. | |
| version (v) | Specifies the DMF tape format version, an integer. This is valid only in unsafe (-u) mode. | |
| volgrp (vg) | Specifies the volume group or allocation group. This field is valid only for LS databases. | |
| wfage (wa) | Specifies the date and time that the volume was written to or freed for reuse. The same as for wfdate, except that it is expressed as age. This is valid only in unsafe (-u) mode. | |
| wfdate (wd) | Specifies the date and time that the volume was written to or freed for reuse, expressed as an integer that reflects raw UNIX time. This is valid only in unsafe (-u) mode. |
The date field keywords (update and wfdate) have a value of either now or raw UNIX time (seconds since January 1, 1970). These keywords display their value as raw UNIX time. The value comparison > used with the date keywords means newer than the value given. For example, >36000 is newer than 10AM on January 1, 1970, and >852081200 is newer than 10AM on January 1, 1997.
The age field keywords (upage and wfage ) let you express time as age, a string in a form such as 8w12d7h16m20s (meaning 8 weeks, 12 days, 7 hours, 16 minutes, and 20 seconds old). The age keywords display their value as an integer followed by w, d, h, m, or s (weeks, days, hours, minutes, and seconds, respectively). The comparison > used with the age keywords means older than the value given (that is, >5d is older than 5 days).
The limit keywords restrict the volumes acted upon:
| Keyword | Description | |
| datalimit (no abbreviation) | Specifies a value in bytes. The directive stops when the sum of dataleft of the volumes processed so far exceeds this value. | |
| recordlimit (rl) | Specifies a number of records; an integer. The directive stops when the number of volumes processed equals this value. | |
| recordorder (ro) | Specifies the order that records are scanned; may be either data or vsn. vsn specifies that the records are scanned in ascending order of the chunk VSN. data specifies that the records are scanned in the order in which they are found in the database, which is fastest but essentially unordered. |
The format keyword selects a format to use for the display. If, for example, you want to display fields in a different order than the default or want to include fields that are not included in the default display, you specify them with the format keyword. Values for format can be default, keyword, or a list of field keywords enclosed in quotation marks.
The flag keywords change the settings of hold flags:
| Keyword | Description | |
| hbadmnt (hb) | Indicates that the volume could not be mounted. If the problem causing the mount to fail is transient, the MSP will clear the flag the next time it attempts to mount the tape and succeeds. Typically this flag indicates a permanent condition that should be investigated and corrected. It is displayed as --------b. Currently, this flag is used only by MSPs. | |
| herr (he) | Indicates that an I/O error has occurred on the volume; displayed as e--------. Currently, this flag is used only by MSPs. | |
| hflags (no abbreviation) | (Display only.) Shows the complete set of hold flags as a 9-character string. Each flag has a specific position and alphabetic value. If the flag is off, a dash (-) is displayed in its position; if the flag is on, the alphabetic character is displayed in that position. | |
| hfree (no abbreviation) | Indicates that the volume has no active data and is available for reuse after HFREE_TIME has expired, displayed as -f-------. See the dmf.conf(5) man page for information about the HFREE_TIME configuration parameter. This is valid only in unsafe (-u) mode. | |
| hfull (hu) | Indicates that the volume cannot hold any more data; displayed as ------u--. For LSs, this flag can also be set if error conditions indicate that no more data should be written to it. | |
| hlock (hl) | Indicates that the tape cannot be used for either input or output. This is a transient condition; the flag will be cleared by the LS after a period of time has passed. Currently used only by the LS. Displayed as ----l----. | |
| hoa (ho) | Indicates that the volume is not to be used for either input or output, displayed as --o------. | |
| hro (hr) | Indicates that the volume is read-only, displayed as ---r-----; this inhibits the MSP from using the volume for output. | |
| hrsv (h*) | Currently unused (reserved); displayed as ----*---- . This is valid only in unsafe (-u) mode. | |
| hsparse (hs) | Indicates that the volume is considered sparse and thus a candidate for a volume merge operation, displayed as -------s- . |
For any field that takes a byte count, you may append the letter k, m, or g (in either uppercase or lowercase) to the integer to indicate that the value is to be multiplied by one thousand, one million, or one billion, respectively.
For information about the role of the dmvoladm command in database recovery, see “Database Recovery” in Chapter 7. For details about dmvoladm syntax, see the man page.
Example 6-4. dmvoladm list directives
The following example illustrates the default format for the list directive when using an MSP. The column marked HFLAGS uses a format similar to the ls -l command in that each letter has an assigned position and its presence indicates that the flag is “on”. The positions spell the string efor*lmusb , representing herr, hfree, hoa, hro, hrsv, hlock, hfull, hsparse, and hbadmnt, respectively.
adm 1>list S03232-S03254
DATA EOT WR/FR
VSN LB DATA LEFT WRITTEN CHUNK HFLAGS AGE
------------------------------------------------------------------
S03232 sl 185.105446 400.000000 10 ------u-- 997d
S03233 sl 177.057792 400.000000 2 ------u-- 495d
S03234 sl 253.573185 400.000000 598 ------u-- 906d
S03235 sl 170.963133 400.000000 18 ------u-- 497d
S03236 sl 194.456616 400.000000 38 ------u-- 915d
S03237 sl 250.533926 400.000000 92 ------u-- 803d
S03238 sl 0.000000 0.000000 1 --------- 114d
S03239 sl 0.000000 0.000000 1 --------- 114d
S03240 sl 0.000000 0.000000 1 --------- 114d
S03241 sl 252.162452 400.000000 325 ------u-- 369d
S03242 sl 166.635861 400.000000 81 ------u-- 631d
S03243 sl 202.468129 400.000000 26 ------u-- 400d
S03244 sl 0.000000 0.000000 1 --------- 96d
S03245 sl 383.047890 400.000000 26 ------u-- 212d
S03246 sl 288.721920 400.000000 5 ------u-- 687d
S03247 sl 261.498716 400.000000 186 ------u-- 691d
S03248 sl 255.480486 400.000000 17 ------u-- 288d
S03249 sl 319.990661 400.000000 526 ------u-- 253d
S03250 sl 0.000000 0.000000 1 --------- 114d
S03251 sl 241.785669 400.000000 533 ------u-- 327d
S03252 sl 1223.947545 1223.947545 157 ------u-- 44d
S03253 sl 386.038988 400.000000 636 ------u-- 136d
S03254 sl 170.798521 400.000000 38 ------u-- 228d |
The following example illustrates using the list command to show only volumes meeting some criterion (in this case, those having their hfree flag set):
adm: 1>list hfree
DATA EOT WR/FR
VSN LB DATA LEFT WRITTEN CHUNK HFLAGS AGE
-------------------------------------------------------------------
003249 sl 0.000000 115.000000 9 -f-r------ 3h
003250 sl 0.000000 115.000000 9 -f-r------ 3h
003251 sl 0.000000 115.000000 10 -f-r------ 3h
003252 sl 0.000000 115.000000 11 -f-r------ 3h
003255 sl 0.000000 115.000000 15 -f-r------ 3h
003258 sl 0.000000 115.000000 13 -f-r------ 3h
003263 sl 0.000000 115.000000 12 -f-r------ 3h
003264 sl 0.000000 0.000000 1 -f-------- 4h
003289 sl 0.000000 0.000000 1 -f-r------ 3h
003290 sl 0.000000 215.000000 29 -f-r------ 3h
003294 sl 0.000000 0.000000 1 -f-------- 4h |
The following example shows one way you can customize the list format to show only the fields that you want to see. The other way is to use the set format command with the same keyword list.
adm 21>list S03232-S03254 format "eotchunk eotzone eotpos"
EOT EOT
VSN CHUNK ZONE EOTPOS
-------------------------------
S03232 10 2 2/4294967295
S03233 2 2 2/4294967295
S03234 598 2 2/4294967295
S03235 18 2 2/4294967295
S03236 38 2 2/4294967295
S03237 92 2 2/4294967295
S03238 1 1 1/4294967295
S03239 1 1 1/4294967295
S03240 1 1 1/4294967295
S03241 325 2 2/4294967295
S03242 81 2 2/4294967295
S03243 26 2 2/4294967295
S03244 1 1 1/4294967295
S03245 26 2 2/4294967295
S03246 5 2 2/4294967295
S03247 186 2 2/4294967295
S03248 17 2 2/4294967295
S03249 526 2 2/4294967295
S03250 1 1 1/4294967295
S03251 533 2 2/4294967295
S03252 157 17 17/2147483648
S03253 636 2 2/4294967295
S03254 38 2 2/4294967295 |
The following example gives a convenient way to show the several flag bits in a way different from their usual representation.
adm 23>list 003232-003254 format "herr hfree hfull hlock hoa hro"
herr hfree hfull hlock hoa hro
VSN
--------------------------------------
003232 off off on off off off
003233 off off off off off off
003234 off off off off off off
003235 off off off off off off
003236 off off on off off off
003237 off off on off off off
003238 off off on off off off
003239 off off on off off off
003240 off off off off off off
003241 off off on off off off
003242 off off on off off off
003243 off off off off off off
003244 off off off off off off
003245 off off on off off off
003246 off off off off off off
003247 off off on off off off
003248 off off on off off on
003249 off on off off off on
003250 off on off off off on
003251 off on off off off on
003252 off on off off off on
003253 off off on off off on
003254 off off on off off on |
The following example shows how to display only those tapes assigned to the volume group named vg9a00. This example is valid with LSs only.
adm 3>list vg=vg9a00
DATA EOT EOT WR/FR
VSN VOLGRP LB DATA LEFT WRITTEN CHUNK ZONE HFLAGS AGE
-------------------------------------------------------------------------------
003210 vg9a00 al 1.048576 1.048576 3 2 --------- 11d
003282 vg9a00 al 11.534336 11.534336 13 2 --------- 7d |
Use the dmatread(8) command to copy all or part of the data from a migrated file back to disk. You might want to do this if, for example, a user accidentally deleted a file and did not discover that the deletion had occurred until after the database entries had been removed by the hard delete procedure. Using backup copies of the databases from before the hard delete was performed, dmatread can restore the data to disk, assuming that the tape volume has not been reused in the meantime.
Example 6-5. Restoring Hard-deleted Files Using dmatread
To copy migrated files back to disk, perform the following steps:
Determine the BFID of the file you want to restore. You can use backup copies of dmdlog or your dbrec.dat files, or a restored dump copy of the deleted file's inode (and the dmattr command).
Using backup copies of the MSP/LS databases, use a dmatread(8) command similar to the following:
dmatread -p /a/dmbackup -B 342984C50000000000084155
342984C50000000000084155 is the BFID of the file to be restored, and /a/dmbackup is the directory containing the backup copies of the MSP databases. Your file will be restored to the current directory as B342984C50000000000084155
DMF does not know the original name of the file; you must manually move the restored data to the appropriate file.
If you have access to chunk and VSN information for the file to be restored, you can use the dmatread -c and -v options and avoid using backup copies of the MSP/LS database. In this case, dmatread will issue messages indicating that the chunk is not found in the current database, but it will continue with the request and restore the file as described in this example.
Use the dmatsnf(8) command to verify the readability of or to audit the contents of MSP/LS volumes. You may also generate text database records that can be applied to the MSP/LS databases (using the load directive in dmcatadm and dmvoladm, respectively), in order to add the contents of a volume to the MSP/LS database (although this is impractical for large numbers of volumes).
dmatsnf can be used to verify one or more tape volumes against the MSP/LS databases. It also can be used to generate journal entries, which can be added to the MSP/LS databases by using the load directive in dmvoladm and dmcatadm.
Use the verifymsp option of the dmaudit(8) command to check the consistency of the DMF daemon and MSP/LS databases after an MSP, LS, DMF, or system failure. This command captures the database files and compares the contents of the daemon database with each MSP/LS database. Any problems are reported to standard output, but no attempt is made to repair them.
This function can also be done directly using dmatvfy(8) after a snapshot has been taken.
The FTP MSP allows the DMF daemon to manage data by moving it to a remote machine. Data is moved to and from the remote machine with the protocol described in RFC 959 (FTP). The remote machine must understand this specific protocol.
| Note:: It is desirable that the remote machine run an operating system based on UNIX, so that the MSP can create subdirectories to organize the offline data. However, this is not a requirement. |
The FTP MSP does not need a private database to operate; all information necessary to retrieve offline files is kept in the daemon database, DMF configuration file, and login information file. The login information file contains configuration information, such as passwords, that must be kept private. As a safeguard, the MSP will not operate if the login information file is readable by anyone other than the system administrator.
The FTP MSP is always waiting for requests to arrive from the DMF daemon, but, to improve efficiency, it holds PUT and DELETE requests briefly and groups similar requests together into a single FTP session. No PUT request will be held longer than 60 seconds. No DELETE request will be held longer than 5 seconds. GET requests are not held. The MSP will stop holding requests if it has a large amount of work to do (more than 1024 individual files or 8 MB of data). The FTP MSP also limits the number of FTP sessions that can be active at once and the rate at which new sessions can be initiated.
After a request has been held for the appropriate amount of time, it enters a ready state. Processing usually begins immediately, but may be delayed if resources are not available.
The following limits affect the maximum number of requests that can be processed:
An administrator-controlled limit on the maximum number of concurrent FTP sessions per MSP (CHILD_MAXIMUM).
An administrator-controlled limit on the number of child processes that are guaranteed to be available for processing delete requests (GUARANTEED_DELETES).
An administrator-controlled limit on the number of child processes that are guaranteed to be available for processing dmget(1) requests (GUARANTEED_GETS).
A system-imposed limit of 85 FTP sessions in any 60-second period. This limit is seldom a concern because of the MSP's ability to transfer many files in one session. Because requests are grouped into batches only when resources are immediately available, GET requests (which are not normally held) are batched when resources are in short supply.
Requests are processed by forking off a child process. The parent process immediately resumes waiting for requests to arrive from the DMF daemon. The child process attempts to initiate an FTP session on the remote FTP server. If the remote machine has multiple Internet Protocol (IP) addresses, all of them are tried before giving up. If the child process cannot connect, it waits 5 minutes and tries again until it succeeds.
Once a connection is established, the child process provides any required user name, password, account, and default directory information to the remote FTP server. PUT, GET, or DELETE operations are then performed as requested by the DMF daemon. PUT, GET, or DELETE operations are not intermixed within a batch. If an individual request does not complete successfully, it does not necessarily cause other requests in the same batch to fail. Binary transfer mode is used for all data transfer.
The stored files are not verbatim copies of the user files. They are stored using the same format used to write tapes, and you can use MSP utilities such as dmatread and dmatsnf to access the data in them.
All DMF MSPs maintain log files named msplog.yyyymmdd in the MSP spool directory which, by default, is SPOOL_DIR/ mspname. SPOOL_DIR is configured in the base object of the configuration file; mspname is the name of the MSP in the daemon object of the configuration file; yyyymmdd is the current year, month, and day.
The activity log shows the arrival of new requests, the successful completion of requests, failed requests, creation and deletion of child processes, and all FTP transactions. Sensitive information (passwords and account information) does not appear in the activity log. In addition, the MSP lists the contents of its internal queues in its activity log if it is given an INTERRUPT signal.
| Note: Because the MSP will continue to create log files and journal files without limit, you must remove obsolete files periodically by configuring the run_remove_logs and run_remove_journals tasks in the configuration file, as described in “Configuring Daemon Maintenance Tasks” in Chapter 2. |
The MSP also recognizes and handles the following messages issued from the DMF daemon:
| Message | Description | |
| CANCEL | Issued when a previously requested action is no longer necessary, for example, when a file being migrated with a PUT request is removed. The MSP is able to cancel a request if it is being held or if it is waiting for resources. A request that has begun processing cannot be canceled and will run to normal completion. | |
| FINISH | Issued during normal shutdown. When the MSP receives a FINISH message, it finishes all requested operations as quickly as it can and then exits. | |
| FLUSHALL | Issued in response to the dmdidle (8) command. When the MSP receives a FLUSHALL message, it finishes all requested operations as quickly as it can. |
 | Caution: If the remote filesystem must be restored to a previous state, inconsistencies may arise: remote files that reappear after being deleted are never removed, and remote files that disappear unexpectedly result in data loss. There is presently no way to detect these inconsistencies. You should avoid situations that require the remote filesystem to be restored to a previous state. |
The disk MSP (dmdskmsp) migrates data into a directory that is accessed on the current system. It uses POSIX file interfaces to open, read, write, and close files. The directory may be NFS-mounted, unless the disk MSP is configured as a disk cache manager (see “Disk MSP and Disk Cache Manager (DCM)”). The data is read and written with root (UID 0) privileges. By default, dmdskmsp stores the data in DMF-blocked format, which allows the MSP to do the following:
Keep metadata with a file
Keep sparse files sparse when they are recalled
Verify that a file is intact on recall
The disk MSP does not need a private database to operate; all information necessary to retrieve offline files is kept in the daemon database and DMF configuration file.
The disk MSP may also be used as an import MSP. In this case, it only permits recalls and copies the data unchanged for a recall.
The disk MSP is always waiting for requests to arrive from the DMF daemon, but, to improve efficiency, it holds PUT and DELETE requests briefly and groups similar requests together into a single session. No PUT request will be held longer than 60 seconds. No DELETE request will be held longer than 5 seconds. GET requests are not held. The MSP will stop holding requests if it has a large amount of work to do (more than 1024 individual files or 8 MB of data).
After a request has been held for the appropriate amount of time, it enters a ready state. Processing usually begins immediately, but may be delayed if resources are not available.
The following limits affect the maximum number of requests that can be processed:
An administrator-controlled limit on the maximum number of concurrent operations per MSP (CHILD_MAXIMUM).
An administrator-controlled limit on the number of child processes that are guaranteed to be available for processing delete requests (GUARANTEED_DELETES).
An administrator-controlled limit on the number of child processes that are guaranteed to be available for processing dmget(1) requests (GUARANTEED_GETS).
Requests are processed by forking off a child process. The parent process immediately resumes waiting for requests to arrive from the DMF daemon.
PUT, GET, or DELETE operations are performed as requested by the DMF daemon. PUT, GET, or DELETE operations are not intermixed within a batch. If an individual request does not complete successfully, it does not necessarily cause other requests in the same batch to fail. Binary transfer mode is used for all data transfer.
The stored files are not verbatim copies of the user files. They are stored using the same format used to write tapes, and you can use MSP utilities such as dmatread and dmatsnf to access the data in them.
All DMF MSPs maintain log files named msplog.yyyymmdd in the MSP spool directory which, by default, is SPOOL_DIR/mspname . SPOOL_DIR is configured in the base object of the configuration file; mspname is the name of the MSP in the daemon object of the configuration file; yyyymmdd is the current year, month, and day).
The log file shows the arrival of new requests, the successful completion of requests, failed requests, and creation and deletion of child processes. In addition, the MSP lists the contents of its internal queues in its activity log if it is given an INTERRUPT signal.
| Note: Because the MSP will continue to create log files and journal files without limit, you must remove obsolete files periodically by configuring the run_remove_logs and run_remove_journals tasks in the configuration file, as described in “Configuring Daemon Maintenance Tasks” in Chapter 2. |
The disk cache manager (DCM) lets you configure the disk MSP to manage data on secondary disk storage, allowing you to further migrate the data to tape as needed. The DCM provides an automated method of using secondary (slower and less-expensive) disk as a fast-access DMF cache for files whose activity levels remain high, while also providing migration to tape for those files requiring less frequent access.
To allow the disk store that is managed by the disk MSP to function as a dynamically managed cache (as opposed to a static store), DCM creates and maintains a filesystem attribute on each file that is created in the MSP STORE_DIRECTORY. This attribute is used by the dmdskfree process to evaluate files for downward migration and for possible removal from the disk cache. For this reason, the DCM STORE_DIRECTORY must be a local XFS or CXFS filesystem mount point with DMAPI enabled.
The DCM supports dual-resident state, in which files reside in the cache and also in a lower volume group. This provides the access speed of a disk file, but allows that cache file to be quickly released without the need to first write it to tape. This is directly analogous to the concept of a dual-state file in the standard DMF-managed filesystem.
Automated movement in the opposite direction (from tape back to the cache) is not available. Any recalls of files that no longer have copies held in the cache will come directly from tape; they are not recalled via the cache and they can only be restored to the cache by an explicit dmmove(8) command.
DMF provides a mechanism to move copies of offline or dual-state files from one MSP or volume group to another. The dmmove(8) command takes a list of such files and moves them to a specified set of MSPs or volume groups. The list of MSPs or volume groups specified to the dmmove command indicates which MSPs or volume groups are to contain migrated copies of a file after the move process is completed. All other migrated copies are hard-deleted unless the dmmove -d option is used to select which copies are to be hard-deleted.
If a file's migrated state is offline, dmmove recalls the file to disk and then remigrates it to the specified MSPs or volume groups. (The one exception to this is that if a disk cache manager disk MSP copy exists, the file will be moved directly from that file copy.) When the migration process is complete, the online copy is removed. The file is recalled to a scratch filesystem that is specified by the MOVE_FS configuration parameter. If the file is dual-state, dmmove does not need to recall the file first, but instead uses the existing online copy.
The dmselect(8) command can be used to determine which files you want to move. dmselect selects files based on age, size, ownership, and MSP criteria. The output from the dmselect command can be used with the dmmove command. The dmmove command also accepts a list of pathnames as input.
See the man pages for dmselect and dmmove for all the possible options and further information.
You can convert IRIX DMF to Linux DMF and also convert Linux DMF to IRIX DMF . This section describes the necessary steps to convert an IRIX DMF to a Linux DMF.
DMF databases on IRIX machines cannot be copied to Linux machines because of binary incompatibility. Instead, they must be dumped to text on the IRIX machine, and the resulting text file must be loaded into the database on the Linux machine. DMF-managed filesystems, that is, filesystems containing user files that DMF has migrated, can be moved from an IRIX machine to a Linux machine.
It is assumed that sites converting DMF from an IRIX to a Linux machine (or vice versa) will obtain the help of SGI customer support; the following documentation is offered to familiarize you with the necessary steps. This procedure assumes the filesystems will be moved, and that this is done before the last step. It does not describe the steps required to move a filesystem.
Procedure 6-1. IRIX to Linux Conversion
Use dmaudit to verify that the DMF databases are valid. For more information, see the dmaudit (8) man page and the DMF Administrator's Guide for SGI InfiniteStorage and DMF Recovery and Troubleshooting Guide for SGI InfiniteStorage. To verify the databases that will actually be moved, you should change the filesystem migration levels in the dmf.conf file to none, run dmdidle, and then ascertain that all DMF activity has stopped before beginning this step. You should also use dmsnap to back up your databases.
Stop DMF on the IRIX system. If DMF is started again on the IRIX system during or after this procedure, the databases captured during step 3 might not reflect reality, and loss of data might result if you use them. To verify the consistency of the DMF databases, use the dmdbcheck(8) command.
Dump all of the DMF databases to text on the IRIX system. This should include the daemon database and the CAT and VOL databases for all tape MSPs and/or tape LSs. For more information, see the dmdump(8) man page.
Set up the /etc/dmf/dmf.conf file on the Linux system. Note that tape MSPs are not supported on Linux. If you do not have any tape MSPs, the conversion will be simpler if you name all of the FTP and DISK MSPs and the tape volume groups and LSs with the same names used on IRIX. This assumes that you do not already have MSPs or volume groups with these names on your Linux system.
If you do change the name of an MSP or volume group, you must convert the daemon database. For more information on how to perform this conversion, see the documentation in the dmconvertdaemon script.
If you do have a tape MSP, it must be converted to a volume group in an LS. Perhaps the easiest way to do this is to make the volume group name the same as the name of the MSP being converted. This method avoids making changes to the daemon database. For more information, see step 6. Use dmcheck to ensure that your new /etc/dmf/dmf.conf file is valid on the Linux system.
Copy the text versions of the databases that you created in step 3 to the Linux machine.
If you have a tape MSP on the IRIX system, run the dmconvertvol script to convert the text version of its VOL database to the format required for a volume group in an LS. Also run the dmconvertcat script to convert the text version of its CAT database to the format required for a volume group in an LS. If the name of the volume group is different from the name of the tape MSP being converted, run the dmconvertdaemon script to convert the text version of the DMF daemon's database. The dmconvertcat, dmconvertvol, and dmconvertdaemon scripts reside in the /usr/lib/dmf/support file. Man pages do not exist for them, each script contains documentation.
Load the database files from the text files on the Linux machine. Use dmdadm to load the daemon database file. Use dmcatadm to load the CAT database for each of the tape LSs. Use dmvoladm to load the VOL database for each of the tape LSs. If you are converting multiple tape MSPs to multiple volume groups within a single LS, you must load each of their databases. See Example 6-6.
Use dmdbcheck to check the consistency of databases on the Linux machine.
Make sure all DMF filesystems are resident on the Linux machine.
Start DMF on the Linux machine and run dmaudit.
Example 6-6. IRIX to Linux Conversion (Single Tape LS)
In the following example, the IRIX machine has a single tape LS, named ls1, and no tape MSPs. The example assumes that the /tmp/dmfdatabases directory has been created, is initially empty, and contains enough space to accommodate the text versions of the databases. The example also assumes that the HOME_DIR configuration parameter is set to /dmf/home on both systems. After completing steps 1 and 2 of Procedure 6-1, the daemon database and the LS databases are dumped to text, as follows:
$ dmdump -c /dmf/home/daemon > /tmp/dmfdatabases/daemon_txt $ dmdump /dmf/home/ls1/tpcrdm.dat > /tmp/dmfdatabases/ls1_cat_txt $ dmdump /dmf/home/ls1/tpvrdm.dat > /tmp/dmfdatabases/ls1_vol_txt |
Next, the files in /tmp/dmfdatabases on the IRIX system are copied to /tmp/dmftxtdb on the Linux system. After creating the DMF configuration file on the Linux system, the databases are loaded on the Linux system, as follows:
$ dmdadm -u -c "load /tmp/dmftxtdb/daemon_txt" $ dmcatadm -m ls1 -u -c "load /tmp/dmftxtdb/ls1_cat_txt" $ dmvoldadm -m ls1 -u -c "load /tmp/dmftxtdb/ls1_vol_txt" |
Now dmdbcheck is run to verify the consistency of the databases, as follows:
$ cd /dmf/home/daemon; dmdbcheck -a dmd_db $ cd /dmf/home/ls1; dmdbcheck -a libsrv_db |
Example 6-7. IRIX to Linux Conversion (Two Tape MSPs)
In the following example, the IRIX machine has two tape MSPs. Their names are msp1 and msp2. The example assumes that the /tmp/dmfdatabases directory has been created, is initially empty, and contains enough space to accommodate the text versions of the databases. This example also assumes that the HOME_DIR configuration parameter is set to /dmf/home on both systems. After completing steps 1 and 2 of Procedure 6-1, the daemon database and the tape MSP databases are dumped to text, as follows:
$ dmdump -c /dmf/home/daemon > /tmp/dmfdatabases/daemon_txt $ dmdump /dmf/home/msp1/tpcrdm.dat > /tmp/dmfdatabases/msp1_cat_txt $ dmdump /dmf/home/msp1/tpvrdm.dat > /tmp/dmfdatabases/msp1_vol_txt $ dmdump /dmf/home/msp2/tpcrdm.dat > /tmp/dmfdatabases/msp2_cat_txt $ dmdump /dmf/home/msp2/tpvrdm.dat > /tmp/dmfdatabases/msp2_vol_txt |
Next, we copy the files in /tmp/dmfdatabases on the IRIX system to /tmp/dmftxtdb on the Linux system. In this example, we assume that msp1 will be converted to a volume group by the name of vgpri in LS ls1. Similarly, msp2 will be converted to a volume group by the name of vgsec in LS ls1. Note that both of these volume groups will be in the same LS. To do this, you must ensure that the VSNs in each of the volume groups are unique.
After creating the /etc/dmf/dmf.conf file on the Linux system, the text versions of the database files are converted. First, the text versions of msp1's files are converted to vgpri:
$ dmconvertcat /tmp/dmftxtdb/msp1_cat_txt > /tmp/dmftxtdb/vgpri_cat_txt $ dmconvertvol /tmp/dmftxtdb/msp1_vol_txt vgpri > /tmp/dmftxtdb/vgpri_vol_txt |
Next, the text versions of msp2's files are converted to vgsec:
$ dmconvertcat /tmp/dmftxtdb/msp2_cat_txt > /tmp/dmftxtdb/vgsec_cat_txt $ dmconvertvol /tmp/dmftxtdb/msp2_vol_txt vgsec > /tmp/dmftxtdb/vgsec_vol_txt |
Since the name for the volume group has not been chosen to be the same as for the MSP being converted, the daemon database must be converted. For each tape MSP being converted to a volume group with a different name, dmconvertdaemon is run. In the following two steps, the first step handles the conversion of msp1 to vgpri in the daemon database. The output of that command is then used as a parameter to dmconvertdaemon in the second step. The second step handles the conversion from msp2 to vgsec in the daemon database.
$ dmconvertdaemon msp1 vgpri /tmp/dmftxtdb/daemon_txt >\ /tmp/dmftxtdb/daemon_pri_txt $ dmconvertdaemon msp2 vgsec /tmp/dmftxtdb/daemon_pri_txt > \ /tmp/dmftxtdb/daemon_cnvt_txt |
Next, the databases are loaded on the Linux system. Since two tape MSPs are being converted to volume groups within the same LS, two files are loaded into the ls1 CAT database, and into the ls1 VOL database:
$ dmdadm -u -c "load /tmp/dmftxtdb/daemon_cnvt_txt" $ dmcatadm -m ls1 -u -c "load /tmp/dmftxtdb/vgpri_cat_txt" $ dmvoldadm -m ls1 -u -c "load /tmp/dmftxtdb/vgpri_vol_txt" $ dmcatadm -m ls1 -u -c "load /tmp/dmftxtdb/vgsec_cat_txt" $ dmvoldadm -m ls1 -u -c "load /tmp/dmftxtdb/vgsec_vol_txt" |
Now dmdbcheck is run to verify the consistency of the databases.
$ cd /dmf/home/daemon; dmdbcheck -a dmd_db $ cd /dmf/home/ls1; dmdbcheck -a libsrv_db |
For an existing MSP-based configuration to take advantage of the additional features of the LS, the existing databases must be converted. Several databases can be converted at the same time, or the conversion can be done in stages over a period of time. You can perform any of the following conversions:
Convert just one MSP's databases to a new volume group in a new LS
Convert an additional MSP to a new volume group within an existing LS
Convert all databases at once
Procedure 6-2. Tape MSP/LS Conversion
Run dmcheck(8)to check the existing configuration.
Copy the production configuration file (/etc/dmf/dmf.conf (2.8 or later) or /etc/dmf/dmbase/host/ hostname/dmf_config(2.7 or earlier)) and replace the definition of the MSPs to be converted with the stanzas defining the equivalent LS components to a new file, /tmp/dmf.conf.new. You might find it useful to examine the sample configuration to be found in /usr/share/doc/dmf-version_number/info/sample/dmf.conf.ls . Over time, many small changes have been made to benefit existing installations as well as new ones.
To replace the definition of the MSPs, you must do the following:
Delete the stanza for the MSP object.
If there are no other references to the device object, remove it.
Create an LS stanza and include the following parameters if they were specified in the MSP definition:
CACHE_DIR
CACHE_SPACE
MAX_CACHE_FILE
MESSAGE_LEVEL
TASK_GROUPS
Replace the MSP's name in the MSP_NAMES (or LS_NAMES) directive in the daemon stanza with the name of this LS.
Create a drive group stanza and include the following parameters if they were specified in the possibly-deleted device object:
BLOCK_SIZE
LABEL_TYPE
MOUNT_SERVICE
MSG_DELAY
OV_ACCESS_MODES
OV_INTERCHANGE_MODES
POSITIONING
POSITION_RETRY
TMF_TMMNT_OPTIONS
VERIFY_POSITION
WRITE_CHECKSUM
Add a MOUNT_SERVICE_GROUP parameter to specify the TMF device group or OpenVault drive group. If TMF_TMMNT_OPTIONS contained a -g specification to provide this information, remove that part of it.
The DRIVE_GROUPS parameter in the LS stanza should refer to this drive group.
Create one volume group stanza per MSP being converted, possibly with the same names as the MSPs they are replacing, and include the following parameters if they were specified in the MSP definitions:
HFREE_TIME
MAX_CHUNK_SIZE
MAX_PUT_CHILDREN
MERGE_CUTOFF
TIMEOUT_FLUSH
Include the ZONE_SIZE parameter from the possibly-deleted device object.
The VOLUME_GROUPS parameter in the drive group stanza should refer to these volume groups. If their names differ from those of the MSPs they are replacing, update the SELECT_MSP/ SELECT_VG policy parameters.
In the task-group that controls filesystem backups with run_full_dump.sh and run_partial_dump.sh , change the DUMP_DEVICE parameter to refer to the drive group rather than to the possibly-deleted device object.
To check this new configuration before placing it into production, before running dmcheck(8), set the DMF_CONFIG environment variable to the absolute path of the file, as follows:
setenv DMF_CONFIG /tmp/dmf.conf.new dmcheck
Run dmaudit(8) and dmdbcheck(8) to confirm that there are no problems with the current databases.
Stop DMF and put the new configuration in place. You can run dmcheck(8) again, if you wish.
Run /usr/lib/dmf/support/dmmsptols, as described in the man page. This process might take some time (even several hours for a large configuration). The selection of MSPs that must be converted at this point is determined by the changes made to the configuration in the previous step.
If there are any problems, the dmmsptols process will instruct you how to back out of the conversion by using the backups it created.
Start DMF and run dmaudit(8) and dmdbcheck(8).
At a later time, you can make additional changes to the configuration to enable or configure new features, such as allocation groups, error recovery, or resource schedulers and watchers. The parameters controlling these are described in Chapter 2, “Configuring DMF”, or elsewhere in this chapter.
When this procedure is followed, the resulting configuration will schedule tape merges for all volume groups to be done at once. This does not cause problems for the LS as it would for the MSP-based configuration, but you might wish to have finer control over the process. You can do this by creating new task-group objects just to control tape merging, and invoking them with a TASK_GROUPS parameter inside the volume groups' stanzas. The other parameters inside the original task-group should still be invoked only from the LS's stanza.
Alternatively, the RUN_TASK parameters can be placed directly in the volume group stanza; they can be specified outside a task-group's stanza.
The improved tape positioning code specified by the drive group's POSITIONING parameter to data will be activated only for data written by the volume group (that is, newly migrated files and files that have been merged from a tape written by the MSP to a new volume group one). The improvement in performance will become more noticeable over time, as a greater proportion of data fits into these categories.
Unlike the MSP, the drive group component of the LS monitors tape use, analyzing any failures, and using this information to avoid future errors.
The drive group component can react to some failures without looking for any patterns of behavior. Among these are the following:
Mounting service failure. If the mounting service is TMF, by default, DMF makes one attempt to restart it. If this attempt does not succeed, DMF notifies the administrator by e-mail and waits for the administrator's intervention. When TMF is back again, DMF resets the auto-restart flag so that if TMF fails again, it will once again make one attempt to restart it.
If OpenVault is the mounting service, by default, no attempt is made to restart it. Instead, an e-mail is sent to the administrator.
A site can set the number of automatic restart attempts by using the drive group's MAX_MS_RESTARTS configuration parameter, but caution and thorough testing are advised. There are many possible failure modes for a mounting service, and automated restarts might not always be appropriate.
Tape volume is not in the tape library. Obviously, this problem will not be fixed by trying again. To prevent further access, the volume is locked by setting the HLOCK flag, as described below, and the user requests that triggered the access attempt are retried on another tape, if possible; otherwise, they are aborted. The administrator is notified by e-mail.
For TMF only, a tape mount was cancelled by an operator or administrator. Although the user requests are retried or aborted, the volume is not disabled. If the volume were disabled, it would be inaccessible for a period of time (default 24 hours) unless dmvoladm were used to preempt this delay. All operators do not necessarily have access to the dmvoladm command.
Because the reason for the cancellation is unknown to DMF, repeated requests for the same volume are quite possible, and the operator might have to cancel each one.
The drive group handles other types of failure by examining the recent history of the tape volume and the tape drive that was used. The drive group maintains records of past tape I/O errors, and uses these to control the way it reacts to future errors.
For example, if a tape has been unusable several times in a row, even though different tape drives were used, the drive group concludes that the problem most likely involves the tape volume rather than the drive. Therefore, it suspends use of that tape for a while, forcing DMF to migrate to a different tape in that volume group, or to recall the file from another tape held by a different volume group. This suspension is usually done by setting the HLOCK flag in the tape's entry in the volume database. This makes the tape inaccessible to the volume group for both reading and writing until it is automatically cleared after REINSTATE_VOLUME_DELAY minutes.
If a variety of volumes fail on a specific drive but are usable on other drives, a drive problem is likely, and the tape drive can be automatically configured down if permitted by the administrator's setting of DRIVES_TO_DOWN to a value higher than its default of zero. When a drive is configured down in this way, it is configured up again after REINSTATE_DRIVE_DELAY minutes.
The analyses of drive and volume errors are performed independently of each other; it is possible for one additional error to result in both the drive and the volume being disabled.
There are several reasons for reinstating drives and volumes after a delay. The most important is that the analyses of previous failures might lead to a faulty conclusion in some situations, such as when DMF is under a very light load, or when multiple failures occur concurrently. A wrong diagnosis might impact DMF's performance, and should not be accepted indefinitely. Disabling a suspected drive or volume for a while is usually enough to break any repetitive cycles of failure. If such patterns re-establish themselves when the reinstatement occurs, the drive group will again analyze the behavior, possibly reaching a different conclusion, and again try to prevent it.
There are some variations from these general reactions. For example, if a tape volume with existing data on it is diagnosed as faulty when appending new data, instead of setting the HLOCK flag, the drive group sets HFULL, which results in the tape being used in a read-only mode until eventually emptied by merges or hard deletion of its files. HFREE_TIME seconds after it becomes empty, it may be placed back into use unless the administrator has decided, possibly as a result of testing it, that it should be deleted or replaced.
In all of these situations, the administrator is notified by e-mail.
If it is considered desirable to return a volume or drive to service earlier than defined in the DMF configuration, the appropriate command (dmvoladm, tmconfig, or ov_drive) can be safely used.
When multiple volume groups are requesting the use of more tape drives than exist in the drive group, the resource scheduler is used to decide which volume groups should wait, and which should be assigned the use of the drives.
The resource scheduler is unaware of non-volume-group activity on the drives in its drive group. Such activity includes MSPs, XFS dumps, and any direct tape use by the system's users, and does not prevent the LS from working properly, though it might be less than optimal.
By default, the resource scheduler uses a round-robin based algorithm, but a site can assign different weightings to different volume groups to meet local requirements. (For more information, see “Resource Scheduler Objects” in Chapter 2).
Some sites will have requirements that cannot be met by a general purpose algorithm. Such sites can write their own resource scheduler algorithms in C++, to be used in place of the supplied one. Instructions can be found in the /usr/share/doc/dmf-version_number/info/sample/RSA.readme file.
You can observe the performance of the LS in two ways. You can monitor its log file with a tool like tail -f, which allows an experienced administrator to follow the flow of events as they happen. You can also use the resource watcher component, when enabled by use of the WATCHER parameter in the libraryserver configuration stanza.
The resource watcher is intended to give the administrator a view of the status of an LS and some of its components. It maintains a set of text files on disk, which are rewritten as events happen. These files can be found in the SPOOL_DIR/lsname/_rwname directory, where SPOOL_DIR is defined in the DMF configuration file, as are the names of the LS and resource watcher (lsname and rwname in the following example). The easiest way to find the precise path is to look in the LS log file for messages like the following:
dmatls rwname.config_changed: Resource Watcher output files will be placed in /dmf/spool/lsname/_rwname at DMF startup or whenever the configuration file is altered or "touch"ed. |
This message is issued at DMF startup or whenever the configuration file is altered or its modification time changes (for example, by using the touch(1) command.
The SPOOL_DIR/lsname/_rwname directory contains files with names ending in .html, which are automatically refreshing HTML files. You can access these files by using a browser running on the same machine. The following example shows an LS page that contains links to drive group pages, and they in turn have links to volume group pages, if the volume groups are active at the time.
netscape file:/dmf/spool/lsname/_rwname/lsname.html |
If running the browser on the DMF machine is inconvenient, you can include the directory in your HTTP server configuration to allow those same pages to be accessed via the web.
This directory also contains files whose names end in .txt, designed to be parsed with programs like awk. The data format is described by comments within those files and can be compared with the equivalent HTML files. If the format of the text ever changes, the version number will change. If the changes are incompatible with previous usage, the number before the decimal point is altered. If they are compatible, the number after the decimal point is altered. An example of compatibility is adding extra fields to the end of existing lines or adding new lines. Programs using these files should check the version number to ensure compatibility. Also, it might be useful to check the DMF version shown by dmversion(1) and the IRIX version from uname(1).