Chapter 2. Configuring DMF

This chapter describes how to configure DMF, verify the configuration, and perform some periodic maintenance tasks.

Overview of the Configuration Steps

The steps outlined in the following procedure are required to configure DMF.

Procedure 2-1. Configuration Steps

  1. Install DMF, ensuring that FLEXlm licensing is set up correctly. Installation is described in the DMF Release and Installation Guide.


    Note: For a description of special configuration issues regarding installation, you must read “Installation Considerations”.


  2. Ensure that your PATH and MANPATH environment variables are set to include DMF paths. See “Setting PATH Environment Variables”.

  3. Invoke dmmaint(8) so that you can create or modify your configuration file. Using dmmaint is described in the DMF Release and Installation Guide.

  4. Determine how you want to complete periodic maintenance tasks. See “Configuring Automated Maintenance Tasks”.

  5. Edit the configuration file to define the base object, daemon object, the objects for daemon maintenance tasks, and objects for automated space management. See “Configuring the Base Object”, through “DMF Policies”.

  6. Define the media-specific process (MSP) or library server (LS) objects. Also define the object for MSP/LS maintenance tasks, set up the MSPs and/or LSs, and configure your mounting service. See “Setting Up Tape MSPs ”, through “Setting up Disk MSPs”.

  7. Verify the configuration with the dmcheck(8) script. See “Verifying the Configuration”.

  8. Start DMF. See “Initializing DMF”.

Installation Considerations

This section discusses installation considerations that will affect how your system is configured.

Configuration File Requirements

The DMF server uses a set of path names in which it stores databases, log and journal files, and temporary file directories. These file systems have the following requirements:

  • HOME_DIR, the base path name for DMF directories in which databases reside, must be a separate file system.

  • JOURNAL_DIR, the base path name for DMF directories in which the daemon and tape MSP/LS database journal files reside, must be a separate file system on a different disk from HOME_DIR.

  • SPOOL_DIR, the base path name used to construct the directory names for DMF directories in which DMF log files reside, must be a separate file system.

  • TMP_DIR, the base path name used to construct the directory names for DMF directories in which DMF puts temporary files such as pipes, should exist, but does not necessarily need to be a separate file system.

  • MOVE_FS, the base path name for the scratch file system used to move files between MSPs or volume groups, has requirements only if you configure more than one MSP or volume group. If you have more than one MSP or volume group, MOVE_FS must be a separate file system, and it must be mounted to enable the DMAPI interface.

All of these configuration requirements are checked by the dmcheck(8) command.

Man Pages

Ensure that you and all DMF users add the correct path to the MANPATH environment variable as described in the DMF Release and Installation Guide, in the chapter titled “Before You Start DMF.” Man pages for DMF commands are installed into the /usr/share/catman/u_man/cat1 directory for IRIX systems and /usr/share/man/man1 for Linux systems.

File System Mount Options

The Data Management API (DMAPI) is the mechanism between the kernel and the XFS file system for passing file management requests between the kernel and DMF. Ensure that you have installed DMAPI and the appropriate patches as listed in the files accessed by the News button on the DMF installation interface (dmmaint(8)).


Caution: For file systems to be managed by DMF, they must be mounted to enable the DMAPI interface. On IRIX systems, you can do this by using the mount -o dmi command or by declaring parameter 4 in the fstab entry to be dmi. On Linux systems, you can do this by using the mount -o dmapi -o mtpt = mountpoint command or by adding dmapi, mtpt = mountpoint to the fourth field in the fstab entry. For more information on the mount or fstab commands, see the man pages. Failure to enable DMAPI for DMF-managed file systems will result in a configuration error.


Mounting Service

Tape mounting services are available through OpenVault or the Tape Management Facility (TMF). The MSP/LS checks mounting service availability when it is started and after each occurrence in which an MSP/LS write child or read child was unable to reserve its drive. If the mounting service is found to be unavailable, the tape MSP/LS does not start any new child processes until the mounting service is once again available.

If the unavailable mounting service is OpenVault, the MSP/LS sends an e-mail message to the administrator, asking that OpenVault be started, and then periodically polls OpenVault until it becomes available, at which time child processes are again allowed to run. For LS, this is the default procedure. You can use MAX_MS_RESTARTS to configure the number of automatic restarts.

If the unavailable mounting service is TMF, the tape MSP/LS not only attempts to initiate tmdaemon if it is not up (based on the exit status of tmstat), but it waits until a TMF device in the configuration pending state is configured up before it resumes processing. If TMF cannot be started or if no devices are configured up, the tape MSP/LS sends e-mail to the administrator and polls TMF until a drive becomes available. For LS, this is the default procedure. You can use MAX_MS_RESTARTS to configure the number of automatic restarts.

Inode Size Configuration

DMF state information is kept within a file system structure called an extended attribute. Extended attributes can be either inside the inode or in attribute blocks associated with the inode. DMF runs much faster when the extended attribute is inside the inode, because this minimizes the number of disk references that are required to determine DMF information. In certain circumstances, there can be a large performance difference between an inode-resident extended attribute and a non-resident extended attribute.

You should configure your file systems to ensure that the extended attribute is always inode-resident. This is done by using the mkfs_xfs command for IRIX systems and the mkfs.xfs command for Linux systems. Declare the inode size to be 512 bytes using the -i size=512 option. File systems that already exist will have to be dumped, recreated, and restored. This change is not mandatory.

Configuring Daemon Database Record Length

A daemon database entry is composed of one or more fixed length records: a base record (dbrec) and zero or more path segment extension (pathseg) records. If the path value that is returned to the daemon by the MSP/LS can fit into the path field of the daemon's dbrec record, DMF does not require pathseg records. If the MSP/LS supplies a path value that is longer than the path field in the dbrec, DMF creates one or more pathseg records.

The default size of the path field of the dbrec is 34 characters. This size allows the default paths returned by the dmatmsp, dmatls, dmdskmsp, and dmftpmsp to fit in the dbrec path field as long as the user name portion of the dmftpmsp or dmdskmsp path (username/bit_file_identifier) is 8 characters or fewer. In almost all cases, you should not need to reconfigure the daemon database record.

The default size of the path field in the pathseg record is 64. For MSP path values which are just slightly over the size of the dbrec path field, this will result in a large amount of wasted space for each record that overflows into the pathseg record. The ideal situation would be to have as few pathseg records as possible.

The advantage of having very few pathseg records lies in increased efficiency for retrieving daemon database records. There is no need to access the pathseg key and data files to retrieve a complete daemon database record.

The disadvantage of using the default path size arises mainly in the tape MSP/LS application in which there is a small amount of wasted space in the daemon's dbrec data file. By extending the default path field size to 34 (8 bytes more than the tape MSP/LS requires), there is a resulting 5% wasted space in the daemon's dbrec data file. For a 10 MB dbrec file, this is 500 Kbytes of wasted space.

For installations that run only the tape MSP or LS and for which the 5% wasted disk space is an important consideration, the size of the path field in the daemon dbrec record can be configured at any time before or after installation. (The same holds true for any installation that might be using the dmftpmsp or dmdskmsp with a different path-generating algorithm or any other MSP that supplies a path longer than 34 characters to the daemon.)

Procedure 2-2. Daemon Database Record Length Configuration

The steps to configure the database entry length are as follows:

  1. If the dmfdaemon is running, use the following command to halt processing.

    /etc/init.d/dmf stop

  2. If a daemon database already exists, perform the following steps:

    1. cd HOME_DIR/daemon (HOME_DIR is the value of HOME_DIR returned by the dmconfig base command)

    2. dmdump -c . > textfile (textfile is the name of a file that will contain the text representation of the current database)

    3. cp dbrec* pathseg* dmd_db.dbd backup_dir (backup_dir is the name of the directory that will hold the old version of the database)

    4. rm dbrec* pathseg* dmd_db.dbd

  3. cd /usr/lib/dmf/rdm

  4. Back up the dmd_db.dbd and dmd_db.ddl files that reside in /usr/lib/dmf/rdm. This will aid in disaster recovery should something go wrong.

  5. Edit dmd_db.ddl to set the new path field lengths for the dbrec and/or pathseg records. For the most efficient use of disk space for the dmatmsp, set the dbrec path size to 26.

  6. Regenerate the new database definition, as follows:

    /usr/lib/dmf/support/dmddlp -drsx dmd_db.ddl

  7. Backup the new versions of dmd_db.dbd and dmd_db.ddl for future reference or disaster recovery.

  8. If the daemon database was dumped to text in step 2, complete the following steps:

    1. cd HOME_DIR/daemon

    2. dmdadm -u -c "load textfile" (textfile was created in step 2)

  9. If the daemon was running in step 1, restart it by executing the following command:

    /etc/init.d/dmf start

Interprocess Communication Parameters

Ensure that, in the operating system configuration file, the following IPC kernel configuration parameters are set equal to or greater than the default: MSGMAX, MSGMNI, MSGSEG, and MSGSSZ. For IRIX systems, the parameters are described in Appendix A of IRIX Admin: System Configuration and Operation. For Linux systems, the full documentation for IPC messages is maintained as a Textinfo file. If the info program is properly installed on your system, you can use the info ipc command to access the information.

Configuring Automated Maintenance Tasks

DMF lets you configure parameters for completing periodic maintenance tasks such as the following:

  • Making backups (full or partial) of user file systems to tape

  • Making backups of DMF databases to disk

  • Removing old log files and old journal files

  • Monitoring DMF logs for errors

  • Running hard deletes

  • Running dmaudit(8)

  • Monitoring the status of tapes in tape MSPs and LSs

  • Merging tapes that have become sparse (and stopping this process at a specified time)

Each of these tasks can be configured in the DMF configuration file through the use of TASK_GROUPS parameters for the DMF daemon and the tape MSP/LS. The tasks are then defined as objects.

For each task you configure, a time expression defines when the task should be done and a script file is executed at that time. The tasks are provided for you in the /usr/lib/dmf directory.

The automated tasks are described in “Configuring Daemon Maintenance Tasks”, for the daemon tasks and in “Configuring Tape Maintenance Tasks ”, for the tape MSP.

Table 2-1provides a summary of the automated maintenance tasks:

Table 2-1. Automated Maintenance Task Summary

Task

Purpose

Parameters

Related Object Type

run_audit

Audit databases

 

daemon

run_copy_databases

Backup DMF databases

DATABASE_COPIES

daemon

run_remove_journals

Remove old journal files

JOURNAL_RETENTION

daemon

run_remove_logs

Remove old log files

LOG_RETENTION

daemon

run_full_dump

Full backup of file system(s)

For restores, see dmxfsrestore(8)

DUMP_DEVICE

DUMP_INVENTORY_COPY

DUMP_FILE_SYSTEMS

DUMP_MIGRATE_FIRST

DUMP_RETENTION

DUMP_VSNS_USED

DUMP_TAPES

daemon

run_hard_deletes

Hard-delete files

Uses DUMP_RETENTION

daemon

run_merge_stop

Stop tape merges

msp/ls

run_partial_dump

Partial backup of file system(s)

Uses parameters set for run_full_dump

daemon

run_scan_logs

Scan log files for errors

 

daemon

run_tape_merge

Merge sparse tapes

DATA_LIMIT

THRESHOLD

VOLUME_LIMIT

msp/ls

run_tape_report

Create tape reports

 

msp/ls

run_compact_tape_report

Create tape reports

 

msp/ls


Setting PATH Environment Variables

To use DMF commands and DMF man pages, set your PATH and MANPATH environment variables. When the software that allows a machine to run the DMF daemon, MSPs, and LSs is installed, the DMF administrator commands and executable files are installed in /usr/sbin. On IRIX systems, man pages are installed in /usr/share/catman/u_man/cat[1,8] . On Linux systems, man pages are installed in /usr/share/man/man[1,8]. This type of installation is called a DMF server installation. User commands are installed in /usr/bin.

Also, beginning with DMF 2.7, it is possible to configure a machine as a DMF client. This configuration installs the software required so that users can execute the DMF user commands on machines that have DMF-managed filesystems exported to them, but never execute as the DMF server host. In this case, the user commands are installed in /usr/bin, and the user man pages are installed in /usr/share/catman/u_man/cat[1,8] for IRIX systems and in /usr/share/man/man[1,8] for Linux systems. The remainder of this section deals with DMF server installations.


Note: If you are not familiar with setting the MANPATH environment variable, you should know that some paths are checked even though they are not listed by default. In other words, even though the command echo $MANPATH appears to indicate that no variable is defined (in ksh it returns no message or in csh it returns the message MANPATH - Undefined variable), certain paths are still searched for man pages. Setting the MANPATH environment variable as described below will supersede the fact that these paths are searched.

If MANPATH has not been set, you should read the man(1) man page to determine the paths that are checked and then include those paths in the commands below.

You can set the user command and man path names in the file /etc/profile for all ksh users and /etc/cshrc for all csh users, or provide a module for users.

Configuration Objects

The configuration file consists of configuration objects and parameters. The file uses the following types of configuration objects:

  • The base object, which defines path name and file size parameters necessary for DMF operation

  • The daemon object, which defines parameters necessary for dmfdaemon(8) operation

  • The file system object, which defines parameters necessary for migrating files in that file system

  • The policy objects, which specify parameters to determine MSP/VG selection, automated space management policies, and/or file weight calculations in automatic space management

  • The MSP objects, which define parameters necessary for that MSP's operation

  • The device objects, which define parameters for the MSP's use of tape devices

  • The taskgroup objects, which define parameters necessary for automatic completion of specific maintenance tasks

  • The library server object, which defines parameters relating to a tape library

  • The drive group object, which defines parameters relating to a pool of tape devices in a specific library

  • The volume group object, which defines parameters relating to a pool of tape volumes mountable on the drives of a specific DG, capable of holding, at most, one copy of user files

  • The resource scheduler object, which defines parameters relating to scheduling of tape devices in a DG when requests from VGs exceed the number of devices available

  • The resource watcher object, which defines parameters relating to the production of files informing the administrator about the status of the library server and its components

DMF configuration objects and parameters are also defined in the dmf.conf(5) man page and in Table 2-3.

Each object is configured by a sequence of lines called a configuration stanza. These have the following general form:

define          object_name
     TYPE            object_type
     parameter-1     value(s)
 ... 
     parameter-n     value(s)
 enddef

For file systems, object_name is the mount point. Otherwise, it is chosen by the administrator. object_type identifies the type (detailed in the following subsections). The parameters and their values depend on the type of the object. These stanzas are case-sensitive and can be indented for readability. The fields can be separated by spaces and/or tabs. Blank lines and all commentary text between a hash character (#) and the end of that line are ignored. Except for comments, any line ending in a back-slash (\) continues onto the next line. Before placing a new configuration into production, it is important to check it by running dmcheck(8).

Configuring the Base Object

The base configuration parameters define path names and file sizes necessary for DMF operation. It is expected that you will modify the path names, although those provided will work without modification. All path names must be unique.

Parameter 

Description

TYPE 

base (type of object).

ADMIN_EMAIL 

E-mail address to which to send output from administrative tasks. The mail can include errors, warnings, and output from any configured tasks. You can specify a list of addresses, separated by spaces.

HOME_DIR 

Base path name used to construct directory names for DMF directories in which databases and related files reside. Generally referred to as HOME_DIR.

JOURNAL_DIR 

Base path name used to construct directory names for DMF directories in which the daemon and tape MSP/LS database journal files will be written. To provide the best chance for database recovery, this directory should be a separate file system and a different physical device from HOME_DIR. Generally referred to as JOURNAL_DIR.

JOURNAL_SIZE 

Maximum size (in bytes) of the database journal file before DMF closes it and starts a new file.

LICENSE_FILE 

Full path name of the file containing the FLEXlm license used by DMF. The default is /var/flexlm/license.dat on IRIX and /etc/flexlm/license.dat on Linux. There is no need to use this parameter if the default is being used.

OV_KEY_FILE 

File containing the OpenVault keys used by DMF. It is usually located in HOME_DIR and called ovkeys. There is no default. (Use this parameter only if you are using OpenVault as your tape mounting service.)

OV_SERVER 

Name returned by the hostname(1) command on the machine on which the OpenVault server is running. This parameter only applies when OpenVault is used as the mounting service. The default value is the host name of the machine on which you are running.

SPOOL_DIR 

Base path name used to construct the directory names for DMF directories in which DMF log files are kept. Generally referred to as SPOOL_DIR.

TMP_DIR 

Base path name used to construct the directory names for DMF directories in which DMF puts temporary files such as pipes. It is also used by scripts for temporary files and is the directory used by default by the tape MSP for caching files if the CACHE_DIR parameter is not defined. Generally referred to as TMP_DIR.


Warning: Do not change the directory names while DMF is running.

If you intend to run the OpenVault library management facility as the mounting service for DMF, you must configure the OV_KEY_FILE and OV_SERVER parameters. If you are running a different mounting service, you do not need these parameters. More configuration steps are necessary to configure DMF to use OpenVault; see “Using OpenVault for Tape MSPs and Drive Groups”.

Procedure 2-3. Base Object Configuration

The following example defines a base object:

define  base
        TYPE            base
        ADMIN_EMAIL     root@dmfserver
        HOME_DIR        /dmf/home
        TMP_DIR         /tmp/dmf
        SPOOL_DIR       /dmf/spool/
        JOURNAL_DIR     /dmf/journals
        JOURNAL_SIZE    10m
        OV_KEY_FILE     /dmf/home/ovkeys
        OV_SERVER       localhost
enddef


Note: Do not use automated space management to manage the HOME_DIR, SPOOL_DIR, or JOURNAL_DIR directories because DMF daemon processes will deadlock if files that they are actively using within these directories are migrated. dmcheck(8) reports an error if any of the HOME_DIR, SPOOL_DIR, or JOURNAL_DIR parameters are also configured as DMF-managed file systems. Configure the daemon_tasks object to manage old log files and journal files in these directories (you can change the namedaemon_tasks to be anything you prefer). See “Configuring Daemon Maintenance Tasks”, for more information.


The following steps explain pertinent information for configuring the base object:

  1. Ensure that TYPE is set to base.

  2. Configure the e-mail address specified by the ADMIN_EMAIL parameter to be the user to whom you want to send the output of the configured tasks described in “Configuring Automated Maintenance Tasks”.

  3. Configure the file system specified by the HOME_DIR configuration parameter (referred to as HOME_DIR) as a separate file system, and restrict its contents to DMF databases and relatively static files such as DMF scripts.

    DMF cannot run if HOME_DIR runs out of space, and such an event is more likely to happen if it is simply another directory in /usr.

  4. Set TMP_DIR to be any file system that can store temporary files. /tmp or a directory below /tmp is a common choice.

  5. Configure the log file directory (referred to as SPOOL_DIR) as a separate file system so that log file growth does not impact the rest of the system.

  6. Ensure that the journal file directory (referred to as JOURNAL_DIR) resides on a physical device completely separate from the one on which HOME_DIR resides. Backup copies of DMF databases should also be stored on the JOURNAL_DIR file system.

  7. Configure the JOURNAL_SIZE parameter to be the maximum size allowable for a journal file before DMF closes it.

  8. If you plan to run OpenVault, configure the OV_KEY_FILE parameter to be the name of the key file that holds security information for OpenVault. For more information, see Procedure 2-13.

  9. If you plan to run OpenVault, configure the OV_SERVER parameter to the name of the server that runs OpenVault. For more information, see Procedure 2-13.

Configuring the DMF Daemon

The daemon object defines configuration parameters necessary for the DMF daemon operation. It is expected that you will modify the values for the path names and MSP names.

Parameter 

Description

TYPE 

dmdaemon (type of object)


Note: This cannot be specified as dmfdaemon. It must be dmdaemon.


MESSAGE_LEVEL 

Specifies the highest message level number that will be written to the daemon log. It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2. For more information on message levels, see “General Message Log File Format”.

MIGRATION_LEVEL 

Sets the highest level of migration service allowed on all DMF file systems (you can configure a lower service level for a specific file system). The value can be none (no migration), user (requests from dmput(1) or dmmigrate(8) only), or auto (automated space management). The default is auto.

MOVE_FS 

Names the scratch file system used by dmmove(8) to move files between MSPs/VGs. There is no default. Necessary only if you wish to use dmmove.

MSP_NAMES 

Names the MSPs and LSs used by the DMF daemon. As a convenience, you can use LS_NAMES instead of MSP_NAMES, but you can specify only one. You must specify a value for MSP_NAMES (or LS_NAMES); there is no default.

The order of the values specified for this parameter is integral to the determination of the MSP or volume group from which the DMF daemon attempts to recall an offline file. If the offline file has more than one copy, DMF uses a specific order when it attempts to recall the file. It searches for a good copy of the offline file in MSP or Library Server order, from the dmdaemon object's MSP_NAMES or LS_NAMES parameter. If one of those names refers to a Library Server, it searches for the copy in drive group order, from the Library Server object's DRIVE_GROUPS parameter. It then searches for the copy in volume group order from the drivegroup object's VOLUME_GROUPS parameter.

TASK_GROUPS 

Names the task groups that contain tasks the daemon should run. They are configured as objects of TYPE taskgroup. There is no default. For more information, see “Configuring Daemon Maintenance Tasks”.

SGI recommends that you use the task groups specified in the sample configuration file, changing the parameters as necessary for your site.

Procedure 2-4. Daemon Configuration

The following example defines a daemon object:

define  daemon
        TYPE                    dmdaemon
        MOVE_FS                 /move_fs
        MIGRATION_LEVEL         auto
        MSP_NAMES               cart1 cart2
        TASK_GROUPS             daemon_tasks dump_tasks
enddef

The following steps explain pertinent information for configuring the daemon object:

  1. Ensure that TYPE is set to dmdaemon. There is no default.


    Note: This cannot be set to dmfdaemon. It must be dmdaemon.


  2. If you have more than one MSP/VG, ensure that the MOVE_FS parameter is set to a file system that can accept temporary files. This must be the root of a DMAPI file system. There is no default.

  3. The MIGRATION_LEVEL parameter determines the level of service for migration to offline media. Migration from offline media (either automatic or manual recall) is not affected by the value of MIGRATION_LEVEL.

    Configure MIGRATION_LEVEL to be none, user, or auto. This value is the highest level you want to allow anywhere in your DMF environment. You can configure a lower level for a specific file system. none means no migration will take place on any DMF file system. user means that users/administrators can perform dmput(1) or dmmigrate(8) commands and no other migration will take place. auto means that you want automated space management on at least one DMF file system. The default is auto. See “DMF Policies”, for information about configuring automated space management.

  4. Configure MSP_NAMES to be the names of the MSPs and/or LSs to be used by this daemon. You will use these names when defining the MSP/LS objects and, for MSPs only, in SELECT_MSP parameters within policies. See Procedure 2-10. As a convenience, you can use LS_NAMES instead of MSP_NAMES, but you can specify only one. You must specify a value for MSP_NAMES (or LS_NAMES; there is no default.

  5. Configure the TASK_GROUPS parameter to the name(s) of the object(s) used to define how periodic maintenance tasks are completed. In the example, daemon_tasks defines the tasks such as scanning and managing log files and journal files. The dump_tasks object defines tasks that back up DMF-managed file systems. You can change the object names themselves (dump_tasks and daemon_tasks) to be any name you like. There is no default value for the object. See “Configuring Daemon Maintenance Tasks” for more information.

Configuring Daemon Maintenance Tasks

You can configure daemon_tasks parameters to manage how the DMF daemon performs the following maintenance tasks:

  • Auditing databases (the run_audit.sh task)

  • Scanning recent log files for errors (the run_scan_logs.sh task)

  • Removing old log files (the run_remove_logs.sh task and the LOG_RETENTION parameter)

  • Removing old journal files (the run_remove_journals.sh task and the JOURNAL_RETENTION parameter)

  • Backing up DMF databases (the run_copy_databases.sh task and the DATABASE_COPIES parameter)

For each of these tasks, you can configure when the task should be run. For some of the tasks, you must provide more information such as destinations or retention times for output.

You can configure dump_tasks parameters to manage how the daemon completes the following tasks to back up the DMF-managed file systems:

  • Fully backing up DMF-managed file systems (the run_full_dump.sh task)

  • Partially backing up DMF-managed file systems (the run_partial_dump.sh task)

  • Hard-deleting files no longer on backup tape (the run_hard_deletes.sh task)

  • Managing the data from the file system dumps (the DUMP_TAPES, DUMP_RETENTION, DUMP_DEVICE, DUMP_MIGRATE_FIRST, DUMP_INVENTORY_COPY, DUMP_FILE_SYSTEMS, and DUMP_VSNS_USED parameters)

For each of these tasks, you can configure when the task is run. To manage the tapes, you must provide information such as tape and device names, retention times for output, whether to migrate files before dumping the file system, and locations for inventory files. Table 2-1, provides a summary of automated maintenance tasks.

Procedure 2-5. Configuring the daemon_tasks Object

The following steps explain how to define a daemon_tasks object. You can change the object name itself (daemon_tasks) to be any name you like.

Do not change the script names.

You may comment out the RUN_TASK parameters for any tasks you do not want to run.

The following example would configure a daemon_tasks object:

define daemon_tasks
        TYPE                 taskgroup
        RUN_TASK             $ADMINDIR/run_audit.sh every day \
                                at 23:00
#
        RUN_TASK             $ADMINDIR/run_scan_logs.sh at 00:01
#
        RUN_TASK             $ADMINDIR/run_remove_logs.sh every \
                                day at 1:00
        LOG_RETENTION        4w
#
        RUN_TASK             $ADMINDIR/run_remove_journals.sh every \
                                day at 1:00
        JOURNAL_RETENTION    4w
#
        RUN_TASK             $ADMINDIR/run_copy_databases.sh \
                             every day at 3:00 12:00 21:00
        DATABASE_COPIES      /save/dmf_home /alt/dmf_home
enddef

  1. Define the object to have the same name that you provided for the TASK_GROUPS parameter of the daemon object. In the example it is daemon_tasks.

  2. Ensure that TYPE is set to taskgroup. There is no default.

  3. Configure the RUN_TASK parameters. DMF substitutes $ADMINDIR in the path with the actual directory containing auxiliary programs and scripts, /usr/lib/dmf at DMF 2.8 or later, or /etc/dmf/dmbase/etc/admin at 2.7 and earlier. When the task is run, it is given the name of the object that requested the task as the first parameter and the name of the task group (in this case daemon_tasks) as the second parameter. The task itself may use the dmconfig(8) command to obtain further parameters from either of these objects.

    All of the RUN_TASK parameters require that you provide a time_expression.

    The time_expression defines when a task should be done. It is a schedule expression that has the following form:

    [every n  period] [at hh:mm[:ss] ...] [on day ...]

    period is one of minute[s], hour[s], day[s], week[s], or month[s].

    n is an integer.

    day is a day of the month (1 through 31) or day of the week (sunday through saturday).

    The following are examples of valid time expressions:

    at 2:00
    every 5 minutes
    at 1:00 on tuesday

    Some of the tasks defined by the RUN_TASK parameters require more information. The following steps specify what you must provide.

    1. The run_audit.sh task runs dmaudit. For this task, provide a time_expression. If it detects any errors, the run_audit.sh task mails the errors to the e-mail address defined by the ADMIN_EMAIL parameter of the base object (described in “Configuring the Base Object”).

    2. The run_scan_logs.sh task scans the DMF log files for errors. For this task, provide a time_expression. If the task finds any errors, it sends e-mail to the e-mail address defined by the ADMIN_EMAIL parameter of the base object.

    3. The run_remove_logs.sh task removes logs that are older than the value you provide by specifying the LOG_RETENTION parameter. You also provide a time_expression to specify when you want the run_remove_logs.sh to run. In the example, log files more than 4 weeks old are deleted each day at 1:00 A.M. Valid values for LOG_RETENTION are a number followed by m[inutes], h[ours], d[ays], or w[eeks].

      The run_remove_journals.sh task removes journals that are older than the value you provide by specifying the JOURNAL_RETENTION parameter. You also provide a time_expression to specify when you want the run_remove_journal.sh to run. In the example, journal files more than 4 weeks old are deleted each day at 1:00 A.M. Valid values for JOURNAL_RETENTION are a number followed by m[inutes], h[ours], d[ays], or w[eeks].


      Note: The run_remove_journals.sh and run_remove_logs.sh tasks are not limited to the daemon logs and journals; they also clear the logs and journals for MSP(s) and LS(s).


    4. The run_copy_databases.sh task makes a copy of the DMF databases. For this task, in addition to a value for time_expression, provide a value for the DATABASE_COPIES parameter that specifies one or more directories. If you specify multiple directories, breaking the directories among multiple disk devices minimizes the chance of losing all the copies of the database.

      The task copies a snapshot of the current DMF databases to the directory with the oldest copy. Integrity checks are done on the databases before the copy is saved. If the checks fail, the copy is not saved, and the task sends e-mail to the e-mail address defined by the ADMIN_EMAIL parameter of the base object.

Procedure 2-6. Configuring the dump_tasks Object

The following steps explain how to define a dump_tasks object. You can change the object name itself (dump_tasks) to be any name you like.

Do not change the script names.

You may comment out the RUN_TASK parameters for any tasks you do not want to run.

The following example would configure a dump_tasks object:

define  dump_tasks
        TYPE                  taskgroup
        RUN_TASK              $ADMINDIR/run_full_dump.sh on \
                                sunday at 00:01
        RUN_TASK              $ADMINDIR/run_partial_dump.sh on \
                                monday tuesday wednesday thursday \
                                friday saturday at 00:01
        RUN_TASK              $ADMINDIR/run_hard_deletes.sh 
                                at 23:00
#
        DUMP_TAPES            HOME_DIR/tapes
        DUMP_RETENTION        4w
        DUMP_DEVICE           SILO_2
        DUMP_MIGRATE_FIRST    yes
        DUMP_INVENTORY_COPY   /save/dump_inventory
enddef

  1. Define the object to have the same name that you provided for the TASK_GROUPS parameter of the daemon object. In the example it is dump_tasks.

  2. Ensure that TYPE is set to taskgroup. There is no default.

  3. Configure the RUN_TASK parameters. See step 3 in Procedure 2-5, for information about $ADMINDIR and time_expression.

    The following steps specify the information you must provide for the tasks to run correctly.

    1. The run_full_dump.sh task runs a full backup of DMF-managed file systems at intervals specified by the time_expression. In the example, the full backup is run each week on Sunday morning one minute after midnight.

    2. The run_partial_dump.sh task backs up only those files in DMF-managed file systems that have changed since the time a full backup was completed. The backups are run at intervals specified by the time_expression. In the example, it is run each day of the week except Sunday, at one minute after midnight.

    3. The run_hard_deletes.sh task removes from the database any files that have been deleted but can no longer be restored because the backup tapes have been recycled (that is, it hard-deletes the files). The backup tapes are recycled at the time interval set by the DUMP_RETENTION parameter described in the next step. For more information on hard-deleting files, see “Soft- and Hard-deletes” in Chapter 7.

    4. Manage the data from the file system dumps by configuring the following parameters:

      DUMP_TAPES
      DUMP_RETENTION
      DUMP_DEVICE
      DUMP_MIGRATE_FIRST
      DUMP_INVENTORY_COPY
      DUMP_FILE_SYSTEMS
      DUMP_VSNS_USED

      The DUMP_TAPES parameter specifies the path of a file that contains tape volume serial numbers (one per line) for the dump tasks to use.

      The DUMP_RETENTION parameter specifies how long the backups of the file system will be kept before the tapes are reused. This is also the value used by the run_hard_deletes.sh task to determine how old soft-deleted database entries must be before removing them from the database. Valid values for DUMP_RETENTION are a number followed by m[inutes], h[ours], d[ays], or w[eeks].

      The DUMP_DEVICE parameter specifies the name of the device object in the configuration file that defines how to mount the tapes that the dump tasks will use. See “Device Objects”, for information about device objects.

      If you set DUMP_MIGRATE_FIRST to YES, the dmmigrate command is run before the dumps are done to ensure that all migratable files are migrated, thus reducing the tapes needed for the dump. The default is NO.

      The DUMP_INVENTORY_COPY parameter specifies the path name of a directory into which are copied the xfsdump(1M) inventory files for the backed-up file systems.

      The DUMP_FILE_SYSTEMS parameter specifies one or more file systems to dump. If not specified, the task dumps all the file systems configured in the configuration file. Use this parameter only if your site needs different dump policies (such as different dump times) for different file systems. It is safest not to specify a value for this parameter and therefore dump all file systems configured.

      The DUMP_VSNS_USED parameter is optional. It specifies the name of a file to which the tasks that dump the file systems will append the VSN, one per line, of each volume used by xfsdump. If you don't specify this parameter, the task uses /dev/null as the file name.

The dump_tasks object employs scripts that call the xfsdump(1m) command in conjunction with the dmtape DMF support program. This mechanism gives you flexible and efficient use of a predetermined set of backup volumes that are automatically allocated to the xfsdump program as needed during the backup. In order to allow you an equally flexible and efficient method for restoring files backed up by the dump_tasks object, the dmxfsrestore(8) command should be used any time a restore is required for a dump_tasks-managed file system. Please see the dmxfsrestore(8) man page for more information on running the command.

Configuring File Systems

You must have a filesystem object for each file system that can migrate files.

The filesystem object parameters are as follows:

Parameter 

Value

TYPE 

filesystem (type of object)

MESSAGE_LEVEL 

Specifies the highest message level number that will be written to the automated space management log (autolog). It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2. For more information on message levels, see “General Message Log File Format”.

MIGRATION_LEVEL 

Sets the level of migration service for the file system. Valid values are none (no migration), user (only user-initiated migration), or auto (automated space management). The migration level actually used for the file system is the lesser of the MIGRATION_LEVEL of the daemon object and this value. The default is auto.

POLICIES 

Specifies the names of the configuration objects defining policies for this file system.

TASK_GROUPS 

Names the task groups that contain tasks the daemon should run. They are configured as objects of TYPE taskgroup. There is no default. Currently there are no defined tasks for file systems.

The following example defines a filesystem object:

define  /c
        TYPE                    filesystem
        MIGRATION_LEVEL         user
        POLICIES                fs_msp
enddef

Procedure 2-7. Configuring filesystem Objects

The following steps explain pertinent information for configuring the above filesystem object:

  1. Ensure that define has a value that is the mount point of the file system you want DMF to manage. Do not use the name of a symbolic link. There is no default.

  2. Ensure that TYPE is set to filesystem. There is no default.

  3. The MIGRATION_LEVEL parameter determines the level of service for migration to offline media. Migration from offline media (either automatic or manual recall) is not affected by the value of MIGRATION_LEVEL.

    Configure MIGRATION_LEVEL to be one of none , user, or auto. none means no migration will take place on this file system. user means that users/administrators can perform dmput(1) or dmmigrate(8) commands but no other migration will take place. auto means that you want automated space management on this file system.

    The default is auto, which means that you do not need to include this line unless you want to specify user or none. See “DMF Policies” and Procedure 2-8, for information about configuring automated space management policies.


    Note: user is the highest migration level that can be associated with a real-time partition.


  4. Use the POLICIES parameter to declare one or more migration policies that will be associated with this file system. Policies are defined with policy objects (see “DMF Policies”). The POLICIES parameter is required; there is no default value. A policy can be unique to each DMF-managed file system, or it can be reused numerous times.

DMF Policies

A policy object is used to specify a migration policy. Three types of migration policies can be defined: automated space management, file weighting, and MSP selection.

The following rules govern the use of policy objects with the POLICIES parameter of the filesystem object:

  • The POLICIES parameter for a file system must specify one and only one MSP selection policy.

  • If the MIGRATION_LEVEL for a file system is auto, the POLICIES parameter for that file system must specify one and only one space management policy.

  • You do not need to specify a weighting policy if the default values are acceptable.

  • You can configure one policy that defines all three groups of policy parameters (space management, file weight, and MSP/VG selection) and share that policy among all the file systems. Alternatively, you might create an MSP/VG selection policy for all file systems and a space management policy (including weighting parameters) for all file systems.

The policy object parameters described below are grouped by function.

Automated Space Management Parameters

DMF lets you automatically monitor file systems and migrate data as needed to prevent file systems from filling. This capability is implemented in DMF with a daemon called dmfsmon(8). After the dmfsmon daemon has been initiated, it will begin to monitor the DMF-managed file system to maintain the level of free space configured (in the configuration file).

Chapter 3, “Automated Space Management”, describes automated space management in more detail.

The following are parameters that control automated space management on a file system:

Parameter

Description

TYPE

policy (type of object)

FREE_DUALSTATE_FIRST

When set to on, dmfsmon will free dual-state files before freeing files it will have to migrate first. The default is off.

FREE_SPACE_DECREMENT

Percentage of file system space by which dmfsmon will decrement FREE_SPACE_MINIMUM if it cannot find enough files to migrate so that the value is reached. The decrement is applied until a value is found that dmfsmon can achieve. If space later frees up, the FREE_SPACE_MINIMUM is reset to its original value. Valid values are between 1 and the value of FREE_SPACE_TARGET. The default is 2.

FREE_SPACE_MINIMUM

Minimum percentage of free file system space that dmfsmon maintains. dmfsmon will begin to migrate files when the available free space for the file system falls below this percentage value. This parameter is required; there is no default.

FREE_SPACE_TARGET

Percentage of free file system space that dmfsmon will try to achieve if free space reaches or falls below FREE_SPACE_MINIMUM. This parameter is required; there is no default.

MIGRATION_TARGET

Percentage of file system capacity that DMF maintains as a reserve of dual-state files whose online space can be freed if free space reaches or falls below FREE_SPACE_MINIMUM. dmfsmon tries to make sure that this percentage of the file system is migrated, migrating, or free after it runs to make space available. This parameter is required; there is no default.


Note: Ideal values for these parameters are highly site-specific, based largely on file system sizes and typical file sizes.



Note: The dump_tasks object employs scripts that call the xfsdump(1m) command in conjunction with the dmtape DMF support program. This mechanism gives you flexible and efficient use of a predetermined set of backup volumes that are automatically allocated to the xfsdump program as needed during the backup. In order to allow you an equally flexible and efficient method for restoring files backed up by the dump_tasks object, the dmxfsrestore(8) command should be used any time a restore is required for a dump_tasks-managed file system. Please see the dmxfsrestore(8) man page for more information on running the command.


File Weighting and MSP and/or VG Selection Parameters

An important part of automatic space management is selecting files to migrate and determining where to migrate them. When DMF is conducting automated space management, it derives an ordered list of files, called a candidate list, and migrates or frees files starting at the top of the list. The ordering of the candidate list is determined by weighting factors that are defined by using weighting-factor parameters in the configuration file.

DMF can be configured to have many MSPs or VGs. Each MSP/VG manages its own set of volumes. The MSP/VG selection parameters allow you to direct DMF to migrate files with different characteristics to different MSPs/VGs.

The file weighting and MSP/VG selection parameters can be used more than once to specify that different files should have different weighting or MSP/VG selection values.

The policy parameters for file weighting are as follows:

Parameter 

Description

AGE_WEIGHT 

Specifies a floating point constant and floating point multiplier to use to calculate the weight given to a file's age. AGE_WEIGHT is calculated as constant + (multiplier * file_age_in_days). If DMF cannot locate values for this parameter, it uses a floating point constant of 1 and a floating point multiplier of 1.

SPACE_WEIGHT 

Specifies a floating point constant and floating point multiplier to use to calculate the weight given to a file's size. SPACE_WEIGHT is calculated as constant + (multiplier * file_disk_space_in_bytes). If DMF cannot locate values for this parameter, it uses a floating point constant of 0 and a floating point multiplier of 0.

The parameter for MSP/VG selection follows:

Parameter 

Description

SELECT_MSP 

Specifies the MSP(s)/VG(s) to use for a file. You can list as many MSP/VG names as you have MSP/VG objects defined. A copy of the file will be migrated to each MSP/VG listed. The special MSP/VG name none means that the file will not be migrated. If you define more than one MSP/VG, separate the names with white space. As a convenience, you can use SELECT_VG instead of SELECT_MSP, and the object can contain a mixture of both forms. If no SELECT_MSP(or SELECT_VG ) parameter applies to a file, it will not be migrated. The parameters are processed in the order they appear in the policy. There is no default.

The file weighting and MSP selection parameters accept an optional when clause to restrict the set of files to which that parameter applies.

This clause has the form when expression.

expression can include any of the following simple expressions:

Expression

Description

age

Days since last modification or last access of the file, whichever is more recent

space

Number of bytes the file occupies on disk (always a multiple of the blocksize, which may be larger or smaller than the length of the file)

gid

Group ID of one or more files

uid

User ID of one or more files

Combine expressions by using and, or, and ().

Use the operators =, >, <, =>, =<, and in to specify values.

The following are examples of valid expressions:

space < 10m            (space used is less than 10 million bytes)
uid <= 123             (file's user ID is less than or equal to 123)
gid = 55               (file's group ID is 55)
age >= 15              (file's age is greater than or equal to 15 days)
space > 1g             (space used is greater than 1 billion bytes)
uid in (10 82-110 200) (file's user ID is 10, between 82 and 110, or 200)
(gid = 55 or uid <= 123) and age < 5
                                          (file's age is greater than 5 days and its
                                          group ID is 55 or its user ID is higher than 123)

Configuring Policies

The following procedures explain how to create policies for automated space management (including file weighting) and MSP/VG selection.

The following example defines a policy object for automated space management:

define  fs_space
        TYPE                     policy
        MIGRATION_TARGET         50
        FREE_SPACE_TARGET        10
        FREE_SPACE_MINIMUM       5
        FREE_DUALSTATE_FIRST     off

        AGE_WEIGHT 0    0.00     when age < 10
        AGE_WEIGHT 1    0.01     when age < 30
        AGE_WEIGHT 10   0.05     when age < 120
        AGE_WEIGHT 50   0.1

        SPACE_WEIGHT  0  0
enddef

Procedure 2-8. Configuring Objects for Automated Space Management

The following steps explain pertinent information for configuring the above policy object:

  1. Ensure that define has a value you set previously in the POLICIES parameter of a filesystem object. There is no default.

  2. Ensure that TYPE is set to policy. There is no default.

  3. Configure automated space management as follows:

    1. Configure MIGRATION_TARGET to an integer percentage of total file system space. DMF attempts to maintain this percentage as a reserve of space that is free or occupied by dual-state files that can be deleted if the file system free space reaches or falls below FREE_SPACE_MINIMUM. The default is 30.

    2. Configure FREE_SPACE_TARGET to an integer percentage of total file system space. DMF will try to achieve this level of free space when free space reaches or falls below FREE_SPACE_MINIMUM. The default is 20.

    3. Configure FREE_SPACE_MINIMUM to an integer percentage of the total file system space that DMF must maintain as free. DMF will begin to migrate files when the available free space for the configured file system reaches or falls below this percentage value. The default is 10.

    4. Configure FREE_DUALSTATE_FIRST to be on if you want DMF to free the space used by dual-state files before it migrates and frees regular files. The default is off.

  4. Configure the age and size weighting factors associated with a file when it is evaluated for migration as follows:

    1. The syntax of the AGE_WEIGHT parameter is a floating-point constant followed by a floating-point multiplier. The age weight is calculated as follows:

      constant + (multiplier x age_in_days)

      Add a when clause to select which files should use these values. DMF checks each AGE_WEIGHT parameter in turn, in the order they occur in the configuration file. If the when clause is present, DMF determines whether the file matches the criteria in the clause. If no clause is present, a match is assumed. If the file matches the criteria, the file weight is calculated from the parameter values. If they do not match, the next instance of that parameter is examined.

      An AGE_WEIGHT of 1 1.0 is used if no AGE_WEIGHT applies for a file.

      In the example policy, files that have been accessed or modified within the last 10 days have a weight of zero. File migration likelihood increases with the length of time since last access because the file will have a greater weight. The final line specifies that files which have not been accessed or modified in 120 days or more have a far greater weight than all other files.

    2. The syntax of SPACE_WEIGHT parameters is a floating-point constant followed by a floating-point multiplier. Calculate the space weight as follows:

      constant + (multiplier x file_disk_space_in_bytes)

      In the example policy, the size of the file does not affect migration because all files have SPACE_WEIGHT of zero.

      A SPACE_WEIGHT of 0 0.0 is used if no SPACE_WEIGHT applies for a file.

    3. Configure negative values to ensure that files are never automatically migrated. For example, you might want to set a minimum age for migration. The following parameter specifies that files that have been accessed or modified within 1 day are never automatically migrated:

      AGE_WEIGHT -1    0.0    when age <= 1 
        

      The following parameter specifies that small files are never automatically migrated:
      SPACE_WEIGHT -1    0    when space <= 4k   


    Note: DMF calculates the size weight and age weight separately. If either value is less than zero, the file is not automatically migrated or freed. Otherwise, the two values are summed to form the file's weight.


The following example defines a policy object for MSP/VG selection:

define  fs_msp
        TYPE                    policy
        SELECT_MSP none         when space < 65536
        SELECT_MSP cart1 cart2  when gid = 22
        SELECT_MSP cart1        when space >= 50m
        SELECT_VG cart2
enddef

Procedure 2-9. Configuring Objects for MSP/VG Selection

The following steps explain pertinent information for configuring the above policy object:


Note: The space expression references the number of bytes the file occupies on disk, which may be larger or smaller than the length of the file. For example, you might use the following line in a policy:
SELECT_VG none when space < 4096



Your intent would be to restrict files smaller than 4 Kbytes from migrating.

However, this line may actually allow files as small as 1 byte to be migrated, because while the amount of data in the file is 1 byte, it will take 1 block to hold that 1 byte. If your file system uses 4-Kbyte blocks, the space used by the file is 4096, and it does not match the policy line.

To ensure that files smaller than 4 Kbytes do not migrate, use the following line:
SELECT_MSP none when space <= 4096



  1. Ensure that define has a value that you set previously in the POLICIES parameter of the filesystem object. There is no default.

  2. Ensure that TYPE is set to policy. There is no default.

  3. Ensure that the MSP/VG name (or names) you specify as the first value of the SELECT_MSP (or SELECT_VG) parameter is either the name of an MSP you set previously in the MSP_NAMES (or LS_NAMES) parameter of the daemon object, or is the name of a VG that is a component of an LS named in that same parameter. There is no default.

  4. Configure MSP/VG selection criteria as follows:

    1. If you want to select an MSP or VG based on file size, use parameters such as the following, which send large files to cart1 and small files to cart2:

      SELECT_MSP cart1        when space >= 50m
      SELECT_MSP cart2        when space >= 65536

    2. If you want certain files to be copied to more than one MSP/VG, use syntax such as the following, which migrates all files that have a group ID of 22 to both of the configured MSPs/VGs:

      SELECT_MSP cart1 cart2  when gid = 22

      Separate multiple MSP/VG names with a blank space.

    3. If you want to ensure that some files are never migrated, you can designate the MSP/VG selection as none. The following line from the sample file ensures that files smaller than 65,536 bytes are not migrated:

      SELECT_MSP none         when space < 65536

Setting Up Tape MSPs

Each MSP you create must have an object defined in the configuration file, which refers to a device object describing the tape drives to be used. Normally, several MSPs share a single device object.

MSP Objects

The tape MSP entry has the following options:

Option 

Description

TYPE 

msp (type of object)

CACHE_DIR 

Directory in which the MSP stores chunks while merging them from sparse tapes. If you do not specify this parameter, DMF uses the value of TMP_DIR from the base object.

CACHE_SPACE 

Amount of disk space (in bytes) that dmatmsp can use when merging chunks from sparse tapes. During merging, small chunks from sparse tapes are cached on disk before being written to a tape. The default is 0, which causes all files to be merged via sockets. For more information, see Step 5.

CHILD_MAXIMUM 

Maximum number of child processes the MSP is allowed to fork. The maximum value is 100; the default is 4.

COMMAND 

Binary file to execute in order to initiate this MSP. For the tape MSP, this value must be dmatmsp.

DISK_IO_SIZE 

Transfer size (in bytes) used when reading from or writing to files within a DMF file system. The value must be between 4096 and 16 million (16m). The default is 65536.

HFREE_TIME 

Minimum number of seconds that a tape no longer containing valid data must remain unused before the MSP overwrites it. The default value is 172,800 seconds (2 days), and the minimum allowed value is 0.

When an MSP removes all data from a tape, it sets the hfree (hold free tape) flag bit in the tape's volume (VOL) database entry to prevent that tape from being immediately reused. The next time the MSP scans the database for volumes after HFREE_TIME seconds have passed, the MSP clears the hfree flag, allowing the tape to be rewritten. If HFREE_TIME is set to 0, the MSP will never clear hfree, so an unused tape will not be reused until you clear its hfree flag manually. For a description of how to set and clear the hfree flag manually, see the dmvoladm man page.

MAX_CACHE_FILE 

Largest chunk (in bytes) that will be merged using the merge disk cache. Larger files are transferred directly via a socket from the read child to the write child. The default is 25% of the CACHE_SPACE value. Valid values are 0 through the value of CACHE_SPACE.

MAX_CHUNK_SIZE 

Specifies that the MSP should break up large files into chunks no larger than this value (specified in bytes) as it writes data to tape. If a file is larger than this size, it is broken up into pieces of the specified size, and, depending on other activity, more than one write child may be used to write the data to tape. If MAX_CHUNK_SIZE is 0 (the default) the MSP only breaks a file into chunks when an end of volume is reached.

MAX_PUT_CHILDREN 

Maximum number of write child processes the MSP will schedule. The default and the maximum are the value of CHILD_MAXIMUM; the minimum is 1.

MERGE_CUTOFF 

Limit at which the MSP stops scheduling tapes for merging. This number refers to the sum of the active and queued children generated from gets, puts, and merges. The default is CHILD_MAXIMUM, which means that if sparse tapes are available, children will be created until there are CHILD_MAXIMUM children, thus using tape efficiently. However, if any recall requests arrive, they will be started before new merges.

Setting this number below CHILD_MAXIMUM reserves some tape units for recalls at the expense of merge efficiency. Setting this number above CHILD_MAXIMUM increases the priority of merges relative to recalls.

MESSAGE_LEVEL 

Highest message level number that will be written to the MSP log. It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2. For more information on message levels, see “General Message Log File Format”.

MIN_TAPES 

Minimum number of unused tapes that can exist in the MSP VOL database before operator notification. If the number of unused tapes falls below MIN_TAPES, the operator will be asked to add new tapes. The default is 10; the minimum is 0.

TAPE_TYPE 

Specifies the name of a device object that describes how the tapes are accessed and used. There is no default. The device object is described in “Device Objects”.

TASK_GROUPS 

Names the task groups that contain tasks the MSP should run. They are configured as objects of TYPE taskgroup. There is no default. See “Configuring Tape Maintenance Tasks ”, for more information.

TIMEOUT_FLUSH 

Minutes after which the MSP will flush files to tape. The default is 120 minutes.

The following example does not use all of the possible options for configuring a tape MSP; it defines two tape MSPs named cart1 and cart2.

define  cart1
        TYPE                    msp
        COMMAND                 dmatmsp
        TAPE_TYPE               SILO_1
        CACHE_SPACE             110m
        CHILD_MAXIMUM           3
        MESSAGE_LEVEL           2
        TASK_GROUP              msp_tasks
enddef
#
define  cart2
        TYPE                    msp
        COMMAND                 dmatmsp
        TAPE_TYPE               SILO_2
        CACHE_SPACE             50m
        CACHE_DIR               /cache
        MAX_CACHE_FILE          50m
        CHILD_MAXIMUM           10
        TASK_GROUP              msp_tasks
enddef

Procedure 2-10. Configuring Tape MSPs

The following steps explain pertinent information for configuring the msp objects:

  1. Ensure that define has a value that you set previously in the MSP_NAMES parameter of the daemon object. There is no default.

  2. Ensure that TYPE is set to msp. There is no default.

  3. Ensure that COMMAND is set to dmatmsp. There is no default.

  4. Define a TAPE_TYPE parameter that names the device type object for the MSP. There is no default. Use the value you set here in defining device objects. See “Device Objects”.

  5. Configure the CACHE_SPACE parameter to be at least twice the configured tape zone size. If you do not set this parameter, DMF will merge tapes via sockets, which means that the read and write children have to synchronize. Using CACHE_SPACE is far more efficient, especially for small files.

    The MSP is able to merge tapes more efficiently if it can stage most of the files to disk. Setting the CACHE_SPACE parameter tells the MSP how much disk space it can use. The MAX_CACHE_FILE parameter specifies the largest file it will place in the CACHE_SPACE. The default for CACHE_SPACE is 0, which causes all data to be transferred by sockets.

    See “Media Concepts” in Chapter 6, for more information on tape zone sizes.

  6. Configure the CHILD_MAXIMUM to be the number of tape drives this MSP can use. The default is 4, and the maximum is 24.

  7. Configure the MESSAGE_LEVEL of an MSP to be higher than 2 (the default) for debugging purposes only. Valid values are 0 to 6.

  8. Configure the MAX_CACHE_FILE to be the size (in bytes) of the largest chunk that will be merged using the merge cache space (defined by CACHE_SPACE). Large files are transferred directly via socket. The largest value you can use is the value of CACHE_SPACE, and the default is 25% of CACHE_SPACE.

  9. Configure the TASK_GROUPS parameter to the name(s) of the object(s) used to define how periodic maintenance tasks are completed. There is no default. See “Configuring Tape Maintenance Tasks ”, for more information.

Device Objects

Each tape device type name you use in the MSP or in the dump_tasks object should be defined as a device object in the configuration file. The parameters you define are based on which mounting service you intend to use.

The following parameters are common to all device objects:

Parameter 

Description

TYPE 

device (type of object)

BLOCK_SIZE 

Block size used when writing tapes from the beginning. The default depends upon the device, with DMF setting defaults as follows:

AMPEX DIS/DST       1199840
DLT                 131072
STK 9840            126976
Other devices       65536

LABEL_TYPE 

Label type used when writing tapes from the beginning. Possible values are nl (no label), sl (standard label, for IBM tapes), and al (ANSI label). The default is al.

MOUNT_SERVICE 

Specifies the mounting service to use. Supported values are openvault and tmf. This parameter is required; there is no default.

MSG_DELAY 

Specifies the number of seconds that all devices in the object can be down before an e-mail message is sent to the administrator and an error message is logged. The default is 0, which means that as soon as DMF notices that the mounting service is up and all of the drives are configured down, it will e-mail a message.

POSITIONING 

Specifies how the tape should be positioned to a zone; either skip or direct. skip specifies the use of tape mark skipping. direct specifies the use of block ID seek capability if the block ID is known. The default is direct.

POSITION_RETRY 

Level of retry in the event of a failure during zone positioning; one of none, lazy, or aggressive. lazy specifies that the MSP will retry if a reasonably fast alternative method of positioning is available. aggressive specifies that the MSP may try more costly and time-consuming alternatives. If the MSP is unable to position to a zone, the MSP aborts all recalls for files with data in that zone (however, DMF does not abort them if a copy exists in another MSP). The default is lazy.

VERIFY_POSITION 

Specifies whether the tape MSP write child should (prior to writing) verify that the tape is correctly positioned and that the tape was properly terminated by the last use. The default is to verify. Specifying no or off turns verification off; anything else ensures verification.

WRITE_CHECKSUM 

Specifies that tape block should be checksummed before writing. If a tape block has a checksum, it is verified when read. The default is on.

ZONE_SIZE 

Specifies approximately how much data the write child should put in a zone. The write child adds files and chunks to a zone until the data written equals or exceeds this value, at which time it writes a tape mark and updates the database. Smaller values allow faster recalls and better recoverability but poorer write performance. The MSP also uses zone size to determine when to start write children. The default is 50 MB.

Device Objects for OpenVault As Mounting Service

The device object may have the following parameters when it is configured for OpenVault:

Parameter 

Description

OV_ACCESS_MODES 

Specifies a list of access mode names that control how data is written to tape. The default value is readwrite when migrating and readonly when recalling. This parameter is optional.

OV_INTERCHANGE_MODES 

Specifies a list of interchange mode names that control how data is written to tape. This can be used to control whether the device compresses data as it is written. This optional parameter is applied when a tape is mounted or rewritten.

Examples of the use of these parameters are provided in Procedure 2-13.

OpenVault requires several configuration steps in addition to configuring the device object. They are described in “Using OpenVault for Tape MSPs and Drive Groups”.

Device Objects for TMF as Mounting Service

Tape mounting can be accomplished by using the Tape Management Facility (TMF). To use TMF as a mounting service, there are no required parameters that you must specify, but the TMF_TMMNT_OPTIONS parameter allows you to specify some tmmnt options:

Parameter 

Description

TMF_TMMNT_OPTIONS 

Specifies command options that should be added to the tmmnt command when mounting a tape.

DMF uses the -Z option to tmmnt, so options controlling block size and label parameters are ignored. Use the BLOCK_SIZE and LABEL_TYPE device parameters instead.

Use -g if the group name is different from the device object's name. Use -i to request compression.

The following example defines a device object for use with TMF:

define  SILO_3
        TYPE                    device
        MOUNT_SERVICE           tmf
        BLOCK_SIZE              131072
        LABEL_TYPE              sl
        TMF_TMMNT_OPTIONS       -g DLT
enddef

Procedure 2-11. Configuring Devices for TMF

The following steps explain pertinent information for configuring the device object for TMF:


Note: DMF uses the -Z option to tmmnt, so options controlling block size and label parameters would be ignored if you were to specify them for the TMF_TMMNT_OPTIONS parameter. Use the BLOCK_SIZE and LABEL_TYPE device parameters instead.


  1. Ensure that define has a value that you set previously in the TAPE_TYPE parameter of the msp object. There is no default.

  2. Ensure that TYPE is set to device. There is no default.

  3. Configure the MOUNT_SERVICE to be tmf.

  4. Configure the BLOCK_SIZE parameter to be the block size used when writing tapes from the beginning. In the example, 131072 is used because DLTs write more efficiently with this blocksize.

  5. Configure the LABEL_TYPE parameter to be the label type used when writing tapes from the beginning. In the example, sl is used to specify standard label for IBM tapes.

  6. Configure the TMF_TMMNT_OPTIONS parameter to specify command options that should be added to the tmmnt command when mounting a tape. In the example, the -g option specifies that the TMF tape group is DLT. If this option on this parameter had not been specified, DMF would have used the name of this device object (in the example, SILO_3).

Setting Up Library Servers

Each object shown in Figure 1-3, must have an object defined in the configuration file. The options shown in the following sections are only the most common. For the complete set, see the dmf.conf(5)man page. For a summary of the parameters and the object to which they apply, see Table 2-3.

Library Server Objects

The entry for a library server, one for each tape library, has the following options:

Option 

Description

TYPE  

libraryserver (type of object)

CACHE_DIR 

Directory in which the VG stores chunks while merging them from sparse tapes. If you do not specify this parameter, DMF uses the value of TMP_DIR from the base object.

CACHE_SPACE 

Amount of disk space (in bytes) that dmatls can use when merging chunks from sparse tapes. During merging, small chunks from sparse tapes are cached on disk before being written to a tape. The default is 0, which causes all files to be merged via sockets.

COMMAND 

Binary file to execute to initiate the LS. This value must be dmatls.

DRIVE_GROUPS 

Names one or more drive groups containing drives that the LS can use for mounting and unmounting volumes. They are configured as objects of type drivegroup. This parameter must be configured. There is no default.

The order of the values specified for this parameter is integral to the determination of the MSP or volume group from which the DMF daemon attempts to recall an offline file. If the offline file has more than one copy, DMF uses a specific order when it attempts to recall the file. It searches for a good copy of the offline file in MSP or Library Server order, from the dmdaemon object's MSP_NAMES or LS_NAMES parameter. If one of those names refers to a Library Server, it searches for the copy in drive group order, from the Library Server object's DRIVE_GROUPS parameter. It then searches for the copy in volume group order from the drivegroup object's VOLUME_GROUPS parameter.

MAX_CACHE_FILE 

Largest chunk (in bytes) that will be merged using the merge disk cache. Larger files are transferred directly via a socket from the read child to the write child. The default is 25% of the CACHE_SPACE value. Valid values are 0 through the value of CACHE_SPACE.

MESSAGE_LEVEL 

Highest message level number that will be written to the LS log, which includes messages from from the LS's components. It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2.

RUN_TASK 

See “Configuring Automated Maintenance Tasks”.

TASK_GROUPS 

Names the task groups that contain tasks the LS should run. They are configured as objects of TYPE taskgroup. There is no default.

WATCHER 

Names the resource watcher that the LS should run. They can be configured as objects of type resourcewatcher, but if the default parameters are acceptable, there is no need to do this. The default is no watcher.

Drive Group Objects

The entry for a drive group, one for each pool of interchangeable drives in a single library, has the following options:

Option 

Description

TYPE 

drivegroup (type of object)

BLOCK_SIZE 

Block size used when writing tapes from the beginning. The default depends upon the device, with DMF setting defaults as follows:

AMPEX DIS/DST       1199840
DLT                 131072
STK 9840            126976
Other devices       65536

DISK_IO_SIZE 

Transfer size (in bytes) used when reading from or writing to files within a DMF file system. The value must be between 4096 and 16 million (16m). The default is 65536.

DRIVE_MAXIMUM 

Maximum number of drives within this drive group that the LS is allowed to attempt to use simultaneously. This can be more or less than the number of drives the LS can physically detect. The maximum is 100; the default is 100 for Drive Groups. If a negative value is specified for DRIVE_MAXIMUM, the Drive Group uses the sum of the number of available drives and DRIVE_MAXIMUM.

DRIVE_SCHEDULER 

Names the resource scheduler that the DG should run for the scheduling of tape drives. They are configured as objects of type resourcescheduler. The default is a resource scheduler of default type and parameters. For the defaults, see “Resource Scheduler Objects”.

DRIVES_TO_DOWN 

An integer value that controls the number of "bad" drives the drive group is allowed to try to configure down. When more than this number are down, whether due to the DG or to external influences such as the system administrator, the DG does not attempt to disable any more. The default of 0 prevents the DG from disabling any.

LABEL_TYPE 

Label type used when writing tapes from the beginning. Possible values are nl (no label), sl (standard label, for IBM tapes), and al (ANSI label). The default is al.

MAX_MS_RESTARTS 

Specifies the maximum number of times DMF can attempt to restart the mounting service (TMF or OpenVault) without requiring administrator intervention. The default and recommended values are 1 for TMF and 0 for OpenVault.

MOUNT_SERVICE 

Specifies the mounting service to use. Possible values are openvault and tmf. The default is openvault.

MOUNT_SERVICE_GROUP 

Specifies the name by which the drive group's devices are known to the mounting service. In the case of TMF, this is the device group name that would be used with the -g option on the tmmnt command. For OpenVault, this is the drive group name that is specified by the ov_drivegroup command.

MOUNT_TIMEOUT 

Specifies the maximum number of minutes to wait for a tape to be mounted. Default is zero, which means forever.

If a tape mount request waits for longer than this period of time, the DG attempts to stop and restart the mount service, in an attempt to force the hanging subsystem to resume normal operation, or to fail solidly.

Do not make this value too restrictive, as any non-LS tape activity (including MSPs and xfsdump) can legitimately delay a VG's tape mount, which could result in this timeout being exceeded.

MSG_DELAY 

Specifies the number of seconds that all drives in the drive group can be down before an e-mail message is sent to the administrator and an error message is logged. The default is 0, which means that as soon as DMF notices that the mounting service is up and all of the drives are configured down, it will e-mail a message.

OV_ACCESS_MODES 

Specifies a list of access mode names that control how data is written to tape. The default value is readwrite when migrating and readonly when recalling. This parameter is optional.

OV_INTERCHANGE_MODES (Open Vault MOUNT_SERVICE only) 

Specifies a list of names to be provided to OpenVault for the firstmount clause when mounting a tape. Use compression to request compression. By default, this list is empty.

POSITIONING 

Specifies how the tape should be positioned. The values can be skip, direct, or data. skip means to use tape mark skipping to the zone. direct means to use block ID seek capability to the zone if the block ID is known. data means the same as direct when the tape is being written. When the tape is being read, data means that the read child will try to determine the block ID of the data being read, and use the block ID seek capability to position there. The default depends on the type of drive, and is either direct or data. If data positioning is specified for a drive whose default is direct, the block ID is calculated by adding an estimate of the number of blocks from the start of the zone to the data being recalled and the block ID of the start of the zone. Not all drives use this format for block ID.

POSITION_RETRY 

Specifies the level of retry in the event of a failure during zone positioning. The values can be none, lazy, or aggressive. With lazy, the VG retries if a reasonably fast alternative means of positioning is available. With aggressive, the VG can try more costly and time consuming alternatives. If the VG is unable to position to a zone, all recalls for files with data in that zone are aborted by the VG (though not by DMF if a copy exists in another VG). The default is lazy, to give the best overall recall time. If you are having trouble getting data from tape, you might want to try aggressive.

REINSTATE_DRIVE_DELAY 

Specifies the number of minutes after which a drive that was configured down by the DG will be automatically reinstated and made available for use again. Zero means it should be left disabled indefinitely. The default is 1440 (one day).

REINSTATE_VOLUME_DELAY 

Specifies the number of minutes after which a volume that had its HLOCK flag set by DMF will be automatically reinstated and made available for use again. Zero means they should be left disabled indefinitely. The default is 1440 (one day).

RUN_TASK 

See “Configuring Automated Maintenance Tasks”.

TASK_GROUPS 

Names the task groups that contain tasks the DG should run. They are configured as objects of TYPE taskgroup. There is no default.

TMF_TMMNT_OPTIONS (TMF MOUNT_SERVICE only) 

Specifies command options that should be added to the tmmnt command when mounting a tape. DMF uses the -Z option to tmmnt to ignore options controlling block size and label parameters. Use the BLOCK_SIZE and LABEL_TYPE device parameters instead. Unlike a tape MSP, there is no need for a -g option here. If it is provided, it must match the value of the MOUNT_SERVICE_GROUP parameter. To request compression, use -i. Options that are ignored are -a, -b, -c, -D, -f, -F, -l, -L, -n, -o, -O, -p, -P, -q, -R, -t, -T, -U, -v, -V, -w, -x , and -X.

VERIFY_POSITION 

Specifies whether the LS write child should (prior to writing) verify that the tape is correctly positioned and that the tape was properly terminated by the last use. The default is to verify. Specifying no or off turns verification off; anything else ensures verification.

VOLUME_GROUPS 

Names the volume group(s) containing volumes that can be mounted on any of the drives within this drive group. They are configured as objects of type volumegroup. This parameter must be configured. There is no default.

The order of the values specified for this parameter is integral to the determination of the MSP or volume group from which the DMF daemon attempts to recall an offline file. If the offline file has more than one copy, DMF uses a specific order when it attempts to recall the file. It searches for a good copy of the offline file in MSP or Library Server order, from the dmdaemon object's MSP_NAMES or LS_NAMES parameter. If one of those names refers to a Library Server, it searches for the copy in drive group order, from the Library Server object's DRIVE_GROUPS parameter. It then searches for the copy in volume group order from the drivegroup object's VOLUME_GROUPS parameter.

WRITE_CHECKSUM 

Specifies that tape block should be checksummed before writing. If a tape block has a checksum, it is verified when read. The default is on.

Volume Group Objects

The entry for a volume group, one for each pool of tape volumes of the same type, usable on the drives of the associated DG, and which is capable of holding at most one copy of user files, has the following options:

Option 

Description

TYPE 

volumegroup (type of object)

ALLOCATION_GROUP 

Name of an allocation group that serves as a source of additional volumes if a volume group runs out of media. Normally, one allocation group is configured to serve multiple volume groups. As a volume's hfree flag is cleared (see HFREE_TIME below) in a volume group, it is immediately returned to the allocation group subject to the restrictions imposed by the configuration parameters ALLOCATION_MAXIMUM and ALLOCATION_MINIMUM. The administrator must ensure that volumes in the allocation group are mountable on drives in the same drive group as any volume group that references the allocation group. It is an error to assign an ALLOCATION_GROUP name that is the same as an existing volume group name. The ALLOCATION_GROUP defines a logical pool of volumes rather than an actual operational volume group.

As allocation groups have no configurable parameters, they have no configuration stanzas of their own; a reference to them in a volume group's ALLOCATION_GROUP parameter is all that is needed to activate them. A volume group that does not define ALLOCATION_GROUP will not use one.

ALLOCATION_MAXIMUM 

Maximum size in number of volumes to which a volume group can grow by borrowing volumes from its allocation group. The minimum value is zero, the maximum is infinity, and the default is infinity. If the volume group already contains ALLOCATION_MAXIMUM or more volumes, no additional volumes are borrowed from the allocation group. If no allocation group is defined, this parameter is meaningless.

ALLOCATION_MINIMUM 

Minimum size in number of volumes to which a volume group can shrink by returning volumes to its allocation group. The minimum value is zero, which is the default, and the maximum is the current value of ALLOCATION_MAXIMUM. If the volume group already contains ALLOCATION_MINIMUM or fewer volumes, no additional volumes are returned to the allocation group. If no allocation group is defined, this parameter is meaningless.

DRIVE_MAXIMUM 

Maximum number of drives within this drive group that this VG is allowed to use simultaneously. The value actually used is the least of the DG's DRIVE_MAXIMUM, this VG's DRIVE_MAXIMUM and the number of drives the DG can physically detect. The maximum is 100; the default is the drive group's DRIVE_MAXIMUM.

HFREE_TIME 

Minimum number of seconds that a tape no longer containing valid data must remain unused before the VG overwrites it. The default value is 172,800 seconds (2 days), and the minimum allowed value is 0.

When an LS removes all data from a tape, it sets the hfree (hold free tape) flag bit in the tape's volume (VOL) database entry to prevent that tape from being immediately reused. The next time the LS scans the database for volumes after HFREE_TIME seconds have passed, the LS clears the hfree flag, allowing the tape to be rewritten. If HFREE_TIME is set to 0, the LS will never clear hfree, so an unused tape will not be reused until you clear its hfree flag manually. For a description of how to set and clear the hfree flag manually, see the dmvoladm man page.

MAX_CHUNK_SIZE 

Specifies that the VG should break up large files into chunks no larger than this value (specified in bytes) as it writes data to tape. If a file is larger than this size, it is broken up into pieces of the specified size, and, depending on other activity, more than one write child may be used to write the data to tape. If MAX_CHUNK_SIZE is 0 (the default) the VG breaks a file into chunks only when an end of volume is reached.

MAX_PUT_CHILDREN 

Specifies the maximum number of write child (dmatwc) processes that will be simultaneously scheduled for the volume group. The maximum value is the value of DRIVE_MAXIMUM for the associated drive group. The minimum value is 1. The default is the value that the volume group uses for DRIVE_MAXIMUM.

MERGE_CUTOFF 

Specifies a limit at which the VG will stop scheduling tapes for merging. This number refers to the sum of the active and queued children generated from gets, puts, and merges. The default value for this option is the value used by the volume group for DRIVE_MAXIMUM. This means that if sparse tapes are available, the volume group will create DRIVE_MAXIMUM number of children, thus using tape resources efficiently. However, if any recall requests arrive for that volume group, they will be started before new merges. Setting this number below DRIVE_MAXIMUM, in effect, reserves some tape units for recalls at the expense of merge efficiency. Setting this number above DRIVE_MAXIMUM increases the priority of merges relative to recalls.

MIN_VOLUMES 

Minimum number of unused volumes that can exist in the library server's volume database for this volume group without operator notification. If the number of unused volumes falls below MIN_VOLUMES, the operator is asked to add new volumes. The default is 10; the minimum is zero. If a volume group has an allocation group configured, MIN_VOLUMES is applied to the sum of the number of unused volumes in the volume group and in its allocation group subject to any ALLOCATION_MAXIMUM restrictions.

PUTS_TIME 

Specifies the minimum number of seconds a volume group waits after it has requested a drive for a write child before it tells a lower priority child to go away. The default is 3600 seconds.

READ_TIME 

Specifies the interval, in seconds, after which the volume group will evaluate whether a read child should be asked to go away (even if it is in the middle of recalling a file) so that a higher priority child can be started. If READ_TIME is 0, the volume group will not do this evaluation. The default is 0.

RUN_TASK 

See “Configuring Automated Maintenance Tasks”.

TASK_GROUP 

Names the task groups that contain tasks the VG should run. They are configured as objects of TYPE taskgroup. There is no default.

TIMEOUT_FLUSH 

Minutes after which the VG will flush files to tape. The default is 120 minutes.

ZONE_SIZE 

Specifies approximately how much data the write child should put in a zone. The write child adds files and chunks to a zone until the data written equals or exceeds this value, at which time it writes a tape mark and updates the database. Smaller values allow faster recalls and better recoverability but poorer write performance. The VG also uses zone size to determine when to start write children. The default is 50 MB.

Resource Scheduler Objects

The entry for a resource scheduler (RS), one for each drive group in a single library, has the following options:

Option 

Description

TYPE 

resourcescheduler (type of object)

ALGORITHM 

The resource scheduling algorithm (RSA) to be used. Two are currently supplied: a simple one called fifo, and a more flexible one called weighted_roundrobin (default).


Note: Sites can write their own RSA to meet specialized needs. Instructions can be found in the /usr/share/doc/dmf-version_number/info/sample/RSA.readme file.


MODULE_PATH 

The path name of a Dynamic Shared Object (library of runtime-loadable routines) that contains an RSA whose name was specified by the ALGORITHM parameter. The default is to use the built-in RSAs.

Other parameters are specific to a particular RSA. There are no parameters for fifo. For weighted_roundrobin, the following apply:

Option 

Description

PENALTY 

Used to reduce the priority of requests from a volume group that is not the next one preferred by the round robin algorithm. It is a multiplier in the range 0.0 - 1.0. Low values result in the urgency assigned by the VG being totally or partially ignored, and high values mean that the urgency is more important than selecting one whose turn ought to be next. The default is 0.7.

WEIGHT 

Used to assign a weighting to one or more volume groups. The ratio of these weightings to each other (within the one drive group) determines the number of opportunities the VG has to obtain drives when they are needed.

The weightings are integers in the range 1 - 99, and need not be unique. For efficiency reasons, small numbers are preferred, especially if large numbers of VGs are defined. Usually, there are multiple WEIGHT lines in the configuration, and a given VG might appear on more than one of them. In such cases, the sum of the weights is used as the effective weight for that VG. Any VGs that do not appear on a WEIGHT line are assigned the default of 5. If there are no WEIGHT lines, all VGs will use this default, resulting in a strict round-robin behavior.

WEIGHT has the following format:


WEIGHT weight vg1 vg2 ...

Resource Watcher Objects

The entry for a resource watcher (RW) is needed only if you wish to change its default parameters; a reference to an RW by the LS is sufficient to activate it. The RW has the following options:

Option 

Description

TYPE 

resourcewatcher (type of object)

HTML_REFRESH 

The refresh rate (in seconds) of the generated HTML pages. The default is 60.

Example

The following code example does not use all of the possible options for configuring an LS. It defines an LS containing a default resource watcher (RW) and one drive group (DG), which in turn contains two volume groups (VGs) sharing an allocation group (AG), and a resource scheduler (RS) to give one VG twice the priority than the other when competing for drives.

The VG objects are slightly different, reflecting that the first one handles all of the recalls in normal circumstances as well as migrations, but the second is usually write-only.

define  ls1
                TYPE                    libraryserver
                COMMAND                 dmatls
                DRIVE_GROUPS            dg1
                CACHE_SPACE             500m
                TASK_GROUPS             ls_tasks
                WATCHER                 rw
        enddef

        define  dg1
                TYPE                    drivegroup
                VOLUME_GROUPS           vg_prim vg_sec
                MOUNT_SERVICE           openvault
                MOUNT_SERVICE_GROUP     drives
                OV_INTERCHANGE_MODES    compression
                DRIVE_SCHEDULER         rs
                DRIVES_TO_DOWN          2
                REINSTATE_DRIVE_DELAY   60
        enddef

        define  rs
                TYPE                    resourcescheduler
                WEIGHT                  10      vg_prim
                WEIGHT                  5       vg_sec
        enddef

        define  vg_prim
                TYPE                    volumegroup
                ALLOCATION_GROUP        ag
        enddef

        define  vg_sec
                TYPE                    volumegroup
                ALLOCATION_GROUP        ag
                DRIVE_MAXIMUM           2
        enddef

The steps in Procedure 2-12 explain pertinent information for configuring each of the LS objects in the previous example.

Procedure 2-12. Configuring a Library Server and Its Components

  1. Ensure that define has a value that you set previously in the LS_NAMES or MSP_NAMES parameter of the daemon object. There is no default.

  2. Ensure that TYPE is set to libraryserver. There is no default.

  3. Ensure that COMMAND is set to dmatls. There is no default.

  4. Specify a DRIVE_GROUPS parameter that names a collection of interchangeable tape drives. The assumption in this example is that there is only one such group. There is no default.

  5. To tell the LS how much disk space it can use, set the CACHE_SPACE parameter. The LS can merge tapes more efficiently if it can stage most of the files to disk. Configure the CACHE_SPACE parameter to be at least twice the configured tape zone size. The default for CACHE_SPACE is 0, which causes all data to be transferred by sockets. For more information on tape zone sizes, see “Media Concepts” in Chapter 6.

  6. Configure the TASK_GROUPS parameter to the name(s) of the object(s) used to define how periodic maintenance tasks are completed. There is no default. For more information, see “Configuring Tape Maintenance Tasks ”.

  7. To observe LS operation through a web browser, define a resource watcher (RW). You need only a reference. Define an RW object only if you want to change its default parameters.

    Assuming that SPOOL_DIR was set in the base object to be /dmf/spool, the URL to use in this example is file://dmf/spool/ls/_rw/ls.html. Text files are generated in the same directory as the HTML files.

  8. Define the drive group (DG) referenced in step 4. Note that there is no COMMAND line; a DG is not an independent program, but a component of an LS.

  9. Define the volume groups (VGs) using the drives managed by this DG with the VOLUME_GROUPS parameter.

  10. Specify the use of OpenVault. Because Open Vault is the default mounting service, this line can be omitted.

  11. Specify the name that the mounting service uses to refer to this group of drives. When using OpenVault, the MOUNT_SERVICE_GROUP line specifies the OpenVault Drive Group to be used. Note that OpenVault uses the same term as does DMF to describe a group of interchangeable tape devices, but the two uses are separate. Their names need not match, though it may be less confusing if they do.

    If using TMF, the MOUNT_SERVICE_GROUP line names the TMF device group name.

  12. Use the OV_INTERCHANGE_MODES and TMF_TMMNT_OPTIONS lines to specify that the drives (OpenVault and TMF, respectively) should be used in compression mode.

  13. Override the default resource scheduler (RS) behavior by referring to an object called rs, to be defined later.

  14. Allow the DG to configure at most two drives down temporarily for 60 minutes for recovery from I/O errors if the drives are faulty and if doing so will result in a more reliable operation. When this happens, the administrator is e-mailed so that maintenance can be performed.

  15. In the rs object, specify that when there are more requests for tape drives than there are drives in the DG, VG vg_prim is to be given access twice as often as vg_sec. The ratio of the numbers is important, but the exact values are not.

  16. Define the VGs. The VOLUME_GROUPS parameter of the DG object and the SELECT_LS or SELECT_MSP lines in the filesystem object(s) refer to them.

  17. Define a common allocation group (AG) called ag. AGs have no configurable parameters, so they have no defining object; just a reference is sufficient. Note that use of an AG is not mandatory.

  18. Include any other VG parameters that you require. For example, one of the previous steps specified that the secondary VG vg_sec can use, at most, two tape drives, so that other drives in this DG are immediately available for use by vg_prim when it needs them.

Using OpenVault for Tape MSPs and Drive Groups

This section describes the steps you must take to configure OpenVault for a tape MSP or a drive group. You must execute OpenVault commands, create security key files, and edit the DMF configuration file.

Procedure 2-13. Configuring DMF to Use OpenVault

The following procedure describes how to make OpenVault and DMF work together. When using OpenVault 1.5 and later versions, you can use the ov_admin script to enable the DMF application. When using earlier versions of OpenVault, you can use the setup script. See the OpenVault Operator's and Administrator's Guide for a description of this script.


Note: The procedure that follows assumes that before you complete the steps described, the OpenVault server is configured and all drives and libraries are configured and OpenVault is running.


  1. On the OpenVault server, add DMF as both a privileged and unprivileged OpenVault application for this host.

    When using versions of OpenVault prior to 1.5, use the setup script, menu item 1, submenu 5.

    When using OpenVault 1.5 or later versions, use the ov_admin script, and select the menu option that allows you to manage applications. Create the DMF application, and activate both a privileged and unprivileged instance of it.

    The application name should be dmf (in lowercase). The instance name should be dmf@hostname where dmf is in lowercase, and hostname is the output of the command hostname -s. For example:

    % hostname -s
    system1

    In this case, dmf@system1 would be the instance name.

  2. Add the DMF application as a valid user to appropriate OpenVault drive groups. The OpenVault drive groups that DMF uses must contain only fungible drives. That is, the drives in the OpenVault drive group must have identical characteristics and accessibility, so that any volume that can be mounted and written on one of the drives can also be mounted and read on any of the other drives within the group. Failure to provide identical mounting and accessibility characteristics to all drives in an OpenVault drive group used by a DMF MSP or Library Server might result in tape mount failures.

    When configuring DMF tape MSPs, ensure that the value for CHILD_MAXIMUM does not exceed the number of drives in the OpenVault drive group.

    When using OpenVault 1.4.x or earlier releases, it is preferable that you use the OpenVault setup script, menu item 2, submenu 7. When using OpenVault 1.5 or later, choose the appropriate item from the ov_admin menu. If for some reason you cannot use the setup or ov_admin script, you can enter the command manually, as follows:

    ov_drivegroup -a drive_group -A dmf

  3. Add DMF as a valid application to appropriate cartridge groups.

    For OpenVault versions prior to 1.5, it is preferable that you use the OpenVault setup script, menu item 2, submenu 8.

    For OpenVault 1.5 and later, the ov_admin script allows you to specify the cartridge groups when the DMF application is created, or after creation of the DMF application, you can choose the menu option that allows you to manage cartridge groups.

    If for some reason you cannot use the setup or the ov_admin script, you can enter the command manually, as follows:

    ov_cartgroup -a tape_group -A dmf

  4. Configure the base object for use with OpenVault:

    define  base
            TYPE            base
            HOME_DIR        /dmf/home
    .
    .
    .
            OV_KEY_FILE     /dmf/home/ovkeys
            OV_SERVER       hostname
    enddef

    1. Configure the OV_KEY_FILE parameter name of the key file that holds security information for OpenVault. It is usually located in HOME_DIR and called ovkeys.

    2. Configure the OV_SERVER parameter to the value returned by the hostname(1) command on the machine on which the OpenVault server is running. This parameter only applies when OpenVault is used as the mounting service. The default value is the host name of the machine on which you are running.

  5. Use the dmov_keyfile(8) command to create the file defined by the OV_KEY_FILE parameter. This command will prompt you for the privileged and unprivileged keys that you defined in step 1.

  6. (This step does not apply to library servers). Configure the MSP's device object for use with OpenVault, as follows:

    define  timber
            TYPE                    device
            MOUNT_SERVICE           openvault
            OV_ACCESS_MODES         readwrite
            OV_INTERCHANGE_MODES    compression
            ZONE_SIZE               200m
    enddef

    1. Ensure that define has a value that you set previously in the TAPE_TYPE parameter of the msp object. There is no default.

    2. Configure TYPE to be device. There is no default.

    3. Configure the MOUNT_SERVICE parameter to be openvault.

    4. Configure the OV_ACCESS_MODES parameter to be a list of access mode names that control how the tape is used. The parameter is optional. The default value is readwrite when migrating and readonly when recalling. Use this parameter to force readwrite.

      The other possible values that OpenVault can use are not configurable in DMF: for rewind/norewind, DMF uses rewind; for variable/fixed, DMF uses variable.

    5. Configure the OV_INTERCHANGE_MODES parameter to be a list of interchange mode names that control how data is written to tape. This can be used to control whether the device compresses data as it is written. This parameter is optional.

      To specify that you want data compressed, use OV_INTERCHANGE_MODES compression

      To force all tapes to be written as DLT4000, use OV_INTERCHANGE_MODES DLT4000

      This parameter is applied when a tape is first used or rewritten.

    6. Configure other parameters relevant to your site. The example sets the ZONE_SIZE parameter to 200 MB. The target zone size is a major factor in determining how much data is written before writing a tape mark and updating the MSP database. Here, the tapes used by the MSP will, in general, have more data written in a zone than DMF uses as a default. Smaller values allow faster recalls and better recovery, but they cause poorer write performance than larger values. The default is 50 MB. See “Media Concepts” in Chapter 6, for more information on how tape zone sizes are determined.

  7. (This step does not apply to MSPs). Configure the library server's drive group object for use with OpenVault. In the drive group object, use the following steps:

    1. Configure the MOUNT_SERVICE parameter to be openvault.

    2. Configure the MOUNT_SERVICE_GROUP parameter to be the name of the OpenVault drive group, as seen in the output from the ov_stat -d command.

    3. Configure the OV_ACCESS_MODES parameter to be a list of access mode names that control how the tape is used. The parameter is optional. The default value is readwrite when migrating and readonly when recalling. Use this parameter to force readwrite.

      The other possible values that OpenVault can use are not configurable in DMF: for rewind/norewind, DMF uses rewind; for variable/fixed, DMF uses variable.

    4. Configure the OV_INTERCHANGE_MODES parameter to be a list of interchange mode names that control how data is written to tape. This can be used to control whether the device compresses data as it is written. This parameter is optional.

      To specify that you want data compressed, use OV_INTERCHANGE_MODES compression

      To force all tapes to be written as DLT4000, use OV_INTERCHANGE_MODES DLT4000

      This parameter is applied when a tape is first used or rewritten.

  8. Make the appropriate cartridges accessible to the MSPs, AGs, or VGs by assigning the cartridges to the DMF application in OpenVault. To do this, you must know the following:

    • Cartridge type name. To determine the cartridge types allowed by a given drive, enter the following:

      ov_stat -c -D  drive | grep base

      The fourth column shown in the output is the cartridge type.

    • Cartridge group. To determine the possible cartridge groups, enter the following:

      ov_cartgroup -l -A dmf

    1. If you already have tapes defined in your MSP or LS database, tell OpenVault about these tapes by entering one of the following:

      dmov_makecarts -g  cartgroup -t  carttype mspname
      dmov_makecarts -g  cartgroup -t  carttype lsname
      dmov_makecarts -g  cartgroup -t  carttype -v vg1, vg2 lsname

      You can replace any of the references to a VG previously mentioned with an allocation group. If the -v parameter is omitted, all VGs and AGs in the specified LS will be processed.

    2. If there are unmanaged cartridges in an OpenVault managed library, you can import the unmanaged cartridges, assign them to DMF, and add them to a database by entering one of the following:

      dmov_loadtapes -l library -g  cartgroup -t carttype  mspname
      dmov_loadtapes -l library -g  cartgroup -t carttype  vgname
      dmov_loadtapes -l library -g  cartgroup -t carttype  agname

      This command will invoke a vi(1) session. In the vi(1) session, delete any cartridges that you do not want added to the database.

    3. If neither of the above cases are appropriate, you can manually configure the cartridges. The following commands can be useful in this effort:

      • To list cartridges in a library, enter the following:

        ov_stat -s -L library

      • To list information on cartridges known to OpenVault, enter the following:

        ov_lscarts -f '.*'

      • To import cartridges into OpenVault and optionally assign them to DMF use the ov_import command.

      • To assign a cartridge known to OpenVault to an application, use the ov_vol command with the -n option.

Using TMF tapes with Tape MSPs and Drive Groups

Use one of the following dmvoladm(8) commands to add tapes to the MSP and/or LS databases:

dmvoladm  -m mspname  -c 'create vsn001-vsn010'
dmvoladm  -l lsname  -c 'create vsn001-vsn010 vg vgname'
dmvoladm  -l lsname  -c 'create vsn001-vsn010 vg agname'

Note that an allocation group is specified by the vg option, just like a volume group.

There is no special procedure to inform TMF of the tapes' existence. TMF assumes that every tape it deals with is in the library or can be provided by an operator, as needed.

Configuring Tape Maintenance Tasks

You can configure parameters for how the tape MSP or LS daemon performs the following maintenance tasks:

  • Creating tape reports (the run_tape_report.sh and run_compact_tape_report.sh tasks)

  • Merging sparse tapes (the run_tape_merge.sh task and the THRESHOLD, VOLUME_LIMIT, and DATA_LIMIT parameters)

  • Stopping tape merges at a specified time (the run_merge_stop.sh task)

For each of these tasks, you can configure when the task is run. For the second task, you must provide more information such as what determines that a tape is sparse and how many tapes can be merged at one time.


Note: The run_remove_journals.sh and run_remove_logs.sh tasks are configured as part of the daemon_tasks object, but these tasks also clear the MSP/LS logs and journals. These tasks are described in “Configuring Daemon Maintenance Tasks”.

Table 2-1, provides a summary of automated maintenance tasks.


The following example explains how to define the msp_tasks object. You can change the object name itself (msp_tasks) to be any name you like.

Do not change the path names or task names.

You may comment out the RUN_TASK parameters for any tasks you do not want to run.

define	msp_tasks
	TYPE			taskgroup
	RUN_TASK	$ADMINDIR/run_tape_report.sh at 00:10
#
	RUN_TASK	$ADMINDIR/run_tape_merge.sh on \
                   monday wednesday friday at 2:00
THRESHOLD          50
#VOLUME_LIMIT      20
#DATA_LIMIT        5g
#
RUN_TASK $ADMINDIR/run_merge_stop.sh at 5:00

Procedure 2-14. Configuring the msp_tasks Object

  1. Define the object to have the same name that you provided for the TASK_GROUPS parameter of the tape msp object. In the example it is msp_tasks.

  2. Ensure that TYPE is set to taskgroup. There is no default.

  3. Configure the RUN_TASK parameters. DMF substitutes $ADMINDIR in the path with the /usr/lib/dmf directory. When the task is run, it is given the name of the object that requested the task as the first parameter and the name of the task group (in this case msp_tasks) as the second parameter. The task itself may use the dmconfig(8) command to obtain further parameters from either of these objects.

    The RUN_TASK parameters require that you provide a time_expression.

    The time_expression defines when a task should be done. It is a schedule expression that has the following form:

    [every n  period] [at hh:mm[:ss] ...] [on day ...]

    period is one of minute[s], hour[s], day[s], week[s], or month[s].

    n is an integer.

    day is a day of the month (1 through 31) or day of the week (sunday through saturday).

    The following are examples of valid time expressions:

    at 2:00
    every 5 minutes
    at 1:00 on tuesday

    The following steps specify the information you must provide for the tasks to run correctly.

    1. The run_tape_report.sh generates a report on the tapes in the MSP tape pool and on MSP activity. In the example, it runs every day at 10 minutes after midnight.

    2. The run_tape_merge.sh task merges sparse tapes. Specify the criteria that DMF uses to determine that a tape is sparse, as follows:

      • Use the THRESHOLD parameter to set an integer percentage of active data on a tape. DMF will consider a tape to be sparse when it has less than this percentage of data that is still active.

      • Use the VOLUME_LIMIT parameter to set the maximum number of tape volumes that can be selected for merging at one time.

      • Use the DATA_LIMIT parameter to set the maximum amount of data (in bytes) that should be selected for merging at one time.

    3. As this might become cumbersome when there are large numbers of VGs configured, an alternative has been provided to run_tape_merge.sh, called run_merge_mgr.sh. This script establishes the needs of the VGs for more tapes, using their MIN_VOLUMES parameters as a guide to expected requirements. The script processes the most urgent ones first, minimizing interference with the production workload. To use this script, perform the following steps:

      1. Define a taskgroup, which is referred to by the drivegroup object (not the VG or LS object).

      2. Specify a RUN_TASK parameter for run_merge_mgr.sh in the taskgroup, and optionally, another for run_merge_stop.sh. You can also specify MESSAGE_LEVEL, THRESHOLD, VOLUME_LIMIT, and DATA_LIMIT parameters.

      3. Ensure that the LS object that refers to this DG has a Resource Watcher defined via the WATCHER parameter.

      4. For each VG, confirm that the value of its MIN_VOLUMES parameter is realistic.

      run_merge_mgr.sh is not available for use with MSPs because it requires the Resource Watcher feature of the Library Server.

    4. Use the run_merge_stop.sh task to shut down volume merging (tape merging) at a time you specify by using a time_expression. This task is an alternative to using the VOLUME_LIMIT and DATA_LIMIT parameters to stop merging at specified points. In the example, the limit parameters are commented out because run_merge_stop.sh is used to control volume merging.

Library Server and MSP Database Records

After you have added the tape MSP/LS information to the configuration file, use the dmvoladm(8) command with the -m option to create any missing directories with the proper labels and to create the volume (VOL) and catalog (CAT) records in the MSP/LS database.

You can follow the steps in Procedure 2-15 for all the tape MSPs/LSs you have defined.


Caution: Each tape MSP/LS must have a unique set of volume serial numbers.


Procedure 2-15. Creating MSP/LS Database Records

The following procedure is shown as an example that assumes you have an MSP called cart1.

  1. If you have not yet done so, set your PATH environment variable to include /usr/sbin. ( See “Setting PATH Environment Variables”.)

  2. Enter the following command and it will respond as shown:

    % dmvoladm -m cart1
    dmvoladm: at rdm_open - created database atmsp_db
    adm: 1>
    The response is an informational message indicating that dmvoladm could not open an existing MSP database, so it is creating a new and empty one. You should get this message the first time you use dmvoladm for an MSP, but never again. The next line is the prompt for dmvoladm directives.

  3. Assume that you will use 200 tapes of type CART with standard labels PA0001 through PA0200.

    After the prompt, enter the following directive:

    adm:1> create PA0001-PA0200

    After entering this directive, you will receive 200 messages, one for each entry created, beginning with the following:

    VSN PA0001 created.
    VSN PA0002 created.

  4. Use the following dmvoladm directive to list all of the tape VSNs in the newly created library:

    adm:2> list all

    Note:: The dmvoladm tapesize field is purely for site documentation and is not used by the MSP. The blocksize field documents the value used when the tape is first written or rewritten. It should not be changed in the database; however, if you want another value, change the BLOCK_SIZE nnn configuration parameter of the device object.


  5. Issue the dmvoladm quit directive to complete setting up the MSP.

    adm:3> quit

Procedure 2-16. Creating LS Database Records

The following procedure is shown as an example that assumes you have an LS called ls1. This LS contains a volume group named vg_pri.

  1. If you have not yet done so, set your PATH environment variable to include /usr/sbin. ( See “Setting PATH Environment Variables”.)

  2. Enter the following command and it will respond as shown:

    % dmvoladm -m ls1
    dmvoladm: at rdm_open - created database libsrv_db
    adm: 1>
    The response is an informational message indicating that dmvoladm could not open an existing LS database, so it is creating a new and empty one. You should get this message the first time you use dmvoladm for an LS, but never again. The next line is the prompt for dmvoladm directives.

  3. Assume that you will use 200 tapes with standard labels VA0001 through VA0200.

    After the prompt, enter the following directive:

    adm:1> create VA0001-VA0200 vg vg_pri

    Note that you are specifying the volume group vg_pri for the tapes being added. It is also valid to specify an allocation group name instead of a volume group name.

    After entering this directive, you will receive 200 messages, one for each entry created, beginning with the following:

    VSN VA0001 created.
    VSN VA0002 created.

  4. Use the following dmvoladm directive to list all of the tape VSNs in the newly created library:

    adm:2> list all

  5. Issue the dmvoladm quit directive to complete setting up the LS.

    adm:3> quit

Setting up FTP MSPs

To enable a file transfer protocol (FTP) MSP, include a name for it on the MSP_NAMES parameter in the daemon object and define an msp object for it in the DMF configuration file.

DMF has the capability to use an FTP MSP to convert a non-DMF file server to DMF with a minimal amount of down time for the switch over, and at site-determined pace. Contact your customer service representative for information about technical assistance with file server conversion.

An FTP MSP object has the following options (defaults are provided here or in Procedure 2-18):

Parameter 

Description

TYPE 

msp (type of object)

CHILD_MAXIMUM 

Maximum number of child processes the MSP is allowed to fork. The default is 4; the maximum is 100.

COMMAND 

Binary file to execute in order to initiate this MSP. For the FTP MSP, this value must be dmftpmsp.

DISK_IO_SIZE 

Transfer size (in bytes) used when reading from or writing to files within a DMF file system. The value must be between 4096 and 16 million (16m). The default is 65536.

FTP_ACCOUNT 

Account ID to use when migrating files to the remote system.

FTP_COMMAND 

Additional commands to send to the remote system. There may be more than one instance of this parameter.

FTP_DIRECTORY 

Directory to use on the remote system.

FTP_HOST 

Internet host name of the remote machine on which files are to be stored.

FTP_PASSWORD 

File containing the password to use when migrating files to the remote system. This file must be owned by root and be only accessible by root.

FTP_PORT 

Port number of the FTP server on the remote system. The default value is the value configured for ftp in the services file.

FTP_USER 

User name to use when migrating files to the remote system.

GUARANTEED_DELETES 

Number of child processes that are guaranteed to be available for processing delete requests. If CHILD_MAXIMUM is nonzero, its value must be greater than the sum of GUARANTEED_DELETES and GUARANTEED_GETS. The default is 1.

GUARANTEED_GETS 

Number of child processes that are guaranteed to be available for processing dmget(1) requests. If CHILD_MAXIMUM is nonzero, its value must be greater than the sum of GUARANTEED_DELETES and GUARANTEED_GETS. The default is 1.

IMPORT_DELETE  

Specifies if the MSP should honor hard-delete requests from the DMF daemon. This parameter applies only if IMPORT_ONLY is set to on. Set IMPORT_DELETE to on if you wish files to be deleted on the destination system when hard deletes are processed.

IMPORT_ONLY 

Specifies that the MSP is used for importing only. Set this parameter ON when the data is stored as a bit-for-bit copy of the file and needs to be available to DMF as part of a conversion. The MSP will not accept dmput(1) requests when this parameter is enabled. The MSP will, by default, ignore hard-delete requests when this parameter is enabled.

When the DMF daemon recalls a file from an IMPORT_ONLY MSP, it makes the file a regular file rather than a dual-state file, and it soft-deletes the MSP's copy of the file.

MESSAGE_LEVEL 

Specifies the highest message level number that will be written to the MSP log. It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2. For more information on message levels, see “General Message Log File Format”.

MVS_UNIT 

Defines the storage device type on an MVS system. This must be specified when the destination is an MVS system. Valid values are 3330, 3350, 3380, and 3390.

NAME_FORMAT 

Remote file name template that creates names for files stored on remote machines. The default is username/bfid (the bfid is the full bfid in hexadecimal).

TASK_GROUPS 

Names the task groups that contain tasks the MSP should run. They are configured as objects of TYPE taskgroup. There is no default. Currently there are tasks defined only for the tape MSP.

The MSP checks the DMF configuration file just before it starts child processes. If the DMF configuration file changed, it is reread.

If CHILD_MAXIMUM is non-zero, its value must be greater than the sum of GUARANTEED_DELETES and GUARANTEED_GETS.

The parameters COMMAND, FTP_HOST, FTP_USER, FTP_PASSWORD, and FTP_DIRECTORY must be present.

The MVS_UNIT parameter affects only IBM machines; they are further described in the dmf.conf(5) man page.


Note: The MSP will not operate if the FTP_PASSWORD file is readable by anyone other than root.

The default value for NAME_FORMAT creates a unique file name and a subdirectory on the remote machine. The subdirectory is named after the file's owner at the time of migration. This default works well if the remote machine runs an operating system based on UNIX. The default may not work at all if the remote machine runs an operating system that is not based on UNIX. The unique file name is the encoded bit-file identifier (bfid) of the file.

Possible substitutes you may specify to create the NAME_FORMAT file name are as follows:

%1 substitutes for the first 32 bits of the bfid in hexadecimal
%2 substitutes for the second 32 bits of the bfid in hexadecimal
%3 substitutes for the third 32 bits of the bfid in hexadecimal
%4 substitutes for the fourth 32 bits of the bfid in hexadecimal
%b substitutes for the full bfid in hexadecimal
%u substitutes for the user name of the file owner
%U substitutes for the user ID of the file owner
%g substitutes for the group name of the file
%G substitutes for the group ID of the file
%% substitutes for the literal % character
%d substitutes the current day of month (two digits)
%H substitutes the current hour (two digits)
%m substitutes the current month (two digits)
%M substitutes the current minute (two digits)
%S substitutes the current second (two digits)
%y substitutes the last two digits of the current year

The %1, %2, %3, %4, and %b substitutions generate uppercase hexadecimal numbers. The NAME_FORMAT must include either %b or %2, %3, %4 in some combination.

The date- and time-related substitutions allow sites with very large numbers of files to spread them over a large number of directories, to minimize subsequent access times.

The following example defines an FTP MSP:

define  ftp
        TYPE                    msp
        COMMAND                 dmftpmsp
        FTP_HOST                fileserver
        FTP_USER                dmf
        FTP_ACCOUNT             dmf.disk
        FTP_PASSWORD            /dmf/ftp/password
        FTP_DIRECTORY           ftpmsp
        FTP_COMMAND             umask 022
enddef

Procedure 2-17. Configuring the ftp Object

The following steps explain pertinent information for configuring the ftp object:

  1. Ensure that define has a value that you set previously in the MSP_NAMES parameter of the daemon object. There is no default.

  2. Ensure that TYPE is set to msp. There is no default.

  3. Ensure that COMMAND is set to dmftpmsp. There is no default.

  4. Set the FTP_USER parameter to the user name to use on the remote FTP server during session initialization. There is no default.

  5. Set the FTP_ACCOUNT parameter (if necessary) to the account to use on the remote FTP server during session initialization. Most FTP servers do not need account information. When account information is required, its nature and format will be dictated by the remote machine and will vary from operating system to operating system. There is no default.

  6. Set the FTP_PASSWORD parameter to the name of the file containing the password to be used on the remote FTP server during session initialization. This file must be owned by root and only be accessible by root. In the example, the password for the user dmf on fileserver is stored in the file /dmf/ftp/password. There is no default.

  7. Set the FTP_DIRECTORY parameter to the directory into which files will be placed on the remote FTP server. There is no default.

  8. If necessary, specify commands to the remote machine's FTP daemon. In the example, the umask for files created is set to 022 (removes write permission for group and other). There is no default.

Setting up Disk MSPs

To enable a disk MSP, include a name for it on the MSP_NAMES parameter in the daemon object and define an msp object for it in the DMF configuration file.

As with the FTP MSP, you can use a disk MSP to convert a non-DMF file server to DMF with a minimal amount of down time for the switch over, and at a site-determined pace. Contact your customer service representative for information about technical assistance with file server conversion.

A disk MSP object has the following options:

Parameter 

Description

TYPE 

msp (type of object)

CHILD_MAXIMUM 

Maximum number of child processes the MSP is allowed to fork. The default is 4; the maximum is 100.

COMMAND 

Binary file to execute in order to initiate this MSP. For the disk MSP, this value must be dmdskmsp.

DISK_IO_SIZE 

Transfer size (in bytes) used when reading from or writing to files within a DMF file system. The value must be between 4096 and 16 million (16m). The default is 65536.

GUARANTEED_DELETES 

Number of child processes that are guaranteed to be available for processing delete requests. The default is 1.

GUARANTEED_GETS 

Number of child processes that are guaranteed to be available for processing dmget(1) requests. The default is 1.

IMPORT_DELETE  

Applies only if IMPORT_ONLY is set to on. Set IMPORT_DELETE to on if you wish files to be deleted in STORE_DIRECTORY when hard deletes are processed.

IMPORT_ONLY 

MSP is used for importing only. Set this parameter on when the data is stored as a bit-for-bit copy of the file and needs to be available to DMF as part of a conversion. The MSP will not accept dmput(1) requests when this parameter is enabled. The MSP will, by default, ignore hard delete requests when this parameter is enabled.

MESSAGE_LEVEL 

Specifies the highest message level number that will be written to the MSP log. It must be an integer between 0 and 6; the higher the number, the more messages written to the log file. The default is 2. For more information on message levels, see “General Message Log File Format”.

NAME_FORMAT 

Template that creates names for files in STORE_DIRECTORY. The default is username/bfid (the bfid is the full bfid in hexadecimal).

STORE_DIRECTORY 

Specifies the directory used to store files for this MSP.

TASK_GROUPS 

Names the task groups that contain tasks the MSP should run. They are configured as objects of TYPE taskgroup. There is no default. Currently there are tasks defined only for the tape MSP.

The default value for NAME_FORMAT creates a unique file name and a subdirectory in the STORE_DIRECTORY. The subdirectory is named after the file's owner at the time of migration. The unique file name is the encoded bit-file identifier of the file. For details, see the description for FTP MSP in the section titled “Setting up FTP MSPs”.

The following example describes setting up a disk MSP:

define  dsk
        TYPE                    msp
        COMMAND                 dmdskmsp
        CHILD_MAXIMUM           8
        GUARANTEED_DELETES      3
        GUARANTEED_GETS         3
        STORE_DIRECTORY         /remote/dir
enddef

Procedure 2-18. Configuring the dsk Object

The following steps explain pertinent information for configuring the dsk object:

  1. Ensure that define has a value that you set previously in the MSP_NAMES parameter of the daemon object. There is no default.

  2. Ensure that TYPE is set to msp. There is no default.

  3. Ensure that COMMAND is set to dmdskmsp. There is no default.

  4. Set the CHILD_MAXIMUM parameter to the maximum number of child processes you want this MSP to be able to fork. The default is 4. The example allows 8.

  5. Set the GUARANTEED_DELETES parameter to the number of child processes that are guaranteed to be available for processing delete requests. The default is 1. The example allows 3.

  6. Set the GUARANTEED_GETS parameter to the number of child processes that are guaranteed to be available for processing dmget requests. The default is 1. The example allows 3.

  7. Set the STORE_DIRECTORY to the directory where files will be stored. This parameter is required; there is no default.

Verifying the Configuration

To verify the DMF configuration, run the dmcheck(8) script. This command checks the configuration file object and parameters, and reports on inconsistencies.

Initializing DMF

The DMF daemon database is created in HOME_DIR/daemon_name as dbrec.dat, dbrec.keys, pathseg.dat, and pathseg.keys. The database definition file (in the same directory) that describes these files and their record structure is named dmd_db.dbd. The database journal file is named dmd_db.yyyymmdd.[hhmmss]. It is created in the directory JOURNAL_DIR/daemon_name (JOURNAL_DIR is specified by the JOURNAL_DIR configuration parameter).

The inst(8) utility on IRIX systems and the rpm(8) utility on Linux systems set up system startup and shutdown scripts to start and stop DMF. You can start the DMF daemon manually by executing the dmfdaemon(8) command and stop it by executing the dmdstop(8) command.

After dmfdaemon is activated, the dmget(1) and dmput(1) user commands can be used to manage file system space manually.

General Message Log File Format

The dmfdaemon, dmlockmgr, dmfsmon, MSP, and LS processes all create message files that are used to track various DMF events. These DMF message log files use the same general naming convention and message format. The message log file names are created using the extension .yyyymmdd, which represents the year, month, and day of log file creation.

Each line in a message log file begins with the time the message was issued, an optional message level, the process ID number, and the name of the program that issued the message.

The optional message level is described below. The remainder of the line contains informative or diagnostic information. The following sections provide details about each of these log files:

Messages in the dmdlog, dmlocklog, and msplog files contain a 2-character field immediately following the time field in each message that is issued. This feature helps to categorize the messages and can be used to extract error messages automatically from these logs. Because the only indication of DMF operational failure may be messages written to the DMF logs, recurring problems can go undetected if you do not check the logs daily.

Possible message types for autolog, dmdlog, msplog, and dmlocklog are defined as follows; the corresponding message level in the configuration file is also provided:

Table 2-2. DMF Log File Message Types

Field

Message type

Message level

E

Error

0

O

Ordinary

0

I

Informative

1

V

Verbose

2

1

Debug level 1

3

2

Debug level 2

4

3

Debug level 3

5

4

Debug level 4

6


Parameter Table

Table 2-3 lists the parameters that can be specified in the /etc/dmf/dmf.conf file and the objects to which they apply. The legend for the abbreviated column headings appears at the end of the table. Please note that the most up-to-date list of parameters is in the dmf.conf(5) man page.

Table 2-3. Parameters for the DMF configuration file

Parameter

BS

DM

DV

DG

DP

FS

FP

LS

PO

RS

RW

TP

TG

VG

 

ADMIN_EMAIL

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

AGE_WEIGHT

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

ALGORITHM

 

 

 

 

 

 

 

 

 

X

 

 

 

 

 

ALLOCATION_GROUP

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

ALLOCATION_MAXIMUM

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

ALLOCATION_MINIMUM

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

BLOCK_SIZE

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

CACHE_DIR

 

 

 

 

 

 

 

X

 

 

 

X

 

 

 

CACHE_SPACE

 

 

 

 

 

 

 

X

 

 

 

X

 

 

 

CHILD_MAXIMUM

 

 

 

 

X

 

X

 

 

 

 

X

 

 

 

COMMAND

 

 

 

 

X

 

X

X

 

 

 

X

 

 

 

DATABASE_COPIES

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DATA_LIMIT

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DISK_IO_SIZE

 

 

 

X

X

 

X

 

 

 

 

X

 

 

 

DRIVES_TO_DOWN

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

DRIVE_GROUPS

 

 

 

 

 

 

 

X

 

 

 

 

 

 

 

DRIVE_MAXIMUM

 

 

 

X

 

 

 

 

 

 

 

 

 

X

 

DRIVE_SCHEDULER

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

DUMP_DEVICE

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DUMP_FILE_SYSTEMS

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DUMP_INVENTORY_COPY

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DUMP_MIGRATE_FIRST

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DUMP_RETENTION

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

DUMP_TAPES

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

FREE_DUALSTATE_FIRST

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

FREE_SPACE_DECREMENT

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

FREE_SPACE_MINIMUM

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

FREE_SPACE_TARGET

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

FTP_ACCOUNT

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_COMMAND

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_DIRECTORY

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_HOST

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_PASSWORD

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_PORT

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

FTP_USER

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

GUARANTEED_DELETES

 

 

 

 

X

 

X

 

 

 

 

 

 

 

 

GUARANTEED_GETS

 

 

 

 

X

 

X

 

 

 

 

 

 

 

 

HFREE_TIME

 

 

 

 

 

 

 

 

 

 

 

X

 

X

 

HOME_DIR

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

HTML_REFRESH

 

 

 

 

 

 

 

 

 

 

X

 

 

 

 

IMPORT_DELETE

 

 

 

 

X

 

X

 

 

 

 

 

 

 

 

IMPORT_ONLY

 

 

 

 

X

 

X

 

 

 

 

 

 

 

 

JOURNAL_DIR

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

JOURNAL_RETENTION

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

JOURNAL_SIZE

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

LABEL_TYPE

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

LICENSE_FILE

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

LOG_RETENTION

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

LS_NAMES

 

X

 

 

 

 

 

 

 

 

 

 

 

 

 

MAX_CACHE_FILE

 

 

 

 

 

 

 

X

 

 

 

X

 

 

 

MAX_CHUNK_SIZE

 

 

 

 

 

 

 

 

 

 

 

X

 

X

 

MAX_MS_RESTARTS

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

MAX_PUT_CHILDREN

 

 

 

 

 

 

 

 

 

 

 

X

 

X

 

MERGE_CUTOFF

 

 

 

 

 

 

 

 

 

 

 

X

 

X

 

MESSAGE_LEVEL

 

X

 

 

X

X

X

X

 

 

 

X

 

 

 

MIGRATION_LEVEL

 

X

 

 

 

X

 

 

 

 

 

 

 

 

 

MIGRATION_TARGET

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

MIN_TAPES

 

 

 

 

 

 

 

 

 

 

 

X

 

 

 

MIN_VOLUMES

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

MODULE_PATH

 

 

 

 

 

 

 

 

 

X

 

 

 

 

 

MOUNT_SERVICE

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

MOUNT_SERVICE_GROUP

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

MOUNT_TIMEOUT

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

MOVE_FS

 

X

 

 

 

 

 

 

 

 

 

 

 

 

 

MSG_DELAY

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

MSP_NAMES

 

X

 

 

 

 

 

 

 

 

 

 

 

 

 

MVS_UNIT

 

 

 

 

 

 

X

 

 

 

 

 

 

 

 

NAME_FORMAT

 

 

 

 

X

 

X

 

 

 

 

 

 

 

 

OV_ACCESS_MODES

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

OV_INTERCHANGE_MODES

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

OV_KEY_FILE

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

OV_SERVER

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

PENALTY

 

 

 

 

 

 

 

 

 

X

 

 

 

 

 

POLICIES

 

 

 

 

 

X

 

 

 

 

 

 

 

 

 

POSITIONING

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

POSITION_RETRY

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

PUTS_TIME

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

READ_TIME

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

REINSTATE_DRIVE_DELAY

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

REINSTATE_VOLUME_DELAY

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

RUN_TASK

 

 

 

 

 

 

 

 

 

 

 

 

X

*

 

SELECT_MSP

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

SELECT_VG

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

SITE_SCRIPT

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

SPACE_WEIGHT

 

 

 

 

 

 

 

 

X

 

 

 

 

 

 

SPOOL_DIR

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

STORE_DIRECTORY

 

 

 

 

X

 

 

 

 

 

 

 

 

 

 

TAPE_TYPE

 

 

 

 

 

 

 

 

 

 

 

X

 

 

 

TASK_GROUPS

 

X

 

 

X

X

X

X

 

 

 

X

 

X

 

THRESHOLD

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

TIMEOUT_FLUSH

 

 

 

 

 

 

 

 

 

 

 

X

 

X

 

TMF_TMMNT_OPTIONS

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

TMP_DIR

X

 

 

 

 

 

 

 

 

 

 

 

 

 

 

VERIFY_POSITION

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

VOLUME_GROUPS

 

 

 

X

 

 

 

 

 

 

 

 

 

 

 

VOLUME_LIMIT

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

WATCHER

 

 

 

 

 

 

 

X

 

 

 

 

 

 

 

WEIGHT

 

 

 

 

 

 

 

 

 

X

 

 

 

 

 

WRITE_CHECKSUM

 

 

X

X

 

 

 

 

 

 

 

 

 

 

 

ZONE_SIZE

 

 

X

 

 

 

 

 

 

 

 

 

 

X

 

* The run_tape_merge.sh and run_merge_stop.sh tasks and their associated parameters can be specified in the VG object.

Legend:

BS: Base
DM: Daemon
DV: Device
DG: Device group
DP: Disk MSP
FS: File system
FP: FTP MSP
LS: Library server
PO: Policy
RS: Resource scheduler
RW: Resource watcher
TP: Tape MSP
TG: Task group
VG: Volume group