Chapter 4. The DMF Daemon
Prev |  | Next

## Chapter 4. The DMF Daemon

The DMF daemon, `dmdaemon(8)`, is the core component of DMF. The daemon passes messages between commands, the MSPs, and the kernel. It also assigns bit file identifiers (bfids) to migrated files and maintains the DMF database entries for offline copies.

When DMF is started, the daemon database is automatically initialized. To start the daemon manually, use the `dmdaemon` command, as follows (assuming your `PATH` environment variable includes `/etc/dmf/dmbase/etc`):

    dmdaemon

Typically, `dmdaemon` should be called as part of the normal system startup procedure by using a direct call in a system startup script in the `/etc/rc2.d` directory.

After `dmdaemon` is activated, the `dmget(1)` and `dmput(1)` user commands can be used to manage file system space manually.

You can restart the daemon by using the `dmdaemon` command.

The following sections provide additional information about the daemon database and daemon processing.

## Daemon Processing

After initialization, `dmdaemon` performs the following steps:

1. Isolates itself as a daemon process.

2. Checks for the existence of other `dmdaemon` processes. If another `dmdaemon` exists, the newer one terminates immediately.

3. Initializes the `dmdaemon` log.

4. Opens the daemon database.

5. Initializes the daemon request socket.

6. Initiates the MSPs.

7. Enters its main request processing.

The daemon uses log files and journal files as described in “Daemon Logs and Journals”.

The main request processing section of the DMF daemon consists of the following sequence:

- The `select(2)` system call, which is used to wait for requests or for a default time-out interval

- A request dispatch switch to read and process requests detected by the `select` call

- A time processor, which checks activities (such as displaying statistics and running the administrator tasks) done on a time-interval basis

This processing sequence is repeated until a stop request is received from the `dmdstop(8)` command. When a normal termination is received, the MSPs are terminated, the database is closed, and the logs are completed.

A typical request to the daemon starts with communication from the requester. The requester is either the kernel (over the DMF device interface) or a user-level request (from the command pipe). A user-level command can originate from the automated space-management commands or from an individual user.

After receipt, the command is dispatched to the appropriate command processor within the daemon. Usually, this processor must communicate with an MSP before completing the specified request. The commands are queued within the daemon and are also queued to a specific group of database entries. All entries referring to the same file share the same bfid. The command is dormant until the reply from the MSP is received or the MSP terminates. When command processing is completed, a final reply is sent to the issuing process, if it still exists.

A final reply usually indicates that the command has completed or an error has occurred. Often, error responses require that you analyze the `dmdaemon` log to obtain a full explanation of the error. An error response issued immediately usually results from an invalid or incorrect request (for example, a request to migrate a file that has no data blocks). A delayed error response usually indicates a database, daemon, or MSP problem.

## DMF Daemon Database and `dmdadm(blank)`

The DMF daemon maintains a database that resides in the directory `HOME_DIR/daemon_name` (HOME_DIR is specified by the `HOME_DIR` configuration parameter). This database contains information about the offline copies of a given file, as well as some information about the original file. The database also contains the bit file identifier (bfid), which is assigned when the file is first migrated.

Other information maintained on a per-entry basis includes the following:

- File size (in bytes)

- MSP name and MSP recall path

- Date and time information, including the following:

  - Time at which the database record was created

  - Time at which the database record was last updated

  - A check time for use by the administrator

  - A soft-delete time, indicating when the entry was soft-deleted

- Original device and inode number

- Base portion of the original file name, if known

The `dmdadm(8)` command provides maintenance services for the daemon database.

`dmdadm` executes directives from `stdin` or from the command line when you use the `-c` option. All directives start with a directive name followed by one or more parameters. Parameters may be positional or keyword-value pairs, depending on the command. White space separates the directive name, keywords, and values.

When you are inside the `dmdadm` interface (that is, when you see the `adm command_number >` prompt), the command has a 30-minute timeout associated with it. If you do not enter a response within 30 minutes of the prompt having been displayed, the `dmdadm` session terminates with a descriptive message. This behavior on all the database administrative commands limits the amount of time that an administrator can lock the daemon and MSP databases from updates.

### `dmdadm` Directives

The `dmdadm` directives are as follows:

Directive |  | Description
`count` |  | Displays the number of records that match the expression provided.
`create` |  | Deletes an existing database record.
`dump` |  | Prints the specified database records to standard out in ASCII; each database field is separated by the pipe character (`|`).
`help` |  | Displays help.
`list` |  | Shows the fields of selected database records. You may specify which fields are shown.
`load` |  | Applies records to the database obtained from running the `dump` directive.
`quit` |  | Stops program execution after flushing any changed database records to disk. The abbreviation `q` and the string `exit` produce the same effect.
`set` |  | Specifies the fields to be shown in subsequent `list` directives.
`update` |  | Modifies an existing database record.

The syntax for the `dmdadm` directives is summarized as follows:

    count selection [limit]
    delete selection [limit]
    dump selection [limit]
    help
    list selection [format]
    load filename
    quit (or q or exit)
    set [format]
    update selection [limit] to fields...

The value for selection can be one of the following:

- A bfid or range of bfids

- The keyword `all`

- A period (`.`), which recalls the previous selection

- An expression involving any of the above, field value comparisons, `and`, `or`, or parentheses.

A field value comparison may use `<` (less than), `>` (greater than), `=` (equal to), `<=` (less than or equal to), or `>=` (greater than or equal to) to compare a field keyword to an appropriate value.

The syntax for selection is as follows:

    selection       ::=    or-expr
    or-expr        ::=    and-expr [ or or-expr  ]
    and-expr      ::=    nested-expr [ and and-expr  ]
    nested-expr    ::=    comparison | ( expression )
    comparison    ::=    bfid-range | field-keyword op  field-value
    op           ::=    < | > | = | >= | <=
    bfid-range      ::=    bfid [ - bfid] | [bfid - [bfid]] | key-macro
    key-macro     ::=    all
    field-keyword   ::=    name or abbreviation of the record field
    field-value     ::=    appropriate value for the field
    bfid          ::=    character representation of the bit file identifier

Thus valid selections could be any of the following:

    305c74b200000010-305c74b200000029
    7fffffff000f4411-
    -305c74b2000004c8
    all
    origsize>1m
    . and origage<7d
    mspkey 456 to origuid 2570

### `dmdadm` Field and Format Keywords

The field keywords listed below specify new values for fields. Some of the keywords are valid only if you also specify the `-u` option.

Keyword |  | Description
`checkage` (`ca`) |  | The time at which the database record was last checked; the same as `checktime`, except that it is specified as age. Valid only in unsafe (`-u`) mode.
`checktime` (`ct`) |  | The time at which the database record was last checked; an integer that reflects raw UNIX time. Valid only in unsafe (`-u`) mode.
`deleteage` (`da`) |  | The time at which the database record was soft-deleted; the same as `deletetime`, except that it is specified as age. Valid only in unsafe (`-u`) mode.
`deletetime` (`dt`) |  | The time at which the database record was soft-deleted; an integer that reflects raw UNIX time. Valid only in unsafe (`-u`) mode.
`mspname` (`mn`) |  | The name of the MSP with which the file is associated; a string of up to 8 characters. Valid only in unsafe (`-u`) mode.
`mspkey` (`mk`) |  | The string that the MSP can use to recall a database record; a string of up to 50 characters. Valid only in unsafe (`-u`) mode.
`origage` (`oa`) |  | Time at which the database record was created; the same as `origtime`, except that it is specified as age.
`origdevice` (`od`) |  | Original device number of the file; an integer.
`originode` (`oi`) |  | Original inode number of the file; an integer.
`origname` (`on`) |  | Base portion of the original file name; a string of up to 14 characters.
`origsize` (`os`) |  | Original size of the file; an integer.
`origtime` (`ot`) |  | Time at which the database record was created; an integer that reflects raw UNIX time.
`origuid` (`ou`) |  | Original user ID of the database record; an integer.
`updateage` (`ua`) |  | Time at which the database record was last updated; the same as `updatetime`, except that it is specified as age.
`updatetime` (`ut`) |  | Time at which the database record was last updated; an integer that reflects raw UNIX time.

The time field keywords (`checktime`, `deletetime`, `origtime`, and `updatetime`) have a value of either `now` or raw UNIX time (seconds since January 1, 1970). These keywords display their value as raw UNIX time. The value comparison `>` used with the date keywords means newer than the value given. For example, `>36000` is newer than 10AM on January 1, 1970, and `>852081200` is newer than 10AM on January 1, 1997.

The age field keywords (`checkage`, `deleteage`, `origage`, and `updateage`) let you express time as age. age is a string in a form such as `8w12d7h16m20s`, meaning 8 weeks, 12 days, 7 hours, 16 minutes, and 20 seconds old. The age keywords display their value as an integer followed by `w`, `d`, `h`, `m`, or `s` (weeks, days, hours, minutes, and seconds, respectively). The comparison `>` used with the age keywords means older than the value given (that is, `>5d` is older than 5 days).

The limit keywords restrict the records acted upon:

Keyword |  | Description
`recordlimit` (`rl`) |  | Limits the number of records acted upon to the value that you specify; an integer.
`recordorder` (`ro`) |  | Specifies the order that records are scanned; may be either `bfid` or `data`. `bfid` specifies that the records are scanned in bfid order. `data` specifies that the records are scanned in the order in which they are found in the database data file. `data` is more efficient for large databases, although it is essentially unordered.

The format keyword selects a format to use for the display. If, for example, you want to display fields in a different order than the default or want to include fields that are not included in the default display, you specify them with the `format` keyword. Values for `format` can be `default`, `keyword`, or a list of field keywords enclosed in quotation marks.

For any field that takes a byte count, you may append the letter `k`, `m`, or `g` (in either uppercase or lowercase) to the integer to indicate that the value is to be multiplied by one thousand, one million, or one billion, respectively.

The following is sample output from the `dmdadm` `list` directive; `recordlimit 20` specifies that you want to see only the first 20 records.

    adm 3>list all recordlimit 20
                BFID     ORIG        ORIG   ORIG MSP      MSP
                          UID        SIZE    AGE NAME     KEY
    ------------------------------------------------------------------------
    305c74b200000010    20934    69140480   537d silo1    88b49f
    305c74b200000013    26444      279290   537d silo1    88b4a2
    305c74b200000014    10634       67000   537d silo1    88b4a3
    305c74b200000016    10634   284356608   537d silo1    88b4a5
    305c74b200000018    10634     1986560   537d silo1    88b4a7
    305c74b20000001b    26444      232681   537d silo1    88b4aa
    305c74b20000001c    10015     7533688   537d silo1    88b4ab
    305c74b200000022    8964     23194990   537d silo1    88b4b1
    305c74b200000023    1294    133562368   537d silo1    88b4b2
    305c74b200000024    10634       67000   537d silo1    88b4b3
    305c74b200000025    10634   284356608   537d silo1    88b4b4
    305c74b200000026    10634     1986560   537d silo1    88b4b5
    305c74b200000027    1294      1114112   537d silo1    88b4b6
    305c74b200000028    10634       25270   537d silo1    88b4b7
    305c74b200000029    1294     65077248   537d silo1    88b4b8
    305c74b20000002b    9244      2740120   537d silo1    88b4ba
    305c74b200000064    9335         9272   537d silo1    88b4f3
    305c74b200000065    9335        10154   537d silo1    88b4f4
    305c74b200000066    9335         4624   537d silo1    88b4f5
    305c74b200000067    9335        10155   537d silo1    88b4f6
    adm 4>

The following example displays the number of records in the database that are associated with user ID `11789` and that were updated during the last five days:

    adm 3>count origuid=11789 and updateage<5d
    72 records found.

### `dmdadm(blank)` Text Field Order

The text field order for daemon records generated by the `dmdump(8)`, `dmdumpj(8)`, and the `dump` directive in `dmdadm` is listed below. This is the format expected by the `load` directives in `dmdadm`:

1. `bfid`

2. `origdevice`

3. `originode`

4. `origsize`

5. `origtime`

6. `updatetime`

7. `checktime`

8. `deletetime`

9. `origuid`

10. `origname`

11. `mspname`

12. `mspkey`

To isolate the `mspname` and `mspkey` from the daemon records soft-deleted fewer than three days ago, use the following command:

    dmdadm -c "dump deleteage<3d and deletetime>0" | awk "-F|" '(print $11,$12}'

## Daemon Logs and Journals

The DMF daemon uses log files to track various types of activity. Journal files are used to track DMF database transactions.

The ASCII log of daemon actions has the following format (SPOOL_DIR refers to the directory specified by the `SPOOL_DIR` configuration parameter):

    SPOOL_DIR/daemon_name/dmdlog.yyyymmdd

The file naming convention is that yyyy, mm, and dd correspond to the date on which the log was created (representing year, month, and day, respectively). Logs are created automatically by the DMF daemon.

| Note: Because the DMF daemon will continue to create log files and journal files without limit, you must remove obsolete files periodically by configuring the `run_remove_logs` and `run_remove_journals` tasks in the configuration file, as described in “Configuring Daemon Maintenance Tasks” in Chapter 2.

The DMF daemon automatically creates journal files that track database transactions. They have the following path name format (JOURNAL_DIR refers to the directory defined by the `JOURNAL_DIR` configuration parameter):

    JOURNAL_DIR/daemon_name/dmd_db.yyyymmdd[.hhmmss]

Existing journal files are closed and new ones created in two circumstances:

- When the first transaction after midnight occurs

- When the journal file reaches size defined by the `JOURNAL_SIZE` configuration parameter

When the first transaction after midnight occurs, the existing open journal file is closed, and the suffix `.235959` is appended to the current file name no matter what the time (or date) of closing. The closed file represents the last (or only) transaction log of the date yyyymmdd. A new journal file with the current date is then created.

When the journal file reaches `JOURNAL_SIZE`, the file is closed and the suffix .hhmmss is added to the name; hh, mm, and ss represent the hour, minute, and second of file closing. A new journal file with the same date but no time is then created.

For example, the following shows the contents of a `JOURNAL_DIR/daemon_name` directory on 15 June 1998:

    dmd_db.19980604.235959  dmd_db.19980612.235959
    dmd_db.19980605.235959  dmd_db.19980613.145514
    dmd_db.19980608.235959  dmd_db.19980613.214233
    dmd_db.19980609.235959  dmd_db.19980613.235959
    dmd_db.19980610.235959  dmd_db.19980614.235959
    dmd_db.19980611.094745  dmd_db.19980615
    dmd_db.19980611.101937
    dmd_db.19980611.110429
    dmd_db.19980611.235959

For every date on which database transactions occurred, there will exist a file with that date and the suffix `.235959`, with the exception of an existing open journal file. Some dates have additional files because the transaction log reached `JOURNAL_SIZE` at a specified time and the file was closed.

You can configure `daemon_tasks` parameters to remove old journal files (using the `run_remove_journals.sh` task and the `JOURNAL_RETENTION` parameter. For more information, see “Configuring Daemon Maintenance Tasks” in Chapter 2.

| Warning: If a daemon database becomes corrupt, recovery consists of applying journals to a backup copy of the database. Database recovery procedures are described in “Database Recovery” in Chapter 7.

Prev | Table of Contents | Next
Chapter 3. Automated Space Management |  | Chapter 5. The DMF Lock Manager