Appendix A. OpenVault Troubleshooting
Prev |  | Next

## Appendix A. OpenVault Troubleshooting

This appendix explains how to resolve error conditions, and includes a description of the possible error messages.

Most OpenVault errors are found in the log file `/usr/OpenVault/var/OVLOG`. They can also be listed with the `ov_msg` `-l` option; see the `ov_msg(1M)` man page.

## Error Conditions

Errors such as tape read and write errors are associated with an application error, whereas conditions such as `cannot find tape` or `cannot associate with driver` are OpenVault errors.

## Accessing OpenVault Messages

OpenVault system messages are appended to the logfile, `/usr/OpenVault/var/OVLOG`.

To view all the warning, error, and operator intervention messages that are contained in the message log on the default server, use the `ov_msg` command; see the `ov_msg(1M)` man page.

## Error Messages and Actions

Refer to the OpenVault release notes for a listing of the error messages that apply to the OpenVault system.

## OpenVault Processes and Files

Table A-1 shows some important OpenVault files.

Table A-1. OpenVault Configuration Files

OpenVault File | What it Controls
--- | ---
`var/corekeys` | File listing clients allowed OpenVault access, possibly with passwords.
`admin/keys` | File storing security passwords for OpenVault applications.
var/OVLOG | Repository of all events and errors that occur within OpenVault.
`mlm/config` | Configuration file for OpenVault server, MLM (media library manager).
`dcp/tapeN/config` | Configuration file for the drive named tapeN.
`lcp/libN/config` | Configuration file for the library named libN.
`dcp/tapeN/logfile` | Event and error log for the drive named tapeN.
`lcp/libN/logfile` | Event and error log for the library named libN.

Table A-2 shows some important OpenVault processes.

Table A-2. OpenVault Processes

OpenVault Process | What It Does
--- | ---
`ovroot` | If `chkconfig` is on, this process mediates and delegates service requests.
`MLM_catalog` | Grants services as requested, and manages the OpenVault catalog.
`MLM_capi` | Started by the server to manage each registered OpenVault application.
`MLM_aapi` | Started by the server to manage each OpenVault administrative application.
`MLM_dcp` | Started by the server to manage each attached drive and its related DCP.
`MLM_lcp` | Started by the server to manage each attached library and its related LCP.
DCPtapeN | The DCP (drive control program) that manages the drive named tapeN.
LCPlibN | The LCP (library control program) that manages the library named libN.
ssi | lmcpd | Product-specific processes to manage ACSLS and IBM-3494, respectively.

## Troubleshooting OpenVault Commands

When trying to troubleshoot OpenVault administrative commands, it's helpful to look inside the command to see what it is doing. The IRIX `par` command shows what system calls a program makes. This is useful, but OpenVault offers a better and more detailed way to see what an OpenVault command is doing. Before any troubleshooting can begin, debugging must be turned on in the OpenVault server, using the following command:

    # /usr/OpenVault/admin/ov_msg -s -t core -m debug

Once this is done, the OpenVault server starts writing a lot more messages to the `OVLOG` file; so you can run a command to see what it is doing. `OVLOG` messages contain a lot of useful information. Here is a sample log entry:

    Apr 26 17:32:44 mlm_aapi:SYSTEM:onlyInstance: (debug) [177600] WRITTEN:
    response whichtask[`5'] accepted ;

This entry shows the following information:

`Apr 26 17:32:44` |  | Current date and time.
`mlm_aapi` |  | The OpenVault process.
`SYSTEM` |  | The application.
`onlyInstance` |  | There are no other such applications.
`(debug)` |  | Level of the message; can be `info`, `error`, `debug`.
`[177600]` |  | Process ID of the OpenVault process.
`WRITTEN: ...` |  | The commands being passed along.

When an OpenVault command is run (or when any connection is made to OpenVault), a process is created to handle the connection. This is listed in `OVLOG` (and in the process table) as `mlm_aapi`. Following this process, you can see what the MLM server sees when it talks to the calling command or application as shown in Example A-1.

Example A-1. `mlm_aapi` Calls and Their PIDs

To get a list of individual `mlm_aapi` calls and their PIDs, run the following command:

    # grep 'welcome version' OVLOG | grep mlm_aapi
    Apr 26 17:32:43 mlm_aapi:SYSTEM:onlyInstance: (debug) [177600] WRITTEN:
          welcome version[`1.0'];
    Apr 26 17:32:57 mlm_aapi:SYSTEM:onlyInstance: (debug) [177685] WRITTEN:
          welcome version[`1.0'];
    Apr 26 17:33:27 mlm_aapi:SYSTEM:onlyInstance: (debug) [177772] WRITTEN:
          welcome version[`1.0'];

This listing shows the AAPI/CAPI reply `welcome` that the `mlm_aapi` sends to the calling command. There is only one `welcome` command per connection, so this should give a unique list of `mlm_aapi` processes.

Once the PID is found, `grep` can be used to locate all of the `OVLOG` messages that relate to that one command or connection. Example A-2, shows an `ov_mount` command:

Example A-2. `ov_mount` Search

    # ov_mount -V A05444 -A dmf

To find the specific PID for that `ov_mount` command, run this `grep` command, which looks for any OpenVault commands showing the volume `A05444` and using the application `dmf`.

    # grep A05444 OVLOG | grep dmf
    Apr 26 14:52:24 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] READ:
        mount match [and( strEQ(VOLUME.'VolumeName' `A05444') strEQ(VOLUME.
        `ApplicationName' `dmf'))] report[VOLUME.'VolumeName' MOUNTLOGICAL.
        `MountLogicalHandle'] reportmode[value]  task[`0'];

Make sure that the time listed in the results corresponds with the time that the `ov_mount` command was run. Now that you found the PID for this particular command, do a `grep` for all messages relating to that command:

    # grep 172088 OVLOG
    Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] WRITTEN:
        welcome version[`1.0'];
    Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] /usr/OpenVault
        /mlm/MLM_aapi: submain(): process_peer() returned 1
    Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] READ: mount
        match[and( strEQ(VOLUME.'VolumeName' `A05443') strEQ(VOLUME.
        `ApplicationName' `dmf'))] report[VOLUME.'VolumeName' MOUNTLOGICAL.
        `MountLogicalHandle'] reportmode[value]  task[`0'];
    Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] WRITTEN:
       response whichtask[`0'] accepted ;
    ...

This shows the entire AAPI conversation from the view of the `mlm_aapi` or OpenVault server. All commands sent and received are shown exactly as transmitted.

Prev | Table of Contents | Next
Chapter 7. Tertiary Storage Management |  | Appendix B. OpenVault Man Pages