Appendix C. Theory of Operations
Prev |  | Next

## Appendix C. Theory of Operations

This appendix is for the system administrator or operator who is familiar with the IRIX NetWorker Administrator's Guide and is interested in more information about NetWorker and how it works. You should be familiar with IRIX administrative concepts, the X Window System interface, and the `nsradmin` command.

The NetWorker reference pages provide complete and detailed information to help you administer NetWorker. The reference pages list the commands in alphabetical order, so that you can verify them easily.

This appendix gives a pictorial overview of the major components of NetWorker and how they are controlled. The pictures illustrate basic points; they are not meant to be detailed or precise. This appendix lists the various NetWorker reference pages that contain details about each section.

You should read this appendix while sitting at your workstation so that you can read the IRIX and NetWorker reference pages, execute NetWorker commands (especially `nsradmin`), and verify the pictures (using IRIX administrative commands like `ps`).

The basic picture elements used in this appendix are shown in Figure C-1.

Figure C-1. Major Components of NetWorker

This appendix explains

- NetWorker media format

- the NetWorker system at rest

- browsing and modifying resources

- recovering files

- backing up the entire system

- automatic network-wide backups

- event processing

- media management

- index management

- disaster recovery

- NetWorker security

## NetWorker Media Format

The NetWorker media format is designed to meet these requirements:

- the ability to multiplex data from several clients simultaneously to maximize performance

- machine architecture independence (such as byte order differences) through the eXternal Data Representation (XDR) format

- filesystem independence, allowing NetWorker to back up heterogeneous clients with different operating systems and filesystems

- the ability to fully utilize media capacity by appending to the media until it is full, and then continuing to additional media

- support for quick seeking on media, by writing occasional file marks

- special handling for certain types of files via the Application Specific Module (ASM) architecture

- the ability to track media through media labels

No other existing media format meets these requirements.

The NetWorker media format is fully documented in the `mm_data`(5) reference page. Third parties are welcome to adopt the format.

## The NetWorker System at Rest

Four daemons continue running on the NetWorker server system even when it is at rest. The two internal server daemons, `nsrmmd` and `nsrmmdbd`, are not meant to be `networker` client-accessible; their only clients are forked copies of `nsrd`. As these internal daemons execute, they may send status messages (or “events”) to the controlling `nsrd`.

The controlling `nsrd` is the root of all NetWorker functionality. It receives status information from the internal daemons and reports the status to commands such as `nsrwatch` and `networker`. The `nsrd` daemon allows the browsing and editing of NetWorker resource descriptors by clients using the `nsradmin` command. Finally, `nsrd` accesses save and recover sessions from network clients and forks “agent” copies of itself to handle each session.

From a monitoring point of view, the NetWorker main window and `nsrwatch` are functionally equivalent. They monitor the dynamic state of the NetWorker server. A very important field in these monitors is the Pending display, which shows you what the server needs to make progress.

The `networker` daemon provides an X Window System graphical interface for all the system configuration tasks. The `nsradmin` daemon provides a character-based administrative interface to the NetWorker server that can be used from any ASCII terminal.

The four daemons that are always running on the NetWorker server and three network-based monitors are shown in Figure C-2.

Figure C-2. NetWorker Monitors and Daemons

Table C-1 lists the relevant reference pages.

Table C-1. Reference Pages for NetWorker at Rest

Reference Page | Comments
--- | ---
`nsradmin`(1M) | Controls all aspects of NetWorker server administration
`nsrwatch`(1M) | Character-based monitor
`nwadmin`(1M) | X Window System interface for NetWorker system administration
`nsrd`(1M) | NetWorker server
`nsrmmd`(1M) | Internal daemon interfaces to all backup devices
`nsrmmdbd`(1M) | Internal daemon that manages the media index

## Browsing and Modifying Resources

You can configure NetWorker by using the X Window System interface or by editing the configuration information (called resources) from an ASCII terminal.

If you are not using the X Window System, you can browse and edit the NetWorker server's resources with the `nsradmin` command. You can either use your IRIX editor to browse and modify the NetWorker server's resource descriptors, or you can use the visual mode of `nsradmin`. See the `nsradmin`(1M) reference page for more information.

If you are using your editor, `nsradmin` copies the resource descriptors from the server into a temporary file on your system. When you are finished making changes, `nsradmin` compares any edits you may have made with the original copies and prompts you for confirmation for any additions, deletions, or changes you may have made to the resource descriptors.

Changes are passed back to the server and are immediately applied to the NetWorker server subsystems that are affected. This way, administrators do not have to kill and restart daemons or reboot systems in order to have changes applied. In addition, the changes can be applied from a client workstation using the administrator's favorite editor. All administrative aspects of a NetWorker server are handled by resource modifications.

The resource types that describe the NetWorker server configuration are shown in Figure C-3.

Figure C-3. NetWorker Resource Types

Table C-2 lists the relevant reference pages.

Table C-2. Reference Pages for Browsing and Modifying Resources

Reference Pages | Comments
--- | ---
`nwadmin`(1M) | X Window System interface
`nsradmin`(1M) | Invokes your editor on resource descriptor copies
`nsr_resource`(5) | Describes resource file format and types
`nsrd`(1M) | NetWorker server checks and applies changes

## Backing Up Files

This section describes the operation of backups, called “saves.”

The `save` command traverses a client's filesystems and backs up a client's files, subject to directives. See “Using Directives” in Chapter 6 for information.

The `save` daemon first initiates a session with the server's controlling `nsrd`. The server accepts a connection if it originates from a privileged port on a client listed in the server's “NSR client” resource descriptor. Once the network connection is accepted, a forked agent `nsrd` handles all subsequent requests from the client.

After session establishment with the server, `save` reads files and sends a save set to its agent `nsrd`. Upon completion, the client calls upon the server to commit the save set. To commit the save set, the agent `nsrd` instructs the `nsrmmd` to commit the save set data to the backup media. At this point, the agent returns from the commit call. `save` then terminates the session and both `save` and the agent `nsrd` exit.

Under normal circumstances, in order to acquire a privileged port, `save` must be a “set uid root” command. Upon acquiring the privileged port, `save` reverts back to the invoking the user ID, unless the user is operator or the user is in the group operator.

The controlling `nsrd` does the original access control, then forks an agent to handle the actual saved data. By forking one agent per save session, the server can handle an arbitrary number of sessions simultaneously.

Figure C-4 diagrams a typical `save` session between client and server.

Figure C-4. Typical `save` Session

Table C-3 lists the relevant reference pages.

Table C-3. Reference Pages for Backing Up Files

Reference Pages | Comments
--- | ---
`nsr`(5) | Documents `save` directives
`nsr_client`(5) | Documents the resource descriptor type “NSR client”
`save`(1M) | Command that saves files to the server
`nsrd`(1M) | NetWorker server daemon checks access control
`nsrexecd`(1M) | NetWorker client execution daemon
`rcmd`(3N) | Discusses privileged ports
`nsr`(1M) | Security section discusses policies and issues
`nsr_data`(5) | Describes basic data types passed from client to server

## Recovering Files

The `recover` command is the counterpart of `save` and subject to similar security and session establishment policies. Once a session is established, two major functions of `recover` are used: file browsing and recovery.

Saved files are browsed by using commands that are familiar to the IRIX user. Since file attributes are kept online, the browsing is possible even when no backup volume is mounted on a device. During browsing, media information may be needed if the user requests information associated with the “Versions” option in the Recover window's View menu.

As users browse, they may build a recover list. Eventually the user may give the `recover` command and the client submits the recovery list to the agent `nsrd` on the server. At this point, the file and media indexes are used to determine the backup volume(s) and position(s) of the desired file(s).

The `nsrmmd` requests backup volumes and reads them until the entire file recovery list is processed.

Clients browse the online indexes via remote procedure calls to their agent daemons. The device and `nsrmmd` are involved only when files are actually requested for recovery. Thus, NetWorker supports multiple concurrent browsers.

Figure C-5 diagrams a typical `recover` session between client and server.

Figure C-5. Typical `recover` Session

The relevant reference page is `recover`(1M), which documents browsing and recovery subcommands.

## Backing Up the Entire System

While it is useful that the `save` command allows individuals to back up their directories, an unattended backup system must back up whole systems according to predefined backup levels. The `savefs` command accomplishes this task.

The `savefs` daemon determines which backup schedule to use for a client by looking for the schedule name in the “NSR client” resource. It then consults the appropriate “NSR schedule” resource to determine which backup level to use (full, incremental, or a level 1–9. Given the save level and a filesystem, `savefs` saves only files within the filesystem that have been modified since the most recently recorded lesser-level save. Upon completion, the current save level and the time at which `savefs` began are recorded in the media database.

If the client has been configured to back up all its filesystems, `savefs` uses all local filesystems.

After compiling information about the local filesystems and when they were last saved, and determining the desired save level, the `savefs` command invokes `save` against each of the client's filesystems.

Figure C-6 diagrams the `savefs` command saving two filesystems in parallel.

Figure C-6. Saving Two Filesystems in Parallel

Table C-4 lists the relevant reference pages.

Table C-4. Reference Pages for Backing Up the Whole System

Reference Page | Comments
--- | ---
`savefs`(1M) | Saves the client's filesystems
`nsr_client`(5) | Determines the client configuration and which filesystems are backed up
`nsr_directive`(5) | Determines which save directives apply to the client
`nsr_schedule`(5) | Determines save level
`save`(1M) | Does the actual save, given many arguments

## Automatic Network-Wide Backups

NetWorker uses the “NSR group” resource to find the start time of a network-wide backup. The pre-configured time is 3:33 each morning. Each “NSR client” resource describes the groups to which it belongs.

The controlling `nsrd` starts the `savegroup` command at the appropriate time for each group that is enabled. The `savegroup` daemon uses the “parallelism” attribute of the “NSR” resource to determine how many client sessions to save concurrently.

Upon completing all client saves, `savegroup` invokes the `saveindex` command to insure that the NetWorker server index is safely backed up.

The NetWorker server `nsrd` starts the nightly saves by invoking `savegroup`. Each client's `savefs` is initiated by `savegroup` in an orderly manner. After the clients are backed up, `saveindex` backs up the server's index.

The daemon `savegroup` also uses `nsrexeced` to run `savefs` on clients. This daemon uses `rshd` if `nsrexecd` is not present and running on a client.

The NetWorker server `nsrd` starts the nightly save by invoking `savegroup`, as illustrated in Figure C-7.

Figure C-7. Starting the Nightly Save

| Tip: Make liberal use of incremental save levels, which are very efficient, since they take minimal backup media space and run quickly.

| Note: The daemon `saveindex` does a level 9 backup of the indexes to promote faster recovery after a disaster.

| Tip: For unattended backups, a NetWorker server with two backup devices is worth more than twice as much as a NetWorker server with only one backup device. Often the NetWorker server with two backup devices is more productive than two NetWorker servers with only one device each. Using a jukebox is the most efficient way to complete unattended backups.

A nightly network backup is shown in Figure C-8. (Not shown are the server's controlling `nsrd` and `nsrmmdbd`, and the client`s `nsrexecd`.)

Figure C-8. Nightly Network Backup

Table C-5 lists the relevant reference pages.

Table C-5. Reference Pages for Automatic Nightly Backups

Reference Page | Comments
--- | ---
`savegroup`(1M) | Conducts the nightly network saves
`nsr_service`(5) | Parallelism attribute controls how many clients save at once
`nsr_group`(5) | Selects the time of nightly saves
`nsrexecd`(1M) | Remote execution system used to start `savefs`
`rshd`(1M) | Remote execution system used to start `savefs` if `nsrexecd` is not present
`savefs`(1M) | Saves each client's filesystems
`saveindex`(1M) | Saves the server's index; detailed in a later section

## Event Processing

Besides reporting routine status to the controlling `nsrd`, the NetWorker subsystems are set to report major events. The administrator can cause any IRIX command to be invoked when an event occurs. He or she does this simply by modifying an “NSR notification” resource on the NetWorker server.

When an event occurs, the controlling `nsrd` inspects all “NSR notification” resources. An action is taken if the event type matches any event type listed in the hidden “event” attribute list, and if the event's priority matches any priority listed in the hidden “priority” attribute. Therefore, one event may trigger any number of actions, and one “NSR notification” may match any number of events.

All NetWorker daemons post events to the controlling `nsrd`. As each event is posted, the controlling `nsrd` matches the event against all NSR notification resources and acts on every match by executing the corresponding IRIX command. NetWorker defaults provide interfaces to the `syslog` system and electronic mail.

Figure C-9 diagrams the notification subsystem.

Figure C-9. Notification Subsystem

The relevant reference page is `nsr_notification`(5), which describes the notification process.

## Media Management

There is a working relationship between `nsrmmd`, the other daemons, the device resources, and the media manipulation commands. As shown, the `nsrmmd` daemon may be writing to one device while the operator brings a second device online. When `nsrmmd` needs a new backup volume, it polls the controlling `nsrd` for the backup volume and the device on which it is located. The `nsrmmd` daemon also records save set information to the `nsrmmdbd` daemon each time a backup volume filemark is written.

During recovery, `nsrmmd` queries `nsrmmdbd` to discover the backup volume and volume-file location of the desired data. The administrator uses the `mminfo` command to display information about the backup volume library.

Figure C-10 diagrams `nsrmmd` and its relation to other processes.

Figure C-10. `nsrmmd` and Other Processes

Table C-6 lists the relevant reference pages.

Table C-6. Reference Pages for Media Management

Reference Page | Comments
--- | ---
`nsrmm`(1M) | Media manager mounts, unmounts, deletes backup volumes
`nsr_device`(5) | Describes and names backup devices
`nsrmmd`(1M) | Writes and reads data to and from the backup devices
`nsrmmdbd`(1M) | Manages the backup media library
`mminfo`(1M) | Displays information about the backup media library

## Index Management

The online index is built during saves, and is queried during browsing and recovering. The `nsrls` command allows the administrator to gather information about the index sizes and record counts. Index entries are purged by the `nsrmm` command after it deletes a backup volume from the media index. The `nsrck` command is automatically invoked after failure (for example, an operating system crash) to guarantee the index database consistency before the NetWorker service is enabled.

Figure C-11 diagrams the file index and its relationship to processes.

Figure C-11. The File Index and Processes

Table C-7 lists the relevant reference pages.

Table C-7. Reference Pages for Index Management

Reference Page | Comments
--- | ---
`nsrmm`(1M) | Media manager deletes backup volumes
`nsrindexd`(1M) | Manages the online index
`nsrls`(1M) | Displays index usage statistics
`nsrck`(1M) | Rebuilds indexes after a hard crash
`nsr_layout`(5) | Explains where all the NetWorker server's files are located

## Disaster Recovery

The last act that `savegroup` performs each morning is backing up the NetWorker server's own index so that the index can be recovered without using itself. Besides saving the index, `saveindex` sends bootstrap recovery information to the printer. Should the server's index ever need to be recovered, this bootstrap information is sufficient to find the saved index on a NetWorker backup volume.

The `saveindex` daemon is invoked by `savegroup`; its job is to save index information vital to the server itself and to print recovery parameters to the line printer. Should the server's indexes ever need to be recovered, these parameters will be fed to the `recoverindex` command. Once the server's indexes are recovered, all other files may be recovered in the normal manner.

Figure C-12 shows the progression of index information from saving to recovering.

Figure C-12. Index Information From Saving to Recovering

Table C-8 lists the relevant reference pages.

Table C-8. Reference Pages for Disaster Recovery

Reference Page | Comments
--- | ---
`savegroup`(1M) | Typically invokes `saveindex`
`saveindex`(1M) | Also covers `recoverindex`
`scanner`(1M) | Used by `recoverindex`; reads raw NetWorker backup volumes
`mm_data`(5) | Describes backup media format
`nsr_crash`(1M) | More general information on recovering after a disk crash

## NetWorker Security

When a client attempts to save files or filesystems or recover them, NetWorker starts the `save`, `savefs`, or `recover` program, respectively. However, before the server accepts the command to start any of these programs, the client must establish a connection with the server. Before the server accepts the connection, it must verify that the user who initiated the save or recover request has the necessary permissions:

- The request for the connection must be made from a secure port on the user's machine. A secure port can be opened only by root, so the `save`, `savefs`, and `recover` programs run setuid to root.

- The server verifies that the user executing the programs at the workstation has permission to save or recover a client's files. For a user to have permission, one of the following criteria must be met:

  - the user's machine name must be equivalent to the client name

  - the user attempting to establish a connection must be a member of the Remote access list in the Clients window

This access control is similar to that used by the `rsh` (remote shell) command, except that instead of using the`/.rhosts` file, NetWorker uses the Remote access list in the Clients window.

Once a connection has been established, the client commands `save` and `recover` set their effective UID to the UID of the user who initiated the command so that all local filesystem and system call access is done as that user. Thus users cannot recover or back up files to which they do not have access. The exception to this rule is that the user name operator and users in the group operator have filesystem access privileges of root. Therefore, the administrator can set up a login or group for the operators who initiate backups and recovers on behalf of other users, without giving the operators root access to client systems.

| Note: Since any user named operator or any user belonging to a group named operator can access all data on all clients, care should be taken in assigning these names.

NetWorker's preconfigured selections allow the clients to browse and recover only their own files. To give other clients recover access to a client's files, the administrator must explicitly add the access to the Remote access list in the Clients window, as explained earlier in this chapter.

You can further tighten access control for the client commands by turning off the set-uid bit. Doing this restricts client system use of the `save`, `savefs`, and `recover` commands to root. To allow access by root and operator, but not by other users, change the group ownership of these commands to operator, and set the mode bits to allow execution by owner and group.

The `savegroup` command initiates the `savefs` command on each client system in a backup group by sending a remote command request to the `nsrexecd` command.

The `nsrexecd` command runs on NetWorker client systems. This command provides a secure and restrictive way for NetWorker to start automatic backups on clients. The `nsrexecd` command allows you to restrict access to a select set of NetWorker servers. When you install a client, `nsrexecd` is started, and statements are added to the appropriate boot files to restart `nsrexecd` each time the client reboots.

Prev | Table of Contents | Next
Appendix B. Recovering From a Disk Crash |  | Appendix D. Command Summary