Chapter 3. Using mpirun to Execute Applications
Prev |  | Next

## Chapter 3. Using `mpirun` to Execute Applications

The `mpirun(1)` command is the primary job launcher for the SGI implementation of MPI. The `mpirun` command must be used whenever a user wishes to run an MPI application on an IRIX or a Linux system. You can run an application on the local host only (the host from which you issued `mpirun`) or distribute it to run on any number of hosts that you specify. Note that several MPI implementations available today use a job launcher called `mpirun` and, because this command is not part of the MPI standard, each implementation's `mpirun` command differs in both syntax and functionality.

## Syntax of the `mpirun` Command

The format of the `mpirun` command is as follows:

    mpirun [global_options ] entry [ : entry ... ]

The global_options operand applies to all MPI executable files on all specified hosts. The following global options are supported:

Option |  | Description
`-a[rray]` array_name |  | (IRIX only) Specifies the array to use when launching an MPI application. By default, Array Services uses the default array specified in the Array Services configuration file, `arrayd.conf`.
`-cpr` |  | ( IRIX systems only.) Allows users to checkpoint or restart MPI jobs that consist of a single executable file running on a single system. The absence of any host names in the `mpirun` command indicates that a job is running on a single system. For example, the following command is valid:
    mpirun -cpr -np 2 ./a.out >&1/dev/null
The following commands are not valid:
    mpirun -cpr 2 ./a.out : 3 ./b.out
    mpirun -cpr hosta -np 2 ./a.out>out 2>&1  mpirun -cpr hosta -np 2 ./a.out>out 2>&1  </dev/null
The first one is not valid because it consists of more than one executable file (`a.out` and `b.out`). The second one is not valid because even if submitted from `hosta`, it specifies a host name. For interactive users, the preferred method of checkpointing the job is by `ASH`. This ensures that all of the user's processes specified in the `mpirun` command, plus daemons associated with the job, will be checkpointed. You can use the `array(1)` command to find the `ASH` of a job. Interactive users should also note that `stdin`, `stdout`, and `stderr` should not be connected to the terminal when this option is being used. Use of this option requires Array Services 3.1 or later. The default behavior will allow for jobs to be checkpointed if the above rules for invoking have been followed, but using the `-cpr` option is recommended because it provides specific error messages instead of silently disabling.
`-d[ir]` path_name |  | Specifies the working directory for all hosts. In addition to normal path names, the following special values are recognized:
`.` | Translates into the absolute path name of the user's current working directory on the local host. This is the default.
`~` | Specifies the use of the value of `$HOME` as it is defined on each machine. In general, this value can be different on each machine.
`-f[ile]` file_name |  | Specifies a text file that contains `mpirun` arguments.
`-h[elp]` |  | Displays a list of options supported by the `mpirun` command.
`-p[refix]` prefix_string |  | Specifies a string to prepend to each line of output from `stderr` and `stdout` for each MPI process. Some strings have special meaning and are translated as follows: `%g` translates into the global rank of the process producing the output. (This is equivalent to the rank of the process in `MPI_COMM_WORLD`.) `%G` translates into the number of processes in `MPI_COMM_WORLD`. `%h` translates into the rank of the host on which the process is running, relative to the `mpirun(1)` command line. `%H` translates into the total number of hosts in the job. `%l` translates into the rank of the process relative to other processes running on the same host. %L translates into the total number of processes running on the host. %@ translates into the name of the host on which the process is running. For examples of the use of these strings, first consider the following code fragment:
    main(int argc, char **argv)
    {
       MPI_Init(&argc, &argv);

       printf("Hello world\n");

       MPI_Finalize();
    }
Depending on how this code is run, the results of running the `mpirun` command will be similar to those in the following examples:
    mpirun -np 2 a.out
    Hello world
    Hello world
    mpirun -prefix ">" -np 2 a.out
    >Hello world
    >Hello world
    mpirun -prefix "%g" 2 a.out
    0Hello world
    1Hello world
    mpirun -prefix "[%g] " 2 a.out
    [0] Hello world
    [1] Hello world
    mpirun -prefix "<process %g out of %G> " 4 a.out
    <process 1 out of 4> Hello world
    <process 0 out of 4> Hello world
    <process 3 out of 4> Hello world
    <process 2 out of 4> Hello world
    mpirun -prefix "%@: " hosta,hostb 1 a.out
    hosta: Hello world
    hostb: Hello world
    mpirun -prefix "%@ (%l out of %L) %g: " hosta 2, hostb 3 a.out
    hosta (0 out of 2) 0: Hello world
    hosta (1 out of 2) 1: Hello world
    hostb (0 out of 3) 2: Hello world
    hostb (1 out of 3) 3: Hello world
    hostb (2 out of 3) 4: Hello world
    mpirun -prefix "%@ (%h out of %H): " hosta,hostb,hostc 2 a.out
    hosta (0 out of 3): Hello world
    hostb (1 out of 3): Hello world
    hostc (2 out of 3): Hello world
    hosta (0 out of 3): Hello world
    hostc (2 out of 3): Hello world
    hostb (1 out of 3): Hello world
`-v[erbose]` |  | Displays comments on what `mpirun` is doing when launching the MPI application.

The entry operand describes a host on which to run a program, and the local options for that host. You can list any number of entries on the `mpirun` command line.

In the common case (same program, multiple data (SPMD)), in which the same program runs with identical arguments on each host, usually, you need to specify only one entry.

Each entry has the following components:

- One or more host names (not needed if you run on the local host)

- Number of processes to start on each host

- Name of an executable program

- Arguments to the executable program (optional)

An entry has the following format:

host_list local_options program program_arguments

The host_list operand is either a single host (machine name) or a comma-separated list of hosts on which to run an MPI program.

The local_options operand contains information that applies to a specific host list. The following local options are supported:

Option |  | Description
`-f[ile]` file_name |  | Specifies a text file that contains `mpirun` arguments (same as global_options.) For more details, see “Using a File for mpirun Arguments”.
`-np` np |  | Specifies the number of processes on which to run.
`-nt` nt |  | This option behaves the same as `-np`.

The program program_arguments operand specifies the name of the program that you are running and its accompanying options.

## Using a File for `mpirun` Arguments

Because the full specification of a complex job can be lengthy, you can enter `mpirun` arguments in a file and use the `-f` option to specify the file on the `mpirun` command line, as in the following example:

    mpirun -f my_arguments

The arguments file is a text file that contains argument segments. White space is ignored in the arguments file, so you can include spaces and newline characters for readability. An arguments file can also contain additional `-f` options.

## Launching Programs on the Local Host Only

For testing and debugging, it is often useful to run an MPI program only on the local host without distributing it to other systems. To run the application locally, enter `mpirun` with the `-np` or `-nt` argument. Your entry must include the number of processes to run and the name of the MPI executable file.

The following command starts three instances of the application `mtest`, to which is passed an arguments list (arguments are optional).

    mpirun -np 3 mtest 1000 "arg2"

You are not required to use a different host in each entry that you specify on the `mpirun(1)` command. You can launch a job that has two executable files on the same host. In the following example, both executable files use shared memory:

    mpirun host_a -np 6 a.out : host_a -np 4 b.out

### Using `mpirun(1)` to Run Programs in Shared Memory Mode

For running programs in MPI shared memory mode on a single host, the format of the `mpirun(1)` command is as follows:

    mpirun -nt[nt ]progname

The `-nt` option specifies the number of tasks for shared memory MPI. A single UNIX process is run with multiple tasks representing MPI processes. The progname operand specifies the name of the program that you are running and its accompanying options.

Originally, the `-nt` option to `mpirun` was supported on IRIX systems for consistency across platforms. Because the default mode of execution on a single IRIX system is to use shared memory, the `-nt` option behaves the same as if you specified the `-np` option to `mpirun`. The following example runs ten instances of `a.out` in shared memory mode on `host_a`:

    mpirun -nt 10 a.out

## Launching a Distributed Program

You can use `mpirun(1)` to launch a program that consists of any number of executable files and processes and distribute it to any number of hosts. A host is usually a single machine, or, for IRIX systems, can be any accessible computer running Array Services software. For available nodes on systems running Array Services software, see the `/usr/lib/array/arrayd.conf` file. Array Services is not supported on Linux systems currently, so an alternate launching mechanism is used.

You can list multiple entries on the `mpirun` command line. Each entry contains an MPI executable file and a combination of hosts and process counts for running it. This gives you the ability to start different executable files on the same or different hosts as part of the same MPI application.

The following examples show various ways to launch an application that consists of multiple MPI executable files on multiple hosts.

The following example runs ten instances of the `a.out` file on `host_a`:

    mpirun host_a -np 10 a.out

When specifying multiple hosts, you can omit the `-np` or `-nt` option, listing the number of processes directly. The following example launches ten instances of `fred` on three hosts. `fred` has two input arguments.

    mpirun host_a, host_b, host_c 10 fred arg1 arg2

The following example launches an MPI application on different hosts with different numbers of processes and executable files, using an array called `test`:

    mpirun -array test host_a 6 a.out : host_b 26 b.out

The following example launches an MPI application on different hosts out of the same directory on both hosts:

    mpirun -d /tmp/mydir host_a 6 a.out : host_b 26 b.out

Prev | Table of Contents | Next
Chapter 2. Building MPI Applications |  | Chapter 4. Thread-Safe MPI