Most existing UNICOS/mk applications and algorithms are written to use PE numbers for communication. Standard PVM notation used only the concept of a PVM task identifier (pvm_tid, whose internal representation is subject to change). To simplify programming, the UNICOS/mk version lets you use PE numbers in place of pvm_tids in many of the PVM functions. An extra function, pvm_get_PE(3), returns the PE number associated with a pvm_tid.

PVM supports the concept of dynamic groups, in which tasks can join and leave groups at any time. The barrier and broadcast functions use these groups for collective synchronization and communications. On a UNICOS/mk system, a static, well-defined group consisting of all the tasks (or PEs) in the partition is referred to as the global group. To simplify programming, PVM has essentially predefined this group by permitting a null name (or, in C, a null char pointer) to be used to refer to this global group. (The Fortran PVM include file, fpvm3.h, contains a declaration of a null character variable, PVMALL.) PVM uses some key optimizations to carry out barriers and broadcasts for the global group.

UNICOS/mk applications can use PVM calls to obtain their own PE number. From C these calls are as follows:

my_pe = pvm_get_PE (pvm_mytid());

From Fortran the calls are as follows:

CALL PVMFMYTID (MYTID)
CALL PVMFGETPE (MYTID,MYPE)

UNICOS/mk applications can use PVM calls to obtain the number of PEs in the partition. From C this is as follows:

n_pes = pvm_gsize(0);

From Fortran the call is as follows:

CALL PVMFGSIZE (PVMALL, NPES)

The variable PVMALL is declared in fpvm3.h.

The UNICOS/mk version of PVM treats data buffers packed using PvmDataInPlace encoding differently than the network version does. In the UNICOS/mk version, such data must not be reused until the data has been unpacked by the receiving PE. You are responsible for any additional synchronization or communication required to ensure this coordination.

The design of buffer memory is based on the following:

Send buffers are never freed by PVM. Once allocated and used, they are kept for later use. However, any incremental memory allocated for a send buffer is freed as soon as it is no longer needed.

The scenario in Table 3-2 shows PVM memory use, using the default settings.

This scenario shows how PVM allocates memory for send buffers. Although 36,128 bytes were allocated, only 32,104 were actually used. The 4096 bytes allocated in the second incremental allocation were used for only 72 bytes.

The following parameters are available for controlling send buffer memory use:

You can set all five parameters by using environment variables, which take effect at PVM initialization time. Four of the five can also be set by calling pvm_setopt(3), which changes the settings dynamically at run-time. (The fifth parameter, initial number of send buffers, affects an initialization time function, and so a run time change would have no effect.) You can call the pvm_getopt(3) function to obtain the current settings for all five parameters.

Only three environment variables are needed to set the five parameters because two of these variables let you set either one or two parameters at once.

By using an environment variable, you set the value for all PEs at once. By calling pvm_setopt(3), you can set different values for different PEs, or you can change a value during the execution of the program. You can, of course, combine the two mechanisms by using the environment variables to set the default values and pvm_setopt(3) to change specific cases.

The following sections discuss how you can use and set these parameters.

setenv PVM_DATA_BUFFERS <number>

setenv PVM_DATA_BUFFERS 10

setenv PVM_DATA_BUFFERS number+increment

setenv PVM_DATA_BUFFERS 10+4

setenv PVM_MAX_PACK initial

setenv PVM_MAX_PACK 16384

setenv PVM_MAX_PACK initial+increment

setenv PVM_MAX_PACK 4096+160000

setenv PVM_TOTAL_PACK limit

setenv PVM_TOTAL_PACK 1048576

In the original scenario (“Simple Scenario, Part 1”), 36,128 bytes of buffer memory were allocated, but only 32,104 were actually used. Memory use could be made most efficient by using PvmDataInPlace encoding, which avoids PVM buffer allocation altogether. But this change may require some additional synchronization within the program, and thus it may not be desirable.

Next in order of simplicity, you could move the large pvm_pkbyte(3) call (with 32,000 bytes) to the end. Consequently, the three small packs would go into the initial 4096 bytes, and just enough bytes would be allocated for the large pvm_pkbyte(3) call.

Instead (or in addition), the following PVM_MAX_PACK settings could be considered to more efficiently manage memory:

setenv PVM_MAX_PACK 32104

This setting ensures that all the memory needed is allocated with the send buffer. If all message traffic looked like this, this would be most efficient. By setting PVM_MAX_PACK to 32104+0, you could verify that no message exceeded this limit.

setenv PVM_MAX_PACK 4096+28008

This setting ensures that the first incremental memory allocation is sufficient for the remaining packs. If most messages fit into 4096 bytes, and the rest fit into 32,104 bytes, this setting limits normal memory use while avoiding unnecessary malloc(3) or free(3) calls for the large messages.

This scenario shows only the memory allocation for a single message. A real application has many messages of different sizes; therefore, while PVM_MAX_PACK settings might help this one message, they might have adverse effects on others.

If only one PE is sending a large message, another approach is to change the source code so that this PE calls pvm_setopt(3) once with PvmMaxPack and perhaps again with PvmMaxPackIncr, each set to the values indicated in the previous setenv commands, prior to packing and sending the large message. For example, you could call pvm_setopt with PVM_MAX_PACK set equal to 32,104, or you could call pvm_setopt with PVM_MAX_PACK set equal to 40 and call pvm_setopt with PvmMaxPackIncr set equal to 28,008.

A pool of memory is allocated in each PE to receive messages from other PEs. When a message is sent, the sending PE uses part of the pool on the receiving PE for the message. At the beginning of various PVM functions, a receiving PE checks for any messages in this pool and clears them out. If too many PEs try to send messages before a PE can clear out the pool, the pool becomes exhausted, and subsequent sends may fail with the PvmOutOfResSMP error.

By default, sends that detect this condition enter a retry loop, in which they delay briefly and then recheck the pool. This loop is performed PVM_RETRY_COUNT times (default is 500), and the PvmOutOfResSMP error is issued at the end of this count. You can adjust this limit up or down as described in Table 3-1. Many applications will find that increasing this count is sufficient to get by the error.

You can also adjust the number of entries in the pool. The default limit is twice the number of PEs or 10, whichever is larger. You can raise or lower this limit by using the PVM_SM_POOL environment variable, described in Table 3-1.

If you are hitting the pool entry limit, you may want to see if the receiving PE can be changed to call pvm_recv(3) or pvm_nrecv(3) sooner. This problem can occur if all PEs are broadcasting to each other and then trying to receive the results. By interspersing the broadcasts with the receives, you may avoid having to raise the limit.

You may also hit the pool entry limit if many messages are being sent to a PE that is busy doing some computation, waiting for I/O, or doing something else that keeps it from entering PVM. Increasing the limit allows such operations to proceed asynchronously; changing the code to operate more synchronously is another option.

The PvmOutOfResBuf error occurs only if you have set the send buffer increment parameter to 0 (see “Send Buffer Increment”, for information on setting this parameter), indicating that you want a fixed number of send buffers. Getting the error indicates that you underestimated the number of buffers that you needed.

A send buffer cannot be reused until the data in it has been copied to the receiving PE. If the data is smaller than the size of a short message (PVM_DATA_MAX, which has a default of 4096 bytes), this copy occurs on the pvm_send(3) call. For larger amounts of data, this copy does not occur until the receiving PE has unpacked that data.

Make sure you are using buffers efficiently. Sometimes users convert code to use PVM, and the code appears as follows:

for (... several PEs ...) {
        pvm_initsend (PvmDataRaw);
        pvm_pkbyte (addr, size,...);
        pvm_send (...);
}

Here, the same data is being sent to each PE. However, a single packed buffer can be used by multiple sends:

pvm_initsend (PvmDataRaw);
pvm_pkbyte (addr, size, ...);
for (... several PEs ...) {
        pvm_send (...);
}

Or the single packed buffer can be used by a more efficient broadcast or multicast such as the following example:

pvm_initsend (PvmDataRaw);
pvm_pkbyte (addr, size, ...);
pvm_mcast (...);

In both cases, a single send buffer is used. The data it contains is not freed until all of the receiving PEs have responded, which may take a while; however, your use of buffers and memory will be reduced. Also, your program will run faster due to the reduced number of function calls.

PVM allows groups to consist of as many PEs as you specify, up to the total number of PEs in the partition. This is a general feature, but for large numbers of PEs it can waste memory. This is especially true if your groups are small relative to the number of PEs.

You can reduce the limit, and thus save memory, in either of two ways:

Remember that the UNICOS/mk version of PVM defines aglobal group, consisting of all PEs in the partition. If you have code in which each PE is joining a global group with your own name (perhaps code ported from a network version of PVM), you should consider using the predefined global group on the UNICOS/mk system. This will simplify your code, and you will get better performance when using barriers across the group or broadcasts to the group.

The following sections discuss several key issues related to the distributed mode. The issues are as follows:

PE Communication

The PVM daemon runs on the UNICOS/mk system. A PE on the UNICOS/mk system communicates with the daemon and with PVM tasks outside its own partition. In theory, any PE can do so. But UNICOS limits the number of open files per application and the number of open sockets in the system. So, if a UNICOS/mk application running on a large number of PEs were to set up communications for each PE, it may hit either or both of these limits.

Socket communications are very slow, especially compared to the speed of communications between PEs. Because much of socket communication is single-threaded in the PVM daemon, the performance cost goes up as more PEs try to communicate at the same time.

For these reasons, by default, only PE 0 establishes communications with the daemon, and you should consider using PVM in this manner. However, you can specify additional PEs by setting the PVM_PE_LIST environment variable, as follows:

setenv PVM_PE_LIST 0,4,8,12 setenv PVM_PE_LIST all

This environment variable must be set for both the PVM daemon pvmd3(1) and the application to read, and both must read the same value.

---

Note: At present, PE 0 always establishes communications with the daemon, even if PE 0 is not specified in PVM_PE_LIST. It is suggested that PVM_PE_LIST specify PE 0, if it is being used, to ensure future compatibility. It is possible that future releases may introduce other mechanisms for controlling access to the daemon.

---

Cross-system Dynamic Groups

You cannot form a dynamic group consisting of tasks from the UNICOS/mk system and another system. You cannot form a dynamic group consisting of tasks from more than one partition within a UNICOS/mk system. You must view group handling on each system and partition as being completely independent. If the UNICOS/mk tasks form a group called MYGROUP, and the tasks in the network also join a group called MYGROUP, the two groups are completely independent. A broadcast from a UNICOS/mk task to MYGROUP sends messages only within that partition; no messages will go outside the partition.

---

Note: In future releases, this limitation might be removed. Therefore, you should not build your application assuming that the two groups are independent; in a later release, they might form a single, combined group.

---

You can use programs and commands a number of different ways to run a distributed application involving the UNICOS/mk system. The following example shows one way.

count = pvm_spawn("mpp.a.out", 0, PvmTaskArch, "CRAY", nproc, tids);

The child task on the UNICOS/mk system does not look very different from one written to run in stand-alone mode. You must be careful to use pvm_tid, instead of PE numbers, when referring to the parent task. You must also be careful that only those PEs that can communicate with the daemon try to do so. You can deal with both of these constraints by calling pvm_parent(3). If this function returns a pvm_tid, that identifier can be used for communication. If the function returns the PvmNoParent error, that PE cannot communicate with the outside world. “PVM Program Development” in Chapter 2, describes how to start the PVM daemon and your parent task.

In distributed mode, PVM uses sockets for communication. Read and write system calls actually transmit control and data across the sockets. Further, a given PVM task may have several sockets open at once: one to its local daemon and, optionally, one or more to specific tasks with which it is communicating.

The following facts have important implications regarding performance:

By default, in distributed mode, only PE 0 communicates with the PVM daemon, but additional PEs can also be permitted to communicate (for more information, see “PE Communication”). If you are interested in performance, think very carefully before using more than one PE to make PVM calls outside the UNICOS/mk partition. This guideline applies regardless of the other options discussed in this chapter.

Because distributed mode is so dependent upon system calls, you should not use it for sending small, frequent messages.

If you are using PVM to communicate between a UNICOS/mk system and a UNICOS system with Cray floating-point hardware, and you specify PvmDataDefault when calling pvm_initsend(3), PVM converts the data between IEEE and Cray formats for all forms of typed data. This is not done very efficiently on the UNICOS end.

You can perform data conversion efficiently on the UNICOS system, however, by using the data conversion functions available in the UNICOS Fortran libraries (see the Application Programmer's Library Reference Manual). If you are using PVM to transfer the data, pack and unpack it with the byte options (pvm_pkbyte(3), pvm_upkbyte(3), or the Fortran BYTE1 option) and then call CRAY2IEG(3) or IEG2CRAY(3), as appropriate. If you are using file I/O, call CRAY2IEG or IEG2CRAY, as appropriate, on the data you are about to write from the UNICOS system or have just read from the UNICOS/mk system.

If you are using file I/O, an easier option is to use a Fortran I/O feature that automatically converts data as it is read or written. These techniques are described in the Application Programmer's I/O Guide.