The WorkShop Debugger lets you debug threaded applications as well as programs that use multiple processes spawned by fork or sproc. You can also control a single process or all members of a process group, attach child processes, and specify that spawned processes inherit traps from the parent process. The Trap Manager provides special commands to facilitate debugging multiple processes by setting traps that apply to the entire process group.
The Multiprocess View window is for use by C, C++, and Fortran users. If you are debugging Ada code, you should use the Task View window available through the View menu of the Main View window (see “Task View” in Appendix A).
Currently, Multiprocess View handles the following multiple process situations:
True multiprocess program, which refers to a tightly integrated system of sproc'd processes, generated by the MIPSpro Automatic Parallelization Option. For more information on parallel processing, see the auto_p(5) man page, or the MIPSpro Automatic Parallelizer Programmer's Guide.
Auto-fork application, which is a process that spawns a child process and then runs in the background.
Fork application, which is a process that spawns child processes and can interact with them.
Locally distributed application, which is an application that involves two different executables running in different processes on the same host coordinated by a rendezvous mechanism.
MPI single system image application, which is an MPI application that runs on the same host.
This chapter discusses the details of multiprocess debugging in WorkShop and includes the following topics:
The Multiprocess View window is brought up by selecting Admin -> Multiprocess View from the menu bar of the Main View window.
This window can display individual processes or operate on a process group. By default, a process group includes the parent process and all descendants spawned by sproc. Processes spawned with fork during the session can be added to the process group automatically when they are created. For a program compiled with the MIPSpro Automatic Parallelization Option, a process group includes all threads generated by the option. Any process to which you have read/write access can also be added to the process group. All sproc 'd processes must be in the same process group, since they share information.
 | Note: Any child process that performs an exec with setuid (set user ID) enabled will not become part of the process group. |
Each process in the session can have a standard main view window session associated with it. However, all processes in a process group appear on a single Multiprocess View window.
When debugging multiprocess applications, you should disable the SIGTERM signal by selecting Views -> Signal Panel from the Main View window menu bar. Although multiprocessing debugging is possible with SIGTERM enabled, the multiprocess application may not terminate gracefully after execution is complete.
The first step in debugging multiple processes is to invoke the Debugger with the parent process. Then select Admin -> Multiprocess View from the menu bar.
The following figure shows a typical Multiprocess View window.
The process display area of the Multiprocess View lists the status of all processes and threads in the process group. For definitions of the various status and states, see “Status of Processes” in Appendix A.
To get more information about a process or thread displayed in the process display area, right-click on the process or thread entry. A Process menu pops up which is applicable to the selected entry. From this menu you can do the following:
change entry focus
create a new window
focus attention to a user-entered thread
show thread-specific details
add or remove an entry
For complete details about the Process menu, see “Controlling Multiple Processes” in Appendix A.
The Multiprocess View window uses the same control buttons as the Main View window with the following exceptions:
Buttons are applied to all processes as a group.
There are no Return, Print, or Run buttons.
Control buttons in the Multiprocess View window have the same effect as clicking the corresponding button in the Main View window of each individual process. For definitions of the buttons, see “Multiprocess View Control Buttons” in Appendix A.
As discussed in Chapter 5, “Setting Traps”, the trap qualifiers [all] and [pgrp] are used in multiprocess analysis. The [all] entry stops or samples all processes when a trap fires. The [pgrp] entry sets the trap in all processes within the process group that contains the trap location. The qualifiers can be entered by default by using the Stop All Default and Group Trap Default selections, respectively, in the Traps menu of Trap Manager. The Trap Manager is brought up from the Views menu of the Main View window.
The Multiprocess View supports a hierarchical view of your pthreaded applications. Select the folder icons of your choosing to get more information about a process or thread.
Perform the following from within the Multiprocess View window to get additional information about a process or thread:
Double-click on a folder icon.
The process display expands to show its pthreads.
Double-click to select the pthread of your choosing.
The call stack for that thread displays.
To add a process, select Add from the Process menu. In the Switch Dialog dialog window, select one of the listed processes or enter a process ID in the Process ID field and click the OK button.
To remove a process, click on the process name in the Multiprocess View window and select Remove from the Process menu. Be aware that a process in a sproc process group cannot be removed. Likewise, you cannot remove a pthread from a pthread group.
The Preferences option in the Config menu brings up the Multiprocess View Preferences dialog. The preferences on this dialog lets you determine when a process is added to the group, specify process behavior, specify the number of call stack levels to display, and so forth.
For details about Multiprocess View Preference options, see “Controlling Preferences” in Appendix A.
This section uses a C program that generates numbers in the Fibonacci sequence to demonstrate the following tasks when using the debugger to debug multiprocess code:
Stopping a child process on a sproc
Using the buttons in the Multiprocess View window
Setting traps in the parent process only
Setting group traps
The fibo program uses sproc to split off a child process, which in turn uses sproc to split off a grandchild process. All three processes generate Fibonacci numbers until stopped. You can find the source for fibo.c in the /usr/demos/WorkShop/mp directory. A listing of the fibo.c source code follows:
#include <stdio.h>
#include <sys/types.h>
#include <sys/prctl.h>
int NumberToCompute = 100;
int fibonacci();
void run(),run1();
int fibonacci(int n)
{
int f, f_minus_1, f_plus_1;
int i;
f = 1;
f_minus_1 = 0;
i = 0;
for (; ;) {
if (i++ == n) return f;
f_plus_1 = f + f_minus_1;
f_minus_1 = f;
f = f_plus_1;
}
}
void run()
{
int fibon;
for (; ;) {
NumberToCompute = (NumberToCompute + 1) % 10;
fibon = fibonacci(NumberToCompute);
printf("%d'th fibonacci number is %d\n",
NumberToCompute, fibon);
}
}
void run1()
{
int grandChild;
errno = 0;
grandChild = sproc(run,PR_SADDR);
if (grandChild == -1) {
perror("SPROC GRANDCHILD");
}
else
printf("grandchild is %d\n", grandChild);
run();
}
void main ()
{
int second;
second = sproc(run1,PR_SADDR);
if (second == -1)
perror("SPROC CHILD");
else
printf("child is %d\n", second);
run();
exit(0);
} |
Perform the following to start, compile the program, and run the Debugger:
Copy the program source from the demo directory as follows:
% cp /usr/demos/WorkShop/mp .
Compile fibo.c by entering the following command:
% cc -g fibo.c -o fibo
Invoke the Debugger on fibo as follows:
% cvd fibo &
Call up the Multiprocess View by selecting Admin -> Multiprocess View from the Main View menu bar.
The next section uses the fibo program to illustrate some of the functionality of the Multiprocess window.
To examine each process as it is created, you must set preferences so that each child process created stops immediately after being created. The following steps show how this can be done:
Select Config -> Preferences from the menu bar in the Multiprocess View window.
Toggle off Resume child after attach on sproc in the Multiprocess View Preferences window.
Toggle off Copy traps to sproc'd processes so you can experiment with setting traps later.
Click on the OK button to accept the changes.
Click on the Run button in the Main View window to execute the fibo program.
Watch the Multiprocess View window, you will see the main process appear and spawn a child process, which stops as soon as it appears. This is because you turned off the Resume child after attach on sproc option. Notice also that the Main View window switched to the stopped child process.
Click on the Stop button in the Multiprocess View window.
The control buttons on the Multiprocess View window may be used to control all processes simultaneously, or the control buttons on any Main View window may be used to control that individual process separately.
Click on the first line (that is, the main process) in the process pane of the Multiprocess View window to highlight this line.
Select Process -> Create new window from the menu bar of the Multiprocess View window.
A new Main View window displays with a debug session for the main process.
Note: You may get a warning that .../write.s is missing. This refers to assembly code and can be ignored. The new Main View window will not have source in its source pane.
Select Views -> Call Stack from the menu bar of the Main View window you just created to create a Call Stack window.
Double-click on the line in the Call Stack window that contains run (). This brings up the fibo.c source for the main process in the Main View window.
Select Admin -> Close from within the Call Stack window to close it.
Click on Cont in the Multiprocess View window. The first child, created in Step 5, now spawns a grandchild process that stops in _nsproc, as shown here:
A Main View window switches to the new stopped process. Click on Stop in the Multiprocess View window.
Repeat steps 7 through 11 to bring up a Main View window for the parent process.
The instructions in this section assume that you have just run the tutorial in “Using Multiprocess View to Control Execution”.
This section shows you how to use the Trap Manager to set traps that affect one or all processes in the fibo process group. For complete information on using the Trap Manager, refer to Chapter 5, “Setting Traps”.
Select Views -> Trap Manager from the Main View window for the parent process. Traps are specific to the processes in the Main View window in which they are set.
Select Display -> Show Line Numbers (from the same Main View window) to turn on line numbering in the source pane, if not already showing.
Click to the left of line 32, to set a breakpoint/stop trap for the parent process. Line 32 reads as follows:
32 Number to Compute = (NumberToCompute + 1) % 10
Line 32 highlights in red to indicate that a breakpoint has been set. A corresponding trap command appears in the Trap text box in the Trap Manager window; and the trap is added to the list on the Active Traps list of the same window. Remember, this trap will affect only the parent process.
Click on the Cont button in the Multiprocess View window. The parent process has stopped, but the other processes are probably still running.
Insert the word pgrp (that is, “process group”) after the word stop in the Trap field of the Trap Manager window.
The trap should now read Stop pgrp at .... As the command suggests, pgrp affects the whole process group.
Click on the Modify button.
The trap now affects two child processes. Watch the Multiprocess View window to see the running processes in the process group stop at the trap on line 32.
Select Traps -> Group Trap Default from the Trap Manager window. Any additional traps that you set using the Trap Manager affect the entire process group. Any previously set traps are not be affected.
Select the text of line 23, found in the source pane of the Main View window associated with the parent process. This line reads as follows:
23 f_minus_1 = f;
Select Traps -> At Source Line from the menu bar of the Trap Manager window. The trap you have just set includes the modifier pgrp.
Select Admin -> Exit from any Main View window to close your session and end this tutorial.
The section of this chapter presents a few standard techniques to assist you in debugging a parallel program. This section shows you how to debug the sample program.
See also Chapter 2, “Basic Debugger Usage” for important related information.
Debugging a multiprocessed program is more involved than debugging a single-processor program. Therefore, you should debug a single-processor version of your program first and try to isolate the problem to a single parallel DO loop.
After you have isolated the problem to a specific DO loop, change the order of iterations in a single-processor version. If the loop can be multiprocessed, then the iterations can execute in any order and produce the same answer. If it cannot be multiprocessed, you will see that changing the order in which the loops execute will cause the single-processor version to produce wrong answers. If wrong answers are produced, you can use standard single-process debugging techniques to find the problem. (See Chapter 2, “Basic Debugger Usage” for important related information.)
If this technique fails, you must debug the multiprocessed version. To do this, compile your code with the -g and -FLIST:=ON flags. The -FLIST:=ON flags save the file containing the multiprocessed DO loop Fortran code in a file called total.w2f.f and a file tital.rii and an rii_files directory.
This section shows you how to debug a small segment of multiprocessed code. The source code for this tutorial, total.f, can be found in the directory /usr/demos/WorkShop/mp.
A listing of this code is as follows:
program driver
implicit none
integer iold(100,10), inew(100,10),i,j
double precision aggregate(100, 10),result
common /work/ aggregate
result=0.
call total(100, 10, iold, inew)
do 20 j=1,10
do 10 i=1,100
result=result+aggregate(i,j)
10 continue
20 continue
write(6,*)' result=',result
stop
end
subroutine total(n, m, iold, inew)
implicit none
integer n, m
integer iold(n,m), inew(n,m)
double precision aggregate(100, 100)
common /work/ aggregate
integer i, j, num, ii, jj
double precision tmp
C$DOACROSS LOCAL(i,ii,j,jj,num)
do j = 2, m-1
do i = 2, n-1
num = 1
if (iold(i,j) .eq. 0) then
inew(i,j) = 1
else
num = iold(i-1,j) +iold(i,j-1) + iold(i-1,j-1) +
& iold(i+1,j) + iold(i,j+1) + iold(i+1,j+1)
if (num .ge. 2) then
inew(i,j) = iold(i,j) + 1
else
inew(i,j) = max(iold(i,j)-1, 0)
end if
end if
ii = i/10 + 1
jj = j/10 + 1
aggregate(ii,jj) = aggregate(ii,jj) + inew(i,j)
end do
end do
end |
In the program, the local variables are properly declared. The inew always appears with j as its second index, so it can be a share variable when multiprocessing the j loop. The iold, m, and n are only read (not written), so they are safe. The problem is with aggregate. The person analyzing this code deduces that, because j is always different in each iteration, j/10 will also be different. Unfortunately, since j/10 uses integer division, it often gives the same results for different values of j.
While this is a fairly simple error, it is not easy to see. When run on a single processor, the program always gets the right answer. Sometimes it gets the right answer when multiprocessing. The error occurs only when different processes attempt to load from and/or store into the same location in the aggregate array at exactly the same time.
Perform the following to debug this code:
Create a new directory for this exercise:
% mkdir demos/mp
cd to the new directory and copy the following program source into it:
% cp /usr/demos/WorkShop/mp .
Edit the total.f file in a shell editor, such as vi:
% vi total.f
Reverse the order of the iterations for demonstration purposes.
Replace
do j = 2, m-1
with
do j = m-1, 2, -1
This still produces the right answer with one process running, but the wrong answer when running with multiple processes. The local variables look right, there are no equivalence statements, and inew uses only simple indexing. The likely item to check is aggregate. Your next step is to look at aggregate with the Debugger.
Compile the program with -g option as follows:
% f77 -g -mp total.f -o total
If your debugging session is not running on a multiprocessor machine, you can force the creation of two threads, for example purposes, by setting an environment variable. If you use the C shell, type:
% setenv MP_SET_NUMTHREADS 2
Is you use the Korn or Bourne shell, type:
$ MP_SET_NUMTHREADS=2 $ export MP_SET_NUMTHREADS
Enter the following to start the Debugger:
% cvd total &
The Main View window displays.
Select Display -> Show Line Numbers from the Main View menu bar to show the line numbers.
Select Source -> Go To Line from the Main View menu bar.
And enter 44.
Line 44 is as follows:
aggregate(ii,jj) = aggregate(ii,jj) + inew(i,j)
You will now set a stop trap at this line, so you can see what each thread is doing with aggregate, ii, and jj. You want this trap to affect all threads of the process group. One way to do this is to turn on trap inheritance in the Multiprocess View Preferences dialog. To open this dialog, select -> Admin -> Multiprocess View from the Main View menu bar to open the Multiprocess View window.
Then, select Config -> Preferences from within the Multiprocess View window.
Another way is to use the Trap Manager to specify group traps, as follows.
Select Views -> Trap Manager from the Main View window menu bar to open the Trap Manager.
Select Traps -> Group Trap Default from the Trap Manager window.
Click-drag to select line 44 in the Main View window.
Open the Trap Manager window from the Main View window menu bar by using Views -> Trap Manager.
Then select Traps -> At Source Line from the Trap Manager window.
This sets a stop trap that reads as follows in the cvd pane of the Main View window:
Stop pgrp at file /usr/demos/WorkShop/mp/total.f line 44
Select Admin -> Multiprocess View from the menu bar in the Main View window to monitor status of the two processes.
You are now ready to run the program.
Click the Run button in the Main View window.
As you watch the Multiprocess View, you see the two processes appear, run, and stop in the function _mpdo_total_1. It is unclear, however, if the Main View window is now relative to the master process, or that it has switched to the slave process.
Right-click on the name of the slave process in the Multiprocess View window and select Process -> Create a new window.
A new window is displayed that launches a debug session for the process. Now, both master and slave processes should display in respective Main View windows.
Invoke the Variable Browser as follows from the Menu Bar of each process: Views -> Variable Browser.
Look at ii and jj in the following figure.
They have the same values in each process; therefore, both processes may attempt to write to the same member of the array aggregate at the same time. So aggregate should not be declared as a share variable. You have found the bug in your parallel Fortran program.
cvd supports the debugging of IRIX 6.4 and 6.5 pthreads. You can view pthread creation and execution through the Multiprocess View window. Through this window you can:
View a hierarchal display of a threaded application
View a process/pthread relationship
Expand individual call stacks
C, C++, and Fortran users should use the Multiprocess View window when debugging pthreads. Ada users should use the Task View window.
The next sections give hints on debugging pthreaded programs, list differences between 6.4 and 6.5 threads, and illustrate how to debug a program that uses IRIX 6.5 pthreads.
The ability to “continue” or “free run” a single POSIX pthread under IRIX 6.5 is available at the user level with WorkShop release 2.8. However, use of this new debugging feature can, in certain specific circumstances, lead to anomalous and possibly confusing behavior. Such behavior occurs when the single thread that is continued or free run encounters either a “blocking” or “scheduling” situation in the operating system or the pthreads library.
When such situations arise, the operating system (or, in some cases, the pthreads library) must take action to dispose of the single continued or free run thread and, possibly, newly created threads. In the course of this action the debugging user will see things occur, with both the single continued or free run thread as well as all other threads, that are confusing because complex thread scheduling algorithms are invoked by both the operating system and the pthreads library to recover from the original blocking or scheduling incident. Debugging true POSIX pthreads is difficult, and users of this new feature, allowing a continue or free run of a single 6.5 POSIX pthread, will gain even more appreciation of this fact.
This feature has been used internally for some time by the WorkShop debugger. The continue or free run of a single 6.5 pthread is used each time a user requests a single thread step-over of a function. The single thread is allowed to free run through the function which is being stepped over. Thus, if any blocking or scheduling situations occur in the course of this stepping over and associated free run of a single thread, then anomalous behavior can, and does, occur. This is described in the following subsections.
Scheduling anomalies may occur when the single 6.5 POSIX pthread which is being continued or free run creates a new pthread via a call to the pthread_create routine. At the time of the call to pthread_create the OS kernel and the pthreads library get into a complex algorithm in deciding how to create the new child pthread. An actual OS kernel thread (OS kernel threads are not available at the user level -- they differ from the user level pthread) must be either created anew or found elsewhere to support the user's new child pthread.
First, assume the OS kernel thread is to be found elsewhere, depending on a vast number of things (for example, number of CPUs, environment variables, and so on). The OS kernel may (this is very non-deterministic) decide to just put the child pthread on a ready queue, in need of an OS kernel thread. Thus the child does nothing immediately.
Meanwhile, if the parent pthread (via a call to the pthread_cond_wait routine) monitors the child pthread's struggle for life, it (the parent) gets parked on a mutex (mutual exchange lock) because the child obviously has not been created yet; it is on the ready queue.
The parent pthread's OS kernel thread becomes available, which causes the OS scheduler to check for work for this newly freed OS kernel thread. It finds the child sitting in the ready queue and assigns the parent's OS kernel thread to the new child pthread. The child then runs to completion and releases its (parent's old) OS kernel thread. The parent, checking for the child's new life via pthread_cond_wait, now recaptures its OS kernel thread and things appear to work correctly.
Now, assume the OS kernel thread required by the new child pthread must be created anew. The child is not placed on the 'ready queue'. Again, this is a non-deterministic decision which depends on a large number of variables (number of CPUs, and so on). The OS kernel creates a new OS kernel thread for the child pthread and “engages” it (the child) to that new OS kernel thread.
However, “marriage” of the OS kernel thread and the new child pthread cannot occur until the new OS kernel thread actually runs. This never occurs because, in allowing the single parent 6.5 pthread to continue or run free, it was requested that only one user pthread be run -- the parent.
If the parent, however, is using pthread_cond_wait to monitor the new life of its child, then it (the parent) is parked on a mutex waiting for the child to run. The parent awaits the child but the child cannot run because only one pthread, the parent, has been requested to run. The debugger displays “running” as the overall status and this is because no events of interest are occurring. Everything is waiting on everything else. Things are not working.
Blocking anomalies occur when the single 6.5 POSIX thread which is being continued or free run encounters a blocking condition in the course of its running. Blocking has three distinct types:
Blocking syscalls in the OS kernel (see “Blocking Kernel System Calls” in Appendix A, for a list). When one of these kernel syscalls is blocked by another thread's usage, the OS kernel decides what the next move is regarding the OS kernel thread attached to the user pthread making the call. Control could just transfer to another application, to disk I/O, or whatever.
These syscalls are all I/O-related. The OS kernel thread is, in effect, “blocked”, and it is immediately available for reassignment. The best example of a blocking kernel syscall is writev, which is used by the common library routine printf.
Various lock blocking in the pthread library, such as mutex (mutual exchange lock). This occurs in user space (libc, user code, and so on). The pthread library senses that a pthread is going to block due to another pthread's usage. Control transfers to the usync_control routine, which eventually calls a blocking kernel syscall (see the preceding item in this list). Again, the OS kernel decides the fate of the associated OS kernel thread. Unexpected things could start running.
Other lock blocking in the pthread library, whereby the pthread library senses that a user pthread is going to block but does not go off to usync_control. Instead it goes to the pthread_scheduler in the pthread library for the disposition of the associated OS kernel thread. The pthread_scheduler then reassigns the associated OS kernel thread to another user pthread and unexpected things could start running.
Observe the following guidelines when debugging pthreaded programs:
Be aware that the cvmain (Main View) for release 2.8 (and later) contains options (such as Continue, StepOver, StepInto, and Return) that are for a single 6.5 pthread -- the pthread that is displayed, or the focus thread. Do not use the Main View options unless you intend to use them for a single thread.
C++ exception handling works per process not per thread.
Using the step over function on a pthread_exit may produce unexpected results.
Use Multiprocess View not Task View.
Use the WorkShop tools instead of dbx for 6.5 pthread debugging whenever possible.
Do not do a Next of printf.
The following differences exist between the IRIX 6.4 and the IRIX 6.5 implementation of pthreads:
In the IRIX 6.5 implementation, many user threads are serviced by a few kernel micro-threads. This can result in data race problems that may occur as kernel micro-threads switch from one user pthread to another.
You can alleviate this problem with the following command:
% ccall pthread_setconcurrency n
n is the number of kernel micro-threads made available to service user threads. In this way, you can increase the number of kernel micro-threads available. However, you must not specify a value for n that is greater than the number of physical CPUs on your system.
The Multiprocess View window shows different sets of threads or processes depending on which version of IRIX you are running. (This is because in IRIX releases before IRIX 6.5, threads are handled as sprocs, not as threads.)
Pthread debugging is highly variable not only from environment to environment but also from IRIX release to IRIX release. Because of this, it is not possible to provide a representative pthread debugging tutorial that can be used by all users.
See “User-Level Continue of Single 6.5 POSIX Pthread”, for an in-depth description of current pthread implementation in IRIX.
When debugging IRIX 6.5 (or greater) pthreads, if you attempt to 'step over' a function call, there is a possibility that pthreads will block. This blocking can occur if you attempt to step-over either a direct or indirect call to one of the following:
One of several blocking pthread library routines (see “Blocking pthreads Library Routines”)
One of several blocking kernel syscalls (see “Blocking Kernel System Calls” in Appendix A, for a list of the syscalls).
If a pthread does block in either of these situations, an internal breakpoint is reached at _SGIPT_sched_block (for blocking pthread library routines) or _SGIPT_libc_blocking (for blocking kernel syscalls).
Without these internal breakpoints, when a pthread blocks, control is returned to the OS kernel, at which point any number of events could occur, including a recycling of the kernel micro-thread attached to the user pthread. This might allow another user pthread to resume execution, thereby causing the debugger to appear to be running or appear to be hung because the original thread which blocked is not allowed to run to its return point (since it had its microthread swapped out underneath it).
The OS kernel uses complex algorithms to determine what action to take when a pthread blocks. The debugger's use of the internal breakpoints allows you to take back a degree of control over these complex algorithms by deciding what to do with a thread that has blocked in either _SGIPT_sched_block or _SGIPT_libc_blocking.
Usually you can simply use Continue All Pthreads to release the blocking condition or continue a different individual pthread (different from the one that blocked).
For OS level 6.5 pthreads, the Libpthread entry point _SGIPT_libc_blocking is entered when a specific pthread blocks in a kernel syscall. See “Blocking Kernel System Calls” in Appendix A, for a list of these syscalls.
There are many library routines that can call one of these blocking system calls; it is impossible to list all such routines which utilize a blocking system call. Users must be knowledgeable enough to know that if, for example, they call the library routine printf, it eventually calls writev() which is a blocking system call and thus may block.
For OS level 6.5 pthreads, the Libpthread entry point _SGIPT_sched_block is entered when a specific pthread blocks in the pthread library. The following routines are known to block:
pthread_cond_wait()
pthread_cond_timedwait()
pthread_mutex_lock()
pthread_join()
pthread_exit()
pthread_rwlock_rdlock()
pthread_rwlock_wrlock()
sem_wait()
cvd supports the debugging of a single system image MPI application. The debugging session is set up so that initially mpirun is being debugged. The following is the typical command line used to invoke cvd on an MPI application:
% cvd mpirun -args -np 2 MPI_app_name |
This example command line indicates that the arguments -np 2 MPI_app_name will be passed to mpirun and that cvd will initially be focused on the mpirun process.
An entry point into the MPI application can be used to set a pending trap (breakpoint) in the MPI application. This breakpoint will be resolved when the Run button is activated and the actual MPI application is running. If the breakpoint target is valid, the MPI application will stop at the breakpoint and further debugging can be done.
The use of the Multiprocess View for debugging MPI applications is very similar to what has been presented in the previous sections of this chapter.
The current implementation does not filter out other processes created by login shells so some extra processes may be shown in the Multiprocess View.