Debuggers that ship with Altix machines are provided by Intel and GNU. The Intel debugger is called idb. Like its GNU counterpart, gdb, it is a command line debugger that can attach to a running process or debug a core file. It supports debugging programs written in all of the languages supported by the Intel compilers and has been improved in the area of debugging multithreaded applications (OpenMP or pthreads). By default, it supports dbx commands though it can also (via option) support gdb commands. Table 5-4 lists some of the more commonly used commands of these debuggers.

A full set of commands supported by the idb and gdb debuggers can be found by listing their respective man pages idb(1) and gdb(1). Documentation on gdb is available at the GNU web site: http://www.gnu.org/software/gdb/documentation/.

In addition to the previously mentioned Etnus TotalView debugger (a discussion of which is beyond the scope of this manual), there also exists a graphical front-end interface to either gdb (by default) or idb called DataDisplayDebugger or ddd. (For information, see http://www.gnu.org/software/ddd/ .)

To invoke ddd running idb in dbx mode type, execute the following:

% ddd --debugger idb --dbx ./a.out

This creates a debugger console window where debugger commands can be typed. This also creates window panes for the source code, disassembled code, and array values. You can use the View menu to switch these panes on and off.

Figure 5-1 shows a typical ddd display.

Figure 5-1. Typical ddd display

Some commonly used commands from Table 5-4 are found in the Program pull-down menu and command.

The following site contains a thorough repository of information about ddd: http://www.gnu.org/manual/ddd/html_mono/ddd.html . The ddd(1) man page is also useful.

Another GUI debugger available for Altix is called the distributed debugging tool (DDT), available from Streamline Computing. DDT focuses on providing support for debugging parallel applications. For more information see: http://www.streamline-computing.com/softwaredivision_1.shtml .

The clock_gettime function returns the current value for the specified clock (passed in by the first parameter clock_id). The value is returned through a pointer to a timespec structure consisting of two long integers containing values for seconds and nanoseconds.

Depending on the clock's resolution, it may be possible to obtain the same time value with consecutive reads of the clock. The time value may also have a higher precision then the resolution of the clock.

The resolution of any clock can be obtained by calling the c lock_getres() function. The resolution of the clock will be returned through a pointer to a timespec structure.

On Altix systems the list of supported clocks differs from those on IRIX. These clocks are:

The CLOCK_SGI_CYCLE and CLOCK_SGI_FAST supported on IRIX systems are not supported on Altix system.

 Example 5-3 gets the resolution of these clocks and then uses CLOCK_PROCESS_CPUTIME_ID to time how long it took to do so.

%cat tr.c
#include <stdio.h>
#include <time.h>
main()
{
   int i;
   struct timespec N;
   i = clock_getres(CLOCK_REALTIME, &N);
   if (i == 0)
      printf(“Resolution is %ld seconds and  %lld nanoseconds for
CLOCK_REALTIME \n”,N.tv_sec, N.tv_nsec);
   i = clock_getres(CLOCK_PROCESS_CPUTIME_ID, &N);
   if (i == 0)
      printf(“Resolution is %ld seconds and  %lld nanoseconds for
CLOCK_PROCESS_CPUTIME_ID\n”,N.tv_sec, N.tv_nsec);
   i = clock_getres(CLOCK_THREAD_CPUTIME_ID, &N);
   if (i == 0)
      printf(“Resolution is %ld seconds and  %lld nanoseconds for
CLOCK_THREAD_CPU_TIME_ID\n”,N.tv_sec, N.tv_nsec);
   i = clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &N); 
   if (i == 0) 
      printf(“Elapsed time is %ld seconds and  %lld nanoseconds
\n”,N.tv_sec, N.tv_nsec);
}
%icc tr.c -lrt
%./a.out
Resolution is 0 seconds and  976562 nanoseconds for CLOCK_REALTIME 
Resolution is 0 seconds and  40 nanoseconds for CLOCK_PROCESS_CPUTIME_ID
Resolution is 0 seconds and  40 nanoseconds for CLOCK_THREAD_CPU_TIME_ID
Elapsed time is 0 seconds and  9125600 nanoseconds 

As mentioned before, the CLOCK_REALTIME clock calls gettimeofday and has a resolution of 1/1024 seconds. The other two clocks provide much better resolution.

MPI applications can take advantage of two portable timing routines provided with the library calls MPI_Wtime() and MPI_Wtick(). Both calls return double precision floating point numbers which represent the time and resolution in seconds respectively. They also read the SN.RTC and have submicrosecond resolution that is synchronized across nodes.

The following is a Fortran example that uses these timing routines.

%cat m.f
          PROGRAM M
          INCLUDE “mpif.h” 
          DOUBLE PRECISION TME1
          DOUBLE PRECISION TME2
          DOUBLE PRECISION ELAPSED
          DOUBLE PRECISION RES
          INTEGER error
          CALL MPI_INIT(error)
          TME1=MPI_WTIME()
          RES=MPI_WTICK()
          PRINT *,”RESOLUTION IS “,RES
          PRINT *,”TIME1 IS “,TME1
          TME2=MPI_WTIME()
          PRINT *,”TIME2 IS “,TME2
          ELAPSED = TME2-TME1
          PRINT *,”ELAPSED TIME IS “,ELAPSED
          CALL MPI_FINALIZE(error)
          END
%ifort m.f -o m -lmpi
%mpirun -np 1 m
 RESOLUTION IS   4.000000000000000E-008
 TIME1 IS    207612.193793240     
 TIME2 IS    207612.196069960     
 ELAPSED TIME IS   2.276720013469458E-003

Performance Tools on Altix are available from both Intel and the open source community (including SGI contributions). The following sections briefly document the following tools:

VTune (see: http://www.intel.com/software/products/vtune/vlin/ ) provides call stack sampling as well as comprehensive support for event based sampling of the Itanium PMU. Two versions of the tool are available. The first requires that the collected data be copied from the Altix to Windows based machine where the analysis takes place under a GUI framework. The second is a command line tool natively hosted on the Itanium system where the data was collected.

For additional information, see http://ssales.corp.sgi.com/products/servers/altix350/intelfaq.html and scroll down to find the comparison chart for VTune 7.1 versus VTune 2.0.

The gprof tool requires that the application being analyzed be compiled with the -pg option of gcc. When run, the resulting program creates a gmon.out file which contains information that can be used to generate three types of reports by the command line based gprof tool:

For further information, see the gprof man page (%man gprof).

The pfmon tool uses the Itanium Performance Monitoring Unit (PMU) to count and sample during runs made on unmodified binaries. It can function on a per-process basis or take a system-wide view on a dedicated CPU or a set of CPUs. It also can monitor events at the user level or at the system level.

The -l option to pfmon lists the (currently 475) supported events. These event names can then be used as arguments to the -e option which specifies which events to monitor. For example, executing the following command will monitor four different events:

%pfmon -ecpu_cycles,ia64_inst_retired_this,nops_retired, \ back_end_bubble_all a.out

Note: that there is no space between the -e option and the name of the first event or between the commas.

This is recommended as the first step in using pfmon to count cycles, instructions, NOPs and back-end stall cycles.

More information about pfmon can be found in its man page (%man pfmon) and a user guide normally installed under /usr/share/doc/pfmon-2.0/pfmon_usersguide.txt.

profile.pl is a Perl script interface to pfmon. It uses dplace to bind the application to specific processors and invoke other Perl scripts to generate a readable report. It requires that the application contain symbol table information (i.e., not be stripped). Table 5-5 shows some commonly used options to profile.pl.

For more information see the profile.pl(1), analyze.pl(1), and makemap.pl(1) man pages.

SGI Histx is a performance analysis tool designed to complement pfmon. The software is designed to run on Altix systems only. Used internally by SGI developers and benchmarkers, the product is offered as a service to SGI customers with a no fee end-user proprietary license via the SGI Download Cool Software (DCS) Web site. Customers wishing to use SGI Histx should be aware that there is no support planned for this product and customers who use it accept it “as is”.

Histx consists of a group of tools:

First there are three data collection programs:

Then there are three filters for performance data postprocessing and display:

The histx command does not have a man page; however, typing the command by itself (or %histx -h) will print relevant options. For example:

%histx
usage: histx [-b width] [-f] [-e source] [-h] [-k] -o file [-s type] [-t signo] command args...
  -b  specify bin bits when using ip sampling: 16,32 or 64 (default: 16)
  -e  specify event source (default: timer@1)
  -f  follow fork (default: off)
  -h  this message (command not run)
  -k  also count kernel events for pm source (default: off)
  -l  include line level counts in ip sampling report (default: off)
  -o  send output to file.<prog>.<pid> (REQUIRED)
  -s  type of sampling (default: ip)
  -t  `toggle' signal number (default: none)

Event sources:
   timer@N       profiling timer events. A sample is recorded
                 every N ticks.
   pm:<event>@N  performance monitor events.  A sample is
                 recorded whenever the number of occurrences of
                 <event> is N larger than the number of occurrences
                 at the time of the previous sample.
   dlatM@N       A sample is recorded whenever the number of
                 loads whose latency exceeded M cycles is N larger
                 than the number at the time of the previous
                 sample. M must be a power of 2 between 4 and
                 4096

Types of sampling:
   ip              Sample instruction pointer
   callstack[N]    Sample callstack.  N, if given, specifies
                   the maximum callstack depth (default: 8)

Notes:
   A list of valid performance monitor <event>s can be found
   in Intel manuals.
   `command' must not be compiled using the `-p' compiler flag
   One tick is about 0.977 milliseconds

Thus

%histx -e timer@1 -o out ./a.out

will generate the output file out.a.out.XXXX (where XXXX is the process id) which provides the number of timer ticks for each function in the a.out file.