Here are the suggested steps for designing and implementing a PMDA:

All PMDAs are launched and controlled by the pmcd process on the local host. Requests from the monitoring tools are received by pmcd and forwarded to the PMDAs. Responses, when required, are returned through pmcd to the clients. The requests fall into a small number of categories and the PMDA must handle each request type. For a DSO PMDA each request type corresponds to a method in the agent. For a daemon PMDA each request translates to a message or protocol data unit (PDU) that may be sent to a PMDA from pmcd.

For daemon PMDA the following request PDUs must be supported:

Each PMDA is associated with a unique domain number that is encoded in the domain field of metric and instance identifiers, and pmcd uses the domain number to determine which PMDA can handle the components of any given client request.

Each PMDA is required to implement a function that handles each of the request types. By implementing these functions as library routines, a PMDA can be implemented as a dynamically shared object (DSO) and attached by PMCD at run time with the dlopen call; see dlopen(3). This eliminates the need for an IPC layer (typically a UNIX pipe) between each PMDA and pmcd, because each request becomes a function call rather than a message exchange. The required library routines are detailed in the section “DSO Interface”.

A PMDA that interacts with pmcd in this fashion must abide by a formal initialization protocol so that pmcd can discover the location of the library routines that are subsequently called with function pointers. When a DSO PMDA is installed, the pmcd configuration file /etc/pmcd.conf is updated to reflect the domain and name of the PMDA, the location of the shared object, and the name of the initialization routine. The initialization sequence is discussed in the section “Initializing a PMDA”.

# cd /var/pcp/pmdas/simple 
# cat /etc/pmcd.conf 
# Name  Id      IPC     IPC Params      File/Cmd 
irix    1       dso     irix_init       libirixpmda.so 
pmcd    2       dso     pmcd_init       pmda_pmcd.so 
proc    3       dso     proc_init       pmda_proc.so 
# ./Install 
You will need to choose an appropriate configuration for installation of the “simple” Performance Metrics Domain Agent (PMDA).
collector   collect performance statistics on this system
  monitor     allow this system to monitor local and/or remote systems
  both        collector and monitor configuration for this system
Please enter c(ollector) or m(onitor) or b(oth) [b] both 
Updating the Performance Metrics Name Space (PMNS) ... 
Installing pmchart view(s) ...
Install simple as a daemon or dso agent? [daemon] dso 
...
Check simple metrics have appeared ... 4 metrics and 6 values
# cat /etc/pmcd.conf 
# Name  Id      IPC     IPC Params      File/Cmd 
irix    1       dso     irix_init       libirixpmda.so 
pmcd    2       dso     pmcd_init       pmda_pmcd.so 
proc    3       dso     proc_init       pmda_proc.so 
simple  253     dso     simple_init     pmda_simple.so  

A DSO PMDA provides the most efficient communication between the PMDA and PMCD. However, this approach has some disadvantages resulting from the DSO PMDA being the same process as pmcd, namely:

Consequently, many PMDAs are implemented as a daemon process.

The libpcp_pmda library is designed to allow simple implementation of a PMDA that runs as a separate process. The library routines provide a message passing layer acting as a generic wrapper that accepts PDUs, makes library calls using the standard DSO PMDA interface, and sends PDUs. Therefore, it is possible to implement a PMDA as a DSO and then install it as either a daemon or a DSO, depending on the presence or absence of the generic wrapper.

The pmcd process launches a daemon PMDA with fork and execv, so a pipe can be easily connected to the PMDA using standard input and output. The pmcd process may also connect to a daemon PMDA using TCP/IP or UNIX domain sockets; see inet(7) or unix(7).

# cd /var/pcp/pmdas/simple 
# ./Install 
... 
Install simple as a daemon or dso agent? [daemon] daemon 
PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
...
# cat /etc/pmcd.conf 
# Name  Id   IPC     IPC Params File/Cmd 
irix    1    dso     irix_init  libirixpmda.so 
pmcd    2    dso     pmcd_init  pmda_pmcd.so 
proc    3    dso     proc_init  pmda_proc.so 
simple  253  pipe    binary     /var/pcp/pmdas/simple/pmdasimple -d 253

When either the cost or latency associated with collecting performance metrics is high, the PMDA implementer may choose to trade off the currency of the performance data to reduce the PMDA resource demands or the fetch latency time.

One scheme for doing this is called a caching PMDA, which periodically instantiates values for the performance metrics and responds to each request from pmcd with the most recently instantiated (or cached) values, as opposed to instantiating current values on demand when the PMCD asks for them.

The Cisco PMDA is an example of a caching PMDA; see the contents of the /var/pcp/pmdas/cisco directory and the pmdacisco(1) reference page.

Domains are autonomous performance areas, such as the operating system or a layered service or a particular application. Metrics are raw performance data for a domain, and typically quantify activity levels, resource utilization or quality of service. Instances are sets of related metrics, as for multiple processors, or multiple service classes, or multiple transaction types.

PCP employs the following simple and uniform data model to accommodate the demands of performance metrics drawn from multiple domains:

The selection of metrics and instances is an important design decision for a PMDA implementer. The metrics and instances for a target domain should have the following qualities:

For each metric, you should also consider these questions:

As with all programming tasks, expect to refine the choice of metrics and instances several times during the development of the PMDA.

Each PMDA must be uniquely identified by PMCD so that requests from clients can be efficiently routed to the appropriate PMDA. The unique identifier, the PMDA's domain, is encoded within the metrics and instance domain identifiers so that they are associated with the correct PMDA, and so that they are unique, regardless of the number of PMDAs that are connected to the pmcd process.

The default domain number for each PMDA is defined in /var/pcp/pmns/stdpmid. This file is a simple table of PMDA names and their corresponding domain number. However, a PMDA does not have to use this domain number—this file is only a guide to help avoid domain number clashes when PMDAs are installed and activated.

The domain number a PMDA uses is passed to the PMDA by pmcd when the PMDA is launched. Therefore, any data structures that require the PMDA's domain number must be set up when the PMDA is initialized, rather than declared statically. The protocol for PMDA initialization provides a standard way for a PMDA to implement this run-time initialization.

---

Tip: Although uniqueness of the domain number in the /etc/pmcd.conf control file used by pmcd(1) is all that is required for successful starting of pmcd and the associated PMDAs, the developer of a new PMDA is encouraged to add the default domain number for each new PMDA to the file /var/pcp/pmns/stdpmid; this file acts as a repository for documenting the known default domain numbers.

---

A PMDA provides support for a collection of metrics. In addition to the obvious performance metrics, and the measures of time, activity and resource utilization, the metrics should also describe how the target domain has been configured, as this can greatly affect the correct interpretation of the observed performance. For example, metrics that describe network transfer rates should also describe the number and type of network interfaces connected to the host.

The metrics should also describe how the PMDA has been configured. For example, if the PMDA was periodically probing a system to measure quality of service, there should be metrics for the delay between probes, the number of probes attempted, plus probe success and failure counters. It may also be appropriate to allow values to be stored (see the pmstore(1) reference page) into the delay metric, so that the delay used by the PMDA can be altered dynamically.

Data Structures

Each metric must be described in a pmDesc structure; see pmLookupDesc(3):

typedef struct { pmID pmid; /* unique identifier */ int type; /* base data type */ pmInDom indom; /* instance domain */ int sem; /* semantics of value */ pmUnits units; /* dimension and units */ } pmDesc;

This structure contains fields for

• a unique identifier (Performance Metric Identifier or PMID) that differentiates this metric from other metrics across the union of all PMDAs

• a data type indicator saying whether the format is an integer (32 or 64 bit, signed or unsigned); float; double; string; or arbitrary aggregate of binary data

• an instance domain identifier that links this metric to an instance domain

• an encoding of the value's semantics (counter, instantaneous, or discrete)

• a description of the value's units based on dimension and scale in the three orthogonal dimensions of space, time, and count (or events)

Symbolic constants of the form PM_TYPE_*, PM_SEM_*, PM_SPACE_*, PM_TIME_*, and PM_COUNT_*, defined in /usr/include/pcp/pmapi.h, may be used to initialize the elements of a pmDesc. The type pmID is an unsigned integer that can be safely cast to a _pmID_int structure, which contains fields defining the metric's (PMDA's) domain, cluster, and item number:

typedef struct { int pad:2; unsigned int domain:8; unsigned int cluster:12; unsigned int item:10; } _pmID_int;

The pad field should be ignored. The domain number should be set at run time when the PMDA is initialized. The PMDA_PMID macro defined in /usr/include/pcp/pmapi.h can be used to set the cluster and item fields at compile time, as these should always be known and fixed for a particular metric.

---

Note: The three components of the PMID should correspond exactly to the three-part definition of the PMID for the corresponding metric in the PMNS described in “Namespace”.

---

A table of pmdaMetric structures should be defined within the PMDA, with one structure per metric. This structure contains a pmDesc structure and a handle that allows PMDA-specific structures to be associated with each metric:

typedef struct { void *m_user; /* for users external use */ pmDesc m_desc; /* metric description */ } pmdaMetric;

For example, m_user could be a pointer to a global variable containing the metric value, or a pointer to a function that may be called to instantiate the metric's value.

static pmdaMetric metrictab[] = {
/* time */ 
 { (void *)0,  
   { PMDA_PMID(0,1), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT, 
     {0, 1, 0, 0, PM_TIME_SEC, 0} }, } 
}; 

Timestamps:         1   3   5   7   9  11 
Value:             10  30  60  80  90 N/A 

Timestamps:         1   3   5   7   9  11 
Semantics: 
Counter           N/A  10  15  10   5 N/A 
Instantaneous      10  30  60  80  90 N/A 
Discrete           10  30  60  80  90  90 

Singular metrics have only one value and no associated instance domain. Some metrics contain a set of values that share a common set of semantics for a specific instance, such as one value per processor, or one value per disk spindle, and so on.

---

Note: The PMDA implementation is solely responsible for choosing the instance identifiers that differentiate instances within the instance domain. The PMDA is also responsible for ensuring the uniqueness of instance identifiers in any instance domain.

---

   M[1]   M[2]   M[3] 
   M[4]   M[5]   M[6] 
or 
   M1[1]  M1[2]  M1[3] 
   M2[1]  M2[2]  M2[3] 

typedef struct { 
    int         i_inst;         /* internal instance identifier */ 
    char        *i_name;        /* external instance identifier */ 
} pmdaInstid; 

typedef struct { 
    pmInDom     it_indom;       /* indom, filled in */ 
    int         it_numinst;     /* number of instances */ 
    pmdaInstid  *it_set;        /* instance identifiers */ 
} pmdaIndom; 

typedef struct { 
        int             pad:2; 
        unsigned int    domain:8;   /* the administrative PMD */ 
        unsigned int    serial:22;  /* unique within PMD */ 
} _pmInDom_int; 

/* 
 * list of instances 
 */ 
static pmdaInstid color[] = {
    { 0, “red” }, { 1, “green” }, { 2, “blue” }
};
static pmdaInstid       *timenow = NULL;
static unsigned int     timesize = 0;
/*
 * list of instance domains
 */
static pmdaIndom indomtab[] = {
#define COLOR_INDOM     0
    { COLOR_INDOM, 3, color },
#define NOW_INDOM       1
    { NOW_INDOM, 0, NULL },
};
/*
 * all metrics supported in this PMDA - one table entry for each
 */
static pmdaMetric metrictab[] = {
/* numfetch */
    { NULL,
      { PMDA_PMID(0,0), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
        { 0,0,0,0,0,0} }, },
/* color */
    { NULL,
      { PMDA_PMID(0,1), PM_TYPE_32, COLOR_INDOM, PM_SEM_INSTANT,
        { 0,0,0,0,0,0} }, },
/* time.user */
    { NULL,
      { PMDA_PMID(1,2), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        { 0, 1, 0, 0, PM_TIME_SEC, 0 } }, },
/* time.sys */
    { NULL,
      { PMDA_PMID(1,3), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        { 0, 1, 0, 0, PM_TIME_SEC, 0 } }, },
/* now */
    { NULL,
      { PMDA_PMID(2,4), PM_TYPE_U32, NOW_INDOM, PM_SEM_INSTANT,
        { 0,0,0,0,0,0 } }, },
};

dp->version.two.fetch = simple_fetch;
dp->version.two.instance = simple_instance;

A suggested approach to writing a PMDA is to write a standalone program to extract the values from the target domain and then incorporate this program into the PMDA framework. This approach avoids concurrent debugging of two distinct problems: the extraction of the data and communication with PMCD.

These are some possible ways of exporting the data from the target domain:

Most of these approaches require some further data processing by the PMDA.

The PCP protocols expect PMDAs to return the current values for performance metrics when requested, and with short delay (low latency). For some target domains, access to the underlying instrumentation may be costly or involve unpredictable delays (for example, if the real performance data is stored on some remote host or network device). In these cases it may be necessary to separate probing for new performance data from servicing PMCD requests.

An architecture that has been used successfully for several PMDAs is to create one or more sproc child processes to obtain information while the main process communicates with pmcd; see sproc(2). At the simplest deployment of this arrangement, the two processes may execute without synchronization.

By contrast, a complex deployment would be one in which the refreshing of the metric values must be atomic, and this may require double buffering of the data structures. It also requires coordination between parent and child processes.

---

Tip: Since PMAPI is not thread-safe, only one PMDA process or thread of control should call any PMAPI routines, and this would typically be the thread servicing requests from the pmcd.

---

One caveat about this style of caching PMDA—it is generally better if the PMDA converts counts to rates based upon consecutive periodic sampling from the underlying instrumentation. By exporting pre-computed rate metrics with “instantaneous” semantics, the PMDA prevents the PCP monitor tools from computing their own rates upon consecutive pmcd fetches (which are likely to return identical values from a caching PMDA).

The pmns file defines the namespace of the PMDA. It is a simple text file that is used during installation to expand the namespace of the PMCD process. The format of this file is described by pmns(4).

Client processes will not be able to access the PMDA's metrics if the pmns file is not defined and installed with the pmnsadd command; see pmnsadd(1). The installed list of metric names and their corresponding PMIDs can be found in /var/pcp/pmns/root.

simple {
    numfetch    SIMPLE:0:0
    color       SIMPLE:0:1
    time
    now         SIMPLE:2:4
}
simple.time {
    user        SIMPLE:1:2
    sys         SIMPLE:1:3
}

simple { 
    numfetch    SIMPLE:0:0 
    color       SIMPLE:0:1 
    usertime    SIMPLE:1:2 
    systime     SIMPLE:1:3 
} 

For each metric defined within a PMDA, the PMDA developer is strongly encouraged to provide both terse and extended help text to describe the metric, and perhaps provide hints about the expected value ranges.

The help text is used to describe each metric in the visualization tools and pminfo with the -T option. The help text is specified in a specially formatted file, normally called help. This file is converted to the expected run-time format using the newhelp command; see newhelp(1). Converted help text files are usually placed in the PMDA's directory below /var/pcp/pmdas as part of the PMDA installation procedure.

@ SIMPLE.1 Instance domain “colour” for simple PMDA
Universally 3 instances, “red” (0), “green” (1) and “blue” (3).
@ SIMPLE.2 Dynamic instance domain “time” for simple PMDA
An instance domain is computed on-the-fly for exporting current time
information. Refer to the help text for simple.now for more details.
@ simple.numfetch Number of pmFetch operations.
The cumulative number of pmFetch operations directed to “simple” PMDA.
This counter may be modified with pmstore(1).
@ simple.color Metrics which increment with each fetch
This metric has 3 instances, designated “red”, “green” and “blue”.
The value of the metric is monotonic increasing in the range 0 to
255, then back to 0.  The different instances have different starting
values, namely 0 (red), 100 (green) and 200 (blue).
The metric values my be altered using pmstore(1).
@ simple.time.user Time agent has spent executing user code
The time in seconds that the CPU has spent executing agent user code.
@ simple.time.sys Time agent has spent executing system code
The time in seconds that the CPU has spent executing agent system code.
@ simple.now Time of day with a configurable instance domain
The value reflects the current time of day through a dynamically
reconfigurable instance domain.  On each metric value fetch request,
the agent checks to see whether the configuration file in
/var/pcp/pmdas/simple/simple.conf has been modified - if it has then
the file is re-parsed and the instance domain for this metric is again
constructed according to its contents.
This configuration file contains a single line of comma-separated time
tokens from this set:
  “sec”  (seconds after the minute),
  “min”  (minutes after the hour),
  “hour” (hour since midnight).
An example configuration file could be:  sec,min,hour
and in this case the simple.now metric would export values for the
three instances “sec”, “min” and “hour” corresponding respectively to
the components seconds, minutes and hours of the current time of day.
The instance domain reflects each token present in the file, and the
values reflect the time at which the PMDA processes the fetch.

Evolution of a PMDA, or more particularly the underlying instrumentation to which it provides access, over time naturally results in the appearance of new metrics and the disappearance of old metrics. This creates potential problems for PMAPI clients and PCP tools that may be required to interact with both new and former versions of the PMDA.

The following guidelines are intended to help reduce the complexity of implementing a PMDA in the face of evolutionary change, while maintaining predictability and semantic coherence for tools using the PMAPI, and for end users of those tools.

Both daemon and DSO PMDAs must handle multiple request types from pmcd. A daemon PMDA communicates with pmcd using the PDU protocol, while a DSO PMDA defines callbacks for each request type. In order to avoid duplicating this PDU processing (in the case of a PMDA that can be installed either as a daemon or as a DSO), and to allow a consistent framework, pmdaMain can be used by a daemon PMDA as a wrapper to handle the communication protocol using the same callbacks as a DSO PMDA.

To further simplify matters, default callbacks are declared in /usr/include/pcp/pmda.h:

Each callback takes a pmdaExt structure as its last argument. This structure contains all the information that is required by the default callbacks in most cases. The one exception is pmdaFetch, which needs an additional callback to instantiate the current value for each supported combination of a performance metric and an instance.

Therefore, for most PMDAs all the communication with pmcd is automatically handled by routines in libpcp.so and libpcp_pmda.so.

static int
trivial_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
    __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
    if (idp->cluster != 0 || idp->item != 0)
        return PM_ERR_PMID;
    else if (inst != PM_IN_NULL)
        return PM_ERR_INST;
    atom->l = time(NULL);
    return 0;
}

static int
simple_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
{
    numfetch++;
    simple_timenow_check();
    simple_timenow_refresh();
    return pmdaFetch(numpmid, pmidlist, resp, pmda);
}

static int
simple_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
    int                 i;
    static int          oldfetch = 0;
    static struct tms   tms;
    __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
    if (inst != PM_IN_NULL &&
        !(idp->cluster == 0 && idp->item == 1) &&
        !(idp->cluster == 2 && idp->item == 4))
        return PM_ERR_INST;
    if (idp->cluster == 0) {
        if (idp->item == 0) {                   /* simple.numfetch */
            atom->l = numfetch;
        }

        else if (idp->item == 1) {              /* simple.color */
            switch (inst) {
            case 0:                             /* red */
                red = (red + 1) % 256;
                atom->l = red;
                break;
            case 1:                             /* green */
                green = (green + 1) % 256;
                atom->l = green;
                break;
            case 2:                             /* blue */
                blue = (blue + 1) % 256;
                atom->l = blue;
                break;
            default:
                return PM_ERR_INST;
            }
        }
        else
            return PM_ERR_PMID;

    else if (idp->cluster == 1) {               /* simple.time */
        if (oldfetch < numfetch) {
            times(&tms);
            oldfetch = numfetch;
        }
        if (idp->item == 2)                     /* simple.time.user */
            atom->d = (tms.tms_utime / (double)CLK_TCK);
        else if (idp->item == 3)                /* simple.time.sys */
            atom->d = (tms.tms_stime / (double)CLK_TCK);
        else
            return PM_ERR_PMID;
     }

    else if (idp->cluster == 2) {
        if (idp->item == 4) {                   /* simple.now */
            if (inst < timesize) {
                /* this loop will always match one of the named */
                /* time constants from the timeslices structure */
                for (i = 0; i < num_timeslices; i++) {
                    if (strcmp(timenow[inst].i_name,
                               timeslices[i].tm_name) == 0) {
                        atom->l = timeslices[i].tm_field;
                        break;
                    }
                }
            }
            else
                return PM_ERR_INST;
        }
        else
            return PM_ERR_PMID;
    }

static int
simple_store(pmResult *result, pmdaExt *pmda)
{
    int         i, j, val, sts = 0;
    pmAtomValue av;
    pmValueSet  *vsp = NULL;
    __pmID_int  *pmidp = NULL;
    for (i = 0; i < result->numpmid; i++) {
        vsp = result->vset[i];
        pmidp = (__pmID_int *)&vsp->pmid;
        if (pmidp->cluster == 0) {  /* storable metrics are cluster0 */
            switch (pmidp->item) {
                case 0:                           /* simple.numfetch */
                    val = vsp->vlist[0].value.lval;
                    if (val < 0) {
                        sts = PM_ERR_SIGN;
                        val = 0;
                    }
                    numfetch = val;
                    break;
                case 1:                             /* simple.color */
                    for (j = 0; j < vsp->numval && sts == 0; j++) {
                        val = vsp->vlist[j].value.lval;
                        if (val < 0) {
                            sts = PM_ERR_SIGN;
                            val = 0;
                        } if (val > 255) {
                            sts = PM_ERR_CONV;
                            val = 255;
                        }

                        switch (vsp->vlist[j].inst) {
                            case 0:                         /* red */
                                red = val;
                                break;
                            case 1:                         /* green */
                                green = val;
                                break;
                            case 2:                         /* blue */
                                blue = val;
                                break;
                            default:
                                sts = PM_ERR_INST;
                        }

                default:
                    sts = PM_ERR_PMID;
                    break;
            }
        }

        else if ((pmidp->cluster == 1 &&
                 (pmidp->item == 2 || pmidp->item == 3)) ||
                 (pmidp->cluster == 2 && pmidp->item == 4)) {
            sts = -EACCES;
            break;
        }
        else {
            sts = PM_ERR_PMID;
            break;
        }
    }
    return sts;
}

PMDA structures used with the pcp_pmda library are defined in /usr/include/pcp/pmda.h.

typedef struct {
    int domain;     /* set/return performance metrics domain id here */
    struct {
        unsigned int pmda_interface : 8;  /* PMDA DSO version */
        unsigned int pmapi_version : 8;   /* PMAPI version */
        unsigned int flags : 16;          /* usage TBD */
    } comm;             /* set/return communication and version info */
    int status;         /* return initialization status here */
    union {
/* Interface Version 1 (PCP 1.0 & PCP 1.1) */
        struct {
            int     (*profile)(__pmProfile *);
            int     (*fetch)(int, pmID *, pmResult **);
            int     (*desc)(pmID, pmDesc *);
            int     (*instance)(pmInDom, int, char *, __pmInResult **);
            int     (*text)(int, int, char **);
            int     (*control)(pmResult *, int, int, int);
            int     (*store)(pmResult *);
        } one;
/* Interface Version 2 (PCP 2.0) */
        struct {
            pmdaExt *ext;
            int     (*profile)(__pmProfile *, pmdaExt *);
            int     (*fetch)(int, pmID *, pmResult **, pmdaExt *);
            int     (*desc)(pmID, pmDesc *, pmdaExt *);
            int     (*instance)(pmInDom, int, char *, __pmInResult **,
                    pmdaExt *);
            int     (*text)(int, int, char **, pmdaExt *);
            int     (*store)(pmResult *, pmdaExt *);
        } two;
    } version;

typedef struct {
    unsigned int e_flags;       /* usage TBD */
    void        *e_ext;         /* usage TBD */
    char        *e_sockname;    /* socket name to pmcd */
    char        *e_name;        /* name of this pmda */
    char        *e_logfile;     /* path to log file */
    char        *e_helptext;    /* path to help text */
    int         e_status;       /* =0 is OK */
    int         e_infd;         /* input file descriptor from pmcd */
    int         e_outfd;        /* output file descriptor to pmcd */
    int         e_port;         /* port to pmcd */
    int         e_singular;     /* =0 for singular values */
    int         e_ordinal;      /* >=0 for non-singular values */
    int         e_direct;       /* =1 if pmid map to meta table */
    int         e_domain;       /* metrics domain */
    int         e_nmetrics;     /* number of metrics */
    int         e_nindoms;      /* number of instance domains */
    int         e_help;         /* help text comes via this handle */
    __pmProfile *e_prof;        /* last received profile */
    pmdaIoType  e_io;           /* connection type to pmcd */
    pmdaIndom   *e_indoms;      /* instance domain table */
    pmdaIndom   *e_idp;         /* instance domain expansion */
    pmdaMetric  *e_metrics;     /* metric description table */
    pmdaResultCallBack e_resultCallBack; /* to clean up pmResult after fetch */
    pmdaFetchCallBack  e_fetchCallBack;  /* to assign metric values in fetch */
    pmdaCheckCallBack  e_checkCallBack;  /* callback on receipt of a PDU */
    pmdaDoneCallBack   e_doneCallBack;   /* callback after PDU is processed */
} pmdaExt;

The initialization process for a PMDA involves opening help text files, assigning callback function pointers, adjusting the metric and instance identifiers to the correct domains, and much more. The initialization of a daemon PMDA also differs significantly from a DSO PMDA, since the pmdaInterface structure is initialized by main or the PMCD process, respectively.

As described in the section “DSO PMDA”, an initialization function is provided by a DSO PMDA and called by PMCD. Using the standard PMDA wrappers, the same routine can also be used as part of the daemon PMDA initialization. This PMDA initialization function is responsible for

If the PMDA uses the common data structures defined for the pcp_pmda library, most of these requirements can be handled by the default pmdaInit routine; see pmdaInit(3).

Because the initialization routine is the only initialization opportunity for a DSO PMDA, the common initialization function should also perform any DSO-specific functions that are required. A default implementation of this functionality is provided by the pmdaDSO routine; see pmdaDSO(3).

void trivial_init(pmdaInterface *dp)
{
    pmdaSetFetchCallBack(dp, trivial_fetchCallBack);
    pmdaInit(dp, NULL, 0,
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}

static int      isDSO = 1;              /* =0 I am a daemon */
void simple_init(pmdaInterface *dp)
{
    if (isDSO)
        pmdaDSO(dp, PMDA_INTERFACE_2, “simple DSO”,
                “/var/pcp/pmdas/simple/help”);
    if (dp->status != 0)
        return;
    dp->version.two.fetch = simple_fetch;
    dp->version.two.store = simple_store;
    dp->version.two.instance = simple_instance;
    pmdaSetFetchCallBack(dp, simple_fetchCallBack);
    pmdaInit(dp, indomtab, sizeof(indomtab)/sizeof(indomtab[0]),
             metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}

In addition to the initialization routine that can be shared by a DSO and a daemon PMDA, a daemon PMDA must also

All these requirements can be handled by default initialization routines in the pcp_pmda library; see pmdaDaemon(3), pmdaGetOpt(3), pmdaOpenLog(3), pmdaConnect(3), and pmdaMain(3).

int
main(int argc, char **argv)
{
    int                 err = 0;
    int                 c = 0;
    pmdaInterface       dispatch;
    char                *p;
    /* trim cmd name of leading directory components */
    pmProgname = argv[0];
    for (p = pmProgname; *p; p++) {
        if (*p == `/')
            pmProgname = p+1;
    }
    isDSO = 0;
    pmdaDaemon(&dispatch, PMDA_INTERFACE_2, pmProgname, SIMPLE,
               “simple.log”, “/var/pcp/pmdas/simple/help”);
    if ((c = pmdaGetOpt(argc, argv, “D:d:i:l:pu:?”, &dispatch, &err)) != EOF)
        err++;
    if (err)
        usage();
    pmdaOpenLog(&dispatch);
    simple_init(&dispatch);
    simple_timenow_check();
    pmdaConnect(&dispatch);
    pmdaMain(&dispatch);
    exit(0);
    /*NOTREACHED*/
}

Thoroughly testing a PMDA with pmcd is difficult, although testing a daemon PMDA is marginally simpler than testing a DSO PMDA. If a DSO PMDA exits, pmcd also exits because they share a single address space and control thread. If the PMDA dumps core, dbx and related tools (see dbx(1)) cannot reasonably explore the generated core image, which includes the pmcd image and any other active DSO PMDAs.

The difficulty in using pmcd to test a daemon PMDA results from pmcd requiring timely replies from the PMDA in response to request PDUs. Although a “timeout” period can be set in /etc/config/pmcd.options, attaching dbx to the PMDA process (or any other long delay) might cause an already running pmcd to close its connection with the PMDA. If timeouts are disabled, pmcd could wait forever to connect with the PMDA.

If you suspect a PMDA has been terminated due to a time out failure, check the pmcd log file, usually /var/adm/pcplog/pmcd.log.

A more robust way of testing a pmcd is to use the dbpmda tool, which is similar to pmcd except that dbpmda provides complete control over the PDUs that are sent to the PMDA, and there are no time limits—it is essentially an interactive debugger for exercising a PMDA. See dbpmda(3) for details.

In addition, careful use of PCP debugging flags can produce useful information concerning a PMDA's behavior; see PMAPI(3) and pmdbg(1) for a discussion of the PCP debugging and tracing framework.

You can activate debugging flags in PMCD and most other PCP tools with the -D command-line option. Supported flags can be listed with the pmdbg command; see pmdbg(1). Setting the debug flag for pmcd in /etc/config/pmcd.options might generate too much information to be useful, especially if there are other clients and PMDAs connected to the pmcd process.

The pmcd debugging flag can also be changed dynamically by storing a new value into the metric pmcd.control.debug:

# pmstore pmcd.control.debug 5 

Most of the pcp_pmda library routines log additional information if the DBG_TRACE_LIBPMDA flag is set within the PMDA; see PMDA(3). The command-line argument -D is trapped by pmdaGetOpt to set the global debugging control variable pmDebug. Adding tests within the PMDA for the trace flags DBG_TRACE_APPL0, DBG_TRACE_APPL1, and DBG_TRACE_APPL2 permits different levels of information to be logged to the PMDA's log file.

All diagnostic, debugging, and tracing output from a PMDA should be written to standard error. By convention, all debugging information is enclosed by preprocessor #ifdef DEBUG statements so that they can be compiled out of the program at a later stage, if required.

    case 0: /* simple.numfetch */ 
        val = vsp->vlist[0].value.lval; 
        if (val < 0) { 
            sts = PM_ERR_SIGN; 
            val = 0; 
        } 
#ifdef DEBUG 
        if (pmDebug & DBG_TRACE_APPL0) { 
            fprintf(stderr,  
                 "simple: %d stored into numfetch", val); 
        } 
#endif 
        numfetch = val; 
        break; 

The dbpmda utility provides a simple interface to the PDU communication protocol. It allows daemon and DSO PMDAs to be tested with most request types, while the PMDA process may be monitored with dbx, par and other diagnostic tools. The reference page dbpmda(1) contains a sample session with the simple PMDA.

The pcp_trace library provides function calls for identifying sections of a program as transactions or events for examination by the trace PMDA, a user command called pmdatrace. The pcp_trace library is described in pmdatrace(3).

The monitoring of transactions using PCP infrastructure begins with a pmtracebegin call. Time is recorded from there to the corresponding pmtraceend call (with matching tag identifier). A transaction in progress can be cancelled by calling pmtraceabort.

A second form of program instrumentation is available with the pmtracepoint function. This is a simpler form of monitoring that exports only the number of times a particular point in a program is passed. The pmtraceobs function has similar semantics, but allows an arbitrary numeric value to be passed to the trace PMDA.

The pmdatrace command is a PMDA that exports transaction performance metrics from application processes using the pcp_trace library; see pmdatrace(1) for details.

For a complete introduction to performance tracing, refer to the Web-based PCP Tutorial, which contains the trace.html file covering this topic.

A PMDA is fully installed when these tasks are completed:

These tasks can be accomplished by a Makefile and an Install script as described below.

The Makefile should include an install target to compile and link the PMDA (as a DSO, or a daemon or both) in the PMDA directory, and in the case of a DSO PMDA, install the shared library in /var/pcp/lib. The clobber target should remove any files created as a by-product of the install target.

You may wish to use /var/pcp/pmdas/simple/Makefile as a template for constructing a new PMDA Makefile; changing the assignment of IAM from simple to newbie would account for most of the required changes.

Since the object format of a DSO PMDA must match the object format of pmcd, which in turn must match the object format of the booted IRIX kernel, there might be multiple DSO targets in the Makefile. For example, see targets mips_o32.pmda_$(IAM).so, mips_n32.pmda_$(IAM).so, and mips_64.pmda_$(IAM).so for the simple PMDA.

The Install script should make use of the generic procedures defined in the script /usr/pcp/lib/pmdaproc.sh, and may be as straightforward as the one used for the trivial PMDA, namely:

# Get the common procedures and variable assignments
#
. /usr/pcp/lib/pmdaproc.sh
# The name of the PMDA
#
iam=trivial
# Do it
#
_setup
_install
exit 0

The following variables may be assigned values to modify the behavior of the _setup and _install procedures from /usr/pcp/lib/pmdaproc.sh.

In addition, the variables do_pmda and do_check will be set to reflect the intention to install the PMDA (as opposed to install just the PMNS) and to check the availability of the metrics once the PMDA is installed. By default each variable is true; however, the command-line options -N and -Q to Install may be used to set the variables to false, as follows: do_pmda (-N) and do_check (-N or -Q).

The variables may also have their assignments changed by the user's response to the common prompt:

You will need to choose an appropriate configuration for installation of the ... Performance Metrics Domain Agent (PMDA).
  collector   collect performance statistics on this system
  monitor     allow this system to monitor local and/or remote systems
  both        collector and monitor configuration for this system

Obviously, for anything but the most trivial PMDA, after calling the _setup procedure, the Install script should also prompt for any PMDA-specific parameters, which are typically accumulated in the args variable and used by the _install procedure.

The detailed operation of the _install procedure involves the following tasks:

There are some PMDA changes that may trick PMCD into thinking nothing has changed, and not restarting the PMDA. Most notable are changes to the PMDA executable. In these cases, you may need to explicitly remove the PMDA (see below), or more drastically, restart pmcd as follows:

# /etc/init.d/pcp start 

When invoked with a -N command-line option, the PMDA Install script may be used to update the PMNS without installing the PMDA. This is typically used on a monitoring system to populate the local PMNS with the names of the performance metrics from a PMDA installed on a remote host running the older PCP 1.x protocols. The -N option also installs pmchart views useful on a monitoring system.

The simplest way to stop a PMDA from running, apart from killing the process, is to remove the entry from /etc/pmcd.conf and signal PMCD to re-read its configuration file. To completely remove a PMDA requires the reverse process of the installation, including an update of the Performance Metrics Name Space (PMNS).

This typically involves a Remove script in the PMDA directory that uses the same common procedures as the Install script described above.

Most PCP tools have their own configuration file format for specifying which metrics to view or to log. By providing “canned” configuration files that monitor key metrics of the new PMDA, users can quickly see the performance of the target system, as characterized by key metrics in the new PMDA.

Any configuration files that are created should be kept with the PMDA and installed into the appropriate directories when the PMDA is installed.

The pmchart command comes with several views for the default PMDAs located at /var/pcp/config/pmchart; see pmchart(1). These views can be used as a basis for defining views relevant to the new PMDA.

Likewise, there are several shell scripts that employ pmview (see pmview(1)) for 3-dimensional visualizations, including dkvis and mpvis; see dkvis(1) and mpvis(1). Only small sections of these scripts require modification to visualize a different set of metrics. Similar scripted front ends could be created to customize pmgadgets icon control panels for a new PMDA; refer to pmgirix(1).

As with all PCP customization, some of the most valuable tools can be created by defining views, scenes, and control-panel layouts that combine related performance metrics from multiple PMDAs or multiple hosts.

Templates for parameterized alarm configurations can be specified using the pmrules command; see pmrules(1), and pmie(1). Rules involving metrics from the new PMDA may be created directly.

Daily logs can be specified in pmlogger configuration files, or with the cron.pmdaily mechanism; see pmlogger(1) and cron.pmdaily(1). The services of cron.pmsnap may be used to incorporate the new performance metrics into charts that may be periodically regenerated and published via a World Wide Web server.