Chapter 11. Thread-Level Parallelism

IRIX supports IEEE standard 1003.1c-1995, “System Application Program Interface—Amendment 2: Threads Extension”; that is, it supports POSIX threads, or pthreads. Pthreads are supported by IRIX 6.2 after the following patches are applied: 1361, 1367, and 1429.

In addition, the Silicon Graphics implementation of the Ada 95 language includes support for multitasking Ada programs. The current implementation of Ada uses an early version of the pthreads library. The next release of Ada will use the POSIX library. For a complete discussion of the Ada 95 task facility, refer to the Ada 95 Reference Manual, which installs with the Ada 95 compiler (GNAT) product.

This chapter contains the following main topics:

Overview of POSIX Threads

A thread is an independent execution state; that is, a set of machine registers, a call stack, and the ability to execute code. When IRIX creates a process, it also creates one thread to execute that process. However, you can write a program that creates many more threads to execute in the same address space. For a comparison of pthreads to processes, see “Thread-Level Parallelism” in Chapter 8.

POSIX threads are similar in some ways to IRIX lightweight processes made with sproc(). You use pthreads in preference to lightweight processes for two main reasons: portability and performance. A program based on pthreads is normally easier to port from another vendor's equipment than a program that depends on a unique facility such as sproc(). Table 11-1 summarizes some of the differences between pthreads and lightweight processes.

Table 11-1. Comparison of Pthreads and Processes

Attribute

POSIX Threads

Lightweight Processes

UNIX Processes

Source portability

Standard interface, portable between vendors

sproc() is unique to IRIX

fork() is a UNIX standard

Creation overhead

Relatively small

Moderately large

Quite large

Block/Unblock (Dispatch) Overhead

Few microseconds

Many microseconds

Many microseconds

Address space

Shared

Shared, or copy on write, or separate

Separate

Memory-mapped files and arenas

Shared

Shared, or copy on write, or separate

Explicit sharing only

Mutual exclusion objects

Mutexes and condition variables; POSIX semaphores; message queues

IRIX semaphores and locks; POSIX semaphores; message queues

IRIX semaphores and locks; POSIX semaphores; message queues

Files, pipes, and I/O streams

Shared single-process file table

Shared or separate file table

Separate file table

Signal masks and signal handlers

Each thread has a mask but handlers are shared

Each process has a mask and its own handlers

Each process has a mask and its own handlers

Resource limits

Single-process limits

Single-process limits

Limits apply to each process separately

Process ID

One PID applies to all threads

PID per process plus share-group PID

PID per process

Effective user and group IDs

Inherited and unchangeable

Inherited, can be changed

Inherited, can be changed

It takes relatively little time to create or destroy a pthread, as compared to creating a lightweight process. On the other hand, threads share all resources and attributes of a single process (except for the signal mask, see “Pthreads and Signals”). If you want each executing entity to have its own set of file descriptors, or if you want to make sure that one entity cannot modify data shared with another entity, you must use lightweight processes or normal processes.

Compiling and Debugging a Pthread Application

A pthread application is a C program that uses some of the POSIX pthreads functions. In order to use these functions, and in order to access the thread-safe versions of the standard I/O macros, you must include the proper header files and link with the pthreads library. You can debug and analyze the compiled program using some of the tools available for IRIX.

Compiling Pthread Source

The header files related to pthreads functions are summarized in Table 11-2.

Table 11-2. Header Files Related to Pthreads

Header

Primary Contents

errno.h

System error codes returned by pthreads functions.

pthread.h

Pthread functions and special pthread data types.

sched.h

The sched_param structure and related functions used in setting thread priorities.

stdio.h

Standard stream I/O macros, including thread-safe versions

sys/types.h

IRIX and standard data types.

limits.h

Some POSIX constants such as _POSIX_THREAD_THREADS_MAX

unistd.h

Constants used when calling sysconf() to query POSIX limits (see the sysconf(3) reference page).

Prior to the inclusion of stdio.h, be sure that the compiler variables _POSIX1C and _NO_ANSIMODE are defined. These variables are set by default in most compiles. Read the header file /usr/include/standards.h (which is included by stdio.h) to see the logic of standard namespace definition.

You can use pthreads with a program compiled to any of the supported execution models: -32 for compatibility with older systems, -n32 for 64-bit data and 32-bit addressing, or -64 for 64-bit addressing.

The pthreads functions are defined in the library libpthread.so. Link with this library using the -lpthread compiler option, which should be the last library on the command line. The compiler chooses the correct library based on the execution model: /usr/lib/libpthread.so, /usr/lib32/libpthread.so, and /usr/lib64/libpthread.so. (However, you must be sure that the needed version of the library is installed; the -n32 and -64 libraries do not install by default.)


Note: The definition of a threaded program or pthread program is: a program that links with libpthread. Do not link with libpthread unless you intend to use the pthread interface, since libpthread replaces many standard library functions.



Tip: Many names in libpthread override names defined in libc. The linker displays warning messages about these overrides. You can silence the warnings with the -Wl,-woff,85 compile option.


Debugging Pthread Programs

The debugging and performance tuning tools distributed with IRIX and the IRIX developer's option can sometimes be used with a threaded program.

Debugging With dbx

The dbx debugger is distributed with the IRIX Developer's Option. Version 7.0 of dbx is required to work properly with pthreads.

When debugging a pthreads program, you must set the following dbx variables:

  • Set $promptonfork to 2.

  • Set $mp_program to 1.

When you set a breakpoint with dbx, it is global to all threads. The first thread to reach the breakpoint trips the breakpoint. This stops execution of the entire process (all threads). If you set the breakpoint in code used by more than one thread, the program could be in a different thread each time it stops. The thread ID is displayed at the stop, as in the display in Example 11-1.

Example 11-1. Debugger Display of Pthread Program


(dbx) showthread all
Thread: Start:               State:     Pid:  Location:
0x10000                      COND-WAIT       _SGIPT_sched_block ["xp.c":966]
0x10001 work_thread          RUNNING     1512 FLOCAL_ALIGN ["workfn.c":864]
0x10002 work_thread          RUNNING     1520 FLOCAL_ALIGN ["workfn.c":850]
0x10003 work_thread          RUNNING     1563 FLOCAL_ALIGN ["workfn.c":866]
0x10004>work_thread          RUNNING     1425 thr_tst ["workfn.c":391]

You can single-step a threaded program as long as you know that only one thread is executing the code through which you are stepping. When you single-step through code that is executed by more than one thread, confusing results can occur. To single-step, dbx sets a breakpoint where the program should stop next. However, breakpoints are global. When you give the next command in one thread, the stop can occur in a different thread.

Debugging With the Workshop Debugger

The Workshop Debugger is part of the Developer Magic package. In version 2.6.2 of this package, the debugger is aware of pthreads. The command line view in the debugger main window can be used to set breakpoints and to produce a display similar to the one in Example 11-1.

Breakpoints set with the Workshop Debugger are global to the program and are taken by the next thread to reach them, as with dbx.

The performance measurement tools of the Developer Magic package do not produce reliable results with a threaded program.

Creating Pthreads

You create a pthread by calling pthread_create(). One argument to this function is a thread attribute object of type pthread_attr_t. You pass a null address to request a thread having default attributes, or you prepare an attribute object to reflect the features you want the thread to have. You can use one attribute object to create many pthreads.

Functions related to attribute objects and pthread creation are summarized in Table 11-3 and described in the following text.

Table 11-3. Functions for Creating Pthreads

Function

Purpose

pthread_attr_init(3P)

Initialize a pthread_attr_t object to default settings.

pthread_attr_setdetachstate(3P)

Set the automatic-detach attribute in a pthread_attr_t object.

pthread_attr_setinheritsched(3P)

Specify whether scheduling attributes come from the attribute object or are inherited from the creating thread.

pthread_attr_setschedparam(3P)

Set the starting thread priority in a pthread_attr_t object.

pthread_attr_setschedpolicy(3P)

Set the scheduling policy in a pthread_attr_t object.

pthread_attr_setstacksize(3P)

Set the stack size attribute in a pthread_attr_t object.

pthread_attr_setstackaddr(3P)

Set the address of memory to use as a stack in a pthread_attr_t object (when you allocate the stack for the new thread).

pthread_attr_destroy(3P)

Uninitialize a pthread_attr_t object.

pthread_create(3P)

Create a new thread based on an attribute object, or with default attributes.


Initial Detach State

After a thread has terminated, it can be “detached.” Detaching means that the pthreads library deletes its information about the thread, possibly releasing some memory (see “Joining and Detaching”). There are three ways to detach a thread:

  • automatically when the thread terminates

  • explicitly by calling pthread_join()

  • explicitly by calling pthread_detach()

You can use pthread_attr_setdetachstate() to specify that a thread should be detached automatically when it terminates. Do this when you know that the thread will not be detached by an explicit function call.

Initial Scheduling Priority and Policy

Scheduling priorities and policies are described under “Scheduling Pthreads”. You can specify an initial scheduling policy by calling pthread_attr_setschedpolicy(), passing one of the policy constants SCHED_FIFO, SCHED_RR, or SCHED_OTHER.

You can specify an initial thread priority in a struct sched_param object in memory (the structure is declared in sched.h). Set the desired priority in the sched_priority field. Pass the structure to pthread_attr_setschedparam().

The pthread_attr_setinheritsched() function is used to specify, in the attribute object, whether a new thread's scheduling policy and priority should be taken from the attribute object, or whether these things should be inherited from the thread that creates the new thread. When you set an attribute object for inheritance, the scheduling policy and priority in the attribute object are ignored.

Thread Stack Allocation

Each pthread has an execution stack area in memory. By default, pthread_create() allocates stack space of the specified size from dynamic memory. When it does so, the stack space is automatically released when the thread is detached.

You use pthread_attr_setstacksize() to specify the size of this stack area. You cannot specify a stack size less than a minimum. You can learn the minimum by calling sysconf() with _SC_THREAD_STACK_MIN (see the sysconf(3C) reference page).

Preallocating Stack Areas

You can instead preallocate stack space from any source of dynamic memory such as malloc(). When you preallocate stack space, you must do the following:

  • Specify the address of the space using pthread_attr_setstackaddr().

    This tells pthread_create() not to allocate space.

  • Specify the size of the allocated space using pthread_attr_setstacksize().

    This enables pthread_create() to initialize the correct starting stack address.

  • Free the stack space when the thread terminates (see “Joining and Detaching”).

There is normally no protection against a thread overrunning the space. If a thread allocates too much automatic data or makes too many nested function calls, it will attempt to modify memory outside the stack space. This might cause a segmentation fault if that memory is not allocated, or it might modify memory used for other purposes.


Tip: When you preallocate stack space, you can create “red zones” around the allocated stacks as follows:


  • Allocate the stack memory in multiples of the system page size aligned on page boundaries (see the getpagesize(2) and memalign(3C) reference pages).

  • Allocate an extra page of memory above and below each stack area.

  • Use the mprotect() function to set the protection of the extra pages to PROT_NONE (see the mprotect(2) reference page).

This procedure creates untouchable pages at each end of the stack area. If the thread misuses its stack, it will usually terminate at once with a segmentation fault. (It is still possible for a thread to call a function that allocates more than a page of automatic variables, and so skips over the “red zone” to modify memory beyond it.)

Caveats Regarding Stack Space

Because thread stack space is taken from dynamic memory, the allocation is charged against the process virtual memory limit, not the process stack size limit as you might expect (see the getrlimit(2) reference page for information on resource limits).

The stack segment of a process is extended automatically up to a (large) system limit as necessary. The stack segment of a pthread is fixed in size. The “first” pthread in a threaded program is no different from any other pthread in this regard. Every pthread has a fixed-size stack. The first pthread has a stack of default size. If your “first” or “main” pthread needs more than the default stack size, the actual first-started pthread must set the desired stack size and create a thread to be “main,” and then terminate.

Executing and Terminating Pthreads

The functions you use to manage the progress of a thread are summarized in Table 11-4 and described in the following topics.

Table 11-4. Functions for Managing Thread Execution

Function

Purpose

pthread_atfork(3P)

Register functions to handle the event of a fork().

pthread_cancel(3P)

Request cancellation of a specified thread.

pthread_cleanup_push(3P)

Register function to handle the event of thread termination.

pthread_cleanup_pop(3P)

Unregister and optionally call termination handler.

pthread_detach(3P)

Detach a terminated thread.

pthread_exit(3P)

Explicitly terminate the calling thread.

pthread_join(3P)

Wait for a thread to terminate and receive its return value.

pthread_once(3P)

Execute initialization function once only.

pthread_self(3P)

Return the calling thread's ID.

pthread_equal(3P)

Compare two thread IDs for equality.

pthread_setcancelstate(3P)

Permit or block cancellation of the calling thread.

pthread_setcanceltype(3P)

Specify deferred or asynchronous cancellation.

pthread_testcancel(3P)

Permit cancellation to take place, if it is pending.


Getting the Thread ID

Call pthread_self() to get the thread ID of the calling thread. A thread can use this thread ID when changing its own scheduling priority, for example (see “Scheduling Pthreads”).

Initializing Static Data

Your program may use static data that should be initialized, but only once. The code can be entered by multiple threads, and might be entered concurrently. How can you ensure that only one thread will perform the initialization?

The answer is to create a variable of type pthread_once_t, statically initialized to the value PTHREAD_ONCE_INIT. In the module code, call pthread_once() passing the addresses of the variable and of an initialization function. The pthreads library ensures that the initialization function is called only once, and that any other threads calling pthread_once() for this variable wait until the first thread completes the call. An example is shown in Example 11-2.

Example 11-2. One-Time Initialization


pthread_once_t first_time_flag = PTHREAD_ONCE_INIT;
elaborate_struct_t uninitialized; /* thing to initialize */
void elaborate_initializer(void); /* function to do it */
int subroutine(...)
{
   ...
   pthread_once(&first_time_flag, elaborate_initializer);
   ...
}

Setting Event Handlers

A thread can establish functions that are called when threads terminate and when the process forks.

Call pthread_cleanup_push() to register a function that is to be called in the event that the current thread terminates, either by exiting or by cancellation. Call pthread_cleanup_pop() to retract this registration and, optionally, to call the handler. These functions are often used in library code, with the push operation done on entry to the library and the pop done upon exit from the library. The push and pop operations are in fact implemented partly as macro code. For this reason, calls to them must be strictly balanced—a pop for each push—and each push/pop pair must appear in a single C lexical scope. A nonstructured jump such as a longjmp (see the setjmp(3) reference page) or goto can cause unexpected results.

Call pthread_atfork() to register three handlers related to a UNIX fork() call. The first handler executes just before the fork() takes place; the second executes just after the fork() in the parent process; the third executes just after the fork() in the child process.

The fork() operation creates a new process with a copy of the calling process's address space, including any locked mutexes or semaphores. Typically, the new process immediately calls exec() to replace the address space with a new program. When this is the case, there is no need for pthread_atfork() (see the exec(2) and fork(2) reference pages). However, if the new process continues to execute with the inherited address space, including perhaps calls to library code that uses pthreads, it may be necessary for the library code to reinitialize data in the address space of the child process. You can do this in the fork event handlers.

Terminating and Being Terminated

A thread begins execution in the function that is named in the pthread_create() call. When it returns from that function, the thread terminates. A thread can terminate earlier by calling pthread_exit(). In either case, the thread returns a value of type void*.

One thread can request early termination of another by calling pthread_cancel(), passing the thread ID of the target thread. A thread can protect itself against cancellation using two built-in status switches:

  • The pthread_setcancelstate() function lets you prevent cancellation entirely (PTHREAD_CANCEL_DISABLE) or permit cancellation (PTHREAD_CANCEL_ENABLE).

  • The pthread_setcanceltype() function lets you decide when cancellation will take place, if it is allowed at all. Cancellation can happen whenever it is requested (PTHREAD_CANCEL_ASYNCHRONOUS) or only at defined points (PTHREAD_CANCEL_DEFERRED).

When you prevent cancellation by setting PTHREAD_CANCEL_DISABLE, a cancellation request is blocked but remains pending until the thread terminates or changes its cancellation state.

The initial state of a thread is PTHREAD_CANCEL_ENABLE and PTHREAD_CANCEL_DEFERRED. In this state, a cancellation request is blocked until the thread calls a function that is a defined cancellation point. The functions that are cancellation points are listed in the pthread_setcanceltype(3P) reference page. A thread can explicitly permit cancellation by calling pthread_testcancel().

Joining and Detaching

Sometimes you do not care when threads terminate—your program starts a set of threads, and they continue until the entire program terminates.

In other cases, threads are created and terminated as the program runs. One thread can find out when another has terminated by calling pthread_join(), specifying the thread ID. The function does not return until the specified thread terminates. The value the specified thread passed to pthread_exit() is returned. At this time, your program can release any resources that you associate with the thread, for example, stack space (see “Thread Stack Allocation”).

The pthread_join() function detaches the terminated thread. If your program does not use pthread_join(), and does continue execution after threads have terminated, you must arrange for terminated threads to be detached in some other way. One way is by specifying automatic detachment when the threads are created (see “Initial Detach State”). Another is to call pthread_detach() at any time after creating the thread, including after it has terminated.

If your program continues for a long time creating threads and letting them terminate, but does not arrange for detaching the completed threads, eventually an error will occur because resources have been used up.

Using Thread-Unique Data

In some designs, especially modules of library code, you need to store data that is both

  • unique to the calling thread

  • persistent from one function call to another

Normally, the only data that is unique to a thread is the contents of its local variables on the stack, and these do not persist between calls. However, the pthreads library provides a way to create persistent, thread-unique data. The functions for this are summarized in Table 11-5.

Table 11-5. Functions for Thread-Unique Data

Function

Purpose

pthread_key_create(3P)

Create a key (class of thread data).

pthread_key_delete(3P)

Delete a key.

pthread_getspecific(3P)

Retrieve this thread's value for a key.

pthread_setspecific(3P)

Set this thread's value for a key.

Your program calls pthread_key_create() to define a new storage key. A storage key represents one kind or class of data. Each thread has a unique instance of this class of data, with an initial value of NULL. The returned key value (of type pthread_key_t) is used by all threads to store and retrieve data of this class.

Any thread can use pthread_getspecific() to retrieve that thread's unique instance of the value stored under this key. A thread can fetch only its own value, which is the value stored by this same thread using pthread_setspecific(). Any thread's stored value is NULL until it stores a new value.

When you create a key, you can specify a destructor function that is called automatically when a thread terminates. The destructor is called as long as the key is still valid and the key value for the terminating thread is not NULL. The destructor receives the thread's value for the key as its argument.

You create keys by calling pthread_key_create(). Keys can be created before any threads are created. However, when you are designing a library module for use from any threaded program, you need to create a key upon first entry to your library code. This is an ideal application for a pthread_once_t variable (see “Initializing Static Data”). The code in Example 11-3 suggests how a threaded module would create a key if necessary, and initialize its contents for the current thread.

Example 11-3. Initializing Thread-Unique Data


typedef struct perThread_s {
   ...items of data unique to thread...
} perThread_t;
pthread_key_t perThreadKey; /* key used to find per-thread info */
pthread_once_t makePerThreadKey = PTHREAD_ONCE_INIT;
/*
|| Destructor function, called when any thread exits with a
|| non-NULL value of perThreadKey.
*/
void deletePerThread(void *arg)
{
   free(arg);
}
/*
|| One-time initializing function, called through pthread_once,
|| to create the perThreadKey.
*/
void createPerThreadKey(void)
{
   pthread_key_create(&perThreadKey,deletePerThread);
}
/*
|| Return the address of this thread's instance of perThread_t,
|| Create the struct if necessary. Create the key if necessary.
*/
struct perThreadInfo *getPerThread(void); 
{
   perThread_t *ppt;
   int ret;
   pthread_once(&makePerThreadKey, createPerThreadKey);
   ppt = pthread_getspecific(perThreadKey);
   if (NULL==ppt)
   {
      ppt = (perThread_t*)malloc(sizeof(perThread_t));
      ...initialize fields of ppt->new per-thread struct...
      ret = pthread_setspecific(perThreadKey,(void*)ppt);
      if (ret) perror("pthread_setspecific()");
   }
   return ppt;
}

The code in Example 11-3 includes the following functions and global variables:

perThreadKey

The key that represents the class of perThread_t structures.

makePerThreadKey

A pthread_once_t variable used to ensure that pthread_key_create() is called only once.

deletePerThread()

Destructor function, passed to pthread_key_create(), called when any thread terminates leaving a non-NULL value under the perThreadKey key.

createPerThreadKey()

Function called via pthread_once() to create perThreadKey.

getPerThread()

Function that can be called from any thread to retrieve that thread's value of perThreadKey. If the key itself has not been defined, the function defines it (calling createPerThreadKey() by way of pthread_once()). If this thread's value of the key is NULL, the function creates and initializes a value, and stores it using pthread_setspecific().


Pthreads and Signals

Signals are an integral part of UNIX programming. For a general overview of signal concepts and numbers, see “Signals” in Chapter 5 and the signal(5) reference page. IRIX supports three different, partly-compatible, signal facilities: BSD signals, SVR4 signals, and POSIX signals. When you are writing a pthreads program, you must be sure to use only the POSIX signal facilities (see “POSIX Signal Facility” in Chapter 5). Do not mix use of other signal functions in a pthreads program or unpredictable results can follow.

Setting Signal Masks

A thread specifies which signals it is willing to receive (see “Signal Blocking and Signal Masks” in Chapter 5). In a program that is linked with the pthreads library this must be done using pthread_sigmask(). Each thread inherits the signal mask of the thread that calls pthread_create(). Typically you set an initial mask in the first thread, so that it can be inherited by all other threads.

When a signal is directed to a specific thread that blocks the signal, the signal remains pending until the thread unblocks the signal. When a signal is directed to the process, it is delivered to the first thread that is not blocking that signal. If all threads block that signal, the signal remains pending until some thread unblocks the signal or the process ends.

While the process runs, a thread can find out which signals are pending by calling sigpending(). This function returns a mask showing the combination of signals pending for the process as a whole and for the calling thread; that is, the signals that could be delivered to the calling thread if the signals were not blocked.

Setting Signal Actions

When a signal is not blocked and is delivered, some action is taken. You specify what that action should be using the sigaction() function. Specify an action for each signal number separately. These actions are set on a process-wide basis, not individually for each thread. Each thread has a private signal mask, but signal actions are specified for all threads in the process. Choose among the following actions for each signal:

SIG_DFL

Default handling, which depends on the specific signal but is either to ignore the signal or to terminate the process, with or without a dump.

SIG_IGN

Ignore the signal, that is, discard it when it is generated. Certain signals cannot be ignored.

(function address)

Signal is delivered by an asynchronous call to the specified function.

When a signal is delivered to a function, you have the option of specifying a function that receives a siginfo_t structure with information about the signal. These and other options are spelled out under “Signal Handling Policies” in Chapter 5.

Receiving Signals Synchronously

You can design a program to receive signals in a synchronous manner instead of asynchronously. To do this, set a mask that blocks all the signals that are to be received synchronously. Then call one of the following three functions:

sigwait(3)

Suspend until one of a specified set of signals is generated, then return the signal number.

sigwaitinfo(3)

Like sigwait(), but returns additional information about the signal.

sigtimedwait(3)

Like sigwaitinfo(), but also returns after a specified time has elapsed if no signal is received.

Using these functions you can write a thread that treats arriving signals as a stream of events to be processed. This is generally the safest program model, much easier to work with than the asynchronous model of signal delivery.

Scheduling Pthreads

By default, the pthreads library schedules the threads of a process in a round-robin fashion. Much of the scheduling machinery is done in the library, within the context of the user process, without assistance from the IRIX kernel. On a multiprocessor, threads can run concurrently.

The scheduling algorithm is controlled by two parameters: a policy and a priority for each thread. These variables are set initially when the thread is created (see “Initial Scheduling Priority and Policy”), and can be modified while the thread is running. The functions used in scheduling are summarized in Table 11-6.

Table 11-6. Functions for Schedule Management

Function

Purpose

pthread_getschedparam(3P)

Get a thread's policy and priority.

pthread_setschedparam(3P)

Set a thread's policy and priority.

sched_get_priority_max(3C)

Return the maximum priority value.

sched_get_priority_min(3C)

Return the minimum priority value.

sched_yield(2)

Relinquish the processor.


Scheduling Policy

There are two scheduling policies in this implementation: first-in-first-out (SCHED_FIFO) and round-robin (SCHED_RR). (The default SCHED_OTHER behaves the same as SCHED_RR.) SCHED_FIFO and SCHED_RR are similar. The round-robin scheduler ensures that when a thread has used a certain maximum amount of time without blocking, it is moved to the end of the queue of threads of the same priority, and can be preempted by other threads.

The details of scheduling are discussed in the pthread_attr_setschedpolicy(3) reference page.

Scheduling Priority

The queues of runnable threads are ordered by thread priority numbers, with a small number representing a low priority, and a larger number representing a higher priority. Threads with higher priorities are chosen to execute before threads with lower priorities.

The sched_get_priority_max() and sched_get_priority_min() functions return the highest and lowest priority numbers. There are at least 32 priority values and the lowest is greater than or equal to 0. You can use these functions to set up a system of relative priorities as suggested by the code in Example 11-4.

Example 11-4. Establishing Relative Priority Levels


#include <sched.h>
int higherP, mediumP, lowerP;
void setRelativePriorities()
{
   int maxP, minP;
   maxP = sched_get_priority_max();
   minP = sched_get_priority_min();
   mediumP = minP + ((maxP-minP)/2);
   higherP = mediumP+1;
   lowerP = mediumP-1;
}

When all threads use one of the three priorities higherP, mediumP, or lowerP, threads that run at higherP will always run in preference to threads at the other two priorities.


Note: There are system functions named sched_get_priority_max() and sched_get_priority_min(); they are documented in “Controlling Scheduling With POSIX Functions” in Chapter 10 and the sched_get_priority_max(2) reference page. However, when you link with libpthread, these names are defined in the pthread library and access the pthread priority values.

A thread can set another's priority or scheduling policy, or both, using pthread_setschedparam(). A simple function to set a specified priority on the current thread, returning the previous value, is shown in Example 11-5.

Example 11-5. Function to Set Own Priority


#include <sched.h> /* struct sched_param */
int setMyPriority(int newP)
{
   pthread_t myTid = pthread_self();
   int ret, oldP, policy;
   struct sched_param sp;
   (void) pthread_getschedparam(myTID,&policy,&sp);
   oldP = sp.sched_priority;
   sp.sched_priority = newP;
   ret = pthread_setschedparam(myTID,policy,&sp);
   if (ret)
   { perror("pthread_setschedparam()"); }
   return oldP;
}

Synchronizing Pthreads

Asynchronous threads using a common address space must cooperate and coordinate their use of shared variables. Independent processes coordinate using the mechanisms described in previous chapters: IRIX semaphores and locks and SVR4 semaphores. Threads cannot use these IPC mechanisms. Threads can coordinate using these mechanisms:

You cannot use IRIX semaphores, locks, and barriers to coordinate between multiple threads within a single program. Nor can you use SVR4 semaphores for this purpose.

Mutexes

A mutex is a software object that stands for the right to modify some shared variable, or the right to execute a critical section of code. A mutex can be owned by only one thread at a time; other threads trying to acquire it wait.

Preparing Mutex Objects

When a thread wants to modify a variable that it shares with other threads, or execute a critical section, the thread claims the associated mutex. This can cause the thread to wait until it can acquire the mutex. When the thread has finished using the shared variable or critical code, it releases the mutex. If two or more threads claim the mutex at once, one acquires the mutex and continues, while the others are blocked until the mutex is released.

A mutex has attributes that control its behavior. The pthreads library contains several functions used to prepare a mutex for use. These functions are summarized in Table 11-7.

Table 11-7. Functions for Preparing Mutex Objects

Function

Purpose

pthread_mutex_init(3P)

Initialize a mutex object based on a pthread_mutexattr_t.

pthread_mutex_destroy(3P)

Uninitialize a mutex object.

pthread_mutexattr_init(3P)

Initialize a pthread_mutexattr_t with default attributes.

pthread_mutexattr_destroy(3P)

Uninitialize a pthread_mutexattr_t.

pthread_mutexattr_getprotocol(3P)

Query the priority protocol in a pthread_mutexattr_t.

pthread_mutexattr_setprotocol(3P)

Set the priority protocol choice in a pthread_mutexattr_t.

pthread_mutexattr_getprioceiling(3P)

Query the minimum priority in a pthread_mutexattr_t.

pthread_mutexattr_setprioceiling(3P)

Set the minimum priority in a pthread_mutexattr_t.

A mutex must be initialized before use. You can do this in one of three ways:

  • Static assignment of the constant PTHREAD_MUTEX_INITIALIZER.

  • Calling pthread_mutex_init() passing NULL instead of the address of a mutex attribute object.

  • Calling pthread_mutex_init() passing a pthread_mutexattr_t object that you have set up with attribute values.

The first two methods initialize the mutex to default attributes. Dynamic initialization should be done only once (see “Initializing Static Data”).

Two attributes can be set in a pthread_mutexattr_t. The priority inheritance protocol is the more important. You can set the priority inheritance protocol using pthread_mutexattr_setprotocol() to one of three values:

PTHREAD_PRIO_NONE

The mutex has no effect on the thread that acquires it.

PTHREAD_PRIO_PROTECT

The thread holding the mutex runs at a priority at least as high as the highest priority of any mutex that it currently holds.

PTHREAD_PRIO_INHERIT

The thread holding the mutex runs at a priority at least as high as the highest priority of any thread blocked on that mutex.

When a low-priority thread has acquired a mutex, and a thread with higher priority claims the mutex and is blocked, a “priority inversion” takes place—a higher-priority thread is forced to wait for one of lower priority. The PTHREAD_PRIO_INHERIT protocol prevents this—when a thread of higher priority blocks, the thread holding the mutex has its priority boosted during the time it holds the mutex.

When round-robin scheduling is used, and a mutex represents a critical section of code, a second problem can arise. If a thread acquires the mutex, enters the critical section, and then is suspended because its time slice is up, other threads can be blocked needlessly waiting for the mutex. The PTHREAD_PRIO_PROTECT protocol prevents this. Using pthread_mutexattr_setprioceiling() you set a priority higher than normal for the mutex. A thread that acquires the mutex runs at this higher priority while it holds the mutex. This keeps it at the front of the round-robin queue until it exits the critical section and releases the mutex.


Tip: PTHREAD_PRIO_NONE uses a faster code path than the other two priority options for mutexes.


Using Mutexes

The functions for claiming, releasing, and using mutexes are summarized in Table 11-8.

Table 11-8. Functions for Using Mutexes

Function

Purpose

pthread_mutex_lock(3P)

Claim a mutex, blocking until it is available.

pthread_mutex_trylock(3P)

Test a mutex and acquire it if it is available, else return an error.

pthread_mutex_unlock(3P)

Release a mutex.

pthread_mutex_getprioceiling(3P)

Query the minimum priority of a mutex.

pthread_mutex_setprioceiling(3P)

Set the minimum priority of a mutex.

To determine where mutexes should be used, examine the memory variables and other objects (such as files) that can be accessed from multiple threads. Create a mutex for each set of shared objects that are used together. Ensure that the code acquires the proper mutex before it modifies the shared objects. You acquire a mutex by calling pthread_mutex_lock(), and release it with pthread_mutex_unlock(). When a thread must not be blocked, it can use pthread_mutex_trylock() to test the mutex and lock it only if it is available.

Condition Variables

A condition variable provides a way in which a thread can temporarily give up ownership of a mutex, wait for a condition to be true, and then reclaim ownership of the mutex, all in a single operation.

Preparing Condition Variables

Like mutexes and threads themselves, condition variables are supplied with a mechanism of attribute objects (pthread_condattr_t objects) and static and dynamic initializers. However, a condition variable has no useful attributes to initialize in this implementation. The functions for initializing one are summarized in Table 11-9.

Table 11-9. Functions for Preparing Condition Variables

Function

Purpose

pthread_cond_init(3P)

Initialize a condition variable based on an attribute object.

pthread_condattr_init(3P)

Initialize a pthread_condattr_t to default attributes.

pthread_condattr_destroy(3P)

Uninitialize a pthread_condattr_t.

A condition variable must be initialized before use. You can do this in one of three ways:

  • Static assignment of the constant PTHREAD_COND_INITIALIZER.

  • Calling pthread_cond_init() passing NULL instead of the address of an attribute object.

  • Calling pthread_cond_init() passing a pthread_condattr_t object that you have set up with attribute values.

The first two methods initialize the variable to default attributes. Dynamic initialization should be done only once (see “Initializing Static Data”).

Using Condition Variables

A condition variable is a software object that represents a test of a Boolean condition. Typically the condition changes because of a software event such as “other thread has supplied needed data.” A thread that wants to wait for that event claims the condition variable, which causes it to wait. The thread that recognizes the event signals the condition variable, releasing one or all threads that are waiting for the event.

A thread holds a mutex that represents a shared resource. While holding the mutex, the thread finds that the shared resource is not complete or not ready. The thread needs to do three things:

  • Give up the mutex so that some other thread can renew the shared resource.

  • Wait for the event that “resource is now ready for use.”

  • Re-acquire the mutex for the shared resource.

These three actions are combined into one using a condition variable. The functions used with condition variables are summarized in Table 11-10.

Table 11-10. Functions for Using Condition Variables

Function

Purpose

pthread_cond_wait(3P)

Wait on a condition variable.

pthread_cond_timedwait(3P)

Wait on a condition variable, returning with an error after a time limit expires.

pthread_cond_signal(3P)

Signal that an awaited event has occurred, releasing at least one waiting thread.

pthread_cond_broadcast(3P)

Signal that an awaited event has occurred, releasing all waiting threads.

The pthread_cond_wait() and pthread_cond_timedwait() functions require two arguments: a mutex that is owned by the calling thread and a condition variable. The mutex is released and the wait begins. When the event is signalled (or the time limit expires), the mutex is reacquired, as if by a call to pthread_mutex_lock().

The POSIX standard explicitly warns that it is possible in some cases for a conditional wait to return early, before the event has been signalled. For this reason, a conditional wait should always be coded in a loop that tests the shared resource for the needed status. These principles are suggested in the code in Example 11-6, which is modelled after an example in the POSIX 1003.1c standard.

Example 11-6. Use of Condition Variables


#include <assert.h>
#include <pthread.h>
typedef int listKey_t;
typedef struct element_s { /* list element */
   listKey_t key;
   struct element_s *next;
   int busyFlag;
   pthread_cond_t notBusy; /* event of no-longer-in-use */
} element_t;
typedef struct listHead_s { /* list head and mutex */
   pthread_mutex_t mutList; /* right to modify the list */
   element_t *head;
} listHead_t;
/*
|| Internal function to find an element in a list, returning NULL
|| if the key is not in the list.
|| A returned element could be in use by another thread (busy).
|| The caller is assumed to hold the list mutex, otherwise
|| the returned value could be made invalid at any time.
*/
static element_t *scanList(listHead_t* lp, listKey_t key)
{
   element_t *ep;
   for (ep=lp->head; (ep) ; ep=ep->next)
   {
      if (ep->key == key) break;
   }
   return ep;
}
/*
|| Public function to find a key in a list, wait until the element
|| is no longer busy, mark it busy, and return it.
*/
element_t *getFromList(listHead_t* lp, listKey_t key)
{
   element_t *ep;
   pthread_mutex_lock(&lp->mutList); /* lock list against changes */
   while ((ep=scanList(lp,key)) && (ep->busyFlag))
   {
      pthread_cond_wait(&ep->notBusy, &lp->mutList); /* (A) */
   }
   if (ep) ep->busyFlag = 1;
   pthread_mutex_unlock(&lp->mutList);
   return ep;
}
/*
|| Public function to release an element returned by getFromList().
*/
void freeInList(listHead_t* lp, element_t *ep)
{
   assert(ep->busyFlag);
   pthread_mutex_lock(&lp->mutList); /* lock list to prevent races */
   ep->busyFlag = 0;
   pthread_cond_signal(&ep->notBusy);
   pthread_mutex_unlock(&lp->mutList);
}
/*
|| Public function to delete a list element returned by getFromList().
*/
void deleteInList(listHead_t* lp, element_t *ep)
{
   element_t **epp;
   assert(ep->busyFlag);
   pthread_mutex_lock(&lp->mutList);
   for (epp = &lp->head; ep != *epp; epp = &((*epp)->next))
   { /* finding anchor of *ep in list */ }
   *epp = ep->next; /* remove *ep from list */
   ep->busyFlag = 0;
   pthread_cond_broadcast(&ep->notBusy);
   pthread_mutex_unlock(&lp->mutList);
   pthread_cond_destroy(&ep->notBusy);
   free(ep);
}

The functions in Example 11-6 implement part of a simple library for managing lists. In a list head, mutList is a mutex object that represents the right to modify any part of the list. The elements of a list can be “busy,” that is, in use by some thread. An element that is busy has a nonzero busyFlag field.

The getFromList() function looks up an element in a specified list, makes that element busy, and returns it. The function begins by acquiring the list mutex. This ensures that the list cannot change while the function is searching the list, and makes it legitimate for the function to change the busy flag in an element.

When it finds the element, the function might discover that the element is already busy. In this case, it must wait for the event “element is no longer busy,” which is represented by the condition variable notBusy in the element. In order to wait for this event, getFromList() calls pthread_cond_wait() passing its list mutex and the condition variable (point “(A)” in the code). This releases the list mutex so that other threads can acquire the list and do their work on other elements.

When any thread wants to release the use of a list element, it calls freeInList(). After clearing the busy flag in the list element, freeInList() announces that the event “element is no longer busy” has occurred, by calling pthread_cond_signal().

This call releases a thread that is waiting at point “(A).” If there is more than one thread waiting for the same element, the first in priority order is released. The released thread re-acquires the list mutex and resumes execution. The first thing it does is to repeat its search of the list for the desired key and, on finding the element again, to test it again for busyness. This repetition is needed because it is possible to get spurious returns from a condition variable.

When a thread wants to delete a list element, it gets the list element by calling getFromList(). This ensures that the element is busy, so no other thread is using it. Then the thread calls deleteInList(). This function changes the list, so it begins by acquiring the list mutex. Then it can safely modify the list pointers. It scans up the list looking for the pointer that points to the target element. It removes the target element from the list by copying its next field to replace the pointer to the target element.

With the element removed from the list, deleteInList() calls pthread_cond_broadcast() to wake up all threads—not just the first thread—that might be waiting for the element to become nonbusy. Each of these threads resumes execution at point “(A)” by attempting to re-acquire the list mutex. However, deleteInList() is still holding the list mutex. The mutex is released; then the other threads can resume execution following point “(A),” but this time when they search the list, the desired key is no longer found.

Meanwhile, deleteInList() uses pthread_cond_destroy() to release any memory that the pthreads library might have associated with the condition variable, before releasing the list element object itself.