Chapter 4. Using the Frame Scheduler
Prev |  | Next

## Chapter 4. Using the Frame Scheduler

The REACT/Pro Frame Scheduler (FRS) makes it easy to structure a real-time program as a family of independent, cooperating activities, running on multiple CPUs, scheduled in sequence at the frame rate of the application.

This chapter contains details on the operation and use of the Frame Scheduler, under these main headings:

- “Frame Scheduler Concepts” details the operation and methods of the Frame Scheduler.

- “Selecting a Time Base” covers the important choice of which source of interrupts should define a frame interval.

- “Using the Scheduling Disciplines” explains the options for scheduling activities of different kinds.

- “Designing an Application for the Frame Scheduler” presents an overview of the steps in the design process.

- “Preparing the System” reviews the system administration steps needed to prepare the CPUs that the Frame Scheduler will use.

- “Implementing a Single Frame Scheduler” outlines the structure of an application that uses one CPU.

- “Implementing Synchronized Schedulers” outlines the structure of an application that needs the power of multiple CPUs.

- “Handling Frame Scheduler Exceptions” describes how overrun and underrun exceptions are dealt with.

- “Using Signals Under the Frame Scheduler” discusses the issue of signal latency and the signals the Frame Scheduler generates.

- “Using Timers with the Frame Scheduler” covers the use of timers with the Frame Scheduler.

- “FRS Kernel-Level Device Driver Interface” documents the way that a kernel-level device driver can generate time-base interrupts for a Frame Scheduler.

## Frame Scheduler Concepts

One Frame Scheduler dispatches selected threads at a real-time rate on one CPU. You can also create multiple, synchronized Frame Schedulers that dispatch concurrent threads on multiple CPUs.

### Frame Scheduler Basics

When a Frame Scheduler takes over scheduling and dispatching threads on one CPU, it isolates the CPU (see “Isolating a CPU from TLB Interrupts” in Chapter 3), and completely supersedes the operation of the normal IRIX scheduler on that CPU. Only threads queued to the Frame Scheduler can use the CPU. IRIX thread dispatching priorities are not relevant on that CPU.

The execution of normal processes, daemons, and pending timeouts are all migrated to other CPUs—typically to CPU 0, which cannot be owned by a Frame Scheduler. All interrupt handling is usually directed away from a Frame Scheduler CPU as well (see “Preparing the System”). However, a Frame Scheduler CPU can be used to handle interrupts, although doing so runs a risk of causing overruns.

### Thread Programming Models

The Frame Scheduler in REACT/pro version 3.2 supports two thread programming models: sprocs and pthreads. Both threading models allow multiprogramming, but sprocs are proprietary to Silicon Graphics, while pthreads are standardized by the IEEE POSIX 1003.1c specification.

In this guide, a thread is defined as an independent flow of execution that consists of a set of registers (including a program counter and a stack).

A traditional IRIX process has a single active thread that starts once the program is executed and runs until the program terminates. A multithreaded process may have several threads active at one time. Hence, a process can be viewed as a receptacle that contains the threads of execution and the resources they share (that is, data segments, text segments, file descriptors, synchronizers, and so forth).

### Frame Scheduling

Instead of scheduling threads according to priorities, the Frame Scheduler dispatches them according to a strict, cyclic rotation governed by a repetitive time base. The time base determines the fundamental frame rate. (See “Selecting a Time Base”.) Some examples of the time base are as follows:

- A specific clocked interval in microseconds

- The Vsync (vertical retrace) interrupt from the graphics subsystem

- An external interrupt (see “External Interrupts”)

- A device interrupt from a specially modified device driver

- A system call (normally used for debugging)

The interrupts from the time base define minor frame s. Together, a fixed number of minor frames make up a major frame. The length of a major frame defines the application's true frame rate. The minor frames allow you to divide a major frame into subframes. Major and minor frames are shown in Figure 4-1.

In the simplest case, you have a single frame rate, such as 60 Hz, and every activity your program does must be done once per frame. In this case, the major and minor frame rates are the same.

In other cases, you have some activities that must be done in every minor frame, but you also have activities that are done less often: in every other minor frame or in every third one. In these cases, you define the major frame so that its rate is the rate of the least-frequent activity. The major frame contains as many minor frames as necessary to schedule activities at their relative rates.

Figure 4-1. Major and Minor Frames

As pictured in Figure 4-1, the Frame Scheduler maintains a queue of threads for each minor frame. Queue each activity thread of your program to a specific minor frame. Determine the order of cyclic execution within a minor frame by the order in which you queue threads. You can do the following:

- Queue multiple threads in one minor frame. They are run in the queued sequence within the frame. All must complete their work within the minor frame interval.

- Queue the same thread to run in more than one minor frame. Say that thread double is to run twice as often as thread solo. You queue double to Q0 and Q2 in Figure 4-1, and queue solo to Q1.

- Queue a thread that takes more than a minor frame to complete its work. If thread sloth needs more than one minor interval, you queue it to Q0, Q1, and Q2 in Figure 4-1, such that it can continue working in all three minor frames until it completes.

- Queue a background thread that is allowed to run only when all others have completed, to use up any remaining time within a minor frame.

All these options are controlled by scheduling disciplines you specify for each thread as you queue it (see “Using the Scheduling Disciplines”).

Typically a frame scheduler is driven by a single interrupt source and contains minor frames having the same duration, but a variable frame scheduler may be used to implement an FRS having multiple interrupt sources and/or minor frames of variable duration (see the `frs_create_vmaster`() function).

The relationship between threads and a Frame Scheduler depends upon the thread model in use, as follows:

- The pthread programming model requires that all threads scheduled by the Frame Scheduler and controlling the Frame Scheduler be system scope threads. These threads must also reside in the same process.

- The `sproc`() and `fork`() programming models do not require that the participating threads reside in the same process.

See “Implementing a Single Frame Scheduler” for details.

#### The FRS Controller Thread

The thread that creates a Frame Scheduler is called the FRS controller thread. It is privileged in these respects:

- Its identifier is used to identify its Frame Scheduler in various functions. If you are using POSIX threads, the FRS controller thread uses a pthread ID; if you are using `sproc`(), the FRS controller process uses a PID.

- It can receive signals when errors are detected by the Frame Scheduler (see “Using Signals Under the Frame Scheduler”).

- It cannot itself be queued to the Frame Scheduler. It continues to be dispatched by IRIX, and executes on a CPU other than the one the Frame Scheduler uses.

### The Frame Scheduler API

An overview of the Frame Scheduler API can be found in the `frs``(3)` man page, which provides a complete listing of all the FRS functions. Separate man pages for each of the FRS functions provide the details of the Frame Scheduler API. The API elements are declared in `/usr/include/sys/frs.h`. The following are some important types that are declared in `/usr/include/sys/frs.h`:

`typedef frs_fsched_info_t` | A structure containing information about one scheduler, including its CPU number, interrupt source and time base, and number of minor frames. Used when creating a Frame Scheduler.
`typedef frs_t` | A structure that identifies a Frame Scheduler.
`typedef frs_queue_info_t` | A structure containing information about one activity thread: the Frame Scheduler and minor frame it uses and its scheduling discipline. Used when enqueuing a thread.
`typedef frs_recv_info_t` | A structure containing error recovery options.
`typedef frs_intr_info_t` | A structure that `frs_create_vmaster`() uses for defining interrupt information templates (see below).

Additionally the pthreads interface adds the following types, as declared in `/usr/include/sys/pthread.h`:

`typedef pthread_t` | An integer identifying the pthread ID.
`typedef pthread_attr_t` | A structure containing information about the attributes of the FRS controller thread.

### Interrupt Information Templates

Variable frame schedulers may drive each minor frame with a different interrupt source, as well as define a different duration for each minor frame. These two characteristics may be used together or separately, and are defined using an interrupt information template.

An interrupt information template consists of an array of frs_intr_info_t data structures, where each element in the array represents a minor frame. For example, the first element in the array represents the interrupt information for the first minor frame, and so on for n minor frames.

The frs_intr_info_t data structure contains two fields for defining the interrupt source and its qualifier: intr_source and intr_qualifier.

The following example demonstrates how to define an interrupt information template for a frame scheduler having minor frames of different duration. Assume the application requires four minor frames, where each minor frame is triggered by the synchronized clock timer, and the duration of each minor frame is as follows: 100 ms, 150 ms, 200 ms, and 250 ms. The interrupt information template may be defined as follows:

    frs_intr_info_t intr_info[4];
    intr_info[0].intr_source    = FRS_INTRSOURCE_CCTIMER;
    intr_info[0].intr_qualifier = 100000;
    intr_info[1].intr_source    = FRS_INTRSOURCE_CCTIMER;
    intr_info[1].intr_qualifier = 150000;
    intr_info[2].intr_source    = FRS_INTRSOURCE_CCTIMER;
    intr_info[2].intr_qualifier = 200000;
    intr_info[3].intr_source    = FRS_INTRSOURCE_CCTIMER;
    intr_info[3].intr_qualifier = 250000;

The following example demonstrates how to define an interrupt information template for a frame scheduler using multiple interrupt sources. Assume the application requires two minor frames, where the first minor frame is triggered by the vertical retrace interrupt and the second minor frame is triggered by the CPU timer. Also assume the vertical retrace interrupt is running at 60 Hz (every 16.6 ms). The following interrupt information template defines the CPU timer interrupt of the second frame to fire 8.3 ms after the vertical retrace interrupt:

    frs_intr_info_t intr_info[2];
    intr_info[0].intr_source    = FRS_INTRSOURCE_VSYNC;
    intr_info[0].intr_qualifier = 0;
    intr_info[1].intr_source    = FRS_INTRSOURCE_CPUTIMER;
    intr_info[1].intr_qualifier = 8300;

Note that 8.3 ms was chosen in the example because it is known that the timer interrupt will fire before the next major frame's vsync interrupt. If 20 ms were chosen for the timer instead, then a sequence error would occur (see section “Sequence Error”) and an error signal would be sent to the controller thread.

Detailed programming examples are available, demonstrating use of variable frame schedulers, which can be found in the `/usr/share/src/react/examples` directory. For more information, see the `frs_create_vmaster(3)` man page.

#### Library Interface for C Programs

The API library functions in the `/usr/lib/libfrs.a` file are summarized in Table 4-1 for convenient reference.

Table 4-1. Frame Scheduler Operations

Operation | Used For | Frame Scheduler API
--- | --- | ---
Create a Frame Scheduler | Process setup | `frs_t*` `frs_create`(`int` cpu, `int` intr_source, `int` intr_qualifier, `int` n_minors, `pid_t` sync_master_pid, `int` num_slaves```)`;
| Process or pthread setup | `frs_t*```` `frs_create_master``(`` ```int` cpu, `int` intr_source, `int` intr_qualifier, `int` n_minors, `int` num_slaves```)`;
| Process or pthread setup | `frs_t`* `frs_create_slave``(` ```int` cpu, `frs_t`* sync_master_frs```)`;
| Process or pthread setup | `frs_t`*`` `frs_create_vmaster``(` ```int` cpu, int n_minors, `int` n_slaves, `frs_intr_info_t` *intr_info```)`;
Queue to an FRS minor frame | Process setup | `int` `frs_enqueue``(` ```frs_t`* frs, `pid_t` pid, `int` minor_frame, `unsigned` `int` discipline```)`;
| Pthread setup | `int` `frs_pthread_enqueue``(` ```frs_t`* frs, `pthread_t` pthread, `int` minor_frame, `unsigned int` discipline```);`
Insert into a queue, possibly changing discipline | Process setup | `int` `frs_pinsert``(` ```frs_t`* frs, `int` minor_frame, `pid_t` target_pid, `int` discipline, `pid_t` base_pid```)`;
| Pthread setup | `int` `frs_pthread_insert``(` ```frs_t`* frs, `int` minor_index, `pthread_t` target_pthread, `int` discipline, `pthread_t` base_pthread```)`;
Set error recovery options | Process setup | `int` `frs_setattr``(` ```frs_t`* frs, `int` minor_frame, `pid_t` pid, `frs_attr_t` attribute, `void`* param```)`;
| Pthread setup | `int` `frs_pthread_setattr``(` ```frs_t`* frs, `int` minor_frame, `pthread_t` pthread, `frs_attr_t` attribute, `void`* param```)`;
Join an FRS (activity is ready to start) | Process or pthread execution | `int` `frs_join``(` ```frs_t`* frs```)`;
Start scheduling (all activities queued) | Process or pthread execution | `int` `frs_start``(` ```frs_t`* frs```)`;
Yield control after completing activity | Process or pthread execution | `int` `frs_yield``(` ```void```)`;`
Pause scheduling at end of minor frame | Process or pthread execution | `int` `frs_stop``(` ```frs_t`* frs```)`;
Resume scheduling at next time-base interrupt | Process or pthread execution | `int` `frs_resume``(` ```frs_t`* frs```)`;
Trigger a user-level FRS interrupt | Process or pthread execution | `int` `frs_userintr``(` ```frs_t`* frs```)`;
Interrogate a minor frame queue | Process or pthread query | `int` `frs_getqueuelen``(` ```frs_t`* frs, `int` minor_index```)`;
| Process query | `int` `frs_readqueue``(` ```frs_t`* frs, int minor_frame, `pid_t` *pidlist```)`;
| Pthread query | `int` `frs_pthread_readqueue``(` ```frs_t`* frs, `int` minor_frame, `pthread_t` *pthreadlist```)`;
Retrieve error recovery options | Process query | `int` `frs_getattr``(` ```frs_t`* frs, `int` minor_frame, `pid_t` pid, `frs_attr_t` attribute, `void`* param```)`;
| Pthread query | `int` `frs_pthread_getattr``(` ```frs_t`* frs, `int` minor_frame, `pthread_t` pthread, `frs_attr_t` attribute, `void`* param```)`;
Destroy FRS and send `SIGKILL` to its FRS controller | Process or pthread teardown | `int` `frs_destroy``(` ```frs_t`* frs```)`;
Remove a process or thread from a queue | Process teardown | `int` `frs_premove``(` ```frs_t`* frs, `int` minor_frame, `pid_t` remove_pid```)`;
| Pthread teardown | `int` `frs_pthread_remove``(` ```frs_t`* frs, `int` minor_frame, `pthread_t` remove_pthread```)`;

#### System Call Interface for Fortran and Ada

Each Frame Scheduler function is available in two ways: as a system call to `schedctl`(), or as one or more library calls to functions in the `frs` library, `/usr/lib/libfrs.a`. The system call is accessible from FORTRAN and Ada programs because both languages have bindings for `schedctl`() (see the `schedctl`(2) man page). The correspondence between the library functions and `schedctl`() calls is shown in Table 4-2.

| Note: The pthread functions for the Frame Scheduler are not supported for FORTRAN applications.

Table 4-2. Frame Scheduler schedctl() Support

Library Function | Schedctl Syntax
--- | ---
`frs_create`() | `int` `schedctl(` ```MPTS_FRS_CREATE, frs_info_t`* frs_info```)`;
`frs_enqueue`() | `int` `schedctl(` ```MPTS_FRS_ENQUEUE, frs_queue_info_t`* frs_queue_info```)`;
`frs_join`() | `int` `schedctl(` ```MPTS_FRS_JOIN, pid_t` frs_master```)`;
`frs_start`() | `int` `schedctl(` ```MPTS_FRS_START, pid_t` frs_master```)`;
`f` `rs_yield`() | `int` `schedctl(` ```MPTS_FRS_YIELD```)`;`
`frs_stop`() | `int` `schedctl(` ```MPTS_FRS_STOP, pid_t` frs_master```)`;
`f` `rs_resume`() | `int` `schedctl(` ```MPTS_FRS_RESUME, pid_t` frs_master```)`;
`frs_destroy`() | `int` `schedctl(` ```MPTS_FRS_DESTROY, pid_t` frs_master```)`;
`frs_getqueuelen`() | `int` `schedctl(` ```MPTS_FRS_GETQUEUELEN, frs_queue_info_t`* frs_queue_info```)`;
`frs_readqueue``()` | `int` `schedctl``(` ```MPTS_FRS_READQUEUE, frs_queue_info_t`* frs_queue_info, `pid_t`* pidlist```)`;
`frs_premove()` | `int` `schedctl``(` ```MPTS_FRS_PREMOVE, frs_queue_info_t`* frs_queue_info```)`;
`frs_pinsert`() | `int` `schedctl``(` ```MPTS_FRS_PINSERT, frs_queue_info_t`* frs_queue_info, `pid_t` *base_pid```)`;
`frs_getattr`() | `int` `schedctl` ```(MPTS_FRS_GETATTR, frs_attr_info_t`* frs_attr_info```)`;
`frs_setattr`() | `int` `schedctl``(` ```MPTS_FRS_SETATTR, frs_attr_info_t`* frs_attr_info```)`;

### Thread Execution

An activity thread that is queued to a Frame Scheduler has the basic structure shown in Example 4-1.

Example 4-1. Skeleton of an Activity Thread

    /* Initialize data structures etc. */
    frs_join(scheduler-handle)
    do
    {
       /* Perform the activity. */
       frs_yield();
    } while(1);
    _exit();

When the thread is ready to start real-time execution, it calls `frs_join`(). This call blocks until all queued threads are ready and scheduling begins (see “Starting Multiple Schedulers”). When `frs_join`() returns, the thread is running in its first minor frame. For more information about `frs_join`(), see `frs_join`(3).

The thread then performs whatever activity is needed to complete the minor frame and calls `frs_yield`(). This gives up control of the CPU until the next minor frame where the thread is queued and executes. For more information about `frs_yield`(), see `frs_yield`(3).

An activity thread is never preempted within a minor frame. As long as it yields before the end of the frame, it can do its assigned work without interruption from other threads (it can be interrupted by hardware interrupts, if any hardware interrupts are allowed in that CPU). The Frame Scheduler preempts the thread at the end of the minor frame.

| Tip: Because an activity thread cannot be preempted, it can often use global data without locks or semaphores. When the thread that modifies a global variable is queued in a different minor frame than the threads that read the variable, there can be no access conflicts between them.

Conflicts are still possible between two threads that are queued to the same minor frame in different, synchronized Frame Schedulers. However, such threads are guaranteed to be running concurrently. This means they can use spin-locks (see “Locks” in Chapter 2) with high efficiency.

| Tip: When a very short minor frame interval is used, it is possible for a thread to have an overrun error in its first frame due to cache misses. A simple variation on the basic structure shown in Example 4-1 is to spend the first minor frame touching a set of important data structures in order to “warm up” the cache. This is sketched in Example 4-2.

Example 4-2. Alternate Skeleton of Activity Thread

    /* Initialize data structures etc. */
    frs_join(scheduler-handle); /* Much time could pass here. */
    /* First frame: merely touch important data structures. */
    do
    {
       frs_yield();
       /* Second and later frames: perform the activity. */
    } while(1);
    _exit();

When an activity thread is scheduled on more than one minor frame in a major frame, it can be designed to do nothing except warm the cache in the entire first major frame. To do this, the activity thread function has to know how many minor frames it is scheduled on, and calls `frs_yield`() that many times in order to pass the first major frame.

### Scheduling Within a Minor Frame

Threads in a minor frame queue are dispatched in the order they appear on the queue (priority is irrelevant). Queue ordering can be modified as follows:

- Appending a thread at the end of the queue with `frs_pthread_enqueue`() or `frs_enqueue`()

- Inserting a thread after a specific target thread via`frs_pthread_insert`() or`frs_pinsert`()

- Deleting a thread in the queue with `frs_pthread_remove`() or `frs_premove`()

See the man pages `frs_enqueue`(3), `frs_pinsert`(3), `frs_premove`(3), and “Managing Activity Threads”.

#### Scheduler Flags frs_run and frs_yield

The Frame Scheduler keeps two status flags per queued thread, named `frs_run` and `frs_yield`. If a thread is ready to run when its turn comes, it is dispatched and its `frs_run` flag is set to indicate that this thread has run at least once within this minor frame.

When a thread yields, its `frs_yield` flag is set to indicate that the thread has released the processor. It is not activated again within this minor frame.

If a thread is not ready (usually because it is blocked waiting for I/O, a semaphore, or a lock), it is skipped. Upon reaching the end of the queue, the scheduler goes back to the beginning, in a round-robin fashion, searching for threads that have not yielded and may have become ready to run. If no ready threads are found, the Frame Scheduler goes into idle mode until a thread becomes available or until an interrupt marks the end of the frame.

#### Detecting Overrun and Underrun

When a time base interrupt occurs to indicate the end of the minor frame, the Frame Scheduler checks the flags for each thread. If the `frs_run` flag has not been set, that thread never ran and therefore is a candidate for an underrun exception. If the `frs_run` flag is set but the `frs_yield` flag is not, the thread is a candidate for an overrun exception.

Whether these exceptions are declared depends on the scheduling discipline assigned to the thread. Scheduling disciplines are explained under “Using the Scheduling Disciplines”).

At the end of a minor frame, the Frame Scheduler resets all `frs_run` flags, except for those of threads that use the Continuable discipline in that minor frame. For those threads, the residual `frs_yield` flags keeps the threads that have yielded from being dispatched in the next minor frame.

Underrun and overrun exceptions are typically communicated via IRIX signals. The rules for sending these signals are covered under “Using Signals Under the Frame Scheduler”.

#### Estimating Available Time

It is up to the application to make sure that all the threads queued to any minor frame can actually complete their work in one minor-frame interval. If there is too much work for the available CPU cycles, overrun errors will occur.

Estimation is simplified by the fact that only the queued threads can execute on a CPU controlled by the Frame Scheduler. You need to estimate the maximum time each thread can consume between one call to `frs_yield`() and the next.

Frame Scheduler threads do compete for CPU cycles with I/O interrupts on the same CPU. If you direct I/O interrupts away from the CPU (see “Isolating a CPU from Sprayed Interrupts” in Chapter 3 and “Redirecting Interrupts” in Chapter 3), then the only competition for CPU cycles (other than a very few essential TLB interrupts) is the overhead of the Frame Scheduler itself, and it has been carefully optimized for least overhead.

Alternatively, you may assign specific I/O interrupts to a CPU used by the Frame Scheduler. In that case, you must estimate the time that interrupt service will consume (see “Maximum Response Time Guarantee” in Chapter 3) and allow for it.

### Synchronizing Multiple Schedulers

When the activities of one frame cannot be completed by one CPU, you need to recruit additional CPUs and execute some activities concurrently. However, it is important that each of the CPUs have the same time base, so that each starts and ends frames at the same time.

You can create one master Frame Scheduler, which owns the time base and one CPU, and as many synchronized (slave) Frame Schedulers as you need, each managing an additional CPU. The slave schedulers take their time base from the master, so that all start minor frames at the same instant.

Each FRS requires its own controller thread. Therefore, to create multiple, synchronized Frame Schedulers, you must create a controller thread for the master and each slave FRS.

Each Frame Scheduler has its own queues of threads. A given thread can be queued to only one CPU. (However, you can create multiple threads based on the same code, and queue each to a different CPU.) All synchronized Frame Schedulers use the same number of minor frames per major frame, which is taken from the definition of the master FRS.

### Starting a Single Scheduler

A single Frame Scheduler is created when the FRS controller thread calls `frs_create_master()` or `frs_create`(). The FRS controller calls `frs_pthread_enqueue`() or `frs_enqueue`() one or more times to notify the new Frame Scheduler of the threads to schedule in each of the minor frames. The FRS controller calls `frs_start`() when it has queued all the threads. Each scheduled thread must call `frs_join`() after it has initialized and is ready to be scheduled.

Each activity thread must be queued to at least one minor frame before it can join the FRS via `frs_join`(). Once all activity threads have joined and the FRS is started by the controller thread, the first minor frame begins executing. For more information about these functions, see the `frs_enqueue`(3), `frs_join`(3), and `frs_start`(3) man pages.

### Starting Multiple Schedulers

A Frame Scheduler cannot start dispatching activities until the following has occurred:

- The FRS controller has queued all the activity threads to their minor frames.

- All the queued threads have done their own initial setup and have joined.

When multiple Frame Schedulers are used, none can start until all are ready.

Each FRS controller notifies its Frame Scheduler that it has queued all activities by calling `frs_start`(). Each activity thread signals its Frame Scheduler that it is ready to begin real-time processing by calling `frs_join`().

A Frame Scheduler is ready when it has received one or more `frs_pthread_enqueue`() or `frs_enqueue`() calls, a matching number of `frs_join`() calls, and an `frs_start`() call for each Frame Scheduler. Each slave Frame Scheduler notifies the master Frame Scheduler when it is ready. When all the schedulers are ready, the master Frame Scheduler gives the downbeat, and the first minor frame begins.

### Pausing Frame Schedulers

Any Frame Scheduler can be made to pause and restart. Any thread (typically but not necessarily the FRS controller) can call `frs_stop`(), specifying a particular Frame Scheduler. That scheduler continues dispatching threads from the current minor frame until all have yielded. Then it goes into an idle loop until a call to `frs_resume`() tells it to start. It resumes on the next time-base interrupt, with the next minor frame in succession. For more information, see the`frs_stop`(3) and `frs_resume`(3) man pages.

| Note: If there is a thread running Background discipline in the current minor frame, it continues to execute until it yields or is blocked on a system service.

Since a Frame Scheduler does not stop until the end of a minor frame, you can stop and restart a group of synchronized schedulers by calling `frs_stop`() for each one before the end of a minor frame. There is no way to restart all of a group of schedulers with the certainty that they start up on the same time-base interrupt.

### Managing Activity Threads

The FRS control thread identifies the initial set of activity threads by calling `frs_pthread_enqueue`() or `frs_enqueue`() prior to starting the Frame Scheduler. All the queued threads must call `frs_join`() before scheduling can begin. However, the FRS controller can change the set of activity threads dynamically while the Frame Scheduler is working, using the following functions:

`frs_getqueuelen`() | Get the number of threads currently in the queue for a specified minor frame.
`frs_pthread_readque ue`() or `frs_readqueue`() | Return the ID values of all queued threads for a specified minor frame as a vector of integers.
`frs_pthread_remove`() or `frs_premove`() | Remove a thread (specified by its ID) from a minor frame queue.
`frs_pthread_insert`() or `frs_pinsert`() | Insert a thread (specified by its ID and discipline) into a given position in a minor frame queue.

Using these functions, the FRS controller can change the queueing discipline (overrun, underrun, continuable) of a thread by removing it and inserting it with a new discipline. The FRS controller can suspend a thread by removing it from its queue; or can restart a thread by putting it back in its queue.

| Note: When an activity thread is removed from the last or only queue it was in, it is returned to the normal IRIX scheduler and can begin to execute on another CPU. When an activity thread is removed from a queue, a signal may be sent to the removed thread (see “Handling Signals in an Activity Thread”). If a signal is sent to it, it begins executing in its specified or default signal handler; otherwise, it simply begins executing following `frs_yield`(). Once returned to the IRIX scheduler, a call to an FRS function such as `frs_yield`() returns an error (this also can be used to indicate the resumption of normal scheduling).

The FRS controller can also queue new threads that have not been scheduled before. The Frame Scheduler does not reject an `frs_pthread_insert`() or `frs_pinsert`() call for a thread that has not yet joined the scheduler. However, a thread must call `frs_join`() before it can be scheduled. For more information, see the `frs_pinsert`(3) man page.

If an queued thread should be terminated for any reason, the Frame Scheduler removes the thread from all queues in which it appears.

## Selecting a Time Base

Your program specifies an interrupt source for the time base when it creates the master (or only) Frame Scheduler. The master Frame Scheduler initializes the necessary hardware resources and redirects the interrupt to the appropriate CPU and handler.

The Frame Scheduler time base is fundamental because it determines the duration of a minor frame, and hence the frame rate of the program. This section explains the different time bases that are available.

When you use multiple, synchronized Frame Schedulers, the master Frame Scheduler distributes the time-base interrupt to each synchronized CPU. This ensures that minor-frame boundaries are synchronized across all the Frame Schedulers.

### On-Chip Timer Interrupt

Each processor chip contains a free-running timer that is used by IRIX for normal process scheduling. This timer is not synchronized between processors, so it cannot be used to drive multiple synchronized schedulers. The on-chip timer can be used as a time base when only one CPU is used.

To use the on-chip timer, specify `FRS_INTRSOURCE_CPUTIMER` as the interrupt source, and the minor frame interval in microseconds, to `frs_create_master`() or `frs_create`().

### High-Resolution Timer

The high-resolution timer and clock is a timer that is synchronous across all processors, and is ideal to drive synchronous schedulers. On Origin, Onyx 2, CHALLENGE, and Onyx systems, this timer is based on the high-resolution counter discussed under “Hardware Cycle Counter” in Chapter 2.

To use this timer, specify FRS_INTRSOURCE_CCTIMER, and specify the minor frame interval in microseconds to `frs_create_master`() or `frs_create`().

The IRIX kernel uses this timer for managing timer events. When your program creates the master Frame Scheduler, the Frame Scheduler migrates all timeout events to CPU 0, leaving the timer on the scheduled CPU free.

The high-resolution timers in all CPUs are synchronized automatically.

### Vertical Sync Interrupt

An interrupt is generated for every vertical retrace by the graphics subsystem (see “Understanding the Vertical Sync Interrupt” in Chapter 3). The frame rate is either 50 Hz or 60 Hz, depending on the installed hardware. This interrupt is especially appropriate for a visual simulator, since it defines a frame rate that matches the graphics subsystem frame rate.

To use the vertical sync interrupt, specify `FRS_INTRSOURCE_VSYNC` to `frs_create_master`() or `frs_create`(). An error is returned if this system is not configured with a graphics subsystem.

When multiple synchronized schedulers are used, the master Frame Scheduler distributes the vertical sync interrupt.

### External Interrupts

An external interrupt is generated via a signal applied to the external interrupt socket on systems supporting such a hardware feature, such as Origin, Challenge, and Onyx systems (see “External Interrupts” in Chapter 6). To use external interrupts as a time base, use the following steps:

1. Redirect the external interrupt to the master frame scheduler (FRS) CPU using the appropriate device administration directive in `/var/sysgen/system/irix.sm`.

2. For the directives take effect, rebuild the kernel using the command `/etc/autoconfig -vf`, and reboot.

3. Specify `FRS_INTRSOURCE_EXTINTR` to `frs_create_master`() or `frs_create`().

For example, in`irix.sm`, a directive similar to the following causes PCI interrupt 4 of the first I/O slot to be handled by CPU 1. (The actual directive depends on the hardware configuration of the target platform.)

    DEVICE_ADMIN: /hw/module/1/slot/io1/baseio/pci/4 INTR_TARGET=/hw/cpunum/1

When multiple synchronized schedulers are used, the master Frame Scheduler receives the interrupt and allocates it simultaneously to the synchronized schedulers.

| Note: When the external interrupt is routed to a CPU, all of the interrupt threads that are triggered from the same physical interrupt will be bound to that CPU as well. When the FRS assumes scheduling responsibility for that CPU, to avoid jitter and potential delay, those threads will be prevented from running. If you still need access to the other devices in the IOC3 module that holds the external interrupt port, you should use another IOC3 module.

### Device Driver Interrupt

A user-written, kernel-level device driver can supply the time-base interrupt (see “FRS Kernel-Level Device Driver Interface”). The Frame Scheduler registers the driver and assigns it a unique registration number, then allocates an interrupt group. The device driver must direct interrupts to it.

To use a device driver as a time base, specify `FRS_INTRSOURCE_DRIVER` and the device driver's registration number to `frs_create_master`() or `frs_create`(). See “Implementing a Single Frame Scheduler”.

### Software Interrupt

A programmed, software-generated interrupt can be used as the time base. Any user process can send this interrupt to the master Frame Scheduler by calling `frs_userintr`().

| Note: Software interrupts are primarily intended for application debugging. It is not feasible for a user process to generate the low-latency and determinism for interrupts required by a real-time application.

To use software interrupts as a time base, specify `FRS_INTRSOURCE_USER` to `frs_create_master`() or `frs_create`().

| Caution: The use of software interrupts has a potential for causing a system deadlock if the interrupt-generating process contends for a resource that is also used by a frame-scheduled activity thread. If any activity thread calls IRIX system functions, the only way to be absolutely sure of avoiding deadlock is for the interrupt-generating process to avoid using any IRIX system functions. Note that C library functions such as `printf`() invoke system functions, and can lead to deadlocks in this case.

### User-Level Interrupts

The user level interrupt (ULI) facility allows a hardware interrupt to be handled by a user process, enabling device drivers to reside at the user-level; see the `uli`(3) man page for details. To use ULI to drive the frame scheduler, specify `FRS_INTRSOURCE_ULI` as the interrupt source to the `frs_create_master`() or `frs_create_vmaster`() functions.

The ULI capability is supported only for Origin and Onyx 2 platforms. The PCI or VME interrupt must be routed to the master Frame Scheduled processor in order for the frame scheduler to recognize it.

| Note: The frame scheduler is invoked after the user-level portion of the interrupt handler has completed servicing the interrupt.

## Using the Scheduling Disciplines

When an FRS controller thread queues an activity thread to a minor frame (using `frs_pthread_enqueue`() or `frs_enqueue`()), it must specify a scheduling discipline that tells the Frame Scheduler how the thread is expected to use its time within that minor frame.

### Real-Time Discipline

In the simplest case, an activity thread starts during the minor frame in which it is queued, and completes its work and yields within the same minor frame.

If the thread is not ready to run (for example, blocked on I/O) during the entire minor frame, an underrun exception is said to occur. If the thread fails to complete its work and yield within the minor frame interval, an overrun exception is said to occur.

The Frame Scheduler calls this strict discipline the Real-time scheduling discipline.

This model could describe a simple kind of simulator in which certain activities—poll the inputs; calculate the new status; update the display—must be repeated in that order during every frame. In this scenario, each activity must start and must finish in every frame. If one fails to start, or fails to finish, the real-time program is broken in some way and must take some action.

However, realistic designs need the flexibility to have threads with the following characteristics:

- Need not start every frame; for instance, threads that sleep on a semaphore until there is work for them to do

- May run longer than one minor frame

- Should run only when time is available, and whose rate of progress is not critical

The other disciplines are used, in combination with Real-time and with each other, to allow these variations.

### Background Discipline

The Background discipline is mutually exclusive with the other disciplines. The Frame Scheduler dispatches a Background thread only when all other threads queued to that minor frame have run and have yielded. Since the Background thread cannot be sure it will run and cannot predict how much time it will have, the concepts of underrun and overrun do not apply to it.

| Note: A thread with the Background discipline must be queued to its frame following all non-Background threads. Do not queue a real-time thread after a Background thread.

### Underrunable Discipline

You specify Underrunable discipline with Real-time discipline to prevent detection of underrun exceptions. You specify Underrunable in the following cases:

- When a thread needs to run only when an event has occurred, such as a lock being released or a semaphore being posted.

- When a thread may need more than one minor frame (see “Using Multiple Consecutive Minor Frames”).

When you specify Real-time+Underrunable, the thread is not required to start in that minor frame. However, if it starts, it is required to yield before the end of the frame or an overrun exception is raised.

### Overrunnable Discipline

You specify Overrunnable discipline with Real-time discipline to prevent detection of overrun exceptions. You specify it in the following cases:

- When it truly does not matter if the thread fails to complete its work within the minor frame—for example, a calculation of a game strategy which, if it fails to finish, merely makes the computer a less dangerous opponent.

- When a thread may need more than one minor frame (see “Using Multiple Consecutive Minor Frames”).

When you specify Overrunnable+Real-time, the thread is not required to call `frs_yield`() before the end of the frame. Even so, the thread is preempted at the end of the frame. It does not have a chance to run again until the next minor frame in which it is queued. At that time it resumes where it was preempted, with no indication that it was preempted.

### Continuable Discipline

You specify Continuable discipline with Real-time discipline to prevent the Frame Scheduler from clearing the flags at the end of this minor frame (see “Scheduling Within a Minor Frame”).

The result is that, if the thread yields in this frame, it need not run or yield in the following frame. The residual `frs_yield` flag value, carried forward to the next frame, applies. You specify Continuable discipline with other disciplines in order to let a thread execute just once in a block of consecutive minor frames.

### Using Multiple Consecutive Minor Frames

There are cases when a thread sometimes or always requires more than one minor frame to complete its work. Possibly the work is lengthy, or possibly the thread could be delayed by a system call or a lock or semaphore wait.

You must decide the absolute maximum time the thread could consume between starting up and calling `frs_yield`(). If this is unpredictable, or if it is predictably longer than the major frame, the thread cannot be scheduled by the Frame Scheduler. Hence, it should probably run on another CPU under the IRIX real-time scheduler.

However, when the worst-case time is bounded and is less than the major frame, you can queue the thread to enough consecutive minor frames to allow it to finish. A combination of disciplines is used in these frames to ensure that the thread starts when it should, finishes when it must, and does not cause an error if it finishes early.

The discipline settings for each frame should be as follows:

First frame |  | Real-time + Overrunnable + Continuable—the thread must start in this frame (not Underrunable) but is not required to yield (Overrunnable). If it yields, it is not restarted in the following minor frame (Continuable).
Intermediate |  | Real-time+Underrunable+Overrunnable+Continuable—the thread need not start (it might already have yielded, or might be blocked) but is not required to yield. If it does yield (or if it had yielded in a preceding minor frame), it is not restarted in the following minor frame (Continuable).
Final frame |  | Real-time+Underrunable—the thread need not start (it might already have yielded) but if it starts, it must yield in this frame (not Overrunnable). The thread can start a new run in the next minor frame to which it is queued (not Continuable).

A thread can be queued for one or more of these multiframe sequences in one major frame. For example, suppose that the minor frame rate is 60 Hz, and a major frame contains 60 minor frames (1 Hz). You have a thread that should run at a rate of 5 Hz and can use up to 3/60 second at each dispatch. You can queue the thread to 5 sequences of 3 consecutive frames each. It could start in frames 0, 12, 24, 36, and 48. Frames 1, 13, 25, 37 and 49 could be intermediate frames, and 2, 14, 26, 38 and 50 could be final frames.

## Designing an Application for the Frame Scheduler

When using the Frame Scheduler, consider the following guidelines when designing your real-time application.

1. Determine the programming model for implementing the activities in your program, choosing among POSIX threads, IRIX`sproc`(), or SVR4`fork`() calls. (You cannot mix pthreads and other disciplines within your program.)

2. Partition the program into activities, where each activity is an independent piece of work that can be done without interruption.

   For example, in a simple vehicle simulator, activities might include “poll the joystick,” “update the positions of moving objects,” “cull the set of visible objects,” and so forth.

3. Decide the relationships among the activities, as follows:

   - Some must be done once per minor frame, others less frequently.

   - Some must be done before or after others.

   - Some may be conditional. For example, an activity could poll a semaphore and do nothing unless an event had completed.

4. Estimate the worst-case time required to execute each activity. Some activities may need more than one minor frame interval (the Frame Scheduler allows for this).

5. Schedule the activities: If all are executed sequentially, will they complete in one major frame? If not, choose activities that can execute concurrently on two or more CPUs, and estimate again. You may have to change the design in order to get greater concurrency.

When the design is complete, implement each activity as an independent thread that communicates with the others using shared memory, semaphores, and locks (see “Synchronization and Communication” in Chapter 2).

The Frame Scheduler is created, stopped, and resumed by a controller thread. The controller thread can also interrogate and receive signals from the Frame Scheduler (see “Signals” in Chapter 2).

A Frame Scheduler seizes its assigned CPU, isolates it, and controls the scheduling on it. It waits for all queued threads to initialize themselves and “join” the scheduler. The FRS begins dispatching the threads in the specified sequence during each frame interval. Errors are monitored (such as a thread that fails to complete its work within its frame) and a specified action is taken when an error occurs. Typically the error action is to send a signal to the controller thread.

## Preparing the System

Before a real-time program executes, you must set up the system in the following ways:

1. Choose the CPU or CPUs that the real-time program will use. CPU 0 (at least) must be reserved for IRIX system functions.

2. Decide which CPUs will handle I/O interrupts. By default, IRIX distributes I/O interrupts across all available processors as a means of balancing the load (referred to as spraying interrupts). CPUs that are used for real-time programs should be removed from the distribution set (see “Redirecting Interrupts” in Chapter 3).

3. If using an external interrupt as a time base, make sure it is redirected to the CPU of the master FRS (see “External Interrupts”).

4. Make sure that none of the real-time CPUs is managing the clock (see “Assigning the Clock Processor” in Chapter 3). Normally the responsibility of handling 10ms scheduler interrupts is given to CPU 0.

Each Frame Scheduler takes care of restricting and isolating its CPU, so that the CPU is used only for threads scheduled by the Frame Scheduler.

## Implementing a Single Frame Scheduler

When the activities of your real-time program can be handled within a major frame interval by a single CPU, your program needs to create only one Frame Scheduler. Examples for implementing a single FRS can be found in the `simple` and `simple_pt` programs, described in Appendix A, “Sample Programs”.

Typically your program has a top-level process (called the controller thread) to handle startup and termination, and one or more activity threads that are dispatched by the Frame Scheduler. The activity threads are typically lightweight threads (pthreads or sprocs), but that is not a requirement—they can also be created with `fork`(); they need not be children of the controller thread. (See, for instance, “Example of Scheduling Separate Programs” in Appendix A.).

In general, these are the steps for setting up a single Frame Scheduler:

1. Initialize global resources such as memory-mapped segments, memory arenas, files, asynchronous I/O, semaphores, locks, and other resources.

2. Lock the shared address space segments. (When `fork`() is used, each child process must lock its own address space.)

3. If using pthreads, create a controller thread; otherwise, the initial thread of execution may be used as the controller thread.

   - Create a system scope attribute structure using `` `pthread_attr_init`() and `` `pthread_attr_setscope`(). See the `pthread_attr_init`(3P) and `pthread_attr_setscope`(3P) references pages for details.

   - Create a system scope controller thread using `pthread_create`() and the attribute structure you just set up. See `pthread_create`(3P) for details.

   - Exit the initial thread, since it cannot execute any FRS operations.

4. Create the Frame Scheduler using `frs_create_master`(), `frs_create_vmaster`(), or `frs_create`() (see the `frs_create`(3) man page for details).

5. Create the activity threads using one of the following interfaces (depending on the thread model being used):

   - pthread_create()

   - sproc()

   - fork()

6. Queue the activity threads on the target minor frame queues, using `frs_pthread_enqueue`() or `frs_enqueue`().

7. Optionally, initialize the Frame Scheduler signal handler to catch frame overrun, underrun, and activity dequeue events (see “Setting Frame Scheduler Signals” and “Setting Exception Policies”). The handlers are set at this time, after creation of the activity threads, so that the activity threads do not inherit them.

8. Use `frs_start`() (Table 4-1) to enable scheduling.

9. Have the activity threads call `frs_join`(). The Frame Scheduler begins scheduling processes as soon as all the activity threads have called `frs_join`().

10. Wait for error signals from the Frame Scheduler and for the termination of child processes.

11. Use `frs_destroy`() to terminate the Frame Scheduler.

12. Tidy up the global resources, as required.

## Implementing Synchronized Schedulers

When the real-time application requires the power of multiple CPUs, you must add one more level to the program design for a single CPU. The program creates multiple Frame Schedulers, one master and one or more synchronized slaves.

### Synchronized Scheduler Concepts

The first Frame Scheduler provides the time base for the others. It is called the master scheduler. The other schedulers take their time base interrupts from the master, and so are called slaves. The combination is called a sync group.

No single thread may create more than one Frame Scheduler. This is because every Frame Scheduler must have a unique FRS controller thread to which it can send signals. As a result, the program has three types of threads:

- A master controller thread that sets up global data and creates the master Frame Scheduler

- One slave controller thread for each slave Frame Scheduler

- Activity threads

The master Frame Scheduler must be created before any slave Frame Schedulers can be created. Slave Frame Schedulers must be specified to have the same time base and the same number of minor frames as the master.

Slave Frame Schedulers can be stopped and restarted independently. However, when any scheduler, master or slave, is destroyed, all are immediately destroyed.

### Implementing a Master Controller Thread

A variety of program designs are possible but the simplest is possibly the set of steps described in the following paragraphs.

The master controller thread performs these steps:

1. Initializes global resource. One global resource is the thread ID of the master controller thread.

2. Creates the master Frame Scheduler using either the `frs_create_master`() or `frs_create_vmaster`() call, and stores its handle in a global location.

3. Creates one slave controller thread for each synchronized CPU to be used.

4. Creates the activity threads that will be scheduled by the master Frame Scheduler and queues them to their assigned minor frames.

5. Sets up signal handlers for signals from the Frame Scheduler (see “Using Signals Under the Frame Scheduler”).

6. Uses `frs_start`() (Table 4-1) to tell the master Frame Scheduler that its activity threads are all queued and ready to commence scheduling.

   The master Frame Scheduler starts scheduling threads as soon as all threads have called `frs_join`() for their respective schedulers.

7. Waits for error signals.

8. Uses `frs_destroy`() to terminate the master Frame Scheduler.

9. Tidies up global resources as required.

### Implementing Slave Controller Threads

Each slave controller thread performs these steps:

1. Creates a synchronized Frame Scheduler using `frs_create_slave`(), specifying information about the master Frame Scheduler stored by the master controller thread. The master FRS must exist. A slave FRS must specify the same time base and number of minor frames as the master FRS.

2. Changes the Frame Scheduler signals or exception policy, if desired (see “Setting Frame Scheduler Signals” and “Setting Exception Policies”).

3. Creates the activity threads that are scheduled by this slave Frame Scheduler, and queues them to their assigned minor frames.

4. Sets up signal handlers for signals from the slave Frame Scheduler.

5. Use `frs_start`() to tell the slave Frame Scheduler that all activity threads have been queued.

   The slave Frame Scheduler notifies the master when all threads have called `frs_join()`. When the master Frame Scheduler starts broadcasting interrupts, scheduling begins.

6. Waits for error signals.

7. Uses `frs_destroy`() to terminate the slave Frame Scheduler.

For an example of this kind of program structure, refer to “Examples of Multiple Synchronized Schedulers” in Appendix A.

| Tip: In this design sketch, the knowledge of which activity threads to create, and on which frames to queue them, is distributed throughout the code of multiple threads, where it might be hard to maintain. However, it is possible to centralize the plan of schedulers, activities, and frames in one or more arrays that are statically initialized. This improves the maintainability of a complex program.

## Handling Frame Scheduler Exceptions

The FRS control thread for a scheduler controls the handling of the Overrun and Underrun exceptions. It can specify how these exceptions should be handled, and what signals the Frame Scheduler should send. These policies have to be set before the scheduler is started. While the scheduler is running, the FRS controller can query the number of exceptions that have occurred.

### Exception Types

The Overrun exception indicates that a thread failed to yield in a minor frame where it was expected to yield, and was preempted at the end of the frame. An Overrun exception indicates that an unknown amount of work that should have been done was not done, and will not be done until the next frame in which the overrunning thread is queued.

The Underrun exception indicates that a thread that should have started in a minor frame did not start. Possibly the thread has terminated. More likely it was blocked in some kind of wait because of an unexpected delay in I/O, or a deadlock on a lock or semaphore.

### Exception Handling Policies

The FRS control thread can establish one of four policies for handling overrun and underrun exceptions. When it detects an exception, the Frame Scheduler can do the following:

- Send a signal to the FRS controller

- Inject an additional minor frame

- Extend the frame by a specified number of microseconds

- Steal a specified number of microseconds from the following frame

The default action is to send a signal (the specific signals are listed under “Setting Frame Scheduler Signals”). The scheduler continues to run. The FRS control thread can then take action, for example, terminating the Frame Scheduler.

#### Injecting a Repeat Frame

The policy of injecting an additional minor frame can be used with any time base. The Frame Scheduler inserts another complete minor frame, essentially repeating the minor frame in which the exception occurred. In the case of an overrun, the activity threads that did not finish have another frame's worth of time to complete. In the case of an underrun, there is that much more time for the waiting thread to wake up. Because exactly one frame is inserted, all other threads remain synchronized to the time base.

#### Extending the Current Frame

The policies of extending the frame, either with more time or by stealing time from the next frame, are allowed only when the time base is an on-chip or high-resolution timer (see “Selecting a Time Base”).

When adding time, the current frame is made longer by a fixed amount of time. Since the minor frame becomes a variable length, it is possible for the Frame Scheduler to drop out of synch with an external device.

When stealing time from the following frame, the Frame Scheduler returns to the original time base at the end of the following minor frame—provided that the threads queued to that following frame can finish their work in a reduced amount of time. If they do not, the Frame Scheduler steals time from the next frame still.

#### Dealing With Multiple Exceptions

You decide how many consecutive exceptions are allowed within a single minor frame. After injecting, stretching, or stealing time that many times, the Frame Scheduler stops trying to recover, and sends a signal instead.

The count of exceptions is reset when a minor frame completes with no remaining exceptions.

### Setting Exception Policies

The `f rs_pthread_setattr`() or `f rs_setattr`() function is used to change exception policies. This function must be called before the Frame Scheduler is started. After scheduling has begun, an attempt to change the policies or signals is rejected.

In order to allow for future enhancements, `frs_pthread_setattr`() or `frs_setattr`() accepts arguments for minor frame number and thread ID; however it currently allows setting exception policies only for all policies and all minor frames. The most significant argument to it is the `frs_recv_info` structure, declared with these fields.

    typedef struct frs_recv_info {
        mfbe_rmode_t  rmode;      /* Basic recovery mode */
        mfbe_tmode_t  tmode;      /* Time expansion mode */
        uint          maxcerr;    /* Max consecutive errors */
        uint          xtime;      /* Recovery extension time */
    } frs_recv_info_t;

The recovery modes and other constants are declared in `/usr/include/sys/frs.h`. The function in Example 4-3 sets the policy of injecting a repeat frame. The caller specifies only the Frame Scheduler and the number of consecutive exceptions allowed.

Example 4-3. Function to Set INJECTFRAME Exception Policy

    int
    setInjectFrameMode(frs_t *frs, int consecErrs)
    {
      frs_recv_info_t work;
      bzero((void*)&work,sizeof(work));
      work.rmode = MFBERM_INJECTFRAME;
      work.maxcerr = consecErrs;
      return frs_setattr(frs,0,0,FRS_ATTR_RECOVERY,(void*)&work);
    }

The function in Example 4-4 sets the policy of stretching the current frame (a function to set the policy of stealing time from the next frame is nearly identical). The caller specifies the Frame Scheduler, the number of consecutive exceptions, and the stretch time in microseconds.

Example 4-4. Function to Set STRETCH Exception Policy

    int
    setStretchFrameMode(frs_t *frs,int consecErrs,uint microSecs)
    {
      frs_recv_info_t work;
      bzero((void*)&work,sizeof(work));
      work.rmode = MFBERM_EXTENDFRAME_STRETCH;
      work.tmode = EFT_FIXED; /* only choice available */
      work.maxcerr = consecErrs;
      work.xtime = microSecs;
      return frs_setattr(frs,0,0,FRS_ATTR_RECOVERY,(void*)&work);
    }

### Querying Counts of Exceptions

When you set a policy that permits exceptions, the FRS controller thread can query for counts of exceptions. This is done with a call to `` `frs_pthread_getattr`() or`` `frs_getattr`(), passing the handle to the Frame Scheduler, the number of the minor frame, and the thread ID of the thread within that frame.

The values returned in a structure of type `frs_overrun_info_t` are the counts of overrun and underrun exceptions incurred by that thread in that minor frame. In order to find the count of all overruns in a given minor frame, you must sum the counts for all threads queued to that frame. If a thread is queued to more than one minor frame, separate counts are kept for it in each frame.

The function in Example 4-5 takes a Frame Scheduler handle and a minor frame number. It gets the list of thread IDs queued to that minor frame, and returns the sum of all exceptions for all of them.

Example 4-5. Function to Return a Sum of Exception Counts (pthread Model)

    #define THE_MOST_TIDS 250
    int
    totalExcepts(frs_t * theFRS, int theMinor)
    {
        int numTids = frs_getqueuelen(theFRS, theMinor);
        int j, sum;
        pthread_t allTids[THE_MOST_TIDS];
        if ( (numTids <= 0) || (numTids > THE_MOST_TIDS) )
            return 0; /* invalid minor #, or no threads queued? */

        if (frs_pthread_readqueue(theFRS, theMinor, allTids) == -1)
            return 0; /* unexpected problem with reading IDs */

        for (sum = j = 0; j<numTids; ++j)
        {
            frs_overrun_info_t work;
            frs_pthread_getattr(theFRS     /* the scheduler */
                        theMinor,          /* the minor frame */
                        allTids[j],        /* the threads */
                        FRS_ATTR_OVERRUNS, /* want counts */
                        &work);            /* put them here */
            sum += (work.overruns + work.underruns);
        }
        return sum;
    }

| Tip: The FRS read queue functions return the number of threads present on the queue at the time of the read. Applications can use this returned value to eliminate calls to `frs_getqueuelen`().

## Using Signals Under the Frame Scheduler

The Frame Scheduler itself sends signals to the threads using it. And threads can communicate by sending signals to each other. In brief, an FRS sends signals to indicate the following:

- The FRS has been terminated

- Overrun or underrun have been detected

- A thread has been dequeued

The rest of this topic details how to specify the signal numbers and how to handle the signals.

### Signal Delivery and Latency

When a process is scheduled by the IRIX kernel, it receives a pending signal the next time the process exits from the kernel domain. For most signals, this could occur under the following conditions:

- When the process is dispatched after a wait or preemption

- Upon return from some system call

- Upon return from the kernel's usual 10-millisecond tick interrupt

(`SIGALRM` is delivered as soon as the kernel is ready to return to user processing after the timer interrupt, in order to preserve timer accuracy.) Thus, for a process that is ready to run, in a CPU that has not been made nonpreemptive, normal signal latency is at most 10 milliseconds, and `SIGALARM` latency is less. However, when the receiving process is not ready to run, or when there are competing processes with higher priorities, the delivery of a signal is delayed until the next time the receiving process is scheduled.

When the CPU is nonpreemptive (see “Making a CPU Nonpreemptive” in Chapter 3), there are no clock tick interrupts, so signals can only be delivered following a system call.

Signal latency can be greater when running under the Frame Scheduler. Like the normal IRIX scheduler, the Frame Scheduler delivers pending signals to a process when it next returns to the process from the kernel domain. This can occur under the following conditions:

- When the process is dispatched at the start of a minor frame where it is queued

- Upon return from some system call

The upper bound on signal latency in this case is the interval between the minor frames to which that process is queued. If the process is scheduled only once in a major frame, it might not receive a signal until a full major frame interval after the signal is sent.

### Handling Signals in the FRS Controller

When a Frame Scheduler detects an Overrun or Underrun exception that it cannot recover from, and when it is ready to terminate, it sends a signal to the FRS controller.

| Tip: Child processes inherit signal handlers from the parent, so a parent should not set up handlers prior to `sproc`() or `fork`() unless they are meant to be inherited.

The FRS controller for a synchronized Frame Scheduler should have handlers for Underrun and Overrun signals. The handler could report the error and issue `frs_destroy`() to shut down its scheduler. An FRS controller for a synchronized scheduler should use the default action for `SIGHUP` (Exit) so that completion of the `frs_destroy`() quietly terminates the FRS controller.

The FRS controller for the master (or only) Frame Scheduler should catch Underrun and Overrun exceptions, report them, and shut down its scheduler.

When an FRS is terminated with `frs_destroy`(), it sends `SIGKILL` to its FRS controller. This cannot be changed; and `SIGKILL` cannot be handled. Hence `frs_destroy`() is equivalent to termination for the FRS controller.

### Handling Signals in an Activity Thread

A Frame Scheduler can send a signal to an activity thread when the thread is removed from any queue using `frs_pthread_remove`() or `frs_premove`() (see “Managing Activity Threads”). The scheduler can also send a signal to an activity thread when it is removed from the last or only minor frame to which it was queued (at which time a thread is returned to normal IRIX scheduling).

In order to have these signals sent, the FRS controller must set nonzero signal numbers for them, as discussed in the following topic, “Setting Frame Scheduler Signals”.

### Setting Frame Scheduler Signals

The Frame Scheduler sends signals to the FRS controller.

| Note: In earlier versions of REACT/pro, the Frame Scheduler sent these signals to all processes queued to that Frame Scheduler as well as the FRS controller. That is no longer the case. You can remove signal handlers for these signals from activity processes, if they exist.

The signal numbers used for most events can be modified. Signal numbers can be queried using `frs_pthread_getattr`(`FRS_ATTR_SIGNALS`) or `frs_getattr`(`FRS_ATTR_SIGNALS`) and changed using `frs_pthread_setattr`(`FRS_ATTR_SIGNALS`) or `frs_setattr`(`FRS_ATTR_SIGNALS`), in each case passing an `frs_signal_info` structure. This structure contains room for four signal numbers, as shown in Table 4-3.

Table 4-3. Signal Numbers Passed in frs_signal_info_t

Field Name | Signal Purpose | Default Signal Number
--- | --- | ---
`sig_underrun` | Notify FRS controller of Underrun. | `SIGUSR1`
`sig_overrun` | Notify FRS controller of Overrun. | `SIGUSR2`
`sig_dequeue` | Notify an activity thread that it has been dequeued with `frs_pthread_remove()` or `frs_premove`(). | 0 (do not send)
`sig_unframesched` | Notify an activity thread that it has been removed from the last or only queue in which it was queued. | `SIGRTMIN`

Signal numbers must be changed before the Frame Scheduler is started. All the numbers must be specified to `frs_pthread_setattr`() or `frs_setattr`(), so the proper way to set any number is to first file the `frs_signal_info_t` using `frs_pthread_getattr`() or `frs_getattr`(). The function in Example 4-6 sets the signal numbers for Overrun and Underrun from its arguments.

Example 4-6. Function to Set Frame Scheduler Signals

    int
    setUnderOverSignals(frs_t *frs, int underSig, int overSig)
    {
      int error;
      frs_signal_info_t work;
      error = frs_pthread_getattr(frs,0,0,FRS_ATTR_SIGNALS,(void*)&work);
      if (!error)
      {
        work.sig_underrun = underSig;
        work.sig_overrun = overSig;
        error = frs_pthread_setattr(frs,0,0,FRS_ATTR_SIGNALS,(void*)&work);
      }
      return error;
    }

### Sequence Error

When `frs_create_vmaster`(`)` is used to create a frame scheduler triggered by multiple interrupt sources, a sequence error signal is dispatched to the controller thread if the interrupts come in out of order. For example, if the first and second minor frame interrupt sources are different, and the second minor frame's interrupt source is triggered before the first minor frame's interrupt source, then a sequence error has occurred.

This type of error condition is indicative of unrealistic time constraints defined by the interrupt information template.

The signal code that represents the occurrence of a sequence error is `SIGRTMIN+1`. This signal cannot be reset or disabled using the `frs_setattr`() interface.

## Using Timers with the Frame Scheduler

In general, interval timers and the Frame Scheduler do not mix. The expiration of an interval is marked by a signal. However, signal delivery to an activity thread can be delayed (see “Signal Delivery and Latency”), so timer latency is unpredictable.

The FRS controller, because it is scheduled by IRIX, not the Frame Scheduler, can use interval timers.

Example 4-7. Minimal Activity Process as a Timer

    frs_join(scheduler-handle)
    do {
       usvsema(frs-controller-wait-semaphore);
       frs_yield();
    } while(1);
    _exit();

## FRS Kernel-Level Device Driver Interface

The Frame Scheduler provides a device driver interface to allow any device with a kernel-level device driver to generate the time-base interrupt. As many as eight different device drivers can support the Frame Scheduler in any one system. The Frame Scheduler distinguishes device drivers by an ID number in the range 0 through 7 that is coded into each driver.

| Note: The structure of an IRIX kernel-level device driver is discussed in the IRIX Device Driver Programming Guide (see “Related Publications and Sites”). The generation of time-base signals can be added as a minor enhancement to a existing device driver.

In order to interact with the Frame Scheduler, a driver provides two routines, one for initialization and one for termination, which it exports during driver initialization. After a master Frame Scheduler has initialized a device driver, the driver calls a Frame Scheduler entry point to signal the occurrence of each interrupt.

### Device Driver Overview

The following sequence of actions occurs when a device driver is used as a source of time-base interrupts for the Frame Scheduler.

1. During its initialization in the pfx `start`() or pfx `init`() entry point, the driver calls a kernel function to specify its unique driver identifier between 0 and 7, and to register its pfx`_frs_func_set`() and pfx`_frs_func_clear`() functions. After this has been done, the Frame Scheduler is aware of the existence of this driver and allows programs to request it as the source of interrupts.

2. Later, a real-time program creates a master Frame Scheduler and specifies this driver by its number as the source of interrupts (see “Device Driver Interrupt”). The Frame Scheduler calls the pfx`_frs_func_set`() registered by this particular driver. This tells the driver that time signals are needed.

3. The device driver calls `frs_handle_driverintr`() each time its interrupt handling routine is entered. This informs the Frame Scheduler that an interrupt has been received.

4. When the Frame Scheduler is being terminated, it invokes pfx`_frs_func_clear`() for the driver it is using. This tells the driver that time signals are no longer needed, and to cease calling `frs_handle_driverintr`() until it is once again initialized by a Frame Scheduler.

Device driver names, device driver structure, configuration files, and related topics are covered in the IRIX Device Driver Programming Guide.

### Registering the Initialization and Termination Functions

A device driver must register two interface functions to make them known to the Frame Scheduler. This call, which occurs during the device driver's own initialization, also makes the driver known as a source of time-base interrupts:

    frs_driver_export( int frs_driver_id,
                      void (*frs_func_set)(intrgroup_t*),
                      void (*frs_func_clear)(void));

The parameter `frs_driver_id` is the driver's identification number. A real-time program specifies the same number to `frs_create_master`() or `frs_create`() to select this driver as the source of interrupts. The identifier is an integer between 0 and 7. Different drivers in the same system must use different identifiers. A typical call resembles the code in Example 4-8.

Example 4-8. Exporting Device Driver Entry Points

    /*
    ** Function called by the example driver to export
    ** its Frame Scheduler interface functions.
    */
    frs_driver_export(3, example_frs_func_set, example_frs_func_clear);

### Frame Scheduler Initialization Function

The device driver must provide a function with the following prototype:

    void pfx_frs_func_set ( intrgroup_t* intrgroup ) ;

A skeleton of an initialization function for a Challenge or Onyx system running under IRIX 6.2 is shown in Example 4-9. The function is called by a new master Frame Scheduler—one that is created with an interrupt source parameter of `FRS_INTRSOURCE_DRIVER` and an interrupt qualifier specifying this device driver's number (see “Device Driver Interrupt”). A device driver is used by only one Frame Scheduler at a time.

The argument intrgroup is passed by the Frame Scheduler to identify the interrupt group it has allocated. A VME device driver must set the hardware devices it manages so that interrupts are directed to this interrupt group. The actual group identifier may be obtained using the macro:

    intrgroup_get_groupid(intrgroup)

The effective destination may be obtained using the following macro:

    EVINTR_GROUPDEST(intrgroup_get_groupid(intrgroup))

Example 4-9. Device Driver Initialization Function

    /*
    ** Frame Scheduler initialization function
    ** for the External Interrupts Driver
    */
    int FRS_is_active = 0;
    int FRS_vme_install = 0;
    void
    example_frs_func_set(intrgroup_t* intrgroup)
    {
       int s;
       ASSERT(intrgroup != 0);
       /*
       ** Step 1 (VME only):
       ** In a VME device driver, set up the hardware to send
       ** the interrupt to the appropriate destination.
       ** This is done with vme_frs_install() which takes:
       ** * (int) the VME adapter number
       ** * (int) the VME IPL level
       ** * the intrgroup as passed to this function.
       */
       FRS_vme_install = vme_frs_install(
          my_edt.e_adap, /* edt struct from example_edtinit */
          ((vme_intrs_t *)my_edt.e_bus_info)->v_brl,
          intrgroup);
       /*
       ** Step 2: any hardware initialization required.
       */
       /*
       ** Step 3: note that we are now in use.
       */
       FRS_is_active = 1;
    }

Only VME device drivers on the CHALLENGE/Onyx need to call `vme_frs_install`() — do not call it on Origin systems. As suggested by the code in Example 4-9, the arguments to `vme_frs_install`() can be taken from data supplied at boot time to the device driver's pfx `edtinit`() function:

- The adapter number is in the `edt.e_adap` field

- The configured interrupt priority level is in the `vme_intrs.v_brl` addressed by the `edt.e_bus_info` field

The pfx `edtinit`() entry point is documented in the IRIX Device Driver Programming Guide.

| Tip: The `vme_frs_install`() function is a dynamic version of the VECTOR configuration statement. You are not required to use the IPL value from the configuration file.

### Frame Scheduler Termination Function

The device driver must provide a function with the following prototype:

    void prfx_frs_func_clear ( void ) ;

A skeleton for this function is shown in Example 4-10. The Frame Scheduler that initialized a device driver calls this function when the Frame Scheduler is terminating. The Frame Scheduler deallocates the interrupt group to which interrupts were directed.

The device driver should clean up data structures and make sure that the device is in a safe state. A VME device driver must call `vme_frs_uninstall`().

Example 4-10. Device Driver Termination Function

    /*
    ** Frame Scheduler termination function
    */
    void
    example_frs_func_clear(void)
    {
       /*
       ** Step 1: any hardware steps to quiesce the device.
       */

       /*
       ** Step 2 (VME only):
       ** Break the link between interrupts and the interrupt
       ** group by calling vme_frs_uninstall() passing:
       ** * (int) the VME adapter number
       ** * (int) the VME IPL level
       ** * the value returned by vme_frs_install()
       */
       vme_frs_uninstall(
          my_edt.e_adap, /* edt struct from example_edtinit */
          ((vme_intrs_t *)my_edt.e_bus_info)->v_brl,
          FRS_vme_install);
       /*
       ** Step 3: note we are no longer in use.
       */
       FRS_is_active = 0;
    }

### Generating Interrupts

A driver has to call the Frame Scheduler interrupt handler from within the driver's nonthreaded interrupt handler using code similar to that shown in Example 4-11. It delivers the interrupt to the Frame Scheduler on that CPU. The function to be invoked is

    void frs_handle_driverintr(void);

Example 4-11. Generating an Interrupt From a Device Driver

    void example_intr()
    {
       /*
       ** Step 1: anything required by the hardware
       */
       /*
       ** Step 2: if connected to the Frame Scheduler, send
       ** an interrupt to it. Flag FRS_is_active is set in
       **

Example 4-9 and cleared in Example 4-10.

       */
       if (FRS_is_active) frs_handle_driverintr();
       /*
       ** Step 3: any additional processing needed.
       */
       return;
    }

It is possible for an interrupt handler to be entered at a time when the Frame Scheduler for its processor is not active; that is, after `frs_destroy`() has been called and before the driver termination function has been entered. The `frs_handle_driverintr`() function checks for this and does nothing when nothing is required.

The call to `frs_handle_driverintr`() must be executed on a CPU controlled by the FRS that is using the driver. The only way to ensure this is to ensure that the hardware interrupt used by this driver is directed to that CPU. In IRIX 6.4 and later, you direct a hardware interrupt to a particular CPU by placing a `DEVICE_ADMIN` directive in the file `/var/sysgen/system/irix.sm`. See comments in that file for the syntax.

Threaded interrupt handlers experience problems when run on a CPU with an FRS. If the physical interrupt is not directed to the CPU, the thread is forced to run elsewhere. If the interrupt is directed, the thread does not get a chance to run until the FRS exits. The FRS schedules only the user threads that are registered with it on its CPUs; it ignores kernel and driver threads bound to those CPUs. Currently, by default, all VME and PCI drivers for IP27 and IP35 systems are threaded.

When writing drivers to be used with the FRS, you should use nonthreaded interrupt handlers. Nonthreaded handlers run out of the interrupt stack and therefore, do not need to be scheduled by the CPU, as do threaded handlers. However, the physical interrupt must still be routed to the desired CPU.

SGI's response time guarantee is not honored when used with nonthreaded interrupt handlers because IRIX has little control over them. Therefore, authors of such drivers should be careful not to spend too much execution time within their handlers. Nonthreaded handlers must not attempt to take any form of blocking lock or call any function that might block.

Example 4-12 shows how to modify the standard VME interrupt registration to use nonthreaded interrupts. The code is based on the sample VME driver found in the IRIX Device Driver Programmer's Guide, chapter 13, in the section titled “Sample VME Device Driver.”

Example 4-12. Registering a Nonthreaded Interrupt Handler

    ...
    #include <sys/iobus.h>
    ...

    int
    rfm_edtinit(edt_t * e)
    {
            ...

            /*
             * Get the device descriptor for our device
             */
            example_dev_desc = device_desc_dup(conn);

            /*
             * Set the non-threaded flag on the device descriptor
             */
            device_desc_flags_set(example_dev_desc,
                   (device_desc_flags_get(conn) | D_INTR_NOTHREAD));

            /*
             * Allocate the interrupt for the device
             */
            intr = vmeio_intr_alloc(conn, example_dev_desc,
                                    ivec, ilev, rfm, 0);

            ...
    }

Prev | Table of Contents | Next
Chapter 3. Controlling CPU Workload |  | Chapter 5. Optimizing Disk I/O for a Real-Time Program