Chapter 12. Makefile Optimization

This chapter presents guidelines for makefile construction, which will help you to make best use of clearmake.

Pathnames in Build Scripts

ClearCase extends the standard UNIX file system, allowing you to use both view-extended and version-extended pathnames:

/view/gamma/usr/hw/src/msg.c         (view-extended pathname)
msg.c@@/main/REL3                 (version-extended pathname)

You can use such pathnames with cleartool and with standard operating system programs. But such names do not work in makefile build scripts — use standard pathnames only.

Declaring Source Dependencies in Makefiles

To implement build avoidance based on files' time-modified stamps, standard make variants require you to declare all the source file dependencies of each build

target. For example, object module hello.o might depend on source files hello.c and hello.h in the same directory:

hello.o: hello.c hello.h
      rm -f hello.o
      cc -c hello.o

Typically, these source files depend on project-specific header files through #include directives, perhaps nested within one another. The standard UNIX files do not change very often, but it is a common programmer's lament that “it didn't compile because someone changed the project's header files without telling me”.

To alleviate this problem, some organizations include every header file dependency in their makefiles. They rely on utility programs (for example, makedepend) to read the source files and determine the dependencies.

clearmake does not require that source-file dependencies be declared in makefiles (but see the next section). The first time a derived object is built, its build script is always executed — thus, the dependency declarations are irrelevant. On rebuilds of a derived object, its configuration record provides a complete list of source-file dependencies, including those on header files.

You can leave source-file dependency declarations in your existing makefiles, but you need not update them as you revise the makefiles. And you need not place source-file dependencies in new makefiles to be used with clearmake.


Note: Even though dependency declarations are not required, you may want to include them in your makefiles, anyway. The principal reason for doing so is portability — you may need to provide your sources to another group (or another company) that is not using ClearCase.


Explicitly-Declared Source Dependencies

ClearCase's automatic build auditing facility tracks only the MVFS objects actually used in the building of a target. Sometimes, however, you may wish to track other objects:

  • the version of a compiler, which is not stored in a VOB

  • the version of the operating system kernel, which is not referenced at all during the build

  • the state of a flag-file, possibly a non-MVFS file, used to force rebuilds

You can force such objects to be recorded in a build's CR as shown in Figure 12-1 by declaring them as dependencies of the makefile target:

Figure 12-1. Explicitly-Declared Source Dependencies


We suggest that you use view-private files as flag files, rather than using non-MVFS files (such as /tmp/flag). In a distributed build, a view-private flag file is guaranteed to be the same object on all hosts; there is no such guarantee for a non-MVFS file.

As an alternative to declaring your C compiler as a build dependency, you might place it (and other tools) in a “tools VOB”. The versions of such tools will automatically be recorded, eliminating the need for explicit dependency declarations. Additional issues in the auditing of build tools are discussed in the next section.

Explicit Dependencies on `Searched-For' Sources

There are situations in which clearmake's configuration lookup algorithm qualifies a derived object, even though an actual target rebuild would produce a different result. Configuration lookup requires that for each object listed in an existing CR, the current view must select the same version of that object. It does not take into account the possibility that a target rebuild might use a different object altogether.

For files that are accessed by explicit pathnames, this situation cannot occur. But it can occur if a file is accessed at build time by a search through multiple directories. For example, the following build script uses a search to locate a library file, libprojutil.a:

hello:
      cc -o hello -L /usr/project/lib -L /usr/local/lib \
            main.o util.o -lprojutil

The command clearmake hello might qualify an existing derived object built with /usr/local/lib/libprojutil.a, even though performing a target rebuild would now use /usr/project/lib/libprojutil.a instead.

clearmake addresses this problem in the same way as some standard make implementations:

  • You must declare the searched-for source object as an explicit dependency in the makefile:

    hello: libprojutil.a
    	...
    

You must use the VPATH macro to specify the set of directories to be searched:

VPATH = /usr/project/lib:/usr/local/lib

Given this makefile, clearmake will use the VPATH (if any) when it performs configuration lookup on libprojutil.a. If a candidate derived object was built with /usr/local/lib/projutil.a, but would be built with /usr/project/lib/projutil.a in the current view, the candidate is rejected.


Note: The VPATH macro is not used for all source dependencies listed in the config rec. It is used only for explicitly-declared dependencies of the target.

Build Tool Dependencies. You can use this mechanism to implement dependencies on build tools. For example, you might track the version of the C compiler used in a build as follows:

msg.o: msg.c $(CC)
       $(CC) -c msg.c

With this makefile, either your VPATH must include the directories on your search path (if the $(CC) value is simply “cc”), or you must use a full pathname as the $(CC) value.

Build-Order Dependencies

In addition to source dependencies, makefiles also contain build-order dependencies. For example:

hello: hello.o libhello.a
      ...
libhello.a: hello_env.o hello_time.o
      ...

These dependencies are buildable objects, and thus are termed subtargets. Executable hello must be built after its subtargets, object module hello.o and library libhello.a, and the library must be built after its subtargets, object modules hello_env.o and hello_time.o.

ClearCase does not automatically detect build-order dependencies; you must include such dependencies in makefiles used with clearmake, just as with other make variants.

Build Sessions, Subsessions, and Hierarchical Builds


Note: Throughout this section, references to “clearmake” should more precisely be “clearmake or clearaudit”. See the clearaudit manual page for more on non-makefile-based building of software.

The following terms are useful in describing the details of ClearCase build auditing:

  • A “top-level” invocation of clearmake starts a build session. The time at which the build session begins becomes the build reference time for the entire build session, as described on “Continuing to Work During a Build / Reference Time”.

  • During a build session, one or more target rebuilds typically take place.

  • Each target rebuild involves the execution of one or more build scripts. (A double-colon target can have multiple build scripts.)

  • During each target rebuild, clearmake conducts a build audit.

Subsessions

A build session can have any number of subsessions, all of which inherit the reference time of the build session. A subsession corresponds to a “nested build” or “recursive make”, started when a clearmake process is invoked in the process family of a higher-level clearmake. Examples of clearmake invocations that start subsessions include:

  • including a clearmake command in a makefile build script executed by clearmake

  • entering a clearmake command in an interactive process started by clearaudit

A subsession begins while a higher-level session is still conducting build audits. The subsession conducts its own build audit(s), independent of the audits of the higher-level session — that is, the audits are not nested or related in any way, other than that they share the same build reference time.

Versions of Elements Created During a Build Session

Any version created during a build session and selected by a LATEST config spec rule will not be visible in that build session. For example, a build might checkin a derived object it has created; subsequent commands in the same build session will not “see” the checked-in version, unless it is selected by a config spec rule that does not involve the version label LATEST.

Coordinating Reference Times of Several Builds

Different build sessions have different reference times. The “best” way to have a series of builds share the same reference time is to structure them as a single, hierarchical build.

An alternative approach is to run all the builds within the same clearaudit session. For example, you might write a shell script, multi_make, that includes several invocations of clearmake (along with other commands). Running the script as follows ensures that all the clearmake builds will be subsessions that share the same reference time:

% clearaudit -c multi_make

Objects Written at More than One Level

Undesirable results occur when the same file is written at two or more session levels (for example, a top-level build session and a subsession): the build audit for the higher-level session does not contain complete information about the file system operations that affected the file. For example:

% clearaudit -c "clearmake shuffle > logfile"

The file logfile may be written both:

  • during the clearaudit build session, by the shell program invoked from clearaudit

  • during the clearmake subsession, when the clearaudit build session is suspended

In this case, clearaudit issues this error message:

Unable to create derived object "logfile"

To work around this limitation, “postprocess” the derived object at the higher level with a copy command:

% clearaudit -c "clearmake shuffle > log.tmp"
% cp log.tmp logfile
% rm log.tmp

No Automatic Creation of Configuration Record Hierarchy

CRs created during a build session and its subsessions are not automatically linked into a single configuration record hierarchy. For more information, see “CR Hierarchies”.

Incremental Updating of Derived Objects

The design of ClearCase's build auditing capability makes it ideal for use with tools that build derived objects “from scratch”. Since such newly-created objects have no “history”, ClearCase can learn everything it needs to know at build time. But this reliance on build-time file-system-level auditing causes ClearCase to record incomplete information for incrementally-updated objects, which do have a history.

From ClearCase's perspective, incremental updating means that an object is partially updated during the builds of multiple makefile targets, instead of being completely generated by the build of one target. clearmake does not incrementally update an existing CR when it builds a target. Instead:

  • Each time a build script incrementally updates an object's file system data, clearmake writes a completely new CR, which describes only the most recent update, not the entire build history.

  • The new CR does not match the desired build configuration for any of the other targets that incrementally update the object.

This results in a situation that is both unstable and incorrect: all incremental-update targets will be rebuilt each time that clearmake is invoked; when it's done, the DO will have the correct file system data, but its CR will not accurately describe the DO's configuration.

Example: Building an Archive

A common incremental-update scenario in “traditional” UNIX environments is the building of an archive (programming library) by ar(1).

A traditional make program treats an archive as a compound object; it can examine the time-modified stamps of the individual components (object modules) in the archive; and it can update the archive by replacing one or more individual object modules. Here is a simple makefile in which a special syntax enables multiple targets to incrementally update a single archive, libvg.a:

libvg.a:: libvg.a(base.o)
libvg.a:: libvg.a(in.o)
libvg.a:: libvg.a(out.o)

If you edit one of the library's sources (for example, out.c); a “traditional” make program uses the special syntax and a .c.a built-in rule to update the library as follows:

  • It looks inside the archive libvg.a, and determines that it includes an out.o that is older than its source file.

  • It compiles a new out.o from out.c.

  • It uses ar to incrementally update libvg.a, replacing the old instance of object module out.o with the newly-built instance.

clearmake does not implement this algorithm, and includes no support for treating an archive as a compound object. ClearCase build-avoidance is based solely on meta-data (CRs), not on any analysis at the file-system-data level. clearmake interprets the above makefile as follows

  • It considers all the libvg.a(...) dependencies to be multiple instances of the same “double-colon” build target.

  • Accordingly, whenever one of those “double-colon” targets requires rebuilding, it rebuilds them all, using the standard .c.a built-in rule. This effectively rebuilds the entire archive libvg.a from scratch.

Thus, clearmake accepts the standard incremental-update syntax, but interprets it in a way that produces a non-incremental build procedure.

Remedies for the Incremental-Update Problem

Some makefile restructuring can ameliorate the situation described above. Often, a restructured build procedure can take advantage of wink-in to compensate for the loss of incremental updating. For example, you might revise the procedure for building the archive libvg.a (discussed in the preceding section) to dispense with the special ar-informed syntax:

libvega.a: base.o in.o out.o
       ar rv libvega.a base.o in.o out.o
base.o:
       cc -c base.c
in.o:
       cc -c in.c
out.o:
       cc -c out.c

Object modules built by this makefile are standard, sharable derived objects; typically, as they libraries sources stabilize over time, most builds of target libvega.a will reuse or wink-in most of the object modules.

Avoid the following restructuring; it will cause a complete rebuild of the archive each time any object module is updated:

base.o: base.c 
       cc -c base.c
       ar rv libvg.a base.o
  .
  . and so on

Additional Incremental-Update Situations

You may encounter incremental updating in other situations, as well. For example, C++ compilers that support parameterized types (templates) often update type map files incrementally as different targets are built. ClearCase includes special makefile rules that store per-target type map files.

Ada compilers often update certain common files in Ada libraries incrementally, as different compilation units are built. There are no current clearmake workarounds to implement per-target CRs for Ada libraries. To produce a CR for an Ada library, you can perform a complete rebuild of the library from its sources in a single clearaudit session.

Build Auditing and Background Processes

The ClearCase build programs — clearmake, clearaudit, and abe — all use the same procedure to produce configuration records:

  1. Sends a request to the host's multiversion file system (MVFS), initiating build auditing.

  2. Invoke one or more child processes (typically, shell processes), in which makefile build scripts or other commands are executed.

  3. Turn off MVFS building auditing.

  4. If all the subprocesses have indicated success by returning a zero exit status, and at least one MVFS file has been created, compute and store one or more configuration records.

Any subprocesses of the child processes invoked in Step #2 inherit the same MVFS build audit. (Recursive invocations of ClearCase build programs conduct their own, independent audits — see <Emphasis>Build Sessions, Subsessions, and Hierarchical Builds.)

A problem can occur if a build script (or other audited command) invokes a background subprocess, and exits without waiting for it to complete. The build program has no knowledge of the background process; it may proceed to Steps #3 and #4 before the background process has finished its work. In such situations, ClearCase cannot guarantee what portion, if any, of the actions of background commands will be reflected in the resulting CR — it depends on system scheduling and timing behavior. Thus, you should strictly avoid using background processes in audited build scripts.