This chapter is for users who wish to learn more about how clearmake accomplishes its work, and to manipulate ClearCase's build management data structures.
In a parallel-development environment, it is likely that many DOs with the same pathname will exist at the same time. For example, suppose that source file msg.c is being developed on three branches concurrently, in three different views. ClearCase builds performed in those three views produce object modules named msg.o. Each of these is a DO, and each has the same standard pathname, say /vobs/proj/src/msg.o.
In addition, each DO can be accessed with ClearCase extended names:
Within each view, a standard UNIX pathname accesses the DO built in that view. This is another example of ClearCase's transparency feature.
msg.o (the derived object in the current view)
You can use a view-extended pathname to access a DO in any view:
/view/drp/vobs/proj/src/msg.o (the derived object in view `drp') /view/R2_integ/vobs/proj/src/msg.o (the derived object in view `R2_integ')Each derived object is cataloged in the VOB database with a unique identifier, it DO-ID, which references it independently of views. The lsdo (“list derived objects”) command can list all DOs created at a specified pathname, regardless of which views (if any) can “see” them:
% cleartool lsdo hello 07-May.16:09 akp "msg.o@@07-May.16:09.623" on neptune 06-May.12:47 akp "msg.o@@06-May.12:47.539" on neptune 01-May.21:49 akp "msg.o@@01-May.21:49.282" on neptune 03-Apr.21:40 akp "msg.o@@01-May.21:40.226" on neptune
Together, a DO's standard name (msg.o) and its DO-ID (07-May.16:09.623) constitute a VOB-extended pathname to that particular derived object. (The extended naming symbol is host-specific; most organizations use the default value, @@.)
Standard software must access a DO through a view, using a standard pathname or view-extended pathname. You can use such names with debuggers, profilers, rm, tar, and so on. Only ClearCase programs can reference a DO using a VOB-extended pathname, and only the DO's meta-data is accessible in this way (see Figure 11-1):
Exception: you can use a VOB-extended pathname with the winkin command, to copy the file system data of any DO into your view. See “Wink-In without Configuration Lookup / The `winkin' Command”.
The following sections discuss how you can use the configuration records with which ClearCase keeps track of builds.
The catcr command displays the configuration record (CR) of a specified derived object (DO). Figure 11-2 shows a CR, with annotations to indicate the various kinds of information in the listing.
Some notes on Figure 11-2:
Directory versions — By default, catcr does not list versions of the VOB directories involved in a build. To list this information, use the - long option:
% cleartool catcr -long util.o directory version /usr/hw/.@@/main/1 <25-Feb-94.16:59:31> directory version /usr/hw/src@@/main/3 <26-Feb-94.20:53:07>Declared dependencies — One of ClearCase's principal features is the automatic detection of source dependencies on MVFS files: versions of elements and objects in view-private storage. In addition, a CR includes non-MVFS objects that are explicitly declared as dependencies in the makefile. Figure 11-2 shows one such declared dependency, on file build.notes.1, located in the non-VOB directory /tmp.
Listing of checked-out versions — Checked-out versions of file elements are highlighted (for example, with boldface or reverse-video). Checked-out versions of directory elements are listed like this:
directory version /usr/hw/src@@/main/CHECKEDOUT <26-Feb...When the elements are subsequently checked in, a listing of the same configuration record shows the updated information. For example,
/usr/hw/src/util.c <25-Feb-94.17:02:27>
... might become ...
/usr/hw/src/util.c@@/main/4 <25-Feb-94.17:02:27>
The actual configuration record contains a ClearCase-internal identifier for each MVFS object. After checkin converts the checked-out version object to a “real” version, catcr changes the way it lists that object.
When deciding whether or not to rebuild a target, clearmake compares one or more CRs of existing DOs with your view's current build configuration. You can use the diffcr command to compare two existing CRs. This command is a valuable diagnostic tool in a parallel development environment.
For example, suppose that a new build of a program dumps core; you have no idea what changed from the preceding build, which ran correctly. A comparison of the CRs might show that the culprit is a change to one header file:
% cleartool diffcr -flat hello@@07-Mar.11:40.217 hello@@08-Mar.12:48.265 ---------------------------- MVFS objects: ---------------------------- . . < First seen in target "hello.o" < 2 /usr/hw/src/hello.h@@/main/5 <20-Mar-94.14:46:00> > First seen in target "hello.o" > 2 /usr/hw/src/hello.h <28-Mar-94.12:48:12> . . |
To get the build working again, you could fix the header file; alternatively, you could isolate yourself from this change by reconfiguring your view to select the old version of the header file. (See “Using Config Spec `Time Rules' to Increase Your View's Isolation”.)
Makefile-based builds of large software systems are almost always hierarchical: you explicitly request a build of a goal target; as necessary, this recursively invokes builds of one or more levels of subtargets. Thus, a single invocation of clearmake can involve the execution of many build scripts. Each execution of a build script is recorded in a separate CR; the result is a CR hierarchy that mirrors the structure of the makefile (Figure 11-3).
The catcr and diffcr commands have options for handling CR hierarchies:
By default, these commands compare individual CRs.
With the - recurse option, they process the entire CR hierarchy of each specified derived object, keeping the individual CRs separate.
With the - flat option, they combine (“flatten”) the CR hierarchy of each specified derived object.
Some ClearCase features automatically process entire CR hierarchies. For example, when the mklabel command attaches version labels to all versions used to build a particular derived object (mklabel -config), it uses the entire CR hierarchy of the specified DO. Similarly, ClearCase maintenance procedures are careful not to “scrub” the CR associated with a deleted DO if it is a member of the CR hierarchy of a higher-level DO.
An individual “parent-child” link in a CR hierarchy is established in either of these ways:
In a target/dependencies line — For example, the following target/dependencies line declares derived objects hello.o, msg.o, and libhello.a to be build dependencies of derived object hello:
hello: hello.o msg.o libhello.a ...
Accordingly, the CR for hello is the parent of the CRs for the . o files and the . a file.
In a build script — For example, in the following build script, derived object libhello.a in another directory is referenced in the build script for derived object hello:
hello: $(OBJS) cd ../lib ; $(MAKE) libhello.a (1) cc -o hello $(OBJS) ../lib/libhello.a (2)Accordingly, the CR for hello is the parent of the CR for libhello.a. Note that the recursive invocation of clearmake in the first line of this build script produces a separate CR hierarchy, which is not necessarily linked to the CR for hello. It is the reading of ../lib/libhello.a in the second line of the build script that creates the link between the CRs of ../lib/libhello.a and hello.
The configuration lookup algorithm used by clearmake guarantees that your builds in a parallel-development environment will be both correct (never reuse an object that is not appropriate) and optimal (always reuse an existing object that is appropriate).
At the time you enter a clearmake command, quite a few wink-in candidates may exist. You cannot simply select the candidate most recently built, because it might have been built in another view, using a completely different set of source versions. Even a single-view scenario demands configuration lookup, given the dynamic nature of views:
Suppose that you build program hello in a view that is configured to select the most recent version of hello.c to which the attribute QAed has been attached with the value "Yes". This turns out to be version 12 on the main branch.
A user discovers a bug in hello that the QA department did not catch. As a result, the QA manager removes the attribute from version 12. Now, version 9 is the most recent version with the attribute, so your view is dynamically reconfigured to access that version.
You enter a clearmake hello command. Since the version of hello.c in the view (/main/9) does not match the version in the config rec of your current instance of hello, clearmake rebuilds the program.
A “standard” make program would have been fooled by the recent time-modified stamp in this situation. The program hello is not out-of-date with respect to version 12 of hello.c, so it is certainly not out-of-date with respect to the even-older version 9. Thus, the standard make algorithm would have declared hello up-to-date, and declined to rebuild it.
clearmake is careful never to wink-in a DO that does not match your view's build configuration. But you can “manually” wink-in any DO to your view, using the winkin command:
% cleartool lsdo hello 08-Mar.12:48 akp "hello@@08-Mar.12:48.265" 07-Mar.11:40 george "hello@@07-Mar.11:40.217" % cleartool winkin hello@@07-Mar.11:40.217 Winked in derived object "hello" % hello Greetings, sakai-san! Your home directory is /net/neptune/home/sakai. It is now Tue Mar 8 12:58:30 1994. |
You might wink-in an DO with the “wrong” configuration in order to run it, to perform a byte-by-byte comparison with another DO, or to perform any other operation that requires access to the DO's file system data.
This section describes the ways in which ClearCase manages derived objects and configuration records. This user-level discussion provides background information for your day-to-day development tasks. We also mention some administrative-level facilities, and provide cross-references to the CASEVision/ClearCase Administration Guide.
You probably think of derived objects (DOs) simply as build targets — the files produced by your compiler (or other translation program). But actually, a DO consists of several parts (Figure 11-4):
VOB database object — Each DO is cataloged in the VOB database, where it is identified both by an extended name that includes both its standard pathname (for example, /usr/hw/src/hello.c) and a unique DO-ID (for example, 23-Feb.08:41.391).
Data container — The “data” portion of a derived object is stored in a standard file within a ClearCase storage area. This file is called a data container — it contains the DO's file system data (as opposed to its meta-data in the VOB database).
Configuration record — Actually, a CR is associated with a DO; it not an intrinsic part of the DO itself. More precisely, a CR is associated with the entire set of sibling DOs created by a particular invocation of a particular build script.
During the lifetime of a DO, both its data container and its CR can migrate between storage areas. When you first create a new DO in your view, its data container and CR are both stored in your view. (The actual pathname of the data container is probably of little concern to you; if you're curious — or you're troubleshooting — you can use the mvfsstorage utility to determine its “real” pathname. See Chapter 15,” Determining a Data Container's Location”, in the CASEVision/ClearCase Administration Guide.)
When you are about to perform a large build, keep in mind the fact that derived objects are initially created in view storage. In the “worst-case” scenario, the disk partition containing your view's private storage area must be able to accommodate building a new DO for every build target. (Conversely, building a large software system might cost your view very little disk space — if someone else recently built it using a similar configuration of sources.)
The first time that the DO is shared with another view — during a clearmake build or with an explicit winkin command — the DO is promoted from view storage to VOB storage:
Promotion allows any number of views to share derived objects without having to communicate directly with each other. For example, a view alpha can be “unaware” of the other views, beta and gamma, with which it shares a derived object. The hosts on which the view storage directories are located need not have network access to each other's disk storage. Information on which views are currently sharing a DO is maintained in the VOB database, and is accessible with the lsdo command:
% cleartool lsdo -l hello.o 10-Mar-94.15:25:52 Allison K. Pak (akp.user@copper) create derived object "hello.o%10-Mar.15:25.213" references: 2 (shared) => copper:/var/tmp/akp/tut/old.vws => copper:/var/tmp/akp/tut/fix.vws |
When a new derived object is created, clearmake sets the DO's reference count to 1, indicating that it is visible in one view. Thereafter, each wink-in of the DO to an additional view increments the reference count. You can also create additional UNIX-level hard links to an existing DO, each of which also increments the reference count.
Such additional hard links are sometimes subject to wink-in:
If the additional hard link was created along with the original DO, in the same build script, then a wink-in of the DO during a subsequent clearmake build also causes a wink-in of the additional hard link.
Additional hard links that you've created “manually” are not winked-in during subsequent builds.
When a program running in any of these views overwrites or deletes a shared derived object, the “link” is broken and the reference count is decremented. That is, the program deletes the view's reference to the DO; the DO itself remains in VOB storage, safe and sound.
This occurs most often when a compiler overwrites an old build target. You can also explicitly remove the derived object with a standard rm(1) command, or a make clean invocation.
It is quite common for a derived object's reference count to become zero. Suppose you build program hello and rebuild it a few minutes later. The second hello overwrites the first hello, decrementing its reference count. Since the reference count probably was 1 (no other view has winked it in), it now becomes 0. Similarly, the reference counts of “old” DOs — even those widely shared — eventually decrease to zero as development proceeds and new DOs replace them.
The lsdo command ignores such DOs by default, but you can use the - zero option to list them:
% cleartool lsdo -zero -long hello.o . . 08-Mar-94.12:47:54 Allison K. Pak (akp.user@cobalt) create derived object "hello.o@@08-Mar.12:47.259" references: 0 . . |
A derived object that is listed with a references: 0 annotation does not currently appear in any view. But some or all of its information may still be available:
If the DO was ever promoted to VOB storage (ever shared), then its data container is still in the VOB storage pool, and its CR is still in the VOB database. You can use catcr and diffcr to work with the CR; you can get to its file system data by performing a clearmake build in an appropriately-configured view, or by using the winkin command.
If the DO was never promoted, then its CR may be gone forever:
% cleartool catcr msg.o@@26-Feb.20:27.453 Config record data no longer available for "msg.o@@26-Feb.20:27.453"
Before a DO becomes shared, ClearCase makes no special effort to preserve it. You can delete it with standard software that removes or overwrites files. Likewise, a subsequent clearmake build in the same view may overwrite data containers in that view. Such activity destroys the data container in view-private storage; typically, it also destroys the associated CR, which is also in view storage.
After a DO becomes shared, however, it can be deleted only with special ClearCase commands. You can delete individual DOs with the rmdo command, but be careful — it will cause errors in processes that are currently attempting to use the deleted DO.
A zero reference count means that the derived object has been deleted, overwritten, or rebuilt in every view that ever used it. This situation calls for scrubbing: automatic deletion of DO-related information from the VOB. This can include the removal of the derived object from the VOB database, removal of its data container from a VOB storage pool (if the DO had ever been shared), and in some cases removal of its associated CR, as well. For more on scrubbing, see Chapter 10, “Periodic Maintenance of the Data Repository”, in the CASEVision/ClearCase Administration Guide.