0

Preface
This book is about porting software between UNIX platforms, the process of taking a software package in source form and installing it on your machine. This doesnt sound like a big deal at rst, but theres more to it than meets the eye: you need to know how to get the software, how to unpack what you get, how to modify the package so that it will compile on your system, how to compile and install the software on your system, and how to deal with problems if they crop up. Nevertheless, it doesnt involve anything that hasnt already been done to death in hundreds of well-written books: you can nd out about getting software from the Internet in The Whole Internet Users Guide and Catalog, by Ed Krol. Unpacking software is basically a matter of using standard tools described in dozens of good introductory textbooks. Compiling programs is so simple that most C textbooks deal with it in passing. Installation is just a matter of copying software to where you want it. Programming is the meat of lots of books on UNIX programming, for example Advanced Programming in the UNIX environment by Richard Stevens, So why yet another book? Most textbooks give you an idealized view of programming: This is the way to do it (and it works). They pay little attention to the ways things can go wrong. UNIX is famed for cryptic or misleading error messages, but not many books go into the details of why they appear or what they really mean. Even experienced programmers frequently give up when trying to port software. The probable advantage of completing the port just isnt worth effort that it takes. In this book, Id like to reduce that effort. If you take all the books I just mentioned, youll have to nd about 3 feet of shelf space to hold them. Theyre all good, but they contain stuff that you dont really want to know about right now (in fact, youre probably not sure if you ever want to know all of it). Maybe you have this pressing requirement to get this debugger package, or maybe you nally want to get the latest version of nethack up and running, complete with X11 support, and the last thing you want to do on the way is go through those three feet of paper. Thats where this book comes in. It covers all issues of porting, from nding the software through porting and testing up to the nal installation, in the sequence in which you perform them. It goes into a lot of detail comparing the features of many different UNIX systems, and offers suggestions about how to emulate features not available on the platform to which you
i

5 February 2005 02:09

ii

are porting. It views the problems from a practical rather than from a theoretical perspective. You probably wont know any more after reading it than you would after reading the in-depth books, but I hope that youll nd the approach more related to your immediate problems.

Audience
This book is intended for anybody who has to take other peoples software and compile it on a UNIX platform. It should be of particular interest to you if youre: A software developer porting software to a new platform. A system administrator collecting software products for your system. A computer hobbyist collecting software off the Internet.

Whatever your interest, I expect that youll know UNIX basics. If youre a real newcomer, you might like to refer to Learning the UNIX Operating System, by Grace Todino, John Strang and Jerry Peek. In addition, UNIX in a Nutshell, available in BSD and System V avours, includes a lot of reference material which I have not repeated in this book. The less you already know, the more use this book is going to be to you, of course. Nevertheless, even if youre an experienced programmer, you should nd a number of tricks to make life easier.

Organization
One of the big problems in porting software is that you need to know everything rst. While writing this book I had quite a problem deciding the order in which to present the material. In the end, I took a two-pronged approach, and divided this book into two major parts: 1. In the rst part, well look at the stages through which a typical port passes: getting the software, extracting the source archives, conguring the package, compiling the software, testing the results, and installing the completed package. In the second part, well take a look at the differences between different avours of UNIX, how they can make life hard for you, and how we can solve the problems.

2.

Operating System Versions

Nearly everything in this book is related to one version or another of UNIX,* and a lot of the text only makes sense in a UNIX context. Nevertheless, it should be of some use to users of other operating systems that use the C programming language and UNIX tools such as make. As in any book about UNIX, its difcult to give complete coverage to all avours. The examples in this book were made with six different hardware/software platforms:
* UNIX is, of course, a registered trademark of its current owner. In this context, I am referring to any operating system that presents a UNIX-like interface to the user and the programmer.

5 February 2005 02:09

Preface

iii

SCO XENIX/386 on an Intel 386 architecture (version 2.3.2). UNIX System V.3 on an Intel 386 architecture (Interactive UNIX/386 version 2.2). UNIX System V.4.2 on an Intel 386 architecture (Consensys V4.2). BSD on an Intel 386 architecture (BSD/386* 1.1 and FreeBSD). SunOS on a Sparc architecture (SunOS 4.1.3). IRIX 5.3 on an SGI Indy Workstation (mainly System V.4).

This looks like a strong bias towards Intel architectures. However, most problems are more related to the software platform than the hardware platform. The Intel platform is unique in offering almost every avour of UNIX that is currently available, and its easier to compare them if the hardware is invariant. I believe these examples to be representative of what you might nd on other hardware. The big difference in UNIX avours is certainly between UNIX System V.3 and BSD, while System V.4 represents the logical sum of both of them. At a more detailled level, every system has its own peculiarities: there is hardly a system available which doesnt have its own quirks. These quirks turn out to be the biggest problem that you will have to ght when porting software. Even software that ported just ne on the previous release of your operating system may suddenly turn into an error message generator.

Conventions used in this book

This book uses the following conventions: Bold is used for the names of keys on the keyboard. Well see more about this in the next section. Italic is used for the names of UNIX utilities, directories and lenames, and to emphasize new terms and concepts when they are rst introduced. Constant Width is used in examples to show the contents of les, the output from commands, program variables, actual values of keywords, for the names of Usenet newsgroups, and in the text to represent commands. Constant Italic is used in examples to show variables for which context-specic substitutions should be made. For example, the variable filename would be replaced by an actual lename. In addition it is used for comments in code examples. Constant Bold is used in examples to show commands or text that would be typed in literally by the user. Most examples assume the use of the Bourne shell or one of its descendents such as the Korn Shell, or the Free Software Foundations bash. Normally the prompt will be shown as the default $, unless it is an operation that requires the superuser, in which case it will be shown as #. When continuation lines are used, the prompt will be the standard >. In cases where the command wouldnt work with the C shell, I present an alternative. In the C shell examples, the prompt is the default %.
* Later versions of this operating system are called BSD/OS.

5 February 2005 02:09

iv

I have tried to make the examples in this book as close to practice as possible, and most are from real-life sources. A book is not a monitor, however, and displays that look acceptable (well, recognizable) on a monitor can sometimes look really bad in print. In particular, the utilities used in porting sometimes print out lines of several hundred characters. I have tried to modify such output in the examples so that it ts on the page. For similar reasons, I have modied the line breaks in some literally quoted texts, and have occasionally squeezed things like long directory listings.

Describing the keyboard

Its surprising how many confusing terms exist to describe individual keys on the keyboard. My favourite is the any key (Press any key to continue). We wont be using the any key in this book, but there are a number of other keys whose names need understanding: The Enter or Return key. Ill call this RETURN. Control characters (characters produced by holding down the CTRL key and pressing a normal keyboard key at the same time). These characters are frequently echoed on the screen as a caret () followed by the character entered. In keeping with other Nutshell books, Ill write control-D as CTRL-D. The ALT key, which emacs afcionados call a META key, works like a second CTRL key, but generates a different set of characters. These are sometimes abbreviated by prexing the character with a tilde () or the characters A-. Although these are useful abbreviations, they can be confusing, so Ill spell these out as CTRL-X and ALT-D, etc. NL is the new line character. In ASCII, it is CTRL-J, but UNIX systems generate it when you press the RETURN key. CR is the carriage return character, in ASCII CTRL-M. Most systems generate it with the RETURN key. HT is the ASCII horizontal tab character, CTRL-I. Most systems generate it when the TAB key is pressed.

Terminology
Any technical book uses jargon and technical terms that are not generally known. Ive tried to recognize the ones used in this book and describe them when they occur. Apart from this, I will be particularly pedantic about the way I use the following terms in this book: program Everybody knows what a program is: a series of instructions to the computer which, when executed, cause a specic action to take place. Source les dont t this category: a source program (a term you wont nd again in this book) is really a program source (a le that you can, under the correct circumstances, use to create a program). A program may, however, be interpreted, so a shell script may qualify as a program. So may something like an emacs macro, whether byte compiled or not (since emacs can interpret uncompiled macros directly).

5 February 2005 02:09

Preface

package

A package is a collection of software maintained in a source tree. At various stages in the build process, it will include source les: les that are part of the distribution. auxiliary les, like conguration information and object les that are not part of the source distribution and will not be installed.

installable les: les that will be used after the build process is complete. These will normally be copied outside the source tree so that the source tree can be removed, if necessary. Some software does not require any conversion: you can just install the sources straight out of the box. We wont argue whether this counts as a package. It certainly shouldnt give you any porting headaches. Well use two other terms as well: building and porting. Its difcult to come up with a hardand-fast distinction between the twowell discuss the terms in Chapter 1, Introduction.

Acknowledgements
Without software developers all over the world, there would be nothing to write about. In particular, the Free Software Foundation and the Computer Sciences Research Group in Berkeley (now defunct) have given rise to an incredible quantity of freely available software. Special thanks go to the reviewers Larry Campbell and Matt Welsh, and particularly to James Cox, Jerry Dunham, and Jrg Micheel for their encouragement and meticulous criticism of what initially was just trying to be a book. Thanks also to Clive King of the University of Aberystwyth for notes on data types and alignment, Steve Hiebert with valuable information about HP-UX, and Henry Spencer and Jeffrey Friedl for help with regular expressions. Finally, I cant nish this without mentioning Mike Loukides and Andy Oram at OReilly and Associates, who gently persuaded me to write a book about porting, rather than just presenting the reader with a brain dump.

5 February 2005 02:09

1
Introduction
One of the features that made UNIX successful was the ease with which it could be implemented on new architectures. This advantage has its down side, which is very evident when you compare UNIX with a single-platform operating system such as MS-DOS: since UNIX runs on so many different architectures, it is not possible to write a program, distribute the binaries, and expect them to run on any machine. Instead, programs need to be distributed in source form, and installation involves compiling the programs for the target hardware. In many cases, getting the software to run may be signicantly more than just typing make.

What is porting?
Its difcult to make a clear distinction between porting and building. In this book, well use three terms: building a package is the planned process of creating an installable software package. This is essentially the content of Chapter 5, Building the package. installation is the planned process of putting an installable software package where users can use it. This is what we talk about in Chapter 9, Installation. Some people use the term porting to describe a software installation requiring undocumented changes to adapt it to a new environment, not including the process of conguration if this is intended to be part of the build process. Although this is a useful denition, it contains an element of uncertainty: when you start, you dont know whether this is going to be a build or a port. Its easier to call the whole process porting, whether you just have to perform a simple build or complicated modications to the source. Thats the way well use the term in this book.

The effort required to port a package can vary considerably. If you are running a SparcStation and get some software developed specically for SparcStations, and the software does not offer much in the way of conguration options, you probably really can get it to run by reading the sources onto disk, and typing make and make install. This is the exception, however, not the rule. Even with a SparcStation, you might nd that the package is written for a different release of the operating system, and that this fact requires signicant modications. A more typical port might include getting the software, conguring the package, building the
1

5 February 2005 02:09

package, formatting and printing the documentation, testing the results and installing les in the destination directories.

How long does it take?

It is very difcult to gauge the length of time a port will take to complete. If a port takes a long time, its not usually because of the speed of the machine you use: few packages take more than a few hours to compile on a fast workstation. Even the complete X11R6 windowing system takes only about 4 hours on a 66 MHz Intel 486 PC. The real time-consumers are the bugs you might encounter on the way: if youre unlucky, you can run into big trouble, and you may nd yourself getting to know the package youre porting much more intimately than you wish, or even having to nd and x bugs. Probably the easiest kind of program to port is free software, that is to say, software that is freely redistributable. As a result of the ease of redistribution, it tends to be ported more frequently and to more platforms, so that conguration bugs get ironed out more evenly than in commercial software. Porting a product like bison* from the Free Software Foundation is usually just a matter of minutes:
$ configure checking how to run the C preprocessor ... messages from configure $ make ... messages from make $ make install

On an Intel 486/66, congure runs for 15 seconds, make runs for about 85 seconds, and make install runs for about 5 secondsall in all, less than two minutes. If everything were that simple, nobody would need this book. On the other hand, this simple view omits a point or two. bison comes with typeset documentation. Like most products of the Free Software Foundation, it is written in texinfo format, which relies on TEX for formatting. It doesnt get formatted automatically. In fact, if you look for the target in the Makele, youll nd that there isnt one: the Makele ignores printed documentation. I consider this a bug in the Makele. Never mind, its easy enough to do it manually:
$ tex bison.texinfo tex: not found

This is a fairly typical occurrence in porting: in order to port a package, you rst need to port three other, more complicated packages. In fact, most ports of bison are made in order to compile some other product, such as the GNU C compiler. In order to get our documentation printed, we rst need to port TEX, which is appropriately depicted in its own printed documentation as a shaggy lion. This is denitely a non-trivial port: TEX consists of dozens of different parts, the source tree varies greatly depending on where you get it from, the whole thing is written in Web, Donald Knuths own private dialect of Pascal, and once you get it to run you
* bison is a parser generator, compatible with yacc.

5 February 2005 02:09

Chapter 1: Introduction

discover that the output (deliberately) does not match any printer available, and that you need a so-called printer driver to output it to your favourite laser printeryet another port. Under these circumstances, it wouldnt be surprising if you give up and rely on the online documentation supplied with bison. bison has two different online reference documents: a man page and something called info, a cross-linked documentation reader from the Free Software Foundation. The man page is two pages long, the info runs to over 200K in ve les. There are no prizes for guessing where the real information is. But how do you run info? Simple: you port the GNU texinfo package. This time its not quite as bad as porting TEX, but its still more difcult than porting bison. This scenario is fairly typical: you set out to port something simple, and everything seems to be ne, and then you nd that a minor part of the port can really take up lots of time. Typically, this is the point where most people give up and make do with what they have achieved. This book is intended to help you go the whole distance.

Why we need to port

There are three main reasons why a port might be more than a simple recompilation: Different operating system. Depending on what features the operating system offers, the program may need to be modied. For example, when porting a program from UNIX to DOS, I will denitely have to do something about le naming conventions. If I port a System V.4 program to BSD I may nd I need to replace STREAMS calls with sockets calls. Different hardware. This is obvious enough with something like a display driver. If the driver you have is designed for a Sun workstation and youre porting it to a PC, you will be involved in some serious rewriting. Even in more mundane circumstances, things like the kind of CPU involved might inuence the program design. Local choices. These includes installation pathnames and cooperation with other installed software. For example, if I use the emacs editor, I may choose to use the etags program to cross-reference my source les; if I use vi, I would probably prefer to use ctags. Depending on the C compiler, I may need to use different compilation options. In many cases, this seems to be similar to the choice of operating system, but there is a signicant difference: in general, changing your kernel means changing your operating system. You can change the C compiler or even the system library without changing the basic system.

Unix flavours
UNIX spent the rst ten years of its existence as the object of computer science research. Developed in Bell Labs (part of AT&T), it was signicantly extended in the University of California at Berkeley (UCB), which started releasing signicant updates, the so-called Berkeley Software Distribution (BSD) in 1977. By the time AT&T decided to commercialize UNIX with System III in the early 80s, the fourth BSD was already available, and both System III and System V drew heavily from it. Nevertheless, the differences were signicant, and

5 February 2005 02:09

despite the advent of System V.4, which basically just added all features available in any UNIX dialect into one package, the differences remain. A good overview of the relationship between the Unixes can be found on page 5 of The Design and the Implementation of the 4.3BSD UNIX Operating System by Sam Lefer, Kirk McKusick, Mike Karels and John Quarterman. In this book I will concentrate on the differences that can be of importance when porting from one avour to another.

Research UNIX
Research UNIX is the original UNIX that has been developed inside Bell Labs since 1969. The last version that became widely available was the Seventh Edition, in 1978. This version can be considered the granddaddy of them all*, and is also frequently called Version 7. In this book, Ill make frequent references to this version. Work on Research UNIX continued until 1993, by which time it had reached the Tenth Edition. Its unlikely that youll have much to do with it directly, but occasionally ideas from Research UNIX trickle into other avours.

Berkeley UNIX (BSD)

The rst Berkeley Software Distribution was derived from the 6th edition in 1977 and ran on PDP-11s only. 2BSD was the last PDP-11 version: 2.11BSD is still available for PDP-11s, if you have a need (and a UNIX source licence). 3BSD was derived from 2BSD and the 7th edition via a short-lived version called 32Vin 1979. Since then, BSD has evolved relatively free of outside borrowings. With the closure of the Computer Science Research Group in Berkeley in autumn 1993 and the release of 4.4BSD in early 1994, the original BSD line has died out, but the public release of the complete sources will ensure the continued availability of Berkeley UNIX for a long time to come. Current BSD systems include BSD/OS (formerly called BSD/386), 386BSD, NetBSD and FreeBSD. These were all originally ports of the BSD Net-2 tape, which was released in 1991, to the Intel 386 architecture. These ports are interesting because they are almost pure BSD and contain no AT&T licensed code. BSD/OS is a commercial system that costs money and supplies support; the other three are available free of charge. It is not clear how long all three free versions will continue to exist side-by-side. 386BSD may already be dead, and the difference between NetBSD and FreeBSD is difcult to recognize. At the time of writing, current versions of BSD/OS and FreeBSD are based on 4.4BSD, and NetBSD is planning to follow suit.

XENIX
XENIX is a version of UNIX developed by Microsoft for Intel architectures in the early 80s. It was based mainly on the System III versions available at the time, though some ideas from other versions were included and a signicant amount of work was put into making it an easier system to live with. Not much effort was put into making it compatible with other versions of UNIX, however, and so you can run into a few surprises with XENIX. SCO still markets it,
* In fact, a number of UNIX avours, including System V and BSD, can trace their origins back to the Sixth Edition of 1976, but they all benetted from modications made in the Seventh Edition.

5 February 2005 02:09

Chapter 1: Introduction

but development appears to have stopped about 1989.

System V
System V was derived from the 6th and 7th editions via System III, with a certain amount borrowed from 4.0BSD. It has become the standard commercial UNIX, and is currently the only avour allowed to bear the UNIX trademark. It has evolved signicantly since its introduction in 1982, with borrowings from Research UNIX and BSD at several points along the way. Currently available versions are V.3 (SCO Open Desktop) and V.4 (almost everybody else). System V.3 lacked a number of features available in other Unixes, with the result that almost all V.3 ports have borrowed signicantly from other versions, mainly 4.2BSD. The result is that you cant really be sure what you have with System V.3 you need to consult the documentation for more information. In particular, vanilla System V.3 supports only the original UNIX le system, with le names length limited to 14 characters and with no symbolic links. It also does not have a standard data communications interface, though both BSD sockets and System V STREAMS have been ported to it. System V.3.2 is, as its name suggests, a version of System V.3. This version includes compatibility with XENIX system calls. As we saw above, XENIX went its own way for some time, resulting in incompatibilities with System V. These XENIX features should be supported by the kernel from System V.3.2 onwards. SCO UNIX is version V.3.2, and includes STREAMS support. System V.4 is the current version of System V. Previous versions of System V were often criticized for lacking features. This cannot be said of System V.4: it incorporates System V.3.2 (which already incorporates XENIX), 4.3BSD, and SunOS. The result is an enormous system which has three different ways to do many things. It also still has signicant bugs. Developing software under System V.4 is an interesting experience. Since the semantics of System V.3 and BSD differ in some areas, System V.4 supplies two separate sets of libraries, one with a System V personality and one with a BSD personality. There are no prizes for guessing which is more reliable: unless you really need to, you should use the System V libraries. When we discuss kernel and library differences in Part 2 of the book, the statement This feature is supported by System V.4 will mean that the System V library interface supports it. The statement This feature is supported by BSD also implies that it should be supported by the BSD library interface of System V.4.

OSF/1
OSF/1 is a comparatively recent development in the UNIX market. It was developed by the Open Systems Foundation, an industry consortium formed as a result of dissatisfaction with AT&Ts policy on UNIX. The kernel is based on CMUs Mach operating system, a so-called microkernel*. The original Mach operating system was styled on Berkeley UNIX. OSF/1 attempts to offer the same functionality as System V, though inevitably some incompatibilities
* A microkernel operating system is an operating system that leaves signicant operating system functionality to external components, usually processes. For example, device drivers and le systems are frequently implemented as separate processes. It does not imply that the complete system is any smaller or less functional than the monolithic UNIX kernel.

5 February 2005 02:09

exist.

POSIX.1
POSIX is a series of emerging IEEE standards applying to operating systems, utilities, and programming languages. The relevant standard for operating systems is IEEE 1003.1-1990, commonly called POSIX.1. It has also been adopted by the International Standards Organization (ISO) as standard ISO/IEC 9945.1:1990. POSIX.1 denes the interface between application programs and the operating system, and makes no demands on the operating system except that it should supply the POSIX.1 interface. POSIX.1 looks very much like a subset of UNIX. In fact, most users wouldnt notice the difference. This makes it easy for UNIX operating systems to supply a POSIX.1 interface. Other operating systems might need much more modication to become POSIX.1 compliant. From a UNIX viewpoint, POSIX.1 does not supply as rich a set of functions as any of the commercially available UNIX avours, so programming to POSIX specications can feel somewhat restrictive. This matter is discussed in the POSIX Programmers Guide by Donald Lewine. Despite these slight disadvantages, POSIX has a great inuence on operating system development: all modern avours of UNIX claim to be POSIX-compliant, although the degree of success varies somewhat, and other systems are also attempting to supply a POSIX.1 interface. The trend is clear: future UNIX-like operating systems will be POSIX-compliant, and if you stick to POSIX features, your porting problems will be over. And I have a supply of bridges for sale, rst come, rst served.

Other flavours
It doesnt take much effort to add a new feature to a kernel, and people do it all the time. The result is a proliferation of systems that mix various features of the leading products and additional features of their own. On top of that, the release of kernel sources to the net has caused a proliferation of free operating systems. Systems that you might well run into include: AIX, IBMs name for its UNIX versions. Current versions are based on System V.3, but IBM has stated an intent to migrate to OSF/1 (IBM is a leading member of the OSF). Compared to System V, it has a large number of extensions, some of which can cause signicant pain to the unwary. HP-UX, Hewlett Packards UNIX system. It is based on System V.3, but contains a large number of so-called BSD extensions. Within HP, it is considered to be about 80% BSDcompliant. Linux, a UNIX clone for the Intel 386 architecture written by Linus Torvalds, a student in Helsinki. It has absolutely no direct connection with traditional UNIX avours, which gives it the unique advantage amongst free UNIXes of not being a potential subject for litigation. Apart from that, it has a vaguely System V-like feeling about it. If you are porting to Linux, you should denitely subscribe to the very active network news groups (comp.os.linux.*).

5 February 2005 02:09

Chapter 1: Introduction

SunOS is the generic name of Sun Microsystems operating systems. The original SunOS was derived from 4.2BSD and 4.3BSD, and until release 4.1 it was predominantly BSD-based with a signicant System V inuence. Starting with version 5.0, it is a somewhat modied version of System V.4. These later versions are frequently referred to as Solaris, though this term properly applies to the complete system environment, including windowing system (OpenWindows), development tools and such, and does not apply only to the System V based versions. Solaris 1.x includes the BSD-based SunOS 4.1 as its kernel; Solaris 2.x includes the System V.4-based SunOS 5.x as its kernel. Ultrix is DECs port of 4.1BSD and 4.2BSD to the VAX and MIPS-based workstations. It is now obsolete and has been replaced by OSF/1.

I would have liked to go into more detail about these versions of UNIX, but doing so would have increased the size of the book signicantly, and even then it wouldnt be possible to guarantee the accuracy: most systems add functionality in the course of their evolution, and information that is valid for one release may not apply to an earlier or a later release. As a result, Ive made a compromise: nearly all UNIX features were introduced either in BSD or System V, so I will distinguish primarily between these two. Where signicant differences exist in other operating systemSunOS 4 is a good example I will discuss them separately. Where does this leave you with, say, NonStop UX version B30? NonStop UX version B is a version of UNIX System V.4 that runs on Tandems Integrity series of fault-tolerant MIPSbased UNIX systems. It includes some additional functionality to manipulate the hardware, and some of the header les differ from the standard System V.4. In addition, it includes a minimal carry-over of BSDisms from the System V.3 version. Obviously, you can start by treating it as an implementation of System V.4, but occasionally you will nd things that dont quite seem to t in. Since its a MIPS-based system, you might try to consider it to be like SGIs IRIX operating system version 5, which is System V.4 for SGIs MIPS-based hardware. Indeed, most IRIX 5.x binaries will also run unchanged on NonStop UX version B, but you will notice signicant differences when you try to port packages that already run on IRIX 5.x. These differences are typical of a port to just about every real-life system. There are very few pure System V.4 or pure BSD systems out thereeverybody has added something to their port. Ultimately, you will need to examine each individual problem as it occurs. Here is a strategy you can use to untangle most problems on UNIX systems: Interpret the error messages to gure out what feature or function call is causing the problem. Typically, the error message will come from the compiler and will point to a specic line in a specic le. Look up the feature or call in this book. Use the description to gure out what the original programmer intended it to do. Figure out how to achieve the same effect on your own system. Sometimes, I recommend a change which you can make and try the program again. If youre not sure how your system works, you can probably nd a manual page for the feature or call, and this book will help you interpret it.

5 February 2005 02:09

Recongure or change the code as necessary, then try building again.

Where you fit in

The effort involved in porting software depends a lot on the package and the way it is maintained. It doesnt make much difference whether the software is subject to a commercial license or is freely available on the net: the people who write and maintain it can never hope to port it to more than a fraction of the platforms available. The result is that there will always be problems that they wont know about. There is also a very good chance that the wellknown and well-used package you are about to port may never have been ported quite that way before. This can have some important consequences: You may run into bugs that nobody has ever seen before in a well-known and well-used package. The package that you ported in ten minutes last year and have been using ever since has been updated, and now you cant get the @&*(&@$( to compile or run.

This also means that if you do run into problems porting a package, your feedback is important, whether or not you can supply a x. If you do supply a x, it should t into the package structure so that it can be included in a subsequent release. To reiterate: it makes very little difference here whether we are talking about free or licensed software. The players involved are different, but the problems are not. In many ways, free software is easier, since there are fewer restrictions in talking about it (if you run into problems porting System V.4, you cant just send the code out on the net and ask for suggestions), and theres a chance that more people will have ported it to more platforms already. Apart from that, everything stays the same.

But can I do it?

Of course, maybe your concern is whether you can do it at all. If youve never ported a program before, you might think that this is altogether too difcult, that youll spend days and weeks of effort and confusion and in the end give it up because you dont understand what is going on, and every time you solve a problem, two new ones spring up in its place. Id like to say Dont worry, with this book nothing can go wrong, but unfortunately things arent always like that. On the other hand, its easy too overestimate the things that can go wrong, or how difcult a port might be. Lets look at the bad news rst: in most cases, you can assume that the worst thing that can happen when you try to port a package is that it wont work, but in some unfortunate cases you may cause your system to panic, especially if you are porting kernel software such as device drivers. In addition, if you are porting system utilities, and they dont work, you could nd that you can no longer perform such essential system functions as starting or shutting down the system. These problems dont occur very often, though, and they should not cause any lasting damage if you religiously back up your system (you do perform regular backups, dont you?).

5 February 2005 02:09

Chapter 1: Introduction

Apart from such possible dangers, there is very little that can go wrong. If you are building a package that has already had been ported to your platform, you should not run into any problems that this book cant help you solve, even if you have negligible background in programming and none in porting.

How to use this book

The way you approach porting depends on how difcult it is. If its a straightforward business, something that has been done dozens of times before, like our example of porting bison above, its just a matter of following the individual steps. This is our approach in the rst part of this book, where we look at the following topics: Getting the software. You might get the sources on tape, on CD-ROM, or by copying them from the Internet. Getting them from this format into a format you can use to compile them may not be as simple as you think. Well look at this subject in Chapter 2, Unpacking the goodies and Chapter 3, Care and feeding of source trees. Congure the package for building. Although UNIX is a relatively well dened operating system, some features are less well dened. For example, there are a number of different ways to perform interprocess communication. Many packages contain alternative code for a number of operating systems, but you still need to choose the correct alternative. People often underestimate this step: it seems simple enough, but in many cases it can be more work than all the rest put together. Conguration is a complicated subject, and various methods have evolved. In Chapter 4, Package conguration, well look at manual conguration, shell scripts, and imake, the X11 conguration solution. Build the package. This is what most people understand by porting. Well look at problems running make in Chapter 5, Building the package, and problems running the C compiler in Chapter 6, Running the compiler. Format and print the documentation, which well investigate in Chapter 7, Documentation. Test the results to make sure that they work. Well look at this in Chapter 8, Testing the package. Well discuss how to do installation correctly, accurately and completely in Chapter 9, Installation. Tidy up after the build. In Chapter 10, Where to go from here, well look at what this entails.

Fortunately, almost no package gives you trouble all the way, but its interesting to follow a port through from getting the software to the nished installation, so as far as is possible Ill draw my examples in these chapters from a few free software packages for electronic mail and Usenet news. Specically, well consider Taylor uucp, the electronic mail reader elm, and C news. In addition, well look at the GNU C compiler gcc, since it is one of the most

5 February 2005 02:09

10

frequently ported packages. Well port them to an Intel 486DX/2-66 machine running BSD/386 Version 1.1.*

Part 2
As long as things go smoothly, you can get through the kind of port described in the rst part of this book with little or no programming knowledge. Unfortunately, things dont always go smoothly. If they dont, you may need to make possibly far-reaching changes to the sources. Part 1 doesnt pay much attention to this kind of modicationthats the topic of part 2 of this book, which does expect a good understanding of programming: In Chapter 11, Hardware dependencies, well look at problems caused by differences in the underlying hardware platform. In the following ve chapters, well look at some of the differences in different UNIX avours. First well look at a number of smaller differences in Chapter 12, Kernel dependencies, then well look at some of the more common problem areas in Chapter 13, Signals, Chapter 14, File systems, Chapter 15, Terminal drivers, and Chapter 16, Timekeeping. Well look at the surprising number of headaches caused by header les in Chapter 17, Header les, and at system library functionality in Chapter 18, Function libraries. Well examine the differences between various avours of the more important tools in Chapter 19, Make, Chapter 20, Compilers, and Chapter 21, Object les and friends.

Finally, there are a number of appendixes: Appendix A, Comparative reference to UNIX data types, describes the plethora of data types that have developed since the advent of ANSI C. Appendix B, Compiler ags, gives you a comparative reference to the compiler ags of many common systems. Appendix C, Assembler directives and ags, gives you a comparative reference to assembler directives and ags. Appendix D, Linker ags, gives you a comparative reference to linker ags. Appendix E, Where to get sources, gives you information on where to nd useful source les, including a number of the packages we discuss in this book.

* With the exception of Taylor uucp, BSD/OS, which at the time was called BSD/386, is supplied with all these packages, so you would only be need to port them if you wanted to modify them or port a new version.

5 February 2005 02:09

Chapter 1: Introduction

11

Preparations
You dont need much to port most packages. Normally everything you needa C compiler, a C library, make and some standard toolsshould be available on your system. If you have a system that doesnt include some of these tools, such as a System V release where every individual program seems to cost extra, or if the tools are so out-of-date that they are almost useless, such as XENIX, you may have problems. If your tools are less than adequate, you should consider using the products of the Free Software Foundation. In particular, the GNU C compiler gcc is better than many proprietary compilers, and is the standard compiler of the Open Software Foundation. You can get many packages directly from the Internet or on CD-ROM. If you are going to be doing any serious porting, I recommend that you get at least the GNU software packages, 4.4BSD Lite, and T X, preferably on CD-ROM. In particular, the GNU software and 4.4BSD Lite contain the E sources to many library functions that may be missing from your system. In addition, many of the GNU packages are available in precompiled binary form from a number of sources. Ill refer to these packages frequently in the text.

5 February 2005 02:09

2
Unpacking the goodies
Before you can start porting, you need to put the sources on disk. We use the term source tree to refer to the directory or hierarchy of directories in which the package is stored. Unpacking the archives may not be as trivial as it seems: software packages are supplied in many different formats, and it is not always easy to recognize the format. In this chapter, well look at how to extract the sources to create the source tree. In Chapter 3, Care and feeding of source trees, well see how the source tree changes in the course of a port, and what you can do to keep it in good shape.

Getting the sources

The standard way to get software sources is on some form of storage medium, such as CDROM or tape. Many packages are also available online via the Internet. The choice is not as simple as it seems:

Software from the Internet

If you have an Internet connection, and if the software is available on the net, its tempting to just copy it across the net with ftp. This may not be the best choice, however. Some packages are very big. The compressed sources of the GNU C compiler, for example, occupy about 6 MB. You cant rely on a typical 56 kb/s line to transfer more than about 2 kilobytes per second.* At this speed, it will take nearly an hour to copy the archives. If youre connected via a SLIP line, it could take several hours. Gaining access to the archive sites is not always trivial: many sites have a maximum number of users. In particular, prep.ai.mit.edu, the prime archive site for gcc, is frequently overloaded, and you may need several attempts to get in. In addition, copying software over the net is not free. It may not cost you money, but somebody has to pay for it, and once you have the software, you need somewhere to store it, so you dont really save on archive media.
* Of course, it should approach 7 kilobytes per second, but network congestion can pull this gure down to a trickle. 13

5 February 2005 02:09

14

Choice of archive medium

If you do choose to get your software on some other medium, you have the choice between CD-ROM and tape. Many archive sites will send you tapes if you ask for them. This may seem like a slow and old-fashioned way to get the software, but the bandwidth is high:* DAT and Exabyte tapes can store 2 GB per tape, so a single tape could easily contain as much software as you can duplicate in a week. In addition, you dont need to make a backup before you start. Software on CD-ROM is not as up-to-date as a freshly copied tape, but its easy to store and reasonably cheap. Many companies make frequent CD editions of the more widely known archive sites for example, Walnut Creek CD-ROM has editions of most commonly known software, frequently pre-ported, and Prime Time Freeware issues a pair of CD-ROMs twice a year with 5 GB of compressed software including lesser-known packages. This can be worth it just to be able to nd packages that you would otherwise not even have known about. If you have already ported a previous version of the package, another alternative is to use diffs to bring the archive up to date. Well look at this on page 29.

Archives
You frequently get pure source trees on CD-ROM, but other media, and also many CD-ROMs, transform the source tree several times: A source tree is usually supplied in an archive, a le containing a number of other les. Like a paper bag around groceries, an archive puts a wrapper around the les so that you can handle them more easily. It does not save any space in fact, the wrapper makes it slightly larger than the sum of its les. Archives make it easier to handle les, but they dont do anything to save space. Much of the information in les is redundant: each byte can have 256 different values, but typically 99% of an archive of text or program sources will consist of the 96 printable ASCII characters, and a large proportion of these characters will be blanks. It makes sense to encode them in a more efcient manner to save space. This is the purpose of compression programs. Modern compression programs such as gzip reduce the size of an archive by up to 90%. If you want to transfer archives by electronic mail, you may also need to encode them to comply with the allowable email character set. Large archives can become very unwieldy. We have already seen that it can take several hours to transfer gcc. If the line drops in this time, you may nd that you have to start the le again. As a result, archives are frequently split into more manageable chunks.

The most common form of archive youll nd on the Internet or on CD-ROM is gzipped tar, a tar archive that has been compressed with gzip. A close second is compressed tar, a tar
* To quote a fortune from the fortune program: Never underestimate the bandwidth of a station wagon full of tapes..

5 February 2005 02:09

Chapter 2: Unpacking the goodies

15

archive that has been compressed with compress. From time to time, youll nd a number of others. In the following sections well take a brief look at the programs that perform these tasks and recover the data.

Archive programs
A number of archive programs are available: tar, the tape archive program, is the all-time favourite. The chances are about 95% that your archive will be in tar format, even if it has nothing to do with tape. cpio is a newer le format that once, years ago, was intended to replace tar. cpio archives suffer from compatibility problems, however, and you dont see them very often. ar is a disk archive program. It is occasionally used for source archives, though nowadays it is almost only used for object le archives. The ar archive format has never been completely standardized, so you get an ar archive from a different machine, you might have a lot of trouble extracting it. Well look at ar formats again in , on page 383. shar is the shell archive program. It is unique amongst archive programs in never using non-printing characters, so shar archives can be sent by mail. You can extract shar archives simply by feeding them to a (Bourne) shell, though it is safer to use a program like unshar.

Living with tar

tar is a relatively easy program to use, but the consequences of mistakes can be far-reaching. In the following sections, well look at how to use tar and how to avoid trouble.

Basic use
When it comes to unpacking software, one or two tar commands can meet all your needs. First, you often want to look at the contents before unpacking. Assuming that the archive is named et1.3.tar, the following command lists the les in the archive:
$ tar tf et1.3.tar et1.3/ et1.3/bell.c pet1.3/bltgraph.c et1.3/BLURB

The t option stands for table of contents, and the f option means use the next parameter in the command (et1.3.tar) as the name of the archive to list. To read in the les that were listed, use the command:
$ tar xfv et1.3.tar et1.3/ et1.3/bell.c pet1.3/bltgraph.c et1.3/BLURB

5 February 2005 02:09

16

The list looks the same, but this time the command actually creates the directory et1.3 if necessary, and then creates the contents. The x option stands for extract, and the f option has the same meaning as before. The v option means verbose and is responsible for generating the list, which gives you the assurance that the command is actually doing something. To bundle some les into an archive, use a command like:
$ tar cvf et1.3.tar et1.3

This command packs everything in the et1.3 directory into an archive named et1.3.tar (which is where we started). The c option stands for create and the v option for verbose. This time, the f means use the next parameter in the command (et1.3.tar) as the archive to create.

Absolute pathnames
Many versions of tar have difculties with absolute pathnames. If you back up a directory /usr/foo, they will only be able to restore it to exactly this directory. If the directory is /usr/bin, and youre trying to restore programs like sh, this could give you serious problems. Some versions of tar have an option to ignore the leading /, and others, such as GNU tar, ignore it unless you tell them otherwise.

Symbolic links
Many versions of tar will only back up a symbolic link, not the le or directory to which it points. This can be very embarrassing if you send somebody a tape with what should be a complete software package, and it arrives with only a single symbolic link.

Tape block size

Many DDS (DAT) drives work better with high blocking factors, such as 65536 bytes per block (128 tape blocks). You can do this with the option b (block size):
$ tar cvfb /dev/tape 128 foo-dir

Unfortunately, this can cause problems too. Some DDS drives cannot read tapes with block sizes of more than 32768 bytes, and some versions of tar, such as SGI IRIS 5.x, cannot handle tapes blocked larger than 20 tape blocks (10240 bytes). This is a show-stopper if you have a tape which is really blocked at more than this size: you just wont be able to read it directly. You can solve this problem by installing GNU tar or piping the archive through dd:
$ dd if=/dev/rmt/ctape0 ibs=128b obs=2b | tar xvf -

File names
Most versions of tar perform lename matching based on the exact text as it appears on the tape. If you want to extract specic les, you must use the names by which they are known in the archive. For example, some versions of tar may end up writing absolute names with two leading slashes (like //usr/bin/sh, for example). This doesnt worry the operating system, which treats multiple leading slashes the same as a single leading slash, but if you want to

5 February 2005 02:09

Chapter 2: Unpacking the goodies

17

extract this le, you need to write:

$ tar x //usr/bin/sh

File name sorting

A tar archive listing with tar tv deliberately looks very much like a listing done with ls -l. There is one big difference, however: ls -l sorts the le names by name before displaying them, whereas tar, being a serial archive program, displays the names in the order in which they occur in the archive. The list may look somewhat sorted, depending on how the archive was created, but you cant rely on it. This means that if you are looking for a le name in an archive, you should not be misled if its not where you expect to nd it: use tools like grep or sort to be sure.

tar: dir - cannot create

With System V systems, you may see things like:
$ tar xvf shellutils-1.9.4.tar tar: shellutils-1.9.4/ - cannot create x shellutils-1.9.4/COPYING, 17982 bytes, 36 tape blocks x shellutils-1.9.4/COPYING.LIB, 25263 bytes, 50 tape blocks tar: shellutils-1.9.4/lib/ - cannot create x shellutils-1.9.4/lib/Makefile.in, 2868 bytes, 6 tape blocks x shellutils-1.9.4/lib/getopt.h, 4412 bytes, 9 tape blocks

This bug has been around so long that you might suspect that it is an insider joke. In fact, it is a benign compatibility problem. The POSIX.2 standard tar format allows archives to contain both directory and le names, although the directory names are not really necessary: assuming it has permission, tar creates all directories necessary to extract a le. The only use of the directory names is to specify the modication time and permissions of the directory. Older versions of tar, including System V tar, do not include the directory names in the archive, and dont understand them when they nd them. In this example, we have extracted a POSIX.2 tar archive on a System V system, and it doesnt understand (or need) the directory information. The only effect is that the directories will not have the correct modication timestamps and possibly not the correct permissions.

Losing access to your files

Some versions of tar, notably System V versions, have another trick in store: they restore the original owner of the les, even if that owner does not exist. That way you can lose access to your les completely if they happen to have permissions like rw-------. You can avoid this by using the o ag (restore ownership to current user). It would be nice to be able to say make a rule of always using the o ag. Unfortunately, other versions of tar dene this ag differently check your man pages for details.

5 February 2005 02:09

18

Multivolume archives
tar can also handle multi-volume archives, in other words archives that go over more than one tape. The methods used are not completely portable: one version of tar may not be able to read multivolume archives written by a different version. Some versions of tar just stop writing data at the end of one tape and continue where they left off at the beginning of the next reel, whereas others write header information on the second tape to indicate that it is a continuation volume. If possible, you should avoid writing multivolume archives unless you are sure that the destination system can read them. If you run into problems with multivolume archives you cant read, you might save the day with something like:
$ (dd if=$TAPE ++ echo 1>&2 Change tapes and press RET ++ read confirmation the name of the variable isnt important ++ dd if=$TAPE ++ echo 1>&2 Change tapes and press RET ++ read confirmation ++ dd if=$TAPE) | tar xvf -

This uses dd to copy the rst tape to stdout, then prints a message and waits for you to press the enter key, copies a second tape, prompts and waits again, and then copies a third tape. Since all the commands are in parentheses, the standard output of all three dd commands is piped into the tar waiting outside. The echo commands need to go to stderr (thats the 1>&2) to get displayed on the terminalotherwise they would be piped into the tar, which would not appreciate it. This only works if the version of tar you use doesnt put any header information (like reel number and a repeat of the le header) at the beginning of the subsequent reels. If it does, and you cant nd a compatible tar to extract it again, the following method may help. Assuming a user of an SCO system has given you a large program foo spread over 3 diskettes, each of which contains header information that your tar doesnt understand, you might enter
$ $ $ $ $ $ $ $ tar x foo mv foo foo.0 tar x foo mv foo foo.1 tar x foo mv foo foo.2 cat foo.* >foo rm foo.* extract first part from first floppy save the first part extract second part from second floppy save the second part extract third part from third floppy save the third part concatenate them and remove the intermediate files

Extracting an archive with tar

Using tar to extract a le is normally pretty straightforward. You can cause a lot of confusion, however, if you extract into the wrong directory and it already contains other les you want to keep. Most archives contain the contents of a single directory as viewed from the parent directory in other words, the name of the directory is the rst part of all le names. All GNU software follows this rule:

5 February 2005 02:09

Chapter 2: Unpacking the goodies

$ tar tvf groff-1.09.tar drwxr-xr-x jjc/staff 0 Feb 19 drwxr-xr-x jjc/staff 0 Feb 19 -rw-r--r-- jjc/staff 607 Sep 21 -rw-r--r-- jjc/staff 1157 Oct 30 -rw-r--r-- jjc/staff 1377 Aug 3 -rw-r--r-- jjc/staff 1769 Aug 10

19

14:15 14:13 12:03 07:38 12:34 15:48

1994 1994 1992 1993 1992 1992

groff-1.09/ groff-1.09/include/ groff-1.09/include/Makefile.sub groff-1.09/include/assert.h groff-1.09/include/cmap.h groff-1.09/include/cset.h

Others, however, show the les from the viewpoint of the directory itselfthe directory name is missing in the archive:
$ tar tvf blaster.tar -rw-r--r-- 400/1 -rw-r--r-- 400/1 -r--r--r-- 400/1 -rw-r--r-- 400/1 -rw-r--r-- 400/1 -rw------- 400/1 -rw-r--r-- 400/1 5666 3638 2117 2420 3408 10247 1722 Feb Feb Feb Feb Feb Feb Feb 14 14 14 14 14 14 14 01:44 01:44 01:44 15:17 01:44 01:44 04:10 1993 1993 1993 1993 1993 1993 1993 README INSTALL LICENSE Makefile sb_asm.s stream.c apps/Makefile

If you have an archive like the rst example, you want to be in the parent directory when you extract the archive; in the second case you need to rst create the directory and then cd to it. If you extract the second archive while in the parent directory, you will face a lot of cleaning up. In addition, there is a good chance that les with names like README, INSTALL and LICENSE may already be present in that directory, and extracting this archive would overwrite them. There are a couple of ways to avoid these problems: Always look at the archive contents with tar t before extracting it. Once you have looked at the archive contents, you can change to the correct directory into which to extract it. In the case of groff above, you might choose a directory name like /mysources*. In the case of blaster, you could create a directory /mysources/blaster and extract into that directory. Alternatively, you can always create a subdirectory and extract there, and then rename the directory. In the rst example, you might create a directory /mysources/temp. After extraction, you might nd that the les were in a directory /mysources/temp/groff-1.09, so you could move them with
$ mv groff-1.09 ..

If they extract directly into temp, you can rename the directory:
$ cd .. $ mv temp groff-1.09

This method may seem easier, but in fact there are a couple of problems with it: You need to choose a directory name that doesnt clash with the real name. Thats why we used the name temp in this example: otherwise it wont be possible to rename the directory in the rst example, since you would be trying to overwrite the directory with one of its own subdirectories.

* A number of shells use the shorthand notation / to refer to your home directory.

5 February 2005 02:09

20

Not all avours of UNIX allow you to move directories.

The command to extract is almost identical to the command to list the archive a clear case for a shell with command line editing:
$ tar tvf groff-1.09.tar list the archive $ tar xvf groff-1.09.tar extract the archive

Frequently your tar archive will be compressed in some way. There are methods for extracting les directly from compressed archives. Well examine these when we look at compression programs on page .

Compression programs
If the archive is compressed, you will need to uncompress it before you can extract les from it. UNIX systems almost invariably use one of three compression formats: compressed les are created with the compress program and extracted with uncompress. They can be up to 70% smaller than the original le. The zcat program will uncompress a compressed le to the standard output. gzipped les are created by gzip and extracted by gunzip. They can be up to 90% smaller than the original le. gunzip will also uncompress compressed or packed les. packed les are obsolete, though you still occasionally see packed man pages. They are created by the pack program and uncompressed by the unpack program. The pcat program will uncompress a packed le to the standard output.

Each of these programs is installed with three different names. The name determines the behavior. For example, gzip is also known as gunzip and zcat:
$ ls -li /opt/bin/gzip /opt/bin/gunzip /opt/bin/zcat 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/gunzip 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/gzip 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/zcat

The -i option to ls tells it to list the inode number, which uniquely identies the le. In this case, you will see that all three names are linked to the same le (and that the link count eld is 3 as a result). You will notice that gzip has also been installed under then name zcat, replacing the name used by compress. This is not a problem, since gzcat can do everything that zcat can do, but it can lead to confusion if you rely on it and one day try to extract a gzipped le with the real zcat.

Encoded files
Most archive programs and all compression programs produce output containing non-printable characters. This can be a problem if you want to transfer the archive via electronic mail, which cannot handle all binary combinations. To solve this problem, the les can be encoded: they are transformed into a representation that contains only printable characters. This has the disadvantage that it makes the le signicantly larger, so it is used only when absolutely

5 February 2005 02:09

Chapter 2: Unpacking the goodies

21

necessary. Two programs are in common use: uuencode is by far the most common format. The companion program uudecode will extract from standard input. btoa format is used to some extent in Europe. It does not expand the le as much as uuencode (25% compared to 33% with uuencode), and is more resistant to errors. You decode the le with the atob program.

Split archives
Many ftp sites split large archives into equal-sized chunks, typically between 256 kB and 1.44 MB (a oppy disk image). Its trivial to combine them back to the original archive: cat will do just that. For example, if you have a set of les base09.000 through base09.013 representing a gzipped tar archive, you can combine them with:
$ cat base09.* > base09.tar.gz

This will, of course, require twice the amount of storage, and it takes time. Its easier to extract them directly:
$ cat base09.* | gunzip | tar xvf drwxr-xr-x root/wheel 0 Aug 23 06:22 1993 ./sbin/ -r-xr-xr-x bin/bin 106496 Aug 23 06:21 1993 ./sbin/chown -r-xr-xr-x bin/bin 53248 Aug 23 06:21 1993 ./sbin/mount_mfs ... etc

cat pipes all archives in alphabetical le name order to gunzip. gunzip uncompresses it and pipes the uncompressed data to tar, which extracts the les.

Extracting a linked file

tar is clever enough to notice when it is backing up multiple copies of a le under different names, in other words so-called hard links. When backing up, the rst time it encounters a le, it copies it to the archive, but if it encounters it again under another name, it simply creates an entry pointing to the rst le. This saves space, but if you just try to extract the second le, tar will fail: in order to extract the second name, you also need to extract the le under the rst name that tar found. Most versions of tar will tell you what the name was, but if you are creating archives, it helps to back up the most-used name rst.

Whats that archive?

All the preceding discussion assumes that you know the format of the archive. The fun begins when you dont. How do you extract it? Your primary indication of the nature of the le is its lename. When archives are created, compressed and encoded, they usually receive a le name sufx to indicate the nature of the le. You may also have come across the term extension, which comes from the MS-DOS world. These sufxes accumulate as various steps proceed. A distribution of gcc might come in a le called gcc-2.5.8.tar.gz.uue. This name gives you the following information:

5 February 2005 02:09

22

The name of the package: gcc. The revision level: -2.5.8. You would expect the name of the root directory for this package to be gcc-2.5.8. The archive format: .tar. Since this is a GNU package, you can expect the name of the uncompressed archive to be gcc-2.5.8.tar. The compression format: .gz (gzip format). The name of the compressed archive would be gcc-2.5.8.tar.gz. The encoding format: .uue (encoded with uuencode).

Some operating systems, notably System V.3 and Linux, still provide le systems which restrict le names to 14 characters. This can lead to several problems.* Archives distributed for these systems frequently use variants on these names designed to make them shorter; gcc-2.5.8.tzue might be an alternate name for the same package. The following table gives you an overview of archive le sufxes you might encounter. Well look at source le sufxes in Chapter 20, Compilers, page
Table 21: Common le name sufxes

Name sufx # ,v .a .arc .arj .cpio .diff .gif .gz .hqx .jpg .lzh .orig .rej .shar .sit .tar .uu

Format Alternate patch reject le name. emacs backup les, also used by some versions of patch. RCS le. Created by ci, extracted by co. ar format. Created by and extracted with ar. Created by and extracted with arc. DOS arj format Created by and extracted with cpio. Difference le, created by diff, can be applied by patch. Graphics Interchange Format gzip format. Created by gzip, extracted with gunzip. HQX (Apple Macintosh) JPEG (graphics format) LHa, LHarc, Larc Original le after processing by patch. patch reject le. Shell archive: created by shar, extracted with any Bourne-compatible shell. Stuff-It (Apple Macintosh) tar format. Created by and extracted with tar. uuencoded le. Created by uuencode, decoded with uudecode.

* If you have one of these systems, and you have a choice of le systems, you can save yourself a lot of trouble by installing one that allows long le names.

5 February 2005 02:09

Chapter 2: Unpacking the goodies Table 21: Common le name sufxes (continued)

23

Name sufx .uue .Z .z .zip .zoo

Format Alternative for .uu Compressed with compress, uncompressed with uncompress, zcat or gunzip. Two different formats: either pack format, compressed by pack, extracted with pcat, or old gzip format, compressed by gzip, extracted with gunzip. Zip (either PKZip or Zip/Unzip) Zoo

Identifying archives
Occasionally youll get an archive whose name gives you no indication of the format. Under these circumstances, nding the kind of archive can be a matter of trial and error, particularly if it is compressed. Here are a couple of ideas that might help:

file
The UNIX le command recognizes a lot of standard le types and prints a brief description of the format. Unfortunately, the le really needs to be a le: le performs some le system checks, so it cant read from standard input. For example,
$ file * 0install.txt: English text base09.000: gzip compressed data - deflate method , original file name , last modified: Mon Aug 23 07:53:21 1993 , max compression os: Unix base09.001: data ...more of same base09.011: DOS executable (COM) man-1.0.cpio: cpio archive tcl7.3.tar.gz: empty tex: directory tk3.6.tar: POSIX tar archive

The information for base09.000 was one output line that wrapped around onto 3 output lines. Most les have certain special values, so-called magic numbers, in specic locations in their headers. le uses a le, usually /etc/magic, which describes these formats. Occasionally it makes a mistakewe can be reasonably sure that the le base09.011 is not a DOS executable, but it has the right number in the right place, and thus fools le. This version of le (from BSD/OS) recognizes base09.000and none of the following pieces of the archive as a gzip archive le, and even extracts a lot of information. Not all versions of le do this. Frequently, it just tells you that the archive is data in this case, the rst assumption should be that the archive is compressed in a format that your version of le doesnt recognize. If the le is packed, compressed or gzipped, gzip expands it, and otherwise it prints an error message, so the next step might look something like:

5 February 2005 02:09

24
$ gunzip < mystery > /tmp/junk $ aha! it didnt complain $ file /tmp/junk /tmp/junk: POSIX tar archive

In this case, we have established that the le mystery is, in fact, a compressed tar archive, though we dont know what kind of compression, since gzip doesnt tell. If le tells you that the le is ASCII or English text, then you can safely look at it with more or less:
$ more strange-file Newsgroups: comp.sources.unix From: [email protected] (Chris Lewis) Subject: v26i014: psroff 3.0, Patch09 Sender: [email protected] Approved: [email protected] Submitted-By: [email protected] (Chris Lewis) Posting-Number: Volume 26, Issue 14 Archive-Name: psroff3.0/patch9 This is official patch 09 for Psroff 3.0. ... intervening lines skipped [email protected] (Chris Lewis)

Patchwrapped: 920128230528 Index: ./lib/lj3.fonts *** /tmp/PATCHold/./lib/lj3.fonts Tue Jan 28 23:03:45 1992 --- ./lib/lj3.fonts Tue Jan 28 23:03:46 1992

This is a plain text patch le: you can pass it straight through the patch program, since patch doesnt worry about junk at the beginning or the end of the le. Well look at patch in depth in Chapter 3, Care and feeding of source trees, page 30.
Newsgroups: comp.sources.unix From: [email protected] (Larry McVoy) Subject: v26i020: perfmon - interface to rstatd(8) Sender: [email protected] Approved: [email protected] ... more stuff omitted #! /bin/sh # This is a shell archive. Remove anything before this line, # then unpack it by saving it into a file and typing "sh file".

As the text tells you, this is a shell archive. To extract it, you can remove all text up to the line starting with #!/bin/sh and extract it with the Bourne shell, or pass it through unshar as it is.
begin 666 magic.gz MXL("_!NRTV5A<W1E<@!-4KV.VS,WO,4W(N;:\9:B+3)T.*1HT*DH M<+3$V+I(HB*2?/V)14W=YMED-\OGW8HE0K0.#[![V/A!4B<(M4_>1C>ZTS MNW&$:<D5>!J9_(0\@:@C?SJ#SU@]IP7V&4L6V=TOAF?Y[N%C#U\@D0B. M!%/PGK+NV[)A\/!*KH)C3[:,!<>"R9T<<KGZC3Z4K9*VUE&B.O"C?H&Q4 MA+,8C"(I2&&/((7&H?![;JX4O0?X]$Y)!\HR3\%U.FT(TE#I>#0YE$*M

5 February 2005 02:09

Chapter 2: Unpacking the goodies

MU$C>%#UPT>&L?WY\ZQKNU_[_S</SN@1226061"15.!K);DF4#4RHFD7 M2;/R8BI/=)5:U*1TMG\W>C=O0PJF]N:(U[L45\B*NIIGPDN%..49+$T%8 MXA7>ZEWS"B;<\3+%O30(.%[%8)TK&<I/O6[6\!M>TPDM"U1+Y3%NXA#K! M28*%RR?MZKA6:NWI5L?&&UM7I1>8,(S05K<!(D+44<N&E$R;OKD%#7!-P M<?66PQR.R73X>E,D0U_"QFUP@YFCJ$&IVST=)2L0:-OH%(QNHF:MMI$>O8 I3#PH#VM<#H4>_]<O$)*>PYU)JPJE7>;*:>5!)4S]9O,/(PQ?IS4#!I end

25

This is a uuencoded le. The rst line contains the word begin, the default security (which you cant change) and the name of the archive (magic.gz). The following lines usually have the same length and begin with the same letter (usually M)this is the encoded length specication for the line. If they dont, something has probably gone wrong in the transmission. The last data line is usually shorter, and thus has a different rst character. Finally, the archive contains two end lines: the rst is usually the single character , and the second is the word end on a line by itself. To extract the le, rst pass it through uudecode, which will create the le magic.gz, then gunzip it to create the le magic. Then you might need to use le to nd out what it is.
$ uudecode < magic.uue $ gunzip magic.gz $ file magic magic: English text

Dont confuse uuencode format with this:

xbtoa5 78 puzzle.gz Begin +,C1(V%L;!!?e@F*(u6!)69ODSn.:h/s&KF-$KGlWA8mP,0BTe$Y<$qSODDdUZO:_0iqn&P/S%8H [AX_&!0:k0$N5WjWlkG?U*XLRJ6"1SE;mJ.kEa#$EL9q3*Bb.c9J@t/K/N>62BM=7Ujbp7$YHN ,m"%IZ93t15j%OV"_S#NMI4;GC_N=%+k5LX,A*uli>IBE@i0T4cP/A#coB""a]![8jgS1L=p6Kit X9EU5N%+(>-N=YU4(aeoGoFH9SqM6#c1(r;;K<aBE/aZRX/:.cbh&9[r.f3bpQJQ&fW:*S_7DW9 6No0QkC7@A0?=YtSYlAc@01eeX;bF/9%&4E627AA6GR!u]3?Zhke.l4*T=U@TF9@1Gs4\jQPjbBm\H K24N:$HKre7#7#jG"KFmedjs!<<*"N xbtoa End N 331 14b E 5c S 75b7 R b506b514

This is a btoa encoded le, probably also gzipped like the previous example. Extract it with btoa -a and then proceed as with uuencoded les.

Whats in that archive?

Now you have discovered the format of the archive and can extract the les from it. Theres a possibility, though, that you dont know what the archive is good for. This is frequently the case if you have a tape or a CD-ROM of an ftp server, and it contains some cryptic names that suggest the les might possibly be of interest. How do you nd out what the package does?

README
By convention, many authors include a le README in the main directory of the package. README should tell you at least:

5 February 2005 02:09

26

The name of the package, and what it is intended to do. The conditions under which you may use it.

For example, the README le for GNU termcap reads:

This is the GNU termcap library -- a library of C functions that enable programs to send control strings to terminals in a way independent of the terminal type. Most of this package is also distributed with GNU Emacs, but it is available in this separate distribution to make it easier to install as -ltermcap. The GNU termcap library does not place an arbitrary limit on the size of termcap entries, unlike most other termcap libraries. See the file INSTALL for compilation and installation instructions. Please report any bugs in this library to [email protected]. You can check which version of the library you have by using the RCS ident command on libtermcap.a.

In some cases, however, there doesnt seem to be any le to tell you what the package does. Sometimes you may be lucky and nd a good man page or even documentation intended to be printed as hardcopysee Chapter 7, Documentation for more information. In many cases, though, you might be justied in deciding that the package is so badly documented that you give up. There may also be les with names like README.BSD, README.SYSV, README.X11 and such. If present, these will usually give specic advice to people using these platforms.

INSTALL file
There may be a separate INSTALL le, or the information it should contain might be included in the README le. It should tell you: A list of the platforms on which the package has been ported. This list may or may not include your system, but either way it should give you a rst inkling of the effort that lies in store. If youre running System V.4, for example, and it has already been ported to your hardware platform running System V.3, then it should be easy. If it has been ported to V.4, and youre running V.3, this can be a completely different matter. A description of how to congure the package (well look at this in Chapter 4, Package conguration). A description of how to build the package (see Chapter 4, Package conguration and Chapter 19, Make for more details on this subject).

It may, in addition, offer suggestions on how to port to other platforms or architectures.

5 February 2005 02:09

Chapter 2: Unpacking the goodies

27

Other files
The package may include other information les as well. By convention, the names are written in upper case or with an initial capital letter, so that they will be stand out in a directory listing. The GNU project software may include some or all of the following les: ABOUT is an alternative name used instead of README by some authors. COPYING and COPYING.LIB are legal texts describing the constraints under which you may use the software. ChangeLog is a list of changes to the software. This name is hard-coded into the emacs editor macros, so its a good chance that a le with this name will really be an emacsstyle change log. MANIFEST may give you a list of the les intended to be in the package. PROBLEMS may help you if you run into problems. SERVICE is supplied by the Free Software Foundation to point you to companies and individuals who can help you if you run into trouble.

A good example of these les is the root directory of Taylor uucp:

$ gunzip </cd0/gnu/uucp/uucp-1.05.tar.gz |tar tvf drwxrwxr-x 269/15 0 May 6 06:10 1994 uucp-1.05/ -r--r--r-- 269/15 17976 May 6 05:23 1994 uucp-1.05/COPYING -r--r--r-- 269/15 163997 May 6 05:24 1994 uucp-1.05/ChangeLog C$

This archive adheres to the GNU convention of including the name of the top-level directory in the archive. When we extract the archive, tar will create a new directory uucp-1.05 and put all the les in it. So we continue:
$ cd /porting/src the directory in which I do my porting $ gunzip </cd0/gnu/uucp/uucp-1.05.tar.gz |tar xf $

After extraction, the resultant directory contains most of the standard les that we discussed above:
$ cd uucp-1.05 $ ls -l total 1724 drwxrwxr-x 7 drwxrwxrwx 44 -r--r--r-1 -r--r--r-1 -r--r--r-1 -rw-r--r-1 -r--r--r-1 -r--r--r-1 -r--r--r-1 -r--r--r-1

grog grog grog grog grog grog grog grog grog grog

wheel wheel wheel wheel wheel wheel wheel wheel wheel wheel

1536 3584 17976 163997 499 14452 4283 7744 23563 32866

May Aug May May May May May May May May

6 19 6 6 6 6 6 6 6 6

06:10 14:34 05:23 05:24 05:24 06:09 05:24 05:24 05:24 05:24

. .. COPYING ChangeLog MANIFEST Makefile.in NEWS README TODO chat.c

5 February 2005 02:09

28
-r--r--r--rwxrwxr-x -r--r--r-...etc 1 grog 1 grog 1 grog wheel wheel wheel 19032 May 6 05:24 config.h.in 87203 May 6 05:27 configure 11359 May 6 05:24 configure.in

5 February 2005 02:09

3
Care and feeding of source trees
In Chapter 2, Unpacking the goodies, we saw how to create an initial source tree. It wont stay in this form for long. During a port, the source tree is constantly changing: Before you can even start, you may apply patches to the tree to bring it up to date. After unpacking and possibly patching, you may nd that you have to clean out junk left behind from a previous port. In order to get it to compile in your environment, you perform some form of conguration, which modies the tree to some extent. Well look at package conguration in Chapter 4, Package conguration. During compilation, you add many new les to the tree. You may also create new subdirectories. After installation, you remove the unneeded les, for example object les and possibly the nal installed les. After cleaning up, you may decide to archive the tree again to save space on disk.

Modifying the source tree brings uncertainty with it: what is original, what have I modied, how do I remove the changes I have made and get back to a clean, well-dened starting point? In this chapter well look at how to get to a clean starting point. Usually this will be the case after you have extracted the source archive, but frequently you need to add patches or remove junk. Well also look at how to build a tree with sources on CD-ROM, how to recognize the changes you have made and how to maintain multiple versions of your software.

Updating old archives

You dont always need to get a complete package: another possibility is that you might already have an older version of the package. If it is large again, for example, the GNU C compiler you might nd it better to get patches and update the source tree. Strictly speaking, a patch is any kind of modication to a source or object le. In UNIX parlance, its almost always a diff, a le that describes how to modify a source le to produce a newer version. Diffs are almost always produced by the diff program, which we describe in Chapter 10,
29

5 February 2005 02:09

30

Where to go from here, page 144. In our case study, we have gcc version 2.5.6 and want to update to 2.5.8. We discover the following les on the le server:
ftp> ls 200 PORT command successful. 150 Opening ASCII mode data connection for -rw-rw-r-- 1 117 1001 10753 Dec 12 19:15 -rw-rw-r-- 1 117 1001 14726 Jan 24 09:02 -rw-rw-r-- 1 117 1001 5955006 Dec 22 14:16 -rw-rw-r-- 1 117 1001 5997896 Jan 24 09:03 226 Transfer complete. ftp>

/bin/ls. gcc-2.5.6-2.5.7.diff.gz gcc-2.5.7-2.5.8.diff.gz gcc-2.5.7.tar.gz gcc-2.5.8.tar.gz

In other words, we have the choice of copying the two diff les gcc-2.5.6-2.5.7.diff.gz and gcc-2.5.7-2.5.8.diff.gz, a total of 25 kB, and applying them to your source tree, or copying the complete 6 MB archive gcc-2.5.8.tar.gz.

Patch
diff les are reasonably understandable, and you can apply the patches by hand if you want, but its obviously easier and safer to use a program to apply the changes. This is the purpose of patch. patch takes the output of the program diff and uses it to update one or more les. To apply the patch, it proceeds as follows: 1. First, it looks for a le header. If it nds any junk before the le header, it skips it and prints a message to say that it has done so. It uses the le header to recognize the kind of diff to apply. It renames the old le by appending a string to its name. By default, the string is .orig, so foo.c would become foo.c.orig. It then creates a new le with the name of the old le, and copies the old le to the new le, modifying it with the patches as it goes. Each set of changes is called a hunk.

2. 3.

The way patch applies the patch depends on the format. The most dangerous kind are ed style diffs, because there is no way to be sure that the text is being replaced correctly. With context diffs, it can check that the context is correct, and will look a couple of lines in each direction if it doesnt nd the old text where it expects it. You can set the number of lines it will look (the fuzz factor) with the -F ag. It defaults to 2. If the old version of the le does not correspond exactly to the old version used to make the diff, patch may not be able to nd the correct place to insert the patch. Except for ed format diffs, it will recognize when this happens, and will print an error message and move the corresponding hunk to a le with the sufx .rej (for reject). A typical example are the patches for X11R5. You might start with the sources supplied on the companion CD-ROM to X Window System Administrators Guide by Linda Mui and Eric Pearce. This CD-ROM includes the complete X11R5 sources to patch level 21. At the time of writing, ve further patches to X11R5 have been released. To bring the source tree up to patch level 26, you would proceed as follows:

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

31

First, read the header of the patch le. As we have seen, patch allows text before the rst le header, and the headers frequently contain useful information. Looking at patch 22, we see:
$ gunzip < /cd0/x11r5/fix22.gz | more X11 R5 Public Patch #22 MIT X Consortium To apply this patch: cd to the top of the source tree (to the directory containing the "mit" and "contrib" subdirectories) and do: patch -p -s < ThisFile Patch works silently unless an error occurs. You are likely to get the following warning messages, which you can ignore:

In this example we have used gunzip to look at the le directly; we could just as well have used GNU zcat. The patch header suggests the ags -s and -p. The -s ag to patch tells it to perform its work silentlyotherwise it prints out lots of information about what it is doing and why. The -p ag is one of the most complicated to use: it species the pathname strip count, how to treat the directory part of the le names in the header. Well look at it in more detail in the section Cant nd le to patch on page 36. This information is important: patch is rather like a chainsaw without a guard, and if you start it without knowing what you are doing, you can make a real mess of its environment. In this case, we should nd that the root of our source tree looks like:
$ cd /usr/x11r5 $ ls -FC mit Imakefile RELNOTES.ms extensions/ rgb/ LABEL bug-report fonts/ server/ Makefile clients/ hardcopy/ util/ Makefile.ini config/ include/ RELNOTES.PS demos/ lib/ RELNOTES.TXT doc/ man/ ... that looks OK, were in the right place $ gunzip < /cd0/x11r5/fix22.gz | patch -p -s

Weve taken another liberty in this example: since the patch le was on CD-ROM in compressed form, we would have needed to extract it to disk in order to patch the way the le header suggests. Instead, we just gunzip directly into the patch program. Its easy to make mistakes when patching. If you try to apply a patch twice, patch will notice, but you can persuade it to reapply the patch anyway. In this section, well look at the havoc that can occur as a result. In addition, well disregard some of the advice in the patch header. This is the way I prefer to do it:
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> patch.log

This invocation allows patch to say what it has to say (no -s ag), but copies both the standard output and the error output to the le patch.log, so nothing appears on the screen. You can, of course, pipe the output through the tee program, but in practice things happen so fast

5 February 2005 02:09

32

that any error message will usually run off the screen before you can read it. It certainly would have done so here: patch.log had a length of 930 lines. It starts with
Hmm... Looks like a new-style context diff to me... The text leading up to this was: -------------------------| Release 5 Public Patch #23 | MIT X Consortium ... followed by the complete header |Prereq: public-patch-22

This last line is one safeguard that patch offers to ensure that you are working with the correct source tree. If patch nds a Prereq: line in the le header, it checks that this text appears in the input le. For comparison, heres the header of mit/bug-report:
To: [email protected] Subject: [area]: [synopsis] [replace with actual area and short description]

VERSION: R5, public-patch-22 [MIT public patches will edit this line to indicate the patch level]

In this case, patch nds the text. When it does, it prints out the corresponding message:
| |*** /tmp/,RCSt1006225 Tue Mar 9 14:40:48 1993 |--- mit/bug-report Tue Mar 9 14:37:04 1993 -------------------------Good. This file appears to be the public-patch-22 version.

This message shows that it has found the text in mit/bug-report. The rst hunk in any X11 diff changes this text (in this case to public-patch-23), so that it will notice a repeated application of the patch. Continuing,
Patching file mit/bug-report using Plan A... Hunk #1 succeeded at 2. Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/,RCSt1005203 Tue Mar 9 13:45:42 1993 |--- mit/lib/X/Imakefile Tue Mar 9 13:45:45 1993 -------------------------Patching file mit/lib/X/Imakefile using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 856. Hunk #3 succeeded at 883. Hunk #4 succeeded at 891. Hunk #5 succeeded at 929. Hunk #6 succeeded at 943. Hunk #7 succeeded at 968. Hunk #8 succeeded at 976. Hmm... The next patch looks like a new-style context diff to me...

This output goes on for hundreds of lines. What happens if you make a mistake and try

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

33

again?
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> patch.log This file doesnt appear to be the public-patch-22 version--patch anyway? [n] y bad choice... Reversed (or previously applied) patch detected! Assume -R? [y] RETURN pressed Reversed (or previously applied) patch detected! Assume -R? [y] RETURN pressed Reversed (or previously applied) patch detected! Assume -R? [y] C$

The rst message is printed because patch didnt nd the text public-patch-22 in the le (in the previous step, patch changed it to read public-patch-23). This message also appears in patch.log. Of course, in any normal application you should immediately stop and check whats gone wrong. In this case, I make the incorrect choice and go ahead with the patch. Worse still, I entered RETURN to the next two prompts. Finally, I came to my senses and hit CTRL-C, the interrupt character on my machine, to stop patch. The result of this is that patch removed the patches in the rst two les (the -R ag tells patch to behave as if the les were reversed, which has the same effect as removing already applied patches). I now have the rst two les patched to patch level 22, and the others patched to patch level 23. Clearly, I cant leave things like this. Two wrongs dont normally make a right, but in this case they do. We do it again, and what we get this time looks pretty much the same as the time before:
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> mit/patch.log Reversed (or previously applied) patch detected! Assume -R? [y] C$

In fact, this time things went right, as we can see by looking at patch.log:
|*** /tmp/,RCSt1006225 Tue Mar 9 14:40:48 1993 |--- mit/bug-report Tue Mar 9 14:37:04 1993 -------------------------Good. This file appears to be the public-patch-22 version. Patching file mit/bug-report using Plan A... Hunk #1 succeeded at 2. Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/,RCSt1005203 Tue Mar 9 13:45:42 1993 |--- mit/lib/X/Imakefile Tue Mar 9 13:45:45 1993 -------------------------Patching file mit/lib/X/Imakefile using Plan A... Hunk #1 succeeded at 1. (lots of hunks succeed) Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/d03300 Tue Mar 9 09:16:46 1993 |--- mit/lib/X/Ximp/XimpLCUtil.c Tue Mar 9 09:16:41 1993 -------------------------Patching file mit/lib/X/Ximp/XimpLCUtil.c using Plan A... Reversed (or previously applied) patch detected! Assume -R? [y]

This time the rst two les have been patched back to patch level 23, and we stop before

5 February 2005 02:09

34

doing any further damage.

Hunk #3 failed
Patch makes an implicit assumption that the patch was created from an identical source tree. This is not always the caseyou may have changed something in the course of the port. The differences frequently dont cause problems if they are an area unrelated to the patch. In this example, well look at how things can go wrong. Lets consider the following situation: during a previous port of X11R5 pl 22,* you ran into some problems in mit/lib/Xt/Selection.c and xed them. The original text read:
if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { Widget w = XtWindowToWidget(dpy, window); RequestWindowRec *requestWindowRec; if (w != NULL && w != widget) widget = w; if (selectWindowContext == 0) selectWindowContext = XUniqueContext();

You had problems with this section, so you commented out a couple of lines:
if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { /* This doesnt make any sense at all - ignore * Widget w = XtWindowToWidget(dpy, window); */ RequestWindowRec *requestWindowRec; /* if (w != NULL && w != widget) widget = w; */ if (selectWindowContext == 0) selectWindowContext = XUniqueContext();

Back in the present, you try to apply patch 24 to this le:

$ gunzip < /cd0/x11r5/fix24.gz | patch -p &> mit/patch.log $

So far so good. But in patch.log we nd

|*** /tmp/da4854 Mon May 17 18:19:57 1993 |--- mit/lib/Xt/Selection.c Mon May 17 18:19:56 1993 -------------------------Patching file mit/lib/Xt/Selection.c using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 70. Hunk #3 failed at 361. Hunk #4 succeeded at 1084. Hunk #5 succeeded at 1199. 1 out of 5 hunks failed--saving rejects to mit/lib/Xt/Selection.c.rej

What does this mean? Theres nothing for it but to look at the les concerned. In x24 we nd
* The abbreviation pl is frequently used to mean patch level.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

*** /tmp/da4854 Mon May 17 18:19:57 1993 --- mit/lib/Xt/Selection.c Mon May 17 18:19:56 1993 *************** *** 1,4 **** this must be hunk 1 ! /* $XConsortium: Selection.c,v 1.74 92/11/13 17:40:46 converse Exp $ */ /*********************************************************** Copyright 1987, 1988 by Digital Equipment Corporation, Maynard, Massachusetts, --- 1,4 ---! /* $XConsortium: Selection.c,v 1.78 93/05/13 11:09:15 converse Exp $ */ /*********************************************************** Copyright 1987, 1988 by Digital Equipment Corporation, Maynard, Massachusetts, *************** *** 70,75 **** --- 70,90 ---this must be hunk 2 Widget w; /* unused */ *************** *** 346,359 **** and this must be hunk 3, the one that failed { Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = req->widget; ... etc *************** *** 1068,1073 **** --- 1084,1096 ---hunk 4 *************** *** 1176,1181 **** --- 1199,1213 ---and hunk 5--at least the count is correct

35

patch put the rejects in Selection.c.rej. Lets look at it:

*************** *** 346,359 **** { Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = req->widget; if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { Widget w = XtWindowToWidget(dpy, window); RequestWindowRec *requestWindowRec; if (w != NULL && w != widget) widget = w; if (selectWindowContext == 0) selectWindowContext = XUniqueContext(); if (XFindContext(dpy, window, selectWindowContext,

! -

5 February 2005 02:09

36
--- 361,375 ---{ Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = XtWindowToWidget(dpy, window); + + + ! if (widget != NULL) req->widget = widget; else widget = req->widget; if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, False, proc, closure); else { RequestWindowRec *requestWindowRec; if (selectWindowContext == 0) selectWindowContext = XUniqueContext(); if (XFindContext(dpy, window, selectWindowContext,

The characters + and - at the beginning of the lines in this hunk identify it as a unied context diff. Well look at them in more detail in Chapter 10, Where to go from here, page 147. Not surprisingly, they are the contents of hunk 3. Because of our x, patch couldnt nd the old text and thus couldnt process this hunk. In this case, the easiest thing to do is to perform the x by hand. To do so, we need to look at the partially xed le that patch created, mit/lib/Xt/Selection.c. The line numbers have changed, of course, but since hunk 3 wasnt applied, we nd exactly the same text as in mit/lib/Xt/Selection.c.orig, only now it starts at line 366. We can effectively replace it by the after text in Selection.c.rej, remembering of course to remove the indicator characters in column 1.

Cant find file to patch

Sometimes youll see a message like:
$ patch -p <hotstuff.diff &>patch.log Enter name of file to patch:

One of the weaknesses of the combination of diff and patch is that its easy to get the le names out of sync. What has probably happened here is that the le names dont agree with your source tree. There are a number of ways for this to go wrong. The way that patch treats the le names in diff headers depends on the -p ag, the so-called pathname strip count: If you omit the -p ag, patch strips all directory name information from the le names and leaves just the lename part. Consider the following diff header:
*** config/sunos4.h --- config/sunos4.h Wed Feb 29 07:13:57 1992 Mon May 17 18:19:56 1993

Relative to the top of the source tree, the le is in the directory cong. If you omit the -p ag, patch will look for the le sunos4.h, not cong/sunos4.h, and will not nd it. If you specify -p, patch keeps the complete names in the headers. If you specify -pn, patch will remove the rst n directory name components in the pathname. This is useful when the diffs contain incorrect base path names. For example, you

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

37

may nd a diff header which looks like:

*** /src/freesoft/gcc-patches/config/sunos4.h Wed Feb 29 07:13:57 1992 --- /src/freesoft/gcc-patches/config/sunos4.h Mon May 17 18:19:56 1993

Unless your source tree also happens to be called /src/freesoft/gcc-patches, patch wont be able to nd the les if you use the -p ag with no argument. Assuming that you are in the root directory of the package (in other words, the parent directory of cong), you really dont want to know about the /src/freesoft/gcc-patches/ component. This pathname consists of four parts: the leading / making the pathname absolute, and the three directory names src, freesoft and gcc-patches. In this case, you can enter
$ patch -p4 <hotstuff.diff &>patch.log

The -p4 tells patch to ignore the rst four pathname components, so it would read thes lenames just as cong/sunos4.h and cong/sunos4.h. In addition to the problem of synchronizing the path names, you may run into broken diffs which dont specify pathnames, even though the les belong to different directories. Well see how easy it is to make this kind of mistake in Chapter 10, Where to go from here, page . For example, you may nd that the diff headers look like:
*** sunos4.h --- sunos4.h Wed Feb 29 07:13:57 1992 Mon May 17 18:19:56 1993

This kind of diff is a real nuisance: you at least need to search for the le sunos4.h, and if youre unlucky youll nd more than one and have to examine the patches to gure out which one is intended. Then you need to give this name to the prompt, and patch should perform the patches. Unfortunately, in a large collection of diffs, this can happen dozens of times.

I cant seem to find a patch in there

Sometimes you will get what looks like a perfectly good unied context diff, but when you run patch against it, you get a message:
$ patch <diffs Hmm... I cant seem to find a patch in there anywhere. $

Some versions of patch dont understand unied diffs, and since all versions skip anything they dont understand, this could be the result. The only thing for it is to get a newer version of patchsee Appendix E, Where to get sources, for details.

Malformed patch
If patch nds the les and understands the headers, you could still run into problems. One of the most common is really a problem in making the diffs:
$ patch <diffs Hmm... Looks like a unified diff to me... The text leading up to this was: --------------------------

5 February 2005 02:09

38
|--- real-programmers.ms Wed Dec 7 13:17:47 1994 |+++ real-programmers.ms Wed Dec 7 14:53:19 1994 -------------------------Patching file real-programmers.ms using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 54. patch: **** malformed patch at line 398: No newline at end of file

Well, it tells you what happened: diff will print this message if the last character in a le is not \n. Most versions of patch dont like the message. You need to edit the diff and remove the offending line.

Debris left behind by patch

At the end of a session, patch leaves behind a number of les. Files of the form lename.orig are the original versions of patched les. The corresponding lenames are the patched versions. The length of the sufx may be a problem if you are using a le system with a limited lename length; you can change it (perhaps to the emacs standard sufx ) with the -b ag. In some versions of patch, is the default. If any patches failed, you will also have les called lename.rej (for rejected). These contain the hunks that patch could not apply. Another common sufx for rejects is #. Again, you can change the sufx, this time with the -r ag. If you have any .rej les, you need to look at them and nd out what went wrong. Its a good idea to keep the .orig les until youre sure that the patches have all worked as indicated.

Pruning the tree

Making clean distribution directories is notoriously difcult, and there is frequently irrelevant junk in the archive. For example, all emacs distributions for at least the last 6 years have included a le etc/COOKIES. As you might guess from the name, this le is a recipe for cookies, based on a story that went round Usenet years ago. This le is not just present in the source tree: since the whole subdirectory etc gets installed when you install emacs, you end up installing this recipe as well. This particular directory contains a surprising number of les, some of them quite amusing, which dont really have much to do with emacs. This is a rather extreme case of a common problem: you dont need some of the les on the distribution, so you could delete them. As far as I know, emacs works just as well without the cookie recipe, but in many cases, you cant be as sure. In addition, you might run into other problems: the GNU General Public License requires you to be prepared to distribute the complete contents of the source tree if so requested. You may think that its an accident that the cookie recipe is in the source tree, but in fact its a political statement*, and you are required by the terms of the GNU General Public License to keep the le in order to give it to anybody who might want it.
* To quote the beginning of the le: Someone sent this in from California, and we decided to extend our campaign against information hoarding to recipes as well as software. (Recipes are the closest thing, not involving computers, to software.)

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

39

This is a rather extreme example, but you might nd any of the following in overgrown trees: Old objects, editor backups and core dumps from previous builds. They may or may not go away with a make clean. Test programs left behind by somebody trying to get the thing to work on his platform. These probably will not go away with a make clean. Formatted documentation. Although the Makele should treat documents like objects when cleaning the tree, a surprising number of packages format and install documentation, and then forget about it when it comes to tidying it away again. Old mail messages, only possibly related to the package. I dont know why this is, but mail messages seem to be the last thing anybody wants to remove, and so they continue to exist for years in many trees. This problem seems to be worse in proprietary packages than in free packages.

The old objects are denitely the worst problem: make cant tell that they dont belong to this conguration, and so they just prevent the correct version of the object being built. Depending on how different the architectures are, you may even nd that the bogus objects fool the linker, too, and you run into bizarre problems when you try to execute.

Save the cleaned archive

If you had to go to any trouble (patches or cleanup) to get to a clean starting point for the port, save the cleaned archive. You wont need it again, of course, but Murphys law will ensure that if you dont save it, you will need it again.

Handling trees on CD-ROM

Its convenient to have your source tree on CD-ROM: you save disk space, and you can be sure that you dont accidentally change anything. Unfortunately, you also cant deliberately change anything. Normal Makeles expect to put their objects in the source tree, so this complicates the build process signicantly. In the next two sections, well look at a couple of techniques that address this problem. Both use symbolic links.

Link trees
You can simulate a writeable tree on disk by creating symbolic links to the sources on CDROM. This way, the sources remain on the CD-ROM, but the objects get written to disk. From your viewpoint, it looks as if all the les are in the same directory. For example, assume you have a CD-ROM with a directory /cd0/src/nd containing the sources to nd:
$ ls -FC /cd0/src/find COPYING Makefile COPYING.LIB Makefile.in ChangeLog NEWS config.status* lib/ configure* locate/ configure.in man/

5 February 2005 02:09

40
INSTALL README find/ xargs/

The / at the end of the le names indicate that these les are directories; the * indicates that they are executables. You could create a link tree with the following commands:
$ cd /home/mysrc/find put the links here $ for i in /cd0/src/find/*; do > ln -s $i . > done $ ls -l see what we got total 16 lrwxrwxrwx COPYING -> /cd0/src/find/COPYING lrwxrwxrwx COPYING.LIB -> /cd0/src/find/COPYING.LIB lrwxrwxrwx ChangeLog -> /cd0/src/find/ChangeLog lrwxrwxrwx INSTALL -> /cd0/src/find/INSTALL lrwxrwxrwx Makefile -> /cd0/src/find/Makefile lrwxrwxrwx Makefile.in -> /cd0/src/find/Makefile.in lrwxrwxrwx NEWS -> /cd0/src/find/NEWS lrwxrwxrwx README -> /cd0/src/find/README lrwxrwxrwx config.status -> /cd0/src/find/config.status lrwxrwxrwx configure -> /cd0/src/find/configure lrwxrwxrwx configure.in -> /cd0/src/find/configure.in lrwxrwxrwx find -> /cd0/src/find/find lrwxrwxrwx lib -> /cd0/src/find/lib lrwxrwxrwx locate -> /cd0/src/find/locate lrwxrwxrwx man -> /cd0/src/find/man lrwxrwxrwx xargs -> /cd0/src/find/xargs

I omitted most of the information that is printed by ls -l in order to get the information on the page: what interests us here is that all the les, including the directories, are symbolic links. In some cases, this is what we want: we dont need to create copies of the directories on the hard disk when a single link to a directory on the CD-ROM does it just as well. In this case, unfortunately, thats not the way it is: our sources are in the directory nd, and thats where we will have to write our objects. We need to do the whole thing again for the subdirectory nd:
$ cd mysource/find change to the source directory on disk $ rm find get rid of the directory symlink $ mkdir find and make a directory $ cd find and change to it $ for i in /cd0/src/find/find/*; do > ln -s $i . > done $ ls -l total 18 lrwxrwxrwx Makefile -> /cd0/src/find/find/Makefile lrwxrwxrwx Makefile.in -> /cd0/src/find/find/Makefile.in lrwxrwxrwx defs.h -> /cd0/src/find/find/defs.h lrwxrwxrwx find -> /cd0/src/find/find/find lrwxrwxrwx find.c -> /cd0/src/find/find/find.c lrwxrwxrwx fstype.c -> /cd0/src/find/find/fstype.c lrwxrwxrwx parser.c -> /cd0/src/find/find/parser.c lrwxrwxrwx pred.c -> /cd0/src/find/find/pred.c lrwxrwxrwx tree.c -> /cd0/src/find/find/tree.c

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

lrwxrwxrwx util.c -> /cd0/src/find/find/util.c lrwxrwxrwx version.c -> /cd0/src/find/find/version.c

41

Yes, this tree really does have a directory called nd/nd/nd, but we dont need to worry about it. Our sources and our Makele are here. We should now be able to move back to the top-level directory and perform the make:
$ cd .. $ make

This is a relatively simple example, but it shows two important aspects of the technique: You dont need to create a symlink for every single le. Although symlinks are relatively small in this case, less than 100 bytesthey occupy up to 1024 bytes of disk space per link, and you can easily nd yourself taking up a megabyte of space just for the links. On the other hand, you do need to make all the directories where output from the build process is stored. You need to make symlinks to the existing les in these directories.

An additional problem with this technique is that many tools dont test whether they have succeeded in creating their output les. If they try to create les on CD-ROM and dont notice that they have failed, you may get some strange and misleading error messages later on.

Object links on CD-ROM

Some CD-ROMs, notably those derived from the Berkeley Net/2 release, have a much better idea: the CD-ROM already contains a symlink to a directory where the object les are stored. For example, the FreeBSD 1.1 CD-ROM version of nd is stored on /cd0/lesys/usr/src/usr.bin/nd and contains:
total 106 drwxrwxr-x 2 bin 2048 Oct 28 1993 . drwxrwxr-x 153 bin 18432 Nov 15 23:28 .. -rw-rw-r-1 bin 168 Jul 29 1993 Makefile -rw-rw-r-1 bin 3157 Jul 29 1993 extern.h -rw-rw-r-1 bin 13620 Sep 7 1993 find.1 -rw-rw-r-1 bin 5453 Jul 29 1993 find.c -rw-rw-r-1 bin 4183 Jul 29 1993 find.h -rw-rw-r-1 bin 20736 Sep 7 1993 function.c -rw-rw-r-1 bin 3756 Oct 17 1993 ls.c -rw-rw-r-1 bin 3555 Jul 29 1993 main.c -rw-rw-r-1 bin 3507 Jul 29 1993 misc.c lrwxrwxr-x 1 root 21 Oct 28 1993 obj -> /usr/obj/usr.bin/find -rw-rw-r-1 bin 7766 Jul 29 1993 operator.c -rw-rw-r-1 bin 4657 Jul 29 1993 option.c -rw-rw-r-1 root 2975 Oct 28 1993 tags

All you have to do in this case is to create a directory called /usr/obj/usr.bin/nd. The Makeles are set up to compile into that directory.

5 February 2005 02:09

42

Tracking changes to the tree

The most obvious modication that you make to a source tree is the process of building: the compiler creates object les* and the loader creates executables. Documentation formatters may produce formatted versions of the source documentation, and possibly other les are created as well. Whatever you do with these les, you need to recognize which ones you have created and which ones you have changed. Well look at these aspects in the following sections.

Timestamps
Its easy enough to recognize les that have been added to the source tree since its creation: since they are all newer than any le in the original source tree, the simple command ls -lt (probably piped into more or less) will display them in the reverse order in which they were created (newest rst) and thus separate the new from the old. Every UNIX le and directory has three timestamps. The le system represents timestamps in the time_t format, the number of seconds elapsed since January 1, 1970 UTC. See Chapter 16, Timekeeping, page 270, for more details. The timestamps are: The last modication timestamp, updated every time the le or directory is modied. This is what most users think of as the le timestamp. You can display it with the ls -l command. The last access timestamp, updated every time a data transfer is made to or from the le. You can display it with the ls -lu command. This timestamp can be useful in a number of different places. The status change timestamp (at least, thats what my header les call it). This is a sort of kludged last modication timestamp for the inode, that part of a le which stores information about the le. The most frequent changes which dont affect the other timestamps are change in the number of links or the permissions, which normally isnt much use to anybody. On the other hand, the inode also contains the other timestamps, so if this rule were enforced rigidly, a change to another timestamp would also change the status change timestamp. This would make it almost completely useless. As a result, most implementations suppress the change to the status change timestamp if only the other timestamps are modied. If you want, you can display the status change timestamp with the ls -lc command.

Whichever timestamp you choose to display with ls -l, you can cause ls to sort by it with the -t ag. Thus, ls -lut displays and sorts by the last access timestamp. Of these three timestamps, the last modication timestamp is by far the most important. There are a number of reasons for this:
* To be pedantic, usually the assembler creates the object les, not the compiler. A kludge is a programming short cut, usually a nasty, untidy one. The New Hackers Dictionary goes to a lot of detail to explain the term, including why it should be spelt kluge and not kludge.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

43

make relies on the last modication timestamp to decide what it needs to compile. If you move the contents of a directory with cp, it changes all the modication timestamps to the time when the copy was performed. If you then type make, you will perform a signicant amount of needless compilation. Its frequently important to establish if two les are in fact the same, in other words, if they have identical content. In the next section well see some programmatic tools that help us with this, but as a rst approximation we can assume that two les with the same name, length and modication timestamp have an identical content, too. The modication timestamp is the most important of these three: you can change the name, but if length and timestamp are the same, theres still a good chance its the same le. If you change the timestamp, you cant rely on the two les being the same just because they have the same name and length. As we have seen above, the last modication timestamp is useful for sorting when you list directories. If youre looking for a le you made the week before last, it helps if it is dated accordingly.

Keeping timestamps straight

Unfortunately, its not as easy to keep timestamps straight. Here are some of the things that can go wrong: If you copy the le somewhere else, traditional versions of cp always set the modication timestamp to the time of copying. ln does not, and neither does mv if it doesnt need to make a physical copy, so either of these are preferable. In addition, more modern versions of cp offer the ag -p (preserve), which preserves the modication timestamp and the permissions. When extracting an archive, cpios default behaviour is to set the modication timestamp to the time of extraction. You can avoid this with the -m ag to cpio. Editing the le changes the modication timestamp. This seems obvious, but you frequently nd that you make a modication to a le to see if it solves a problem. If it doesnt help, you edit the modication out again, leaving the le exactly as it was, except for the modication timestamp, which points to right now. A better strategy is to save the backup le, if the editor keeps one, or otherwise to rename the original le before making the modications, then to rename it back again if you decide not to keep the modications. In a network, its unusual for times to be exactly the same. UNIX machines are not very good at keeping the exact time, and some gain or lose as much as 5 minutes per day. This can cause problems if you are using NFS. You edit your les on one machine, where the clocks are behind, and compile on another, where the clocks are ahead. The result can be that objects created before the last edit still have a modication timestamp that is more recent, and make is fooled into believing that it doesnt need to recompile. Similar problems can occur when one system is running with an incorrect time zone setting.

5 February 2005 02:09

44

cmp
A modication timestamp isnt infallible, of course: even if EOF, timestamp and name are identical, there still can be a lingering doubt as to whether the les really are identical. This doubt becomes more pronounced if you seee something like:
$ ls -l total 503 -rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw... etc

1 1 1 1 1

grog grog grog grog grog

wheel wheel wheel wheel wheel

1326 28871 4259 4515 33690

May May May May May

1 1 1 1 1

01:00 01:00 01:00 01:00 01:00

a29k-pinsn.c a29k-tdep.c a68v-nat.c alpha-nat.c alpha-tdep.c

Its a fairly clear bet that somebody has done a touch on all the les, and their modication timestamps have all been set to midnight on May 1.* The cmp program can give you certainty:
$ cmp foo.c ../orig/foo.c compare with the original $ echo $? show exit status 0 0: all OK $ cmp bar.c ../orig/bar.c bar.c ../orig/bar.c differ: char 1293, line 39 $ echo $? show exit status 1 1: they differ

Remember you can tell the shell to display the exit status of the previous command with the shell variable $?. In the C shell, the corresponding variable is called $status. If the contents of the les are identical, cmp says nothing and returns an exit status 0. If they are, it tells you where they differ and returns 1. You can use the exit status in a shell script. For example, the following Bourne shell script (it doesnt work with csh) compares les that are in both the current tree (which is the current working directory) and the original tree (../orig) and makes a copy of the ones that have changed in the directory ../changed.
$ for i in *; do > if [ -f ../orig/$i ]; then > cmp $i ../orig/$i 2>&1 >/dev/null > if [ $? -ne 0 ]; then > cp -p $i ../changed > fi > fi > done check all files in the directory it is present in the orig tree compare them theyre different make a copy

There are a couple of points to note about this example: Were not interested in where the les differ, or in even seeing the message. We just want to copy the les. As a result, we copy both stdout and stderr of cmp to /dev/null, the UNIX bit bucket.

* Midnight? That looks like 1 a.m. But remember that UNIX timestamps are all in UTC, and thats 1 a.m. in my time zone. This example really was done with touch.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

45

When copying, we use -p to ensure that the timestamps dont get changed again.

An example updating an existing tree

Chances are that before long you will have an old version of gcc on your system, but that you will want to install a newer version. As we saw on page 29, the gzipped archive for gcc is around 6 MB in size, whereas the patches run to 10 KB or 15 KB, so we opt to get diffs from prep.ai.mit.edu to update version 2.6.1 to 2.6.3. Thats pretty straightforward if you have enough disk space: we can duplicate the complete source tree and patch it. Before doing so, we should check the disk space: the gcc source tree with all objects takes up 110 MB of disk space.
$ cd /porting/srcmove to the parent directory $ mkdir gcc-2.6.3 make a directory for the new tree $ cd gcc-2.6.1 move to the old directory $ tar cf - . | (cd ../gcc-2.6.3;tar xf -) and copy all files* $ cd ../gcc-2.6.3 move to new directory $ make clean and start off with a clean slate $ gunzip < /C/incoming/gcc-2.6.1-2.6.2.tar.gz | patch -p | tee patch.log Hmm... Looks like a new-style context diff to me... The text leading up to this was: -------------------------|Changes for GCC version 2.6.2 from version 2.6.1: | |Before applying these diffs, go to the directory gcc-2.6.1. Remove all |files that are not part of the distribution with the command | | make distclean | |Then use the command | | patch -p1 | |feeding it the following diffs as input. Then rename the directory to |gcc-2.6.2, re-run the configure script, and rebuild the compiler. | |diff -rc2P -x c-parse.y -x c-parse.c -x c-parse.h -x c-gperf.h -x cexp.c -x bi-parser.c -x objc-parse.y -x objc-parse.c |-x TAGS -x gcc.?? -x gcc.??s -x gcc.aux -x gcc.info* -x cpp.?? -x cpp.??s -x cpp.aux -x cpp.info* -x cp/parse.c -x cp/pa |rse.h gcc-2.6.1/ChangeLog gcc-2.6.2/ChangeLog |*** gcc-2.6.1/ChangeLog Tue Nov 1 21:32:40 1994 |--- gcc-2.6.2/ChangeLog Sat Nov 12 06:36:04 1994 -------------------------File to patch:

Oops, these patches contain the directory name as well. As the diff header indicates, we can solve this problem by supplying the -p1 ag to patch. We can also solve the problem by
* When moving directories with tar, it may not seem to be important whether you say tar c . or tar c *--but it is. If you say *, you will miss out any le names starting with . (period).

5 February 2005 02:09

46

moving up one level in the directory hierarchy, since we have stuck to the same directory names. This message also reminds us that patch is very verbose, so this time we enter:
$ gunzip < /C/incoming/gcc-2.6.1-2.6.2.tar.gz | patch -p1 -s | tee patch.log 1 out of 6 hunks failed--saving rejects to cccp.c.rej $

What went wrong here? Lets take a look at cccp.c.rej and cccp.c.orig. According to the hunk, line 3281 should be
if (ip->macro != 0)

The hunk wants to change it to

if (output_marks)

However, line 3281 of cccp.orig is

if (output_marks)

In other words, we had already applied this change, probably from a message posted in gnu.gcc.bugs. Although the patch failed, we dont need to worry: all the patches had been applied. Now we have a gcc-2.6.2 source tree in our directory. To upgrade to 2.6.3, we need to apply the next patch:
$ gunzip < /C/incoming/gcc-2.6.2-2.6.3.diff.gz | patch -p1 -s | tee -a patch.log

We use the -a option to patch here to keep both logspossibly overkill in this case. This time there are no errors. After patching, there will be a lot of original les in the directory, along with the one .rej le. We need to decide when to delete the .orig les: if something goes wrong compiling one of the patched les, its nice to have the original around to compare. In our case, though, we have a complete source tree of version 2.6.2 on the same disk, and it contains all the original les, so we can remove the .orig les:
$ find . -name "*.orig" -print | xargs rm

We use xargs instead of -exec rm {} \; because its faster: -exec rm starts a rm process for every le, whereas xargs will put as many le names onto the rm command line as possible. After cleaning up the tree, we back it up. Its taken us a while to create the tree, and if anything goes wrong, wed like to be able to restore it to its initial condition as soon as possible.

5 February 2005 02:09

4
Package conguration
Programs dont run in a vacuum: they interface with the outside world. The view of this outside world differs from location to location: things like host names, system resources, and local conventions will be different. Theoretically, you could change the program sources every time you install a package on a new system, but besides being a pain, its very errorprone. All modern packages supply a method of conguration, a simplied way of adapting the sources to the environment in which the program will run. In this chapter, well look at common conguration conventions. We can divide system differences into one of three categories: The kind of hardware the package will run on. A compiler needs to generate the correct machine instructions, and an X server needs to know how to transfer data to the display hardware. Less well-written programs have hardware dependencies that could have been avoided with some forethought. Well look at this in more detail in Chapter 11, Hardware dependencies. The system software with which it will interact. Differences between UNIX systems are signicant enough that it will be necessary to make certain decisions depending on the system avour. For example, a communications program will need to know what kind of network interface your system has. Programs that come from other systems may need signicant rewriting to conform with UNIX library calls. Well look at these dependencies in part 2 of this book, from Chapter 12, Kernel dependencies to Chapter 21, Object les and friends. The local conguration. These may include obvious things like the system name, aspects of program behaviour, information about tools used locally, or local system conventions.

In this chapter, well look at what local conguration entails, and how we tell the package about our chosen conguration.

47

5 February 2005 02:09

48

Installation paths
Your system conguration may place constraints on where you can install the software. This is not normally a problem for individual systems, but on a large, heterogeneous network it could require more consideration. Traditionally, non-system software has been installed in the hierarchy /usr/local. This is not an sthetically pleasing location: the hierarchy can become quite large, and in a network many systems might share the directory. One of the best thought-out descriptions of a modern le system structure is in the UNIX System V Application Binary Interface, which is also similar to structures used by SunOS and the newer BSD variants. In brief, it species the following top-level directories: / The root directory. /dev The directory tree containing device les. /etc Directory for machine-specic conguration les. /opt Directory for add-on software. /usr This directory used to be the other le system on a UNIX machine. In the System V ABI it has lost most of its importance. The ABI states uses only for /usr/bin and /usr/share, and the name /usr has lost its original meaning: the ABI species /usr only as a location for system les that users may wish to access. /usr/bin is intended for Utility programs and commands for the use of all applications and users. In practice, its better to use this directory for system programs only. /usr/share The System V ABI states that /usr/share is intended for architecture-independent shareable les. In practice, those versions of System V that still have man pages put them in /usr/share/man, and terminfo data are stored in /usr/share/lib/terminfo. The rest of the directory may contain a few other odds and ends, but these two directories make up over 99% of the content. The choice of the location /usr/share is not a happy choice: rstly, it is frequently a separate le system, but it must be mounted on a non-root le system, and secondly the man pages arent really architecture-independent. The choice makes more sense from the point of view of the Unix Systems Group, who are concerned only with pure System V: the man pages are mainly independent of hardware architecture. However, in a real-world net you probably have two or three different operating systems, each with their own man pages. /var This directory contains les that are frequently modied. Typical subdirectories are /var/tmp for temporary les and /var/spool for printer output, uucp and news. The System V ABI does not say anything about where to store user les. The Seventh Edition typically stored them as a subdirectory of /usr, but newer systems have tended to store them in a directory called /home.

5 February 2005 02:09

Chapter 4: Package conguration

49

The /opt hierarchy resembles that of /usr. A typical structure is: /opt/bin for executables. /opt/man for man pages not /opt/share/man, unlike the structure in /usr. /opt/lib for additional les used by executables. In particular, this directory could contain library archives for compilers, as well as the individual passes of the compilers. /opt/<pkg> This is where the System V ABI places individual package data. Not many other systems follow it. /opt/lib/<pkg> This is where most packages place private data. Using the /opt hierarchy has one disadvantage: you may not want to have a separate le system. In modern systems, the solution is simple enough: place the directory where you want it, and create a symbolic link /opt that points to it. This works only if your system has symbolic links, of course, so I have come to a compromise: I use /opt on systems with symbolic links, and /usr/local on systems without symbolic links. Many packages compile pathnames into the code, either because its faster that way, or because its easier. As a result, you should set the path names before compilationdont put off this task until youre ready to install, or you may run into problems where the packages are all nicely installed in the correct place and look for data in the wrong directories.

Preferred tools
Many of the most popular software packages are alternative tools. Free software such as gcc, emacs and perl have become so popular that they are frequently supplied with proprietary system releases, and many other systems have ported them and use them as standard tools. If you want to use such programs, you need to tell the conguration routines about them. Depending on the tools you use, you may also need to change the ags that you pass to them. For example, if you compile with gcc, you may choose to include additional compiler ags such as -fstrength-reduce, which is specic to gcc.

Conveying configuration information

The goal of conguration is to supply the conguration information to the program sources. A good conguration mechanism will hide this from you, but its helpful to understand what its doing. In this section, well look under the covers you can skip it if it looks too technical. There are a number of possible ways to use conguration information: for example, the package may have separate communication modules for STREAMS and sockets, and the conguration routines may decide which of the two modules to compile. More typically, however, the conguration routines convey conguration information to the package by dening preprocessor variables indicating the presence or absence of a specic feature. Many packages provide this information in the make variable CFLAGSfor example, when you make bash, the GNU Bourne Again Shell, you see things like

5 February 2005 02:09

50
$ make gcc -DOS_NAME="FreeBSD" -DProgram=bash -DSYSTEM_NAME="i386" \ -DMAINTAINER="[email protected]" -O -g -DHAVE_SETLINEBUF -DHAVE_VFPRINTF \ -DHAVE_UNISTD_H -DHAVE_STDLIB_H -DHAVE_LIMITS_H -DHAVE_GETGROUPS \ -DHAVE_RESOURCE -DHAVE_SYS_PARAM -DVOID_SIGHANDLER -DOPENDIR_NOT_ROBUST \ -DINT_GROUPS_ARRAY -DHAVE_WAIT_H -DHAVE_GETWD -DHAVE_DUP2 -DHAVE_STRERROR \ -DHAVE_DIRENT -DHAVE_DIRENT_H -DHAVE_STRING_H -DHAVE_VARARGS_H -DHAVE_STRCHR \ -DHAVE_STRCASECMP -DHAVE_DEV_FD -D"i386" -D"FreeBSD" -DSHELL -DHAVE_ALLOCA \ -I. -I. -I././lib/ -c shell.c

The -D arguments pass preprocessor variables that dene the conguration information. An alternative method is to put this information in a le with a name like cong.h. Taylor uucp does it this way: in cong.h you will nd things like:
/* If your compiler supports prototypes, set HAVE_PROTOTYPES to 1. */ #define HAVE_PROTOTYPES 1 /* Set ECHO_PROGRAM to a program which echoes its arguments; if echo is a shell builtin you can just use "echo". */ #define ECHO_PROGRAM "echo" /* The following macros indicate what header files you have. Set the macro to 1 if you have the corresponding header file, or 0 if you do not. */ #define HAVE_STDDEF_H 1 /* <stddef.h> */ #define HAVE_STDARG_H 1 /* <stdarg.h> */ #define HAVE_STRING_H 1 /* <string.h> */

I prefer this approach: you have all the conguration information in one place, it is documented, and its more reliable. Assuming that the Makele dependencies are correct, any change to cong.h will cause the programs to be recompiled on the next make. As we will see in Chapter 5, Building the package, page 68, this usually doesnt happen if you modify the Makele. Typically, conguration information is based on the kind of operating system you run and the kind of hardware you use. For example, if you compile for a Sparc II running SunOS 4.1.3, you might dene sparc to indicate the processor architecture used and sunos4 to indicate the operating system. Since SunOS 4 is basically UNIX, you might also need to dene unix. On an Intel 486 running UnixWare you might need to dene i386 for the processor architecture,* and SVR4 to indicate the operating system. This information is then used in the source les as arguments to preprocessor #ifdef commands. For example, the beginning of each source le, or a general conguration le, might contain:
#ifdef i386 #include "m/i386.h" #endif #ifdef sparc #include "m/sparc.h" #endif

* Why not i486? The processor is an Intel 486, but the architecture is called the i386 architecture. You also use i386 when compiling for a Pentium.

5 February 2005 02:09

Chapter 4: Package conguration

#ifdef sunos4 #include "s/sunos4.h" #endif #ifdef SVR4 #include "s/usg-4.0.h" #endif

51

You can get yourself into real trouble if you dene more than one machine architecture or more than one operating system. Since conguration is usually automated to some extent, the likelihood of this is not very great, but if you end up with lots of double denitions when compiling, this is a possible reason. Conguration through the preprocessor works nicely if the hardware and software both exactly match the expectations of the person who wrote the code. In many cases, this is not the case: looking at the example above, note that the le included for SVR4 is s/usg-4.0.h, which suggests that it is intended for UNIX System V release 4.0. UnixWare is System V release 4.2. Will this work? Maybe. It could be that the conguration mechanism was last revised before System V.4.2 came out. If you nd a le s/usg-4.2.h, its a good idea to use it instead, but otherwise its a matter of trial and error. Most software uses this approach, although it has a number of signicant drawbacks: The choices are not very detailed: for example, most packages dont distinguish between Intel 386 and Intel 486, although the latter has a oating point coprocessor and the former doesnt. There is no general consensus on what abbreviations to use. For UnixWare, you may nd that the correct operating system information is determined by USG (USG is the Unix Systems Group, which, with some interruption,* is responsible for System V), SYSV, SVR4, SYSV_4, SYSV_4_2 or even SVR3. This last can happen when the conguration needed to be updated from System V.2 to System V.3, but not again for System V.4. The choice of operating system is usually determined by just a couple of differences. For example, base System V.3 does not have the system call rename, but most versions of System V.3 that you will nd today have it. System V.4 does have rename. A software writer may use #ifdef SVR4 only to determine whether the system has the rename system call or not. If you are porting this package to a version of System V.3.2 with rename, it might be a better idea to dene SVR4, and not SVR3. Many aspects attributed to the kernel are in fact properties of the system library. As we will see in the introduction to Part 2 of this book, there is a big difference between kernel functionality and library functionality. The assumption is that a specic kernel uses the library with which it is supplied. The situation is changing, though: many companies sell systems without software development tools, and alternative libraries such as the GNU C library are becoming available. Making assumptions about the library based on the kernel was never a good ideanow its completely untenable. For example, the GNU C

* The rst USG was part of AT&T, and was superseded by UNIX Systems Laboratories (USL). After the sale of USL to Novell, USL became Novells UNIX Systems Group.

5 February 2005 02:09

52

library supplies a function rename where needed, so our previous example would fail even on a System V.3 kernel without a rename system call if it uses the GNU C library. As you can imagine, many packages break when compiled with the GNU C library, through their own fault, not that of the library. In the example above, it would make a whole lot more sense to dene a macro HAS_RENAME which can be set if the rename function is present. Some packages use this method, and the GNU project is gradually working towards it, but the majority of packages base their decisions primarily on the combination of machine architecture and operating system. The results of incorrect conguration can be far-reaching and subtle. In many cases, it looks as if there is a bug in the package, and instead of reconguring, you can nd yourself making signicant changes to the source. This can cause it to work for the environment in which it is compiled, but to break it for anything else.

What do I need to change?

A good conguration mechanism should be able to decide the hardware and software dependencies that interest the package, but only you can tell it about the local preferences. For example, which compiler do you use? Where do you want to install the executables? If you dont know the answers to these questions, theres a good chance that youll be happy with the defaults chosen by the conguration routines. On the other hand, you may want to use gcc to compile the package, and to install the package in the /opt hierarchy. In all probability, youll have to tell the conguration routines about this. Some conguration routines will look for gcc explicitly, and will take it if they nd it. In this case, you may have a reason to tell the conguration routines not to use gcc. Some packages have a number of local preferences: for example, do you want the package to run with X11 (and possibly fail if X isnt running)? This sort of information should be in the README le.

Creating configuration information

A number of conguration methods exist, none of them perfect. In most cases you dont get a choice: you use the method that the author of the package decided upon. The rst signicant problem can arise at this point: what method does he use? This is not always easy to gure out it should be described in a le called README or INSTALL or some such, but occasionally you just nd cryptic comments in the Makele. In the rest of this chapter well look at conguration via multiple Makele targets, manual conguration, shell scripts, and imake, the X11 conguration mechanism. In addition, the new BSD make system includes a system of automatic conguration: once it is set up, you dont have to do anything, assuming you already have a suitable Makele. Well look at this method in more detail in Chapter 19, Make, page 323.

5 February 2005 02:09

Chapter 4: Package configuration

53

Multiple Makefile targets

Some packages anticipate every possibility for you and supply a customized Makele. For example, when building unzip, a free uncompression utility compatible with the DOS package PK-ZIP, you would nd:
$ make If youre not sure about the characteristics of your system, try typing "make generic". If the compiler barfs and says something unpleasant about "timezone redefined," try typing "make clean" followed by "make generic2". One of these actions should produce a working copy of unzip on most Unix systems. If you know a bit more about the machine on which you work, you might try "make list" for a list of the specific systems supported herein. And as a last resort, feel free to read the numerous comments within the Makefile itself. Note that to compile the decryption version of UnZip, you must obtain the full versions of crypt.c and crypt.h (see the "Where" file for ftp and mail-server sites). Have an excruciatingly pleasant day.

As the comments suggest, typing make generic should work most of the time. If it doesnt, looking at the Makele reveals a whole host of targets for a number of combined hardware/software platforms. If one of them works for you, and you can nd which one, then this might be an easy way to go. If none does, you might nd yourself faced with some serious Makele rewriting. This method has an additional disadvantage that it might compile with no problems and run into subtle problems when you try to execute itfor example, if the program expects System V sigpause and your system supplies BSD sigpause,* the build process may complete without detecting any problems, but the program will not run correctly, and you might have a lot of trouble nding out why.

Manual configuration
Modifying the Makele or cong.h manually is a better approach than multiple Makele targets. This seemingly arduous method has a number of advantages: You get to see what is being changed. If you have problems with the resultant build, its usually relatively easy to pin-point them. Assuming that the meanings of the parameters are well documented, it can be easier to modify them manually than run an automated procedure that hides much of what it is doing. If you nd you do need to change something, you can usually do it fairly quickly. With an automated script, you may need to go through the whole script to change a single minor parameter.

On the down side, manual conguration requires that you understand the issues involved: you cant do it if you dont understand the build process. In addition, you may need to repeat it every time you get an update of the package, and it is susceptible to error.
* See Chapter 13, Signals, pages 190 and 192 for further information.

5 February 2005 02:09

54

Configuration scripts
Neither multiple Makele targets nor manual modication of the Makele leave you with the warm, fuzzy feeling that everything is going to work correctly. It would be nice to have a more mechanized method to ensure that the package gets the correct information about the environment in which it is to be built. One way to do this is to condense the decisions you need to make in manual conguration into a shell script. Some of these scripts work very well. A whole family of conguration scripts has grown up in the area of electronic mail and news. Heres part of the conguration script for C news, which for some reason is called build:
$ cd conf $ build This interactive command will build shell files named doit.root, doit.bin, doit.news, and again.root to do all the work. It will not actually do anything itself, so feel free to abort and start again. C News wants to keep most of its files under a uid which preferably should be all its own. Its programs, however, can and probably should be owned by another user, typically the same one who owns most of the rest of the system. (Note that on a system running NFS, any program not owned by "root" is a gaping security hole.) What user id should be used for news files [news]? RETURN pressed What group id should be used for news files [news]? RETURN pressed What user id should be used for news programs [bin]? RETURN pressed What group id should be used for news programs [bin]? RETURN pressed Do the C News sources belong to bin [yes]? no You may need to do some of the installation procedures by hand after the software is built; doit.bin assumes that it has the power to create files in the source directories and to update the news programs. It would appear that your system is among the victims of the 4.4BSD / SVR4 directory reorganization, with (e.g.) shared data in /usr/share. Is this correct [yes]? RETURN pressed This will affect where C News directories go. We recommend making the directories wherever they have to go and then making symbolic links to them under the standard names that are used as defaults in the following questions. Should such links be made [yes]? no

We chose not to use the symbolic links: the script doesnt say why this method is recommended, they dont buy us anything, and symbolic links mean increased access time. The conguration script continues with many more questions like this. Well pick it up at various places in the book. The exibility of a shell script is an advantage when checking for system features which are immediately apparent, but most of them require that you go through the whole process from start to nish if you need to modify anything. This can take up to 10 minutes on each occasion, and they are often interactive, so you cant just go away and let it do its thing.

5 February 2005 02:09

Chapter 4: Package configuration

55

GNU package configuration

Most GNU project packages supply another variety of conguration script. For more details, see Programming with GNU Software, by Mike Loukides. GNU conguration scripts sometimes expect you to know the machine architecture and the operating system, but they often attempt to guess if you dont tell them. The main intention of the conguration utility is to gure out which features are present in your particular operating system port, thus avoiding the problems with functions like rename discussed on page 51. Taylor uucp uses this method:
$ sh configure checking how to run the C preprocessor checking whether -traditional is needed checking for install checking for ranlib checking for POSIXized ISC checking for minix/config.h checking for AIX checking for -lseq checking for -lsun checking whether cross-compiling checking for lack of working const checking for prototypes checking if #! works in shell scripts checking for echo program checking for ln -s

see page 351 the install program, page 128 see page Interactive POSIX extensions? MINIX specific IBM UNIX libseq.a needed? libsun.a? see page 339 does the compiler understand function prototypes? is echo a program or a builtin? do we have symbolic links? (page 218)

This method makes life a whole lot easier if the package has already been ported to your particular platform, and if you are prepared to accept the default assumptions that it makes, but can be a real pain if not: You may end up having to modify the conguration scripts, which are not trivial. Its not always easy to congure things you want. In the example above, we accepted the default compiler ags. If you want maximum optimization, and the executables should be installed in /opt/bin instead of the default /usr/local/bin, running congure becomes signicantly more complicated:*
$ CFLAGS="-O3 -g" sh configure --prefix=/opt

The scripts arent perfect. You should really check the resultant Makeles, and you will often nd that you need to modify them. For example, the conguration scripts of many packages, including the GNU debugger, gdb, do not allow you to override the preset value of CFLAGS. In other cases, you can run into a lot of trouble if you do things that the script didnt expect. I once spent a couple of hours trying to gure out the behaviour of the GNU make conguration script when porting to Solaris 2.4:

* This example uses the feature of modern shells of specifying environment variables at the beginning of the command. The program being run is sh, and the denition of CFLAGS is exported only to the program being started.

5 February 2005 02:09

56
$ CFLAGS="O3 -g" configure --prefix=/opt creating cache ./config.cache checking for gcc... gcc checking whether we are using GNU C... yes checking how to run the C preprocessor... gcc -E checking whether cross-compiling... yes

Although this was a normal port, it claimed I was trying to cross-compile. After a lot of experimentation, I discovered that the conguration script checks for cross-compilation by compiling a simple program. If this compilation fails for any reason, the script assumes that it should set up a cross-compilation environment. In this case, I had mistakenly set my CFLAGS to O3 -gof course, I had meant to write -O3 -g. The compiler looked for a le O3 and couldnt nd it, so it failed. The conguration script saw this failure and assumed I was cross-compiling. In most cases, you need to re-run the conguration script every time a package is updated. If the script runs correctly, this is not a problem, but if you need to modify the Makele manually, it can be a pain. For example, gdb creates 12 Makeles. If you want to change the CFLAGS, you will need to modify each of them every time you run congure. Like all conguration scripts, the GNU scripts have the disadvantage of only conguring things they know about. If your man program requires pre-formatted man pages, you may nd that there is no way to congure the package to do what you want, and you end up modifying the Makele after you have built it.

Modifying automatically build Makeles is a pain. An alternative is to modify Makele.in, the raw Makele used by congure. That way, you will not have to redo the modications after each run of congure.

imake
imake is the X11 solution to package conguration. It uses the C preprocessor to convert a number of conguration les into a Makele. Here are the standard les for X11R6: Imake.tmpl is the main conguration le that is passed to the C preprocessor. It is responsible for including all the other conguration les via the preprocessor #include directive. Imake.cf determines the kind of system upon that imake is running. This may be based on preprocessor variables supplied by default to the preprocessor, or on variables compiled in to imake. site.def describes local preferences. This is one of the few les that you should normally consider modifying. As its name implies, <vendor>.cf has a different name for each platform. Imake.tmpl decides which le to include based on the information returned by Imake.cf. For example, on BSD/OS the le bsdi.cf will be included, whereas under SunOS 4 or Solaris 2 the le sun.cf will be included.

5 February 2005 02:09

Chapter 4: Package conguration

57

Imake.rules contains preprocessor macros used to dene the individual Makele targets. Imakele is part of the package, not the imake conguration, and describes the package to imake.

You dont normally run imake directly, since it needs a couple of pathname parameters: instead you have two possibilities: Run xmkmf, which is a one-line script that supplies the parameters to imake. Run make Makele. This assumes that some kind of functinoal Makele is already present in the package.

Strangely, make Makele is the recommended way to create a new Makele. I dont agree: one of the most frequent reasons to make a new Makele is because the old one doesnt work, or because it just plain isnt there. If your imake conguration is messed up, you can easily remove all traces of a functional Makele and have to restore the original version from tape. xmkmf always works, and anyway, its less effort to type. Once you have a Makele, you may not be nished with conguration. If your package contains subdirectories, you may need to create Makeles in the subdirectories as well. In general, the following sequence will build most packages:
$ $ $ $ $ xmkmf make Makefiles make depend make make install run imake against the Imakefile create subordinate Makefiles run makedepend against all Makefiles make the packages install the packages

These commands include no package-dependent parametersthe whole sequence can be run as a shell script. Well, yes, there are minor variations: make Makeles fails if there are no subordinate Makeles to be made, and sometimes you have targets like a make World instead of make or make all, but in general its very straightforward. If your imake conguration les are set up correctly, and the package that you are porting contains no obscenities, this is all you need to know about imake, which saves a lot of time and is good for your state of mind. Otherwise, check Software Portability with imake, by Paul DuBois, for the gory details.

5 February 2005 02:09

5
Building the package
Now we have congured our package and were ready to build. This is the Big Moment: at the end of the build process we should have a complete, functioning software product in our source tree. In this chapter, well look at the surprises that make can have in store for you. You can nd the corresponding theoretical material in Chapter 19, Make.

Preparation
If youre unlucky, a port can go seriously wrong. The rst time that error messages appear thick and fast and scroll off the screen before you can read them, you could get the impression that the packages were built this way deliberately to annoy you. A little bit of preparation can go a long way towards keeping you in control of whats going on. Here are some suggestions:

Make sure you have enough space

One of the most frequent reasons of failure of a build is that the le system lls up. If possible, ensure that you have enough space before you start. The trouble is, how much is enough? Hardly any package will tell you how much space you need, and if it does it will probably be wrong, since the size depends greatly on the platform. If you are short on space, consider compiling without debugging symbols (which take up a lot of space). If you do run out of space in the middle of a build, you might be able to save the day by stripping the objects with strip, in other words removing the symbols from the le.

Use a windowing system

The sheer size of a complicated port can be a problem. Like program development, porting tends to be an iterative activity. You edit a le, compile, link, test, go back to edit the le, and so on. Its not uncommon to nd yourself having to compare and modify up to 20 different les in 5 different directories, not to mention running make and the debugger. In adddition, a single line of output from make can easily be 5000 or 10000 characters long, many times the screen capacity of a conventional terminal.
59

5 February 2005 02:09

60

All of these facts speak in favour of a windowing system such as X11, preferably with a highresolution monitor. You can keep your editor (or editors, if they dont easily handle multiple les) open all the time, and run the compiler and debugger in other windows. If multiple directories are involved, its easier to maintain multiple xterms, one per directory, than to continually change directories. A correctly set up xterm will allow you to scroll back as far as you want I nd that 250 lines is adequate.

Keep a log file

Sooner or later, youre going to run into a bug that you cant x immediately: you will have to experiment a bit before you can x the problem. Like nding your way through a labyrinth, the rst time through you will probably not take the most direct route, and its nice to be able to nd your way back again. In the original labyrinth, Theseus used a ball of string to nd his way both in and out. The log le, a text le describing what youve done, is the computer equivalent of the ball of string, so you should remember to roll it up again. If youre running an editor like emacs, which can handle multiple les at a time, you can keep the log in the editor buffer and remove the notes again when you back out the changes. In addition to helping you nd your way out of the labyrinth, the log will also be of use later when you come to install an updated version of the software. To be of use like this, it helps to keep additional information. For example, here are some extracts from a log le for the gcc:
Platform: Revision: Date ported: Ported by: Compiler used: Library: SCO UNIX System V.3.2.2.0 2.6.0 25 August 1994 Greg Lehey, LEMIS rcc, gcc-2.6.0 SCO

0. configure i386-unknown-sco --prefix=/opt. It sets local_prefix to /usr/local anyway, and wont listen to --local_prefix. For some reason, config decides that it should be cross-compiling. 1. function.c fails to compile with the message function.c: 59: no space. Compile this function with ISC gcc-2.5.8. 2. libgcc.a was not built because config decided to cross-compile. Re-run config with configure i386-*-sco --prefix=/opt, and do an explicit make libgcc.a. 3. crtbegin.o and crtend.o were not built. Fix configure: --- configure Tue Jul 12 01:25:53 1994 +++ configure Sat Aug 27 13:09:27 1994 @@ -742,6 +742,7 @@ else tm_file=i386/sco.h tmake_file=i386/t-sco + extra_parts="crtbegin.o crtend.o"

5 February 2005 02:09

Chapter 5: Building the package

fi truncate_target=yes ;;

61

Keeping notes about problems you have with older versions helps a lot: this example represents the results of a considerable time spent debugging the make procedure. If you didnt have the log, youd risk tripping over this problem every time.

Save make output

Typically, to build a package, after you have congured it, you simply type
$ make

Then the reworks start. You can sit and watch, but it gets rather boring to watch a package compile for hours on end, so you usually leave it alone once you have a reasonable expectation that it will not die as soon as you turn your back. The problem is, of course, that you may come back and nd a lot of gobbldegook on the screen, such as:
make[5]: execve: ../../config/makedepend/makedepend: No such file or directory make[5]: *** [depend] Error 127 make[5]: Leaving directory /cdcopy/SOURCE/X11/X11R6/xc/programs/xsetroot depending in programs/xstdcmap... make[5]: Entering directory /cdcopy/SOURCE/X11/X11R6/xc/programs/xstdcmap checking ../../config/makedepend/makedepend over in ../../config/makedepend first... make[6]: Entering directory /cdcopy/SOURCE/X11/X11R6/xc/config/makedepend gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -fwritable-strings -O \ -I../../config/imake -I../.. OSDefines -DSYSV -DSYSV386 -c include.c gcc: OSDefines: No such file or directory In file included from include.c:30: def.h:133: conflicting types for getline /opt/include/stdio.h:505: previous declaration of getline Broken pipe

This is from a real life attempt to compile X11R6, normally a fairly docile port. The target makedepend failed to compile, but why? The reason has long since scrolled off the screen.* You can have your cake and eat it too if you use tee to save your output:
$ make 2>&1 | tee -a Make.log

This performs the following actions: It copies error output (le descriptor 2) to standard output (le descriptor 1) with the expression 2>&1. It pipes the combined standard output to the program tee, which echos it to its standard output and also copies it to the le Make.log.

* Well, there is a clue, but its very difcult to see unless you have been hacking X11 congurations longer than is good for your health. OSDefines is a symbol used in X11 conguration. It should have been replaced by a series of compiler ags used to dene the operating system to the package. In this case, the X11 conguration was messed up, and nothing dened OSDefines, so it found its way to the surface.

5 February 2005 02:09

62

In this case, I specied the -a option, which tells tee to append to any existing Make.log. If I dont supply this ag, it will erase any previous contents. Depending on what youre doing, you may or may not want to use this ag.

If youre not sure what your make is going to do, and especially if the Makele is complicated, consider using the -n option. This option tells make to perform a dry run: it prints out the commands that it would execute, but doesnt actually execute them. These comparatively simple conventions can save a lot of pain. I use a primitive script called Make which contains just the single line:
make 2>&1 $* | tee -a Make.log

Its a good idea to always use the same name for the log les so that you can nd them easily.

Standard targets
Building packages consists of more than just compiling and linking, and by convention many Makeles contain a number of targets with specic meanings. In the following sections well look at some of the most common ones.

make depend
make depend creates a list of dependencies for your source tree, and usually appends it to the Makele. Usually it will perform this task with makedepend, but sometimes you will see a depend target that uses gcc with the -M ag or cpp. depend should be the rst target to run, since it inuences which other commands need to be executed. Unfortunately, most Makeles dont have a depend target. Its not difcult to write one, and it pays off in the reduction of strange, unaccountable bugs after a rebuild of the package. Heres a starting point:
depend: makedepend *.[ch]

This will work most of the time, but to do it correctly you need to analyze the structure of the package: it might contain les from other languages, or some les might be created by shell scripts or special conguration programs. Hopefully, if the package is this complicated, it will also have a depend target. Even if you have a depend target, it does not always work as well as you would hope. If you make some really far-reaching changes, and things dont work the way you expect, its worth starting from scratch with a make clean to be sure that the make still works.

make all
make all is the normal way to perform the build. Frequently, it is the default target (the rst target in the Makele), and you just need to enter make. This target typically rebuilds the package but does not install it.

5 February 2005 02:09

Chapter 5: Building the package

63

make install
make install installs the compiled package into the local system environment. The usage varies considerably; well look at this target in more detail in Chapter 9, Installation, page 126.

make clean
make clean normally removes everything that make all has madethe objects, executables and possibly auxiliary les. You use it after deciding to change a compiler, for example, or to save space after you have nished an installation. Be careful with make clean: there is no complete agreement about exactly what it removes, and frequently you will nd that it doesnt remove everything it should, or it is too eager and removes lots of things it shouldnt. make clean should remove everything that make all can make again the intermediate and installable les, but not the conguration information that you may have taken days to get right.

make stamp-halfway
Occasionally you see a target like make stamp-halfway. The commands perform a lot of other things, and at the end just create an empty le called stamp-halfway. This is a short cut to save lots of complicated dependency checking: the presence of this le is intended to indicate that the rst half of the build is complete, and that a restart of make can proceed directly to the second half. Good examples of this technique can be found in the Makele for the GNU C compiler, and in the X11 source tree, which uses the name DONE for the stamp le.

Problems running make

Ideally, running make should be simple:
$ make all lots of good messages from make

Things dont always go this smoothly. You may encounter a number of problems: You may not be able to nd a Makele, or the targets dont work the way you expect. make may not be able to make any sense of the Makele. The Makele may refer to non-existent les or directories. make seems to run, but it doesnt rebuild things it should, or it rebuilds things it shouldnt. You cant nd anything thats wrong, but make still produces obscure error messages.

In the following sections well look at each of these problems. Heres an overview of the

5 February 2005 02:09

64

types of error message well consider:

Table 51: Problems running make

Problem Argument list too long "$! nulled, predecessor circle" "Circular dependency dropped" "Commands commence before rst target" Comments in command lists "Graph cycles through target Incorrect continuation lines Incorrect dependencies make forgets the current directory "Missing separator - stop" Missing targets No dependency on Makele No Makele Nonsensical targets Problems with make clean Problems with subordinate makes Prompts in Makeles Subordinate makes Syntax errors from the shell Trailing blanks in variables Unable to stop make Wrong avour of make Wrong Makele

page 74 71 71 70 69 71 73 68 70 70 66 68 64 71 72 68 74 72 71 69 71 66 66

Missing Makefile or targets

Sometimes make wont even let you in the doorit prints a message like:
$ make all Dont know how to make all. Stop.

The rst thing to check here is whether there is a Makele. If you dont nd Makele or makele, check for one under a different name. If this is the case, the author should have documented where the Makele comes fromcheck the README les and other documentation that came with the package. You may nd that the package uses separate Makeles for different architectures. For example, Makele may be correct only if you are compiling in a BSD environment. If you want to compile for a System V machine, you may need to specify a different Makele:

5 February 2005 02:09

Chapter 5: Building the package

$ make -f Makefile.sysv

65

This is a pain because its so easy to make a mistake. In extreme cases the compiler will successfully create objects, but they will fail to link. Other possibilities include: The Makele is created by the conguration process, and you havent congured yet. This would be the case if you nd an Imakele (from which you create a Makele with xmkmfsee Chapter 4, Package conguration, page 57), or Makele.in (GNU conguresee page 55). The directory you are looking at doesnt need a Makele. The Makele in the parent directory, also part of the source tree, could contain rules like:
foo/foo: foo/*.c ${CC} foo/*.c -o foo/foo

In other words, the executable is made automatically when you execute make foo/foo in the parent directory. As a rule, you start building in the root directory of a package, and perform explicit builds in subdirectories only if something is obviously wrong. The author of the package doesnt believe in Makeles, and has provided a shell script instead. You often see this with programs that originated on platforms that dont have a make program. There is really nothing to build the package: the author is used to doing the compilation manually. In this case, your best bet is to write a Makele from scratch. The skeleton in Example 5-1 will get you a surprisingly long way. The empty targets are to remind you what you need to ll in:

Example 51:
SRCS = OBJS = ${SRCS:.c=.o} CC=gcc CFLAGS=-g -O3 LDFLAGS=-g BINDIR=/opt/bin LIBDIR=/opt/lib MANDIR=/opt/man MAN1DIR=man1 INFODIR=/opt/info PROGRAM= name of all: list of C source files corresponding object files file name of compiler flags for compiler flags for linker

finished program

$(PROGRAM) ${CC} ${LDFLAGS} -o ${PROGRAM} ${OBJS}

man: doc: install: all

5 February 2005 02:09

66 Example 51: (continued)

depend: makedepend ${SRCS} clean: rm -f \#* * core $(PROGRAM) *.o

Missing targets
Another obvious reason for the error message might be that the target all doesnt exist: some Makeles have a different target name for each kind of system to which the Makele has been adapted. The README le should tell you if this is the case. One of the more unusual examples is gnuplot. You need to enter
$ make All $ make x11 TARGET=Install

The better ones at least warn yousee Chapter 4, Package conguration, page 53, for an example. I personally dont like these solutions: its so much easier to add the following line at the top of the Makele:
BUILD-TARGET = build-bsd

The rst target would then be:

all: ${BUILD-TARGET}

If you then want to build the package for another architecture, you need only change the single line dening BUILD-TARGET.

make doesnt understand the Makefile

Sometimes make produces messages that make no sense at all: the compiler tries to compile the same le multiple times, each time giving it a different object name, or it claims not to be able to nd les that exist. One possible explanation is that various avours of make have somewhat different understandings of default rules. In particular, as we will see in Chapter 19, Make, there are a number of incompatibilities between BSD make and GNU make. Alternatively, make may not even be tryining to interpret the Makele. Somebody could have hidden a le called makele in the source tree. Most people today use the name Makele for makes description le, probably because its easier to see in an ls listing, but make always looks for a le called makele (with lower case m) rst. If you are using GNU make, it rst looks for a le called GNUmakele before checking for makele and Makele.

5 February 2005 02:09

Chapter 5: Building the package

67

make refers to non-existent files

Building a package refers to a large number of les, and one of the most frequent sources of confusion is a le that cant be found. There are various avours of this, and occasionally the opposite happens, and you have trouble with a le that make nds, but you cant nd. To analyse this kind of problem, its helpful to know how make is referring to a le. Here are some possibilities: make may be looking for a dependent le, but it cant nd it, and it cant nd a rule to build it. In this case you get a message like:
$ make make: *** No rule to make target config.h. Stop.

make may not be able to locate a program specied in a command. You get a message like:
$ make foo.o /bin/cc -c foo.o -o foo.c make: execve: /bin/cc: No such file or directory make: *** [foo.o] Error 127

The compilers and other programs started by make also access les specied in the source. If they dont nd them, youll see a message like
$ make foo.o gcc -c foo.c -o foo.o foo.c:1: bar.h: No such file or directory make: *** [foo.o] Error 1

No matter where the le is missing, the most frequent reasons why it is not found are: The package has been congured incorrectly. This is particularly likely if you nd that the package is missing a le like cong.h. The search paths are incorrect. This could be because you congured incorrectly, but it also could be that the conguration programs dont understand your environment. For example, its quite common to nd Makeles with contents like:
AR AS CC LD = = = = /bin/ar /bin/as /bin/cc /bin/cc

Some older versions of make need this, since they dont look at the PATH environment variable. Most modern versions of make do look at PATH, so the easiest way to x such a Makele is to remove the directory component of the denitions.

5 February 2005 02:09

68

Problems with subordinate makes

Occasionally while building, the compiler complains about a le that doesnt seem to be there. This can be because the make is running in a subdirectory: large projects are frequently split up into multiple subdirectories, and all the top level Makele does is to run a number of subordinate makes. If it is friendly, it also echos some indication of where it is at the moment, and if it dies you can nd the le. Newer versions of GNU make print messages on entering and leaving a directory, for example:
make[1]: Entering directory /cdcopy/SOURCE/Core/glibc-1.08.8/assert make[1]: Nothing to be done for subdir_lib. make[1]: Leaving directory /cdcopy/SOURCE/Core/glibc-1.08.8/assert

If neither of these methods work, you have the option of searching for the le:
$ find . -name foo.c -print

or modifying the Makele to tell you whats going on.

make doesnt rebuild correctly

One of the most insidious problems rebuilding programs occurs when make doesnt rebuild programs correctly: theres no easy way to know that a module has been omitted, and the results can be far-reaching and time-consuming. Lets look at some possible causes of this kind of problem.

Incorrect dependencies
One weakness of make is that you have to tell it the interdependencies between the source les. Unfortunately, the dependency specications are very frequently incorrect. Even if they were correct in the source tree as delivered, changing conguration ags frequently causes other header les to be included, and as a result the dependencies change. Make it a matter of course to run a make depend after reconguring, if this target is suppliedsee page 62 for details on how to make one.

No dependency on Makefile
What happens if you change the Makele? If you decide to change a rule, for example, this could require recompilation of a program. To put it in make terms: all generated les depend on the Makele. The Makele itself is not typically included in the dependency list. It really should be, but that would mean rebuilding everything every time you change the Makele, and in most cases its not needed. On the other hand, if you do change your Makele in the course of a port, its a good idea to save your les, do a make clean and start all over again. If everything is OK, it will build correctly without intervention.

5 February 2005 02:09

Chapter 5: Building the package

69

Other errors from make

The categories we have seen above account for a large proportion of the error messages you will see from make, but there are many others as well. In this section, well look at other frequent problems.

Trailing blanks in variables

You dene a make variable with the syntax:
NAME = Definition # optional comment

The exact Denition starts at the rst non-space character after the = and continues to the end of the line or the start of the comment, if there is one. You can occasionally run into problems with things like:
MAKE = /opt/bin/make # in case something else is in the path

When starting subsidiary makes, make uses the value of the variable MAKE as the name of the program to start. In this case it is /opt/bin/make it has trailing blanks, and the exec call fails. If youre lucky, you get:
$ make make: dont know how to make make . stop.

This message does give you a clue: there shouldnt be any white space between the name of the target and the following period. On the other hand, GNU make is friendly and tidies up trailing blanks, so it says:
$ make /opt/bin/make subdir note the space before the target name "subdir" make: execve: /opt/bin/make: No such file or directory make: *** [suball] Error 127

The only clue you have here is the length of the space on the rst line. Its relatively easy to avoid this sort of problem: avoid comments at the end of denition lines.

Comments in command lists

Some versions of make, notably XENIX, cant handle rules of the form
doc.dvi: doc.tex tex doc.tex # do it again to get the references right tex doc.tex # same thing again

The rst comment causes make to think that the rule is completed, and it stops. When you x this problem by removing the comment, you run into a second one: it doesnt understand the second comment either. This time it produces an error message. Again, you need to remove the comment.

5 February 2005 02:09

70

make forgets the current directory

Occasionally, it looks as if make has forgotten what you tell it. Consider the following rule:
docs: cd doc ${ROFF} ${RFLAGS} doc.ms > doc.ps

When you run it, you get:

$ make docs cd doc groff -ms doc.ms >doc.ps gtroff: fatal error: cant open doc.ms: No such file or directory make: *** [docs] Error 1

So you look for doc.ms in doc, and its there. Whats going on? Each command is run by a new shell. The rst one executes the cd doc and then exits. The second one tries to execute the groff command. Since the cd command doesnt affect the parent environment, it has no further effect, and youre still in the original directory. To do this correctly, you need to write the rule as:
docs: cd doc; \ ${ROFF} ${RFLAGS} doc.ms > doc.ps

This causes make to consider both lines as a single line, which is then passed to a single shell. The semicolon after the cd is necessary, since the shell sees the command as a single line.

Missing separator - stop

This strange message is usually made more complicated because it refers to a line that looks perfectly normal. In all probability it is trying to tell you that you have put leading spaces instead of a tab on a command line. BSD make expects tabs, too, but it recovers from the problem, and the message it prints if they are missing is much more intelligible:
"Makefile", line 21: warning: Shell command needs a leading tab

Commands commence before first target

This message, from System V make, is trying to tell you that you have used a tab character instead of spaces at the beginning of the denition of a variable. GNU make does not have a problem with thisit doesnt even mention the fact so you might see this in a Makele written for GNU make when you try to run it with System V make. BSD make cannot handle tabs at the beginning of denitions either, and produces the message:
"Makefile", line 3: Unassociated shell command "CC=gcc" Fatal errors encountered -- cannot continue

5 February 2005 02:09

Chapter 5: Building the package

71

Syntax errors from the shell

Many Makeles contain relatively complicated shell script fragments. As we have seen, these are constrained to be on one line, and most shells have rather strange relationship between new line characters and semicolons. Heres a typical example:
if test -d $(texpooldir); then exit 0; else mkdir -p $(texpooldir); fi

This example is all on one line, but you can break it anywhere if you end each partial line with a backslash (\). The important thing here is the placement of the semicolons: a rule of thumb is to put a semicolon where you would otherwise put a newline, but not after then or else. For more details, check your shell documentation.

Circular dependency dropped

This message comes from GNU make. In System V make, it is even more obscure:
$! nulled, predecessor circle

BSD make isnt much more help:

Graph cycles through docs

In each case, the message is trying to tell you that your dependencies are looping. This particular example was caused by the dependencies:
docs: man-pages

man-pages: docs

In order to resolve the dependency docs, make rst needs to resolve man-pages. But in order to resolve man-pages, it rst needs to resolve docsa real Catch 22 situation. Real-life loops are, of course, usually more complex.

Nonsensical targets
Sometimes the rst target in the Makele does nothing useful: you need to explicitly enter make all in order to make the package. There is no good reason for this, and every reason to x itsend the mods back to the original author if possible (and be polite).

Unable to stop make

Some Makeles start a number of second and third level Makeles with the -k option, which tells make to continue if the subsidiary Makele dies. This is quite convenient if you want to leave it running overnight and collect all the information about numerous failures the next morning. It also makes it almost impossible to stop the make if you want to: hitting the QUIT key (CTRL-C or DEL on most systems) kills the currently running make, but the top-level make just starts the next subsidiary make. The only thing to do here is to identify the top-level make and stop it rst, not an easy thing to do if you have only a single screen.

5 February 2005 02:09

72

Problems with make clean

make clean is supposed to put you back to square one with a build. It should remove all the les you created since you rst typed make. Frequently, it doesnt achieve this result very accurately: It goes back further than that, and removes les that the Makele doesnt know how to make.* Other Makeles remove conguration information when you do a make clean. This isnt quite as catastrophic, but you still will not appreciate it if this happens to you after you have spent 20 minutes answering conguration questions and xing incorrect assumptions on the part of the conguration script. Either way: before running a make clean for the rst time, make sure that you have a backup. make clean can also start off by doing just the opposite: in early versions of the GNU C library, for example, it rst compiled some things in order to determine what to clean up. This may work most of the time, but is still a Bad Idea: make clean is frequently used to clean up after some catastrophic mess, or when restarting the port on a different platform, and it should not be able to rely on being able to compile anything. Yet another problem with make clean is that some Makeles have varying degrees of cleanliness, from clean via realclean all the way to squeakyclean. There may be a need for this, but its confusing for casual users.

Subordinate makes
Some subordinate makes use a different target name for the subsidiary makes: you might write make all, but make might start the subsidiary makes with make subdirs. Although this cannot always be avoided, it makes it difcult to debug the Makele. When modifying Makeles, you may frequently come across a situation where you need to modify the behaviour of only one subsidiary make. For example, in many versions of System V, the man pages need to be formatted before installation. Its easy to tell if this applies to your system: if you install BSD-style unformatted man pages, the man program will just display a lot of hard-to-read nroff source. Frequently, xing the Makele is more work than you expect. A typical Makele may contain a target install that looks like:
install: for dir in ${SUBDIRS}; do \ echo making $@ in $$dir; \ cd $$dir; ${MAKE} ${MDEFINES} $@; \ cd ..; \ done

make $@ expands to make install. One of these subdirectories is the subdirectory doc,
* If this does happen to you, dont despair just yet. Check rst whether this is just simple-mindedness on the part of the Makelemaybe there is a relatively simple way to recreate the les. If not, and you forgot to make a backup of your source tree before you started, then you can despair.

5 February 2005 02:09

Chapter 5: Building the package

73

which contains the documentation and requires special treatment for the catman pages: they need to be formatted before installation, whereas the man pages are not formatted until the rst time they are referencedsee Chapter 7, Documentation, page 99 for further information. The simplest solution is a different target that singles out the doc and makes a different target, say install-catman. This is untidy and requires some modications to the variable SUBDIRS to exclude doc. A simpler way is to create a new target, install-catman, and modify all Makeles to recognize it:
install-catman install-manman: for dir in ${SUBDIRS}; do \ echo making $@ in $$dir; \ cd $$dir; ${MAKE} ${MDEFINES} $@; \ cd ..; \ done

In the Makeles in the subdirectories, you might then nd targets like

install-catman: ${MANPAGES} for i in $<; do ${NROFF} -man $$i > ${CATMAN}/$i; done install-manman: ${MANPAGES} for i in $<; do cp $$i > ${MANMAN}/$i; done

The rule in the top-level Makele is the same for both targets: you just need to know the name to invoke it with. In this example we have also renamed the original install target so that it doesnt get invoked accidentally. By removing the install target altogether, you need to make a conscious decision about what kind of man pages that your system wants. Were not done yet: we now have exactly the situation we were complaining about on page 66: it is still a nuisance to have to remember make install-catman or make install-manman. We can get round this problem, too, with
INSTALL_TYPE=install-catman install: ${INSTALL_TYPE}

After this, you can just enter make install, and the target install performs the type of installation specied in the variable INSTALL_TYPE. This variable needs to be modied from time to time, but it makes it easier to avoid mistakes while porting.

Incorrect continuation lines

Makeles frequently contain numerous continuation lines ending with \. This works only if it is the very last character on the line. A blank or a tab following the backslash is invisible to you, but it really confuses make. Alternatively, you might continue something you dont want to. Consider the following Makele fragment, taken from an early version of the Makele for this book:
PART1 = part1.ms config.ms imake.ms make.ms tools.ms compiler.ms obj.ms \ documentation.ms testing.ms install.ms epilogue.ms

5 February 2005 02:09

74

At some point I decided to change the sequence of chapters, and removed the le tools.ms. I was not completely sure I wanted to do this, so rather than just changing the Makele, I commented out the rst line and repeated it in the new form:
# PART1 = part1.ms config.ms imake.ms make.ms tools.ms compiler.ms obj.ms \ PART1 = part1.ms config.ms imake.ms make.ms compiler.ms obj.ms \ documentation.ms testing.ms install.ms epilogue.ms

This works just neat rst. In fact, it turns out that make treats all three lines as a comment, since the comment nished with a \ character. As a result, the variable PART1 remained undened. If you comment out a line that ends in \, you should also remove the \.

Prompts in Makefiles
If you do the Right Thing and copy your make output to a log le, you may nd that make just hangs. The following kind of Makele can cause this problem:
all: checkclean prog

checkclean: @echo -n "Make clean first? " @read reply; if [ "$$reply" = y ]; then make clean; fi

If you run make interactively, you will see:

$ make Make clean first?

If you copy the output to a le, of course, you dont see the prompt, and it looks as if make is hanging. This doesnt mean its a bad idea to save your make output: its generally a bad idea to put prompts into Makeles. There are some exceptions, of course. The Linux conguration program is a Makele, and to interactively congure the system you enter make cong.

Arg list too long

Sometimes make fails with this message, especially if you are running a System V system. Many versions of System V limit the argument list to 5120 byteswell look at this in more detail in Chapter 12, Kernel dependencies, page 169. Modern versions of System V allow you to rebuild the kernel with a larger parameter list: modify the tuneable parameter ARG_MAX to a value in the order of 20000. If you cant do this, there are a couple of workarounds: The total storage requirement is the sum of the length of the argument strings and the environment strings. Its very possible that you have environment variables that arent needed in this particular situation (in fact, if youre like me, you probably have environment variables that you will never need again). If you remove some of these from your shell startup le, you may get down below the limit. You might be able to simplify expressions. For example, if your Makele contains a line like

5 February 2005 02:09

Chapter 5: Building the package

clean: rm -rf *.o *.a *.depend * core ${INTERMEDIATES}

75

you can split it into

clean: rm -rf *.o rm -rf *.a *.depend * core ${INTERMEDIATES}

In most large trees, the *.o lenames constitute the majority of the arguments, so you dont need more than two lines. Even after the previous example, you might nd that the length of the *.o parameters is too long. In this case, you could try naming the objects explicitly:
clean: rm rm rm rm -rf -rf -rf -rf [a-f]*.o [g-p]*.o [r-z]*.o *.a *.depend * core ${INTERMEDIATES}

Alternatively, you could specify the names explicitly in the Makele:

OBJ1S = absalom.o arthur.o ... fernand.o OBJ2S = gerard.o guillaume.o ... pierre.o OBJ3S = rene.o roland.o ... zygyszmund.o OBJS = ${OBJ1S} ${OBJ2S} ${OBJ3S} clean: rm -rf ${OBJ1S} rm -rf ${OBJ2S} rm -rf ${OBJ3S}

Yet another method involves the use of the xargs program. This has the advantage of not breaking after new les have been added to the lists:
clean: find . -name "*.o" -print | xargs rm -f

This chops up the parameter list into chunks that wont overow the system limits.

Creating executable files

The xargs method is not much help if you want to build an executable le. If the command that fails looks like
${PROG}: ${CC} ${ALLOBJS} -o ${PROG}

there are some other possibilities. You might be able to shorten the pathnames. If you are building in a directory /next-release/SOURCE/sysv/SCO/gcc-2.6.0, and every le name in ALLOBJS is absolute, its much easier to exceed the limit than if the directory name was, say, /S. You could use a symbolic link to solve this problem, but most systems that dont support ARG_MAX also dont have symbolic links.*

5 February 2005 02:09

76

If this doesnt work, you could place the les in a library, possibly using xargs:
${PROG}: rm libkludge.a echo ${ALLOBJS} | xargs ar cruv libkludge.a ${CC} libkludge.a -o ${PROG}

This looks strange, since theres no object le, but it works: by the time it nds the name libkludge.a, the linker has already loaded the object le crt0.o (see Chapter 21, Object les and friends, page 368), and is looking for a symbol main. It doesnt care whether it nds it in an object le or a library le.

Modifying Makefiles
Frequently enough, you nd that the Makele is inadequate. Targets are missing, or some error occurs that is almost untraceable: you need to x the Makele. Before you do this, you should check whether you are changing the correct Makele. Some packages build a new Makele every time you run make. In particular, you frequently see Makeles that start with text like
# Makefile generated by imake - do not edit!

You can follow this advice or not: it depends on you and what you are doing: If you are just trying to gure out what the Makele is trying (and presumably failing) to do, its nice to know that you can subsequently delete your modied Makele and have it automatically remade. Once you have found out why the Makele is doing what it is, you need to x the source of the Makele. This is not usually too difcult: the input les to the Makele generation phase typically dont look too different from the nished Makele. For example, Makele.in in the GNU packages is a skeleton that is processed by m4, and except for the m4 parameters Makele.in looks very similar to the nished Makele. Finding the way back to the Imakele from the Makele requires a little more understanding of the imake process, but with a little practice its not that difcult.

* If you are on a network with other machines with more modern le systems, you could work around this problem by placing the les on the other system and accessing them via NFS.

5 February 2005 02:09

6
Running the compiler
In the previous chapter, we looked at building from the viewpoint of make. The other central program in the build process is the compiler, which in UNIX is almost always a C compiler. Like make, the compiler can discover a surprising number of problems in what ostensibly debugged source code. In this chapter, well look at these problems and how to solve them. In well look at how the compiler works and how the various avours of C differ. Although we restrict our attention to the C compiler, much of what we discuss relates to other compilers as well, particularly of course to C++. This chapter expects a certain understanding of the C language, of course, but dont be put of if youre still a beginner: this is more about living with C than writing it. Information from the compiler can come in a number of forms: The compiler may issue warnings, which are informational messages intended to draw attention to possible program errors. Their reliability and their value varies signicantly: some are a sure-re indication that something is wrong, while others should be taken with a pinch of salt. The compiler may issue error messages, indicating its conviction that it cannot produce a valid output module. This also usually means that the compiler will not create any output les, though you cant always rely on this. The compiler may fail completely, either because of an internal bug or because it realizes that it no longer understands the input sufciently to continue.

Compiler warnings
Its easy to make mistakes when writing programs, but it used to be even easier: nowadays, even the worst compilers attempt to catch dubious constructs and warn you about them. In this section, well look at what they can and cant do. Before compilers worried about coding quality, the program lint performed this task. lint is still around, but hardly anybody uses it any more, since it doesnt always match the compiler being used. This is a pity, because lint can catch a number of dubious situations that evade most compilers.

77

5 February 2005 02:09

78

Modern compilers can recognize two kinds of potential problems: Problems related to dubious program text, like
if (a = 1) return;

The rst line of this example is almost superuous: if I allocate the value 1 to a, I dont need an if to tell me what the result will be. This is probably a typo, and the text should have been
if (a == 1) return;

Problems related to program ow. These are detected by the ow analysis pass of the optimizer. For example:
int a; b = a;

The second line uses the value of a before it has been assigned a value. The optimizer notices this omission and may print a warning. In the following sections, well examine typical warning messages, how they are detected and how reliable they are. Ill base the sections on the warning messages from the GNU C compiler, since the it has a particularly large choice of warning messages, and since it is also widely used. Other compilers will warn about the same kind of problems, but the messages may be different. Table 6-1 gives an overview of the warnings well see.
Table 61: Overview of warning messages

5 February 2005 02:09

Chapter 6: Running the compiler Table 61: Overview of warning messages (continued)

79

Kind of warning Changing non-volatile automatic variables Character subscripts to arrays Dequalifying types Functions with embedded extern denitions Implicit conversions between enums Implicit return type Incomplete switch statements Inconsistent function returns Increasing alignment requirements Invalid keyword sequences in declarations Long indices for switch Missing parentheses Nested comments Signed comparisons of unsigned values Trigraphs Uninitialized variables

page 82 80 81 84 82 79 82 79 81 83 82 83 83 80 83 80

Implicit return type

K&R C allowed programs like
main () { printf ("Hello, World!\n"); }

ANSI C has two problems with this program: The function name main does not specify a return type. It defaults to int. Since main is implicitly an int function, it should return a value. This one does not.

Both of these situations can be caught by specifying the -Wreturn-type option to gcc. This causes the following messages:
$ gcc -c hello.c -Wreturn-type hello.c:2: warning: return-type defaults to int hello.c: In function main: hello.c:4: warning: control reaches end of non-void function

Inconsistent function returns

The following function does not always return a dened value:

5 February 2005 02:09

80
foo (int x) { if (x > 3) return x - 1; }

If x is greater than 3, this function returns x - 1. Otherwise it returns with some uninitialized value, since there is no explicit return statement for this case. This problem is particularly insidious, since the return value will be the same for every invocation on a particular architecture (possibly the value of x), but this is a by-product of the way the compiler works, and may be completely different if you compile it with a different compiler or on some other architecture.

Uninitialized variables
Consider the following code:
void foo (int x) { int a; if (x > 5) a = x - 3; bar (a); ... etc

Depending on the value of x, a may or may not be initialized when you call bar. If you select the -Wuninitialized compiler option, it warns you when this situation occurs. Some compilers, including current versions of gcc place some limitations on this test.

Signed comparisons of unsigned values

Occasionally you see code of the form
int foo (unsigned x) { if (x >= 0) ... etc

Since x is unsigned, its value is always >= 0, so the if is superuous. This kind of problem is surprisingly common: system header les may differ in opinion as to whether a value is signed or unsigned. The option -W causes the compiler to issue warnings for this and a whole lot of other situations.

Character subscripts to arrays

Frequently, the subscript to an array is a character. Consider the following code:
char iso_translate [256] = /* translate table for ISO 8859-1 to LaserJet */ { codes for the first 160 characters

5 February 2005 02:09

Chapter 6: Running the compiler

0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, ... etc }; #define xlate(x) iso_translate [x]; char *s; for (*s = buf; *s; s++) *s = xlate (*s); /* pointer in buf */

81

The intention of xlate is to translate text to a form used by older model HP LaserJet printers. This code works only if the char *s is unsigned. By default, the C char type is a signed value, and so the characters 0x80 to 0xff represent a negative array offset, and the program attempts (maybe successfully) to access a byte outside the table iso_translate. gcc warns about this if you set the option -Wchar-subscripts.

Dequalifying types
The following code fragment can cause problems:
char *profane; void foo (const char *holy) { profane = holy;

The assignment of holy to profane loses the qualier const, and the compiler complains about the fact. On the other hand, this is valid:
profane = (char *) holy;

This doesnt make it a better idea: holy is supposed to be unchangeable, and here you are removing this qualier. If you specify the -Wcast-qual option to gcc, it complains if you use a cast to remove a type qualier such as const.

Increasing alignment requirements

Many processors require that specic data types be aligned on specic boundaries, and the results can be spectacular if they are notsee Chapter 11, Hardware dependencies, page 158, for more details. We can easily outsmart the C compiler with code like:
void foo (char *x) { int *ip = (int *) x;

In this case, there is a good chance that the int * pointer ip requires a specic alignment and is not allowed to point at any address in memory the way the char pointer x is allowed to do. If you specify the -Wcast-align option to gcc, it warns you of such assignments.

5 February 2005 02:09

82

Implicit conversions between enums

One of the advantages of enums is that they make type checking easierwell look at that in more detail in Chapter 20, Compilers, page 339. If you specify the -Wenum-clash option to gcc, and youre compiling C++, it warns about sloppy use of enums.

Incomplete switch statements

A frequent cause of error in a switch statement is that the index variable (the variable that decides which case is chosen) may assume a value for which no case has been specied. If the index variable is an int of some kind, there is not much you can do except include a default clause. If the index variable is an enum, the compiler can check that case clauses exist for all the possible values of the variable, and warns if they do not. It also warns if case clauses exist for values that are not dened for the type of the index variable. Specify the -Wswitch option for these warnings.

long indices for switch

In some dialects of pre-ANSI C, you could write things like
foo (x) long x; { switch (x) { ... etc

This is no longer allowed in ANSI C: indices for switch must evaluate to an int, even if int and long have the same length. gcc issues a warning about long indices in switch unless you specify the -traditional option.

Changing non-volatile automatic variables

Under certain circumstances, a signal handler might modify a local automatic variable if the function has called setjmpsee Chapter 13, Signals, page 200 for more details. gcc options this situation as a warning if you specify the -W option. This is a complicated problem: It can occur only during an optimizing compilation, since the keyword volatile has meaning only in these circumstances. In addition, the situation is recognized only by the optimizer. The optimizer cannot recognize when a longjmp could be performed. This depends on semantics outside the scope of the optimizer. As a result, it could issue this warning when there is, in fact, no danger.

5 February 2005 02:09

Chapter 6: Running the compiler

83

Invalid keyword sequences in declarations

Currently, it is permissible to write declarations like
int static bad_usage;

Here the storage class specier static comes after the type specier int. The ANSI Standard still permits this, but declares the usage to be obsolescent. gcc issues a warning when it encounters this and the option -W has been set.

Trigraphs
Trigraphs (see Chapter 20, Compilers, page 342) are no error, at least according to the ANSI Standard. The Free Software Foundation makes no bones about their opinion of them, and so gcc supplies the option -Wtrigraphs, which prints a warning if any trigraphs occur in the source code. Since this works only if the option -trigraphs is used to enable them, it is not clear that this is of any real use.

Nested comments
Occasionally you see code like
void foo (int x) { int y; y = bar (); if (y == 4) ... etc

/* state information /* initialize y */

The code looks reasonable, and it is syntactically correct C, but in fact the comment after the declaration of y is not terminated, so it includes the whole of the next line, which is almost certainly not the intention. gcc recognizes this if it nds the sequence /* in a comment, and warns of this situation if you specify the -Wcomment option.

Missing parentheses
What value does the following code return?
int a = 11 << 4 & 7 << 2 > 4;

The result is 0, but the real question is: in what order does the compiler evaluate the expression? You can nd the real answer on page 53 of K&R, but you dont want to do that all the time. We can re-write the code as
int a = (11 << 4) & ((7 << 2) > 4);

This makes it a lot clearer what is intended. gcc warns about what it considers to be missing parentheses if you select the -Wparentheses option. By its nature, this option is subjective, and you may nd that it complains about things that look ne to you.

5 February 2005 02:09

84

Functions with embedded extern definitions

K&R C allowed you to write things like
int datafile; foo (x) { extern open (); datafile = open ("foo", 0777); }

The extern declaration was then valid until the end of the source le. In ANSI C, the scope of open would be the scope of foo: outside of foo, it would no longer be known. gcc issues a warning about extern statements inside a function denition unless you supply the -traditional option. If you are using -traditional and want these messages, you can supply the -Wnested-externs option as well.

Compiler errors
Of course, apart from warnings, you frequently see error messages from the compilerthey are the most common reason for a build to fail. In this section, well look at some of the more common ones.

Undefined symbols
This is one of the most frequent compiler error messages you see during porting. At rst sight, it seems strange that the compiler should nd undened symbols in a program that has already been installed on another platform: if there are such primitive errors in it, how could it have worked? In almost every case, you will nd one of the following problems: The denition you need may have been #ifdefed out. For example, in a manually congured package, if you forget to specify a processor architecture, the package may try to compile with no processor denitions, which is sure to give rise to this kind of problem. The symbol may have been dened in a header le on the system where it was developed. This header le is different on your system, and the symbol you need is never dened. You may be looking at the wrong header les. Some versions of gcc install xed copies of the system header les in their own private directory. For example, under BSD/386 version 1.1, gcc version 2.6.3 creates a version of unistd.h and hides it in a private directory. This le omits a number of denitions supplied in the BSDI version of unistd.h. You can conrm which header les have been included by running gcc with the -H option. In addition, on page 86 we look at a way to check exactly what the preprocessor did.

The second problem is surprisingly common, even on supposedly identical systems. For

5 February 2005 02:09

Chapter 6: Running the compiler

85

example, in most versions of UNIX System V.4.2, the system header le link.h denes information and structures used by debuggers. In UnixWare 1.0, it denes information used by some Novell-specic communications protocols. If you try to compile gdb under UnixWare 1.0, you will have problems as a result: the system simply does not contain the denitions you need. Something similar happens on newer System V systems with POSIX.1 compatibility. A program that seems formally correct may fail to compile with an undened symbol O_NDELAY. O_NDELAY is a ag to open, which species that the call to open should not wait for completion of the request. This can be very useful, for example, when the open is on a serial line and will not complete until an incoming call occurs. The ag is supported by almost all modern UNIX ports, but it is not dened in POSIX.1. The result is that the denition is carefully removed if you compile dening -D_POSIX_SOURCE. You might think that this isnt a problem, and that you can replace O_NDELAY with the POSIX.1 ag O_NONBLOCK. Unfortunately, the semantics of O_NONBLOCK vary from those of O_NDELAY: if no data is available, O_NONBLOCK returns -1, and O_NDELAY returns 0. You can make the change, of course, but this requires more modications to the program, and you have a strraighforward alternative: #undef _POSIX_SOURCE. If you do this, you may nd that suddenly other macros are undened, for example O_NOCTTY. System V.4 only denes this variable if _POSIX_SOURCE is set. Theres no simple solution to this problem. It is caused by messy programming style: the programmer has mixed symbols dened only by POSIX.1 with those that are not dened in POSIX.1. The program may run on your current system, but may stop doing so at the next release.

Conflicts between preprocessor and compiler variables

Occasionally youll see things that seem to make absolutely no sense at all. For example, porting gcc, I once ran into this problem:
gcc -c -DIN_GCC -g -O3 -I. -I. -I./config \ -DGCC_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/include\" \ -DGPLUSPLUS_INCLUDE_DIR=\"/opt/lib/g++-include\" \ -DCROSS_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/sys-include\" \ -DTOOL_INCLUDE_DIR=\"/opt/i386--sysv/include\" \ -DLOCAL_INCLUDE_DIR=\"/usr/local/include\" \ -DSTD_PROTO_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0\" \ ./protoize.c ./protoize.c:156: macro puts used without args

Looking at this part of protoize.c, I found lots of external denitions:

extern extern extern extern extern extern extern int int int int int int int fflush (); atoi (); puts (); fputs (); fputc (); link (); unlink ();

5 February 2005 02:09

86

Line 156 is, not surprisingly, the denition of puts. But this is a denition, not a call, and certainly not a macro. And why didnt it complain about all the other denitions? There were many more than shown here. In cases like this, its good to understand the way the compiler works well look at this in more detail in Chapter 20, Compilers, on page 348. At the moment, we just need to recall that programs are compiled in two stages: rst, the preprocessor expands all preprocessor denitions and macros, and then the compiler itself compiles the resultant output, which can look quite different. If you encounter this kind of problem, theres a good chance that the compiler is not seeing what you expect it to see. You can frequently solve this kind of riddle by examining the view of the source that the compiler sees, the output of the preprocessor. In this section, well look at the technique I used to solve this particular problem. All compilers will allow you to run the preprocessor separately from the compiler, usually by specifying the -E option see your compiler documentation for more details. In this case, I was running the compiler in an xterm*, so I was able to cut and paste the complete 8-line compiler invocation as a command to the shell, and all I needed to type was the text in bold face:
$ gcc -c -DIN_GCC -g -O3 -I. -I. -I./config \ -DGCC_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/include\" \ -DGPLUSPLUS_INCLUDE_DIR=\"/opt/lib/g++-include\" \ -DCROSS_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/sys-include\" \ -DTOOL_INCLUDE_DIR=\"/opt/i386--sysv/include\" \ -DLOCAL_INCLUDE_DIR=\"/usr/local/include\" \ -DSTD_PROTO_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0\" \ ./protoize.c -E -o junk.c $

If you dont have xterm, you can do the same sort of thing by editing the make log (see Chapter 5, Building the package, page 60), which will contain the invocation as well. junk.c starts with:
# 1 "./config.h" 1

# 1 "./config/i386/xm-i386.h" 1 40 empty lines # 1 "./tm.h" 1 19 empty lines # 1 "./config/i386/gas.h" 1 22 empty lines

This le seems to consist mainly of empty lines, and the lines that arent empty dont seem to be C! In fact, the # lines are C (see the line directive in Chapter 20, Compilers, page 344), except that in this case the keyword line has been omitted. The empty lines are where comments and preprocessor directives used to be. The error message referred to line 156 of protoize.c, so I searched for lines with protoize.c on them. I found a number of them:
* xterm is a terminal emulator program that runs under X11. If you dont use X11, you shouldfor example, it makes this particular technique much easier.

5 February 2005 02:09

Chapter 6: Running the compiler

$ grep protoize.c junk.c # 1 "./protoize.c" # 39 "./protoize.c" 2 # 59 "./protoize.c" 2 # 62 "./protoize.c" 2 # 63 "./protoize.c" 2 ... etc # 78 "./protoize.c" 2 # 222 "./protoize.c"

87

Clearly, the text was between lines 78 and 222. I positioned on the line after the marker for line 78 and moved down (156 - 78) or 78 lines. There I found:
extern int fflush (); extern int atoi (); extern int ((fputs(( ), stdout) || (( stdout )->__bufp < ( stdout )->__put_limit ? (int) (unsigned char) (*( stdout )->__bufp++ = (unsigned char) ( 0 )) :__flshfp (( stdout ), (unsigned char) ( 0 ))) == (-1) ) ? (-1) : 0) ; extern int fputs (); extern int fputc (); extern int link (); extern int unlink ();

Well, at any rate this made it clear why the compiler was complaining. But where did this junk come from? It can be difcult to gure this out. With gcc you can use the -dD option to keep the preprocessor denitionsunfortunately, the compiler still removes the other preprocessor directives. I used -dD as well, and found in junk.c:
# 491 "/opt/include/stdio.h" 2 25 lines missing extern int fputs (__const char *__s, FILE *__stream) ; /* Write a string, followed by a newline, to stdout. */ extern int puts (__const char *__s) ; #define puts(s) ((fputs((s), stdout) || __putc(0, stdout) == EOF) ? EOF : 0)

This looks strange: rst it declares puts as an external function, then it denes it as a macro. Looking at the original source of stdio.h, I found:
/* Write a string, followed by a newline, to stdout. */ extern int puts __P ((__const char *__s)); #ifdef __OPTIMIZE__ #define puts(s) ((fputs((s), stdout) || __putc(0, stdout) == EOF) ? EOF : 0) #endif /* Optimizing. */

No, this doesnt make sense its a real live bug in the header le. At the very least, the declaration of puts () should have been in an #else clause. But thats not the real problem: it doesnt worry the preprocessor, and the compiler doesnt see it. The real problem is that protoize.c is trying to do the work of the header les and dene puts again. There are many programs that try to out-guess header les: this kind of denition breaks them all. There are at least two ways to x this problem, both of them simple. The real question is, what is the Right Thing? System or library header les should be allowed to dene macros

5 February 2005 02:09

88

instead of functions if they want, and an application program has no business trying to do the work of the header les, so it would make sense to x protoize.c by removing all these external denitions: apart from this problem, theyre also incompatible with ANSI C, since they dont describe the parameters. In fact, I chose to remove the denition from the header le, since that way I only had to do the work once, and in any case, its not clear that the denition really would run any faster. Preprocessor output usually looks even more illegible than this, particularly if lots of clever nested #defines have been performed. In addition, youll frequently see references to nonexistant line numbers. Here are a couple of ways to make it more legible: Use an editor to put comments around all the #line directives in the preprocessor output, and then recompile. This will make it easier to nd the line in the preprocessor output to which the compiler or debugger is referring; then you can use the comments to follow it back to the original source. Run the preprocessor output through a program like indent, which improves legibility considerably. This is especially useful if you nd yourself in the unenviable position of having to modify the generated sources. indent is not guaranteed to maintain the same number of lines, so after indenting you should recompile.

Other preprocessors
There are many other cases in which the source le you use is not the source le that the compiler gets. For example, yacc and bison take a grammar le and make a (more or less illegible) .c le out of it; other examples are database preprocessors like Informix ESQL, which takes C source with embedded SQL statements and converts it into a form that the C compiler can compile. The preprocessors output is intended to be read by a compiler, not by humans. All of these preprocessors use lines beginning with # to insert information about the original line numbers and source les into their output. Not all of them do it correctly: if the preprocessor inserts extra lines into the source, they can become ambiguous, and you can run into problems when using symbolic debuggers, where you normally specify code locations by line number.

Syntax errors
Syntax errors in previously functional programs usually have the same causes as undened symbols, but they show their faces in a different way. A favourite one results from omitting /usr/include/sys/types.h. For example, consider bar.c:
#include <stdio.h> #ifdef USG #include <sys/types.h> #endif ushort num; int main (int argc, char *argv []) {

5 February 2005 02:09

Chapter 6: Running the compiler

num = atoi (argv [1]); printf ("First argument: %d\n", num); }

89

If you compile this under BSD/OS, you get:

$ gcc -o bar bar.c bar.c:6: parse error before num bar.c:6: warning: data definition has no type or storage class

Theres an error because ushort hasnt been dened. The compiler expected a type specier, so it reported a syntax error, not an undened symbol. To x it, you need to dene the type specied see Appendix A, Comparative reference to UNIX data types for a list of the more common type speciers.

Virtual memory exhausted

You occasionally see this message, particularly when youre using gcc, which has a particular hunger for memory. This may be due to unrealistically low virtual memory limits for your system by default, some systems limit total virtual memory per process to 6 MB, but gcc frequently requires 16 or 20 MB of memory space, and on occasion it can use up to 32 MB for a single compilation. If your system has less than this available, increase the limit accordingly. Dont forget to ensure that you have enough swap space! Modern systems can require over 100 MB of swap space. Sometimes this doesnt help. gcc seems to have particular difculties with large data denitions; bit map denitions in X11 programs are the sort of things that cause problems. xphoon, which displays a picture of the current phase of the moon on the root window, is a good example of a gcc-breaker.

Compiler limits exceeded

Some compilers have difculties with complicated expressions. This can cause cc1, the compiler itself, to fail with messages like expression too complicated or out of tree space. Fixing such problems can be tricky. Straightforward code shouldnt give the compiler indigestion, but some nested #denes can cause remarkable increases in the complexity of expressions: in some cases, a single line can expand to over 16K of text. One way to get around the problem is to preprocess the code and then break the preprocessed code into simpler expressions. The indent program is invaluable here: preprocessor output is not intended to be human-readable, and most of the time it isnt.

Running compiler passes individually

Typical compilers run four distinct passes to compile and link a programsee Chapter 20, Compilers, page 348, for more details. Sometimes running the passes separately can be useful for debugging a compilation:

5 February 2005 02:09

90

If you nd yourself with header les that confuse your preprocessor, you can run a different preprocessor, collect the output and feed it to your compiler. Since the output of the preprocessor is not machine-dependent, you could even do this on a different machine with different architecture, as long as you ensure that you use the correct system header les. By convention, the preprocessor output for foo.c would be called foo.isee Chapter 20, Compilers, page 348 for a list of intermediate le sufxes though it usually does no harm if you call it foo.c and pass it through the preprocessor again, since there should no longer be anything for the second preprocessor to do. If you want to report a compiler bug, its frequently a good idea to supply the preprocessor output: the bug might be dependent on some header le conict that doesnt exist on the system where the compiler development takes place. If you suspect the compiler of generating incorrect code, you can stop compilation after the compiler pass and collect the generated assembler output.

Incorrect code from compiler

Compilers sometimes generate incorrect code. Incorrect code is frequently difcult to debug because the source code looks (and might be) perfect. For example, a compiler might generate an instruction with an incorrect operand address, or it might assign two variables to a single location. About the only thing you can do here is to analyze the assembler output. One kind of compiler bug is immediately apparent: if the code is so bad that the assembler cant assemble it, you get messages from the assembler. Unfortunately, the message doesnt usually tell you that it comes from the assembler, but the line numbers change between the compiler and the assembler. If the line number seems completely improbable, either because it is larger than the number of lines in your source le, or because it seems to have nothing to do with the context of that line, there is a chance that the assembler produced the message. There are various ways to conrm which pass of the compiler produced the message. If youre using gcc, the simplest one is to use the -v option for the compiler, which announces each pass of compilation as it starts, together with the version numbers and parameters passed to the pass. This makes it relatively easy to gure out which pass is printing the error messages. Otherwise you can run the passes individually see Chapter 20, Compilers, page 348 for more details.

5 February 2005 02:09

7
Documentation
Ask any real guru a question, so the saying goes, and he will reply with a cryptic RTFM.* Cynics claim this is even the answer to the question Where can I nd the manual? All too often, programmers consider documentation a necessary (or even unnecessary) evil, and if it gets done at all, its usually the last thing that gets done. This is particularly evident when you look at the quality of documentation supplied with some free software packages (though many free packages, such as most of those from the Free Software Foundation, are very well documented). The quality and kind of the documentation in source packages varies wildly. In Chapter 2, Unpacking the goodies, page 25, we looked at the documentation that should be automatically supplied with the package to describe what it is and how to install it. In this chapter, well look at documentation that is intended for use after you have installed the package. The documentation you get with a package is usually in one of the following formats: man pages, the traditional on-line documentation for UNIX, which are formatted with nroff. info les, used with the GNU projects info on-line documentation reader. Unformatted roff, T X, or texinfo hardcopy documentation. E Preformatted documentation in PostScript or .dvi format, or occasionally in other formats such as HP LaserJet.

We know where we want to get tothe formatted documentationbut we dont always know where to start, so its easier to look at documentation in reverse order: rst, well look at the end result, then at the formatters, and nally at the input les.

Preformatted documentation
Occasionally you get documentation that has been formatted so that you can print it on just about any printer, but this doesnt happen very much: in order to achieve this, the text must be free of any frills and formatted so that any typewriter can print it. Nearly any printer
* Read The Manualthe F is usually silent. 91

5 February 2005 02:09

92

nowadays is capable of better results, so preformatted les are usually supplied in a format that can print high quality printout on a laser printer. The following three are about the only ones you will come across: PostScript is a specialized programming language for printers, and the printed data are in fact embedded in the program. This makes it an extremely exible format. .dvi is the format that is output by T X. In order to print it, you need a TEX driver. E Unlike PostScript and .dvi, the Hewlett-Packard LaserJet format is not portable: you need a LaserJet-compatible printer to print it. The LaserJet format is obsolescent: even many LaserJet printers made today also support PostScript, and there are programmatic ways to print PostScript on other laser printers, so there is little motivation for using the much more restrictive LaserJet format.

PostScript
PostScript is the current format of choice. Because it is a programming language, it is much more exible than conventional data formats. For example, it is easily scalable. You can take a le intended for a phototypesetter with a resolution of 2540 bpi and print it on a laser printer, and it will come out correctly.* In addition, better quality printers perform the formatting themselves, resulting in a considerable load reduction for the computer. A large number of printers and all modern phototypesetters can process PostScript directly. If your printer doesnt handle PostScript, you can use programs like ghostscript, which interpret PostScript programs and output in a multitude of other formats, including LaserJet, so even if you have a LaserJet, it can be a better idea to use PostScript format. ghostscript is distributed by the Free Software Foundation see Appendix E, Where to get sources. ghostscript can also display PostScript les on X displays. Most PostScript les are encoded in plain ASCII without any control characters except newline (though that doesnt make them easy to read). Even when you include special characters in your text, they appear in the PostScript document as plain ASCII sequences. Its usually pretty easy to recognize PostScript, even without the le program. Heres the start of a draft version of this chapter:
%!PS-Adobe-3.0 %%Creator: groff version 1.09 %%CreationDate: Thu Aug 18 17:34:24 1994 %%DocumentNeededResources: font Times-Bold

The data itself is embedded in parentheses between the commands. Looking at a draft of this text, we see things like
(It)79.8 273.6 Q 2.613(su)-.55 G .113 (sually pretty easy to recognize a PostScript program, e)-2.613 F -.15 (ve)-.25 G 2.614(nw).15 G .114(ithout the)-2.614 F F2(\214le)2.614 E F1 (program--here)79.8 285.6 Q 2.5(st)-.55 G(he start of a draft v)-2.5 E

* You may have to wait a while before a few megabytes of font information are transferred and processed, but eventually you get your document.

5 February 2005 02:09

Chapter 7: Documentation

93

Problems with PostScript

PostScript doesnt pose too many problems, but occasionally you might see one of these: Missing fonts PostScript documents include information about the fonts they require. Many fonts are built in to printers and PostScript display software, but if the fonts are not present, the system chooses a default value which may have little in common with the font which the document requested. The default font is typically Courier, which is fixed-width, and the results look terrible. If this happens, you can nd the list of required fonts with the following:
$ grep %%.* font mumble.ps %%DocumentNeededResources: font Garamond-BookItalic %%+ font Times-Roman %%+ font Garamond-Light %%+ font Garamond-LightItalic %%+ font Courier %%+ font Garamond-Book %%+ font Courier-Bold %%IncludeResource: font Garamond-BookItalic %%IncludeResource: font Times-Roman %%IncludeResource: font Garamond-Light %%IncludeResource: font Garamond-LightItalic %%IncludeResource: font Courier %%IncludeResource: font Garamond-Book %%IncludeResource: font Courier-Bold (%%DocumentNeededResources: font Times-Bold)131.711 327.378 S F1 1.281

This extracts the font requests from the PostScript le: in this case, the document requires Times Roman, Courier and Garamond fonts. Just about every printer and software package supplies Times Roman and Courier, but Garamond (the font in which this book is written) is less common. In addition, most fonts are copyrighted, so you probably wont be able to nd them on the net. If you have a document like this in PostScript format, your choices are: Reformat it with a different font if you have the source. Get the Garamond fonts.

Edit the le and change the name of the font to a font with similar metrics (in other words, with similar size characters). The results wont be as good, but if the font you nd is similar enough, they might be acceptable. For example, you might change the text Garamond to Times Roman. Wrong font type Most PostScript fonts are in plain ASCII. You may also come across Type 2 PostScript and display PostScript, both of which include binary data. Many printers cant understand the binary format, and they may react to it in an unfriendly way. For example, my National KX-P 4455 printer just hangs if I copy display PostScript to it. See the section format conversion below for ways to solve this dilemma.

5 February 2005 02:09

94

.dvi format
One of the goals of T X was to be able to create output for just about any printer. As we will E see, old versions of troff, the main competitor, were able to produce output only for a very limited number of phototypesetters. Even if you have one of them in your ofce, its unlikely that you will want to use it for printing out a draft of a 30-page paper. The T X solution, which was later adopted by troff in ditroff (device independent troff), was to E output the formatted data in a device-independent format, .dvi, and leave it to another program, a so-called driver, to format the les in a format appropriate to the output device. Unlike PostScript, .dvi contains large numbers of control characters and characters with the sign bit set, and is not even remotely legible. Most versions of le know about .dvi format.

Format conversion
Not so long ago your choice of documentation software determined your output format. For example, if you used T X, you would get .dvi output, and you would need a T X driver to print E E it. Nowadays, its becoming easier to handle le formats. GNU troff will output in .dvi format if you wish, and programs are available to convert from .dvi to PostScript and back again. Heres a list of conversions you might like to perform see Appendix E, Where to get sources for how to get software to perform them. A number of programs convert from .dvi to PostScriptfor example, dvips. Theres no good reason to want to convert from PostScript to .dvi, so there are no programs available. .dvi is not much use in itselfit needs to be tranformed to a nal printer form, and if you have PostScript output, you can do that directly with ghostscript (see below) without going via .dvi. To display .dvi les on an X display, use SeeTeX. To convert from .dvi to a printer output format, use one of the dvi2xxx programs. To convert from PostScript to a printer format, use ghostscript. To display PostScript on an X display, you can also use ghostscript, but ghostview gives you a better interface. To convert PostScript with binary data into ASCII, use t1ascii.

roff and friends

The original UNIX formatting program was called roff (for run-off). It is now completely obsolete, but it has a number of descendents: nroff is a comparatively simple formatter designed to produce output for plain ASCII displays and printers. troff is a more sophisticated formatter designed to produce output for phototypesetters. Many versions create output only for the obsolete APS-5 phototypesetter, and you need

5 February 2005 02:09

Chapter 7: Documentation

95

postprocessing software to convert this output to something that modern typesetters or laser printers understand. Fortunately, versions of troff that produce PostScript output are now available. ditroff (device independent troff) is a newer version of troff that produces output in a device-independent intermediate form that can then be converted into the nal form by a conversion program. This moves the problem of correct output format from troff to the conversion program. Despite the terminology, this device-independent format is not the same as .dvi format. groff is the GNU project troff and nroff replacement. In troff mode it can produce output in PostScript and .dvi format.

All versions of roff share the same source le syntax, though nroff is more restricted in its functionality than troff. If you have a usable version of troff, you can use it to produce properly formatted hardcopy versions of the man pages, for example. This is also what xman (the X11 manual browser) does.

formatting with nroff or troff

troff input bears a certain resemblance to the traces left behind when a y falls into an inkwell and then walks across a desk. The rst time you run troff against a le intended for troff, the results may be less than heartening. For example, consider the following passage from the documentation of the Revision Control System RCS. When correctly formatted, the output is:
Besides the operations ci and co, RCS provides the following commands: ident rcs rcsclean rcsdiff rcsfreeze rcsmerge rlog extract identication markers change RCS le attributes remove unchanged working les (optional) compare revisions record a conguration (optional) merge revisions read log messages and other information in RCS les

A synopsis of these commands appears in the Appendix. 2.1 Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker $Id$ into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form $Id: lename revisionnumber date time author state locker $

To format it, you can try

$ troff rcs.ms >rcs.ps

This assumes the use of groff or another avour of troff that creates PostScript output (thus the

5 February 2005 02:09

96

name rcs.ps for the output le). If you do this, you get an output that looks like:
Besides the operations ci and co, RCS provides the following commands: tab(%); li l. ident%extract identication markers rcs%change RCS le attributes rcsclean%remove unchanged working les (optional) rcsdiff%compare revisions rcsfreeze%record a conguration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS les A synopsis of these commands appears in the Appendix. Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker Id into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form Id: lename revisionnumber date time author state locker

Most of the text seems to be there, but it hasnt been formatted at all (well, it has been right justied). What happened? Almost every troff or roff input document uses some set of macros. You can dene your own macros in the source, of course, but over time a number of standard macro packages have evolved. They are stored in a directory called tmac. In the days of no confusion, this was /usr/lib/tmac, but nowadays it might equally well be /usr/share/tmac (for systems close to the System V.4 ABIsee Chapter 4, Package conguration, page 48, for more details) or /usr/local/groff/tmac for GNU roff. The name is known to troff either by environment variables or by instinct (the path name is compiled into the program). troff loads specic macros if you specify the name of the le as an argument to the -m ag. For example, to specify the man page macros /usr/lib/tmac/an, you would supply troff with the parameter -man. man makes more sense than an, so these macros are called the man macros. The names of other macro packages also usually grow an m at the beginning. Some systems change the base name of the macros from, say, /usr/lib/tmac/an to /usr/lib/tmac/tmac.an. Most versions of troff supply the following macro packages: The man (tmac/an) and mandoc (tmac/andoc) packages are used to format man pages. The mdoc (tmac/doc) package is used to format hardcopy documents, including some man pages. The mm (tmac/m) macros, the so-called memorandum macros, are described in the documentation as macros to format letters, reports, memoranda, papers, manuals and books. It doesnt describe what you shouldnt use them for. The ms (tmac/s) macros were the original macros supplied with the Seventh Edition. They are now claimed to be obsolescent, but you will see them again and again. This book was formatted with a modied version of the ms macros. The me (tmac/e) macros are another, more recent set of macros which originated in Berkeley.

There is no sure-re way to tell which macros a le needs. Here are a couple of possibilities: The le name sufx might give a hint. For example, our le is called rcs.ms, so there is a very good chance that it wants to be formatted with -ms.

5 February 2005 02:09

Chapter 7: Documentation

97

The program grog, which is part of groff, examines the source and guesses the kind of macro set. It is frequently wrong. The only other way is trial and error. There arent that many different macro sets, so this might be a good solution.

In this case, our le name suggests that it should be formatted with the ms macros. Lets try that:
$ troff rcs.ms >rcs.ps

Now we get:
Besides the operations ci and co, RCS provides the following commands: tab(%); li l. ident%extract identication markers rcs%change RCS le attributes rcsclean%remove unchanged working les (optional) rcsdiff%compare revisions rcsfreeze%record a conguration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS les A synopsis of these commands appears in the Appendix. 2.1 Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker $Id$ into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form $Id: lename revisionnumber date time author state locker $

Well, it doesnt look quite as bad, but its still not where we want to be. What happened to that list of program names? troff does not do all the work by itself. The tabular layout of the program names in this example is done by the preprocessor tbl, which handles tables. Before we let troff at the document, we need to pass it through tbl, which replaces the code
.TS tab(%); li l. ident%extract identification markers rcs%change RCS file attributes rcsclean%remove unchanged working files (optional) rcsdiff%compare revisions rcsfreeze%record a configuration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS files .TE

with a couple of hundred lines of complicated and illegible troff instructions to build the table. To get the desired results, we need to enter:
$ tbl rcs.ms | troff -ms >rcs.ps

nroff, troff and groff use a number of preprocessors to perform special functions. They are:

5 February 2005 02:09

98

soelim replaces .so statements (which correspond to C #include statements) with the contents of the le to which the line refers. The roff programs do this too, of course, but the other preprocessors dont, so if the contents of one of the les is of interest to another preprocessor, you need to run soelim rst. refer processes references. pic draws simple pictures. tbl formats data in tabular form. eqn formats equations.

Unless you know that the document youre formatting doesnt use any of these preprocessors, or formatting takes a very long time, its easier to use them all. There are two possible ways to do this: You can pipe from one processor to the next. This is the standard way:
$ soelim rcs.ms | refer | pic | tbl | eqn | troff -ms The soelim preprocessor reads in the document, and replaces any .so commands by the contents of the le to which they refer. It then passes the output to refer, which processes any textual references and passes it to pic, which processes any pictures it may nd, and passes the result to tbl. tbl processes any tables and passes its result to eqn, which processes any equations before passing the result to troff.

Some versions of troff invoke the preprocessors themselves if passed appropriate ags. For example, with groff:
Table 71: Starting preprocessors from groff Flag -e -t -p -s -R

Processor eqn tbl pic soelim refer

Starting the preprocessors from troff not only has the advantage of involving less typingit also ensures that the preprocessors are started in the correct sequence. Problems can arise if you run eqn before tbl, for example, when there are equations within tables. See Typesetting tables with tbl by Henry McGilton and Mary McNabb for further details.

Other roff-related programs

As you can see, the troff system uses a large number of programs. Once they were relatively small, and this was the UNIX way. Now they are large, but there are still a lot of them. Apart from the programs we have already seen, you could encounter the GNU variants, which can

5 February 2005 02:09

Chapter 7: Documentation

99

optionally be installed with a name beginning in gfor example, GNU eqn may be installed as geqn if the system already has a different version of eqn. indxbib and lookbib (sometimes called lkbib) process bibliographic references, and are available in the groff package if you dont have them. groff also includes a number of other programs, such as grops, and grotty, which you dont normally need to invoke directly.

Man pages
Almost from the beginning, UNIX had an on-line manual, traditionally called man pages. You can peruse man pages with the man program, or you can print them out as hardcopy documentation. Traditionally, man pages are cryptic and formalized: they were