There are not many operating systems that anyone has ever described as 鈥榝un鈥�. Indeed, the
friction and labor of development under most other environments has been aptly compared to
kicking a dead whale down the beach. The kindest adjectives one normally hears are on the
order of 鈥渢olerable鈥� or 鈥渘ot too painful鈥�. In the Unix world, by contrast, the OS is normally
seen not as an adversary to be clubbed into doing one's bidding by main effort but rather as
an actual positive help.

This has real economic significance. The fun factor started a virtuous circle early in Unix's
history. People liked Unix, so they built more programs for it that made it nicer to use. Today
people build entire, production-quality open-source Unix systems as a hobby. To understand
how remarkable this is, ask yourself when you last heard of anybody cloning OS/360 or
VAX VMS or Microsoft Windows for fun.

The 鈥渇un鈥� factor is not trivial from a design point of view, either. The kind of people who
become programmers and developers have 鈥渇un鈥� when the effort they have to put out to do a
task challenges them, but is just within their capabilities. 鈥淔un鈥� is therefore a sign of peak
efficiency. Painful development environments waste labor and creativity; they extract huge
hidden costs in time, money, and opportunity.

If Unix were a failure in every other way, the Unix engineering culture would be worth
understanding for the ways it keeps the fun in development 鈥� because that fun is a sign that
it makes developers efficient, effective, and productive.

The lessons of Unix can be applied elsewhere

Unix programmers have accumulated decades of experience while pioneering operating-
system features we now take for granted. Even non-Unix programmers can benefit from
studying that Unix experience. Because Unix makes it relatively easy to apply good design
principles and development methods, it is an excellent place to learn them.

Other operating systems generally make good practice rather harder, but even so some of the
Unix culture's lessons can transfer. Much Unix code (including all its filters, its major
scripting languages, and many of its code generators) will port directly to any operating
system supporting ANSI C (for the excellent reason that C itself was a Unix invention and
the ANSI C library embodies a substantial chunk of Unix's services!).

Prev Up Next

What Unix gets wrong

Home Basics of the Unix philosophy

Basics of the Unix philosophy

Prev Chapter 1. Philosophy

Next



Basics of the Unix philosophy

The 鈥楿nix philosophy鈥� originated with Ken Thompson's early meditations on how to design
a small but capable operating system with a clean service interface. It grew as the Unix
culture learned things about how to get maximum leverage out of Thompson's design. It
absorbed lessons from many sources along the way.

The Unix philosophy is not a formal design method. It wasn't handed down from the high
fastnesses of theoretical computer science as a way to produce theoretically perfect software.
Nor is it that perennial executive's mirage, some way to magically extract innovative but
reliable software on too short a deadline from unmotivated, badly managed and underpaid
programmers.

The Unix philosophy (like successful folk traditions in other engineering disciplines) is
bottom-up, not top-down. It is pragmatic and grounded in experience. It is not to be found in
official methods and standards, but rather in the implicit half-reflexive knowledge, the
expertise that the Unix culture transmits. It encourages a sense of proportion and skepticism
鈥� and shows both by having a sense of (often subversive) humor.

Doug McIlroy, the inventor of pipes and one of the founders of the Unix tradition, famously
summarized it this way (quoted in A Quarter Century of Unix [Salus]):

This is the Unix philosophy. Write programs that do one thing and do it well.
Write programs to work together. Write programs to handle text streams,
because that is a universal interface.

McIlroy later expanded on this [BSTJ]:

(i) Make each program do one thing well. To do a new job, build afresh rather
than complicate old programs by adding new features.

(ii) Expect the output of every program to become the input to another, as yet
unknown, program. Don't clutter output with extraneous information. Avoid
stringently columnar or binary input formats. Don't insist on interactive input.

(iii) Design and build software, even operating systems, to be tried early,
ideally within weeks. Don't hesitate to throw away the clumsy parts and
rebuild them.

(iv) Use tools in preference to unskilled help to lighten a programming task,
even if you have to detour to build the tools and expect to throw some of them
out after you've finished using them.

Rob Pike, one of the great early masters of C, offers a slightly different angle in Notes on C
Programming [Pike]:

Rule 1. You can't tell where a program is going to spend its time. Bottlenecks
occur in surprising places, so don't try to second guess and put in a speed hack
until you've proven that's where the bottleneck is.

Rule 2. Measure. Don't tune for speed until you've measured, and even then
don't unless one part of the code overwhelms the rest.

Rule 3. Fancy algorithms are slow when n is small, and n is usually small.
Fancy algorithms have big constants. Until you know that n is frequently going
to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)

Rule 4. Fancy algorithms are buggier than simple ones, and they're much
harder to implement. Use simple algorithms as well as simple data structures.

Rule 5. Data dominates. If you've chosen the right data structures and
organized things well, the algorithms will almost always be self-evident. Data
structures, not algorithms, are central to programming. [3]

Rule 6. There is no Rule 6.

Ken Thompson, the man who designed and implemented the first Unix, reinforced Pike's rule
4 with a gnomic maxim worthy of a Zen patriarch:

When in doubt, use brute force.

More of the Unix philosophy was implied not by what these elders said but by what they did

and the example Unix itself set. Looking at the whole, we can abstract the following ideas:

1.
Rule of Modularity: Write simple parts connected by clean interfaces.
2.
Rule of Composition: Design programs to be connected to other programs.
3.
Rule of Clarity: Clarity is better than cleverness.
4.
Rule of Simplicity: Design for simplicity; add complexity only where you must.
5.
Rule of Transparency: Design for visibility to make inspection and debugging easier.
6.
Rule of Robustness: Robustness is the child of transparency and simplicity.
7.
Rule of Least Surprise: In interface design, always do the least surprising thing.
8.
Rule of Repair: When you must fail, fail noisily and as soon as possible.
9.
Rule of Economy: Programmer time is expensive; conserve it in preference to
machine time.
10.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you
can.
11.
Rule of Representation: Use smart data so program logic can be stupid and robust.
12.
Rule of Separation: Separate policy from mechanism; separate interfaces from
engines.
13.
Rule of Optimization: Prototype before polishing. Get it working before you optimize
it.
14.
Rule of Diversity: Distrust all claims for 鈥渙ne true way鈥�.
15.
Rule of Extensibility: Design for the future, because it will be here sooner than you
think.

If you're new to Unix, these principles are worth some meditation. Software-engineering
texts recommend most of them; but most other operating systems lack the right tools and
traditions to turn them into practice, so most programmers can't apply them with any
consistency. They come to accept blunt tools, bad designs, overwork, and bloated code as
normal 鈥� and then wonder what Unix fans are so annoyed about.

Rule of Modularity: Write simple parts connected by clean
interfaces.

As Brian Kernignan once observed, 鈥淐ontrolling complexity is the essence of computer
programming.鈥� Debugging dominates development time, and getting a working system out
the door is usually less a result of brilliant design than it is of managing not to trip over your
own feet too many times.

Assemblers, compilers, flowcharting, procedural programming, structured programming,
鈥渁rtificial intelligence鈥�, fourth-generation languages, object orientation, and software-
development methodologies without number have been touted and sold as a cure for this
problem. All have failed, if only because they 鈥榮ucceeded鈥� by escalating the normal level of
program complexity to the point where (once again) human brains could barely cope. As
Fred Brooks famously observed [Brooks], there is no silver bullet.

The only way to write complex software that won't fall on its face is to hold its global
complexity down 鈥� to build it out of simple parts connected by well-defined interfaces, so
that most problems are local and you can have some hope of upgrading a part without
breaking the whole.

Rule of Composition: Design programs to be connected with
other programs.

It's hard to avoid programming overcomplicated monoliths if none of your programs can talk
to each other.

Unix tradition puts a lot of emphasis on writing programs that read and write simple, textual,
stream-oriented, device-independent formats. Under classic Unix, as many programs as
possible are written as simple filters, which take a simple text stream on input and process it
into another simple text stream on output.

Despite popular mythology, this is not because Unix programmers hate graphical user
interfaces. It's because if you don't write programs that accept and emit simple text streams,
it's much more difficult to hook them together.

Text streams are to Unix tools as messages are to objects in an object-oriented setting. The
simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate
forms of IPC, such as remote procedure calls, show a tendency to involve programs with
each others' internals too much.

GUIs can be a very good thing. Complex binary data formats are sometimes unavoidable by
any reasonable means. But before writing a GUI, it's wise to ask if the tricky interactive parts
of your program can be segregated into one piece and the workhorse algorithms into another,
with a simple command stream or application protocol connecting the two. Before devising a
tricky binary format to pass data around, it's worth experimenting to see if you can make a
simple textual format work and accept a little parsing overhead in return for being able to
hack the data stream with general-purpose tools.

When such a serialized, protocol-like interface is not natural, proper Unix design is to at least
organize as many of the application primitives as possible into a library with a well-defined
API. This opens up the possibility that the application can be called by linkage, or to glue
multiple interfaces on it for different tasks.

(We discuss these issues in detail in Chapter 6 (Multiprogramming).)

Rule of Clarity: Clarity is better than cleverness.

Because maintenance is so important and so expensive, write programs as if the most
important communication they do is not to the computer that executes them but to the human
beings who will read and maintain the source code in the future (including yourself).

In the Unix tradition, the implications of this advice go beyond just commenting your code.
Good Unix practice also embraces choosing your algorithms and implementations for future
maintainability. Buying a small increase in performance with a large increase in the
complexity and obscurity of your technique is a bad trade 鈥� not merely because complex
code is more likely to harbor bugs, but also because complex code will be harder to read for
future maintainers.

Code that is graceful and clear, on the other hand, is less likely to break 鈥� and more likely to

be instantly comprehended by the next person to have to change it. This is important,
especially when that next person might be yourself some years down the road.

Rule of Simplicity: Design for simplicity; add complexity only
where you must.

There are many pressures which tend to make programs more complicated (and therefore
more expensive and buggy). One is technical machismo. Programmers are bright people who
are (justly) proud of their ability to handle complexity and juggle abstractions. Often they
compete with their peers to see who can build the most intricate and beautiful complexities.
Just as often, their ability to design outstrips their ability to implement and debug, and the
result is expensive failure.

Even more often (at least in the commercial software world) excessive complexity comes
from project requirements that are based on the marketing fad of the month rather than the
reality of what customers want or software can actually deliver. Many a good design has
been smothered under marketing's pile of 鈥渃heck-list features鈥� 鈥� features which, often, no
customer will ever use. And a vicious circle operates; the competition thinks it has to
compete with chrome by adding more chrome. Pretty soon, massive bloat is the industry
standard and everyone is using huge, buggy programs not even their developers can love.

Either way, everybody loses in the end.

The only way to avoid these traps is to encourage a software culture that actively resists bloat
and complexity 鈥� an engineering tradition that puts a high value on simple solutions, looks
for ways to break program systems up into small cooperating pieces, and reflexively fights
attempts to gussy up programs with a lot of chrome (or, even worse, to design programs
around the chrome).

That would be a culture a lot like Unix's.

Rule of Transparency: Design for visibility to make
inspection and debugging easier.

Because debugging often occupies three-quarters or more of development time, work done
early to ease debugging can be a very good investment. A particularly effective way to
accomplish this is to design for transparency and discoverability.

A software system is transparent when you can look at it and immediately understand what
it is doing and how. It is discoverable when it has facilities for monitoring and display of
internal state so that your program not only functions well but can be seen to function well.

Designing for these qualities will have implications throughout a project. At minimum, it
implies that debugging options should not be minimal afterthoughts. Rather, they should be
designed in from the beginning 鈥� from the point of view that the program should be able to
both demonstrate its own correctness and communicate the original developer's mental
model of the problem it solves to future developers.

In order for a program to demonstrate its own correctness, it needs to be using input and
output formats sufficiently simple so that the proper relationship between valid input and
correct output is easy to check.

The objective of designing for transparency and discoverability should also encourage simple
interfaces that can easily be manipulated by other programs 鈥� in particular, test and
monitoring harnesses and debugging scripts.

Rule of Robustness: Robustness is the child of transparency
and simplicity.

Most software is buggy because most programs are too complicated for a human brain to
understand all at once. When you can't reason correctly about the guts of a program, you
can't be sure it's correct, and you can't fix it if it's broken.

It follows that the way to make programs that aren't buggy is to make their internals easy for
human beings to reason about. There are two main ways to do that: transparency and and
simplicity.

We observed above that software is transparent when you can look at it and immediately see
what is going on. It is simple when what is going on is uncomplicated enough for a human
brain to reason about all the potential cases without strain.

Modularity (simple parts, clean interfaces) is a way to organize programs to make them
simpler. There are other ways to fight for simplicity. Here's another one:

Rule of Least Surprise: In interface design, always do the

least surprising thing.

The easiest programs to use are those which demand the least new learning from the user 鈥�
or, to put it another way, the easiest programs to use are those that connect to the user's pre-
existing knowledge most effectively.

Therefore, avoid gratuitous novelty and excessive cleverness in interface design 鈥� if you're
writing a calculator program, 鈥�+鈥� should always mean addition! When designing an interface,
model it on the interfaces of functionally similar or analogous programs with which your
users are likely to be familiar.

Pay attention to tradition. The Unix world has rather well-developed conventions about
things like the format of configuration and run-control files, command-line switches, and the
like. These traditions exist for a good reason 鈥� to tame the learning curve. Learn and use
them.

(We'll cover many of these traditions in Chapters 5 (Textuality) and 10 (Configuration).)

Rule of Repair: Repair what you can 鈥� but when you must
fail, fail noisily and as soon as possible.

Software should be transparent in the way that it fails as well as in normal operation. It's best
when software can cope with unexpected conditions by adapting to them, but the worst kinds
of bugs are those in which the repair doesn't succeed and the problem quietly causes
corruption that doesn't show up until much later.

Therefore, write your software to cope with incorrect inputs and its own execution errors as
gracefully as possible 鈥� but when it cannot, make it fail in a way that makes diagnosis of the
problem as easy as possible.

Consider also Postel's Prescription[4]: 鈥淏e liberal in what you accept, and conservative in
what you send.鈥� Postel was speaking of network service programs, but the underlying idea is
more general. Well-designed programs cooperate with other programs by making as much
sense as they can from ill-formed inputs; they either fail noisily or pass strictly clean and
correct data to the next program in the chain.

Rule of Economy: Programmer time is expensive; conserve