NetBSD: Not Just for Toasters

gvim

http://www.phptr.com/articles/article.asp?p=655422&rl=1

NetBSD: Not Just for Toasters

Early this month (October 2006), the NetBSD project is due to release the next version of its OS. NetBSD is the oldest of the free BSD-derived operating systems still in active development, beginning life as a patch set for 386BSD. It is also the one with which I am least familiar, so I asked some of the developers to tell me what was so special about their system.

As with the rest of the BSD family, NetBSD is a complete operating system—kernel and userland—derived from the Berkeley Software Distribution version of UNIX. By the time it reached version 1.0, it already had support for six platforms, with more being added every release. The current total is 57, including 17 different CPU types.

Of Course, It Runs NetBSD
NetBSD has a well-deserved reputation for portability. Part of this reputation comes from the driver layer, which makes use of an abstraction layer known as the Modular Portability Layer (MPL). This layer enables a single driver to be easily used on all architectures by hiding details of exactly how the host talks to the hardware and dramatically reduces the amount of work needed to port it to a new architecture. It was a running joke for years that NetBSD was the OS to run on your toaster, and last year the team proved it by demonstrating a NetBSD-enabled toaster.

This portability was responsible for my first introduction to NetBSD. With version 7.0, RedHat dropped support for the SPARC32 architecture. While looking for an alternative to run on some old SparcStations that were being used as X terminals, I came across NetBSD. At the time, Linux’s handling of the SPARC32 MMU had some major issues (I don’t know if they have been fixed now), and NetBSD was noticeably faster. NetBSD has given a new lease on life to a lot of old hardware. Ports to systems such as old Apple 68K-based machines mean that it is often a very cheap way of getting a taste of UNIX.

Although the number of ports is impressive, the team is quick to point out that this does not necessarily correspond to portability. Geert Hendrickx explains:

You should understand that "portability" is not the same as "number of ports," though. Making an actual port still requires 1) manpower and 2) hardware, thus money, both of which NetBSD certainly has much less than (for example, Linux). And Linux ports are often quite "brute forced," involving a lot of code rewriting.

When NetBSD is ported to a new platform, this typically involves writing the code that sits at the bottom of the abstraction layer and writing drivers for any required hardware. In some cases, the drivers may already exist, but it is common for a new platform to have things such as keyboard controllers that are not present in existing systems, and so need adding. Hubert Feyer explained that the focus on portability is as important as ever:

With the ever-growing number of devices, not only in the traditional computer market but also more and more in the embedded area, portability is the key to NetBSD. Without proper infrastructure for writing portable code, like bus and DMA abstractions and a driver framework that sits on top of that, we could not deliver an operating system that runs on more than 50 hardware platforms and 14(?) CPU architectures—all from one source tree, including kernel, userland, and the X Window System.

This approach makes it a lot easier to maintain ports. When a new feature, such as the native threading support added in 2.0, is added higher up the abstraction stack, all the ports automatically gain it. This is in contrast with a system such as Linux, in which a number of ports are close to being forks and require a lot of pack-porting to bring up to feature-parity with the main branch. This portability, along with the permissive license, has attracted a number of hardware developers to NetBSD. It is found in a number of systems that the end user would not even think of as needing an operating system, such as Ricoh laser printers and photocopiers. When I asked why system developers would choose Linux over NetBSD, Hubert Feyer made this suggestion:

The primary reason why a lot of companies are choosing Linux over NetBSD, though, would be hype: Linux gets a lot more media attention; many people don’t even know that NetBSD exists. As a consequence, there is a lot more commercial software and support for Linux, and unfortunately this seems to form a closed loop. Fortunately, there are a number of companies that look beyond the latest hype and recognize the strengths of NetBSD: companies like Avocent, Sony, Brocade, Force 10, Wasabi Systems, NEC, Ricoh, and others.

It's All About the Applications
An operating system is nothing without applications to run on it. This is one area in which NetBSD excels. The packaging system, pkgsrc, has grown beyond its roots and is now available for several other platforms, including the other BSDs; Linux; and several flavors of commercial UNIX, such as Solaris and Mac OS X. DragonFly BSD has adopted it as its native packaging system, and Sun has contributed hardware to the project to encourage better Solaris support. Geert Hendrickx describes some of the attractions of pkgsrc:

pkgsrc provides a solid, consistent, portable framework for installing and managing an ever-growing number of third-party packages, both in source and in binary form. pkgsrc doesn’t let packages install files where they want it; they must all fit into a coherent (and configurable)directory structure. pkgsrc features package selection by license (users can specify which software licenses they accept and which not), automatic checking of packages against a list of known vulnerabilities, etc.

Originally, pkgsrc started as a fork of FreeBSD’s ports system. Since then, it has grown enormously. Like the rest of NetBSD, pkgsrc has a focus on portability, and most of the packages in it have been tested to compile on all supported architectures and operating systems. This, of course, comes with its own problems.

Many of the architectures supported by NetBSD are very slow. Although a 5MHz MicroVAX or a 25MHz 68000 might be fast enough to run some applications usefully, building 6,000 packages on one would take a very, very long time. Joerg Sonnenberger suggested a few approaches that are being used to address this problem. The first is emulation:

It’s certainly faster to build packages for Vax in an emulator running on a beefy amd64 box than to build them natively. Partial CPU emulation of the userland could work, too—similar to our Linux emulation, and I think some people are working on it, but I don’t know the status of it.

The partial CPU emulation referred to here was the subject of a paper published a couple of years ago. This approach is particularly interesting; building an emulator that can run configure scripts and GCC quickly is a very different problem from building a general purpose emulator. For one thing, system calls can be executed on the host machine in much the same way in which system call translation works: When the emulator encounters a system call, it performs any required byte-order manipulations and calls the real system call. This is quite easy to do on NetBSD because the system calls are all in the same place on all systems.

Some programs that are frequently invoked, such as the shell, can be implemented natively, which can speed things up a lot. Although this is a fascinating direction from a research perspective, there is a slightly simpler option, namely using distcc:

All configuration and preprocessing stages would run on the native host, but the actual cross-compiling can be done on faster machines on the network. pkgsrc supports using distcc as a compiler.

The distcc program is a wrapper around another compiler, which allows the distributing of a build job across a network of machines. This means that the heavy lifting work of compiling can be done on other machines; as long as the native machine is fast enough to run the configure script, most of the rest of the work can be delegated.

The ideal solution would be for all the packages to support cross-building as well as the base system does; you can build NetBSD almost anywhere, including on Windows, which makes it particularly nice for developing embedded systems that don’t have the necessary power to perform a native build. Unfortunately, a large number of open source programs use build environments designed without cross-building in mind, and it would require a huge amount of effort to move all of them to a new system.

The Virtual Machine
In recent years, Xen has been a focus point for NetBSD development. Benchmarks of the Xen version 2.0 port showed that it was a faster Domain 0 host than Linux, even when running Linux guests.

The new 3.1 release is the first to support Xen 3.0. Currently, this is only domU (guest OS) support. Users wanting to run Xen virtual machines on a NetBSD host need to either stick with Xen 2.0 or wait for NetBSD 4.0, which is currently in beta and scheduled for release later in the year. Support for Xen 3.0 dom0 is in the development branch and will appear in NetBSD 4.0 when this is branched, but it has not been back-ported to the 3.x branch.

One of the aims of Xen is for the hypervisor to do as little as possible, delegating as much as it can to the domain 0 guest, and each release moves more functionality from Xen to the dom0 OS. To get some idea of what was required to add Xen 3.0 support, I asked Manuel Bouyer, the NetBSD/xen port maintainer:

The main difference is that dom0 and domU talk to each other using the xenstore interface instead of control messages. Code for xenstore had to be added. Then, the virtual device front-end and back-ends had to be adapted to the xenstore interface. There were also minor differences in some hypercalls. For dom0, access to PCI devices is also a bit different; especially the APCI interrupt routing, which was done by the Xen kernel has now to be done by dom0.’

The Technology
NetBSD only comparatively recently added support for multithreading and it did so in a very ambitious way. Threading implementations can roughly be divided into three categories, depending on the number of kernel scheduler entities are mapped to threads. The simplest model is 1:N, which means that all threading is implemented in userspace using timer events. The model used by most is 1:1, meaning all thread scheduling is handled by the kernel.

The third approach is to adopt a M:N model, in which some threads are scheduled by the kernel, whereas others are handled in userspace. This is the most flexible; it means that fewer kernel resources are needed and allows faster, userspace-only, context switching to be used where appropriate. This is also the hardest to get right. Solaris used to use this model, but moved to 1:1 recently. FreeBSD 5.x also introduced an M:N model, but the team is now moving back to a 1:1 approach. The NetBSD team seems to be sticking with M:N. I asked what the secret was. Bill Studenmund gave this answer:

As to what the secret is, a main issue has been getting the exact internal functioning correct. This version of the library has a number of critical code sections (usually a few lines of code each) fixed.

The benchmarks certainly look promising, and Bill adds:

The current library has been used in a number of high-performance applications and is stable.

One side-effect of NetBSD’s focus on portability is the fact that the system is very clean, with well-separated abstraction layers. Geert Hendrickx summed up the NetBSD philosophy with this:

We strive to build a clean, open source, portable, standards-compliant, secure, and performant UNIX-like operating system, and I think we’re doing a good job at that. Many people regard NetBSD as a "reference" UNIX implementation, centered around code quality.

Thanks to all the members of the NetBSD team who answered questions in time for this article, and especially to Geert Hendrickx for his assistance.