- 论坛徽章:
- 0
|
Content: 1. Introduction 2. Configuring the master and the slaves 3. Configuring the DHCP server 4. Configuring the TFTP server and PXE Linux Bootloader and/or Etherboot 5. Configuring the NFS server 6. Completing the slave filesystem
1. Introduction
About this HOWTO
This HOWTO will help you setup diskless workstations based on the Gentoo Linux distribution. We intend to make this as user friendly as possible and cater to the Linux newbie, because every one of us was one at a certain point :) While an experienced user could easily tie the multiple HOWTOs available on diskless nodes and networking together we hope that this guide can ease the installation for all interested users, geeks or not.
What is a diskless machine?
A diskless machine is a PC without any of the usual boot devices such as hard disks, floppy drives or CD-ROMs. The diskless node boots off the network and needs a server that will provide it with storage space as a local hard disk would. From now on we call the server the master, while the diskless machine gets called the slave (what's in a name :). The slave node needs a network adapter that supports PXE booting or Etherboot; check Etherboot.org for support listings. Most modern cards support PXE and many built-in adapters on motherboards will also work.
Before you start
You should have Gentoo installed on your master node and enough space on the master to store the file systems of the slave nodes you want to host. Also make sure you have one interface to the internet separated from the local area connection.
2. Configuring the master and the slaves
About kernels
Note: If you are going to cluster your nodes into an openMosix cluster, make sure you use the patched kernel for openMosix. It can be found in portage under sys-kernel/openmosix-sources. You should read our openMosix HOWTO to learn how to compile your kernel for openMosix.
The kernel is the software that sits between your hardware and all other software you have loaded on your machine, essentially the heart of a kernel based operating system. When your computer is started, the BIOS executes the instructions found at the reserved boot space of your hard drive. These instructions are typically a boot loader that loads your kernel. After your kernel has been loaded all processes are handled by the kernel.
For more information on kernels and kernel configuration you might want to check out the kernel HOWTO.
Configuring the master kernel
The master kernel can be as large and as customized as you would like but there are a few required kernel options you need to select. Go into your kernel configuration menu by typing:
Code Listing 2.1: Editing the master's kernel configuration
# cd /usr/src/linux
# make menuconfig
You should get a grey and blue GUI that offers a safe alternative to manually editing the /usr/src/linux/.config file. If your kernel is currently functioning well you might want to save the current configuration file by exiting the GUI and type:
Code Listing 2.2: Backing up the master's kernel configuration
# cp .config .config_working
Go into the following sub-menus and make sure the listed items are checked as built-in (and NOT as modular). The options show below are taken from the 2.6.10 kernel version. If you use a different version, the text or sequence might differ. Just make sure you select at least those shown below.
Code Listing 2.3: master's kernel options
Code maturity level options --->
Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking options --->
Packet socket
Unix domain sockets
TCP/IP networking
IP: multicasting
[ ] Network packet filtering (replaces ipchains)
File systems --->
Network File Systems --->
NFS server support
Provide NFSv3 server support
If you want to access the internet through your master node and/or have a
secure firewall make sure to add support for iptables
Network packet filtering (replaces ipchains)
IP: Netfilter Configuration --->
Connection tracking (required for masq/NAT)
IP tables support (required for filtering/masq/NAT)
If you want to use packet filtering, you can add the rest as modules later. Make sure to read the Gentoo Security Handbook Chapter about Firewalls on how to set this up properly.
Note: These kernel configuration options should only be added to your system specific configuration options and are not meant to completely replace your kernel configuration.
After you have re-configured the master's kernel you will want to rebuild it:
Code Listing 2.4: Recompiling the master's kernel and modules
# make && make modules_install
(Make sure /boot is mounted before copying to it)
# cp arch/i386/boot/bzImage /boot/bzImage-master
Then add an entry for that new kernel into lilo.conf or grub.conf depending on which bootloader you are using and make the new kernel the default one. Now that the new bzImage has been copied into your boot directory all you will have to do is reboot the system in order to load these new options.
About the slave kernel
It is recommended that you compile the slave kernel without any modules, since loading and setting them up via remote boot is a difficult and unnecessary process. Additionally, the slave kernel should be as small and compact as possible in order to efficiently boot from the network. We are going to compile the slave's kernel in the same place where the master was configured.
To avoid confusion and wasting time it is probably a good idea to backup the master's configuration file by typing:
Code Listing 2.5: Backing up the master's kernel configuration
# cp /usr/src/linux/.config /usr/src/linux/.config_master
Now we will want to configure the slave's kernel in the same fashion we configured the master's kernel. If you want to start with a fresh configuration file you can always recover the default /usr/src/linux/.config file by typing:
Code Listing 2.6: Getting a clean kernel configuration
# cd /usr/src/linux
# cp .config_master .config
Now go into the configuration GUI by typing:
Code Listing 2.7: Editing the slave's kernel configuration
# cd /usr/src/linux
# make menuconfig
You will want to make sure you select the following options as built-in and NOT as kernel modules:
Code Listing 2.8: slave's kernel options
Code maturity level options --->
Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking support
Networking options --->
Packet socket
Unix domain sockets
TCP/IP networking
IP: multicasting
IP: kernel level autoconfiguration
IP: DHCP support (NEW)
File systems --->
Network File Systems --->
file system support
Provide NFSv3 client support
Root file system on NFS
Note: An alternative to having an dhcp server is setting up a BOOTP server.
Important: It is important that you add your network adapter into the kernel (and not as a module) on the nodes. Using modules however is generally not a problem for diskless nodes.
Now the slave's kernel needs to be compiled. You have to be careful here because you don't want to mess up the modules (if any) you have built for the master:
Code Listing 2.9: Compiling the slave kernel
# cd /usr/src/linux
# make
Now create the directory on the master that will be used to hold slaves' files and required system files. We use /diskless but you may choose any location you like. Now copy the slave's bzImage into the /diskless directory:
Note: If you are using different architectures you might want to save each config into .config_arch. Do the same with the images: save them into the /diskless as bzImage_arch.
Code Listing 2.10: Copying the slave kernel
# mkdir /diskless
# cp /usr/src/linux/arch/i386/boot/bzImage /diskless
Configuring a preliminary slave file system
The master and slave filesystems can be tweaked and changed a lot. Right now we are only interested in getting a preliminary filesystem of appropriate configuration files and mount points. First we need to create a directory within /diskless for the first slave. Each slave needs it's own root file system because sharing certain system files will cause permission problems and hard crashes. You can call these directories anything you want but I suggest using the slaves IP addresses as they are unique and not confusing. The static IP of our first slave will be, for instance, 192.168.1.21:
Code Listing 2.11: Creating a remote root directory
# mkdir /diskless/192.168.1.21
Various configuration files in /etc need to be altered to work on the slave. Copy the master's /etc directory onto your new slave root by typing:
Code Listing 2.12: Creating /etc for the slave's filesystem
# cp -r /etc /diskless/192.168.1.21/etc
Still this filesystem isn't ready because it needs various mount points and directories. To create them, type:
Code Listing 2.13: Creating mount points and directories in the slave's filesystem
# mkdir /diskless/192.168.1.21/home
# mkdir /diskless/192.168.1.21/dev
# mkdir /diskless/192.168.1.21/proc
# mkdir /diskless/192.168.1.21/tmp
# mkdir /diskless/192.168.1.21/mnt
# chmod a+w /diskless/192.168.1.21/tmp
# mkdir /diskless/192.168.1.21/mnt/.initd
# mkdir /diskless/192.168.1.21/root
# mkdir /diskless/192.168.1.21/sys
# mkdir /diskless/192.168.1.21/var
# mkdir /diskless/192.168.1.21/var/empty
# mkdir /diskless/192.168.1.21/var/lock
# mkdir /diskless/192.168.1.21/var/log
# mkdir /diskless/192.168.1.21/var/run
# mkdir /diskless/192.168.1.21/var/spool
# mkdir /diskless/192.168.1.21/usr
# mkdir /diskless/192.168.1.21/opt
(for openMosix only)
# mkdir /diskless/192.168.1.21/mfs
Most of these "stubs" should be recognizable to you; stubs like /dev, /proc or /sys will be populated when the slave starts, the others will be mounted later. You should also change the /diskless/192.168.1.21/etc/conf.d/hostname file to reflect the hostname of the slave. Binaries, libraries and other files will be populated later in this HOWTO right before you attempt to boot the slave.
Even though /dev is populated by udev later on, you need to create the console entry. If not, you will receive the error "unable to open initial console".
Code Listing 2.14: Creating console entry in the /dev
# mknod /diskless/192.168.1.21/dev/console c 5 1
3. Configuring the DHCP server
About the DHCP server
DHCP stands for Dynamic Host Configuration Protocol. The DHCP server is the first computer the slaves will communicate with when they PXE boot. The primary purpose of the DHCP server is to assign IP addresses. The DHCP server can assign IP addresses based on hosts ethernet MAC addresses. Once the slave has an IP address, the DHCP server will tell the slave where to get its initial file system and kernel.
Before you get started
There are several things you will want to make sure are working before you begin. First check your network connectivity:
Code Listing 3.1: Checking networking configurations
# ifconfig eth0 multicast
# ifconfig -a
You will want to make sure you have have an eth0 device running. It should look something like this:
Code Listing 3.2: A properly working eth0 device
eth0 Link encap:Ethernet HWaddr 00:E0:83:16:2F:D6
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26460491 errors:0 dropped:0 overruns:2 frame:0
TX packets:32903198 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:100
RX bytes:2483502568 (2368.4 Mb) TX bytes:1411984950 (1346.5 Mb)
Interrupt:18 Base address:0x1800
It's important that it says MULTICAST, if it doesn't then you will have to recompile your kernel to include multicast support.
Installing the DHCP server
If your network does not already have a DHCP server installed you will need to install one:
Code Listing 3.3: Installing the dhcp server
# emerge dhcp
If your network already has a DHCP server installed you will have to edit the configuration file to get the PXE boot to function correctly.
Configuring the DHCP server
There is only one configuration file you will have to edit before starting the DHCP server: /etc/dhcp/dhcpd.conf. Copy and edit the provided sample file:
Code Listing 3.4: Editing the dhcp server's configuration file
# cp /etc/dhcp/dhcpd.conf.sample /etc/dhcp/dhcpd.conf
# nano -w /etc/dhcp/dhcpd.conf
The general layout of the file is set up in an indented fashion and looks like this:
Code Listing 3.5: Sample dhcpd.conf layout
# global options here
ddns-update-style none;
shared-network LOCAL-NET {
# shared network options here
subnet 192.168.1.0 netmask 255.255.255.0 {
# subnet network options here
host slave{
# host specific options here
}
group {
# group specific options here
}
}
}
The shared-network block is optional and should be used for IPs you want to assign that belong to the same network topology. At least one subnet must be declared and the optional group block allows you to group options between items. A good example of dhcpd.conf looks like this:
Code Listing 3.6: Sample dhcpd.conf
# DHCP configuration file for DHCP ISC 3.0
ddns-update-style none;
# Definition of PXE-specific options
# Code 1: Multicast IP address of boot file server
# Code 2: UDP port that client should monitor for MTFTP responses
# Code 3: UDP port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
# to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
# a MTFTP transfer
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option PXE.discovery-control code 6 = unsigned integer 8;
option PXE.discovery-mcast-addr code 7 = ip-address;
subnet 192.168.1.0 netmask 255.255.255.0 {
class "pxeclients" {
match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
option vendor-class-identifier "PXEClient";
vendor-option-space PXE;
# At least one of the vendor-specific PXE options must be set in
# order for the client boot ROMs to realize that we are a PXE-compliant
# server. We set the MCAST IP address to 0.0.0.0 to tell the boot ROM
# that we can't provide multicast TFTP (address 0.0.0.0 means no
# address).
option PXE.mtftp-ip 0.0.0.0;
# This is the name of the file the boot ROMs should download.
filename "pxelinux.0";
# This is the name of the server they should get it from.
# Use the master's IP
next-server 192.168.1.1;
}
# If you are using etherboot with a non specific image
class "etherboot" {
if substring (option vendor-class-identifier, 0, 9) = "Etherboot" {
filename "/diskless/vmlinuz";
}
}
pool {
max-lease-time 86400;
default-lease-time 86400;
# This prevents unlisted machines from getting an IP
deny unknown-clients;
}
host slave21 {
# Use your slave's MAC address
hardware ethernet 00:40:63:C2:CA:C9;
# Give your slave a static IP
fixed-address 192.168.1.21;
server-name "master";
# Use your gateway IP, if required
option routers 192.168.1.1;
# Use your DNS IP, if required
option domain-name-servers 192.168.1.1;
option domain-name "mydomain.com";
# Use your slave hostname
option host-name "slave21";
# Etherboot and pxe boot with a mac specific image
option root-path "/diskless/192.168.1.21";
if substring (option vendor-class-identifier, 0, 9) = "Etherboot" {
filename "/vmlinuz_arch";
} else if substring (option vendor-class-identifier, 0,9) ="PXEClient" {
filename "/pxelinux.0";
}
}
}
Note: There is nothing prohibiting the use of both PXE boot and Etherboot together. The above Code Listing is merely an example; if you have issues, please consult the DHCPd documentation.
The IP address after next-server will be asked for the specified filename. This IP address should be the IP of the tftp server, usually the same as the master's IP address. The filename is relative to the /diskless directory (this is due to the tftp server specific options which will be covered later). Inside the host block, the hardware ethernet option specifies a MAC address, and fixed-address assigns a fixed IP address to that particular MAC address. The host-name option is probably a good idea to include and is just the hostname of a particular slave. There is a pretty good man page on dhcpd.conf with options that are beyond the scope of this HOWTO. You can read it by typing:
Code Listing 3.7: Viewing the man pages for dhcpd.conf
# man dhcpd.conf
Starting the DHCP server
Before you start the dhcp initialisation script edit the /etc/conf.d/dhcp file so that it looks something like this:
Code Listing 3.8: Sample /etc/conf.d/dhcp
IFACE="eth0"
# insert any other options needed
The IFACE variable is the device you wish to run your DHCP server on, in our case eth0. Adding more arguments to the IFACE variable can be useful for a complex network topology with multiple Ethernet cards. To start the dhcp server type:
Code Listing 3.9: Starting the dhcp server on the master
# /etc/init.d/dhcp start
To add the dhcp server to your start-up scripts type:
Code Listing 3.10: Adding the dhcp server to the master's default run level
# rc-update add dhcp default
Troubleshooting the DHCP server
To see if a node boots you can take a look at /var/log/syslog.log. If the node successfully boots, the syslog.log file should have some lines at the bottom looking like this:
Code Listing 3.11: Sample log file entries created by dhcp
DHCPDISCOVER from 00:00:00:00:00:00 via eth0
DHCPOFFER on 192.168.1.21 to 00:00:00:00:00:00 via eth0
DHCPREQUEST for 192.168.1.21 from 00:00:00:00:00:00 via eth0
DHCPACK on 192.168.1.21 to 00:00:00:00:00:00 via eth0
Note: This log file can also help you discover the slaves' MAC addresses.
If you get the following message it probably means there is something wrong in the configuration file but that the DHCP server is broadcasting correctly.
Code Listing 3.12: Sample dhpc server error
no free leases on subnet LOCAL-NET
Every time you change the configuration file you must restart the DHCP server. To restart the server type:
Code Listing 3.13: Restarting the dhcp server on the master
# /etc/init.d/dhcpd restart
4. Configuring the TFTP server and PXE Linux Bootloader and/or Etherboot
About the TFTP server
TFTP stands for Trivial File Transfer Protocol. The TFTP server is going to supply the slaves with a kernel and an initial filesystem. All of the slave kernels and filesystems will be stored on the TFTP server, so it's probably a good idea to make the master the TFTP server.
Installing the TFTP server
A highly recommended tftp server is available as the tftp-hpa package. This tftp server happens to be written by the author of SYSLINUX and it works very well with pxelinux. To install simply type:
Code Listing 4.1: Installing the tfp server
# emerge tftp-hpa
Configuring the TFTP server
Edit /etc/conf.d/in.tftpd. You need to specify the tftproot directory with INTFTPD_PATH and any command line options with INTFTPD_OPTS. It should look something like this:
Code Listing 4.2: Sample /etc/conf.d/in.tftpd
INTFTPD_PATH="/diskless"
INTFTPD_OPTS="-l -v -s ${INTFTPD_PATH}"
The -l option indicates that this server listens in stand alone mode so you don't have to run inetd. The -v indicates that log/error messages should be verbose. The -s /diskless specifies the root of your tftp server.
Starting the the TFTP Server
To start the tftp server type:
Code Listing 4.3: Starting the master's tftp server
# /etc/init.d/in.tftpd start
This should start the tftp server with the options you specified in the /etc/conf.d/in.tftpd. If you want this server to be automatically started at boot type:
Code Listing 4.4: Adding the tftp server to the master's default run level
# rc-update add in.tftpd default
About PXELINUX
This section is not required if you are only using Etherboot. PXELINUX is the network bootloader equivalent to LILO or GRUB and will be served via TFTP. It is essentially a tiny set of instructions that tells the client where to locate its kernel and initial filesystem and allows for various kernel options.
Before you get started
You will need to get the pxelinux.0 file which comes in the SYSLINUX package by H. Peter Anvin. You can install this package by typing:
Code Listing 4.5: Installing syslinux
# emerge syslinux
Setting up PXELINUX
Note: This isn't needed for Etherboot
Before you start your tftp server you need to setup pxelinux. First copy the pxelinux binary into your /diskless directory:
Code Listing 4.6: Setting up the remote bootloader
# cp /usr/lib/syslinux/pxelinux.0 /diskless
# mkdir /diskless/pxelinux.cfg
# touch /diskless/pxelinux.cfg/default
This will create a default bootloader configuration file. The binary pxelinux.0 will look in the pxelinux.cfg directory for a file whose name is the client's IP address in hexadecimal. If it does not find that file it will remove the rightmost digit from the file name and try again until it runs out of digits. Versions 2.05 and later of syslinux first perform a search for a file named after the MAC address. If no file is found, it starts the previously mentioned discovery routine. If none is found, the default file is used.
Code Listing 4.7: Files that PXE looks for in pxelinux.cfg/ in sequence
(Leading 01 means Ethernet, next bytes match our slave's MAC address)
01-00-40-63-c2-ca-c9
(Assigned IP in hexadecimal)
C0A80115
C0A8011
C0A801
C0A80
C0A8
C0A
C0
C
default
Note: These are all in lowercase.
Let's start with the default file:
Code Listing 4.8: Sample pxelinux.cfg/default
DEFAULT /bzImage
APPEND ip=dhcp root=/dev/nfs nfsroot=192.168.1.1:/diskless/192.168.1.21
The DEFAULT tag directs pxelinux to the kernel bzImage that we compiled earlier. The APPEND tag appends kernel initialisation options. Since we compiled the slave kernel with NFS_ROOT_SUPPORT, we will specify the nfsroot here. The first IP is the master's IP and the second IP is the directory that was created in /diskless to store the slave's initial filesystem.
About Etherboot
Note: This isn't required if you are using PXE boot.
Etherboot boots network boot images from a TFTP server. As the PXE this is equivalent to LILO or GRUB. The mknbi utility enables you to create different images using different options.
Before you get started
You will need to get the mknbi (utility for making tagged kernel images useful for netbooting) package to create your Etherboot images. This tool will create a preconfigured kernel image from your original kernel. This contains the boot options as shown further down.
Code Listing 4.9: Installing mknbi
# emerge mknbi
Setting up Etherboot
In this section we will create a simple etherboot image. As the dhcp server gives out the clients root-path in the "option root-path" dhcp.conf, we do not have to include this here. More details can be found in the mknbi manual.
Code Listing 4.10: mknbi manual
# man mknbi
Making the boot images. This will create a ELF bootable image capable of passing dhcp and the rootpath to the kernel. Also forcing the kernel to browse the network for a dhcp server.
Code Listing 4.11: making netboot images
# mkelf-linux -ip=dhcp /diskless/bzImage > /diskless/vmlinuz
Note: For the arch specific images you have to type bzImage_arch and vmlinuz_arch.
Troubleshooting the network boot process
There are a few things you can do to debug the network boot process. Primarily you can use a tool called tcpdump. To install tcpdump type:
Code Listing 4.12: Installing tcpdump
# emerge tcpdump
Now you can listen to various network traffic and make sure your client/server interactions are functioning. If something isn't working there are a few things you might want to check. First make sure that the client/server is physically connected properly and that the networking cables are not damaged. If your client/server is not receiving requests on a particular port make sure that there is no firewall interference. To listen to interaction between two computers type:
Code Listing 4.13: Listening to client and server interaction via tcpdump
# tcpdump host client_ip and server_ip
You can also use tcpdump to listen on particular port such as the tftp port by typing:
Code Listing 4.14: Listening to the tftp server
# tcpdump port 69
A common error you might receive is: "PXE-E32: TFTP open time-out". This is probably due to firewall issues. If you are using TCPwrappers, you might want to check /etc/hosts.allow and etc/hosts.deny and make sure that they are configured properly. The client should be allowed to connect to the server.
5. Configuring the NFS server
About the NFS server
NFS stands for Network File System. The NFS server will be used to serve directories to the slave. This part can be somewhat personalized later, but right now all we want is a preliminary slave node to boot diskless.
About Portmapper
Various client/server services do not listen on a particular port, but instead rely on RPCs (Remote Procedure Calls). When the service is initialised it listens on a random port and then registers this port with the Portmapper utility. NFS relies on RPCs and thus requires Portmapper to be running before it is started.
Before you start
The NFS Server needs kernel level support so if you don't have this you should recompile your master's kernel. To double check your master's kernel configuration type:
Code Listing 5.1: Checking for NFS specific options
# grep NFS /usr/src/linux/.config_master
You should see output that looks something like this if your kernel has been properly configured:
Code Listing 5.2: Proper NFS specific options in the master's kernel configuration
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
# CONFIG_NETFILTER is not set
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V4 is not set
# CONFIG_NFSD_TCP is not set
Installing the NFS server
The NFS package that can be acquired through portage by typing:
Code Listing 5.3: Installing nfs-utils
# emerge nfs-utils
This package will emerge a portmapping utility, nfs server, and nfs client utilities and will automatically handle initialisation dependencies.
Configuring the NFS server
There are three major configuration files you will have to edit:
Code Listing 5.4: Nfs configuration files
/etc/exports
/diskless/192.168.1.21/etc/fstab
/etc/conf.d/nfs
The /etc/exports file specifies how, to who and what to export through NFS. The slave's fstab will be altered so that it can mount the NFS filesystems that the master is exporting.
A typical /etc/exports for the master should look something like this:
Code Listing 5.5: Sample master /etc/exports
# one line like this for each slave
/diskless/192.168.1.21 192.168.1.21(sync,rw,no_root_squash,no_all_squash)
# common to all slaves
/opt 192.168.1.0/24(sync,ro,no_root_squash,no_all_squash)
/usr 192.168.1.0/24(sync,ro,no_root_squash,no_all_squash)
/home 192.168.1.0/24(sync,rw,no_root_squash,no_all_squash)
# if you want to have a shared log
/var/log 192.168.1.21(sync,rw,no_root_squash,no_all_squash)
The first field indicates the directory to be exported and the next field indicates to who and how. This field can be divided in two parts: who should be allowed to mount that particular directory, and what the mounting client can do to the filesystem: ro for read only, rw for read/write; no_root_squash and no_all_squash are important for diskless clients that are writing to the disk, so that they don't get "squashed" when making I/O requests. The slave's fstab file, /diskless/192.168.1.21/etc/fstab, should look like this:
Code Listing 5.6: Sample slave fstab
# these entries are essential
master:/diskless/192.168.1.21 / nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
master:/opt /opt nfs sync,hard,intr,ro,nolock,rsize=8192,wsize=8192 0 0
master:/usr /usr nfs sync,hard,intr,ro,nolock,rsize=8192,wsize=8192 0 0
master:/home /home nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
none /proc proc defaults 0 0
# useful but superfluous
master:/var/log /var/log nfs hard,intr,rw 0 0
(if you are setting up an openMosix cluster only)
none /mfs mfs dfsa=1 0 0
In this example, master is just the hostname of the master but it could easily be the IP of the master. The first field indicates the directory to be mounted and the second field indicates where. The third field describes the filesystem and should be NFS for any NFS mounted directory. The fourth field indicates various options that will be used in the mounting process (see mount(1) for info on mount options). Some people have had difficulties with soft mount points so we made them all hard, but you should look into various /etc/fstab options to make your cluster more efficient.
The last file you should edit is /etc/conf.d/nfs which describes a few options for nfs when it is initialised and looks like this:
Code Listing 5.7: Sample master /etc/conf.d/nfs
# Config file for /etc/init.d/nfs
# Number of servers to be started up by default
RPCNFSDCOUNT=8
# Options to pass to rpc.mountd
RPCMOUNTDOPTS=""
You should change RPCNFSDCOUNT to the number of diskless nodes on the network.
Starting the NFS server
You should start the nfs server with its init script located in /etc/init.d by typing:
Code Listing 5.8: Starting the master's nfs server
# /etc/init.d/nfs start
If you want to this script to start when the system boots simply type:
Code Listing 5.9: Adding the nfs server to the master's default run level
# rc-update add nfs default
6. Completing the slave filesystem
Copy the missing files
We will now make the slave's file system in sync with the master's and provide the necessary binaries while still preserving slave specific files.
Code Listing 6.1: Creating a slave filesystem
# rsync -avz /bin /diskless/192.168.1.21
# rsync -avz /sbin /diskless/192.168.1.21
# rsync -avz /lib /diskless/192.168.1.21
Note: The reason for rsync -avz instead of cp is to maintain symlinks and permissions
Initialisation scripts
The default scripts will try to run checkroot which does not make sense on your slave nodes. The hard way out is to manually edit the /diskless/192.168.1.21/sbin/rc script but this is cumbersome, dangerous and could break if you decided to sync your node file system again and forgot to leave this script alone. The trick is to have a /fastboot file when your system boots. This file tells checkroot not to run any file system check. But it will also erase the file when it has finished the initialisation process. That is why we need to create this file again at the end of the init process like this:
Code Listing 6.2: Preventing init scripts from running a file system check
(Create the /fastboot file for next reboot)
# touch /diskless/192.168.1.21/fastboot
(Create the /fastboot file on each boot)
# echo "touch /fastboot" >> /diskless/192.168.1.21/etc/conf.d/local.start
Since the unmounting of network filesystems should be done as late as possible, you have to edit your /etc/init.d/netmount and have it read:
Code Listing 6.3: Editing /etc/init.d/netmount
depend() {
before *
Note: Baselayout version 1.11.* and later does not need this.
If you do this on a live system, don't forget to run depscan.sh to fix the service dependencies. You can safely ignore any warning about the collision with /etc/init.d/checkroot, since you are disabling it via the fastboot file you set up in the previous paragraph.
You need as many init scripts under /diskless/192.168.1.21/etc/runlevels as you need services on your diskless nodes. It all depends on what you want your slaves to do.
Warning: Do not use the rc-update program to add or remove scripts from the slave runlevels when logged on your master. This would change your master runlevels. You need to create the links manually or log into your slave nodes using ssh or connect a screen and keyboard to your slave.
Code Listing 6.4: Typical slave runlevels
/diskless/192.168.1.21/etc/runlevels/:
total 16
drwxr-xr-x 2 root root 4096 2003-11-09 15:27 boot
drwxr-xr-x 2 root root 4096 2003-10-01 21:10 default
drwxr-xr-x 2 root root 4096 2003-03-13 19:05 nonetwork
drwxr-xr-x 2 root root 4096 2003-02-23 12:26 single
/diskless/192.168.1.21/etc/runlevels/boot:
total 0
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 bootmisc -> /etc/init.d/bootmisc
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 checkfs -> /etc/init.d/checkfs
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 clock -> /etc/init.d/clock
lrwxrwxrwx 1 root root 23 2003-10-18 17:28 consolefont -> /etc/init.d/consolefont
lrwxrwxrwx 1 root root 22 2003-10-18 17:28 domainname -> /etc/init.d/domainname
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 hostname -> /etc/init.d/hostname
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 keymaps -> /etc/init.d/keymaps
lrwxrwxrwx 1 root root 22 2003-10-18 17:28 localmount -> /etc/init.d/localmount
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 modules -> /etc/init.d/modules
lrwxrwxrwx 1 root root 18 2003-10-18 17:28 net.lo -> /etc/init.d/net.lo
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 netmount -> /etc/init.d/netmount
lrwxrwxrwx 1 root root 21 2003-10-18 17:28 rmnologin -> /etc/init.d/rmnologin
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 urandom -> /etc/init.d/urandom
/diskless/192.168.1.21/etc/runlevels/default:
total 0
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 distccd -> /etc/init.d/distccd
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 local -> /etc/init.d/local
lrwxrwxrwx 1 root root 16 2003-10-18 17:28 sshd -> /etc/init.d/sshd
lrwxrwxrwx 1 root root 21 2003-10-18 17:28 syslog-ng -> /etc/init.d/syslog-ng
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 vixie-cron -> /etc/init.d/vixie-cron
/diskless/192.168.1.21/etc/runlevels/nonetwork:
total 0
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 local -> /etc/init.d/local
/diskless/192.168.1.21/etc/runlevels/single:
total 0
Now is a good time to boot your slave and cross your fingers. It works? Congratulations, you are now the proud owner of (a) diskless node(s) :)
Content: 1. Introduction 2. Configuring the master and the slaves 3. Configuring the DHCP server 4. Configuring the TFTP server and PXE Linux Bootloader and/or Etherboot 5. Configuring the NFS server 6. Completing the slave filesystem
1. Introduction
About this HOWTO
This HOWTO will help you setup diskless workstations based on the Gentoo Linux distribution. We intend to make this as user friendly as possible and cater to the Linux newbie, because every one of us was one at a certain point :) While an experienced user could easily tie the multiple HOWTOs available on diskless nodes and networking together we hope that this guide can ease the installation for all interested users, geeks or not.
What is a diskless machine?
A diskless machine is a PC without any of the usual boot devices such as hard disks, floppy drives or CD-ROMs. The diskless node boots off the network and needs a server that will provide it with storage space as a local hard disk would. From now on we call the server the master, while the diskless machine gets called the slave (what's in a name :). The slave node needs a network adapter that supports PXE booting or Etherboot; check Etherboot.org for support listings. Most modern cards support PXE and many built-in adapters on motherboards will also work.
Before you start
You should have Gentoo installed on your master node and enough space on the master to store the file systems of the slave nodes you want to host. Also make sure you have one interface to the internet separated from the local area connection.
2. Configuring the master and the slaves
About kernels
Note: If you are going to cluster your nodes into an openMosix cluster, make sure you use the patched kernel for openMosix. It can be found in portage under sys-kernel/openmosix-sources. You should read our openMosix HOWTO to learn how to compile your kernel for openMosix.
The kernel is the software that sits between your hardware and all other software you have loaded on your machine, essentially the heart of a kernel based operating system. When your computer is started, the BIOS executes the instructions found at the reserved boot space of your hard drive. These instructions are typically a boot loader that loads your kernel. After your kernel has been loaded all processes are handled by the kernel.
For more information on kernels and kernel configuration you might want to check out the kernel HOWTO.
Configuring the master kernel
The master kernel can be as large and as customized as you would like but there are a few required kernel options you need to select. Go into your kernel configuration menu by typing:
Code Listing 2.1: Editing the master's kernel configuration
# cd /usr/src/linux
# make menuconfig
You should get a grey and blue GUI that offers a safe alternative to manually editing the /usr/src/linux/.config file. If your kernel is currently functioning well you might want to save the current configuration file by exiting the GUI and type:
Code Listing 2.2: Backing up the master's kernel configuration
# cp .config .config_working
Go into the following sub-menus and make sure the listed items are checked as built-in (and NOT as modular). The options show below are taken from the 2.6.10 kernel version. If you use a different version, the text or sequence might differ. Just make sure you select at least those shown below.
Code Listing 2.3: master's kernel options
Code maturity level options --->
Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking options --->
Packet socket
Unix domain sockets
TCP/IP networking
IP: multicasting
[ ] Network packet filtering (replaces ipchains)
File systems --->
Network File Systems --->
NFS server support
Provide NFSv3 server support
If you want to access the internet through your master node and/or have a
secure firewall make sure to add support for iptables
Network packet filtering (replaces ipchains)
IP: Netfilter Configuration --->
Connection tracking (required for masq/NAT)
IP tables support (required for filtering/masq/NAT)
If you want to use packet filtering, you can add the rest as modules later. Make sure to read the Gentoo Security Handbook Chapter about Firewalls on how to set this up properly.
Note: These kernel configuration options should only be added to your system specific configuration options and are not meant to completely replace your kernel configuration.
After you have re-configured the master's kernel you will want to rebuild it:
Code Listing 2.4: Recompiling the master's kernel and modules
# make && make modules_install
(Make sure /boot is mounted before copying to it)
# cp arch/i386/boot/bzImage /boot/bzImage-master
Then add an entry for that new kernel into lilo.conf or grub.conf depending on which bootloader you are using and make the new kernel the default one. Now that the new bzImage has been copied into your boot directory all you will have to do is reboot the system in order to load these new options.
About the slave kernel
It is recommended that you compile the slave kernel without any modules, since loading and setting them up via remote boot is a difficult and unnecessary process. Additionally, the slave kernel should be as small and compact as possible in order to efficiently boot from the network. We are going to compile the slave's kernel in the same place where the master was configured.
To avoid confusion and wasting time it is probably a good idea to backup the master's configuration file by typing:
Code Listing 2.5: Backing up the master's kernel configuration
# cp /usr/src/linux/.config /usr/src/linux/.config_master
Now we will want to configure the slave's kernel in the same fashion we configured the master's kernel. If you want to start with a fresh configuration file you can always recover the default /usr/src/linux/.config file by typing:
Code Listing 2.6: Getting a clean kernel configuration
# cd /usr/src/linux
# cp .config_master .config
Now go into the configuration GUI by typing:
Code Listing 2.7: Editing the slave's kernel configuration
# cd /usr/src/linux
# make menuconfig
You will want to make sure you select the following options as built-in and NOT as kernel modules:
Code Listing 2.8: slave's kernel options
Code maturity level options --->
Prompt for development and/or incomplete code/drivers
Device Drivers --->
Networking support
Networking options --->
Packet socket
Unix domain sockets
TCP/IP networking
IP: multicasting
IP: kernel level autoconfiguration
IP: DHCP support (NEW)
File systems --->
Network File Systems --->
file system support
Provide NFSv3 client support
Root file system on NFS
Note: An alternative to having an dhcp server is setting up a BOOTP server.
Important: It is important that you add your network adapter into the kernel (and not as a module) on the nodes. Using modules however is generally not a problem for diskless nodes.
Now the slave's kernel needs to be compiled. You have to be careful here because you don't want to mess up the modules (if any) you have built for the master:
Code Listing 2.9: Compiling the slave kernel
# cd /usr/src/linux
# make
Now create the directory on the master that will be used to hold slaves' files and required system files. We use /diskless but you may choose any location you like. Now copy the slave's bzImage into the /diskless directory:
Note: If you are using different architectures you might want to save each config into .config_arch. Do the same with the images: save them into the /diskless as bzImage_arch.
Code Listing 2.10: Copying the slave kernel
# mkdir /diskless
# cp /usr/src/linux/arch/i386/boot/bzImage /diskless
Configuring a preliminary slave file system
The master and slave filesystems can be tweaked and changed a lot. Right now we are only interested in getting a preliminary filesystem of appropriate configuration files and mount points. First we need to create a directory within /diskless for the first slave. Each slave needs it's own root file system because sharing certain system files will cause permission problems and hard crashes. You can call these directories anything you want but I suggest using the slaves IP addresses as they are unique and not confusing. The static IP of our first slave will be, for instance, 192.168.1.21:
Code Listing 2.11: Creating a remote root directory
# mkdir /diskless/192.168.1.21
Various configuration files in /etc need to be altered to work on the slave. Copy the master's /etc directory onto your new slave root by typing:
Code Listing 2.12: Creating /etc for the slave's filesystem
# cp -r /etc /diskless/192.168.1.21/etc
Still this filesystem isn't ready because it needs various mount points and directories. To create them, type:
Code Listing 2.13: Creating mount points and directories in the slave's filesystem
# mkdir /diskless/192.168.1.21/home
# mkdir /diskless/192.168.1.21/dev
# mkdir /diskless/192.168.1.21/proc
# mkdir /diskless/192.168.1.21/tmp
# mkdir /diskless/192.168.1.21/mnt
# chmod a+w /diskless/192.168.1.21/tmp
# mkdir /diskless/192.168.1.21/mnt/.initd
# mkdir /diskless/192.168.1.21/root
# mkdir /diskless/192.168.1.21/sys
# mkdir /diskless/192.168.1.21/var
# mkdir /diskless/192.168.1.21/var/empty
# mkdir /diskless/192.168.1.21/var/lock
# mkdir /diskless/192.168.1.21/var/log
# mkdir /diskless/192.168.1.21/var/run
# mkdir /diskless/192.168.1.21/var/spool
# mkdir /diskless/192.168.1.21/usr
# mkdir /diskless/192.168.1.21/opt
(for openMosix only)
# mkdir /diskless/192.168.1.21/mfs
Most of these "stubs" should be recognizable to you; stubs like /dev, /proc or /sys will be populated when the slave starts, the others will be mounted later. You should also change the /diskless/192.168.1.21/etc/conf.d/hostname file to reflect the hostname of the slave. Binaries, libraries and other files will be populated later in this HOWTO right before you attempt to boot the slave.
Even though /dev is populated by udev later on, you need to create the console entry. If not, you will receive the error "unable to open initial console".
Code Listing 2.14: Creating console entry in the /dev
# mknod /diskless/192.168.1.21/dev/console c 5 1
3. Configuring the DHCP server
About the DHCP server
DHCP stands for Dynamic Host Configuration Protocol. The DHCP server is the first computer the slaves will communicate with when they PXE boot. The primary purpose of the DHCP server is to assign IP addresses. The DHCP server can assign IP addresses based on hosts ethernet MAC addresses. Once the slave has an IP address, the DHCP server will tell the slave where to get its initial file system and kernel.
Before you get started
There are several things you will want to make sure are working before you begin. First check your network connectivity:
Code Listing 3.1: Checking networking configurations
# ifconfig eth0 multicast
# ifconfig -a
You will want to make sure you have have an eth0 device running. It should look something like this:
Code Listing 3.2: A properly working eth0 device
eth0 Link encap:Ethernet HWaddr 00:E0:83:16:2F:D6
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26460491 errors:0 dropped:0 overruns:2 frame:0
TX packets:32903198 errors:0 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:100
RX bytes:2483502568 (2368.4 Mb) TX bytes:1411984950 (1346.5 Mb)
Interrupt:18 Base address:0x1800
It's important that it says MULTICAST, if it doesn't then you will have to recompile your kernel to include multicast support.
Installing the DHCP server
If your network does not already have a DHCP server installed you will need to install one:
Code Listing 3.3: Installing the dhcp server
# emerge dhcp
If your network already has a DHCP server installed you will have to edit the configuration file to get the PXE boot to function correctly.
Configuring the DHCP server
There is only one configuration file you will have to edit before starting the DHCP server: /etc/dhcp/dhcpd.conf. Copy and edit the provided sample file:
Code Listing 3.4: Editing the dhcp server's configuration file
# cp /etc/dhcp/dhcpd.conf.sample /etc/dhcp/dhcpd.conf
# nano -w /etc/dhcp/dhcpd.conf
The general layout of the file is set up in an indented fashion and looks like this:
Code Listing 3.5: Sample dhcpd.conf layout
# global options here
ddns-update-style none;
shared-network LOCAL-NET {
# shared network options here
subnet 192.168.1.0 netmask 255.255.255.0 {
# subnet network options here
host slave{
# host specific options here
}
group {
# group specific options here
}
}
}
The shared-network block is optional and should be used for IPs you want to assign that belong to the same network topology. At least one subnet must be declared and the optional group block allows you to group options between items. A good example of dhcpd.conf looks like this:
Code Listing 3.6: Sample dhcpd.conf
# DHCP configuration file for DHCP ISC 3.0
ddns-update-style none;
# Definition of PXE-specific options
# Code 1: Multicast IP address of boot file server
# Code 2: UDP port that client should monitor for MTFTP responses
# Code 3: UDP port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
# to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
# a MTFTP transfer
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option PXE.discovery-control code 6 = unsigned integer 8;
option PXE.discovery-mcast-addr code 7 = ip-address;
subnet 192.168.1.0 netmask 255.255.255.0 {
class "pxeclients" {
match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
option vendor-class-identifier "PXEClient";
vendor-option-space PXE;
# At least one of the vendor-specific PXE options must be set in
# order for the client boot ROMs to realize that we are a PXE-compliant
# server. We set the MCAST IP address to 0.0.0.0 to tell the boot ROM
# that we can't provide multicast TFTP (address 0.0.0.0 means no
# address).
option PXE.mtftp-ip 0.0.0.0;
# This is the name of the file the boot ROMs should download.
filename "pxelinux.0";
# This is the name of the server they should get it from.
# Use the master's IP
next-server 192.168.1.1;
}
# If you are using etherboot with a non specific image
class "etherboot" {
if substring (option vendor-class-identifier, 0, 9) = "Etherboot" {
filename "/diskless/vmlinuz";
}
}
pool {
max-lease-time 86400;
default-lease-time 86400;
# This prevents unlisted machines from getting an IP
deny unknown-clients;
}
host slave21 {
# Use your slave's MAC address
hardware ethernet 00:40:63:C2:CA:C9;
# Give your slave a static IP
fixed-address 192.168.1.21;
server-name "master";
# Use your gateway IP, if required
option routers 192.168.1.1;
# Use your DNS IP, if required
option domain-name-servers 192.168.1.1;
option domain-name "mydomain.com";
# Use your slave hostname
option host-name "slave21";
# Etherboot and pxe boot with a mac specific image
option root-path "/diskless/192.168.1.21";
if substring (option vendor-class-identifier, 0, 9) = "Etherboot" {
filename "/vmlinuz_arch";
} else if substring (option vendor-class-identifier, 0,9) ="PXEClient" {
filename "/pxelinux.0";
}
}
}
Note: There is nothing prohibiting the use of both PXE boot and Etherboot together. The above Code Listing is merely an example; if you have issues, please consult the DHCPd documentation.
The IP address after next-server will be asked for the specified filename. This IP address should be the IP of the tftp server, usually the same as the master's IP address. The filename is relative to the /diskless directory (this is due to the tftp server specific options which will be covered later). Inside the host block, the hardware ethernet option specifies a MAC address, and fixed-address assigns a fixed IP address to that particular MAC address. The host-name option is probably a good idea to include and is just the hostname of a particular slave. There is a pretty good man page on dhcpd.conf with options that are beyond the scope of this HOWTO. You can read it by typing:
Code Listing 3.7: Viewing the man pages for dhcpd.conf
# man dhcpd.conf
Starting the DHCP server
Before you start the dhcp initialisation script edit the /etc/conf.d/dhcp file so that it looks something like this:
Code Listing 3.8: Sample /etc/conf.d/dhcp
IFACE="eth0"
# insert any other options needed
The IFACE variable is the device you wish to run your DHCP server on, in our case eth0. Adding more arguments to the IFACE variable can be useful for a complex network topology with multiple Ethernet cards. To start the dhcp server type:
Code Listing 3.9: Starting the dhcp server on the master
# /etc/init.d/dhcp start
To add the dhcp server to your start-up scripts type:
Code Listing 3.10: Adding the dhcp server to the master's default run level
# rc-update add dhcp default
Troubleshooting the DHCP server
To see if a node boots you can take a look at /var/log/syslog.log. If the node successfully boots, the syslog.log file should have some lines at the bottom looking like this:
Code Listing 3.11: Sample log file entries created by dhcp
DHCPDISCOVER from 00:00:00:00:00:00 via eth0
DHCPOFFER on 192.168.1.21 to 00:00:00:00:00:00 via eth0
DHCPREQUEST for 192.168.1.21 from 00:00:00:00:00:00 via eth0
DHCPACK on 192.168.1.21 to 00:00:00:00:00:00 via eth0
Note: This log file can also help you discover the slaves' MAC addresses.
If you get the following message it probably means there is something wrong in the configuration file but that the DHCP server is broadcasting correctly.
Code Listing 3.12: Sample dhpc server error
no free leases on subnet LOCAL-NET
Every time you change the configuration file you must restart the DHCP server. To restart the server type:
Code Listing 3.13: Restarting the dhcp server on the master
# /etc/init.d/dhcpd restart
4. Configuring the TFTP server and PXE Linux Bootloader and/or Etherboot
About the TFTP server
TFTP stands for Trivial File Transfer Protocol. The TFTP server is going to supply the slaves with a kernel and an initial filesystem. All of the slave kernels and filesystems will be stored on the TFTP server, so it's probably a good idea to make the master the TFTP server.
Installing the TFTP server
A highly recommended tftp server is available as the tftp-hpa package. This tftp server happens to be written by the author of SYSLINUX and it works very well with pxelinux. To install simply type:
Code Listing 4.1: Installing the tfp server
# emerge tftp-hpa
Configuring the TFTP server
Edit /etc/conf.d/in.tftpd. You need to specify the tftproot directory with INTFTPD_PATH and any command line options with INTFTPD_OPTS. It should look something like this:
Code Listing 4.2: Sample /etc/conf.d/in.tftpd
INTFTPD_PATH="/diskless"
INTFTPD_OPTS="-l -v -s ${INTFTPD_PATH}"
The -l option indicates that this server listens in stand alone mode so you don't have to run inetd. The -v indicates that log/error messages should be verbose. The -s /diskless specifies the root of your tftp server.
Starting the the TFTP Server
To start the tftp server type:
Code Listing 4.3: Starting the master's tftp server
# /etc/init.d/in.tftpd start
This should start the tftp server with the options you specified in the /etc/conf.d/in.tftpd. If you want this server to be automatically started at boot type:
Code Listing 4.4: Adding the tftp server to the master's default run level
# rc-update add in.tftpd default
About PXELINUX
This section is not required if you are only using Etherboot. PXELINUX is the network bootloader equivalent to LILO or GRUB and will be served via TFTP. It is essentially a tiny set of instructions that tells the client where to locate its kernel and initial filesystem and allows for various kernel options.
Before you get started
You will need to get the pxelinux.0 file which comes in the SYSLINUX package by H. Peter Anvin. You can install this package by typing:
Code Listing 4.5: Installing syslinux
# emerge syslinux
Setting up PXELINUX
Note: This isn't needed for Etherboot
Before you start your tftp server you need to setup pxelinux. First copy the pxelinux binary into your /diskless directory:
Code Listing 4.6: Setting up the remote bootloader
# cp /usr/lib/syslinux/pxelinux.0 /diskless
# mkdir /diskless/pxelinux.cfg
# touch /diskless/pxelinux.cfg/default
This will create a default bootloader configuration file. The binary pxelinux.0 will look in the pxelinux.cfg directory for a file whose name is the client's IP address in hexadecimal. If it does not find that file it will remove the rightmost digit from the file name and try again until it runs out of digits. Versions 2.05 and later of syslinux first perform a search for a file named after the MAC address. If no file is found, it starts the previously mentioned discovery routine. If none is found, the default file is used.
Code Listing 4.7: Files that PXE looks for in pxelinux.cfg/ in sequence
(Leading 01 means Ethernet, next bytes match our slave's MAC address)
01-00-40-63-c2-ca-c9
(Assigned IP in hexadecimal)
C0A80115
C0A8011
C0A801
C0A80
C0A8
C0A
C0
C
default
Note: These are all in lowercase.
Let's start with the default file:
Code Listing 4.8: Sample pxelinux.cfg/default
DEFAULT /bzImage
APPEND ip=dhcp root=/dev/nfs nfsroot=192.168.1.1:/diskless/192.168.1.21
The DEFAULT tag directs pxelinux to the kernel bzImage that we compiled earlier. The APPEND tag appends kernel initialisation options. Since we compiled the slave kernel with NFS_ROOT_SUPPORT, we will specify the nfsroot here. The first IP is the master's IP and the second IP is the directory that was created in /diskless to store the slave's initial filesystem.
About Etherboot
Note: This isn't required if you are using PXE boot.
Etherboot boots network boot images from a TFTP server. As the PXE this is equivalent to LILO or GRUB. The mknbi utility enables you to create different images using different options.
Before you get started
You will need to get the mknbi (utility for making tagged kernel images useful for netbooting) package to create your Etherboot images. This tool will create a preconfigured kernel image from your original kernel. This contains the boot options as shown further down.
Code Listing 4.9: Installing mknbi
# emerge mknbi
Setting up Etherboot
In this section we will create a simple etherboot image. As the dhcp server gives out the clients root-path in the "option root-path" dhcp.conf, we do not have to include this here. More details can be found in the mknbi manual.
Code Listing 4.10: mknbi manual
# man mknbi
Making the boot images. This will create a ELF bootable image capable of passing dhcp and the rootpath to the kernel. Also forcing the kernel to browse the network for a dhcp server.
Code Listing 4.11: making netboot images
# mkelf-linux -ip=dhcp /diskless/bzImage > /diskless/vmlinuz
Note: For the arch specific images you have to type bzImage_arch and vmlinuz_arch.
Troubleshooting the network boot process
There are a few things you can do to debug the network boot process. Primarily you can use a tool called tcpdump. To install tcpdump type:
Code Listing 4.12: Installing tcpdump
# emerge tcpdump
Now you can listen to various network traffic and make sure your client/server interactions are functioning. If something isn't working there are a few things you might want to check. First make sure that the client/server is physically connected properly and that the networking cables are not damaged. If your client/server is not receiving requests on a particular port make sure that there is no firewall interference. To listen to interaction between two computers type:
Code Listing 4.13: Listening to client and server interaction via tcpdump
# tcpdump host client_ip and server_ip
You can also use tcpdump to listen on particular port such as the tftp port by typing:
Code Listing 4.14: Listening to the tftp server
# tcpdump port 69
A common error you might receive is: "PXE-E32: TFTP open time-out". This is probably due to firewall issues. If you are using TCPwrappers, you might want to check /etc/hosts.allow and etc/hosts.deny and make sure that they are configured properly. The client should be allowed to connect to the server.
5. Configuring the NFS server
About the NFS server
NFS stands for Network File System. The NFS server will be used to serve directories to the slave. This part can be somewhat personalized later, but right now all we want is a preliminary slave node to boot diskless.
About Portmapper
Various client/server services do not listen on a particular port, but instead rely on RPCs (Remote Procedure Calls). When the service is initialised it listens on a random port and then registers this port with the Portmapper utility. NFS relies on RPCs and thus requires Portmapper to be running before it is started.
Before you start
The NFS Server needs kernel level support so if you don't have this you should recompile your master's kernel. To double check your master's kernel configuration type:
Code Listing 5.1: Checking for NFS specific options
# grep NFS /usr/src/linux/.config_master
You should see output that looks something like this if your kernel has been properly configured:
Code Listing 5.2: Proper NFS specific options in the master's kernel configuration
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
# CONFIG_NETFILTER is not set
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V4 is not set
# CONFIG_NFSD_TCP is not set
Installing the NFS server
The NFS package that can be acquired through portage by typing:
Code Listing 5.3: Installing nfs-utils
# emerge nfs-utils
This package will emerge a portmapping utility, nfs server, and nfs client utilities and will automatically handle initialisation dependencies.
Configuring the NFS server
There are three major configuration files you will have to edit:
Code Listing 5.4: Nfs configuration files
/etc/exports
/diskless/192.168.1.21/etc/fstab
/etc/conf.d/nfs
The /etc/exports file specifies how, to who and what to export through NFS. The slave's fstab will be altered so that it can mount the NFS filesystems that the master is exporting.
A typical /etc/exports for the master should look something like this:
Code Listing 5.5: Sample master /etc/exports
# one line like this for each slave
/diskless/192.168.1.21 192.168.1.21(sync,rw,no_root_squash,no_all_squash)
# common to all slaves
/opt 192.168.1.0/24(sync,ro,no_root_squash,no_all_squash)
/usr 192.168.1.0/24(sync,ro,no_root_squash,no_all_squash)
/home 192.168.1.0/24(sync,rw,no_root_squash,no_all_squash)
# if you want to have a shared log
/var/log 192.168.1.21(sync,rw,no_root_squash,no_all_squash)
The first field indicates the directory to be exported and the next field indicates to who and how. This field can be divided in two parts: who should be allowed to mount that particular directory, and what the mounting client can do to the filesystem: ro for read only, rw for read/write; no_root_squash and no_all_squash are important for diskless clients that are writing to the disk, so that they don't get "squashed" when making I/O requests. The slave's fstab file, /diskless/192.168.1.21/etc/fstab, should look like this:
Code Listing 5.6: Sample slave fstab
# these entries are essential
master:/diskless/192.168.1.21 / nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
master:/opt /opt nfs sync,hard,intr,ro,nolock,rsize=8192,wsize=8192 0 0
master:/usr /usr nfs sync,hard,intr,ro,nolock,rsize=8192,wsize=8192 0 0
master:/home /home nfs sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
none /proc proc defaults 0 0
# useful but superfluous
master:/var/log /var/log nfs hard,intr,rw 0 0
(if you are setting up an openMosix cluster only)
none /mfs mfs dfsa=1 0 0
In this example, master is just the hostname of the master but it could easily be the IP of the master. The first field indicates the directory to be mounted and the second field indicates where. The third field describes the filesystem and should be NFS for any NFS mounted directory. The fourth field indicates various options that will be used in the mounting process (see mount(1) for info on mount options). Some people have had difficulties with soft mount points so we made them all hard, but you should look into various /etc/fstab options to make your cluster more efficient.
The last file you should edit is /etc/conf.d/nfs which describes a few options for nfs when it is initialised and looks like this:
Code Listing 5.7: Sample master /etc/conf.d/nfs
# Config file for /etc/init.d/nfs
# Number of servers to be started up by default
RPCNFSDCOUNT=8
# Options to pass to rpc.mountd
RPCMOUNTDOPTS=""
You should change RPCNFSDCOUNT to the number of diskless nodes on the network.
Starting the NFS server
You should start the nfs server with its init script located in /etc/init.d by typing:
Code Listing 5.8: Starting the master's nfs server
# /etc/init.d/nfs start
If you want to this script to start when the system boots simply type:
Code Listing 5.9: Adding the nfs server to the master's default run level
# rc-update add nfs default
6. Completing the slave filesystem
Copy the missing files
We will now make the slave's file system in sync with the master's and provide the necessary binaries while still preserving slave specific files.
Code Listing 6.1: Creating a slave filesystem
# rsync -avz /bin /diskless/192.168.1.21
# rsync -avz /sbin /diskless/192.168.1.21
# rsync -avz /lib /diskless/192.168.1.21
Note: The reason for rsync -avz instead of cp is to maintain symlinks and permissions
Initialisation scripts
The default scripts will try to run checkroot which does not make sense on your slave nodes. The hard way out is to manually edit the /diskless/192.168.1.21/sbin/rc script but this is cumbersome, dangerous and could break if you decided to sync your node file system again and forgot to leave this script alone. The trick is to have a /fastboot file when your system boots. This file tells checkroot not to run any file system check. But it will also erase the file when it has finished the initialisation process. That is why we need to create this file again at the end of the init process like this:
Code Listing 6.2: Preventing init scripts from running a file system check
(Create the /fastboot file for next reboot)
# touch /diskless/192.168.1.21/fastboot
(Create the /fastboot file on each boot)
# echo "touch /fastboot" >> /diskless/192.168.1.21/etc/conf.d/local.start
Since the unmounting of network filesystems should be done as late as possible, you have to edit your /etc/init.d/netmount and have it read:
Code Listing 6.3: Editing /etc/init.d/netmount
depend() {
before *
Note: Baselayout version 1.11.* and later does not need this.
If you do this on a live system, don't forget to run depscan.sh to fix the service dependencies. You can safely ignore any warning about the collision with /etc/init.d/checkroot, since you are disabling it via the fastboot file you set up in the previous paragraph.
You need as many init scripts under /diskless/192.168.1.21/etc/runlevels as you need services on your diskless nodes. It all depends on what you want your slaves to do.
Warning: Do not use the rc-update program to add or remove scripts from the slave runlevels when logged on your master. This would change your master runlevels. You need to create the links manually or log into your slave nodes using ssh or connect a screen and keyboard to your slave.
Code Listing 6.4: Typical slave runlevels
/diskless/192.168.1.21/etc/runlevels/:
total 16
drwxr-xr-x 2 root root 4096 2003-11-09 15:27 boot
drwxr-xr-x 2 root root 4096 2003-10-01 21:10 default
drwxr-xr-x 2 root root 4096 2003-03-13 19:05 nonetwork
drwxr-xr-x 2 root root 4096 2003-02-23 12:26 single
/diskless/192.168.1.21/etc/runlevels/boot:
total 0
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 bootmisc -> /etc/init.d/bootmisc
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 checkfs -> /etc/init.d/checkfs
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 clock -> /etc/init.d/clock
lrwxrwxrwx 1 root root 23 2003-10-18 17:28 consolefont -> /etc/init.d/consolefont
lrwxrwxrwx 1 root root 22 2003-10-18 17:28 domainname -> /etc/init.d/domainname
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 hostname -> /etc/init.d/hostname
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 keymaps -> /etc/init.d/keymaps
lrwxrwxrwx 1 root root 22 2003-10-18 17:28 localmount -> /etc/init.d/localmount
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 modules -> /etc/init.d/modules
lrwxrwxrwx 1 root root 18 2003-10-18 17:28 net.lo -> /etc/init.d/net.lo
lrwxrwxrwx 1 root root 20 2003-10-18 17:28 netmount -> /etc/init.d/netmount
lrwxrwxrwx 1 root root 21 2003-10-18 17:28 rmnologin -> /etc/init.d/rmnologin
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 urandom -> /etc/init.d/urandom
/diskless/192.168.1.21/etc/runlevels/default:
total 0
lrwxrwxrwx 1 root root 19 2003-10-18 17:28 distccd -> /etc/init.d/distccd
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 local -> /etc/init.d/local
lrwxrwxrwx 1 root root 16 2003-10-18 17:28 sshd -> /etc/init.d/sshd
lrwxrwxrwx 1 root root 21 2003-10-18 17:28 syslog-ng -> /etc/init.d/syslog-ng
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 vixie-cron -> /etc/init.d/vixie-cron
/diskless/192.168.1.21/etc/runlevels/nonetwork:
total 0
lrwxrwxrwx 1 root root 17 2003-10-18 17:28 local -> /etc/init.d/local
/diskless/192.168.1.21/etc/runlevels/single:
total 0
Now is a good time to boot your slave and cross your fingers. It works? Congratulations, you are now the proud owner of (a) diskless node(s) :)
本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u/11445/showart_97089.html |
|
免费注册
发表于 2006-04-09 00:00