Creating a private network with Xen virtual machin

spooky

This post is intended for people
who are trying to set up a private network with Xen virtual machines.
(For my notes on installation of Xen,
see part I
.)
As such, it assumes that you know what Xen is, what dom0 and domU are,
and how to configure domU and xend. Particularly, you should have read
the XenNetworking wiki page
.
The basic problem being solved is that I don't want to maintain static
IP addresses for each domU, nor do I want to maintain any static
host↔IP address mapping. Instead, I want to give domU machines an IP
address over DHCP, and set their hostname in DNS automatically, at
least from the point of view of the machine running Xen. Of course, if
other machines on the physical LAN use dom0's DNS server, they, too,
will be able to see the domU machines.
This is similar in concept to
Xen3 and a Virtual Network
on the OpenSUSE wiki, but different in implementation.
            
                        Networking: Background
Whenever a domU is started, it receives at least one virtual Ethernet
card, eth0. That card is hard-wired to a virtual interface, vifU.0, in
dom0. The final network architecture depends on what the vif-script
does with that vifU.0 interface.
The network-script and vif-script options in xend-config.sxp simply name scripts in /etc/xen/scripts which set up or tear down networking for each virtual machine. Don't forget to do sudo /etc/init.d/xend restart
after changing xend-config.sxp. For more information on the
configurations which come with Xen (bridged/routed/NAT), see their
respective scripts, or the XenNetworking article linked to at the top
of this post.
In bridged mode (the network-bridge and vif-bridge scripts), the
physical ethernet card of dom0, and all vifU.0 endpoints, are connected
to a virtual bridge, xenbr0. The bridge works like a real bridge,
learning which MAC addresses are at which ports, and sending traffic
that arrives from any card to the appropriate destination, or to all
ports if the destination is unknown. This is how any domU started can
appear as peers on the same network that has dom0. This also means that
domU should be given a sane IP address for the physical network, or use
DHCP to get an address from the physical network's DHCP server. In
other words, bridged mode does not really create a "virtual network" at
all; it simply connects all Xen domains to the real network.
In routed mode (network-route and vif-route, naturally), each vifU.N
interface must have a static IP address, and the vifU.N card is given a
static route in dom0 for that single address. The bridge interface,
xenbr0, is not used in this configuration, and will only be visible in
dom0 if you've used bridge-mode since rebooting.
Routed mode is fundamentally incompatible with DHCP. If a domU doesn't
have a static address, the vif-route script doesn't assign any route to
the new domU. Since the DHCP packet is addressed to the new IP address,
it follows the default route onto the physical LAN, instead of going to
the domU which actually needs it. On the other hand, bridged mode
operates below the IP level, which makes it difficult to block DHCP
requests from traveling out to the physical branch of the LAN.
netfilter actually does seem powerful enough to do it, but I couldn't
figure out how.
DHCP for domU
To make DHCP work, add a line to the domU's configuration file (e.g. /etc/xen/xen1.cfg):
kubikal$ sudo -e /etc/xen/xen1.cfg
dhcp = "dhcp"
vif  = [ 'mac=aa:cc:00:00:00:01' ]
kubikal$ sudo xm shutdown xen1
kubikal$ sudo xm create /etc/xen/xen1.cfg
Now xen1 will be able to get an address from any DHCP server that it
can see, either the existing one for the physical LAN in bridged mode,
or the one we set up in routed+DHCP mode.
Customized Routing
Unable to do what I wanted with the default Xen scripts, I decided to customize my own. It turns out that the network-route script was suitable for my purposes, but vif-route was not. I created a new script, called vif-private-subnet, which simply adds the vif to a bridge, and brings it up. Make sure this script is executable after placing it in /etc/xen/scripts. Here is the code for the script:
#!/bin/bash
#============================================================================
# /etc/xen/vif-private-subnet
#
# Script for configuring a vif on a private subnet, visible only to dom0.
# The hotplugging system will call this script if it is specified either in
# the device configuration given to Xend, or the default Xend configuration
# in /etc/xen/xend-config.sxp.  If the script is specified in neither of those
# places, then vif-bridge is the default.
#
# Usage:
# vif-route-bridge (add|remove|online|offline)
#
# Environment vars:
# vif         vif interface name (required).
# bridge      bridge interface name (default br0).
# XENBUS_PATH path to this device's details in the XenStore (required).
#============================================================================
dir=$(dirname "$0")
. "$dir/vif-common.sh"
bridge="${bridge:-br0}"
case "$command" in
    online)
        brcmd='addif'
        cmdprefix=''
        ;;
    offline)
        do_without_error ifdown ${vif}
        brcmd='delif'
        cmdprefix='do_without_error'
        ;;
esac
${cmdprefix} brctl ${brcmd} ${bridge} ${vif}
r2=$?
${cmdprefix} ifconfig ${vif} up
r1=$?
log debug "vif-route-bridge $command for $vif on $bridge: ifconfig=>$r1 brctl=>$r2"
if [ "$command" == "online" ]
then
  success
fi
To keep the Xen side as simple as possible, I configured the bridge
outside of the Xen scripts. (Unlike the bridge scripts, which take care
of creating the bridge themselves.) This is the relevant part of /etc/network/interfaces:
# Create the backend Xen bridge
auto br0
iface br0 inet static
        # bridge_ports tells ifupdown that this is a bridge
        bridge_ports none
        # assign address for dom0 to use on the virtual net
        address 192.168.0.254
        netmask 255.255.255.0
        # Fix checksum errors (DHCP in domU fails without this)
        post-up ethtool -K br0 tx off
# Statically configure eth0 so the DSL modem can have a static route
# through eth0 to the 192.168.0.x network on br0. This lets machines
# on the LAN talk to the domUs by IP. The gateway here is the DSL
# modem.
auto eth0
iface eth0 inet static
        address 192.168.11.100
        netmask 255.255.255.0
        gateway 192.168.11.254
I'm not clear on the reasons yet, but giving the bridge an IP
automatically lets dom0 use it as a virtual network card. No extra
interface needs to be created and attached to the bridge, and no route
needs to be created manually. Do note, however, that the gateway given
for eth0 needs to have a static route in order for the domU machines to
be visible from the LAN. That route should direct traffic for 192.168.0.0/24 to 192.168.11.100, the public IP of kubikal.
Serving DHCP
I installed dnsmasq for serving
DHCP to the domU machines. dnsmasq also serves DNS, and can link DHCP
addresses to hostnames. I decided to have '.xen' as the domain for the
virtual network, and the IP range was set in the bridge configuration
above. For symmetry with the DSL modem, dom0 also sits at the last
address on the virtual network, 192.168.0.254.
This is the minimal set of options needed in /etc/dnsmasq.conf:
domain-needed
resolv-file=/etc/resolv.upstream.conf
local=/xen/
user=dnsmasq
group=daemon
domain=xen
dhcp-range=192.168.0.1,192.168.0.253,3d
dhcp-leasefile=/var/lib/dnsmasq/leases
I moved the initial /etc/resolv.conf to the mentioned /etc/resolv.upstream.conf, and then set /etc/resolv.conf
to direct all traffic to 127.0.0.1, where dnsmasq listens. All the DHCP
clients, by default, are given the address of dom0 as a DNS server, and
thus also resolve names using dnsmasq.
The final piece of the puzzle is to make the domU send a hostname to
dnsmasq. Without that hostname, dnsmasq doesn't know what the domU
should be called, and therefore can't add it to DNS. On Debian, the hostname line in /etc/network/interfaces only applies to the DHCP clients which are not installed by default. dhclient3 does its work using /etc/dhcp3/dhclient.conf instead. Once send host-name
is uncommented and set to "xen1.xen", then the DNS query returns the
correct IP address. (And I begin to think of a better name than
xen1....)
Is that all?
It's been a long, complicated process, with a number of false starts. Hopefully, I didn't miss anything. Let me know if I did.

本文来自ChinaUnix博客，如果查看原文请点：http://blog.chinaunix.net/u/31253/showart_1134255.html