Xen Networking

I've been working on evaluating Xen for all the reasons most companies look into virtualization. One of the most confusing parts of Xen to learn is the way it does networking. This is because the Xen team has devised a default configuration that is quite flexible, but also takes some time to get used to. I'm not even going to talk about that, because I think there's a better way.

This is a description of how to set up networking for Xen on CentOS 5.0 using 802.1q VLAN trunking with a bridge on dom0 for each VLAN. In my setup, eth0 is dedicated to dom0 tasks like live migration, iSCSI, and console access. eth1 is dedicated as a VLAN trunk. NO VLAN interfaces are configured with addresses in dom0, which should help reduce security implications.

Files:/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:19:D1:4D:CD:D1
IPADDR=192.168.1.25
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=00:19:D1:4D:CD:D2
ONBOOT=yes
# note, no address configured

/etc/sysconfig/network-scripts/ifcfg-eth1.4

DEVICE=eth1.4
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=br4
# again, no address configured

/etc/sysconfig/network-scripts/ifcfg-br4

DEVICE=br4
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
DELAY=0
STP=off

The files above only demonstrate the standard way to configure 802.1q VLAN's and bridges under modern Redhat-derivative distributions like RHEL, CentOS, and Fedora. The nice part is, we're almost done. Only a small change is necessary in Xen, so that it no longer brings up xenbr0. I don't want to pay for the overhead of networking through a bridge for my iSCSI, even if it is a very small cost.

Modify /etc/xen/xend-config.sxp and change "(network-script ...)" to "(network-script /bin/true)". Comment the "(vif-script ...)" line(s) out completely. You don't need them anymore.

Reboot dom0.

When the box comes back up, log in and look around at your networking configuration. Use "ifconfig" or "ip link show", "brctl", and "cat /proc/net/vlan/config".

Modify your VM configurations to point at the correct bridges. Something like the following should work fine.

    vif = [ 'mac=00:16:3e:01:fb:fe, bridge=br4' ]

xend will still create the point-to-point (vif) interfaces and connect them to the bridge for you. It doesn't need the helper scripts because the bridges are already all set up and ready to have interfaces bound to them.

This is currently a bit outside the beaten path for Xen netw0rking, so you get to keep all the pieces if it breaks things, has an affair with your refrigerator, or any other such sideeffects. If you have questions, leave comments on this post and I'll try to answer.