Xen Networking

I've been working on evaluating Xen for all the reasons most companies look into virtualization. One of the most confusing parts of Xen to learn is the way it does networking. This is because the Xen team has devised a default configuration that is quite flexible, but also takes some time to get used to. I'm not even going to talk about that, because I think there's a better way.

This is a description of how to set up networking for Xen on CentOS 5.0 using 802.1q VLAN trunking with a bridge on dom0 for each VLAN. In my setup, eth0 is dedicated to dom0 tasks like live migration, iSCSI, and console access. eth1 is dedicated as a VLAN trunk. NO VLAN interfaces are configured with addresses in dom0, which should help reduce security implications.

Files:/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:19:D1:4D:CD:D1
IPADDR=192.168.1.25
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=00:19:D1:4D:CD:D2
ONBOOT=yes
# note, no address configured

/etc/sysconfig/network-scripts/ifcfg-eth1.4

DEVICE=eth1.4
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=br4
# again, no address configured

/etc/sysconfig/network-scripts/ifcfg-br4

DEVICE=br4
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
DELAY=0
STP=off

The files above only demonstrate the standard way to configure 802.1q VLAN's and bridges under modern Redhat-derivative distributions like RHEL, CentOS, and Fedora. The nice part is, we're almost done. Only a small change is necessary in Xen, so that it no longer brings up xenbr0. I don't want to pay for the overhead of networking through a bridge for my iSCSI, even if it is a very small cost.

Modify /etc/xen/xend-config.sxp and change "(network-script ...)" to "(network-script /bin/true)". Comment the "(vif-script ...)" line(s) out completely. You don't need them anymore.

Reboot dom0.

When the box comes back up, log in and look around at your networking configuration. Use "ifconfig" or "ip link show", "brctl", and "cat /proc/net/vlan/config".

Modify your VM configurations to point at the correct bridges. Something like the following should work fine.

    vif = [ 'mac=00:16:3e:01:fb:fe, bridge=br4' ]

xend will still create the point-to-point (vif) interfaces and connect them to the bridge for you. It doesn't need the helper scripts because the bridges are already all set up and ready to have interfaces bound to them.

This is currently a bit outside the beaten path for Xen netw0rking, so you get to keep all the pieces if it breaks things, has an affair with your refrigerator, or any other such sideeffects. If you have questions, leave comments on this post and I'll try to answer.

Moving sucks.

Well, I'm all moved in to the apartment in San Diego. Almost, that is. My belongings are in the space, but as it goes, it's all still in boxes and it's a daunting task to go through and organize everything. Most importantly, though, I'm missing my wife and pets. In both senses of "missing."

Life the last month has been interesting. My Vehicross blew up. I'm in the process of figuring out what to do with it right now and will probably sell it. It did provide me an excuse to buy a new motorcycle, so I got a brand-new Suzuki V-Strom 650 (a.k.a. the "Wee-Strom"). I enjoyed riding my previous bike, a 1981 Kawasaki 440LTD, but this bike is an absolute dream as far as upgrades go! So, it is really helping fight off the depression that comes with losing a beloved vehicle.

My job is awesome and I'm learning a ton about how to work on a massive scale and how the MMO game industry works. Linux and Perl still shine as the best hammer any guy could have a toolbox full of. I'm blown away by how cool and accessible everybody is at work. I'm so used to the extremely conservative atmosphere, that I still have a hard time adjusting to the relaxed environment.

I'm spending some time reimplementing some tools similar to what I had at my old job because they really did make life better. One is a nice wrapper around CGI::Application and some of its plugins (like Plugin::TT and Plugin::AutoRunmode) to make writing one-off apps very fast and efficient. I looked at Catalyst & co. but they're just too darned much infrastructure for the kinds of apps I'm writing. I also couldn't get a handle on what the future is for Catalyst. It looks like Maypole is pretty much rolled into Catalyst (I could be wrong, so don't quote me). I didn't see anything else compelling out there as far as Perl goes. Django looked good, but I still have a hard time reaching for Python. It's not my usual language bigotry - it's common sense this time. I'm so busy already that taking time to learn another language isn't in the cards right now. I know these things have 20-minute tutorials, but I also have to consider the fact that all the other sysadmins I know, know Perl. Very few know Python or any other language for that matter.

In other Perl-related news, I'm looking for somebody to take over maintenance on Nagios::Object on CPAN. I really enjoyed getting it out there, but it's tough keeping up with the updates to Nagios these days since I don't have any installations that I work on. Just drop a mail to tobert@gmail.com and we'll talk about it. There is a queue of updates that I need to get out but haven't had the time to get them all wrapped up and tested so I can make a release. I guess I'll just have to throw it together and put it out there so people can beat on it...