tobert.org

Over-breeding & Culling EC2 Instances for IO Performance

2011-09-20T23:06:00.000-07:00

I've heard other people talk about this on Twitter or at conferences, but as far as I can remember, nobody has described the nuts and bolts of finding tolerable-performance ephemeral disks in EC2.

I recently spun up a 12-node Cassandra cluster in EC2 and, since it's a database, I decided to do some basic tire-kicking and learned a few things along the way.

Rule: always zero your ephemerals if you care about performance.

Why: Amazon is likely using sparse files to back ephemerals (and probably EBS, I have no experience there). This makes perfect sense, because:

you get free thin provisioning, so unused disk doesn't go to waste
Xen supports it well
it's easy to manage lots & lots of them
it's trivial to export over all common network block protocols (e.g. AoE, iSCSI)

Because there is an extra step of allocating a backing block for a sparse file for every block in your VM, performance will be all over the map while zeroing the disks.

Script #1:

I usually launch my zeroing script with cl-run.pl --list burnin -s zero-drives.sh. The "burnin" list is just all the ec2 hostnames, one per line, in ~/.dsh/machines.burnin.

Culling round 1: Look at the raw throughput of all of the nodes and cull anything that looks abnormally low. For example, when building the aforementioned cluster, I kept getting really obviously bad instances in one of the us-east-1 AZ's.This is what I saw when using my cluster netstat tool for a batch of m1.xlarge's in us-east-1c.

I immediately culled off everything doing under 10k IOPS for more than a minute. If you examine the per-disk stats with iostat -x 2, you'll usually see one disk with insanely high (>1000ms) latency all the time. There are certainly false-negatives at this phase, but I don't really care since instances are cheap and time is not. I ended up starting around 30 instances in that one troublesome AZ to find 3 with sustainable IOPS in the most trivial of tests (dd).

When I think I have enough obviously tolerable nodes for a race, I kick off another zero round. Once the load levels out a little, I take a snapshot I like of the cl-netstat.pl output and process it in a hacky way to sort by IOPS and add which EC2 zone the instance is in and its instance ID so I can kill the losers without digging around. Here's an example from a round of testing I did for a recent MySQL cluster deployment:

I picked the top few instances from each AZ and terminated the rest. Job done.

This is a pretty crude process in many ways. It's very manual, it requires a lot of human judgement, and most importantly, dd if=/dev/zero not a good measure of real-world performance. This process is just barely good enough to cull the worst offenders in EC2, which seem to be quite common in my recent experience.

In the future, I will likely automate most of this burn-in process and add some real-world I/O generation, probably using real data.

New Experiment Language Choices

2011-07-21T00:39:00.000-07:00

(edit: I'm still tinkering with the ideas and have started learning Go, but will probably end up doing this project in either python or bash)

My evolving side project is a configuration management suite that:

1.) uses as much of git as possible and sensible
2.) lightweight
3.) simple things must be very fast turnaround to/from thought<->production
4.) all configs (and binaries if you like) versioned bit-for-bit for all time

I'll write more about the overall system design later; I've written a bit of an intro and will post it when I have code to present (here's what I have in mind for fs permissions: gist) I'm implementing some tools now that I've got the branching & config strategy worked out. I could whip out a lot of it in Perl pretty handily, but I suspect it'll be a one-man show forever if I go that route. At work, almost everything is done in Ruby, which I still don't really like much. Plus two of the leading CFM tools are already in Ruby, so I feel like it's time for something different. So, here's what I'm considering and why:

1) Go
  +) generates regular binaries, no VM to maintain on nodes
  +) good C compatibility, already has libgit2 bindings
  +) garbage collected, first-class strings
  +) (yes, +) DNA from C, Erlang, Plan9, and Inferno/Limbo
     e.g.) lightweight processes, channels, goroutines
  -) I'd have to learn from scratch
  -) AFAIK nobody on my team knows/uses it (obscure)
2) C
  +) compiled, works literally everywhere, knows your mom intimately
  +) good match with git, openssl/gnutls, and libgit2
  +) igraph looks neat for smart path & network
  -) manual memory, thread, and process management :(
  -) I'm rusty as the Titanic
3) Perl >=5.8
  +) installed everywhere
  +) I'm good/fast at it, prototype in a couple weeks
  +) many git utilities are written in perl, good match
  -) declining popularity, fractured community
  -) modern libs require too much baggage (e.g. Moose)
  -) XS is the devil, no usable libgit2 bindings
4) Python
  +) installed everywhere I care about
  +) thriving community
  +) I have some experience (though still much slower than perl)
  +) libgit2 bindings exist
  +) some of the git tools are already python
  -) difficult to bundle with zero dependencies outside the git branch
  -) v2.5 -> v3.x transition is in progress, which to choose? (prob 2.7)
5) Java
  +) just kidding!
6) Javascript/NodeJS
  -) I'm not kidding
  +) garbage collected, closures, functional style available, naturally async
  +) growing popularity, modules I'd need are fairly mature (relative to node.js)
  -) rapidly changing ecosystem/interfaces

To be honest, I'm really leaning towards Go right now but it'd add a month or so before I have something useful.

Comments, suggestions? Don't hold back, you know I wouldn't ;)

My Christmas Wish

2010-12-23T13:42:00.001-08:00

I wrote this email to my (non-technical) family with the subject, "My Christmas Wish."

Folks,

In light of the most recent "Zero Day" security vulnerability in Internet Explorer, I am writing this to ask all of you who still use it to consider switching browsers. IE is by far the most ubiquitous source of junk on Windows computers outside of unsafe downloads. Other browsers do much more to protect you.

So-called "Zero Day" exploits occur on a regular basis and leave you vulnerable to attack for weeks at a time. For example, the Chinese attacks on Google relied on vulnerabilities in Internet Explorer to steal users' account information. Google now has a policy tightly restricting the use of Windows and IE to only employees who need it for testing purposes.

You don't have to do much to pick up malware via websites. There are myriad ways for attackers to deliver malicious code that you will never see, and often your virus scanners cannot detect either.

http://isc.sans.edu/diary.html?storyid=10132
http://www.microsoft.com/technet/security/advisory/2488013.mspx

It's incredibly easy to avoid the whole mess. Install another, faster and more secure browser. Delete the IE icon from your desktop and quickstart bars, especially if others use your computer. Set the other browser of your choice as the default. Here's a list of modern, secure browsers, in my personal order of preference.

http://www.google.com/chrome
http://www.mozilla.com/en-US/firefox/
http://www.opera.com/
http://www.apple.com/safari/

If you can't see yourself using another browser, at least use Chrome for your sensitive data and IE for everything else. That way at least your bank accounts and health records will be slightly harder for attackers to steal.

Lastly, if you haven't done it in a while, check for updates on the other programs you use regularly, especially instant messaging like Skype, AOL, MSN, and Yahoo!. Many programs have an automatic update feature under the "Help" menu that will automatically do all the dirty work.

-Al

Ride Report 2010-09-05

2010-09-05T21:49:00.000-07:00

I recently started commuting to work on my bicycle. I recently hit 230lbs., which I've always kept in my head as my maximum weight I'll let myself get to. The weight problem leaves me with a conundrum. I love eating and I hate exercising. Fortunately, I've always enjoyed bicycling, so the answer was to start converting my 2009 Marin Alpine Trail 29er to a commuter and combine exercise with my commute. Riding a bicycle to work only takes me an extra 15 minutes if traffic is bad, giving me 1.5hrs of exercise every day with only a half hour added to my commute. That's not mentioning the reduced road rage, gas, mileage on my car, and starting my day energized.

I've ridden into work about 10 times now. It's about 10 miles each way depending on my route, which I think is in the sweet spot of, "far enough to make it worth getting smelly and still under an hour commute."

Latest Work Route

Since tomorrow is a holiday and I wanted to check out some panniers at REI, I decided to ride so I could keep my body guessing. Route

The ride went fine and I felt pretty good the whole time. It was really nice to be able to wear my camelback (Costco version) and stay hydrated without dinking with a water bottle. I picked up some new grips and a pack of shot bloks. I looked at the Ortlieb panniers which look very nice, but are way too expensive for my current level of commitment. The laptop bag alone was something like $150. I might go after one later after I've totally adapted to bicycle commuting, but I'm not spending $300 on bags until I'm sure I'll use them for a 1000+ miles.

Defeated, I rode home and browsed around the web looking at what people are doing with custom mods. There's some really creative stuff out there, most interestingly, using plastic ammo cans. I drove over to a Stevens Creek Surplus, but all they had were the classic steel cans (though they had an awesome camping section). On the way home I thought I'd stop at Home Depot and see if they had anything interesting. They did. I bought two "Husky 16 in. Hang-Up Tool Bag", some velcro straps, and a two packs of flat bungee cords.

Tomorrow, I'll figure out how to mount the bags to my satisfaction, or return them and go for the plain aluminum briefcase (can't find HD link).

Now, I'm off to google some ideas for pannier hooks.

Blocking SSH Attacks

2010-08-12T12:12:00.000-07:00

I run this on almost any server that isn't behind another firewall blocking $SSH_PORT. I got this from somewhere else, but it was a long time ago and I don't remember the source anymore.


SSH_PORT=22  # I think ssh on !22 is silly, but to each their own
TRUSTED_SUBNET_OR_HOST=mybox.homeip.com # or a subnet you trust

iptables -A ssh_drop -j DROP -m comment --comment "SSH attack drop"
iptables -A INPUT  -p tcp -m tcp -s $TRUSTED_SUBNET_OR_HOST --dport $SSH_PORT  -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m state --state NEW --dport $SSH_PORT -m recent --name sshattack --set
iptables -A INPUT -p tcp --dport $SSH_PORT -m state --state NEW -m recent --name sshattack --rcheck --seconds 60 --hitcount 3 -j LOG --log-prefix 'SSH Attack DROP: '
iptables -A INPUT -p tcp --dport $SSH_PORT -m state --state NEW -m recent --name sshattack --rcheck --seconds 60 --hitcount 3 -j ssh_drop

Basically, this counts connection attempts on port 22 and starts dropping attempts after the third one (obviously adjust --hitcount to taste). It times out after 60 seconds, so I don't accidentally lock myself out. I see a few attacks a day start then go away after the drop. I've never seen one come back after the window reopens (unless it was myself trying to "attack"). It's also 0 maintenance unlike the userspace solutions. A lot of people like tcp wrappers, but that still lets attackers hammer the port. Using both works nicely.

This and DenyHosts don't help with distributed attacks, which is why I'm moving towards key-only logins and maybe picking up some YubiKeys. (yubico.com)

Virtualization Madness

2008-04-06T11:50:00.000-07:00

I now have all of my hardware for the virtualization project I've been working on and have been doing final testing and setup configuration lately. I've had the awesome opportunity to really torture the VM setup on a test cluster in the lab. I started out with testing storage solutions on a 10x10 (physical x virtual) cluster of Dell 1950's with 8GB of RAM and two Core 2 Duo's. I still have the demo NetApp 3070 that proved out NFS storage for VM images, and now I have all of the upgrades in the 1950's to push them to 32GB of RAM and hardware RAID controllers. I've learned quite a bit in the process of all of this testing and thought I'd share some tips here.

Many of the VM's I'll be running in production will come from a P2V migration, but because my predecessors were smart enough to concentrate all of our custom content under a single mountpoint, most of the systems can be migrated to new OS images (and thus upgrade ancient OS's in the process).

Tip #1: when building servers (physical or virtual), ignore the FHS for your local content. The common place these days is /srv, so put anything that does not come in a distribution package here. Backups are as easy as "tar -czvf /someplace/`hostname -s`-srv.tar.gz /srv". Migrations and cluster scaleouts similarly easy: "rsync -ave ssh /srv/./ username@newbox:/srv". When a package really wants you to conform to FHS, work around it with symlinks.

Edit: it was pointed out to me that FHS actually recommends /srv. To be honest, I haven't looked at it in years. In any case, my point remains valid, since most distros interpret FHS in their own way and still put things like web content and database files under /var.

Ok, so that wasn't so much about virtualization. A somewhat little known fact is that almost all of the virtualization players out there, be it Xen, KVM, VMWare, or even Microsoft, is that the VM's themselves are not actually that hard to migrate between them. You only have to figure it out once then, especially for Linux VM's, you can script it and do them in bulk. Probably the two most valuable tools for this are kpartx and qemu-img. Xen doesn't seem to install qemu-img with its Qemu stuff, but it's well-worth keeping around on your dom0's.

Tip #2: learn to use kpartx and qemu-img, even if you're using LVM or individual LUN's for your VM's. qemu-img can read and write raw, vmdisk, qcow2, and a few other formats and is pretty deft at enabling sparse file support, which is pretty nifty. For instance, if you download a VMWare appliance and want to run it under Xen, it's trivial to convert to a raw image with "qemu-img convert vmappliance.img -O raw vmappliance-raw.img". kpartx is nice because it will map out partitions within an image or LVM volume using device mapper. So once that image is created, do "kpartx -va vmappliance-raw.img" then you can mount the partitions without messing around with weird offset options to losetup.

One of the problems I've run into quite a lot over the last couple years' of playing with Xen & co. is that most initramfs scripts output far too fragile and stupid environments. With the availability of busybox and gobs of RAM these days, there is absolutely no reason I should have to screw around for hours rebooting a box because these filesystems are not smart enough to drop into a debug shell when things go wrong. I have published a simplistic script that I occasionally use to build initrd's at http://tobert.org/unix/index.html. But often, for support reasons, it's not practical to run a custom generation script. With the 2.6 Linux kernel, it's actually way easier to edit these buggers than it was back in the day, since now they're simply compressed cpio archives.

Tip #3: learn how to hack initrd's and get those years of your life back. Here's how to tear it apart:

mkdir /tmp/initrd
cd /tmp/initrd
gunzip -c /boot/initrd-`uname -r`.img |cpio -ivdm

The first thing to look at is the "init" script. For instance, when CLVM locking is stopping you from getting to single-user, simply crack that file open and comment out all the LVM initialization code. It's mostly a simple shell script. Another trick is to copy busybox into the bin directory, symlink lash, then add a "/bin/lash -i" to the init script right before root gets mounted. To put everything together again, you have to use the "newc" cpio format, so the command is (from the top of the initrd):

find . |cpio -oH newc |gzip -c > /boot/initrd-`uname -r`.img

To save yourself a lot of frustration, I highly recommend playing around with initramfs hacking in VM's first, since the hack/reboot/fail/reboot/hack/reboot cycle is so much faster.

Xen is pretty neat and it's nice how it's integrated with EL5 so I can just use yum to keep up to date. While I'm deploying Xen for my production stuff in the coming weeks, I'm watching closely for KVM to reach a level of maturity where I can start migrating over to it. I expect this to happen this year, but I won't go anywhere near it for production until it starts surviving my torture tests (another post, another day). For some more eloquent writing about why KVM can be better, check out Ulrich Drepper's LiveJournal, specifically here and here. So, what can you do to keep your VM's easy to migrate when something better comes along? Tip #1 takes you a long ways, since even if you have to reinstall the OS, it's a pretty trivial operation (especially if you use Cobbler).

Tip #4: don't tie your VM's too tightly to one solution. Obviously, the first step is to use libvirt rather than XenAPI. Once I figured out all of the bits & pieces, it only took about an hour - mostly waiting for the damned computers - to get all my test VM's converted from Xen to KVM paravirt. kpartx was invaluable since it let me mount the VM filesystems from the host. All of my VM's are on NetApp NFS, so a simple shell loop made quick work of mounting all 100 filesystems in my test cluster.

cd /net/vm-disks
for vmdisk in *.img
do
 mkdir -p /mnt/$vmdisk
 # run kpartx and grab partition #1 all at once
 DEVICE=`kpartx -v -a /net/vm-disks/$vmdisk |head -n 1 \
          |awk '{print $3}'`
 mount /dev/mapper/$DEV /mnt/$vmdisk
done

Once they're all mounted, it's pretty easy to loop through all of them and make a change, such as copy in a new /etc/modprobe.conf or an updated initramfs with virtio network/block drivers. I'm especially excited about KVM virtio-net with NFS root, since virtio-net is shaping up to be quite a bit faster than xennet.

# install a normal kernel
cd /mnt
for vmdisk in *.img
do
 chroot /mnt/$vmdisk yum -y install kernel
done
# and so on ...

You might even get away with some of these tricks on Windows VM's using ntfstools and ntfs-3g.

Tip #5: when searching for best practices and tuning information, there is a lot of excellent documentation available for us Xen and KVM users in the form of VMWare documentation. For example, I've had really good luck with reading NetApp's docs for NetApp + VMWare (the block alignment and Oracle RAC on NFS docs are particularly good). When vendors say "we really don't do much with Xen," I ask them for VMWare whitepapers instead. Most of the concepts are the same regardless of the hypervisor, so learn both sets of terminology and make the best of all the great documentation out there.

As always, remember to make backups ...

Xen Networking

2007-09-07T08:18:00.001-07:00

I've been working on evaluating Xen for all the reasons most companies look into virtualization. One of the most confusing parts of Xen to learn is the way it does networking. This is because the Xen team has devised a default configuration that is quite flexible, but also takes some time to get used to. I'm not even going to talk about that, because I think there's a better way.

This is a description of how to set up networking for Xen on CentOS 5.0 using 802.1q VLAN trunking with a bridge on dom0 for each VLAN. In my setup, eth0 is dedicated to dom0 tasks like live migration, iSCSI, and console access. eth1 is dedicated as a VLAN trunk. NO VLAN interfaces are configured with addresses in dom0, which should help reduce security implications.

Files:/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:19:D1:4D:CD:D1
IPADDR=192.168.1.25
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
HWADDR=00:19:D1:4D:CD:D2
ONBOOT=yes
# note, no address configured

/etc/sysconfig/network-scripts/ifcfg-eth1.4

DEVICE=eth1.4
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=br4
# again, no address configured

/etc/sysconfig/network-scripts/ifcfg-br4

DEVICE=br4
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
DELAY=0
STP=off

The files above only demonstrate the standard way to configure 802.1q VLAN's and bridges under modern Redhat-derivative distributions like RHEL, CentOS, and Fedora. The nice part is, we're almost done. Only a small change is necessary in Xen, so that it no longer brings up xenbr0. I don't want to pay for the overhead of networking through a bridge for my iSCSI, even if it is a very small cost.

Modify /etc/xen/xend-config.sxp and change "(network-script ...)" to "(network-script /bin/true)". Comment the "(vif-script ...)" line(s) out completely. You don't need them anymore.

Reboot dom0.

When the box comes back up, log in and look around at your networking configuration. Use "ifconfig" or "ip link show", "brctl", and "cat /proc/net/vlan/config".

Modify your VM configurations to point at the correct bridges. Something like the following should work fine.

    vif = [ 'mac=00:16:3e:01:fb:fe, bridge=br4' ]

xend will still create the point-to-point (vif) interfaces and connect them to the bridge for you. It doesn't need the helper scripts because the bridges are already all set up and ready to have interfaces bound to them.

This is currently a bit outside the beaten path for Xen netw0rking, so you get to keep all the pieces if it breaks things, has an affair with your refrigerator, or any other such sideeffects. If you have questions, leave comments on this post and I'll try to answer.

Moving sucks.

2007-05-19T20:30:00.000-07:00

Well, I'm all moved in to the apartment in San Diego. Almost, that is. My belongings are in the space, but as it goes, it's all still in boxes and it's a daunting task to go through and organize everything. Most importantly, though, I'm missing my wife and pets. In both senses of "missing."

Life the last month has been interesting. My Vehicross blew up. I'm in the process of figuring out what to do with it right now and will probably sell it. It did provide me an excuse to buy a new motorcycle, so I got a brand-new Suzuki V-Strom 650 (a.k.a. the "Wee-Strom"). I enjoyed riding my previous bike, a 1981 Kawasaki 440LTD, but this bike is an absolute dream as far as upgrades go! So, it is really helping fight off the depression that comes with losing a beloved vehicle.

My job is awesome and I'm learning a ton about how to work on a massive scale and how the MMO game industry works. Linux and Perl still shine as the best hammer any guy could have a toolbox full of. I'm blown away by how cool and accessible everybody is at work. I'm so used to the extremely conservative atmosphere, that I still have a hard time adjusting to the relaxed environment.

I'm spending some time reimplementing some tools similar to what I had at my old job because they really did make life better. One is a nice wrapper around CGI::Application and some of its plugins (like Plugin::TT and Plugin::AutoRunmode) to make writing one-off apps very fast and efficient. I looked at Catalyst & co. but they're just too darned much infrastructure for the kinds of apps I'm writing. I also couldn't get a handle on what the future is for Catalyst. It looks like Maypole is pretty much rolled into Catalyst (I could be wrong, so don't quote me). I didn't see anything else compelling out there as far as Perl goes. Django looked good, but I still have a hard time reaching for Python. It's not my usual language bigotry - it's common sense this time. I'm so busy already that taking time to learn another language isn't in the cards right now. I know these things have 20-minute tutorials, but I also have to consider the fact that all the other sysadmins I know, know Perl. Very few know Python or any other language for that matter.

In other Perl-related news, I'm looking for somebody to take over maintenance on Nagios::Object on CPAN. I really enjoyed getting it out there, but it's tough keeping up with the updates to Nagios these days since I don't have any installations that I work on. Just drop a mail to tobert@gmail.com and we'll talk about it. There is a queue of updates that I need to get out but haven't had the time to get them all wrapped up and tested so I can make a release. I guess I'll just have to throw it together and put it out there so people can beat on it...

New Job, New City

2007-04-04T07:18:00.000-07:00

I've been putting off shutting down tobert.org because I've been really busy with another project: finding a new job. I applied and interviewed at four well-known technical companies on the west coast and accepted an offer from Sony Online Entertainment on April 2, 2007.

It was a weird experience for me, having mostly applied for standard corporate jobs in West Michigan in the past. Actually, I've never had multiple offers before which made the decision process difficult. Any one of these companies would be awesome to work for. All of the people were extremely smart and passionate. I spent the entire weekend driving my wife, family, and my friends crazy talking incessantly about what to do.

What it came down to was my gut reaction towards the companies and the cities they were in. A lot of people might think I'm full of it when I say it, but money was not the object. I didn't take the highest paying job even after rolling in cost of living differences.

I ended up asking myself some "soft" questions:
- abstract the job from the company and its reputation; do I still like it?
- would I choose the city regardless of job and income? Would Krissy (my wife)?
- what does the professional and community theatre community look like for Krissy?
- do I like the physical environment (desk/workstation/lighting/dress code/etc.)?
- do I feel comfortable with the political environment?
- is there opportunity to make things better? Is it cleanup or creation?
- what career options are available in the next 5 years?

Anyways, I'm not going to mention the names of the other three companies. I don't want anybody to think they're not awesome companies to work for, because that would be untrue. I would work at any one of them and it has been both a burden and an honor to choose between them.

In case you're somebody looking for tips on interviewing, here's the most important and non-obvious-to-me lesson I learned. Think really hard about whether or not to reveal the names and offers from other companies to your recruiters. It may mean better offers, it might not. It can also cause some other side effects that I found less than pleasant, so in the future, I'm going to keep my big mouth shut. Somebody warned me and they were right, so if you ever read this, you're welcome to your "I told you so."

Lastly, I have to say goodbye to six years at Priority Health. They accepted my resignation with dignity and warm wishes, which I truly appreciate. I will miss my friends there and wish them all the best.

First Post!

2007-02-26T16:57:00.000-08:00

Hi, my name is Al Tobey. For now, go to http://www.tobert.org to learn the grossly-out-of-date details.

I'm investigating whether or not I want to move http://www.tobert.org to Blogger. I'm doing it to save time/money and to take advantage of integration between the various Google tools, where it exists. I've also often considered posting more often, which I avoid because my current setup is still a PITA.

The main snag I'm working on right now is how to post code examples. So far, it looks like nobody is really doing it well. Most of the "solutions" I've found involve stuff like sed 's/ /\ /g', which isn't very elegant when HTML has tags specifically for this kind of thing.

Previously, I defaulted to using the <pre> tag for code examples. So far, I've tried <blockquote> and <p style="white-space: pre;"> (which I learned about at http://www.htmldog.com/ptg/archives/000077.php)

Anyways, I'm going to keep editing this post adding copies of the below code until it works.

Using <pre>:


#!/bin/ksh

for file in *.html
do
    # this should be indented by 4 spaces
    sed 's/ /\&nbsp;/g' < "$file" > "$file.new"
    mv "$file.new" "$file"
done

Problem solved:: It appears that <pre> is ignored in the preview you get in the edit posts view, but not on the published page.

64-bit perl on HP-UX (PARISC or Itanium)

2005-10-12T19:30:00.000-07:00

Here's something that took me a few hours to get dialed in. This is all set for linking DBD::Oracle and DBD::Sybase to their respective 64-bit client libraries without hassle. I'm assuming the HP ANSI C compiler is installed. I never even tried with gcc due to some custom extensions that require the HP compiler.


CC=/opt/ansic/bin/cc \
CXX=/opt/aCC/bin/aCC \
CFLAGS="+DD64 +Z -I/opt/perl-5.8/local/include" \
CXXFLAGS=$CFLAGS \
LDFLAGS="-L/opt/perl-5.8/local/lib -L/usr/lib/hpux64" \
PATH=/opt/ansic/bin:/usr/ccs/bin:/opt/perl-5.8/local/bin:/opt/perl-5.8/bin:/usr/bin:/usr/local/bin:/bin:/sbin \
PERL=/opt/perl-5.8/bin/perl \ 
MAKE=/usr/local/bin/gmake \
./Configure -der \
            -A prepend:libswanted='cl pthread ' \
            -Dprefix=/opt/perl-5.8 \
            -Dinstallprefix=/opt/perl-5.8 \
            -Duselargefiles \
            -Dusethreads \
            -Duse64bitall \
            -Dyacc='/usr/local/bin/bison -y' \
            -Uusemallocwrap \
            -Dcf_by="Al Tobey" \
            -Dperladmin='tobert@gmail.com' \
            -Dcf_email='tobert@gmail.com'

make
make test
make install

(cd /usr/include; /opt/perl-5.8/bin/h2ph -r -l .)

2005-09-20T16:00:00.000-07:00

pdksh 5.2.14 on HP-UX 11.23 Itanium: If you grab the source for pdksh and try to compile it on Itanium HP-UX, it will tend to dump core with SIGBUS. SIGBUS is a signal sent to processes when they access unaligned memory, usually in C unions or structs. pdksh has a union in it that ends up unaligned on Itanium, which is very particular about such things. HP-UX is also fussy.

Go into the source tree and open up "alloc.c" in your favorite editor. Search for "union Cell", which is the union declaration that is causing the trouble. Change the last member "double djunk; /* alignement */" to "long double djunk; /* alignemnt */." On Itanic, long double is 128 bits and double is 64 bits. Pushing the union out to 128 bits makes it align right all the time, even though it does eat up a bit more memory now.

I also have a patch at work that logs all commands run by root to syslog with the username of the owner of the terminal device. It's a fancy way to avoid typing "sudo" in front of everything while still logging every command for auditing purposes. Unfortunately, I probably can't post that patch.

root-tail on nautilus/kde/xfdesktop

2004-06-30T18:00:00.000-07:00

I hacked root-tail to work with Nautilus, KDE, and hopefully XFCE4's xfdesktop. I took the toon_root.c file from xpenguins and :r'd it into root-tail.c. A few modifications later and I had a working root-tail for nautilus.

Download the tarball: root-tail-xpenguins.tar.gz or just root-tail.c. Here it is in HTML: root-tail.c.html.

Screenshot (40k)

2004-06-30T12:00:00.000-07:00

I wrote a couple SNMP plugins for Nagios to check disk space and running processes. They are now in the contrib directory in the Nagios Plugins package.
They both should work on Windows 2000 (and up) and Novell. HP-UX, unfortunately, does not ship with the Host-Resource MIB by default, so I'm using the Net-SNMP daemon at work. It's a little flakey, but gets the job done. SNMP Research provides an extended SNMP daemon for HP-UX and other operating systems that includes the host resource MIB.

I'm also author and maintainer of the Nagios::Object and Nagios::Cmd modules on CPAN.

2003-12-02T09:00:00.000-08:00

Here is a drop-in replacement for Rajeev Kumar's bootprofile, written in (bourne|korn|posix) shell. Everything required [on my rh9/rawhide laptop] is in /bin or /sbin. It's nice when you like to put everything on separate LVM volumes like I do. Here it is: bootprofile.sh. View now: bootprofile.sh.txt. Last update: December 02, 2003.
Screenshot (46k)