Building a Budget Beowulf

(This page is in progress. Excuse the ommissions, typos, etc.)

[Todo: clean up intro] Cheap Beowulf for experimenting with , learning the trade offs, software development, etc. Although small and relatively low-powered, the same principles apply to a high end system, but the latter would use high end node hardware and networking. This doc doesn't go into that in any detail, for more information see "Beowulf Cluster Computing with Linux" (photo right) for more info -- or for serious work, go with a preconfigured system from an experienced company (such as Aspen Systems, Inc.).

The book. (Order it from Amazon.com)

Determine your constraints

Constraints may include budget, space/power/etc, cluster size, what you're going to do with it, etc. We'll assume pretty generic experimentation here, base more on available cheap hardware rather than optimizing for a specific task (render farm vs numerical analysis vs modeling, etc) although in general faster and more memory is better (of course).

In my case the design was influenced by a used computer sale: Dell GXi desktop boxes for $15 each, including P166-MMX, 64 MB RAM, 2GB HD, and built in 100TX NIC. Low horsepower these days but a bargain otherwise (no CD drive -- which isn't really needed in a cluster node -- but has a floppy). [photos] I was figuring a budget in the $100-$150 neighborhood, and bought 5 of the machines. (Later I thought I should have bought more -- but by then they were sold out.) This gives me 4 slave nodes and 1 master.

You'll also need a network switch (or a hub if you're desperate, but a switch reduces network collisions and these days costs about the same as a hub). I got an 8-port 10/100 switch for $35, although in the final cluster I swapped it for the 5-port switch I'd been using in my office. [photos] I already had a spare old monitor and mouse, I picked up an extra (used) keyboard for $2. (Typical used prices for mouse and monitor would be about $1 and $50.)

Network diagram of my setup.

If you want to splurge, you can get a KVM (keyboard, video, mouse) switch to switch the keyboard, monitor and mouse between the different nodes, but it really isn't necessary -- you'll be controlling the nodes over the network, and you can just manually plug the cables into each machine for the initial check out and software install. A mechanical 4-way KVM switch is about $30, but will start to drive you crazy when the contacts get dirty and start degrading the video signals or glitching when you switch. For $60 you can find cheap electronic KVMs, and you can go way up from there depending on how fancy you want (eg keyboard controlled switching, scanning, daisy chaining, etc). For regular mult-computer use, get an electronic KVM switch -- although again, you don't need one for the Budget Beowulf. I didn't bother with one.

2. Housing

where are you going to put all your boxes? The "traditional" method for clusters made of cheap surplus computers -- usually various shapes and sizes -- is LOBOS, Lots Of Boxes On Shelves. Modular shelving of the sort made with steel frame and particle-board shelves works pretty well. Desktop (pizza-box) machines stack, although that can make it awkward to get into a given machine if you need to fix the hardware. Keep in mind that you'll want to access both the front and back of the machines.

As it happens, I had an old typewriter table that turned out to be a perfect fit for the low-profile Dell boxes I bought. (You can't lose them all!) Since I'm only stacking them four high, I can live with having to disassemble the stack if I need to get into one. [photos] To ensure that the computers had a solid base within the legs of the typewriter table, I made a platform of 1/2 plywood to fit. The legs themselves are snug enough around the computer cases to hold them, but you might want to run a strap around everything to help keep it all in place, particularly if your setup is movable (mine is on wheels). [photos]

Cabling:

Making custom cables is a bit of a pain but cheaper than buying, and you can make them the exact length you need, but go ahead and buy them if you don't have access to a crimping tool. [photos]

Use a power bar with built-in surge protection, and mount it at a convenient location on your rack. You don't have to keep your cables neat, but it looks better and it'll keep you from tripping over them. If your cable runs are long (mine aren't), you might want to label the cable ends so you know which is which. [photos]

Configuring the Hardware.

I wanted the cluster on its own subnet, with the master node serving as the gateway to the rest of my household LAN. This meant adding another NIC (network interface card) to the master node. I also wanted to add a CD-ROM drive to it to simplify software installation, although I could have done all that over the network. It turns out that the slim desktop model Dells had a relatively low power power supply, 145 Watt, so I need to pay attention to the power requirements of anything I added. The NIC (I had a spare 3C905) doesn't consume much, and of the couple of spare CD-ROM drives I had, I chose the lower power one. [photo] In a large cluster, power consumption (and related heat output) can be a significant factor, so plan accordingly. Opening up one of the Dells to install the extra NIC and CD-ROM drive was pretty straightforward. [photos]

Setup.

Before assembling everything, and after a very quick check to make sure all the machines would boot up, I took a few minutes to blow accumulated dust out of the machines and particularly off the PSU fans. [photos] The next step is to configure the BIOS on each node. In particular, you want it to ignore "missing keyboard" errors on boot (unless you're using a KVM). I also made sure the time was set and that they would boot off a floppy if present. [photo?] Once I was happy that the node hardware worked and was configured enough to allow me to do the OS install (via floppy and network), I installed the machines into their "rack", connected network cabling and power, and was ready to install software. [photo?]

Software Install.

Since I wanted to be able to run the cluster entirely self-contained (isolated from the house LAN) at times, I did a full install (SuSE Linux 8.1) on the Master node. Well, not quite full, that'd be about twice as much software as the 2G hard drive could hold. I skipped the games and office productivity software. SuSE Linux supports network installs using "AutoYaST", similar to Solaris's Jumpstart if you've used that. Create a boot floppy with a configuration file, and the install software downloads the needed packages from an install server on your network. More about that can be found in the Network Install HOWTO at The Linux Documentation Project (www.tldp.org)

[Fill in the details here].

Configuring Clustering Software.

[Todo]

Running Parallel Programs.

[Todo]

Links and References.

[Todo]