Chaingang is a Beowulf-class supercomputer, which consists of a pile-of-PCs connected together through a private network and running under the Linux operating system. As described below, all the components used in the building of chaingang were off-the-shelf or commodity components, which means that we spent roughly $18K to get a very significant computing punch. As you'll see in the pictures and descriptions, our system is pretty vanilla, and looks exactly like other Beowulfs out there (a bunch of boxes sitting on a shelf, with hundreds of feet of cables of different sorts all over the place...). We use it mainly for computational chemistry calculations (ab initiocalculations, molecular dynamics simulations, etc.).
Chaingang in now the flagship of the West Center for Computer-Aided Drug Discovery at the Department of Chemistry & Biochemistry (DC&B), University of the Sciences in Philadelphia (USP).
The idea of chaingang started in August of 1999. At this time we were pondering different options for a powerful multiprocessor number-cruncher for our computational chemistry applications. We looked into low-end SGI systems, but even a system with only 4 processors was in the 35 to 50K range. Larger mainframes start at around 100K, and then prices go up steadily.
Then we bumped, more or less by accident, into the web-page of the Stone-Soup Supercomputer, a Beowulf-class supercomputer at the Oak Ridge National Laboratory. More research into the subject and some pipe dreaming indicated that we could build a pretty decent number-cruncher for less than 20K. More research led us to a countless number of web-pages, all with invaluable information about different designs, building strategies, software, benchmarks, and tons more.
Next came some preliminary testing. We went down to the University's Information Technology Department, were John Masciantonio (both him and the rest of the people at IT were a huge help throughout) gave us a pile of beaten-up Pentium-75 Dell and Gateway boxes. We installed Linux in all of them, linked them together (at this time through a painfully slow and congested 10BT hub), and voila, we had minigang, our first test Beowulf cluster with one server and 4 nodes. We performed some tests using the ab initiopackage GAMESS, and found that it performed slightly faster than one of our SGI O2 R10000 workstations!!! Furthermore, the total cost of this machine was $0.00, a real bargain. At this point, we knew that a Beowulf cluster was the answer to our computational needs...
Hardware and Parts (including prices)
At this point we started preparing our 'dream' shopping list. We decided on a 16-node cluster plus a server, and started looking for prices on the web. After much looking and hunting, we decided for the folks at thelinuxstore. They had good prices, good support, and we had purchased a dual PIII compute server earlier from them that is a sweet machine. Mary Joyce was our contact there, and she was and still is really helpful whenever problems show up.
So we prepared a shopping list, and sent it to them. This is the current configuration of the server and nodes in chaingang. The total prices are included.
Server
| SuperMicro P6SBS (SCSI) 440BX 100MHz ATX with AGP |
| Intel Pentium III 450 MHz with 512KB cache |
| 256MB DIMM SDRAM 100MHz |
| (2) 3Com 3C905B 10/100MB PCI Ethernet Card |
| Seagate Barracuda 18GB UltraWide SCSI Hard Drive (**) |
| 3dfx VooDoo 3 3000 AGP w/16MB |
| Mitsumi 1.44 Floppy Drive |
| ATX Case |
| 40 X CD ROM |
| 19 inch Gem monitor |
| Keyboard & Mouse |
| Total - Server |
$2,387.00 |
Nodes (everything
x 16)
| SuperMicro P6SBA 440BX 100MHz ATX w/AGP |
| Intel Pentium III 450 MHz with 512KB cache |
| 128MB DIMM SDRAM 100MHZ (*) |
| 3Com 3C905B 10/100Mbit PCI Ethernet Card |
| 4.3GB Seagate Medalist 4311 Ultra ATA/66 |
| S3 Virge 4 MB PCI or AGP Video Card |
| Mitsumi 1.44 Floppy Drive |
| ATX Case |
| Total - 16 Nodes |
$13,327.00 |
Switch
| 3Com SuperStack II 3300 100MB 24-port switch |
$1,549.00 |
Micellaneous
| Wiretech rack with 4 shelves (Staples) |
| Belden CAT 5 cable - 200 feet spool (Datawarehouse) |
| Kit with RJ45 connectors, and network tools (Datawarehouse) |
| (4) Power strips (Walmart) |
| Total - Micellaneous |
$400.00 |
| Grand Total |
$17,663.00 |
We prepared a small tutorial on how we built the server and nodes, and on how we installed the software. It also describes how to wire the network and power you'll need on a system like chaingang.
(*) NOTE 1: In August of 2000 we overhauled chaingang and installed 128 MB of extra memory on all nodes, bringing the total memory of the system to 4.45 GB.
(**) NOTE 2: In September of 2000, and after upgrading the kernel of the server, we also added a large 73 GB drive to chaingang.
Architecture of the private network
As we said initially, our system is pretty vanilla. We followed the simplest network configuration possible. All nodes and the server connected to the SuperStack II switch, and the extra NIC on the server wired to the public network:
This is by far the easiest way to go, and the one that has been proven to work by countless people.
This section describes the software we have installed on chaingang.
System Software
All the nodes have Linux running the 2.2.5-15 kernel, as it comes in the RedHat 6.0 distribution CD. On the server we had to upgrade from the 2.2.5-15 to the 2.2.16-3 kernel to get a 73 GB SCSI drive to work properly. Some folks have clusters running Windows-NT, or mixed environments of Linux and Windows-NT, but everyone we spoke with said that NT was way too unstable. Furthermore, who wants to pay for an unstable OS? Go Linux, period.
The parallel environment packages we use depends on the code we run. Most of the programs use MPI (Message Passing Interface) as a communication protocol. We have installed a free version of MPI, MPICH, mainly from the Argonne National Laboratory, which works really nice.
Although we have not worked with it so far, we also installed PVM, or Parallel Virtual Machine, from the Oak Ridge National Laboratory.
Finally, some programs have their own commercial parallel environment tools. The computational chemistry package Gaussian98 uses Linda to run on our workstation cluster.
We also have a set of tools to manage the cluster, which we obtained from different sources. The idea has been not so much to develop sophisticated tools, but to do what we need to do faster and more effectivelly.
Applications
All the applications we have installed on chaingang are related to molecular modeling and computational chemistry.
We have a copy of Gaussian98, revision a.7, which, as said before, runs on the cluster using Linda. We use the code for different types of ab initiocalculations, such as geometry optimizations, charge derivation, and transition state studies. The main use of Gaussian98 on chaingang, however, has been the calculation of 13C chemical shifts for polysaccharides at different Hartree-Fock and DFT theory levels. A poster presented at the ACS Meeting in New Orleans on August of 1999 with preliminary results on this can be found here.
After making several tweaks, we also got the ab initiopackage GAMESS (both versions from 1998 and 1999) up and running. These use either MPI (98), and either MPI or TCP/IP sockets (99) as communication platforms. In the near future, we will place a tutorial describing the changes to GAMESS 98 and 99 we had to do to get them working with MPI.
We have installed and are in the process of configuring MDynaMyx, from the Arrhenius Laboratory , Stockholm University. This program is used for molecular dynamic simulations of solvated systems, particularly oligosaccharides and polypeptides. It uses MPI as its communication platform.
We also have a copy of the parallel molecular dynamics program NAMD, from the Theoretical Biophysics Group, University of Illinois at Urbana-Champaign (UIUC). We still have to install and configure the package, which requires the CHARM++ and Converse from the Parallel Programming Laboratory, UIUC, to run in parallel.
Finally, we will receive AMBER 5.1 in the near future. Although there are no hints on how to make this work with MPI on a Linux cluster that we know of, the parallel implementation for Cray supercomputers runs using MPI, so there is probably a way to make it work with Intel/Linux.
Some Benchmarks
These will be here as soon as we get them...
Here is a very small list of sites with information about Beowulf clusters and how to build them. Only some sites are listed, others can be found within these primary sites.
Additional useful links will be added periodically to this list.
A lot of people was involved with the design and construction of chaingang. Dr. Guillermo Moyna coordinated the efforts, and spent quite a bit of time building nodes and installing software. Mike DeGrazia and Chet Swalina, undergraduate students in Dr. Moyna's lab, built several of the nodes, and helped around the WC-CADD during installation. Farzad Korbafad and Fran Zappaterini also pitched in, building some of the nodes and solving some of the keyboardless booting issues. Dr. Randy J. Zauhar helped with the original design of the beast.
