ProjectOur goal was to construct a mini-Beowulf cluster our of 4 Power Macs that the Creighton Physics department no longer had any other use for. This was a summer project undertaken mostly to learn about building Beowulf clusters. The cluster was never fully completed because we all went our separate ways at the end of the summer. I don't know that this is the either the only or the best way to do this; it's a record of what we found to work and the obstacles we overcame.
PersonelJiro Fujita, Lenore Horner, Roger Loucks
General Requirementsa controller computer - must have 2 ethernet cards, one to the outside world and one to a switch (or hub) connecting to the rest of the world
nodes - need one ethernet card
a switch (or hub)
cables for all of the above
It's useful to have a monitor switch so one monitor/keyboard/mouse can be used for installing Linux on each machine. Note if it's not actively terminated you must have the screen and keyboard switched to whatever system is booting at the time until you get the password prompt. In headless mode, terminators may be needed on the terminal ports.
A CD (bought or burned) of one of the PPC linux distributions.
A CD containing LinuxDisks.
Specifics of our system
ControllerPower Mac 7100/80 with a 1GB hard drive and internal CD and 32MB RAM
Nodes3 PowerMac 6100/60's with 1GB drives and no internal CD and 72MB RAM
Switch/Hub5 channel hub
SoftwareYellowDog Linux 1.2 http://matrix.tesi.dsi.unimi.it/pub/yellowdoglinux-1.2/YellowDog/
FWB Toolkit CD
Setting Up the System
Install LinuxPPC 2000 Q4 on ControllerGo to http://nubus-pmac.sourceforge.net/ and download the kernal with Yellowdog 1.2 installer, the stable kernal for YD1.2, and MkLinux Booter.
Unzip the first to get Mach Kernal and the second to get a MkLinux folder. See below for final locations of these items.
If necessary download an appropriate version of Stuffit Expander to unstuff the kernal.
METHOD 1Use FWB to repartition drive (without erasing the things we just downloaded).
Boot from the CD.
Open HDT PE alias (italics).
Select the hard disk and from Devices menu choose View Partitions.
Select Macintosh HD from the resulting list then click resize volume and choose a new size. Because it already has a system installed, you'll have to choose a size larger than the system etc. occupies, but small enough to easily get the other partitions out of the free space thus created.
Then make the other spaces out of Free Space using the New Volumes command in the Devices menu.
Finally resize the biggest space to take up the remaining room.
Note the partition numbers of the four partitions below.
/boot = 20MB (remember that this must be configured as Macintosh standard could actually be 16M)
/root = 50MB (Linux PPC)
swap = 32MB (choose other and then type Apple_Unix-SVR2))
/usr = ~900MB (rest)
Note that we are going to have to be very careful about what we install.
Now create a system folder in the /boot partition.
Copy to it the folder Fonts and the files Finder and System as well as the applications BBedit Lite and SimpleText.
Create the folders Preferences, Extensions, Control Panels, Apple Menu Items, Startup Items, Shutdown Items.
In the Extensions folder copy the Apple CD-ROM extension so we can read CDs, MkLinux booter, and Mach Kernal and in an untitled folder the correctly named and expanded stable kernal for later.)
In Control Panels put MkLinux (this is the booter application).
In Preferences put lilo.conf and MkLinux preferences.
Rebuild the desktop by restarting with command-option held until you get a query.
Make sure no other disk has a system folder with Finder in it. If it does, hide it in an unnamed folder.
METHOD 2 (if you have appletalk connection to another computer and a floppy drive)Download the materials above and create a folder equivalent to the /boot partition above. Add Fetch and Netscape Navigator for insurance.
Download from www.apple.com the 7.5 network access floppy image and put it on a floppy disk.
Download HD Setup 7.3.5 from www.apple.com
Download from www.apple.com ResEdit.
Use ResEdit to make the setup recognise non-Apple disk per www.euronet.nl/users/ernstoud/patch.html and name result Apple HD Setup 7.3.5 linux.
Add the patched setup to the folder.
Save this folder on an Appletalk accessible machine so for handy use in later installations.
The floppy will allow us to boot and access this material at anytime regardless of what we've done to the hard drive.
Format the HD using the patched Setup over the network.
Initialize the disk. (Make sure it's the right one, the local hard drive seems to be what comes up naturally.)
Partition the disk.
Select the big partition and click remove.
Draw a box around chunks of the gray area. A dialog appears. Type the size desired then click on a designation in the box on the left.
When all partitions have been created, click "Done".
Quite the installer when all partitions have been chosen.
Copy the prepared things to the /boot sector via the netboot floppy disk.
Use an Apple CD to install a minimal system to the /boot sector
Installing the System(also see nubus mailing list archived and searchable on www.linuxppc.org)
Reboot the system.
Run MkLinux from the control panels under the Apple menu.
Pick Custom to edit the lilo.conf file with Simple Text (So you need simple text somewhere, but it needn't be in the /boot partition.)
Edit the line so that you have rootdev=/dev/*** where *** is the designation for the /boot partition you have gotten from FWB tool.
At the bottom of the file put a line mach_options=ramdisk=9182.
Leave at least one empty line at the bottom of the file.
In the installer, you will have to go through fdisk to assign mountpoints.
Note that you want to say "done" immediately at the partitioning stage to go on to give mount points.
Note that assigning /boot is forbidden. Swap should automatically come up with Linux_Swap as mount point when you go to the assignment dialog box. Only /usr and / can be assigned now. The next box acknowledges the swap space.
Packages to installcomponents: Mac, XWindow, ,Gnome, KDE, Networked workstation, Networked manager workstation
in development/languagescpp (found out later we should have installed this)
in development/librariesglibc-devel (configure later shows that we need this for the file crt1.o)
in development/systemkernel-header (this is necessary for glibc-devel)
in system environement/basechkfontpath
in system environment/deamonsinetd
in system environment/kernal
in system environement/librariesfnlib
in system environment/shells
in user interface/desktopscontrol-center
in user interface/Xgdm
in user interface/x hardware supportXconfigurator
Do not permit installation of packages to fix broken dependencies!
Follow directions in http://www.infernosoft.com/beowoof/installingLinux.html regarding the mouse quoted below.
After it has installed all the packages, there will be a window that says 'An error occurred
during setup "Configure Mouse" of the install.' Select "Menu." Go through each of the items
one by one, skipping the mouse item. The items you need to do are...
Install Bootloader - write down the modification you need to make to the lilo.conf file.
Then go back to the top of the list and do Continue Install.
It finishes and puts "Complete" in upper right corner of the screen. At this point you can
reboot. Sometimes it breaks in this step. It seems to help to let it rest for a few minutes before
When you're done, reboot the Mac. When the MkLinux dialog box comes up, choose
Replace the kernel with the stable one.
Throw the old System Folder:Extensions:Mach Kernel file away
Copy the stable one from your Mac OS Mac to System Folder:Extensions
Rename it Mach Kernel.
Edit the lilo.conf file
should be what the step Install Bootloader told you.
Uncomment the line
The ramdisk line should be commented out.
Reboot the Mac and let it boot into Linux.
The anomaly in fdisk (prohibition on assigning mount point /boot) is followed by no /etc/fstab being written.
Reboot into MacOS.
Use BBedit (or any text editor in which Linux line endings can be written) to create an fstab file. (In BBedit check Unix line endings and put in hard wraps.)
Use LinuxDisks to put this file in the /etc folder of the unix files system. (Expand the hard-disk, and double-click / to see the file system. Then drag-and-drop.)
Reboot into Linux.
Configure X Windows.
Reboot. Select MkLinux. You should see the usual Linux startup stuff followed by a Linux
Once you have a usable command line going, you need to do a few Linux administration chores.
Add the CD ROM to the fstab.
Append the line
/dev/scd0 /mnt/cdrom iso9660 noauto,ro 0 0
to the file /etc/fstab . Sometimes the existing line with /dev/cdrom works fine. If so, then
just leave it.
Set the X mouse to the Apple Desktop Bus mouse.
Warning: Do not run #mouseconfig all by itself with no parameters. It will crash your
computer. (Well, it did mine, anyway.) And do not run this from inside the Yellow Dog
install program. It will crash your computer. (Well, it did mine, anyway.)
Mount the Yellow Dog installer CD ROM
Stick it in the drive, then issue the commands
#mount /dev/scd0 /mnt/cdrom (Unless you didn't have to alter the fstab for the cd in which case use /dev/cdrom.)
Install the Mac X server.
#rpm -i XFree86-Xpmac-3.3.6-11.2.ppc.rpm
Create a symbolic link.
#ln -s Xpmac XF68_FBDev
so that when Xconfigurator looks for XF86_FBDev it uses Xpmac. (Yes, that's 68, not
Run the X configurator.
For Interex two-button mouse, choose Mac two-button and do _not_ choose three-button emulation.
(If you want to change resolution/depth from Linux, use fbset. If it's not installed, do "yup
install fbset" and "man fbset". For example, "fbset -xres 832 -yres 624 -depth 8" should
change the resolution/depth to 832x624-8bit. If this doesn't work, check that ariel2fb is
being used. "cat /proc/fb" should show "0 ariel2fb".)
Even if Xconfigurator crashes with a core dump, X will now run.
5. Start X Windows.
If all works right, you will have a workable X system. If X ever does anything seriously
unhappy, press command-F1. This will kill X and give you some debug messages. You
might be able to make sense of them and reinstall the missing bits.
Once you've got X working correctly, you may want your Linux box to boot into X by
default. Edit the file /etc/inittab. Change the line
so that it reads
For some reason, make was not installed so install it on the controller computer from the YellowDog rpms. See procedure above for installing pmac for syntax.
Download MPICH from http://www-unix.mcs.anl.gov/mpi/mpich/ or more directly from anonymous ftp from ftp.mcs.anl.gov in the directory pub/mpi.
There are instructions on that site for expansion, installation, and testing. Follow them. Elaborations follow.
Got to init 3 for speed. This system is taxed by X.
Root cannot telnet or rsh. The installation and use must be by
a user or member of a user group owning the directories.
The final - in the gunzip line is important (for tar).
The tar file should be downloaded into a directory in a partition big enough to hold things or errors will result (i.e. /usr not /).
Email from Anthony Chan indicates lib won't appear until after "configure" and "make".
In the directory of the unpacked mpich type ./configure --prefix=/usr/local/mpich-1.2.1.
(It seems to be necessary to use this flag at least; failure leads to "make install" giving an error. If things are missing - say cpp, errors will indicate that. If a mystery file is reported missing, try to find a computer that has the file and run rpm -qf path/file on it. This will tell what rpm it comes from.)
Now that configure has run, from the same directory type make.
The lib file has appeared now in the directory.
We actually still need to install the stuff by running "make install" as root (or we don't have authority to write some necessary file).
Create the file /etc/hosts.equiv per instructions. (Two lines: localhosts and your-machine-name).
Install ftp server.
Uncomment telnet, rshd, and ftp lines in /etc/inetd.conf then run /etc/rc.d/init.d/inet restart.
If this file isn't present, go back and rpm inetd.
If telnet still fails install telnet-server.
Run make testing.
(If it fails, and the failure notes an inability to contact a machine (which will be itself) then rsh or telnet is not fully implemented or you may be root.)
Now we need to tell the controller about the other machines.
Edit the file mpich/util/machines/machines.xxxx, * to contain names
of machines of architecture xxxx. The xxxx matches the arch given when mpich was configured.
We also need to make the second ethernet card on the controller work.
dmesg |grep eth1 - to verify the computer knows it has a second card
ifconfig eth1 up - to activate the second card
ifconfig -i - to check that it's activated -- should see etho0, eth1, and lo.
ifconfig eth1 down
ifconfig eth1 192.168.0.254 (this is a fake address with 254 being the default gateway designator.)
ifconfig -i to check results (look at the addresses and see if they make sense)
The second card and the nodes all need to have IP addresses in the same subnet. They are internal addresses so we don't need to worry about assigning fixed addresses.
Go to the nodes and do essentially the same thing: ifconfig eth0 192.168.0.(diff numbers for each node).
Must tell controller the route to its nodes.
route add xxx.yyy.z.0 gw xxx.yyy.z.1 (if .1 is the controller)
Try pinging the controller to check whether things work. (No one should be in the uplink slot on the hub).
This leaves the nodes able to find the controller, but not to see past the controller (even with an explicit IP address).
We also need to have each node running rsh and telnet.
ftp turns out to be useful for correcting installation errors in machines without a CD drive.
Currently, the installation tests on the controller, but we have not
succeeded in getting it to run on the nodes. However, this may be
a problem with rsh and telnet still (very, very slow) and not mpich.