Project

Our goal was to construct a mini-Beowulf cluster our of 4 Power Macs that the Creighton Physics department no longer had any other use for. This was a summer project undertaken mostly to learn about building Beowulf clusters. The cluster was never fully completed because we all went our separate ways at the end of the summer. I don't know that this is the either the only or the best way to do this; it's a record of what we found to work and the obstacles we overcame.

Personel

Jiro Fujita, Lenore Horner, Roger Loucks

Equipment

General Requirements

a controller computer - must have 2 ethernet cards, one to the outside world and one to a switch (or hub) connecting to the rest of the world
nodes - need one ethernet card
a switch (or hub)
cables for all of the above
It's useful to have a monitor switch so one monitor/keyboard/mouse can be used for installing Linux on each machine. Note if it's not actively terminated you must have the screen and keyboard switched to whatever system is booting at the time until you get the password prompt. In headless mode, terminators may be needed on the terminal ports.
A CD (bought or burned) of one of the PPC linux distributions.
A CD containing LinuxDisks.

Specifics of our system

Controller

Power Mac 7100/80 with a 1GB hard drive and internal CD and 32MB RAM

Nodes

3 PowerMac 6100/60's with 1GB drives and no internal CD and 72MB RAM

Switch/Hub

5 channel hub

Software

YellowDog Linux 1.2 http://matrix.tesi.dsi.unimi.it/pub/yellowdoglinux-1.2/YellowDog/

FWB Toolkit CD

Setting Up the System

Install LinuxPPC 2000 Q4 on Controller

Go to http://nubus-pmac.sourceforge.net/ and download the kernal with Yellowdog 1.2 installer, the stable kernal for YD1.2, and MkLinux Booter.
Unzip the first to get Mach Kernal and the second to get a MkLinux folder. See below for final locations of these items.
If necessary download an appropriate version of Stuffit Expander to unstuff the kernal.

METHOD 1

Use FWB to repartition drive (without erasing the things we just downloaded).

Boot from the CD.
Open HDT PE alias (italics).
Select the hard disk and from Devices menu choose View Partitions.
Select Macintosh HD from the resulting list then click resize volume and choose a new size. Because it already has a system installed, you'll have to choose a size larger than the system etc. occupies, but small enough to easily get the other partitions out of the free space thus created.
Then make the other spaces out of Free Space using the New Volumes command in the Devices menu.
Finally resize the biggest space to take up the remaining room.
Note the partition numbers of the four partitions below.

Partition Sizes
/boot = 20MB (remember that this must be configured as Macintosh standard could actually be 16M)
/root = 50MB (Linux PPC)
swap = 32MB (choose other and then type Apple_Unix-SVR2))
/usr = ~900MB (rest)

Note that we are going to have to be very careful about what we install.

Now create a system folder in the /boot partition.
Copy to it the folder Fonts and the files Finder and System as well as the applications BBedit Lite and SimpleText.
Create the folders Preferences, Extensions, Control Panels, Apple Menu Items, Startup Items, Shutdown Items.
In the Extensions folder copy the Apple CD-ROM extension so we can read CDs, MkLinux booter, and Mach Kernal and in an untitled folder the correctly named and expanded stable kernal for later.)
In Control Panels put MkLinux (this is the booter application).
In Preferences put lilo.conf and MkLinux preferences.
Rebuild the desktop by restarting with command-option held until you get a query.
Make sure no other disk has a system folder with Finder in it. If it does, hide it in an unnamed folder.

METHOD 2 (if you have appletalk connection to another computer and a floppy drive)

Download the materials above and create a folder equivalent to the /boot partition above. Add Fetch and Netscape Navigator for insurance.
Download from www.apple.com the 7.5 network access floppy image and put it on a floppy disk.
Download HD Setup 7.3.5 from www.apple.com
Download from www.apple.com ResEdit.
Use ResEdit to make the setup recognise non-Apple disk per www.euronet.nl/users/ernstoud/patch.html and name result Apple HD Setup 7.3.5 linux.
Add the patched setup to the folder.
Save this folder on an Appletalk accessible machine so for handy use in later installations.
The floppy will allow us to boot and access this material at anytime regardless of what we've done to the hard drive.

Format the HD using the patched Setup over the network.
Initialize the disk. (Make sure it's the right one, the local hard drive seems to be what comes up naturally.)
Partition the disk.
Select the big partition and click remove.
Draw a box around chunks of the gray area. A dialog appears. Type the size desired then click on a designation in the box on the left.
When all partitions have been created, click "Done".
Quite the installer when all partitions have been chosen.
Copy the prepared things to the /boot sector via the netboot floppy disk.
Use an Apple CD to install a minimal system to the /boot sector

Installing the System

(also see nubus mailing list archived and searchable on www.linuxppc.org)

Reboot the system.
Run MkLinux from the control panels under the Apple menu.
Pick Custom to edit the lilo.conf file with Simple Text (So you need simple text somewhere, but it needn't be in the /boot partition.)
Edit the line so that you have rootdev=/dev/*** where *** is the designation for the /boot partition you have gotten from FWB tool.
At the bottom of the file put a line mach_options=ramdisk=9182.
Leave at least one empty line at the bottom of the file.

In the installer, you will have to go through fdisk to assign mountpoints.
Note that you want to say "done" immediately at the partitioning stage to go on to give mount points.
Note that assigning /boot is forbidden. Swap should automatically come up with Linux_Swap as mount point when you go to the assignment dialog box. Only /usr and / can be assigned now. The next box acknowledges the swap space.

Packages to install

components: Mac, XWindow, ,Gnome, KDE, Networked workstation, Networked manager workstation

in amusements/games

in amusements/graphics

in applications/archiving

sharutils

in applications/communications

in applications/databases

in applications/editors

gedit

in applications/emulators

in applications/engineering

in applications/file

hfsutils

in applications/internet

ftp
rsh
telnet

in applications/multimedia

in applications/network

in applications/productivity

in applications/publishing

in applications/system

autorun
control-panel
gnome-linuxconf
gnome-utils
gnorpm
kdeadmin
kdeutils
kernalcfg
kpackage
macutils
magicdev
mkxauth
pmac-utils
timetool
tksysv
usermode

in applications/text

in base

in development/debuggers

in development/languages

cpp (found out later we should have installed this)
gcc
gcc-c++
gcc-f77
tcl
tclx
tix
tk
tkinter

in development/libraries

glibc-devel (configure later shows that we need this for the file crt1.o)

in development/system

kernel-header (this is necessary for glibc-devel)

in development/tools

make

in documentation

in system environement/base

chkfontpath
ipchains
yp-tools

in system environment/deamons

inetd
iputils
lpr
pidentd
portmap
rusers
tcp_wrappers
XFree86-xfs
ypserv

in system environment/kernal

in system environement/libraries

fnlib
freetype
gnome-libs
gtk+
gtk-engines
imlib
imlib-cfgeditor
kdelibs
libgr
libgtop
libjpeg
libpng
libtiff
libungif
libxml
qt
qt1x
XFree86-libs
xpm

in system environment/shells

in user interface/desktops

control-center
enlightenment
enlightenment-conf
gmc
gnome-core
kdebase
dkebase-lowcolor-icons
kdesupport
switchdesk
switchdesk-gnome
switchdesk-kde
wmconfig

in user interface/X

gdm
gqview
rxvt
urw-fonts
X11R6-contrib
XFree86
XFree86-100dpi-fonts
XFree86-75dpi-fonts
xinitrc

in user interface/x hardware support

Xconfigurator

Do not permit installation of packages to fix broken dependencies!

Wait!
Follow directions in http://www.infernosoft.com/beowoof/installingLinux.html regarding the mouse quoted below.

After it has installed all the packages, there will be a window that says 'An error occurred
   during setup "Configure Mouse" of the install.' Select "Menu." Go through each of the items
   one by one, skipping the mouse item. The items you need to do are...
      Networking
      Time Zone
      Services
      Root Password
      Install Bootloader - write down the modification you need to make to the lilo.conf file.
      Configure X
      Exit Install
   Then go back to the top of the list and do Continue Install.
It finishes and puts "Complete" in upper right corner of the screen. At this point you can
   reboot. Sometimes it breaks in this step. It seems to help to let it rest for a few minutes before
   rebooting.
When you're done, reboot the Mac. When the MkLinux dialog box comes up, choose
   Macintosh.
Replace the kernel with the stable one.
      Throw the old System Folder:Extensions:Mach Kernel file away
      Copy the stable one from your Mac OS Mac to System Folder:Extensions
      Rename it Mach Kernel.
Edit the lilo.conf file
      the line
      rootdev=/dev/sdb5
      should be what the step Install Bootloader told you.
      Uncomment the line
      mach_options= video=ariel2fb:
      The ramdisk line should be commented out.
Reboot the Mac and let it boot into Linux.

The anomaly in fdisk (prohibition on assigning mount point /boot) is followed by no /etc/fstab being written.
Reboot into MacOS.
Use BBedit (or any text editor in which Linux line endings can be written) to create an fstab file. (In BBedit check Unix line endings and put in hard wraps.)
Use LinuxDisks to put this file in the /etc folder of the unix files system. (Expand the hard-disk, and double-click / to see the file system. Then drag-and-drop.)
Reboot into Linux.

Configure X Windows.
Reboot. Select MkLinux. You should see the usual Linux startup stuff followed by a Linux
prompt.
Once you have a usable command line going, you need to do a few Linux administration chores.
Add the CD ROM to the fstab.
   Append the line
   /dev/scd0 /mnt/cdrom iso9660 noauto,ro 0 0
   to the file /etc/fstab . Sometimes the existing line with /dev/cdrom works fine. If so, then
   just leave it.
Set the X mouse to the Apple Desktop Bus mouse.
   #mouseconfig ADB
   Warning: Do not run #mouseconfig all by itself with no parameters. It will crash your
   computer. (Well, it did mine, anyway.) And do not run this from inside the Yellow Dog
   install program. It will crash your computer. (Well, it did mine, anyway.)
Mount the Yellow Dog installer CD ROM
   Stick it in the drive, then issue the commands
   #mount /dev/scd0 /mnt/cdrom   (Unless you didn't have to alter the fstab for the cd in which case use /dev/cdrom.)
   #cd /mnt/cdrom/YellowDog/RPMS
Install the Mac X server.
   #rpm -i XFree86-Xpmac-3.3.6-11.2.ppc.rpm

Create a symbolic link.
   #cd /usr/X11R6/bin
   #rm XF68_FBDev
   #ln -s Xpmac XF68_FBDev
   so that when Xconfigurator looks for XF86_FBDev it uses Xpmac. (Yes, that's 68, not
   86.)
Run the X configurator.
   #Xconfigurator
For Interex two-button mouse, choose Mac two-button and do _not_ choose three-button emulation.
   (If you want to change resolution/depth from Linux, use fbset. If it's not installed, do "yup
   install fbset" and "man fbset". For example, "fbset -xres 832 -yres 624 -depth 8" should
   change the resolution/depth to 832x624-8bit. If this doesn't work, check that ariel2fb is
   being used. "cat /proc/fb" should show "0 ariel2fb".)
Even if Xconfigurator crashes with a core dump, X will now run.
5. Start X Windows.
   #startx
   If all works right, you will have a workable X system. If X ever does anything seriously
   unhappy, press command-F1. This will kill X and give you some debug messages. You
   might be able to make sense of them and reinstall the missing bits.
   Once you've got X working correctly, you may want your Linux box to boot into X by
   default. Edit the file /etc/inittab. Change the line
   id:3:initdefault:
   so that it reads
   id:5:initdefault:

Additional tools

For some reason, make was not installed so install it on the controller computer from the YellowDog rpms. See procedure above for installing pmac for syntax.

See http://www-unix.mcs.anl.gov/mpi/.

Download MPICH from http://www-unix.mcs.anl.gov/mpi/mpich/ or more directly from anonymous ftp from ftp.mcs.anl.gov in the directory pub/mpi.

There are instructions on that site for expansion, installation, and testing. Follow them. Elaborations follow.

Got to init 3 for speed. This system is taxed by X.

Root cannot telnet or rsh. The installation and use must be by a user or member of a user group owning the directories.
The final - in the gunzip line is important (for tar).
The tar file should be downloaded into a directory in a partition big enough to hold things or errors will result (i.e. /usr not /).
Email from Anthony Chan indicates lib won't appear until after "configure" and "make".
In the directory of the unpacked mpich type ./configure --prefix=/usr/local/mpich-1.2.1.
(It seems to be necessary to use this flag at least; failure leads to "make install" giving an error. If things are missing - say cpp, errors will indicate that. If a mystery file is reported missing, try to find a computer that has the file and run rpm -qf path/file on it. This will tell what rpm it comes from.)
Now that configure has run, from the same directory type make.
The lib file has appeared now in the directory.
We actually still need to install the stuff by running "make install" as root (or we don't have authority to write some necessary file).
Create the file /etc/hosts.equiv per instructions. (Two lines: localhosts and your-machine-name).
Enable telnet.
Install ftp server.
Uncomment telnet, rshd, and ftp lines in /etc/inetd.conf then run /etc/rc.d/init.d/inet restart.
If this file isn't present, go back and rpm inetd.
If telnet still fails install telnet-server.
Run make testing.
(If it fails, and the failure notes an inability to contact a machine (which will be itself) then rsh or telnet is not fully implemented or you may be root.)

Now we need to tell the controller about the other machines.
Edit the file mpich/util/machines/machines.xxxx, * to contain names
of machines of architecture xxxx. The xxxx matches the arch given when mpich was configured.
We also need to make the second ethernet card on the controller work.
dmesg |grep eth1 - to verify the computer knows it has a second card
ifconfig eth1 up - to activate the second card
ifconfig -i - to check that it's activated -- should see etho0, eth1, and lo.
ifconfig eth1 down
ifconfig eth1 192.168.0.254 (this is a fake address with 254 being the default gateway designator.)
ifconfig -i to check results (look at the addresses and see if they make sense)
The second card and the nodes all need to have IP addresses in the same subnet. They are internal addresses so we don't need to worry about assigning fixed addresses.
Go to the nodes and do essentially the same thing: ifconfig eth0 192.168.0.(diff numbers for each node).
Must tell controller the route to its nodes.
route add xxx.yyy.z.0 gw xxx.yyy.z.1 (if .1 is the controller)
Try pinging the controller to check whether things work. (No one should be in the uplink slot on the hub).
This leaves the nodes able to find the controller, but not to see past the controller (even with an explicit IP address).

We also need to have each node running rsh and telnet.
ftp turns out to be useful for correcting installation errors in machines without a CD drive.

Currently, the installation tests on the controller, but we have not succeeded in getting it to run on the nodes. However, this may be a problem with rsh and telnet still (very, very slow) and not mpich.