Partitions:

Network settings:

Skip the package selection stage (tasksel and dselect); install the following packages manually afterwards with apt-get. Some of them will ask configuration questions, but in many cases we will replace the configuration files with our own later anyway.

emacs21
emacs21-el
emacs-goodies-el
less
ssh
wget
unp
host
tcpdump
gawk
rsh-server
rsh-client
cfengine
slocate
blas-dev
blas-doc
blas-test
atlas2-sse2
atlas2-sse2-dev
atlas2-headers
atlas-doc
python2.2
python2.2-numeric
python-numeric
python-numeric-ext
python2.2-numeric
python2.2-numeric-ext
python-elisp
rstat-client
rstatd
rusers
rusersd
libgsl0-dev
gsl-bin
gsl-ref-psdoc
netcdf-bin
netcdfg3
netcdfg-dev
python-netcdf
netcdf-doc
gri
gri-html-doc
gri-ps-doc
grace
gv
g77
f2c
xlibs-dev
tcl8.3
tk8.3
xserver-common
xbase-clients
xfonts-base
groff
linuxdoc-tools
tetex-base
tetex-bin
texinfo
tetex-extra
linuxdoc-tools-text
linuxdoc-tools-latex
linuxdoc-tools-info
transfig
python-dev
python2.2-dev
postfix
postfix-doc
xclip
dnsmasq
netperf
mozilla
libstdc++2.10
gcc-3.0

As well, run tasksel -n install c-dev.

Disable CTRL-ALT-DELETE rebooting. In /etc/inittab, comment out

ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

Contents of /etc/networks.

boiler 192.168.253.0
localnet 142.103.140.0

Create a scratch directory.

mkdir /scratch
chmod 1777 /scratch

Since the nodes won't be in a DNS directory, we need to explicitly list them in /etc/hosts.

127.0.0.1       localhost

142.103.140.90  steamengine.physics.ubc.ca      steamengine

192.168.253.250 piston00.boiler piston00
192.168.253.1   piston01.boiler piston01
192.168.253.2   piston02.boiler piston02
192.168.253.3   piston03.boiler piston03
192.168.253.4   piston04.boiler piston04
192.168.253.5   piston05.boiler piston05
192.168.253.6   piston06.boiler piston06
192.168.253.7   piston07.boiler piston07
192.168.253.8   piston08.boiler piston08
192.168.253.9   piston09.boiler piston09
192.168.253.10  piston10.boiler piston10
192.168.253.11  piston11.boiler piston11
192.168.253.12  piston12.boiler piston12
192.168.253.13  piston13.boiler piston13
192.168.253.14  piston14.boiler piston14
192.168.253.15  piston15.boiler piston15

Delete /etc/hosts.deny, as we will use a single file, /etc/hosts.allow:

# Localhost can access everything
ALL:     127.0.0.1: ALLOW

# Also OK with physics ip interface
ALL:     142.103.140.96: ALLOW

# Let all nodes access everything
ALL:     192.168.253. : ALLOW

# Let everyone ssh here.
sshd:        ALL : ALLOW
# Let everyone use X forwarding over X
sshdfwd-X11: ALL: ALLOW

# Everything else is DENIED; send root an email.
ALL: ALL: spawn (/bin/echo "Attempted access to %s from %c" | /usr/bin/mail -s "inetd access violation" root) & : DENY

Enable X-forwarding. In /etc/ssh/sshd_config:

X11Forwarding yes

Create a /etc/hosts.equiv and also copy it to /root/.rhosts.

piston00
piston01
piston02
piston03
piston04
piston05
piston06
piston07
piston08
piston09
piston10
piston11
piston12
piston13
piston14
piston15

We are going to configure postfix to deliver mail for the other nodes.

Contents of /etc/postfix/main.cf:

# Do not change these directory settings - they are critical to Postfix
# operation.
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
program_directory = /usr/lib/postfix

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
setgid_group = postdrop
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no
#myhostname = piston00.boiler
myhostname = steamengine.physics.ubc.ca
mydomain = boiler
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = localhost localhost.boiler steamengine steamengine.physics.ubc.ca piston00 piston00.boiler $mydomain
relayhost = 
mynetworks = 127.0.0.0/8, 192.168.253.0/24
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +

Contents of /etc/mailserver:

steamengine.physics.ubc.ca

Run apt-get install ntpdate ntp-simple, use servers 132.246.168.148, 142.103.237.225 and 192.35.82.50 .

We're going to use berserk as a print server.

Edit /etc/cups/client.conf and add

ServerName berserk.physics.ubc.ca

First install iptables.

Generate firewall rules at http://morizot.net/firewall/gen.

Internet Interface: eth0
Type of Internet IP Address: Static Internet IP Address
Static IP Address: 142.103.140.96
Gateway/Firewall
  Internal Network Interface: bond0
  Internal Network IP Address: 192.168.253.1
  Internal Network: 192.168.253.0/24
  Internal Network Broadcast: 192.168.253.255
  Advanced Network Options
    Internal DHCP Server
  Allow InboundServices
    SSH
    TimeServer (NTP)

Copy the generated firewall script to /etc/init.d/easy-firewall, make it executable, and run update-rc.d easy-firewall defaults 13 to create start-up links.

Since we're going to install all the nodes, it's most efficient to create a (partial) local Debian mirror.

First install some stuff we need.

apt-get install debmirror rsync fmirror

The following script will create and/or update the local mirror.

# /usr/local/bin/mkdebmirror2

ALLOPTS="--getcontents --nosource --arch=i386 --passive --progress"

debmirror /var/debmirror/debian $ALLOPTS --host=mirror.direct.ca --root=/pub/linux/debian --dist=woody,woody-proposed-updates

debmirror /var/debmirror/debian-non-US $ALLOPTS --host=mirror.direct.ca --root=/pub/linux/debian-non-US --dist=woody,woody-proposed-updates --section=non-US/main,non-US/contrib,non-US/non-free

debmirror /var/debmirror/debian-security $ALLOPTS --host=security.debian.org --root=debian-security --dist=woody/updates

fmirror -V 2 -S -s mirror.direct.ca -r /pub/linux/debian/dists/woody/main/disks-i386 -l /var/debmirror/debian/dists/woody/main/disks-i386

We're going to have some custom debian packages, so it will be useful to have a place to put them where we can access them with apt. Make a script /usr/local/bin/update-custom-debs which copies any debs which are in /root/debs to /var/debmirror/steamengine-custom. Don't forget to create the latter directory.

# /usr/local/bin/update-custom-debs
cp /root/debs/*.deb /var/debmirror/steamengine-custom
rm /var/debmirror/steamengine-custom/Packages
cd /var/debmirror/steamengine-custom ; apt-ftparchive packages . > Packages

Allow nfs access to this directory by adding to following to /etc/exports.

/var/debmirror     192.168.253.0/255.255.255.0(ro) 142.103.140.96(ro,no_root_squash)

Now we can replace /etc/apt/sources.list with the following.

# Local mirror of debian
deb file:/var/debmirror/debian/ woody main non-free contrib
deb file:/var/debmirror/debian/ woody-proposed-updates main non-free contrib
deb file:/var/debmirror/debian-non-US woody/non-US main contrib non-free
deb file:/var/debmirror/debian-non-US woody-proposed-updates/non-US main contrib non-free
deb file:/var/debmirror/steamengine-custom ./

# Source (Woody and Testing)
deb-src ftp://mirror.direct.ca/pub/linux/debian/ woody main non-free contrib
deb-src http://non-us.debian.org/debian-non-US woody/non-US main contrib non-free
deb-src ftp://mirror.direct.ca/pub/linux/debian/ testing main non-free contrib
deb-src http://non-us.debian.org/debian-non-US testing/non-US  main contrib non-free

# Security
deb http://security.debian.org/ woody/updates main contrib non-free

There are certain packages which we need to get from the testing version of Debian, usually because the version from the stable branch lacks some feature which we require.

You can find detailed instructions for backporting and compiling debs from the Debian Reference Manual.

First make sure we have everything that we need

apt-get install fakeroot dpkg-dev gcc make autoconf libtool manpages-dev debhelper doc++ xsltproc docbook-xsl ldp-docbook-xsl

As an example, here are the commands to backport ifenslave.

Here are all the packages we want to backport:

With the modified apt-sources, we can install a recent 2.4 serier kernel. (It's in woody-proposed-updates.)

apt-get install kernel-image-2.4.20-2-686

Before rebooting, add the following line to /etc/modules, because this kernel doesn't auto-load the driver module for the network hardware (bummer).

3c59x

This is a little complicated, because we have to compile the kernel modules. Here's how it's done. This mostly follows the instructions in /usr/share/doc/kernel-package/README.modules.

Now that we have the kernel modules as Debian packages, we can install them.

For this particular computer (any node), add these lines to /etc/modules. (Running sensors-detect will give this result.)

# I2C adapter drivers
i2c-i801
# I2C chip drivers
adm1025

Once the modules have been loaded, the CPU temperature and other information can be determined with sensors.

This cluster uses channel-bonded networking, which combines two physically separate 100MBit Ethernet networks (two switches and two network cards per node) to double network through-put. It works well, except that network booting (BOOTP and DHCP) don't work at all over channel-bonded ethernet. Therefore, we're going to have to configure thing so we can switch between normal Ethernet for node installation, and channel-bonded ethernet for standard cluster operation.

Modify /etc/network/interfaces as shown.

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
	address 142.103.140.96
	netmask 255.255.255.0
	network 142.103.140.0
	broadcast 142.103.140.255
	gateway 142.103.140.254

iface eth1 inet static
	address 192.168.253.250
	netmask 255.255.255.0
	broadcast 192.168.253.255

auto bond0
iface bond0 inet static
    address 192.168.253.250
    netmask 255.255.255.0
    broadcast 192.168.253.255
    up ifenslave bond0 eth1
    up ifenslave bond0 eth2
    post-down ifconfig eth2 down
    post-down ifconfig eth1 down

To switch to channel-bonded networking, use these commands.

Similarly, to switch from channel-bonded to regular ethernet, run

For now leave channel-bonding off so we can install the nodes. When the nodes are installed, you can use the following commands to test through-put and verify that channel-bonding is operating correctly. (If netperf is running on the target node.)

update-cluster is tool to maintain all the config files on the master node which require a list in some format of all nodes. The Debian package should be installed already (we back-ported it). Make an /etc/update-cluster/cluster.xml file.

<?xml version ="1.0"?>
<!DOCTYPE cluster SYSTEM "/usr/share/update-cluster/cluster.dtd">
<cluster>
  <host>
    <ip>192.168.253.250</ip>
    <hostname>piston00</hostname>
    <notes>master</notes>
  </host>
  <host>
    <ip>192.168.253.1</ip>
    <hostname>piston01</hostname>
    <mac>0:4:75:8d:94:6</mac>
  </host>
  <host>
    <ip>192.168.253.2</ip>
    <hostname>piston02</hostname>
    <mac>0:4:75:8d:94:7</mac>
  </host>
  <host>
    <ip>192.168.253.3</ip>
    <hostname>piston03</hostname>
    <mac>0:4:75:8d:94:8</mac>
  </host>
  <host>
    <ip>192.168.253.4</ip>
    <hostname>piston04</hostname>
    <mac>0:4:75:8d:f1:33</mac>
  </host>
  <host>
    <ip>192.168.253.5</ip>
    <hostname>piston05</hostname>
    <mac>0:4:75:8c:e2:6f</mac>
  </host>
  <host>
    <ip>192.168.253.6</ip>
    <hostname>piston06</hostname>
    <mac>0:4:75:8d:92:69</mac>
  </host>
  <host>
    <ip>192.168.253.7</ip>
    <hostname>piston07</hostname>
    <mac>0:4:75:8d:f0:b3</mac>
  </host>
  <host>
    <ip>192.168.253.8</ip>
    <hostname>piston08</hostname>
    <mac>0:1:3:27:f:2f</mac>
  </host>
  <host>
    <ip>192.168.253.9</ip>
    <hostname>piston09</hostname>
    <mac>0:4:75:8d:f1:31</mac>
  </host>
  <host>
    <ip>192.168.253.10</ip>
    <hostname>piston10</hostname>
    <mac>0:1:3:34:5a:72</mac>
  </host>
  <host>
    <ip>192.168.253.11</ip>
    <hostname>piston11</hostname>
    <mac>0:4:75:8d:f1:3</mac>
  </host>
  <host>
    <ip>192.168.253.12</ip>
    <hostname>piston12</hostname>
    <mac>0:1:3:24:25:f5</mac>
  </host>
  <host>
    <ip>192.168.253.13</ip>
    <hostname>piston13</hostname>
    <mac>0:1:3:d6:99:6f</mac>
  </host>
  <host>
    <ip>192.168.253.14</ip>
    <hostname>piston14</hostname>
    <mac>0:4:75:8d:f1:30</mac>
  </host>
  <host>
    <ip>192.168.253.15</ip>
    <hostname>piston15</hostname>
    <mac>0:4:75:8d:92:f5</mac>
  </host>
</cluster>

We don't actually want update-cluster to change the /etc/hosts file, so delete the file /usr/lib/update-cluster/hosts.updatelist.

I have found the C3 Cluster Command and Control tools to be very useful. There is no debian package, but you can download and install the sources (all in python, so no compilation required). They will install into /opt/c3-3.

A minor change must be made. Change the first line of all the executable files to

#!/usr/bin/env python

Make the following changes to /etc/profile.

PATH="/opt/c3-3:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games"
export C3_RSH=rsh

And to /root/.bashrc :

PATH=/opt/c3-3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11
export C3_RSH=rsh

We can use cpush to update the password-related files if we make a file /etc/cpush/passwd .

/etc/passwd
/etc/group
/etc/shadow

Now we can update the entire cluster after adding users on the master node with

Install matlab from the CD into /usr/nfs/matlab6.5.

Install Lahey Fortran from the CD in the usual way.

Now create links to the executables.

Add the following to the end of /etc/ld.so.conf.

/usr/local/lf9560/lib

Add the following to /etc/manpath.conf.

MANDATORY_MANPATH			/usr/local/lf9560/manuals/man/lf95

We need to recompile the debian MPI packages to get it to work for Fortran 90/95 programs. We will use the Lahey fortran compiler.

Edit /root/debs/mpich-1.2.5/debian/rules and change the line which sets CONFIGURE_OPTIONS.

CONFIGURE_OPTIONS= -rsh=/usr/bin/rsh --with-romio=-file_system=nfs --enable-f90modules -fc=lf95 -f90=lf95

Now build and then install mpich

We can now compile and install some other things which depend on mpi.

   INTFACE = -DAdd_
   F77            = lf95
   F77NO_OPTFLAGS = 
   F77FLAGS       = $(F77NO_OPTFLAGS) 

#build: build-lam build-mpich
build: build-mpich

#install: install-mpich install-lam
install: install-mpich

#binary-arch: binary-arch-lam binary-arch-mpich
binary-arch: binary-arch-mpich

F77           = lf95
NOOPT         = 
F77FLAGS      = $(NOOPT)
CDEFS         = -DAdd_ -DNO_IEEE $(USEMPI)
BLASLIB       = -llapack -lblas -lf2c

#build: build-lam build-mpich build-pvm
build: build-mpich

#install: install-dirs install-mpich install-lam install-pvm
install: install-dirs install-mpich

# the testing binaries
#       echo *** building static testing binaries for mpich ***
#       BASEDIR=$(topdir) MPI=mpich BUILD=shared make exe
#       set -e ;\
#       cd TESTING ;\
#       for i in $$(find -name 'x*'  -maxdepth 1 ! -name 'x*-lam'); do \
#         mv $$i $$i-mpich ;\
#       done

#       install TESTING/x*-mpich \
#         `pwd`/debian/scalapack-mpich-test/usr/lib/scalapack

TO DO: compile and document the following.

Java is necessary for the maui scheduler. Get the Debian java packages (j2re1.3, j2sdj1.3 and j2se-common) from here and put them in /var/debmirror/steamengine-custom. Run

Maui needs a couple of other java bits.

Add to /etc/profile:

PATH="/opt/c3-3:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/local/javacc-3.0/bin"

Add to /root/.bashrc:

export PATH=/opt/c3-3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/local/javacc-3.0/bin

Maui requires the mysql database.

Of course, I'm not going to put the real password in these docs.

Create /root/.my.cnf.

[mysql]
user            = root
password        = new-password

[mysqladmin]
user            = root
password        = new-password 

Edit /etc/mysql/my.cnf, change the following lines as shown.

set-variable    = max_allowed_packet=16M
#skip-networking

/etc/init.d/mysql restart

Before we install, we create a mauid user. In /etc/passwd

mauid:x:499:499:mauid:/tmp:/bin/false

In /etc/group

mauid:x:499:

Make sure the mail for this user goes to root. Add to /etc/aliases

mauid: root

Now get and install maui.

Now edit the /usr/nfs/maui/share/maui.properties file. I show here only the important or changed settings (from the default).

maui.name=Steam Engine
maui.email=root@localhost
maui.host=piston00
#wiki.udp.multicast=true
wiki.tcp.class=unm.maui.wikid.DaemonTCPListener
wiki.tcp.port=31115
wiki.prolog.remote=/usr/nfs/maui/share/prolog-node.sh
wiki.epilog.remote=/usr/nfs/maui/share/epilog-node.sh
wiki.slots=1
worker.hosts.check=false

Now run

We're going to create an epilog script /usr/nfs/maui/share/epilog-node.sh which cleans up after a user's job finishes.

#!/bin/sh
set -o xtrace
if /usr/bin/test ! -z "$MAUI_JOB_USER" ; then
	if /usr/bin/test x"$MAUI_JOB_USER" != 'xroot' ; then

		# Kill straggling user's process.
		#
		for i in `ps auxwww | grep "^$MAUI_JOB_USER " | grep -v grep | awk '{print $2}'` ; do
			if ps -p $i >/dev/null 2>&1 ; then
				kill $i 2>&1
			fi
		done
		sleep 2
		for i in `ps auxwww | grep "^$MAUI_JOB_USER " | grep -v grep | awk '{print $2}'` ; do
			if ps -p $i >/dev/null 2>&1 ; then
				kill -9 $i 2>&1
			fi
		done
		sleep 2

                # Aggressively delete everything in scratch.
		rm -rf /scratch/* >/dev/null 2>&1

	fi
fi

We need a start-up script /etc/init.d/maui-control. (Make it executable.)

#!/bin/sh

# description: Maui Scheduler server daemon
 
#
# Script to start maui scheduler and remote daemons
#

case "$1" in
	'start')
		# Start MauiME
		echo -n 'Starting Maui Scheduler: '
		# scheduler needs to run at higher priority than most
		# processes
		nice -n -5 /usr/nfs/maui/bin/mauictl start > /dev/null 2>&1
		if [ $? -eq 0 ] ; then
			echo 'done'
		else
			echo 'ERROR!'
		fi
		;;

	'stop')
		# Stop MauiME
		echo -n 'Stopping Maui Scheduler: '
		/usr/nfs/maui/bin/mauictl stop > /dev/null 2>&1
		if [ $? -eq 0 ] ; then
			echo 'done'
		else
			echo 'ERROR!'
		fi
		;;

	*)
		# usage
		echo "usage: $0 start|stop"
		exit 1
		;;
esac

Configure maui to start at boot-time.

update-rc.d maui-control defaults 98 01

We have to make some small changes to /usr/nfs/maui/bin/runmpi_eth. Modify this line

mpirun="/usr/bin/mpirun"

Also change ssh commands to rsh.

Now we put maui on the standard path. Add to /etc/profile:

PATH="/usr/nfs/maui/bin:/opt/c3-3:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/usr/local/javacc-3.0/bin"

Add to /root/.bashrc:

export PATH=/usr/nfs/maui/bin:/opt/c3-3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/local/javacc-3.0/bin

Add man path directory to /etc/manpath.config:

MANPATH_MAP   /usr/nfs/maui/bin       /usr/nfs/maui/man