3. Node installation with FAI (Fully Automatic Installation)

We're going to install the nodes using the automated installer FAI. Once FAI is configured, the nodes can eb installed, or reinstalled, as needed with practically no manual intervention, and no manual node configuration at all.

FAI has an execellent manual, which you can find on the FAI homepage.

3.1. Download and install FAI

As usual, the first step is to make sure that we have all the debian packages which we require.

apt-get install nfs-kernel-server debootstrap netboot bootp tftpd

Now get the FAI packages and install.

wget http://www.informatik.uni-koeln.de/fai/download/fai-kernels_1.5.3_i386.deb http://www.informatik.uni-koeln.de/fai/download/fai_2.4.1_all.deb

dpkg -i fai-kernels_1.5.3_i386.deb fai_2.4.1_all.deb

Here's some more stuff which we'll need to support our 3-com network cards.

wget http://www.informatik.uni-koeln.de/fai/download/imggen-1.00

mv imggen-1.00 /usr/local/bin

chmod +x /usr/local/bin/imggen-1.00

ln -s imggen-1.00 /usr/local/bin/imggen

3.2. Basic configuration

Here's the /etc/fai/fai.conf file which I used.

installserver=piston00
ftpserver=piston00
debdist=woody
FAI_DEBOOTSTRAP="$debdist file:/var/debmirror/debian"
FAI_DEBOOTSTRAP_OPTS="--arch i386 --exclude=pcmcia-cs,ppp,pppconfig,pppoe,pppoeconf,dhcp-client"
FAI_DEBMIRROR=$installserver:/var/debmirror
MNTPOINT=/var/debmirror
FAI_SOURCES_LIST="deb file:$MNTPOINT/debian $debdist main contrib non-free
deb file:$MNTPOINT/debian/ $debdist-proposed-updates main contrib non-free
deb file:$MNTPOINT/debian-non-US $debdist/non-US main contrib non-free
deb file:$MNTPOINT/debian-security/ $debdist/updates main contrib non-free
deb file:$MNTPOINT/steamengine-custom ./"
#FAI_LOCAL_REPOSITORY="deb file:/fai/files packages/"
NFSROOT_PACKAGES="expect"
# pw is: fai
FAI_ROOTPW="56hNVqht51tzc"
KERNELPACKAGE=/usr/lib/fai/kernel/kernel-image-2.4.20_fai1_i386.deb
NFSROOT_ETC_HOSTS="192.168.253.250 piston00"
SERVERINTERFACE=eth1
FAI_REMOTESH=rsh
FAI_REMOTECP=rcp
FAI_CONFIGDIR=/var/fai/config
FAI_LOCATION=$installserver:$FAI_CONFIGDIR
FAI_BOOT="bootp"
NFSROOT=/usr/lib/fai/nfsroot
FAI=/fai
OS_TYPE=`uname -s |  tr /A-Z/ /a-z/`

Now run fai-setup.

I don't quite like everything it does, so after change /etc/exports to the following and restart the nfs server.

# /etc/exports
/var/debmirror 192.168.253.0/255.255.255.0(ro,no_root_squash) 142.103.140.96(ro,no_root_squash)
/var/fai/config 192.168.253.0/255.255.255.0(ro)
/usr/lib/fai/nfsroot 192.168.253.0/255.255.255.0(ro,no_root_squash)
/usr/nfs 192.168.253.0/255.255.255.0(ro,no_root_squash)
/home 192.168.253.0/255.255.255.0(rw,no_root_squash)

3.3. BIOS configuration on the nodes

To enable network booting on the nodes, in the BOIS select the first boot item as network card, second as hard drive. Then hit CTRL-ALT-B when prompted, and select method TCP/IP, protocol BOOTP, on failure boot second device.

3.4. bootp

We need to enable the bootp service, so uncomment this line from /etc/inetd.conf, then restart inetd with /etc/init.d/inetd restart.

bootps          dgram   udp     wait    root    /usr/sbin/bootpd        bootpd -i -t 120

Make links for particular nodes to particular kernels (all the same in fact).

cd /boot/fai

ln -s installimage_3com piston01

ln -s installimage_3com piston02

and so on.

I collected all the network card MAC addresses by running

tcpdump -i eth1 -qte broadcast and port bootpc > n1mac

and then turning on all the nodes sequentially.

Here's the /etc/bootptab.

.faiglobal:\
	:ms=1024:\
	:hd=/boot/fai:\
	:hn:bs=auto:\
	:rp=/usr/lib/fai/nfsroot:

.failocal:\
	:tc=.faiglobal:\
	:sa=piston00:\
	:ts=piston00:\
	:gw=192.168.253.250:\
	:sm=255.255.255.0:\
	:dn=boiler:\
	:T172="verbose syslogd createvt":\
	:nt=piston00:

# now one entry for each install client
# Comment these out if you *don't* want to reinstall them
#piston01:ha=0x0004758D9406:ip=192.168.253.1:bf=piston01:tc=.failocal:
#piston02:ha=0x0004758D9407:ip=192.168.253.2:bf=piston02:tc=.failocal:
#piston03:ha=0x0004758D9408:ip=192.168.253.3:bf=piston03:tc=.failocal:
#piston04:ha=0x0004758DF133:ip=192.168.253.4:bf=piston04:tc=.failocal:
#piston05:ha=0x0004758CE26F:ip=192.168.253.5:bf=piston05:tc=.failocal:
#piston06:ha=0x0004758D9269:ip=192.168.253.6:bf=piston06:tc=.failocal:
#piston07:ha=0x0004758DF0B3:ip=192.168.253.7:bf=piston07:tc=.failocal:
#piston08:ha=0x000103270F2F:ip=192.168.253.8:bf=piston08:tc=.failocal:
#piston09:ha=0x0004758DF131:ip=192.168.253.9:bf=piston09:tc=.failocal:
#piston10:ha=0x000103345A72:ip=192.168.253.10:bf=piston10:tc=.failocal:
#piston11:ha=0x0004758DF103:ip=192.168.253.11:bf=piston11:tc=.failocal:
#piston12:ha=0x0001032425F5:ip=192.168.253.12:bf=piston12:tc=.failocal:
#piston13:ha=0x000103D6996F:ip=192.168.253.13:bf=piston13:tc=.failocal:
#piston14:ha=0x0004758DF130:ip=192.168.253.14:bf=piston14:tc=.failocal:
#piston15:ha=0x0004758D92F5:ip=192.168.253.15:bf=piston15:tc=.failocal:

3.5. Target configuration

Copy the whole directory /usr/share/fai/templates/ to /var/fai/config. By editing files in here, we can customize our installation. I advise that you read the FAI manual, because it's rather complex. I've modified the DEFAULT target, since we're going to use this installation of FAI onyl to install the cluster nodes.

3.5.1. /var/fai/config/class

I have only the following files in this directory.

/var/fai/config/class/01alias


#! /bin/sh

# echo architecture in upper case
uname -s | tr /a-z/ /A-Z/
[ -x "`which dpkg`" ] && dpkg --print-installation-architecture | tr /a-z/ /A-Z/

/var/fai/config/class/06hwdetect


#! /bin/sh

# load ide kernel modules
modprobe -a ide-probe-mod ide-disk ide-cd

# probe for scsi devices and set classes

for d in `discover --module scsi`; do
        modprobe $d  >> $moduleslog 2>&1
        case "$d" in
	    aic7xxx)
              # define a class for adaptec special work to do
	      newclasses="$newclasses ADAPTEC" ;;
        esac
done

modprobe -a sd_mod sr_mod
# are any SCSI devices attached?
if grep -q "^Attached devices: none" /proc/scsi/scsi; then
    :
else
    newclasses="SCSI $newclasses"
    # we want also get the filename in front of each line. So do not use cat, use grep
    [ "$verbose" ] && cat /proc/scsi/scsi
    [ "$debug" ] && grep -H . /proc/scsi/*/*
fi

set_disk_info  # calculate number of available disks
save_dmesg # save new boot messages (from loading modules)

/var/fai/config/class/11modules.source


#! /bin/sh

# load kernel modules
# messages of loading modules are written to syslogd and read with dmesg.
# for all classes which are defined before this script is called, look for
# a $class.mod file and execute it.

conffiles="$classes $HOSTNAME"

for cfile in $conffiles ; do
    if [ -f "${cfile}.mod" ]; then
	.  ${cfile}.mod  >> $moduleslog 2>&1
    fi
done

unset conffiles cfile

/var/fai/config/class/DEFAULT.var


UTC=yes
time_zone=Canada/Pacific
kernelversion=2.4.20-2
kernelimage="kernel-image-$kernelversion-686"
rootpw="(removed for security)"
hserver=piston00
bserver=piston00

/var/fai/config/class/LAST.var


#! /bin/bash

FAI_ACTION=install
#FAI_ACTION=sysinfo
[ "$debug" ] && echo "FAI_ACTION now defined as $FAI_ACTION"

/var/fai/config/class/default


BASE
NET
NETWORK
BOOT
MBR
LILO
NOTCPD
NOSECURETTY
NTP
NONIS
HOME_CLIENT
SCRATCH
USR_NFS_MOUNT
USR_LOCAL_RSYNC

/var/fai/config/class/piston01.var


moduleslist="3c59x bonding i2c-i801 adm1025"

/var/fai/config/class/piston02.var, piston02.var,... piston15.var


moduleslist="3c59x e100 bonding i2c-i801 adm1025"

Note

piston01 has two 3com cards, the other nodes have a 3com and an intel card, hence the different kernel modules required.

3.5.2. /var/fai/config/disk-config

I have only one file in this directory.

/var/fai/config/disk_config/DEFAULT


# <type> <mountpoint> <size in mb> [mount options]     [;extra options]

disk_config hda
primary  swap        500      sw                   
primary  /           1000-    rw,errors=remount-ro ;-j ext3

3.5.3. /var/fai/config/package_config

I have only one file in this directory.

/var/fai/config/package_config/DEFAULT



PACKAGES install
hdparm debconf bootpc parted cron
rstat-client rstatd rusers rusersd rsh-client rsh-server
pciutils sysutils time strace libpam-cracklib
tcsh tcsh-i18n file less cfengine rsync psmisc jove
linuxlogo

PACKAGES install
ifenslave
ntp-simple ntpdate
nfs-common
emacs21 unp gawk python2.2 elvis vim
blas-dev blas-test
atlas2-sse2 atlas2-sse2 atlas2-headers
python python-numeric python-numeric-ext
python2.2 python2.2-numeric python2.2-numeric-ext
python-dev python2.2-dev
netcdf-bin netcdfg3 netcdfg python-netcdf
libgsl0 libgsl0 gsl-bin
jmon
mpich
scalapack-mpich-dev
petsc2.1.3
sfftw2 sfftw-dev p4fftwgel2 p4fftwgel-dev
blacs-mpich-test blacs-test-common
f2c
lm-sensors-2.4.20-2-686 i2c-2.4.20-2-686 lm-sensors sensord
libstdc++2.10 gcc-3.0

PACKAGES taskinst
c-dev

As of April 26, 2005, this file has changed, the diff is below.


14c14
< blas-dev blas-test
---
> blas-dev
29c29
< libstdc++2.10 gcc-3.0
---
> gcc-3.0
32a33,35
> 
> PACKAGES install
> bc gri iproute libmpich-p4mpd1.0 mpich-mpd

3.5.4. /var/fai/config/scripts

For the following files, I mostly show the diffs from the original templates provided with FAI.

/var/fai/config/scripts/DEFAULT/S01 (diff)


68,75c68
<     rsync $vopt --delete -a $bserver:/usr/local/ $target/usr/
< }
< 
< ifclass OPT_RSYNC && {
<     echo "Syncing /opt via $bserver"
<     vopt=
<     [ "$verbose" ] && vopt=-v
<     rsync $vopt --delete -a $bserver:/opt $target/
---
>     rsync $vopt --delete -a $LOGUSER@$bserver:/usr/local/ $target/usr/local/
79c72
< #chmod a+rw $target/dev/fd*
---
> chmod a+rw $target/dev/fd*
81c74
< #chmod a+r $target/dev/sr* 
---
> chmod a+r $target/dev/sr* 
115c108
< #ifclass LINUX && fcopy /etc/exim/exim.conf /etc/aliases
---
> ifclass LINUX && fcopy /etc/exim/exim.conf /etc/aliases
157,158d149
< 
< ifclass USR_NFS_MOUNT && mkdir $target/usr/nfs

As of April 26, 2005, this file has changed. The diff is below (this was needed to keep the script working. It seems that some command syntax has changed over the years).


68c68,69
<     rsync $vopt --delete -a $bserver:/usr/local/ $target/usr
---
>     mkdir -p $target/usr/local
>     rsync $vopt --delete -a $bserver:/usr/local/ $target/usr/local/

/var/fai/config/scripts/DEFAULT/S20 (diff)


14,15c14,15
< #        ${target}/dev include=fd* mode=666   action=fixall r=1
< #        ${target}/dev include=sr* mode=444   action=fixall r=1
---
>         ${target}/dev include=fd* mode=666   action=fixall r=1
>         ${target}/dev include=sr* mode=444   action=fixall r=1
103,104c103
< #	  AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid 0 0"
< 	  AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid,noac,rsize=8192,wsize=8192 0 0"
---
> 	  AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid 0 0"
119,121c118,121
<    USR_NFS_MOUNT::
< 	{ ${target}/etc/fstab
< 	  AppendIfNoSuchLine "${bserver}:/usr/nfs /usr/nfs nfs ro 0 0"
---
>    SCRATCH::
> 	{ ${target}/etc/exports
> 	  AutoCreate
> 	  AppendIfNoSuchLine "/scratch${tab}${tab} @faiclients(rw,no_root_squash)"
123,128d122
< 
< #   SCRATCH::
< #	{ ${target}/etc/exports
< #	  AutoCreate
< #	  AppendIfNoSuchLine "/scratch${tab}${tab} @faiclients(rw,no_root_squash)"
< #	}

/var/fai/config/scripts/DEFAULT/S50


#!/bin/sh

echo "Duplicating files from master node"

DUPFILES="/etc/passwd /etc/group /etc/shadow /etc/hosts /etc/networks /etc/hosts.equiv /root/.rhosts /etc/mpd.conf"
for F in $DUPFILES
do
  rcp piston00:$F ${target}${F}
  # might have problems with too many connections; slow down connection rate
  sleep 1
done
DUPDIR="/etc/maui"
for D in $DUPDIR
do
  rcp -r piston00:$D ${target}/etc
  # might have problems with too many connections; slow down connection rate
  sleep 1
done
chroot $target chown mauid /etc/maui/maui.key

echo "Installing start-up scripts for maui"

# Install start-up scripts for maui
fcopy /etc/init.d/maui-node
chroot $target /usr/sbin/update-rc.d maui-node defaults 98 01

# Create log directory with permissions for user mauid
mkdir $target/var/log/maui
chown -f 499.499 $target/var/log/maui

echo "/usr/local/lf9560/lib" >> $target/etc/ld.so.conf
chroot $target ldconfig

echo "installing java"
chroot $target "yes 'yes' | apt-get install j2re1.3 > /fai/tmp/java-install.log 2>&1"

echo "Making matlab links"
ln -sf /usr/nfs/matlab6.5/bin/matlab $target/usr/local/bin/matlab
ln -sf /usr/nfs/matlab6.5/bin/matlab $target/usr/local/bin/matlab6.5

As of April 26, 2005, this file has changed. The diff is below (this was needed to keep the script working. It seems that some command syntax has changed over the years).


47c47
< chroot $target "yes 'yes' | apt-get install j2re1.3 > /fai/tmp/java-install.log 2>&1"
---
> chroot $target yes 'yes' | chroot $target apt-get install j2re1.3 > /tmp/fai/java-install.log 2>&1
51a52,55
> 
> # We want to hold these packages (added Apr. 26, 2005, by Scott Webster)
> chroot $target echo "scalapack-mpich-dev hold" | chroot $target dpkg --set-selections
> chroot $target echo "scalapack1-mpich hold" | chroot $target dpkg --set-selections

/var/fai/config/scripts/LAST (diff)


21c21
< echo "$FAI_DEBMIRROR $MNTPOINT nfs ro 0 0" >> $target/etc/fstab
---
> echo "#$FAI_DEBMIRROR $MNTPOINT nfs ro 0 0" >> $target/etc/fstab

/var/fai/config/scripts/NETWORK/S40 (diff)


8c9
< 
---
> auto eth0
10,24c11,14
<     address $IPADDR
<     netmask $NETMASK
<     broadcast $BROADCAST
<     gateway $GATEWAYS
< 
< auto bond0
< iface bond0 inet static
<     address $IPADDR
<     netmask $NETMASK
<     broadcast $BROADCAST
<     gateway $GATEWAYS
<     up ifenslave bond0 eth0
<     up ifenslave bond0 eth1
<     post-down ifconfig eth1 down
<     post-down ifconfig eth0 down
---
> address $IPADDR
> netmask $NETMASK
> broadcast $BROADCAST
> gateway $GATEWAYS
26c16
< #echo "localnet $NETWORK" > $target/etc/networks
---
> echo "localnet $NETWORK" > $target/etc/networks

3.5.5. /var/fai/config/files

/var/fai/config/files/etc/nullmailer/defaultdomain/DEFAULT


boiler

/var/fai/config/files/etc/nullmailer/defaulthost/DEFAULT


steamengine.physics.ubc.ca

/var/fai/config/files/etc/nullmailer/remotes/DEFAULT


piston00.boiler smtp

/var/fai/config/files/etc/resolv.conf/DEFAULT


search physics.ubc.ca
nameserver 192.168.253.250

/var/fai/config/files/etc/init.d/maui-node/DEFAULT


#!/bin/sh

# description: Maui Scheduler node daemon
#
# Script to start node daemons.  Copy this file to /etc/rc.d on all
# nodes.
#

case "$1" in
	'start')
		# Start Node Daemon
		echo -n 'Starting Node Daemon: '
		# Make sure daemon is not running
		/usr/nfs/maui/bin/nodectl stop > /dev/null 2>&1
		sleep 1
                # Really kill maui!
                /bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill > /dev/null 2>&1
		sleep 1
                # Really really kill maui!
                /bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill -s 9 > /dev/null 2>&1
		sleep 1
		# node daemon needs to run at higher priority than
		# most processes
		nice -n -5 /usr/nfs/maui/bin/nodectl start > /dev/null 2>&1
		if [ $? -eq 0 ] ; then
			echo 'done'
		else
			echo 'ERROR!'
		fi
		;;

	'stop')
		# Stop Node Daemon
		echo -n 'Stopping Node Daemon: '
		/usr/nfs/maui/bin/nodectl stop > /dev/null 2>&1
		sleep 1
                # Really kill maui!
                /bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill > /dev/null 2>&1
		sleep 1
                # Really really kill maui!
                /bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill -s 9 > /dev/null 2>&1
		echo 'done'
		;;

	*)
		# usage
		echo "usage: $0 start|stop"
		exit 1
		;;
esac

/var/fai/config/files/root/.bashrc


if [ "$PS1" ]; then
    export PS1='\[\033[34m\]#\[\033[0m\] '

    # If this is an xterm set the title to user@host
    case $TERM in
    xterm*|rxvt*)
        PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
        ;;
    *)
        ;;
    esac

fi

export PATH=/usr/lib/mpich-mpd/bin:/opt/c3-3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11

3.5.6. /var/fai/config/hooks

/var/fai/config/hooks/instsoft.DEFAULT


#! /bin/sh

# This file needs to exist or the installation of atlas2-sse2 barfs.
touch $target/etc/ld.so.conf

# Replace exim with nullmailer
fcopy -r /etc/nullmailer
$ROOTCMD apt-get --purge install nullmailer exim-

# set up kernel-img.conf so that the initrd kernel gets installed without prompting
# otherwise it aborts and certain packages don't get installed
# Scott Webster, Apr. 25, 2005
echo "do_initrd = Yes" >> $target/etc/kernel-img.conf
echo "do_symlinks = Yes" >> $target/etc/kernel-img.conf

3.6. Installing one or more nodes

This is described in the reinstallation section.