We're going to install the nodes using the automated installer FAI. Once FAI is configured, the nodes can eb installed, or reinstalled, as needed with practically no manual intervention, and no manual node configuration at all.
FAI has an execellent manual, which you can find on the FAI homepage.
As usual, the first step is to make sure that we have all the debian packages which we require.
apt-get install nfs-kernel-server debootstrap netboot bootp tftpd
Now get the FAI packages and install.
wget http://www.informatik.uni-koeln.de/fai/download/fai-kernels_1.5.3_i386.deb http://www.informatik.uni-koeln.de/fai/download/fai_2.4.1_all.deb
dpkg -i fai-kernels_1.5.3_i386.deb fai_2.4.1_all.deb
Here's some more stuff which we'll need to support our 3-com network cards.
wget http://www.informatik.uni-koeln.de/fai/download/imggen-1.00
mv imggen-1.00 /usr/local/bin
chmod +x /usr/local/bin/imggen-1.00
ln -s imggen-1.00 /usr/local/bin/imggen
Here's the /etc/fai/fai.conf file which I used.
installserver=piston00 ftpserver=piston00 debdist=woody FAI_DEBOOTSTRAP="$debdist file:/var/debmirror/debian" FAI_DEBOOTSTRAP_OPTS="--arch i386 --exclude=pcmcia-cs,ppp,pppconfig,pppoe,pppoeconf,dhcp-client" FAI_DEBMIRROR=$installserver:/var/debmirror MNTPOINT=/var/debmirror FAI_SOURCES_LIST="deb file:$MNTPOINT/debian $debdist main contrib non-free deb file:$MNTPOINT/debian/ $debdist-proposed-updates main contrib non-free deb file:$MNTPOINT/debian-non-US $debdist/non-US main contrib non-free deb file:$MNTPOINT/debian-security/ $debdist/updates main contrib non-free deb file:$MNTPOINT/steamengine-custom ./" #FAI_LOCAL_REPOSITORY="deb file:/fai/files packages/" NFSROOT_PACKAGES="expect" # pw is: fai FAI_ROOTPW="56hNVqht51tzc" KERNELPACKAGE=/usr/lib/fai/kernel/kernel-image-2.4.20_fai1_i386.deb NFSROOT_ETC_HOSTS="192.168.253.250 piston00" SERVERINTERFACE=eth1 FAI_REMOTESH=rsh FAI_REMOTECP=rcp FAI_CONFIGDIR=/var/fai/config FAI_LOCATION=$installserver:$FAI_CONFIGDIR FAI_BOOT="bootp" NFSROOT=/usr/lib/fai/nfsroot FAI=/fai OS_TYPE=`uname -s | tr /A-Z/ /a-z/`
Now run fai-setup.
I don't quite like everything it does, so after change /etc/exports to the following and restart the nfs server.
# /etc/exports /var/debmirror 192.168.253.0/255.255.255.0(ro,no_root_squash) 142.103.140.96(ro,no_root_squash) /var/fai/config 192.168.253.0/255.255.255.0(ro) /usr/lib/fai/nfsroot 192.168.253.0/255.255.255.0(ro,no_root_squash) /usr/nfs 192.168.253.0/255.255.255.0(ro,no_root_squash) /home 192.168.253.0/255.255.255.0(rw,no_root_squash)
To enable network booting on the nodes, in the BOIS select the first boot item as network card, second as hard drive. Then hit CTRL-ALT-B when prompted, and select method TCP/IP, protocol BOOTP, on failure boot second device.
We need to enable the bootp service, so uncomment this line from /etc/inetd.conf, then restart inetd with /etc/init.d/inetd restart.
bootps dgram udp wait root /usr/sbin/bootpd bootpd -i -t 120
Make links for particular nodes to particular kernels (all the same in fact).
cd /boot/fai
ln -s installimage_3com piston01
ln -s installimage_3com piston02
and so on.
I collected all the network card MAC addresses by running
tcpdump -i eth1 -qte broadcast and port bootpc > n1mac
and then turning on all the nodes sequentially.
Here's the /etc/bootptab.
.faiglobal:\ :ms=1024:\ :hd=/boot/fai:\ :hn:bs=auto:\ :rp=/usr/lib/fai/nfsroot: .failocal:\ :tc=.faiglobal:\ :sa=piston00:\ :ts=piston00:\ :gw=192.168.253.250:\ :sm=255.255.255.0:\ :dn=boiler:\ :T172="verbose syslogd createvt":\ :nt=piston00: # now one entry for each install client # Comment these out if you *don't* want to reinstall them #piston01:ha=0x0004758D9406:ip=192.168.253.1:bf=piston01:tc=.failocal: #piston02:ha=0x0004758D9407:ip=192.168.253.2:bf=piston02:tc=.failocal: #piston03:ha=0x0004758D9408:ip=192.168.253.3:bf=piston03:tc=.failocal: #piston04:ha=0x0004758DF133:ip=192.168.253.4:bf=piston04:tc=.failocal: #piston05:ha=0x0004758CE26F:ip=192.168.253.5:bf=piston05:tc=.failocal: #piston06:ha=0x0004758D9269:ip=192.168.253.6:bf=piston06:tc=.failocal: #piston07:ha=0x0004758DF0B3:ip=192.168.253.7:bf=piston07:tc=.failocal: #piston08:ha=0x000103270F2F:ip=192.168.253.8:bf=piston08:tc=.failocal: #piston09:ha=0x0004758DF131:ip=192.168.253.9:bf=piston09:tc=.failocal: #piston10:ha=0x000103345A72:ip=192.168.253.10:bf=piston10:tc=.failocal: #piston11:ha=0x0004758DF103:ip=192.168.253.11:bf=piston11:tc=.failocal: #piston12:ha=0x0001032425F5:ip=192.168.253.12:bf=piston12:tc=.failocal: #piston13:ha=0x000103D6996F:ip=192.168.253.13:bf=piston13:tc=.failocal: #piston14:ha=0x0004758DF130:ip=192.168.253.14:bf=piston14:tc=.failocal: #piston15:ha=0x0004758D92F5:ip=192.168.253.15:bf=piston15:tc=.failocal:
Copy the whole directory /usr/share/fai/templates/ to /var/fai/config. By editing files in here, we can customize our installation. I advise that you read the FAI manual, because it's rather complex. I've modified the DEFAULT target, since we're going to use this installation of FAI onyl to install the cluster nodes.
I have only the following files in this directory.
/var/fai/config/class/01alias
#! /bin/sh # echo architecture in upper case uname -s | tr /a-z/ /A-Z/ [ -x "`which dpkg`" ] && dpkg --print-installation-architecture | tr /a-z/ /A-Z/
/var/fai/config/class/06hwdetect
#! /bin/sh
# load ide kernel modules
modprobe -a ide-probe-mod ide-disk ide-cd
# probe for scsi devices and set classes
for d in `discover --module scsi`; do
modprobe $d >> $moduleslog 2>&1
case "$d" in
aic7xxx)
# define a class for adaptec special work to do
newclasses="$newclasses ADAPTEC" ;;
esac
done
modprobe -a sd_mod sr_mod
# are any SCSI devices attached?
if grep -q "^Attached devices: none" /proc/scsi/scsi; then
:
else
newclasses="SCSI $newclasses"
# we want also get the filename in front of each line. So do not use cat, use grep
[ "$verbose" ] && cat /proc/scsi/scsi
[ "$debug" ] && grep -H . /proc/scsi/*/*
fi
set_disk_info # calculate number of available disks
save_dmesg # save new boot messages (from loading modules)
/var/fai/config/class/11modules.source
#! /bin/sh
# load kernel modules
# messages of loading modules are written to syslogd and read with dmesg.
# for all classes which are defined before this script is called, look for
# a $class.mod file and execute it.
conffiles="$classes $HOSTNAME"
for cfile in $conffiles ; do
if [ -f "${cfile}.mod" ]; then
. ${cfile}.mod >> $moduleslog 2>&1
fi
done
unset conffiles cfile
/var/fai/config/class/DEFAULT.var
UTC=yes time_zone=Canada/Pacific kernelversion=2.4.20-2 kernelimage="kernel-image-$kernelversion-686" rootpw="(removed for security)" hserver=piston00 bserver=piston00
/var/fai/config/class/LAST.var
#! /bin/bash FAI_ACTION=install #FAI_ACTION=sysinfo [ "$debug" ] && echo "FAI_ACTION now defined as $FAI_ACTION"
/var/fai/config/class/default
BASE NET NETWORK BOOT MBR LILO NOTCPD NOSECURETTY NTP NONIS HOME_CLIENT SCRATCH USR_NFS_MOUNT USR_LOCAL_RSYNC
/var/fai/config/class/piston01.var
moduleslist="3c59x bonding i2c-i801 adm1025"
/var/fai/config/class/piston02.var, piston02.var,... piston15.var
moduleslist="3c59x e100 bonding i2c-i801 adm1025"
piston01 has two 3com cards, the other nodes have a 3com and an intel card, hence the different kernel modules required.
I have only one file in this directory.
/var/fai/config/disk_config/DEFAULT
# <type> <mountpoint> <size in mb> [mount options] [;extra options] disk_config hda primary swap 500 sw primary / 1000- rw,errors=remount-ro ;-j ext3
I have only one file in this directory.
/var/fai/config/package_config/DEFAULT
PACKAGES install hdparm debconf bootpc parted cron rstat-client rstatd rusers rusersd rsh-client rsh-server pciutils sysutils time strace libpam-cracklib tcsh tcsh-i18n file less cfengine rsync psmisc jove linuxlogo PACKAGES install ifenslave ntp-simple ntpdate nfs-common emacs21 unp gawk python2.2 elvis vim blas-dev blas-test atlas2-sse2 atlas2-sse2 atlas2-headers python python-numeric python-numeric-ext python2.2 python2.2-numeric python2.2-numeric-ext python-dev python2.2-dev netcdf-bin netcdfg3 netcdfg python-netcdf libgsl0 libgsl0 gsl-bin jmon mpich scalapack-mpich-dev petsc2.1.3 sfftw2 sfftw-dev p4fftwgel2 p4fftwgel-dev blacs-mpich-test blacs-test-common f2c lm-sensors-2.4.20-2-686 i2c-2.4.20-2-686 lm-sensors sensord libstdc++2.10 gcc-3.0 PACKAGES taskinst c-dev
As of April 26, 2005, this file has changed, the diff is below.
14c14 < blas-dev blas-test --- > blas-dev 29c29 < libstdc++2.10 gcc-3.0 --- > gcc-3.0 32a33,35 > > PACKAGES install > bc gri iproute libmpich-p4mpd1.0 mpich-mpd
For the following files, I mostly show the diffs from the original templates provided with FAI.
/var/fai/config/scripts/DEFAULT/S01 (diff)
68,75c68
< rsync $vopt --delete -a $bserver:/usr/local/ $target/usr/
< }
<
< ifclass OPT_RSYNC && {
< echo "Syncing /opt via $bserver"
< vopt=
< [ "$verbose" ] && vopt=-v
< rsync $vopt --delete -a $bserver:/opt $target/
---
> rsync $vopt --delete -a $LOGUSER@$bserver:/usr/local/ $target/usr/local/
79c72
< #chmod a+rw $target/dev/fd*
---
> chmod a+rw $target/dev/fd*
81c74
< #chmod a+r $target/dev/sr*
---
> chmod a+r $target/dev/sr*
115c108
< #ifclass LINUX && fcopy /etc/exim/exim.conf /etc/aliases
---
> ifclass LINUX && fcopy /etc/exim/exim.conf /etc/aliases
157,158d149
<
< ifclass USR_NFS_MOUNT && mkdir $target/usr/nfs
As of April 26, 2005, this file has changed. The diff is below (this was needed to keep the script working. It seems that some command syntax has changed over the years).
68c68,69 < rsync $vopt --delete -a $bserver:/usr/local/ $target/usr --- > mkdir -p $target/usr/local > rsync $vopt --delete -a $bserver:/usr/local/ $target/usr/local/
/var/fai/config/scripts/DEFAULT/S20 (diff)
14,15c14,15
< # ${target}/dev include=fd* mode=666 action=fixall r=1
< # ${target}/dev include=sr* mode=444 action=fixall r=1
---
> ${target}/dev include=fd* mode=666 action=fixall r=1
> ${target}/dev include=sr* mode=444 action=fixall r=1
103,104c103
< # AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid 0 0"
< AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid,noac,rsize=8192,wsize=8192 0 0"
---
> AppendIfNoSuchLine "${hserver}:/home /home nfs rw,nosuid 0 0"
119,121c118,121
< USR_NFS_MOUNT::
< { ${target}/etc/fstab
< AppendIfNoSuchLine "${bserver}:/usr/nfs /usr/nfs nfs ro 0 0"
---
> SCRATCH::
> { ${target}/etc/exports
> AutoCreate
> AppendIfNoSuchLine "/scratch${tab}${tab} @faiclients(rw,no_root_squash)"
123,128d122
<
< # SCRATCH::
< # { ${target}/etc/exports
< # AutoCreate
< # AppendIfNoSuchLine "/scratch${tab}${tab} @faiclients(rw,no_root_squash)"
< # }
/var/fai/config/scripts/DEFAULT/S50
#!/bin/sh
echo "Duplicating files from master node"
DUPFILES="/etc/passwd /etc/group /etc/shadow /etc/hosts /etc/networks /etc/hosts.equiv /root/.rhosts /etc/mpd.conf"
for F in $DUPFILES
do
rcp piston00:$F ${target}${F}
# might have problems with too many connections; slow down connection rate
sleep 1
done
DUPDIR="/etc/maui"
for D in $DUPDIR
do
rcp -r piston00:$D ${target}/etc
# might have problems with too many connections; slow down connection rate
sleep 1
done
chroot $target chown mauid /etc/maui/maui.key
echo "Installing start-up scripts for maui"
# Install start-up scripts for maui
fcopy /etc/init.d/maui-node
chroot $target /usr/sbin/update-rc.d maui-node defaults 98 01
# Create log directory with permissions for user mauid
mkdir $target/var/log/maui
chown -f 499.499 $target/var/log/maui
echo "/usr/local/lf9560/lib" >> $target/etc/ld.so.conf
chroot $target ldconfig
echo "installing java"
chroot $target "yes 'yes' | apt-get install j2re1.3 > /fai/tmp/java-install.log 2>&1"
echo "Making matlab links"
ln -sf /usr/nfs/matlab6.5/bin/matlab $target/usr/local/bin/matlab
ln -sf /usr/nfs/matlab6.5/bin/matlab $target/usr/local/bin/matlab6.5
As of April 26, 2005, this file has changed. The diff is below (this was needed to keep the script working. It seems that some command syntax has changed over the years).
47c47 < chroot $target "yes 'yes' | apt-get install j2re1.3 > /fai/tmp/java-install.log 2>&1" --- > chroot $target yes 'yes' | chroot $target apt-get install j2re1.3 > /tmp/fai/java-install.log 2>&1 51a52,55 > > # We want to hold these packages (added Apr. 26, 2005, by Scott Webster) > chroot $target echo "scalapack-mpich-dev hold" | chroot $target dpkg --set-selections > chroot $target echo "scalapack1-mpich hold" | chroot $target dpkg --set-selections
/var/fai/config/scripts/LAST (diff)
21c21 < echo "$FAI_DEBMIRROR $MNTPOINT nfs ro 0 0" >> $target/etc/fstab --- > echo "#$FAI_DEBMIRROR $MNTPOINT nfs ro 0 0" >> $target/etc/fstab
/var/fai/config/scripts/NETWORK/S40 (diff)
8c9 < --- > auto eth0 10,24c11,14 < address $IPADDR < netmask $NETMASK < broadcast $BROADCAST < gateway $GATEWAYS < < auto bond0 < iface bond0 inet static < address $IPADDR < netmask $NETMASK < broadcast $BROADCAST < gateway $GATEWAYS < up ifenslave bond0 eth0 < up ifenslave bond0 eth1 < post-down ifconfig eth1 down < post-down ifconfig eth0 down --- > address $IPADDR > netmask $NETMASK > broadcast $BROADCAST > gateway $GATEWAYS 26c16 < #echo "localnet $NETWORK" > $target/etc/networks --- > echo "localnet $NETWORK" > $target/etc/networks
/var/fai/config/files/etc/nullmailer/defaultdomain/DEFAULT
boiler
/var/fai/config/files/etc/nullmailer/defaulthost/DEFAULT
steamengine.physics.ubc.ca
/var/fai/config/files/etc/nullmailer/remotes/DEFAULT
piston00.boiler smtp
/var/fai/config/files/etc/resolv.conf/DEFAULT
search physics.ubc.ca nameserver 192.168.253.250
/var/fai/config/files/etc/init.d/maui-node/DEFAULT
#!/bin/sh
# description: Maui Scheduler node daemon
#
# Script to start node daemons. Copy this file to /etc/rc.d on all
# nodes.
#
case "$1" in
'start')
# Start Node Daemon
echo -n 'Starting Node Daemon: '
# Make sure daemon is not running
/usr/nfs/maui/bin/nodectl stop > /dev/null 2>&1
sleep 1
# Really kill maui!
/bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill > /dev/null 2>&1
sleep 1
# Really really kill maui!
/bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill -s 9 > /dev/null 2>&1
sleep 1
# node daemon needs to run at higher priority than
# most processes
nice -n -5 /usr/nfs/maui/bin/nodectl start > /dev/null 2>&1
if [ $? -eq 0 ] ; then
echo 'done'
else
echo 'ERROR!'
fi
;;
'stop')
# Stop Node Daemon
echo -n 'Stopping Node Daemon: '
/usr/nfs/maui/bin/nodectl stop > /dev/null 2>&1
sleep 1
# Really kill maui!
/bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill > /dev/null 2>&1
sleep 1
# Really really kill maui!
/bin/ps -u mauid -o pid --no-headers | /usr/bin/xargs /bin/kill -s 9 > /dev/null 2>&1
echo 'done'
;;
*)
# usage
echo "usage: $0 start|stop"
exit 1
;;
esac
/var/fai/config/files/root/.bashrc
if [ "$PS1" ]; then
export PS1='\[\033[34m\]#\[\033[0m\] '
# If this is an xterm set the title to user@host
case $TERM in
xterm*|rxvt*)
PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
;;
*)
;;
esac
fi
export PATH=/usr/lib/mpich-mpd/bin:/opt/c3-3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11
/var/fai/config/hooks/instsoft.DEFAULT
#! /bin/sh # This file needs to exist or the installation of atlas2-sse2 barfs. touch $target/etc/ld.so.conf # Replace exim with nullmailer fcopy -r /etc/nullmailer $ROOTCMD apt-get --purge install nullmailer exim- # set up kernel-img.conf so that the initrd kernel gets installed without prompting # otherwise it aborts and certain packages don't get installed # Scott Webster, Apr. 25, 2005 echo "do_initrd = Yes" >> $target/etc/kernel-img.conf echo "do_symlinks = Yes" >> $target/etc/kernel-img.conf
This is described in the reinstallation section.