Setting up a computational cluster (HPC), part 2

Now that we can easily provide DHCP, DNS and TFTP and a debian image for all the nodes, we want to make it easy to maintain the cluster and setup user management. For maintaining packages and configuration etc we use Puppet on Debian. So awesome!
NOTE: remember to add "puppet" and "puppetmaster" in /etc/hosts on the server, so dnsmasq can provide DNS! Otherwise puppet agent will not know where to connect.

Setup Puppet on the master

Install the puppetmaster on the master node

apt-get install puppetmaster

A nice addition to the puppet service is the stdlib.

puppet module install puppetlabs-stdlib

regular expression and autosign.conf for fast deployment
See the configuration of puppet here

• https://github.com/charnley/debian-cluster-configuration/tree/master/etc/puppet/manifests

Where NIS is setup as

## User Authentication
# set NIS domain
file {"/etc/defaultdomain": source => "puppet:///modules/nodes/defaultdomain"} ->
# set yp server
file {"/etc/yp.conf": source => "puppet:///modules/nodes/yp.conf"} ->
# install NIS
package {"nis": ensure => installed} ->
# update passwd, shadow and gshadow
file_line {'update passwd': path => '/etc/passwd', line => '+::::::'} ->
file_line {'update shadow': path => '/etc/shadow', line => '+::::::::'} ->
file_line {'update group': path => '/etc/group', line => '+:::'} ->
file_line {'update gshadow': path => '/etc/gshadow', line => '+:::'}

Where NFS is setup as

## Network File System
package {"nfs-common": ensure => installed}
file_line {'nfs home':
path => '/etc/fstab',
line => '192.168.10.1:/home /home nfs rw,hard,intr 0 0',
require => Package["nfs-common"],
notify => Exec['nfs mount'],
}
file_line {'nfs opt':
path => '/etc/fstab',
line => '192.168.10.1:/opt /opt nfs rw,hard,intr 0 0',
require => Package["nfs-common"],
notify => Exec['nfs mount'],
}
exec {'nfs mount':
command => '/bin/mount -a',
path => '/usr/local/bin',
refreshonly => true,
}

You can either set autosign.conf in the puppet folder to just sign everything, and sign the nodes as the connect via

puppet cert sign --all

Setup of the node

Puppet is installed via det preseed configuration by adding

d-i pkgsel/include string openssh-server puppet facter

Puppet needs to connect and get a certificate signed by the server. This is either done by autosign or by manually signing the nodes

puppet agent --test --waitforcert 60

Puppet is then either run by manually or by adding puppet to the /etc/rc.local to be run on every boot.
/etc/rc.local

#!/bin/bash
echo -n "Waiting for network."
while ! ip addr show | grep -F "inet 192.168.10" >> /dev/null
do
    sleep 1
    echo -n "."
done
# Run puppet for node
echo "Running puppet..."
echo "boot $(date)" >> /root/puppet-node.log
puppet agent -t | while read line; do
    echo $line
    echo $line >> /root/puppet-node.log
done

Setting up a computational cluster (HPC), part 1

So by the power elimination I got put in charge of administration/setup of the local cluster system for the theoretical/computational chemistry department. The current system was completely out-dated, and made it impossible to apt-get update/upgrade, so with the addition of additional 60+ nodes from another cluster it was up to me to save the system! Which practically means I had to set it up from scratch. And a lot of googling. So much googling.

So here is what I did.

First thing first, I wanted it easily maintainable and scalable. There is no way I wanted to install software manually on all the nodes, which means all installation and setup needs to be done automatically from the masternode (frontend).

This was done via PXE/TFTP booting, and installing of a netboot Debian image (with a few extra packages). After the Debian installation, package management and configuration of the nodes is done via Puppet.

To speed things up, the whole installation is done via a local apt-get mirror on the master node. This also insures that all the packages are exactly the same version.

What you need of physical hardware:

• a frontend computer (192.168.10.1) (probably with two ethernet ports)
• Nx nodes (192.168.10.x)
• switch(s)
• Ethernet cables

The frontend:

• hosts a apt-mirror
• hosts all the user accounts (NIS/ypbind)
• hosts home folder (for NFS)
• running {DHCP, TFTP, DNS} (via DNSMASQ) and has PXE image
• running Puppetmaster
• running apache
• running slurm

the nodes:

• uses the frontend for apt-get server
• uses frontend NIS for all user accounts
• network mounted home folder (NFS)
• running puppet agent
• running slurm deamon

Setup of the master

Setup apt-mirror

We want all the nodes to have the same packages installed, also on the frontend, for consistency. The way this is implemented is to have local copy of the apt-get server. You will need apache for http requests.

apt-get install apt-mirror
mkdir /srv/apt # basepath
vi /etc/apt/mirror.list # edit and set basepath in

Remember to add debian-installer to the repository list, or the netboot (later on) will have trouble installing debian. Your mirror list should look something like this;

/etc/apt/mirror.list

set base_path    /srv/apt
set nthreads     20
set _tilde       0

deb http://ftp.dk.debian.org/debian/ jessie main main/debian-installer
deb-src http://ftp.dk.debian.org/debian/ jessie main

After configuration, apt-mirror and create a symbolic link in your apache webfolder. Apt-mirror will take a few hours to download (approximate 70-90gb)

apt-mirror
cd /var/www
sudo ln -s /srv/apt/mirror/ftp.dk.debian.org/debian debian # create symbolic link to the mirror

Now we edit our source list to point and our own mirror instead of the internet

/etc/apt/source.list

deb http://192.168.10.1/debian/ jessie main

so we know that we are using same packages as the nodes. Now to update our system we need too;

on the frontend

apt-mirror
apt-update
apt-upgrade

on the nodes

apt-update
apt-upgrade

Setup DHCP, DNS and TFTP with DNSMASQ

The first thing to setup is the DHCP server on the frontend, and because we want to run a DNS server as well, the easiest service to setup is dnsmasq, instead of ics-dhcp etc.

apt-get install dnsmasq

after installation we configure the server with /etc/dnsmasq.conf.

We want to serve DHCP on eth0 interface with TFTP/PXE boot in range 192.168.10.x. For all nodes the mac addresses are then registered

/etc/dnsmasq.conf

interface=eth0
dhcp-range=192.168.10.10,192.168.10.255,72h
# tftp boot
dhcp-boot=pxelinux.0,pxeserver,192.168.10.1
pxe-service=x86PC, "Install Debian", pxelinux
enable-tftp
tftp-root=/srv/tftp
# log
log-queries
log-dhcp
log-facility=/var/log/dnsmasq
# nodes
dhcp-host=00:33:64:b1:83:94,node-hostname

We server internet on eth1 and local dhcp on eth0, so we setup a static ip on eth0

/etc/network/interface

auto eth0
iface eth0 inet static    
address 192.168.10.1
netmask 255.255.255.0

Notice the dhcp-host line where I couple a mac-address to a hostname. The same hostname is then added to the /etc/hosts, for example;

192.168.10.23   node-hostname

Setup PXE booting image

download netboot/netboot.tar.gz for the version of Debian you are using, and setup the PXE boot;

mkdir /srv/tftp
cd /srv/tftp
tar -xf netboot.tar.gz
vi pxelinux.cfg/default

edit and setup PXE to use a preseed configuration. If you are unsure what to put in your preseed script, you can always manually install debian and check the debconf-get-selections --installer > preseed.cfg output after the installation, or look at this guide https://www.debian.org/releases/stable/amd64/apbs04.html.en

pxelinux.cfg/default

default install
label install
    menu label ^Install
    menu default
    kernel debian-installer/amd64/linux
    append initrd=debian-installer/amd64/initrd.gz auto=true priority=critical url=http://192.168.

The preseed cfg is placed in the apache http folder so it can be loaded over the net. Remember to setup the mirror settings to use the local mirror on the frontend.

/var/www/sunray-preseed.cfg

• https://github.com/charnley/debian-cluster-configuration/blob/master/var/www/sunray-preseed.cfg

Setup NIS and NFS

Next is setup of user management and network shared folders (home and opt).

apt-get install nfs-common nfs-kernel-server

Set the mount drives

/etc/exports

/home 192.168.10.1/255.255.255.0(rw,sync,no_subtree_check)
/opt 192.168.10.1/255.255.255.0(rw,sync,no_subtree_check)

and run

nfs-kernel-server restart
showmount -e

And now for NIS

apt-get install nis

give it a NIS domain (remember it, mine was "sunray-nis")

Setup the master to be the server, by editing the file /etc/defaults/nis making sure that you have the following lines:

NISSERVER=true
NISCLIENT=false

Once this is done you need to control which machines are allowed to access the NIS server. Do this by editing the file /etc/ypserv.securenets as in the following example:

# Restrict to 192.168.1.x
255.255.255.0 192.168.1.0

Run the configuration for NIS

/usr/lib/yp/ypinit -m

and restart nis

service nis restart

Next "setting up puppet/nodes"