Thursday 11 June 2015

Setting up a computational cluster (HPC), part 2

Now that we can easily provide DHCP, DNS and TFTP and a debian image for all the nodes, we want to make it easy to maintain the cluster and setup user management. For maintaining packages and configuration etc we use Puppet on Debian. So awesome!
NOTE: remember to add "puppet" and "puppetmaster" in /etc/hosts on the server, so dnsmasq can provide DNS! Otherwise puppet agent will not know where to connect.

Setup Puppet on the master

Install the puppetmaster on the master node
apt-get install puppetmaster
A nice addition to the puppet service is the stdlib.
puppet module install puppetlabs-stdlib
regular expression and autosign.conf for fast deployment
See the configuration of puppet here
Where NIS is setup as
## User Authentication
# set NIS domain
file {"/etc/defaultdomain": source => "puppet:///modules/nodes/defaultdomain"} ->
# set yp server
file {"/etc/yp.conf": source => "puppet:///modules/nodes/yp.conf"} ->
# install NIS
package {"nis": ensure => installed} ->
# update passwd, shadow and gshadow
file_line {'update passwd': path => '/etc/passwd', line => '+::::::'} ->
file_line {'update shadow': path => '/etc/shadow', line => '+::::::::'} ->
file_line {'update group': path => '/etc/group', line => '+:::'} ->
file_line {'update gshadow': path => '/etc/gshadow', line => '+:::'}
Where NFS is setup as
## Network File System
package {"nfs-common": ensure => installed}
file_line {'nfs home':
path => '/etc/fstab',
line => ' /home nfs rw,hard,intr 0 0',
require => Package["nfs-common"],
notify => Exec['nfs mount'],
file_line {'nfs opt':
path => '/etc/fstab',
line => ' /opt nfs rw,hard,intr 0 0',
require => Package["nfs-common"],
notify => Exec['nfs mount'],
exec {'nfs mount':
command => '/bin/mount -a',
path => '/usr/local/bin',
refreshonly => true,
You can either set autosign.conf in the puppet folder to just sign everything, and sign the nodes as the connect via
puppet cert sign --all

Setup of the node

Puppet is installed via det preseed configuration by adding
d-i pkgsel/include string openssh-server puppet facter
Puppet needs to connect and get a certificate signed by the server. This is either done by autosign or by manually signing the nodes
puppet agent --test --waitforcert 60
Puppet is then either run by manually or by adding puppet to the /etc/rc.local to be run on every boot.
echo -n "Waiting for network."
while ! ip addr show | grep -F "inet 192.168.10" >> /dev/null
    sleep 1
    echo -n "."
# Run puppet for node
echo "Running puppet..."
echo "boot $(date)" >> /root/puppet-node.log
puppet agent -t | while read line; do
    echo $line
    echo $line >> /root/puppet-node.log

Setting up a computational cluster (HPC), part 1

So by the power elimination I got put in charge of administration/setup of the local cluster system for the theoretical/computational chemistry department. The current system was completely out-dated, and made it impossible to apt-get update/upgrade, so with the addition of additional 60+ nodes from another cluster it was up to me to save the system! Which practically means I had to set it up from scratch. And a lot of googling. So much googling.

So here is what I did.

First thing first, I wanted it easily maintainable and scalable. There is no way I wanted to install software manually on all the nodes, which means all installation and setup needs to be done automatically from the masternode (frontend).

This was done via PXE/TFTP booting, and installing of a netboot Debian image (with a few extra packages). After the Debian installation, package management and configuration of the nodes is done via Puppet.

To speed things up, the whole installation is done via a local apt-get mirror on the master node. This also insures that all the packages are exactly the same version.

What you need of physical hardware:

  • a frontend computer ( (probably with two ethernet ports)
  • Nx nodes (192.168.10.x)
  • switch(s)
  • Ethernet cables

The frontend:

  • hosts a apt-mirror
  • hosts all the user accounts (NIS/ypbind)
  • hosts home folder (for NFS)
  • running {DHCP, TFTP, DNS} (via DNSMASQ) and has PXE image
  • running Puppetmaster
  • running apache
  • running slurm

the nodes:

  • uses the frontend for apt-get server
  • uses frontend NIS for all user accounts
  • network mounted home folder (NFS)
  • running puppet agent
  • running slurm deamon

Setup of the master

Setup apt-mirror

We want all the nodes to have the same packages installed, also on the frontend, for consistency. The way this is implemented is to have local copy of the apt-get server. You will need apache for http requests.

apt-get install apt-mirror
mkdir /srv/apt # basepath
vi /etc/apt/mirror.list # edit and set basepath in

Remember to add debian-installer to the repository list, or the netboot (later on) will have trouble installing debian. Your mirror list should look something like this;


set base_path    /srv/apt
set nthreads     20
set _tilde       0

deb jessie main main/debian-installer
deb-src jessie main

After configuration, apt-mirror and create a symbolic link in your apache webfolder. Apt-mirror will take a few hours to download (approximate 70-90gb)

cd /var/www
sudo ln -s /srv/apt/mirror/ debian # create symbolic link to the mirror

Now we edit our source list to point and our own mirror instead of the internet


deb jessie main

so we know that we are using same packages as the nodes. Now to update our system we need too;

on the frontend


on the nodes



The first thing to setup is the DHCP server on the frontend, and because we want to run a DNS server as well, the easiest service to setup is dnsmasq, instead of ics-dhcp etc.

apt-get install dnsmasq

after installation we configure the server with /etc/dnsmasq.conf.

We want to serve DHCP on eth0 interface with TFTP/PXE boot in range 192.168.10.x. For all nodes the mac addresses are then registered


# tftp boot
pxe-service=x86PC, "Install Debian", pxelinux
# log
# nodes

We server internet on eth1 and local dhcp on eth0, so we setup a static ip on eth0


auto eth0
iface eth0 inet static    

Notice the dhcp-host line where I couple a mac-address to a hostname. The same hostname is then added to the /etc/hosts, for example;   node-hostname

Setup PXE booting image

download netboot/netboot.tar.gz for the version of Debian you are using, and setup the PXE boot;

mkdir /srv/tftp
cd /srv/tftp
tar -xf netboot.tar.gz
vi pxelinux.cfg/default

edit and setup PXE to use a preseed configuration. If you are unsure what to put in your preseed script, you can always manually install debian and check the debconf-get-selections --installer > preseed.cfg output after the installation, or look at this guide


default install
label install
    menu label ^Install
    menu default
    kernel debian-installer/amd64/linux
    append initrd=debian-installer/amd64/initrd.gz auto=true priority=critical url=http://192.168.

The preseed cfg is placed in the apache http folder so it can be loaded over the net. Remember to setup the mirror settings to use the local mirror on the frontend.


Setup NIS and NFS

Next is setup of user management and network shared folders (home and opt).

apt-get install nfs-common nfs-kernel-server

Set the mount drives



and run

nfs-kernel-server restart
showmount -e

And now for NIS

apt-get install nis

give it a NIS domain (remember it, mine was "sunray-nis")

Setup the master to be the server, by editing the file /etc/defaults/nis making sure that you have the following lines:


Once this is done you need to control which machines are allowed to access the NIS server. Do this by editing the file /etc/ypserv.securenets as in the following example:

# Restrict to 192.168.1.x

Run the configuration for NIS

/usr/lib/yp/ypinit -m

and restart nis

service nis restart

Next "setting up puppet/nodes"

Monday 17 February 2014

Compiling and setting up GAMESS

Small guide on how to setup the QM software GAMESS on a normal Ubuntu computer and work in parallel with multiple nodes (via sockets). Loosely this is based on this guide on how to compile GAMESS.

I'm going to pretend that you are working on Ubuntu 12.04 LTS, but I'm sure you can relate it to whatever distribution you are working on.

Download and compile

2. Run config


and answer the questions. Answer them truthfully.

3. Compile DDI

DDI is used to run the GAMESS executable, compile it by

make ddi

4. Compile GAMESS

After compiling DDI and setting up the config, we can just write


and everything will happen automatically. You can add the flag "-j4" if you have 4 CPU's and want the compiling to go a little faster. It takes a few minutes.

Note: Compiling without Math library

Note on compiling without a math library, like MKL. In some versions of GAMESS the linking will fail with error message like "including missing subroutines for vector operations failed". This is what the math libraries are used for. For Ubuntu you can install  BLAS and LAPACK easily with

sudo apt-get install libblas3gf libblas-doc libblas-dev
sudo apt-get install liblapack3gf liblapack-doc liblapack-dev

Now we just need to change the ./liked script and add some compiler flags. If you are compiling with gfortran then you need to find and update the following two lines in ./lked under the gfortran section.

- set BLAS='blas.o'
- set MATHLIBS=' '
+ set BLAS=''
+ set MATHLIBS='-lblas -llapack '

and similar if you are using ifort to compile edit the section in lked and set the correct compiler flags.
and then it should liked alright.

Update rungms

To run GAMESS there is a included a run script in the root folder, which needs to be updated located in the beginning, for the scratch folder and GAMESS path. so, edit (using VIM, if you are awesome, EMACS, if you are not).

vi rungms

and set following paths to the correct (and obviously not my username);

set SCR=/home/charnley/scr
set USERSCR=/home/charnley/scr
set GMSPATH=/home/charnley/opt/gamess

Note: GAMESS doesn't work?!

Pro tip by +Anders Steen Christensen
If this is the first time you are trying to get GAMESS working on your Linux machine, you will need to set kernel.shmmax to 1610612736. This is done by either;

/sbin/sysctl -w kernel.shmmax=1610612736

or if you don't want to do the command (or don't have root access) every time you boot up, open the file called:


and add the following line:


Running the GAMESS testcase (exam)

Included in GAMESS is a list of input files to be tested to see if the software is working as it should. This is also useful to run if you do changes to the source code. Go to the root of the GAMESS folder and write.


This will then output all the exams. If all is passed, then GAMESS should be working alright.

Setting up cluster calculation

If you want to have GAMESS working with a cluster system then there are a few things to change. This is to get the parallelization‎ with sockets to work.

1. Update rungms

You'll need to edit the part of rungms that checks the hostname of the current node. As present the rungms check if you are running calculations on any of the Iowa nodes (where GAMESS is developed), and you are probably not.

Find the following if-statement and switch;

if ($NCPUS > 1) then
    switch (`hostname`)

and locate the default switch case, and outcomment the "exit", just because the rungms doesn't regonize your hostname. If you are using a cluster system, you are probably using some kind of queuing system, like PBS og LoadLeveler. Here is what you need to change on the specific system;

1.1 PBS specific

PBS should be working out-of-the box. As long as you have fixed the above hostname exit. If you don't need the user-scratch file, you can set \$USERSCR to "/scratch/\$PBS_JOBID" as \$SCR, under

if ($SCHED == PBS) then

1.2 LoadLeveler specific

As default, rungms does not contain loadleveler default settings, so you will need to setup the following test.


under 'batch schedular' section, and then add the case

if ($SCHED == LOADL) then
    set SCR=/scratch/$LOADL_STEP_ID

If you don't use the $USERSCR output, you can set that too the local scratch folder as well.

2. Setting up input file

If you are running the calculation across multiple nodes, you'll need to tell the GAMESS executable how many nodes you are using, this is done simply by stating, fx. for 10 nodes;



in the input file, with mwords being how much memory you want to allocate.

Remember: Remember that you need 1 space before the sections on GAMESS input files.

The end

And that's it. So simple and easy out-of-the-box. You should be running loads of QM calculations now.

Happy GAMESS'ing

Friday 26 April 2013

Calculate RMSD from two XYZ files

I want to calculate the RMSD (Root-mean-square deviation) between two molecule structures in XYZ format. And after googling around I concluded that the easiest way to do it was to use pymol. However being a CLI user, I do not want to download the files and open up a GUI all the time, I just want a script that can do it via a terminal. Time for a little google and python project.

Calculating the RMSD mathematically for two sets of xyz coordinates for $n$ particles is straight forward:

$$\mathrm{RMSD}(v,w) = \sqrt{  \frac{1}{n} \sum_{i=1}^n ((v_{ix} - w_{ix})^2 + (v_{iy} - w_{iy})^2 + (v_{iz} - w_{iz})^2 ) }$$

However this is without taking into account that the two molecules could be identical and only translated in space. To solve this we need to position the molecules in the same center and rotate one onto the other.

The problem is solved by first find the centroid for both molecules and translating both molecules to the center of the coordinate system. Then we need an algorithm to align the molecules by rotation. For this I found Kabsch algorithm from 1976.

"It is a method for calculating the optimal rotation matrix, that minimzes the RMSD between two paired set of points.

The algorithm is nicely written out on wikipedia, so it was straight forward to implement (still took me a little time though). So I wont go into details of it here.

However it is clear that the centering the molecules using their centeroid could possibly not be the best way of finding the minimal RMSD between two vector sets. So +Lars Bratholm got the idea of using a fitting function, fitting the center of the molecules and using Kabsch to calculate a minimal RMSD.

You can find the code and examples here:



The output will then be "pure rmsd", "rmsd after rotation" and "rmsd after fit".

Note; for a testcase I calculated the rmsd for a molecule set with 140 atoms, to be ~0.15, and when I did the same calculation in pymol I got 0.092. However pymol did state that it was cutting 20 atoms, but didn't state which, where as my script takes all the atoms into account.

Happy RMSD'ing everyone.

Monday 26 November 2012

PM6 in GAMESS, Part 3, a new file is introduced

Good news and bad news. After working with a fortran 77 file called mndod.F and mopac 7.1 (First mopac Fortran 95 version) for a while, it seemed like a lot of subroutines was missing in this old file. So recently we wrote James Stewart (Mr. Mopac) asking for potential missing files (with the missing subroutines).  James, luckily enough, instead of finding more files for us, found a updated version of the mndod.F file with the missing subroutines.

This was great, but it also meant I had to start over with respect to mapping and compiling of this file.

The mndod.f, for historical purpose, is the d-integrals from the old mndo-d method (created by Walter Theil), and then converted to work with MOPAC from version 7+, for AM1-D and on.
To follow the naming convention of GAMESS file structure, this holy file will now be known as mpcintd.src (because, yeah, d-integrals).

So I started over with mapping (see figure above), fixing compile errors, updating and deleting common blocks. This would not be possible without the source from Mopac 7.1 (available on, because the new mndod.f file lacked a lot of comments and documentation, referencing to unknown common-blocks and subroutines.

This was a lot of work, but still faster than rewriting mopac 7.1, because this version of mopac uses interfaces and modules, and so a lot of subroutine headers would need to be written before I could compile it, yet alone test it. No Thanks.

After working a weeks time on implementing the new code/subroutines, the result is now I get the correct nuclear repulsion term, but I do not get the correct electronic energy. This results in correct 'NUCLEAR ENERGY' (gamess output), but incorrect electronic energy, and therefore incorrect Heat of Formation.

Update list from last blogpost;

Step 1: Get mndod.f90 compiled with gamess
Step 2: Integration: Make IF(PM6) and mndod code instead of gamess with pm6 parameters and more
Step 3: Integration: Make IF(PM6) and run fock-d 1 and 2 in mpcg()
Step 4: Find out why it does not work and solve the problem
Step 5: Celebration

Clearly on step 4, trying to make it work.

Stay tuned for the dramatic conclusion of implementation of PM6 in GAMESS!

Friday 31 August 2012

PM6 in GAMESS, Part 2

Okay, so I'm still working on implementing PM6 integrals in GAMESS.

I got the source code from MOPAC 7.1 which includes d-integrals for the MNDO-D method (which is what Jimmy Stewart is using for PM6 in the newest MOPAC (hopefully), which originates from a program written by Walter Thiel).

So the strategy is simply to 'export' the subroutines / modules from MOPAC 7.1 needed to replicate the d-integrals in GAMESS (written in Fortran 90),  and 'import' them into GAMESS-US.
Now, the semi-emperical part of GAMESS-US is actually based on a older version of MOPAC (written in Fortran 77) so the subroutines should be very similar to the code I'll be trying to import.

First part of this mission is too map the relevant subroutines in both GAMESS and MOPAC. And hopefully I'll be able too se a pattern and merge the 'trees'.

The map for MOPAC is:

And the map for GAMESS is:

Now I just need an idea for merging the two trees. Since Stewart based his d-integrals on code he got from Thiel it seems like most of the subroutines is collected in a single file called mndod.F90 (fitting name, lol).

This means I 'just' (I beginning to hate that work) need to copy-and-paste the file into GAMESS and make sure the file is hooked to GAMESS common blocks instead of the fortran 90 modules from MOPAC. So step 1: Include the file and make it compile (which is a lot of rewriting interfaces and modules into the actual file so it is a standalone solution.)

The highlighted area is only the first part of the problem though. After the fock matrix has been put together with the new and cool d-items the matrix needs to be solved and we need the fockd1 and fockd2 for that. They are conveniently also put in the same file with the rest of the subroutines.

Furthermore I have been told by +Jan Jensen that I need to watch out for 'guessmo' subroutine when implementing the new integrals. As described in his figure;

So to recap, implementation in 5 easy steps (said in a very television kitchen accent):

Step 1: Get mndod.f90 compiled with gamess (using gamess common blocks instead of mopac modules and interfaces)
Step 2: Integration: Make IF(PM6) and run the mndod code instead of gamess with pm6 parameters and more.
Step 3: Integration: Make IF(PM6) and run the fock-d 1 and 2 instead in mpcg()
Step 4: Find out why it does not work and solve the problem.
Step 5: Celebration.

To be continued!

Tuesday 28 August 2012

PM6 in GAMESS, Part 1

Okay, so I'm working on implementing the semi-empirical method PM6 (by Jimmy "Mopac" Stewart) in GAMESS-US.

The status is; (before I started working) GAMESS has up to and including PM3 already implemented. So the idea is just to update the SE parameters and substitute the subroutines necessary to get PM6 working. Without prior knowledge to GAMESS this really did not sound like a big deal, as the differences between PM6 and PM3 only lies in the way the parameters are used (roughly). The parameterization of PM3 (and AM1) is utilised in the core-core repulsion term (nuclear repulsion) of the Heat of Formation to compensate for the aproximations made in SE methods. Heat of Formation is calcuated acordingly:

$$\Delta H_f = E_{\rm Elect} + E_{\rm Core} - \sum_{A}^{} E_{el}^{A} + \sum_{A}^{} \Delta H_{f}^{A}$$

The emperical parameters from PM3 is fitted via a scaleing factor on the core-core term to fit the experimental heat of formation for the molecule. Fitting of the data and derivation of the parameters was done my Jimmy Stewart, for his program MOPAC where the methods were original implemented. The PM3 core-core repulsion term looks like this;

$$E_n(A,B) = Z_A Z_B \langle s_A s_A | s_B s_B \rangle \left ( 1 + e^{-\alpha_A R_{AB}} + e^{-\alpha_B R_{AB}} \right ) $$

which is then summed over all nuclear repulsions/interactions between any atom A and B. This core-core term needs to be substituted with the new term from PM6:

$$E_n(A,B) = Z_A Z_B \langle s_A s_A | s_B s_B \rangle \left ( 1 + x_{AB} e^{-\alpha_{AB} (R_{AB} + 3 \cdot 10^{-4} R_{AB}^6)} \right )$$

Note that the $\alpha$ parameter is now a di-atomic parameter unlike the mono-atomic parameter is PM3. Another parameter $x$ is also introduced, but that is 'pritty much it'. (there are also a Lennard Jones Term and a van der Waals term, but that is for another blog post). The parameters are all located in the PM6 article, but Jimmy Stewart was kind of enough to send his files including his implementation of PM6 core equation and the list of all parameters. This saved me alot of pointless typing time, so thanks!

Okay, so after implementing the new PM6 specific code and the corresponding parameters, I discovered that the result did not match its MOPAC equivalent. In fact nether electronic, core or the total energy fit the MOPAC value. This was happening for single point energy calculation even for very small molecules (even water). For reference I did a similar test for PM3 and AM1, and found that the already implemented methods results did not fit it's rightfull energies (MOPAC energy) with the same order of magnitude as PM6, which was quickly discovered to be size dependt. This is clearly shown in the below figure, which shows single point energy calculations on a carbon chain from 1 carbon to 20 carbons

Energy difference $\Delta E$ is calculated from mopac energy minus the corresponding gamess energy.

Arrgghh! How am I going to implement a new method, when the already implemented methods varies this much from the original program?

Okay, so the problem was that the SE part of GAMESS was based on a very old version of MOPAC, and so we figured alot of the energy deviation must be originating from the lack of update on physical constants. The MOPAC integrals use two physical constants to calculate the integrals in atomic units, namely bohr radius and electron volts, so by using grep I found all the places where the constants/variables were defined (which was alot!), and then updated acording to the constant defined on MOPAC's website using a common block, instead of alot of local instances.

This resulted in

Okay, but is this better? Hell yeah! The total energy is clearly more stable compared to MOPAC energy, which is the energy that matters most. The deviation in the nuclear and electronic energy looks very much linear which hints to more constants needed to be updated. Note I have only updated the constants located in MOPAC part of GAMESS and therefor only effects the semi-empirical part of GAMESS.

However the effect is there, and even though the energy is working now, it will prove a problem for people who wants to reproduce data already calculated with GAMESS. So be warned GAMESS users, keep a copy of your GAMESS when the PM6 update is integrated in GAMESS-US.

PM6 Gradient:

The integration of gradient was actually really easy, because GAMESS only uses numerical gradients for semi-emperical calculations.

Am I done? Unfortunately no. To get PM6 fully working I need to implement the d-integrals from MOPAC. As it is now only s- and p-orbitals are used for calculating the integrals. Is that easy? No.

To be continued...

tldr; PM6, PM3 and AM1 did not work as expected, which was partially fixed by updating physical constants in the semi-empirical part of GAMESS. PM6 energy and gradient now works up to including Ne, but will need d-orbitals before it is fully operational.