Delft3d revision 5.01.00.2163 setup up on Ubuntu 12.04 (Prescise Pangolin).
Inspired by:
It is worth reading MpichCluster howto for background information on setting up clusters with Ubuntu. The code below was tested on Ubuntu 12.04 ( Precise Pangolin ) Desktop.
If you are setting up a remote machine and you would like a remote graphical interface, FreeNX is a useful fast secure way of running one. Windows client available here
export https_proxy=your_proxy.ac.uk:8080
sudo apt-add-repository ppa:freenx-team`
sudo sed -i 's/quantal/precise/g' /etc/apt/sources.list.d/freenx-team-ppa-precise.list
sudo apt-get update
sudo apt-get install freenx
wget https://bugs.launchpad.net/freenx-server/+bug/576359/+attachment/1378450/+files/nxsetup.tar.gz
tar -xvf nxsetup.tar.gz
sudo cp nxsetup /usr/lib/nx/nxsetup
sudo /usr/lib/nx/nxsetup --install
sudo echo -e "\n#Use unity 2d for client sessions\nCOMMAND_START_GNOME='gnome-session --session=ubuntu-2d'"|sudo tee -a /etc/nxserver/node.conf
Common libraries required for building Delft3d, this should be done on all nodes.
sudo apt-get install gcc gfortran automake autoconf subversion libreadline6-dev bison flex g++ libtool autotools-dev nfs-server nfs-client libexpat1-dev
On your cluster master make a directory for sharing via NFS and add it to /etc/export then restart NFS.
sudo mkdir /mirror
echo "/mirror *(rw,sync)" | sudo tee -a /etc/exports
sudo service nfs-kernel-server restart
On the slave nodes of your cluster edit /etc/fstab with root privileges
master-hostname:/mirror /mirror nfs
On all nodes if you are going to use hostnames make sure the hostname is associated with an IP address in /etc/hosts
Download and install MPICH
wget http://www.mpich.org/static/downloads/1.4.1/mpich2-1.4.1.tar.gz
tar -xvf mpich2-1.4.1.tar.gz
cd mpich2-1.4.1/
Command for building mpich
sudo ./configure CFLAGS=-fPIC FFLAGS=-fPIC FC=gfortran --enable-shared --enable-lib-depend --prefix=/mirror/mpich2
sudo make && sudo make install
If you are using subversion behind a proxy you need to associate the remote domain with the proxy by editing .subversion/servers in your home directory and having something like the following around line 100.
[groups]
group1 = *.deltares.nl
[group1]
http-proxy-host = yourproxy.ac.uk
http-proxy-port = 8080
Then checkout the repository out as below and enter the source directory.
cd /mirror
sudo svn checkout https://svn.oss.deltares.nl/repos/delft3d/tags/5.01.00.2163 delft3d_repository
cd delft3d_repository/src
Edit build.sh edit lines 243-245 and 247 if using gfortran to point to mpich install at /mirror/mpich2
sudo ./build.sh -gnu -64bit
Add a user mpiu and give that user ownership of the NFS share directory.
sudo adduser mpiu sudo
cd /mirror
sudo chown -R mpiu:mpiu
To run a flow simulation you will need the directory containing d_hydro.exe on your path. If following the above setup add it with:
export PATH=$PATH:/mirror/deflt3d/bin/lnx/flow2d3d/bin
For running wave models you should have wave/bin, swan/bin and swan/scripts on your path
export PATH=$PATH:/mirror/delft3d/bin/lnx/flow2d3d/bin:/mirror/delft3d/bin/lnx/wave/bin/:/mirror/delft3d/bin/lnx/swan/bin/:/mirror/delft3d/bin/lnx/swan/scripts/
Also Delft3d expects the LD_LIBRARY_PATH to contain the directory with
export LD_LIBRARY_PATH=/mirror/deflt3d/bin/lnx/flow2d3d/bin:/mirror/delft3d/bin/lnx/swan/lib
Also D3D_HOME should be exported
export D3D_HOME=/mirror/delft3d/bin/lnx
It is worth putting similar commands to all of the above in the .profile directory of the user account you are going to be running the simulations on.
You may need to make certain files executable, as below:
chmod +x d_hydro.exe
It is important that the same users exist on both master and slave nodes with identical id.
Depending on how your simulation is setup you may find there is some contention for disk writes, on your cluster master or whichever machine is acting as the NFS server. Is it possible to alleviate the disk writes bottleneck by using a ramdisk if you have sufficient RAM to hold the simulation files in memory, although sufficient space must be allocated to the ramdisk or NEFIS error 8001 Data file ’%s’ has already been opened will occur. You can setup a ramdisk with the following commands:
mkdir /tmp/ramdisk
sudo mount -t tmpfs -o size=5G tmpfs /tmp/ramdisk
To make this share accessible to other nodes you will have to add it to /etc/exports restart nfs on the master and add the /tmp/ramdisk to the /etc/fstab on any slave nodes, as shown for the mirror share above.
To run a flow simulation in parallel it is necessary to include the directory containing mpiexec on the path
export PATH=$PATH:/mirror/mpich2/bin
A machinefile is a simple text file which can be setup specifying the number of processes to be run on each node, if you are using hostnames they must be linked to an ip address in /etc/hosts
cluster-master:2
cluster-slave-000:2
A simple shell script can be setup to execute mpiexec passing the machinefile and then calling d_hydro.
mpiexec -machinefile ~/machinefile d_hydro.exe config_flow2d3d.xml
d_hydro expects an xml file which is a structued text file specifying the mdf file
<?xml version='1.0' encoding='iso-8859-1'?>
<DeltaresHydro start="flow2d3d">
<flow2d3d
MDFile = 'test.mdf'
>
</flow2d3d>
</DeltaresHydro>
Depending on the complexity of the model the simulation can only be broken down in a limited number of processes before the simulation could fail. There is generally no benefit in increasing the number of process on a particular node beyond the number of the physical cores on that node.
It doesn't appear to be currently possible to run models with domain decomposition across multiple nodes in the same manner as standard flow simulations. It should be possible to run the different domains on different nodes in Linux, but this has yet to be demonstrated. A flow with domain decomposition simulation does make use of multiple cores on a single node.
Coupled models are especially sensitive to files left from previous simulation as noted in this thread
On Linux a coupled model can be run from two terminals, one starting d_hydro.exe input.mdf and the other starting wave.exe input.mdw, getting around hanging issues which can be experienced when using the gui. No attempt has been made to run either part of the coupled model with MPICH yet, SWAN appears to make use of all available cores, d_hydro doesn't by default.
ClusterSSH offers a simple way to manage multiple machines from a GUI for the purposes of updating and upgrading libraries. Byobu offers an improved alternative to GNU Screen for persisting remote shell session with multiple tabs across multiple clients.