How to set up a grid of computers using the Mac OS X desktop version.
Any machine can be an agent. Configure the agent using the Sharing Pane of System Preferences. See Managing Xgrid Agents.
How to set up a grid of computers using the Mac OS X desktop version.
Any machine can be an agent. Configure the agent using the Sharing Pane of System Preferences. See Managing Xgrid Agents.
For basic applications, MPI is as easy to use as any other message-passing system. The sample scripts below contain the complete communications skeleton for a data (or embarrassingly) parallel problem using the mpi4py package.
Within the code is a description of the few functions needed to write typical parallel applications.
mpi-submit.py - Parallel application with simple partitioning: unbalanced load.
mpi-submit2.py - Parallel application with master/slave scheme: dynamically balanced load.
Spark
If you have (larger-than-memory) petabytes of JSON/XML/CSV files, a simple workflow, and a thousand node cluster
Dask
If you have (larger-than-memory) 10s-1000s of gigabytes of binary or numeric data (e.g., HDF5, netCDF4, CSV.gz), complex algorithms, and a (single) large multi-core workstation
Some (trivial?) commands to work on a remote machine using the SSH, FTP and HTTP protocols.
Generate SSH keys for using a SSH connection without having to enter the password every time you log in.
Generate the key (local machine) [Skip this step if you have already the keys in your machine!]:
You need the package manager Homebrew (if not installed), see http://brew.sh/
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Get Spark 'Source Code':
http://spark.apache.org/downloads.html
Triton, TCC and TSCC, are clusters at the San Diego Supercomputer Center (SDSC): http://www.sdsc.edu/services/hpc/hpc_systems.html#tscc
Account balance:
Although I'm not a big fan of Fortran, here is a simple example of parsing Unix-like command line arguments to a program following the modern standard of F2003. Note that the following example also works for any F90/F95 code compiled with gfortran or g95.
Example:
$ ./thisprog -h
usage: ./thisprog [-h] [-a ARG_A] [-b ARG_B] [-c]