###First Clone the latest version of the model from bitbucket.
###Upload code to TACC
You can use scp, rsync, even git, whatever. I use rsync:
rsync -raz . jdh33@stampede.tacc.utexas.edu:~/projects/emodps
when you're in the highest level directory of the project (the dot .) the nice thing about rsync is that it only transfers files that were added or modified relative to the destination.
###Compile Go into the "parallel" folder
make -f makefile-tacc
This will create a couple of executables. The one you care about is ./RedRiver4Par
###Settings: historical or stochastic
Edit mainP_redriver4.cpp (contains the main function to run this program)
On line 21, choose either
model_redriver4 RedRiverProblem("settings_redriver4r_MC.txt");
or
model_redriver4 RedRiverProblem("settings_redriver4r.txt");
Anywhere you see MC is Monte Carlo. You can check out those text files to see how Matteo set them up -- it's pretty nice from a usability standpoint. I'm 99% sure that's the only change you need to make to switch between stochastic/historical.
###Other considerations
On line 85 of mainP_redriver4.cpp notice it wants to print to the output directory. This doesn't exist so you'll have to create it on TACC.
(So in terms of the project directory it's emodps/parallel/output)
Runtime estimates. We timed the historical model at about 1.38 sec/NFE on the Cube (500K NFE, 0.75 hr, 256 procs.) The stochastic model should be about 20x longer because it's running 20 realizations of synthetic streamflow.
###Submit
There are two job scripts in the parallel directory for TACC. Modify as needed -- right now there's one for small test jobs, and one for larger experiments.
tacc-job-script.sh submits to the development queue. tacc-large submits to the regular queue.
See here for queue information and other useful info:
https://portal.tacc.utexas.edu/user-guides/stampede#running
To submit the job, on the command line run:
sbatch tacc-job-script.sh
There are a lot of people using TACC so it may take on the order of hours for a job to start. So it's useful to include your email address in the job script to be notified when it actually runs. You can view the status of your jobs with showq -u <your_id>. Cancel jobs with scancel <job_id>.
Job submission is managed by SLURM (Simple Linux Uniform Resource Manager), which is why these commands are different from the Cube which runs PBS. But for everything you can do on a PBS system, there's an analogous command for SLURM, you just have to do some googling.
###Output
Your set(s) will be located in parallel/output/RedRiver_S<#>.set for each seed #.
There are three objectives -- the last three columns. Not sure how many DV there are but it's > 100.
Check Matteo's powerpoints for the equations for these objectives. You can scp them down to your computer to plot with matlab or something,
scp jdh33@stampede.tacc.utexas.edu:~/emodps/parallel/output/* .
###System hours to use When you log in, it prints the time left on the allocation:
______________________________________________________________________________
----------------------- Project balances for user jdh33 -----------------------
| Name Avail SUs Expires | |
| TG-EAR090013 2236240 2015-09-30 | |
------------------------- Disk quotas for user jdh33 --------------------------
| Disk Usage (GB) Limit %Used File Usage Limit %Used |
| /home1 0.0 5.0 0.30 631 150000 0.42 |
| /work 1.0 1024.0 0.09 964 3000000 0.03 |
-------------------------------------------------------------------------------
So we have about 2.2 million hours total. For first experiments don't use more than ~200K. The Red River project as a whole should only use about half of the allocation (1 million hrs) as per Pat.