baseline-generation-and-comparison

baseline-generation-and-comparison

#
>git checkout bishtgautam/lnd/lateral-vsfm-infrastructure

# Let's look at the graph
>git log --oneline --decorate --graph
* c425c1a (HEAD, origin/thorntonpe/lnd/implement-architecture-v2, origin/bishtgautam/lnd/lateral-vsfm-infrastructure, bishtgautam/lnd/lateral-vsfm-infrastructure) Updates VSFM solver tolerance after an unacceptable solution
* ff07479 Adds options to include lateral flows in VSFM
* 46115a8 Updates VSFM solver in ALM to saves internal/boundary mass flux
* da638f5 Turns on lateral subsurface flow in ALM
* 99c6fe6 Adds seepage face BC in ALM VSFM solver
* e991824 Minor ALM modifications to avoid compiler failures
* d2ccd94 Updates to ALM suburface mesh for VSFM
* 973d89e ALM reads additional data specified in MPAS format
* 060672f Adds initial infrastructre to support lateral subsurface flow in ALM
* a482ef8 Updates to equation-of-state in ALM for VSFM
* 6995f91 Assigns soil properties to ghost/halo colums in ALM
* 1fc8a0d Fixes the mapping of neighboring grid cells in ALM
* f4a25ec Adds subroutine to exchange column-level ALM across MPI tasks
* e7fbfac Updates the check for ALM ghost subgrid hierarchy
* 741154e Adds x/y field for each ALM grid cell
* 7750c3c Adds subroutine to count ALM ghost subgrid hierarchy
* 47067da Fixes preprocessor statement in ALM-VSFM
* 3a0893b Adds update of ALM's ghost subgrid structure
* b5aa2e8 Adds more functionality for laterally connected ALM domain
* df50d5a Adds subroutines to get/set values for ALM subgrid categories
* 4bf9682 Initial implementation of a lateral connectivity framework for ALM
* 0a4dd12 Consolidates the initialization of PETSc in ALM
* 4bc25cb Adds support for decomposing ALM grid via graph partitioning
* 8300f8a Adds a subroutine in ALM to read grid-to-grid connectivity
* ccd76ef Adds namelist option to specify domain decompisition for ALM.
* 6b8cc6a (origin/bishtgautam/lnd/topo-effects-on-rad) Adds 2nd order effects of topography on surface radiation
* 75d146c Incroporates effect of topography on downwelling solar radation
* a803322 Adds a function to compute solar zenith angle
*   f720c5d Merge branch 'jayeshkrishna/cime/change_perl5lib_doc_filename' (PR #813)
|\  
| * c0426b9 (origin/jayeshkrishna/cime/change_perl5lib_doc_filename) Replacing colons in man page filenames to underscores
* |   d69c8a8 Merge branch 'worleyph/cime/comp_timer_attribution' (PR #790)
|\ \  
| * | 6c606f0 (origin/worleyph/cime/comp_timer_attribution) adding component and phase specific prefixes to timer names


# From the graph above, f720c5d is the starting tag of master.
#
# (Btw, another way to find out the commits that are in a particular branch 
#  but not in master is:
#  git log master..bishtgautam/lnd/lateral-vsfm-infrastructure --oneline --graph
# )

# So, one should do the following:
# Step-1: Generate the baselines using the tag f720c5d, and
# Step-2: Checkout thorntonpe/lnd/implement-architecture-v2 and make comparison against baselines generated in Step-1.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

# Step-1: Generate baseline
>git checkout f720c5d
M	components/mpas-cice/model
M	components/mpas-o/model
M	components/mpasli/model
Note: checking out 'f720c5d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at f720c5d... Merge branch 'jayeshkrishna/cime/change_perl5lib_doc_filename' (PR #813)

>git submodule update --init
Submodule path 'components/mpas-cice/model': checked out 'fd1e8802a257b0347ad5a80bf83d41eda0780c53'
Submodule path 'components/mpas-o/model': checked out '23155b16bba6eb59d666bcf1f11b6dc0c5053962'
Submodule path 'components/mpasli/model': checked out 'c5ef15e9e669f231118f6c76ae2b3a8664aca1c2'


>cd cime/scripts

# We will generate baselines for only 'acme_land_developer'

# Define the directory to hold the baseline 
# Note: Following assumpitions are made in these notes:
#    - Tests are run on Titan.
#    - User is using a bash shell.
export MY_BASELINE_DIR=/lustre/atlas/scratch/gb9/cli112/acme_baselines

# Which project to charge for the HPC usage
export PROJECT=cli112

# Do you want to use a name for the baseline? 
# I prefer to use the git hash that is being used to generate the baselines.
export BASELINE_NAME=f720c5d

# Let's set TEST_ID to be same as BASELINE_NAME
export TEST_ID=${BASELINE_NAME}

# -v: Verbose option
# -g: Generate the baseline

>./create_test \
acme_land_developer \
--baseline-root ${MY_BASELINE_DIR} \
-p ${PROJECT} \
-b ${BASELINE_NAME} \
-t ${TEST_ID} \
-g \
-v 

# The casenames would be named *.G.* to denote one is generating the baselines

# Now go get some coffee/tea, as it will take a while to compile all the cases and submit the code.

# After the cases have been successfully compiled and submit, one can check that 
# status of test by the cs.status.YYYYMMDD_HHMMSS file that was created in
# /ccs/home/gb9/acme_scratch/cli112/
#
# cd /ccs/home/gb9/acme_scratch/cli112/;
# ./cs.status.YYYYMMDD_HHMMSS

# After all simulations finish, the baseline would be saved in ${MY_BASELINE_DIR}/pgi/${BASELINE_NAME}.
# Note, pgi directory is automatically created because the default compiler on Titan is PGI.


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

# Step-2: Compare against previously generated baselines

>git checkout thorntonpe/lnd/implement-architecture-v2

Previous HEAD position was f720c5d... Merge branch 'jayeshkrishna/cime/change_perl5lib_doc_filename' (PR #813)
Branch bishtgautam/lnd/lateral-vsfm-infrastructure set up to track remote branch bishtgautam/lnd/lateral-vsfm-infrastructure from origin.
Switched to a new branch 'bishtgautam/lnd/lateral-vsfm-infrastructure'

>git submodule update --init

>cd cime/scripts

# We are comparing baselines for only 'acme_land_developer'

# Let's use the settings as the last time
export MY_BASELINE_DIR=/lustre/atlas/scratch/gb9/cli112/acme_baselines
export PROJECT=cli112

# IMPORTANT: One needs to use the same BASELINE_NAME as in Step-1 because one wants
#            to compare against the baselines previously generated in Step-1.
export BASELINE_NAME=f720c5d

# I prefer to use the the has as test-id
export TEST_ID=`git log -n 1 --format=%h`

# -v: Verbose option
# -c: Compare baseline

>./create_test \
acme_land_developer \
--baseline-root ${MY_BASELINE_DIR} \
-p ${PROJECT} \
-b ${BASELINE_NAME} \
-t ${TEST_ID} \
-c \
-v 

# The casenames would be named *.C.* to denote one is comparing against previously generated baselines

# It is time for some coffee/tea.

# Similar to the last time, a new cs.status.YYYYMMDD_HHMMSS would be created and
# you can check the status of test by:
#
# cd /ccs/home/gb9/acme_scratch/cli112/;
# ./cs.status.YYYYMMDD_HHMMSS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Thanks for the great instructions!
In Step 2, should the second "export BASELINE_NAME ..." be "export TEST_ID ..." ?
Are you expecting that everything from commit f720c5d to commit c425c1a is BFB? If not, then the comparison to baseline will fail. I was thinking that for my branch development I need to be comparing against baselines generated at its branch-point, c425c1a. Of course, if all the commits on your branch ahead of origin/master are BFB, then it won't make any difference.