Pseudocode for PR testing in Jenkins

1-intro.md

Assumptions (Git structure)

1. There are multiple git repositories which must be assembled into a full build.
2. There is a command which creates a working build (eg "drush make" or "civibuild create") by fetching the appropriate code from all the repositories.
3. A single job in Jenkins can be associated with a single git repository, and new PR's/change-requests will trigger a build. Jenkins will checkout that single repository.
4. Conclusion: We need a way to use our a single build command (eg "drush make" or "civibuild create") so that most of the build uses canonical code -- but one small part of the build (one particular directory or another) imitates the repository configured in Jenkins. The new command "git cloneinto" will address this.

Assumptions (Performance/optimization)

1. The build command (eg "drush make" or "civibuild create") is I/O intensive.
2. It is faster (less I/O) to use "git checkout " than "rm -rf oldrepo; git clone -b "
3. The file-structure of the build (ie the placement of the subordinate git repos) rarely changes.
4. Conclusion: Each job should have a single source-tree which is retained across various test runs. When executing a new test-run, it is preferrable to massage an existing source-tree to have the right files (eg "git reset --hard; git checkout") rather than to delete and recreate. The new commands "git scan branch-save" and "git scan branch-load" will address this.

Setup Instructions

1. Install Jenkins master node
2. Install any Jenkins slave nodes
3. On any nodes which will test PRs, pre-install buildkit (eg to /srv/buildkit)
4. Create Jenkins job like one of the examples below. (Note: The code in the examples is all the same, except for lines marked with "^^^^".

2-civicrm-org-job.sh

#!/bin/bash

## This is an example Jenkins job.
##
## Job Name: CiviCRM-Drupal-PR
## Job Description: Monitor civicrm-drupal.git for proposed changes
##    For any proposed changes, run a few "smoke tests"         
## Job Git Repo: https://github.com/civicrm/civicrm-drupal.git
## Job Git Path: WORKSPACE/pr-repo
## Job Triggers: GitHub pull-requests
## Job xUnit Files: WORKSPACE/output/*.xml

## Pre-requisite: We only test PR's for main-line branches.
case "$ghprbTargetBranch"
  4.3|4.4|4.5|master)
    echo "PR test allowed for $ghprbTargetBranch."
    ;;
  *)
    echo "PR test not allowed for $ghprbTargetBranch"
    exit 1
    ;;
esac

## Environment
BLDKIT="/srv/buildkit"
BLDNAME=lowercase("$JOB_NAME-$ghprbTargetBranch")
BLDDIR="$BLDKIT/build/$BLDNAME"
BRANCH_LIST="$BLDKIT/build/$BLDNAME.branches"
PATH="$BLDKIT/bin:$PATH"

## On the first exeuction, we need to do a full install
if [ ! -d "$BLDDIR" ]; then
  civibuild create "$BLDNAME" --type drupal-clean --civi-ver $ghprbTargetBranch --url "http://$BLDNAME"
  ## ^^^^ this command is likely to change depending version & build config

  ## Save the names of the various branches/remotes   
  ## e.g. "sites/all/modules/civicrm" ==> "origin/4.4"
  ## e.g. "sites/all/modules/civicrm/drupal" ==> "origin/7.x-4.4"
  pushd "$BLDDIR"
    git scan branch-save > "$BRANCHLIST"
  popd
fi

## On every iteration, reset code+DB
pushd "$BLDDIR"
  ## Always start out with the latest canonical code from the proper branches
  git scan branch-load < "$BRANCHLIST"
  git scan up --hard

  ## Load the code identified by Jenkins as the target
  git cloneinto "$WORKSPACE/pr-repo" "$BLDDIR/sites/all/modules/civicrm/drupal"
  ## ^^^^ this command is likely to change depending on the particular repo used in this job

  ## Reinitialize the Drupal+Civi databases
  civibuild reinstall "$BLDNAME"
popd

## Run the tests
pushd "$BLDDIR/sites/all/modules/civicrm/tools"
  ./scripts/phpunit --junit-xml "$WORKSPACE/output/1.xml" WebTest_Generic_CheckDashboardTest
  ## ^^^^ this command is likely to change depending on how many tests you want to run for any PR
popd

3-wmff-job.sh

#!/bin/bash

## This is an example Jenkins job.
##
## Job Name: WMFF-CRM-Civi-PR
## Job Description: Monitor wikimedia-fundraising-crm-civicrm.git for changes.
##    For any proposed changes, run a few "smoke tests"         
## Job Git Repo: https://github.com/wikimedia/wikimedia-fundraising-crm-civicrm
## Job Git Path: WORKSPACE/pr-repo
## Job Triggers: Gerrit changes against master
## Job xUnit Files: WORKSPACE/output/*.xml

BLDKIT="/srv/buildkit"
BLDNAME="$JOB_NAME"
BLDDIR="$BLDKIT/build/$BLDNAME"
BRANCH_LIST="$BLDKIT/build/$BLDNAME.branches"
PATH="$BLDKIT/bin:$PATH"

## On the first exeuction, we need to do a full install
if [ ! -d "$BLDDIR" ]; then
  civibuild create "$BLDNAME" --type wmff --url "http://${BLDNAME}"
  ## ^^^^ this command is likely to change depending version & build config

  ## Save the names of the various branches/remotes   
  ## e.g. "sites/all/modules/civicrm" ==> "origin/4.4"
  ## e.g. "sites/all/modules/civicrm/drupal" ==> "origin/7.x-4.4"
  pushd "$BLDDIR"
    git scan branch-save > "$BRANCHLIST"
  popd
fi

## On every iteration, reset code+DB
pushd "$BLDDIR"
  ## Always start out with the latest canonical code from the proper branches
  git scan branch-load < "$BRANCHLIST"
  git scan up --hard

  ## Load the code identified by Jenkins as the target
  git cloneinto "$WORKSPACE/pr-repo" "$BLDDIR/civicrm"
  ## ^^^^ this command is likely to change depending on the particular repo used in this job

  ## Reinitialize the Drupal+Civi databases
  civibuild reinstall "$BLDNAME"
popd

## Run the tests
pushd "$BLDDIR"
  phpunit --junit-xml "$WORKSPACE/output/1.xml"
  ## ^^^^ this command is likely to change depending on how many tests you want to run for any PR
popd

4-new-commands.md

The script introduces a few new commands:

• git scan branch-save > FILE - Searches for git repos and notes the branch/remote being used
• git scan branch-load < FILE - For each sub-repo, call "git checkout"
• git scan up --force - Like the existing "git scan up", this gets the latest code. However, instead of passively avoiding conflicts (eg with "git pull --ff-only"), it aggressively works to match the upstream branch (eg deleting local changes and local commits)
• git cloneinto SRC_REPO TGT_REPO -- Given two existing repos, set TGT to exactly match the code from SRC.

Notably, these "git scan" helpers should work with any git-based build process (incl. composer, drush-make, buildkit, and bash scripts) and many different build-configurations (eg the "drupal-clean" which is just bare Drupal+Civi; or "drupal-demo" which has several modules/extensions preinstalled; or some third configuration with its own mix of modules).

5-random-thoughts.md

It seems undesirable to reproduce so much code between the job definitions -- only three statements are different (1: the initial build; 2: the cloneinto; 3: the test execution). Should find a better abstraction.