branch_deployment_spec.md

Overview

compudeploy creates, deploys and updates CiviCRM websites

Requirements

• The creation command should take a single repository URL
• The host for the site should be flexible: local host, local vagrant, local docker, remote host, AWS
• The type of site should be flexible: a profile or a "full site" repository

Commands

Add Server

Adds a server configuration for deployment

compubuild add-server --type=remote --name=testing1 --target=<target>

• name: The name of the configuration
• type: What kind of server is it. For example jenkins, remote, aws, local or docker
• target: Not required for all types, but if a server is remote points to where it is located

Create

Creates a new site

compubuild create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --name=testing1 \
  --git-ref=CGP-my-feature-branch \
  --git-merge-ref=CGP-my-workstream-branch \
  --cms-db=./dumps/drupal-test.sql \
  --civi-db=./dumps/civi-test.sql

• name: The name of the site
• server: Which server it should be deployed to
• url: The URL for the site
• build-dir: Where on the server the site should be created
• repo: What repository to use to create the site
• git-ref: Optional branch of the repository to check out before building
• git-merge: Optional branch to merge into git-ref before building
• cms-db: Allows import of a certain database before running upgraders for Drupal. Supports local and remote files
• civi-db: Allows import of a certain database before running upgraders for CiviCRM. Supports local and remote files

Deploy

Deploys code changes to an existing site

compubuild deploy
  --name=testing1 \
  --git-ref=CGP-my-feature-branch \
  --git-merge-ref=CGP-my-workstream-branch
  --skip-upgraders \
  --cms-db=scp://123.23.234.42/dumps/drupal.mysql \
  --civi-db=scp://123.23.234.42/dumps/civicrm.mysql

• name: The name of the site
• git-ref: Optional branch of the repository to check out
• git-merge: Optional branch to merge into git-ref
• skip-upgraders: Whether to run the Drupal and CiviCRM upgraders during deployment
• cms-db: Allows import of a certain database before running upgraders for Drupal. Supports local and remote files
• civi-db: Allows import of a certain database before running upgraders for CiviCRM. Supports local and remote files

Destroy

Destroy all resources related to a site: db, files, vhosts

compubuild destroy --name=mysite

• name: The name of the site to destroy

Snapshot

Takes a snapshot of the database

compubuild snapshot --name=mysite --snapshot-name=pre-release

• name: the name of the site
• snapshot-name: the name to store the snapshot as, defaults to "default"

Restore

Restores the databases using a snapshot name or defaults to "default" if none provided

compubuild restore --name=mysite --snapshot-name=pre-release

• name: the name of the site
• snapshot-name: the name to store the snapshot as, defaults to "default"

Exec

Executes a command on a site and returns the result

compubuild exec --name=mysite --dir=profiles/compuclient --cmd=phpunit4

• name: The name of the site
• dir: The directory from where the command should be executed
• cmd: The command to run

Use Cases

I want to create a local development site for a client site

add-server --type=local --name=local
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/home/michael/sites/mysite \
  --url=mysite.local \
  --server=local \
  --name=local-mysite

I want to create a local development site using Vagrant

add-server --type=vagrant --name=vagrant1
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/home/michael/sites/mysite \
  --url=mysite.local \
  --server=vagrant1 \
  --name=local-mysite

I want to create a client site on a testing server

add-server --type=remote --name=testing1 --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --name=testing1

I want to do manual testing of a PR for a client site

add-server --type=remote --name=testing1 --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --git-ref=CGP-my-new-branch \
  --name=pr221 

I want to run automated tests for a PR for a client site

add-server --type=remote --name=testing1 --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --git-ref=CGP-my-new-branch \
  --name=pr221 
exec --dir=/var/www/mysite/profiles/compuclient phpunit4

I want to deploy a site permanently to AWS

add-server --type=aws --name=aws1
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=aws1

I want to deploy a site permanently to a remote host

add-server --type=remote --name=mysite --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=mysite 

I want to upgrade an existing remote site

add-server --type=remote --name=mysite --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=mysite 
  --name=mysite
deploy --name=mysite --git-ref=1.1.0

I want to make a backup before release, try release and restore site if it fails

add-server --type=remote --name=mysite --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --name=mysite \
  --server=mysite \
  --git-ref=1.0.0
snapshot --name=mysite --name=pre-relase
deploy --name=mysite --git-ref=1.1.0 #returns failure
restore --name=mysite --name=pre-release
deploy --name=mysite --git-ref=1.0.0

I want to deploy a branch to a test site, using a dump of the database

add-server --type=remote --name=mysite --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --git-ref=CGP-my-new-branch \
  --name=pr221 \
  --cms-db=./../dumps/brochure_drupal.mysql \
  --civi-db=./../dumps/brochure_civicrm.mysql

I want to deploy a branch to a test site, using the latest anonymized version of the database

add-server --type=remote --name=dev1 --type=aws
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --server=testing1 \
  --git-ref=CGP-my-new-branch \
  --name=pr221 \
  --cms-db=scp://123.23.234.42/dumps/drupal.mysql \
  --civi-db=scp://123.23.234.42/dumps/civicrm.mysql

I want to clean up an old site to save memory

add-server --type=remote --name=mysite --target=184.112.23.23
create --repo=http://bitbucket.org/compucorp/mysite \
  --build-dir=/var/www/mysite \
  --url=mysite.civihr.net \
  --server=testing1 \
  --git-ref=CGP-my-new-branch \
  --name=pr221 
destroy pr221

Milestones

• [] create brochure-site from master in Vagrant
• [] create brochure-site from master in Docker-
• [] create profile site from compuclient master in Vagrant
• [] create brochure-site and use imported db dumps
• [] create brochure-site based of different branch
• [] create brochure-site with different branch and merge staging
• [] create profile site in remote host

@mickadoo a few questions:

• What kind of configuration will be created by the add-server command?
• Does the tool really need to support all of these deployment targets? I think that we can limit the target to only local if we can guarantee that we'll always use it on the server where the site will run. This will probably make things much much simpler
• I'd suggest removing the git- prefix from the params name. It is very unlikely, but we might decide to not user git in the future.
• Keep in mind that the branch structure might be three levels deep (staging -> epic -> feature) and we will probably want all these merged when deploying feature. My suggestion is to have three params: base-branch, workstream-branch and branch, where staging would be the base-branch, epic would be the workstream-branch and feature the branch. This can also be useful when you want to test a hotfix, which can be merged to both master and staging
• Why would I want to skip upgraders during a deployment? Is this for when we need to add some data to the site before the upgraders run so that we can check if they actually work?
• What do you think of making the create command to, optionally, accepted a tarball with the site instead of a repo URL?

@davialexandre

What kind of configuration will be created by the add-server command?

For local sites nothing happens, for vagrant or remote sites it just stores configuration for where a server is located. For AWS or other cloud providers I thought it could be an option to spin up a new instance using their API. In this way we would have the option to create new instances on the fly when a PR came in and destroy them after a PR is merged.

I think (based on your comment below about "we'll always use it on the server where the site will run") that maybe you're thinking this is a direct replacement for civibuild. I'd like it to be more than that; a tool to allow you to clone a client site locally and set it up using a database dump, allow you to deploy to multiple servers, potentially orchestrating deployments to all client sites, and also accept requests from webhooks to automatically deploy a certain branch for a client.

Does the tool really need to support all of these deployment targets? I think that we can limit the target to only local if we can guarantee that we'll always use it on the server where the site will run. This will probably make things much much simpler

Maybe some of them are too much, I'll probably remove Docker for now but originally added it because I wanted to test this with Docker first. I REALLY don't want to install all the shitty old versions of MySQL, PHP and all the other junk I need to get D7 + CiviCRM running on my host machine so I created a Dockerfile with all the libraries I needed and mounted the tool inside it, using the docker container as my "local"

I'd suggest removing the git- prefix from the params name. It is very unlikely, but we might decide to not user git in the future.

Sure I can do that, although I'm not sure if just ref is enough information to go on. Originally I would have thought branch would be fine, but I wanted something to show the user that any git reference (branch / commit hash / tag) is fine. Any suggestions?

Keep in mind that the branch structure might be three levels deep (staging -> epic -> feature) and we will probably want all these merged when deploying feature. My suggestion is to have three params: base-branch, workstream-branch and branch, where staging would be the base-branch, epic would be the workstream-branch and feature the branch. This can also be useful when you want to test a hotfix, which can be merged to both master and staging

I was planning to always merge staging in but it might be better to change this to make it more configurable. I would like to keep this as flexible as possible to cater for all workflow. Do you think making the merge reference a comma separated list would be an option? Something like --git-ref=CGP-111-myfeature --merge-refs=staging,CGP-120-myworkstream?

Why would I want to skip upgraders during a deployment? Is this for when we need to add some data to the site before the upgraders run so that we can check if they actually work?

Exactly, I'm just thinking of when we actually need to test upgraders for CiviHR how we need to do a manual deployment. I thought it would be nice if we added an option to not run the upgraders and also make it configurable whether we want to reset using a snapshot or not.

What do you think of making the create command to, optionally, accepted a tarball with the site instead of a repo URL?

I haven't used tarballs much, I know sometimes they're used to reduce the code in production but if we are going to always use them do you think you could give me a bit more information on why we need to support them? From my (basic) understanding it seems cleaner to just use git to checkout code. I checked a few articles, seems to be no clear consensus on which is better, but I'm happy to add support for tarball deployment instead of git.

1.For Creating a new site. Usually we need the anonymized db here if its a development server, or just the ability to create databases. In any case you need to provide with database credentials, perhaps information that needs to be provides in the "add server" step for the root mysql user and the user and databases that will be created under the create new site stage.

2.For Snapshots as i understand it you will be backups of directory and db saving them with a date or name stamp so you can restore right?

3.Should we be supplying git credentials or should they exist on the server?