Instructions for getting GoCD running on Amazon Linux

gocd_on_amazon.md

This might help someone who wants to set up GoCD (https://www.gocd.io/) on an Amazon EC2 machine with the Amazon Linux AMI. I'm doing this from a Mac.

Preliminary notes

I'm not great at this stuff but I don't think I've done anything here terribly wrong. It will at least get you started. A person who is better at this sort of thing would probably use Docker.

Create the EC2 instance

In the Amazon console, select the Amazon Linux AMI. Choose the m4.xlarge instance type. I'm running the Go server and two agents on this machine. It is likely overkill -- I'm going to monitor the machine and downgrade if it makes sense, or install a third agent.

On the 'Add Storage' step, add a new 8 GB volume with the device /dev/sdb selected. We will use this to store Go's artifacts, as their docs recommend storing artifacts away from the rest of Go.

Add security group rules for HTTP / TCP / 80 / 0.0.0.0/0 and HTTPS / TCP / 443 / 0.0.0.0/0.

When you go to launch the instance, a popup will ask you to create a new key pair or select an existing one. To create a new one, give it a name (e.g., 'gocd') and click the download button. Save it in your Downloads folder. Click 'Launch Instance'.

While the instance is launching, copy the private key file to your ssh directory and restrict the permissions:

cp ~/Downloads/gocd.pem ~/.ssh
chmod 400 ~/.ssh/gocd.pem

Go grab the 'IPv4 Public IP' from your instance's description in the AWS console. Edit the file ~/.ssh/config and add this:

Host gocd
  HostName [your IPv4 Public IP]
  User ec2-user
  IdentityFile ~/.ssh/gocd.pem

This will let you type ssh gocd to log into your machine. Do that now to make sure everything is working. Apply available updates by typing sudo yum update.

Setup DNS

This is optional but the Apache instructions below will need to be changed if you choose not to do it.

Go to the Elastic IPs section in the Amazon console and allocate a new address. Associate it to your GoCD EC2 instance. This wipes out your existing public IP address, so edit ~/.ssh/config and put your new Elastic IP in for the HostName.

Use your DNS provider to set up a new A record to point gocd.yourdomain.com at the Elastic IP address.

Install the Go server

First install Java 1.8 with sudo yum install -y java-1.8.0-openjdk. Then uninstall Java 1.7 with sudo yum remove java-1.7.0-openjdk. My understanding is that the order matters because uninstalling Java 1.7 first will also uninstall dependencies that we want to hold onto.

Install the Go server:

echo "
[gocd]
name     = GoCD YUM Repository
baseurl  = https://download.gocd.io
enabled  = 1
gpgcheck = 1
gpgkey   = https://download.gocd.io/GOCD-GPG-KEY.asc
" | sudo tee /etc/yum.repos.d/gocd.repo

sudo yum install -y go-server

Install and configure Apache httpd

sudo yum install -y httpd24
sudo service httpd start
sudo chkconfig httpd on

That last line is to make it start up automatically on system boot.

You should now be able to go to http://gocd.yourdomain.com and see the Apache HTTP server test page.

Make an httpd configuration file (sudo vi /etc/httpd/conf.d/gocd.conf) and put this in it:

<VirtualHost *:80>
  ServerName gocd.yourdomain.com

  ProxyPass         "/"  "http://localhost:8153/"
  ProxyPassReverse  "/"  "http://localhost:8153/"
  ProxyPreserveHost On
</VirtualHost>

Restart httpd: sudo service httpd restart.

Start the Go server: sudo /etc/init.d/go-server start

If you go to http://gocd.yourdomain.com, you should now see the Go server's 'add pipeline' page.

We will add SSL capability later.

Password-protect your Go server

I recommend using the oauth plugin (see https://github.com/gocd-contrib/gocd-oauth-login), but even if you use that you're instructed to first set up an admin account using Go's built-in password mechanism. Here's how:

sudo htpasswd -c -s /etc/go/passwd yourusername

That will prompt you to enter a password. Once you're done, in your Go server UI go to the 'Server Configuration' page and set the Password File Path to /etc/go/passwd. Save and it will redirect you to the authentication page. Check that you can log in.

Prepare your second EBS volume

Based on the instructions at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html, prepare the xvdb volume and mount it at the '/artifacts' dir:

sudo mkfs -t ext4 /dev/xvdb
sudo mkdir /artifacts
sudo mount /dev/xvdb /artifacts

To make this happen on bootup, back up the fstab file (sudo cp /etc/fstab /etc/fstab.orig) and then grab the UUID that sudo file -s /dev/xvdb spits out. Edit the fstab file sudo vi /etc/fstab and put this line at the end:

UUID=[your_uuid]       /artifacts   ext4    defaults,nofail        0       2

Run sudo mount -a and if it doesn't print anything out, it worked.

In the Go server configuration UI, set the artifacts directory to be /artifacts. Make the go user the owner: sudo chown go:go /artifacts. Then restart the Go server: sudo /etc/init.d/go-server restart.

Installing Go agents

The first agent is easy to install: sudo yum install -y go-agent. Start it: sudo /etc/init.d/go-agent. You should now see it listed at http://gocd.yourdomain.com/go/agents

For the second one, do this (based on https://docs.gocd.io/current/advanced_usage/admin_install_multiple_agents.html)

sudo ln -s /etc/init.d/go-agent /etc/init.d/go-agent-1
sudo ln -s /usr/share/go-agent /usr/share/go-agent-1
sudo cp /etc/default/go-agent /etc/default/go-agent-1
sudo mkdir /var/{lib,log}/go-agent-1
sudo chown go:go /var/{lib,log}/go-agent-1
sudo chgrp go /etc/default/go-agent-1

Then start it: sudo /etc/init.d/go-agent-1

Make it start automatically on boot: sudo chkconfig go-agent-1 on.

SSL

This is based on the instructions here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-an-instance.html

Install mod_ssl: sudo yum install -y mod24_ssl

Then get an SSL cert for gocd.yourdomain.com and upload the files to the server:

scp -i ~/.ssh/gocd.pem gocd_yourdomain_com.crt gocd_yourdomain_com.key gocd_yourdomain_com_bundle.pem ec2-user@yourpublicip:~

Copy your certificate files to the appropriate places and fix up their permissions:

sudo cp gocd_yourdomain_com.key /etc/pki/tls/private
sudo cp gocd_yourdomain_com.crt /etc/pki/tls/certs
sudo cp gocd_yourdomain_com_bundle.pem /etc/pki/tls/certs

sudo chmod 600 /etc/pki/tls/certs/gocd_yourdomain_com.crt
sudo chmod 644 /etc/pki/tls/certs/gocd_yourdomain_com_bundle.pem
sudo chmod 600 /etc/pki/tls/private/gocd_yourdomain_com.key

Edit the relevant lines in /etc/httpd/conf.d/ssl.conf to match:

SSLCertificateFile /etc/pki/tls/certs/gocd_yourdomain_com.crt
SSLCertificateKeyFile /etc/pki/tls/private/gocd_yourdomain_com.key
SSLCertificateChainFile /etc/pki/tls/certs/gocd_yourdomain_com_bundle.pem

Edit /etc/httpd/conf.d/gocd.conf and add this block at the end:

<VirtualHost *:443>
  ServerName gocd.yourdomain.com

  # Proxy everything over to the GoCD server
  ProxyPass         /  http://localhost:8153/
  ProxyPassReverse  /  http://localhost:8153/
  ProxyPreserveHost On
  RequestHeader set X-Forwarded-Proto "https"

  <Location />
    Order allow,deny
    Allow from all
  </Location>

  # SSL configuration
  SSLEngine on

  SSLCertificateFile /etc/pki/tls/certs/gocd_yourdomain_com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/gocd_yourdomain_com.key
  SSLCertificateChainFile /etc/pki/tls/certs/gocd_yourdomain_com_bundle.pem
</VirtualHost>

Restart httpd: sudo service httpd restart

Going to https://gocd.yourdomain.com should work.

Next steps

You'll need to install whatever software your agents need to run your tests (e.g., a database, ruby, etc.) and configure pipelines in Go.

I tried this on Amazon Linux 2 AMI as-well. The only thing I had to change was to use service go-server to manage the server instead of /etc/init.d/go-server