Skip to content

Instantly share code, notes, and snippets.

View claydanford's full-sized avatar
🌮
sup

Clay Danford claydanford

🌮
sup
View GitHub Profile
@claydanford
claydanford / s3-static-website.tf
Last active June 3, 2019 15:20
S3 Static Website Example
resource "random_uuid" "name" {}
resource "aws_s3_bucket" "bucket" {
bucket = "${var.application}-${random_uuid.name.result}"
acl = "public-read"
policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
@claydanford
claydanford / nmg-jenkins-api-key.sh
Created June 5, 2019 19:26
Jenkins master creates an API key, and publishes it to SSM parameter store.
#!/bin/bash
if [ ! -f "/var/lib/jenkins/api_key.txt" ]; then
CRUMB=$(curl -s http://localhost:8080/crumbIssuer/api/json --user admin:${admin_password} | jq -r .crumb)
API_KEY=$(curl -s -X POST http://localhost:8080/me/descriptorByName/jenkins.security.ApiTokenProperty/generateNewToken --user admin:${admin_password} --data newTokenName=agent-token -H "Jenkins-Crumb: $CRUMB" | jq -r .data.tokenValue)
echo $API_KEY > /var/lib/jenkins/api_key.txt
fi
aws ssm put-parameter --name "${api_ssm_parameter}" --value "$(cat /var/lib/jenkins/api_key.txt)" --type "SecureString" --overwrite --region ${aws_region}
@claydanford
claydanford / nmg-jenkins-agent-connection.yaml
Last active June 5, 2019 20:45
Jenkins agent retrieves the API key and master instance infromation.
#cloud-config
runcmd:
- MASTER_ID=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ${master_asg} --query 'AutoScalingGroups[0].Instances[0].InstanceId' --output text --region ${aws_region}) && echo $MASTER_ID > /opt/master_id.txt
- AGENT_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id) && echo $AGENT_ID > /opt/agent_id.txt
- MASTER_IP=$(aws ec2 describe-instances --instance-ids $(cat /opt/master_id.txt) --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text --region ${aws_region}) && echo $MASTER_IP > /opt/master_ip.txt
- API_KEY=$(aws ssm get-parameter --name "${api_ssm_parameter}" --with-decryption --region ${aws_region} --output text --query 'Parameter.Value') && echo $API_KEY > /opt/api_key.txt
@claydanford
claydanford / nmg-jenkins-agent-swarm.sh
Last active June 5, 2019 20:46
Jenkins agent connects via the SWARM plugin, or terminates.
#!/bin/bash
runuser -l jenkins -c "nohup java -jar /var/lib/jenkins/swarm-client-${swarm_version}.jar -master http://$(cat /opt/master_ip.txt):8080 -description $(cat /opt/agent_id.txt) -username admin -password $(cat /opt/api_key.txt) -name $(cat /opt/agent_id.txt) -labels $(cat /opt/agent_id.txt) -executors ${executors} -fsroot /tmp -retry 0 -disableClientsUniqueId" || aws ec2 terminate-instances --instance-ids $(cat /opt/agent_id.txt) --region ${aws_region}
@claydanford
claydanford / nmg-jenkins-scaling-out-agents.sh
Created June 5, 2019 20:47
Jenkins master node polls for busy executors and scales.
#!/bin/bash
for i in {1..3};
do
AGENT_ASG="$(aws autoscaling describe-auto-scaling-groups --region ${aws_region} | jq -r '.AutoScalingGroups[] | select(.AutoScalingGroupName | startswith("${application}-agent-")) | .AutoScalingGroupName')"
BUILD_ACTIVE_QUEUE_VALUE1="$(curl -s -m 60 -u 'admin:APIKEY' 'http://localhost:8080/computer/api/json' | jq -r '.busyExecutors')"
BUILD_ACTIVE_QUEUE_VALUE2="$(curl -s -m 60 -u 'admin:APIKEY' 'http://localhost:8080/jqs-monitoring/api/json' | jq -r '.buildQueue.numberOfJobs')"
BUILD_ACTIVE_QUEUE_VALUE=$(( $BUILD_ACTIVE_QUEUE_VALUE1 + $BUILD_ACTIVE_QUEUE_VALUE2 ))
@claydanford
claydanford / nmg-jenkins-scaling-in-agents.sh
Created June 5, 2019 20:48
Jenkins master node polls for idle executors and scales in.
#!/bin/bash
for i in {1..3};
do
AGENT_ASG="$(aws autoscaling describe-auto-scaling-groups --region ${aws_region} | jq -r '.AutoScalingGroups[] | select(.AutoScalingGroupName | startswith("${application}-agent-")) | .AutoScalingGroupName')"
BUILD_ACTIVE_QUEUE_VALUE1="$(curl -s -m 60 -u 'admin:APIKEY' 'http://localhost:8080/computer/api/json' | jq -r '.busyExecutors')"
BUILD_ACTIVE_QUEUE_VALUE2="$(curl -s -m 60 -u 'admin:APIKEY' 'http://localhost:8080/jqs-monitoring/api/json' | jq -r '.buildQueue.numberOfJobs')"
BUILD_ACTIVE_QUEUE_VALUE=$(( $BUILD_ACTIVE_QUEUE_VALUE1 + $BUILD_ACTIVE_QUEUE_VALUE2 ))
@claydanford
claydanford / nmg-jenkins-version.tf
Created June 5, 2019 20:49
The default variables for the Jenkins and SWARM versions.
variable "jenkins_version" {
description = "The version number of Jenkins to use on the master. Change this value when a new version comes out, and it will update the launch configuration and the autoscaling group."
default = "2.164.3"
}
variable "swarm_version" {
description = "The version of swarm plugin to install on the agents. Update by updating this value."
default = "3.15"
}
@claydanford
claydanford / nmg-jenkins-user-data-template.tf
Created June 5, 2019 20:50
The Terraform configuration, generating the agent user-data.
data "template_file" "agent_write_files" {
template = "${file("${path.module}/init/agent-write-files.cfg")}"
vars {
agent_logs = "${aws_cloudwatch_log_group.agent_logs.name}"
aws_region = "${var.region}"
executors = "${var.executors}"
swarm_version = "${var.swarm_version}"
}
}
@claydanford
claydanford / nmg-jenkins-terraform-cloud-init-template.tf
Created June 5, 2019 20:52
This data source puts rendered template parts together for generating cloud-init user-data.
data "template_cloudinit_config" "agent_init" {
gzip = false
base64_encode = false
part {
filename = "agent.cfg"
content_type = "text/cloud-config"
content = "${data.template_file.agent_write_files.rendered}"
}
@claydanford
claydanford / nmg-jenkins-external-userdata.tf
Created June 5, 2019 20:54
How to generate external user-data and pass it into the terrform module.
module "jenkins_ha_agents" {
source = "neiman-marcus/jenkins-ha-agents/aws"
snipped for brevity…
extra_agent_userdata = "${data.template_file.extra_agent_userdata.rendered}"
extra_agent_userdata_merge = "list(append)+dict(recurse_array)+str()"
extra_master_userdata = "${data.template_file.extra_master_userdata.rendered}"
extra_master_userdata_merge = "list(append)+dict(recurse_array)+str()"
}