jenkins_slave.log

Started by user Michael de Silva
09:41:21 Connecting to https://api.github.com using bsodmike/******
Obtained Jenkinsfile from cbc695e524bef85e0bff0e0bc52b60a532784cec
[Pipeline] properties
[Pipeline] node
Running on Docker Builder (i-0c250582627987aa0) in /var/jenkins/workspace/x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Debug Slave User GUID)
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ echo Which user running this node:
Which user running this node:
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ whoami
ubuntu
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ echo Get UID:
Get UID:
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ id -u
1000
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ echo GID for docker group:
GID for docker group:
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ getent group docker
docker:x:999:ubuntu
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ echo Current groups:
Current groups:
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ groups
ubuntu adm dialout cdrom floppy sudo audio dip video plugdev netdev lxd
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Prep Build Tools img)
[Pipeline] checkout
Cloning the remote Git repository
Cloning repository git@github.com:bsodmike/jenkins-simple-app.git
 > git init /var/jenkins/workspace/x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA # timeout=10
Fetching upstream changes from git@github.com:bsodmike/jenkins-simple-app.git
 > git --version # timeout=10
using GIT_SSH to set credentials simple-app Deploy Key
 > git fetch --tags --progress git@github.com:bsodmike/jenkins-simple-app.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url git@github.com:bsodmike/jenkins-simple-app.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url git@github.com:bsodmike/jenkins-simple-app.git # timeout=10
Fetching upstream changes from git@github.com:bsodmike/jenkins-simple-app.git
using GIT_SSH to set credentials simple-app Deploy Key
 > git fetch --tags --progress git@github.com:bsodmike/jenkins-simple-app.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin1.url git@github.com:bsodmike/jenkins-simple-app.git # timeout=10
Fetching upstream changes from git@github.com:bsodmike/jenkins-simple-app.git
using GIT_SSH to set credentials simple-app Deploy Key
 > git fetch --tags --progress git@github.com:bsodmike/jenkins-simple-app.git +refs/pull/*/head:refs/remotes/origin/pr/*
Checking out Revision cbc695e524bef85e0bff0e0bc52b60a532784cec (fix/jenkins_slave_docker_user)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f cbc695e524bef85e0bff0e0bc52b60a532784cec
 > git rev-list 0aa7f709a93fbc4bf15898e8ffdc32f51afc113b # timeout=10
[Pipeline] sh
[x_jenkins_slave_docker_user-RI5P2HVITDHDB26VODIKEPPWIXLML6YS6SLECBPGSU6GCUSHX5LA] Running shell script
+ docker build -t inertialbox/build-tools:snapshot .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.26/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=inertialbox%2Fbuild-tools%3Asnapshot&ulimits=null: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

slave.init

#!/bin/sh

# Author: Michael de Silva <michael@inertialbox.com>
#
# AMI: ami-40d28157 (Ubuntu 16.04 Xenial)
# - Remote user: ubuntu
# - Remote FS root: /var/jenkins
#
# Ref: https://gist.github.com/VJftw/0e3607c5ce7caa85ff9125658dc1fac5
sudo apt-get update
sudo apt-get install -y \
  apt-transport-https \
  ca-certificates \
  ca-certificates-java

echo "--> Setting up Docker repository"
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update && apt-cache policy docker-engine

echo "--> Installing Docker, Python, Java, Git, pv"
sudo apt-get -y update
sudo apt-get install -y \
    linux-image-extra-$(uname -r) \
    build-essential \
    docker-engine \
    openjdk-8-jdk \
    git \
    python3 \
    python3-dev \
    pv

echo "--> Adding user to Docker group"
sudo usermod -aG docker ubuntu

echo "--> Starting Docker"
sudo service docker start

echo ""
echo "--> Fix user/groups to run Docker"
echo ""

DOCKER_SOCKET=/var/run/docker.sock
DOCKER_GROUP=docker

if [ -S ${DOCKER_SOCKET} ]; then
    DOCKER_GID=$(stat -c '%g' ${DOCKER_SOCKET})
    sudo groupadd -for -g ${DOCKER_GID} ${DOCKER_GROUP}
    sudo usermod -aG ${DOCKER_GROUP} ubuntu
fi

echo "--> Installing pip, invoke, docker-py, invoke-docker-flow"
sudo curl -O https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
sudo pip install \
    invoke \
    docker-py \
    invoke-docker-flow

echo ""
echo "--> Creating Jenkins Workspace"
echo ""
sudo mkdir -p /var/jenkins
sudo chown -R ubuntu:ubuntu /var/jenkins
sudo chmod -R 770 /var/jenkins

echo ""
echo "--> Add Swap Space"
echo ""
SWAP=/mnt/swap1
if [ ! -f $SWAP ]; then
    sudo dd if=/dev/zero of=$SWAP bs=1M count=2K
    sudo chmod 600 $SWAP
    sudo mkswap $SWAP
    # add new swap to config and start using it
    echo "$SWAP none swap defaults 0 0" | sudo tee -a /etc/fstab
    sudo swapon -a
fi

echo "--> Restart Docker"
sudo service docker restart

echo "--> Docker Status"
sudo docker images

echo "--> Current groups"
sudo groups

echo ""
echo "--> Done!"
echo ""

slave.log

--> Creating Jenkins Workspace


--> Add Swap Space

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 30.8325 s, 69.7 MB/s
Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=636956c7-1962-42b1-b10d-48d32eb2ac3e
/mnt/swap1 none swap defaults 0 0
--> Restart Docker
--> Docker Status
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
--> Current groups
root

--> Done!

Mar 18, 2017 9:35:03 AM null
INFO: Verifying that java exists
openjdk full version "9-internal+0-2016-04-14-195246.buildd.src"
Mar 18, 2017 9:35:04 AM null
INFO: Copying slave.jar
Mar 18, 2017 9:35:09 AM null
INFO: Launching slave agent (via Trilead SSH2 Connection): java  -jar /tmp/slave.jar
<===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 2.62
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

The plot thickens!

Once the ec2 slave starts to process the Jenkinsfile within the repo pulled onto the Jenkins slave notice how it does not list the docker group for the ubuntu user.

I tried running either sudo su - $USER or sudo newgrp docker at the end of the slave init script based on this advice — however, this would just cause the slave to hang indefinitely. Basically, we need to re-initialise the group assignment within the existing session/shell.

At the moment, I can only get the docker run to work from within the slave agent (whilst parsing the Jenkinsfile that is) if I disconnect and re-connect to the slave EC2 agent, simply because Jenkins does this as a new SSH connection, and logs into a new session.

Solved by taking a different approach — the ami used for the slave is now built with a basic install of Ubuntu LTS + Docker CE; this simplifies the init script to only install dependencies needed for the final build on the slave.