Skip to content

Instantly share code, notes, and snippets.

@cvega

cvega/README.md Secret

Last active May 18, 2020 00:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cvega/66e0e5a2815e9b923da2c616f74dea7e to your computer and use it in GitHub Desktop.
Save cvega/66e0e5a2815e9b923da2c616f74dea7e to your computer and use it in GitHub Desktop.
Download ZIP
SonarQube
Raw
README.md

Basic Jenkins and Sonarqube setup (05/15/2020)

Just some notes I put together to help keep me better organized. No guarentees made.

First things first, bring up an instance some where some how. We need access to root, including ports 22, and 443. Once that's up point your DNS A records to the public IP.

jenkins.hauntedmansion.io. 300	IN	A	52.11.156.122
sonar.hauntedmansion.io.   300	IN	A	52.11.156.122

I'm running in EC2 and using Amazon Linux. I followed these steps to bring up a docker environment which worked perfectly for my needs:

sudo yum update -y
sudo yum install docker git -y

sudo sysctl -w vm.max_map_count=262144
sudo sysctl -w fs.file-max=100000

sudo sh -c "echo 'vm.max_map_count=262144' >> /etc/sysctl.conf"
sudo sh -c "echo 'fs.file-max=100000' >> /etc/sysctl.conf"
sudo sysctl -p

sudo usermod -a -G docker ec2-user
sudo sh -c "echo \"OPTIONS='--default-ulimit nofile=65536:65536'\" >> /etc/sysconfig/docker"

newgrp docker
sudo service docker start
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

Once your docker environment is online you can run a docker compose file. This will bring up 4 services and multiple mount points. Jenkins, Sonarqube, Postgres, and Nginx. All traffic is proxied via Nginx to the specific container. This configuration also provides a cert-bot capability and will automatically setup TLS once it's running and your A records can be resolved by the certificate issuer (Let's Encrypt).

Before running the docker compose file you'll want to have a couple of folders set up. I fought a battle with the Sonarqube container and lost. In a nutshell, it's installation expands itself (tar.gz) during the docker build. This prevents you from mounting the volume locally which I wanted to do. Sure, you can mount the volume, but now that dir is empty and Sonarqube cant function. So I ended up downloading the of version of Sonarqube I needed, unzipping it into my local directory and then using that as the mount point, I also had to copy the dockers run.sh script. I do not like this container. I may suggest that their entry script determine if the application is installed and if not expand on first run.

You can get the source here:

https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.1.zip

Sonarqube also has weird permissions so you may need to

chown -R 999:docker sonarqube/

I also created a conf.d directory for nginx with two files:

For Jenkins (conf.d/jenkins.conf):

server {
    listen              443 ssl;
    server_name jenkins.hauntedmansion.io;
    ssl_certificate     /etc/letsencrypt/live/jenkins.hauntedmansion.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins.hauntedmansion.io/privkey.pem;

    client_max_body_size 20M;

    location / {
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_pass         http://jenkins:8080;
        proxy_read_timeout 90;
        proxy_redirect     http://127.0.0.1:8080 https://jenkins.hauntedmansion.io;
    }
}

For Sonarqube: (conf.d/sonar.conf):

server {
    listen              443 ssl;
    server_name sonar.hauntedmansion.io;
    ssl_certificate     /etc/letsencrypt/live/sonar.hauntedmansion.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/sonar.hauntedmansion.io/privkey.pem;

    client_max_body_size 20M;

    location / {
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_pass         http://sonarqube:9000;
        proxy_read_timeout 90;
        proxy_redirect     http://127.0.0.1:9000 https://sonar.hauntedmansion.io;
    }
}

So far our project directory looks like this:

drwxr-xr-x  7 ec2-user       docker    119 May 17 21:54 .
drwx------  5 ec2-user       ec2-user  138 May 17 19:06 ..
drwxr-xr-x  2 ec2-user       docker     44 May 17 19:06 conf.d
drwxr-xr-x 12 libstoragemgmt docker    191 May 16 20:01 sonarqube

You will also need to copy that docker run.sh to sonarqube/bin. You can grab it here:

https://raw.githubusercontent.com/SonarSource/docker-sonarqube/82aff0792a91eb1a58baea66e7651c8e69fa18ce/7.9.1-community/run.sh

Last but not least, the docker-compose.yml I used to bring it all up.

version: "3"

services:

  nginx:
    container_name: nginx-certbot
    restart: unless-stopped
    image: staticfloat/nginx-certbot
    networks:
      - local
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
    environment:
      CERTBOT_EMAIL: cvega@github.com
    volumes:
      - ./conf.d:/etc/nginx/user.conf.d:ro
      - letsencrypt:/etc/letsencrypt

  sonarqube:
    image: sonarqube:7.9.1-community
    user: "999"
    networks:
      - local
    environment:
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
      - SONARQUBE_JDBC_USERNAME=sonar
      - SONARQUBE_JDBC_PASSWORD=sonar
    volumes:
      - ./sonarqube:/opt/sonarqube

  db:
    image: postgres
    networks:
      - local
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - ./postgresql:/var/lib/postgresql
      - ./postgresql_data:/var/lib/postgresql/data

  jenkins:
    image: jenkins/jenkins:2.222.1
    networks:
      - local
    volumes:
      - ./jenkins:/var/jenkins_home

networks:
  local:

volumes:
  letsencrypt:

If all goes well you can run the following and visit your sites in a few moments:

docker-compose -f "docker-compose.yml" up --build

You end up with a directory that looks like this once the cluster is up, you can access all of the important pieces of the applications locally.

drwxr-xr-x  7 ec2-user       docker    119 May 17 21:54 .
drwx------  5 ec2-user       ec2-user  138 May 17 19:06 ..
drwxr-xr-x  2 ec2-user       docker     44 May 17 19:06 conf.d
-rw-r--r--  1 ec2-user       docker   1086 May 16 20:49 docker-compose.yml
drwxr-xr-x 23 ec2-user       docker   4096 May 17 19:47 jenkins
drwxr-xr-x  3 ec2-user       docker     18 May 16 19:37 postgresql
drwx------ 19 libstoragemgmt docker   4096 May 16 21:26 postgresql_data
drwxr-xr-x 12 libstoragemgmt docker    191 May 16 20:01 sonarqube

Last but not least, script format to make this even easier (not saying this is universal but it worked on EC2 base Amazon 2 Linux m5.xlarge instance)

#!/usr/bin/env bash

sudo yum update -y
sudo yum install docker git -y

sudo sysctl -w vm.max_map_count=262144
sudo sysctl -w fs.file-max=100000

sudo sh -c "echo 'vm.max_map_count=262144' >> /etc/sysctl.conf"
sudo sh -c "echo 'fs.file-max=100000' >> /etc/sysctl.conf"
sudo sysctl -p

sudo usermod -a -G docker ec2-user
sudo sh -c "echo \"OPTIONS='--default-ulimit nofile=65536:65536'\" >> /etc/sysconfig/docker"

newgrp docker
sudo service docker start
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

mkdir project
cd project
mkdir -p conf.d/

curl -L https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.1.zip -o sonarqube-7.9.1.zip
unzip sonarqube-7.9.1.zip
mv sonarqube-7.9.1/ sonarqube/
curl -L https://raw.githubusercontent.com/SonarSource/docker-sonarqube/82aff0792a91eb1a58baea66e7651c8e69fa18ce/7.9.1-community/run.sh
 -o sonarqube/bin/run.sh
sudo chown -R 999:docker sonarqube/

JENKINS="conf.d/jenkins.conf"
/bin/cat <<EOM >$JENKINS
server {
    listen              443 ssl;
    server_name jenkins.hauntedmansion.io;
    ssl_certificate     /etc/letsencrypt/live/jenkins.hauntedmansion.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins.hauntedmansion.io/privkey.pem;

    client_max_body_size 20M;

    location / {
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_pass         http://jenkins:8080;
        proxy_read_timeout 90;
        proxy_redirect     http://127.0.0.1:8080 https://jenkins.hauntedmansion.io;
    }
}
EOM

SONAR="conf.d/sonar.conf"
/bin/cat <<EOM >$SONAR
server {
    listen              443 ssl;
    server_name sonar.hauntedmansion.io;
    ssl_certificate     /etc/letsencrypt/live/sonar.hauntedmansion.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/sonar.hauntedmansion.io/privkey.pem;

    client_max_body_size 20M;

    location / {
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_pass         http://sonarqube:9000;
        proxy_read_timeout 90;
        proxy_redirect     http://127.0.0.1:9000 https://sonar.hauntedmansion.io;
    }
}
EOM

DOCKER="docker-compose.yml"
/bin/cat <<EOM >$DOCKER
version: "3"

services:

  nginx:
    container_name: nginx-certbot
    restart: unless-stopped
    image: staticfloat/nginx-certbot
    networks:
      - local
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
    environment:
      CERTBOT_EMAIL: cvega@github.com
    volumes:
      - ./conf.d:/etc/nginx/user.conf.d:ro
      - letsencrypt:/etc/letsencrypt

  sonarqube:
    image: sonarqube:7.9.1-community
    user: "999"
    networks:
      - local
    environment:
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
      - SONARQUBE_JDBC_USERNAME=sonar
      - SONARQUBE_JDBC_PASSWORD=sonar
    volumes:
      - ./sonarqube:/opt/sonarqube

  db:
    image: postgres
    networks:
      - local
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - ./postgresql:/var/lib/postgresql
      - ./postgresql_data:/var/lib/postgresql/data

  jenkins:
    image: jenkins/jenkins:2.222.1
    networks:
      - local
    volumes:
      - ./jenkins:/var/jenkins_home

networks:
  local:

volumes:
  letsencrypt:
EOM

docker-compose -f "docker-compose.yml" up --build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment