Rails, Docker, Kubernets & ECS

rails_docker_ecs.txt

# source code: https://github.com/Apress/deploying-rails-w-docker/tree/master/webapp

$ sudo docker run -it --rm --user "$(id -u):$(id -g)" -v "$PWD":/usr/src/app -w /usr/src/app rails rails new --skip-bundle --api --database postgresql webapp
cd webapp
$ sudo docker run --rm -v "$PWD":/usr/src/app -w /usr/src/app ruby:2.3 bundle install

-----------------

## adding pushion passenger

/webapp

$ touch webapp.conf
server {
  listen 80;
  server_name _;
  root /home/app/webapp/public;
  passenger_enabled on;
  passenger_user app;
  passenger_ruby /usr/bin/ruby2.3;
}

-----------------
$ touch rails-env.conf

env SECRET_KEY_BASE;
env DATABASE_URL;
env DATABASE_PASSWORD;

-----------------
$ touch Dockerfile

FROM phusion/passenger-ruby23:0.9.19

# Set correct environment variables.
ENV HOME /root

# Use baseimage-docker's init process.
CMD ["/sbin/my_init"]

# Additional packages: we are adding the netcat package so we can
# make pings to the database service
RUN apt-get update && apt-get install -y -o Dpkg::Options::="--force-confold" netcat

# Enable Nginx and Passenger
RUN rm -f /etc/service/nginx/down

# Add virtual host entry for the application. Make sure
# the file is in the correct path
RUN rm /etc/nginx/sites-enabled/default
ADD webapp.conf /etc/nginx/sites-enabled/webapp.conf

# In case we need some environmental variables in Nginx. Make sure
# the file is in the correct path
ADD rails-env.conf /etc/nginx/main.d/rails-env.conf

# Install gems: it's better to build an independent layer for the gems
# so they are cached during builds unless Gemfile changes
WORKDIR /tmp
ADD Gemfile /tmp/
ADD Gemfile.lock /tmp/
RUN bundle install

# Copy application into the container and use right permissions: passenger
# uses the app user for running the application
RUN mkdir /home/app/webapp
COPY . /home/app/webapp
RUN usermod -u 1000 app
RUN chown -R app:app /home/app/webapp
WORKDIR /home/app/webapp


# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

EXPOSE 80

-----------------
$ touch setup.sh

#!/bin/sh

echo "Waiting PostgreSQL to start on 5432..."

while ! nc -z postgres 5432; do
  sleep 0.1
done

echo "PostgreSQL started"

bin/rails db:migrate

-----------------
$ chmod +x setup.sh
$ touch docker-compose.yml

version: '2'
services:
  webapp_setup:
    build: .
    depends_on:
      - postgres
    environment:
      - PASSENGER_APP_ENV=development
    entrypoint: ./setup.sh
  webapp:
    container_name: webapp
    build: .
    depends_on:
      - postgres
      - webapp_setup
    environment:
      - PASSENGER_APP_ENV=development
    ports:
      - "80:80"
    volumes:
      - .:/home/app/webapp
  postgres:
    image: postgres:9.5.3
    environment:
      - POSTGRES_PASSWORD=mysecretpassword
      - POSTGRES_USER=webapp
      - POSTGRES_DB=webapp_development
    volumes_from:
      - postgres_data
  postgres_data:
      image: postgres:9.5.3
      volumes:
        - /var/lib/postgresql/data
      command: /bin/true


-----------------
# start docker deamon in local
$ sudo systemctl start docker
$ sudo docker-compose up
# optional 
$ docker-compose build


-----------------
/config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  user: webapp
  password: mysecretpassword
  host: postgres
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

development:
  <<: *default
  database: webapp_development
  
-----------------
# start docker deamon in local
$ sudo systemctl start docker
$ sudo docker-compose up
# optional 
$ sudo docker-compose build

-----------------
# test
curl -I localhost


-----------------
## to run the environment, always do it with this command
$ sudo systemctl start docker
$ rvm use 2.7.1
$ sudo docker-compose up
$ curl -I localhost
## just for when I have kubernetes ready (not neccesary at the beginning)
$ minikube start --vm-driver=virtualbox

-----------------
## adding a new resource
sudo docker-compose run --rm webapp bin/rails g scaffold articles title:string body:text

## That’s going to create the necessary files for the scaffold, and since we have a volume
for this project, we also have those files locally. That’s a workflow you have to learn when
working with containers. Locally you’re only editing files, but all the tasks and executions
happen inside the container, so you normally want to run commands with docker-
compose and add the --rm flag so the container is deleted after, or you may want to keep
an open connection inside the container by using docker exec -it webapp bash, and
run all your commands from there.

-----------------
## migrating database
sudo docker-compose run --rm webapp bin/rails db:migrate

-----------------
## creating the test database
sudo docker-compose run --rm webapp bash -c "RAILS_ENV=test bin/rails db:create"
## run the tests
sudo docker-compose run --rm webapp bash -c "RAILS_ENV=test bin/rake"

-----------------
## Let’s test the end point for creating articles.
$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my first article","body":"Lorem ips\ um dolor sit amet, consectetur adipiscing elit..."}' http://localhost/articles

-----------------
## rails console
sudo docker-compose exec webapp bin/rails c

-----------------
# installing a new gem
$ # kill the process with C-c (Control + c) a)
# Gemfile
gem "figaro"
# console
$ sudo docker-compose build
$ sudo docker-compose up
$ curl -I localhost


-----------------
## Log Issues with Docker
config/application.rb
config.logger = Logger.new(STDOUT)
$ # kill the process with C-c (Control + c) a)
$ sudo docker-compose build
$ sudo docker-compose up -d && sudo docker-compose logs -f
## in another tab
$ curl -I localhost

-----------------
## Pushing code to DockerHub (This will help us with Kubernets and AWS)
$ sudo docker login
username:
password:

## go to DockerHub, login and create a public repository with the name of the app: "webapp" in this case
### WARNING: Image needs to be public, if not Kubernetes is going to do an error. So make image public

$ touch .dockerignore

# Ignore bundler config.
/.bundle
# Ignore all logfiles and tempfiles.
/log/*
/tmp/*
!/log/.keep
!/tmp/.keep
# Ignore Byebug command history file.
.byebug_history

$ git init
$ git add .
$ git commit -m
$ LC=$(git rev-parse --short HEAD)
$ sudo docker build -t your_dockerhub_username/webapp:${LC} .
$ sudo docker push your_dockerhub_username/webapp:${LC}

-----------------
## Those steps for generating an image and pushing it to DockerHub can be automated easily with a bash script
$ touch push.sh
$ chmod +x push.sh

#!/bin/sh
LC=$(git rev-parse --short HEAD)
sudo docker build -t danielmorales1202/webapp:${LC} .
sudo docker push danielmorales1202/webapp:${LC}

$ ./push.sh

-----------------
## Installing AWS CLI
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
$ aws --version

## Configuring the AWS CLI
## get tokens: http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html

$ aws configure

AWS Access Key ID [None]: xxxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxx
Default region name [None]: us-east-1
Default output format [None]: json

## credentials & config under:
~/.aws/credentials
/.aws/config

## test
$ aws ec2 describe-vpcs --region us-east-1 --query="Vpcs[*].{ID:VpcId,tags:Tags[0]}"
$ aws rds describe-db-instances --db-instance-identifier webapp-postgres --query 'DBInstances[*].{Status:DBInstanceStatus}'
$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-a0e9c0c7" --query="Subnets[*].SubnetId"

-----------------
### Kubernetes
$ mkdir -p kube
$ mkdir -p kube/deployments
$ mkdir -p kube/jobs

## deployments (PostgreSQL)
$ touch kube/deployments/postgres-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: webapp
spec:
  ports:
    - port: 5432
  selector:
    app: webapp
    tier: postgres
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  labels:
    app: webapp
spec:
  selector:
    matchLabels:
      app: webapp
      tier: postgres
  template:
    metadata:
      labels:
        app: webapp
        tier: postgres
    spec:
      containers:
      - image: postgres:9.5.3
        name: postgres
        env:
        - name: POSTGRES_PASSWORD
          value: mysecretpassword
        - name: POSTGRES_USER
          value: webapp
        - name: POSTGRES_DB
          value: webapp_development
        ports:
        - containerPort: 5432
          name: postgres
          
-----------------
#Job
$ touch kube/jobs/setup-job.yaml
  
apiVersion: batch/v1
kind: Job
metadata:
  name: setup
spec:
  template:
    metadata:
      name: setup
    spec:
      containers:
      - name: setup
        image: danielmorales1202/webapp:1ca63f8
        command: ["/bin/bash", "./setup.sh"]
        env:
        - name: PASSENGER_APP_ENV
          value: development
      restartPolicy: Never
      
-----------------
$ touch kube/deployments/webapp-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  ports:
    - port: 80
  selector:
    app: webapp
    tier: frontend
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
      tier: frontend
  template:
    metadata:
      labels:
        app: webapp
        tier: frontend
    spec:
      containers:
      - image: danielmorales1202/webapp:1ca63f8
        name: webapp
        env:
        - name: PASSENGER_APP_ENV
          value: development
        ports:
        - containerPort: 80
          name: webapp
        imagePullPolicy: Always

-----------------
## Install & Run Minikube
$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
$ kubectl version --client

Install virtualbox following this tutorial: https://websiteforstudents.com/install-virtualbox-latest-on-ubuntu-16-04-lts-17-04-17-10/

$ minikube start --vm-driver=virtualbox
$ minikube ip
$ minikube dashboard

## commands with erros in my machine
$ kubectl create -f kube/deployments/postgres-deployment.yaml
## solve with this
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl apply -f kube/deployments/postgres-deployment.yaml

## continue
$ kubectl describe deployment postgres
$ kubectl describe Pod postgres

## working
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl get Pods
$ kubectl create -f kube/deployments/webapp-deployment.yaml
$ kubectl get Pods
$ kubectl logs webapp-7946c57864-btkc4 (following a name of one of the webapp replicas)
$ minikube service webapp

## let's make a post request
$ minikube ip
$ kubectl describe service webapp
## example: http://192.168.99.100:30763/
$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my first article","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit..."}' http://192.168.99.100:30763/articles

-----------------
## Deploying to production

$ kubectl config current-context
$ export NUM_NODES=2
$ export NODE_SIZE=t2.small
$ source ~/.zshrc
$ export KUBERNETES_PROVIDER=aws; curl -sS https://get.k8s.io | bash
$ kubectl config current-context
$ sudo docker-compose run --rm webapp bin/rake secret RAILS_ENV=production
## output Token: 7b66186fd493ac8302449d17........ copy this to config/secrets.ym

## in config/database.yml
production:
  <<: *default
  host: postgres
  database: webapp_production
  username: webapp
  password: mysecretpassword

## open the kube/deployments/postgres.yml and change:
env:
  - name: POSTGRES_PASSWORD
    value: mysecretpassword
  - name: POSTGRES_USER
    value: webapp
  - name: POSTGRES_DB
    value: webapp_production

$ kubectl apply -f kube/deployments/postgres-deployment.yaml
$ kubectl describe service postgres

## open the kube/jobs/setup-job.yaml and change:
env:
- name: PASSENGER_APP_ENV
  value: production
  
## new setup file
$ touch setup.production.sh
$ chmod +x setup.production.sh

#!/bin/sh
echo "Waiting PostgreSQL to start on 5432..."
while ! nc -z postgres 5432; do
  sleep 0.1
done
echo "PostgreSQL started"
bin/rails db:migrate RAILS_ENV=production

## open the kube/jobs/setup-job.yaml and change:
containers:
  - name: setup
    image: danielmorales1202/webapp:1ca63f8
    command: ["/bin/bash", "./setup.production.sh"]

## push
git add .
git commit -m "add production templates"
./push.sh
git rev-parse --short HEAD (#to get the new tag)

## open kube/jobs/setup-job.yaml and change the tag of the image
image: danielmorales1202/webapp:f5e0a11

$ kubectl delete -f kube/jobs/setup-job.yaml
$ kubectl create -f kube/jobs/setup-job.yaml

$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods

## open kube/deployments/webapp-deployment.yaml
apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  ports:
    - port: 80
  selector:
    app: webapp
    tier: frontend
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
      tier: frontend
  template:
    metadata:
      labels:
        app: webapp
        tier: frontend
    spec:
      containers:
      - image: danielmorales1202/webapp:f5e0a11
        name: webapp
        env:
        - name: PASSENGER_APP_ENV
          value: production
        ports:
        - containerPort: 80
          name: webapp
        imagePullPolicy: Always

## create / apply
$ kubectl apply -f kube/deployments/webapp-deployment.yaml
$ kubectl describe service webapp

-----------------
## Adding persistence
$ aws ec2 create-volume --region us-west-2 --availability-zone us-west-2a --size 10 --volume-type gp2
## "VolumeId": "vol-08abc774f44d1f8f1"
## open kube/deployments/postgres-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: webapp
spec:
  ports:
    - port: 5432
  selector:
    app: webapp
    tier: postgres
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  labels:
    app: webapp
spec:
  selector:
    matchLabels:
      app: webapp
      tier: postgres
  template:
    metadata:
      labels:
        app: webapp
        tier: postgres
    spec:
      containers:
      - image: postgres:9.5.3
        name: postgres
        env:
        - name: POSTGRES_PASSWORD
          value: mysecretpassword
        - name: POSTGRES_USER
          value: webapp
        - name: POSTGRES_DB
          value: webapp_production
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
          name: postgres
        volumeMounts:
          - name: postgres-persistent-storage
            mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-persistent-storage
          awsElasticBlockStore:
            volumeID: vol-08abc774f44d1f8f1
            fsType: ext4
## run
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl create -f kube/deployments/postgres-deployment.yaml
$ kubectl describe rs postgres
$ aws ec2 describe-volumes --volume-ids vol-08abc774f44d1f8f1 --region us-west-2
$ kubectl delete job/setup
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl create -f kube/deployments/postgres-deployment.yaml

-----------------
## Updating application
$ sudo docker-compose run --rm webapp bin/rails g migration AddSlugToArticles slug:string

## ojoo: I Need to open atom with root permission: 
/webapp
$ sudo atom .

# in articles#create

def create
  @article = Article.new(article_params)
  @article.slug = @article.title.parameterize

  if @article.save
    render json: @article, status: :created, location: @article
  else
    render json: @article.errors, status: :unprocessable_entity
  end
end

$ git add .
$ git commit -m ...
$ ./push.sh
$ git rev-parse --short HEAD

## change image code

  containers:
      - name: setup
        image: danielmorales1202/webapp:56b2244
      
$ kubectl delete jobs/setup
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl apply -f kube/deployments/webapp-deployment.yaml

$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my second article","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit..."}' http://a333dae17845a11e6b47b06103f11903-585648094.us-west-2.elb.amazonaws.com/articles

-----------------
### Automation Scripts
$ mkdir deploy
$ mv push.sh deploy

# inside deploy/push.sh add

#!/bin/sh
set -x

LC=$(git rev-parse --short HEAD)
sudo docker build -f Dockerfile -t danielmorales1202/webapp:${LC} .
sudo docker push danielmorales1202/webapp:${LC}
kubectl set image deployment webapp webapp=danielmorales1202/webapp:${LC}

## in console
$ kubectl delete jobs/setup

# Open the kube/jobs/setup-job.yaml and change the image to
image: pacuna/webapp:LAST_COMMIT


## create migrate fil
$ touch deploy/migrate.sh
$ chmod +x deploy/migrate.sh

## And add:
#!/bin/sh
set -x

# get latest commit hash
LC=$(git rev-parse --short HEAD)

# delete current migrate job
kubectl delete jobs/setup || true

# replace LAST_COMMIT with latest commit hash output the result to a tmp file
sed "s/webapp:LAST_COMMIT/webapp:$LC/g" kube/jobs/setup-job.yaml > setup-job.yaml.tmp

# run the updated tmp job in the cluster and then delete the file
kubectl create -f setup-job.yaml.tmp &&
  rm setup-job.yaml.tmp
  
## command
And that’s it! Now if you want to make changes to your code and deploy a new
version, you just have to commit your changes and then run deploy/push.sh. If you want
run migrations, you can just run deploy/migrate.sh. You can even build another script
that uses both for every deployment, which is what we will later do with Jenkins

#### COMMENTS ON THIS
- I NEED TO REPEAT THE WHOLE PROCESS FROM SCRATCH
- THIS NEW PROCESS NEEDS TO BE DONE IN A NEW FOLDER WITH A NEW PROJECT
- THE WAY TO DO THIS IS FOLLOWING THE ORIIGNAL BOOK AND THIS GIST AT THE SAME TIME AND CORRECTING THE ISSUES
- WHEN I CUT THIS PROCESS BECAUSE I HAD TO STOP MY STUDY AND I CONTINUE NEXT DAY, I HAVE TO HAVE IN MIND THAT I NEED TO RE-RUN SOME COMMANDS. THIS IS VERY IMPORTANT, AND CAN BREAK THE WHOLE PROCESS. SO I NEED TO DO ALL OF THIS THE SAME DAY
- THERE ARE TWO MAIN OPTIONS ACCORDING TO THE BOOK: ONE IS USING KUBERNETES AND THE OTHER IS USING INSTANCES FROM ECS AWS (THIS LATTER IS DIFFERENT FROM EC2). SO I NEED TO RUN EVERYTHING FIRST WITH KUBERNETES WITH ONE PROJECT AND NEXT DO IT WITH ECS
- BEFORE DOING ALL OF THIS FOR A REAL PROJECT I NEED TO KEEP THE SAME PROJECT MENTIONED ON THE BOOK (API ONLY)
- ONE I HAVE DONE ALL OF THIS I NEED TO DELETE CLUSTERS AND INTANCES FROM AWS, BECAUSE IS A PAID SERVICE