Skip to content

Instantly share code, notes, and snippets.

@danielmoralesp
Last active Jun 26, 2021
Embed
What would you like to do?
Rails, Docker, Kubernets & ECS
# source code: https://github.com/Apress/deploying-rails-w-docker/tree/master/webapp
$ sudo docker run -it --rm --user "$(id -u):$(id -g)" -v "$PWD":/usr/src/app -w /usr/src/app rails rails new --skip-bundle --api --database postgresql webapp
cd webapp
$ sudo docker run --rm -v "$PWD":/usr/src/app -w /usr/src/app ruby:2.3 bundle install
-----------------
## adding pushion passenger
/webapp
$ touch webapp.conf
server {
listen 80;
server_name _;
root /home/app/webapp/public;
passenger_enabled on;
passenger_user app;
passenger_ruby /usr/bin/ruby2.3;
}
-----------------
$ touch rails-env.conf
env SECRET_KEY_BASE;
env DATABASE_URL;
env DATABASE_PASSWORD;
-----------------
$ touch Dockerfile
FROM phusion/passenger-ruby23:0.9.19
# Set correct environment variables.
ENV HOME /root
# Use baseimage-docker's init process.
CMD ["/sbin/my_init"]
# Additional packages: we are adding the netcat package so we can
# make pings to the database service
RUN apt-get update && apt-get install -y -o Dpkg::Options::="--force-confold" netcat
# Enable Nginx and Passenger
RUN rm -f /etc/service/nginx/down
# Add virtual host entry for the application. Make sure
# the file is in the correct path
RUN rm /etc/nginx/sites-enabled/default
ADD webapp.conf /etc/nginx/sites-enabled/webapp.conf
# In case we need some environmental variables in Nginx. Make sure
# the file is in the correct path
ADD rails-env.conf /etc/nginx/main.d/rails-env.conf
# Install gems: it's better to build an independent layer for the gems
# so they are cached during builds unless Gemfile changes
WORKDIR /tmp
ADD Gemfile /tmp/
ADD Gemfile.lock /tmp/
RUN bundle install
# Copy application into the container and use right permissions: passenger
# uses the app user for running the application
RUN mkdir /home/app/webapp
COPY . /home/app/webapp
RUN usermod -u 1000 app
RUN chown -R app:app /home/app/webapp
WORKDIR /home/app/webapp
# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
EXPOSE 80
-----------------
$ touch setup.sh
#!/bin/sh
echo "Waiting PostgreSQL to start on 5432..."
while ! nc -z postgres 5432; do
sleep 0.1
done
echo "PostgreSQL started"
bin/rails db:migrate
-----------------
$ chmod +x setup.sh
$ touch docker-compose.yml
version: '2'
services:
webapp_setup:
build: .
depends_on:
- postgres
environment:
- PASSENGER_APP_ENV=development
entrypoint: ./setup.sh
webapp:
container_name: webapp
build: .
depends_on:
- postgres
- webapp_setup
environment:
- PASSENGER_APP_ENV=development
ports:
- "80:80"
volumes:
- .:/home/app/webapp
postgres:
image: postgres:9.5.3
environment:
- POSTGRES_PASSWORD=mysecretpassword
- POSTGRES_USER=webapp
- POSTGRES_DB=webapp_development
volumes_from:
- postgres_data
postgres_data:
image: postgres:9.5.3
volumes:
- /var/lib/postgresql/data
command: /bin/true
-----------------
# start docker deamon in local
$ sudo systemctl start docker
$ sudo docker-compose up
# optional
$ docker-compose build
-----------------
/config/database.yml
default: &default
adapter: postgresql
encoding: unicode
user: webapp
password: mysecretpassword
host: postgres
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
<<: *default
database: webapp_development
-----------------
# start docker deamon in local
$ sudo systemctl start docker
$ sudo docker-compose up
# optional
$ sudo docker-compose build
-----------------
# test
curl -I localhost
-----------------
## to run the environment, always do it with this command
$ sudo systemctl start docker
$ rvm use 2.7.1
$ sudo docker-compose up
$ curl -I localhost
## just for when I have kubernetes ready (not neccesary at the beginning)
$ minikube start --vm-driver=virtualbox
-----------------
## adding a new resource
sudo docker-compose run --rm webapp bin/rails g scaffold articles title:string body:text
## That’s going to create the necessary files for the scaffold, and since we have a volume
for this project, we also have those files locally. That’s a workflow you have to learn when
working with containers. Locally you’re only editing files, but all the tasks and executions
happen inside the container, so you normally want to run commands with docker-
compose and add the --rm flag so the container is deleted after, or you may want to keep
an open connection inside the container by using docker exec -it webapp bash, and
run all your commands from there.
-----------------
## migrating database
sudo docker-compose run --rm webapp bin/rails db:migrate
-----------------
## creating the test database
sudo docker-compose run --rm webapp bash -c "RAILS_ENV=test bin/rails db:create"
## run the tests
sudo docker-compose run --rm webapp bash -c "RAILS_ENV=test bin/rake"
-----------------
## Let’s test the end point for creating articles.
$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my first article","body":"Lorem ips\ um dolor sit amet, consectetur adipiscing elit..."}' http://localhost/articles
-----------------
## rails console
sudo docker-compose exec webapp bin/rails c
-----------------
# installing a new gem
$ # kill the process with C-c (Control + c) a)
# Gemfile
gem "figaro"
# console
$ sudo docker-compose build
$ sudo docker-compose up
$ curl -I localhost
-----------------
## Log Issues with Docker
config/application.rb
config.logger = Logger.new(STDOUT)
$ # kill the process with C-c (Control + c) a)
$ sudo docker-compose build
$ sudo docker-compose up -d && sudo docker-compose logs -f
## in another tab
$ curl -I localhost
-----------------
## Pushing code to DockerHub (This will help us with Kubernets and AWS)
$ sudo docker login
username:
password:
## go to DockerHub, login and create a public repository with the name of the app: "webapp" in this case
### WARNING: Image needs to be public, if not Kubernetes is going to do an error. So make image public
$ touch .dockerignore
# Ignore bundler config.
/.bundle
# Ignore all logfiles and tempfiles.
/log/*
/tmp/*
!/log/.keep
!/tmp/.keep
# Ignore Byebug command history file.
.byebug_history
$ git init
$ git add .
$ git commit -m
$ LC=$(git rev-parse --short HEAD)
$ sudo docker build -t your_dockerhub_username/webapp:${LC} .
$ sudo docker push your_dockerhub_username/webapp:${LC}
-----------------
## Those steps for generating an image and pushing it to DockerHub can be automated easily with a bash script
$ touch push.sh
$ chmod +x push.sh
#!/bin/sh
LC=$(git rev-parse --short HEAD)
sudo docker build -t danielmorales1202/webapp:${LC} .
sudo docker push danielmorales1202/webapp:${LC}
$ ./push.sh
-----------------
## Installing AWS CLI
$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
$ aws --version
## Configuring the AWS CLI
## get tokens: http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html
$ aws configure
AWS Access Key ID [None]: xxxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxx
Default region name [None]: us-east-1
Default output format [None]: json
## credentials & config under:
~/.aws/credentials
/.aws/config
## test
$ aws ec2 describe-vpcs --region us-east-1 --query="Vpcs[*].{ID:VpcId,tags:Tags[0]}"
$ aws rds describe-db-instances --db-instance-identifier webapp-postgres --query 'DBInstances[*].{Status:DBInstanceStatus}'
$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-a0e9c0c7" --query="Subnets[*].SubnetId"
-----------------
### Kubernetes
$ mkdir -p kube
$ mkdir -p kube/deployments
$ mkdir -p kube/jobs
## deployments (PostgreSQL)
$ touch kube/deployments/postgres-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: webapp
spec:
ports:
- port: 5432
selector:
app: webapp
tier: postgres
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: webapp
spec:
selector:
matchLabels:
app: webapp
tier: postgres
template:
metadata:
labels:
app: webapp
tier: postgres
spec:
containers:
- image: postgres:9.5.3
name: postgres
env:
- name: POSTGRES_PASSWORD
value: mysecretpassword
- name: POSTGRES_USER
value: webapp
- name: POSTGRES_DB
value: webapp_development
ports:
- containerPort: 5432
name: postgres
-----------------
#Job
$ touch kube/jobs/setup-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: setup
spec:
template:
metadata:
name: setup
spec:
containers:
- name: setup
image: danielmorales1202/webapp:1ca63f8
command: ["/bin/bash", "./setup.sh"]
env:
- name: PASSENGER_APP_ENV
value: development
restartPolicy: Never
-----------------
$ touch kube/deployments/webapp-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: webapp
labels:
app: webapp
spec:
ports:
- port: 80
selector:
app: webapp
tier: frontend
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
app: webapp
spec:
replicas: 3
selector:
matchLabels:
app: webapp
tier: frontend
template:
metadata:
labels:
app: webapp
tier: frontend
spec:
containers:
- image: danielmorales1202/webapp:1ca63f8
name: webapp
env:
- name: PASSENGER_APP_ENV
value: development
ports:
- containerPort: 80
name: webapp
imagePullPolicy: Always
-----------------
## Install & Run Minikube
$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
$ kubectl version --client
Install virtualbox following this tutorial: https://websiteforstudents.com/install-virtualbox-latest-on-ubuntu-16-04-lts-17-04-17-10/
$ minikube start --vm-driver=virtualbox
$ minikube ip
$ minikube dashboard
## commands with erros in my machine
$ kubectl create -f kube/deployments/postgres-deployment.yaml
## solve with this
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl apply -f kube/deployments/postgres-deployment.yaml
## continue
$ kubectl describe deployment postgres
$ kubectl describe Pod postgres
## working
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl get Pods
$ kubectl create -f kube/deployments/webapp-deployment.yaml
$ kubectl get Pods
$ kubectl logs webapp-7946c57864-btkc4 (following a name of one of the webapp replicas)
$ minikube service webapp
## let's make a post request
$ minikube ip
$ kubectl describe service webapp
## example: http://192.168.99.100:30763/
$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my first article","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit..."}' http://192.168.99.100:30763/articles
-----------------
## Deploying to production
$ kubectl config current-context
$ export NUM_NODES=2
$ export NODE_SIZE=t2.small
$ source ~/.zshrc
$ export KUBERNETES_PROVIDER=aws; curl -sS https://get.k8s.io | bash
$ kubectl config current-context
$ sudo docker-compose run --rm webapp bin/rake secret RAILS_ENV=production
## output Token: 7b66186fd493ac8302449d17........ copy this to config/secrets.ym
## in config/database.yml
production:
<<: *default
host: postgres
database: webapp_production
username: webapp
password: mysecretpassword
## open the kube/deployments/postgres.yml and change:
env:
- name: POSTGRES_PASSWORD
value: mysecretpassword
- name: POSTGRES_USER
value: webapp
- name: POSTGRES_DB
value: webapp_production
$ kubectl apply -f kube/deployments/postgres-deployment.yaml
$ kubectl describe service postgres
## open the kube/jobs/setup-job.yaml and change:
env:
- name: PASSENGER_APP_ENV
value: production
## new setup file
$ touch setup.production.sh
$ chmod +x setup.production.sh
#!/bin/sh
echo "Waiting PostgreSQL to start on 5432..."
while ! nc -z postgres 5432; do
sleep 0.1
done
echo "PostgreSQL started"
bin/rails db:migrate RAILS_ENV=production
## open the kube/jobs/setup-job.yaml and change:
containers:
- name: setup
image: danielmorales1202/webapp:1ca63f8
command: ["/bin/bash", "./setup.production.sh"]
## push
git add .
git commit -m "add production templates"
./push.sh
git rev-parse --short HEAD (#to get the new tag)
## open kube/jobs/setup-job.yaml and change the tag of the image
image: danielmorales1202/webapp:f5e0a11
$ kubectl delete -f kube/jobs/setup-job.yaml
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
## open kube/deployments/webapp-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: webapp
labels:
app: webapp
spec:
ports:
- port: 80
selector:
app: webapp
tier: frontend
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
app: webapp
spec:
replicas: 3
selector:
matchLabels:
app: webapp
tier: frontend
template:
metadata:
labels:
app: webapp
tier: frontend
spec:
containers:
- image: danielmorales1202/webapp:f5e0a11
name: webapp
env:
- name: PASSENGER_APP_ENV
value: production
ports:
- containerPort: 80
name: webapp
imagePullPolicy: Always
## create / apply
$ kubectl apply -f kube/deployments/webapp-deployment.yaml
$ kubectl describe service webapp
-----------------
## Adding persistence
$ aws ec2 create-volume --region us-west-2 --availability-zone us-west-2a --size 10 --volume-type gp2
## "VolumeId": "vol-08abc774f44d1f8f1"
## open kube/deployments/postgres-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: webapp
spec:
ports:
- port: 5432
selector:
app: webapp
tier: postgres
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: webapp
spec:
selector:
matchLabels:
app: webapp
tier: postgres
template:
metadata:
labels:
app: webapp
tier: postgres
spec:
containers:
- image: postgres:9.5.3
name: postgres
env:
- name: POSTGRES_PASSWORD
value: mysecretpassword
- name: POSTGRES_USER
value: webapp
- name: POSTGRES_DB
value: webapp_production
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
name: postgres
volumeMounts:
- name: postgres-persistent-storage
mountPath: /var/lib/postgresql/data
volumes:
- name: postgres-persistent-storage
awsElasticBlockStore:
volumeID: vol-08abc774f44d1f8f1
fsType: ext4
## run
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl create -f kube/deployments/postgres-deployment.yaml
$ kubectl describe rs postgres
$ aws ec2 describe-volumes --volume-ids vol-08abc774f44d1f8f1 --region us-west-2
$ kubectl delete job/setup
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl delete -f kube/deployments/postgres-deployment.yaml
$ kubectl create -f kube/deployments/postgres-deployment.yaml
-----------------
## Updating application
$ sudo docker-compose run --rm webapp bin/rails g migration AddSlugToArticles slug:string
## ojoo: I Need to open atom with root permission:
/webapp
$ sudo atom .
# in articles#create
def create
@article = Article.new(article_params)
@article.slug = @article.title.parameterize
if @article.save
render json: @article, status: :created, location: @article
else
render json: @article.errors, status: :unprocessable_entity
end
end
$ git add .
$ git commit -m ...
$ ./push.sh
$ git rev-parse --short HEAD
## change image code
containers:
- name: setup
image: danielmorales1202/webapp:56b2244
$ kubectl delete jobs/setup
$ kubectl create -f kube/jobs/setup-job.yaml
$ Pods=$(kubectl get Pods --selector=job-name=setup --output=jsonpath={.items..metadata.name})
$ kubectl logs $Pods
$ kubectl apply -f kube/deployments/webapp-deployment.yaml
$ curl -H "Content-Type: application/json" -X POST -d '{"title":"my second article","body":"Lorem ipsum dolor sit amet, consectetur adipiscing elit..."}' http://a333dae17845a11e6b47b06103f11903-585648094.us-west-2.elb.amazonaws.com/articles
-----------------
### Automation Scripts
$ mkdir deploy
$ mv push.sh deploy
# inside deploy/push.sh add
#!/bin/sh
set -x
LC=$(git rev-parse --short HEAD)
sudo docker build -f Dockerfile -t danielmorales1202/webapp:${LC} .
sudo docker push danielmorales1202/webapp:${LC}
kubectl set image deployment webapp webapp=danielmorales1202/webapp:${LC}
## in console
$ kubectl delete jobs/setup
# Open the kube/jobs/setup-job.yaml and change the image to
image: pacuna/webapp:LAST_COMMIT
## create migrate fil
$ touch deploy/migrate.sh
$ chmod +x deploy/migrate.sh
## And add:
#!/bin/sh
set -x
# get latest commit hash
LC=$(git rev-parse --short HEAD)
# delete current migrate job
kubectl delete jobs/setup || true
# replace LAST_COMMIT with latest commit hash output the result to a tmp file
sed "s/webapp:LAST_COMMIT/webapp:$LC/g" kube/jobs/setup-job.yaml > setup-job.yaml.tmp
# run the updated tmp job in the cluster and then delete the file
kubectl create -f setup-job.yaml.tmp &&
rm setup-job.yaml.tmp
## command
And that’s it! Now if you want to make changes to your code and deploy a new
version, you just have to commit your changes and then run deploy/push.sh. If you want
run migrations, you can just run deploy/migrate.sh. You can even build another script
that uses both for every deployment, which is what we will later do with Jenkins
#### COMMENTS ON THIS
- I NEED TO REPEAT THE WHOLE PROCESS FROM SCRATCH
- THIS NEW PROCESS NEEDS TO BE DONE IN A NEW FOLDER WITH A NEW PROJECT
- THE WAY TO DO THIS IS FOLLOWING THE ORIIGNAL BOOK AND THIS GIST AT THE SAME TIME AND CORRECTING THE ISSUES
- WHEN I CUT THIS PROCESS BECAUSE I HAD TO STOP MY STUDY AND I CONTINUE NEXT DAY, I HAVE TO HAVE IN MIND THAT I NEED TO RE-RUN SOME COMMANDS. THIS IS VERY IMPORTANT, AND CAN BREAK THE WHOLE PROCESS. SO I NEED TO DO ALL OF THIS THE SAME DAY
- THERE ARE TWO MAIN OPTIONS ACCORDING TO THE BOOK: ONE IS USING KUBERNETES AND THE OTHER IS USING INSTANCES FROM ECS AWS (THIS LATTER IS DIFFERENT FROM EC2). SO I NEED TO RUN EVERYTHING FIRST WITH KUBERNETES WITH ONE PROJECT AND NEXT DO IT WITH ECS
- BEFORE DOING ALL OF THIS FOR A REAL PROJECT I NEED TO KEEP THE SAME PROJECT MENTIONED ON THE BOOK (API ONLY)
- ONE I HAVE DONE ALL OF THIS I NEED TO DELETE CLUSTERS AND INTANCES FROM AWS, BECAUSE IS A PAID SERVICE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment