Deploy Rails 5.2.3 app in Ec2(ubuntu) + Nginx + Redis + Mina-Deploy

deploy-rails-ec2.md

Compute EC2 and setup instance and deploy Rails 5.2.3:

Ref: https://medium.com/@manishyadavv/how-to-deploy-ruby-on-rails-apps-on-aws-ec2-7ce55bb955fa

Before Launching Instance in AWS Create new security group and add below rules:

HTTP	TCP	80	0.0.0.0/0
HTTP	TCP	80	::/0
PostgreSQL	TCP	5432	0.0.0.0/0
SSH	TCP	22	0.0.0.0/0
HTTPS	TCP	443	::/0

Note: PostgreSQL 5432 is not required if application & database in the same instance

And then launch instance and get .pem file download it somewhere

$ chmod 400 my-rails-app-staging.pem
$ ssh -i "my-rails-app-staging.pem" ubuntu@1.2.3.5

You are logged into your instance:

$ sudo apt-get update
$ sudo apt-get install autoconf bison build-essential libssl-dev libyaml-dev
zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev
git-core curl libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev systemd

Allow 443, 80, 22 ports if not yet

sudo ufw allow ssh
sudo ufw allow 22
sudo ufw allow 443
sudo ufw allow 8080
sudo ufw allow 80
sudo ufw status 

if status returns 'inactive' then enable it

 sudo ufw enable
 sudo ufw status 

Ruby Installation

REF: https://gorails.com/deploy/ubuntu/18.04

Install node js

$ cd /tmp/
$ curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
$ sudo apt-get install -y nodejs
$ node -v
-> v12.10.0
$ npm -v
-> 6.10.3

Install yarn

$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt-get update && sudo apt-get install --no-install-recommends yarn
$ yarn -v
-> 1.17.3
$ sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common libffi-dev dirmngr gnupg apt-transport-https ca-certificates redis-server redis-tools nodejs yarn

Install mjml(if used for emails)

$ cd
$ sudo npm install -g mjml
$ sudo npm install -g npm

Now that we have our dependencies installed, we can begin installing Ruby.

$ cd
$ git clone https://github.com/rbenv/rbenv.git ~/.rbenv
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(rbenv init -)"' >> ~/.bashrc
$ source ~/.bashrc
type rbenv ## to see if rbenv is installed correctly
$ git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
$ echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
$ source ~/.bashrc
To get list of current versions

$ rbenv install -l

To list all local versions
$ rbenv install --list-all / -L


$ rbenv install 2.7.2
$ rbenv global 2.7.2
$ ruby -v ## must display 2.7.2 if installed correctly
$ which ruby ## must show the fully qualified path of the executable
$ echo "gem: --no-document" > ~/.gemrc ## to skip documentation while installing gem
$ gem install bundler
$ rbenv rehash ## latest version of rbenv apparently don't need this. Nevertheless, lets use it to avoid surprises.
$ bundle -v
   -> Bundler version 2.0.2

———————————

setup PostgreSQL and its users

$ sudo apt-get install postgresql postgresql-contrib libpq-dev

Then create DB user and his roles:

$ sudo -i -u postgres
$ postgres@~$ psql
That’s the psql command line. We can now enter a command to see what users are installed
$ postgres=# \du
Create User:
$ postgres=# create user may_app_dbuser with password 'yourpassword';
$ postgres=# alter user may_app_dbuser superuser createrole createdb replication bypassrls;
$ postgres=# \du

O/P

Role name	Roles	Member of
may_app_dbuser	Superuser, Create role, Create DB, Replication, Bypass RLS	{}
postgres	Superuser, Create role, Create DB, Replication, Bypass RLS	{}

Create database:

$ create database my_rails_app_staging owner may_app_dbuser;

Install Redis

REF: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04

$ sudo apt update
$ sudo apt install redis-server
        OR through download
$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make

Then,

$ redis-cli ping
    -> PONG

Changing default configuration Security fixes bind it to localhost only:

$ sudo vim /etc/redis/redis.conf

Locate this line and make sure it is uncommented (remove the # if it exists): bind 127.0.0.1 ::1

below change only for if it is downloaded instead of apt-get

And the set supervised as system

$ sudo vim /etc/redis/redis.conf

Update supervised no to

supervised system

Then, restart the Redis service to reflect the changes you made to the configuration file Redis Restart:

  $ sudo systemctl restart redis.service
  $ sudo systemctl status redis.service
  $ sudo systemctl stop redis.service

If made any changes redis.conf then restart the redis

$ sudo systemctl restart redis

Install ImageMagick

Ref: https://www.tecmint.com/install-imagemagick-on-debian-ubuntu/

$ sudo apt update
$ sudo apt-get install build-essential
$ sudo apt-get install autoconf automake autotools-dev libtool pkg-config
$ sudo apt-get install checkinstall 
$ sudo apt-get install libx11-dev libxext-dev zlib1g-dev 
$ sudo apt-get install libjpeg-dev libfreetype6-dev libxml2-dev

-- OR Single command to use all delegates --------

 Ref selected answer:

  https://askubuntu.com/questions/1042436/how-to-install-delegate-libraries-for-image-magick-7-0-7

$ sudo apt-get install autoconf automake autopoint autotools-dev build-essential chrpath \
cm-super-minimal debhelper dh-autoreconf dh-exec dh-strip-nondeterminism doxygen \
doxygen-latex dpkg-dev fonts-lmodern g++ g++-7 gcc gcc-7 gir1.2-harfbuzz-0.0 graphviz \
icu-devtools libann0 libasan4 libatomic1 libbz2-dev libc-dev-bin libc6-dev \
libcairo-script-interpreter2 libcairo2-dev libcdt5 libcgraph6 libcilkrts5 \
libclang1-6.0 libdjvulibre-dev libexif-dev libexpat1-dev libfftw3-bin libfftw3-dev \
libfftw3-long3 libfftw3-quad3 libfile-stripnondeterminism-perl libfontconfig1-dev \
libfreetype6-dev libgcc-7-dev libgdk-pixbuf2.0-dev libglib2.0-dev libglib2.0-dev-bin \
libgraphite2-dev libgts-0.7-5 libgvc6 libgvpr2 libharfbuzz-dev libharfbuzz-gobject0 \
libice-dev libicu-dev libicu-le-hb-dev libicu-le-hb0 libiculx60 libilmbase-dev \
libitm1 libjbig-dev libjpeg-dev libjpeg-turbo8-dev libjpeg8-dev liblab-gamut1 \
liblcms2-dev liblqr-1-0-dev liblsan0 libltdl-dev liblzma-dev libmime-charset-perl \
libmpx2 libopenexr-dev libpango1.0-dev libpathplan4 libpcre16-3 libpcre3-dev \
libpcre32-3 libpcrecpp0v5 libperl-dev libpixman-1-dev libpng-dev libpotrace0 \
libptexenc1 libpthread-stubs0-dev libpython-stdlib libquadmath0 librsvg2-bin \
librsvg2-dev libsigsegv2 libsm-dev libsombok3 libstdc++-7-dev libsynctex1 \
libtexlua52 libtexluajit2 libtiff-dev libtiff5-dev libtiffxx5 libtool libtool-bin \
libtsan0 libubsan0 libunicode-linebreak-perl libwmf-dev libx11-dev libxau-dev \
libxcb-render0-dev libxcb-shm0-dev libxcb1-dev libxdmcp-dev libxext-dev libxft-dev \
libxml2-dev libxml2-utils libxrender-dev libxt-dev libzzip-0-13 linux-libc-dev m4 \
make pkg-config pkg-kde-tools po-debconf preview-latex-style python python-minimal \
python2.7 python2.7-minimal python3-distutils python3-lib2to3 tex-common \
texlive-base texlive-binaries texlive-extra-utils texlive-font-utils \
texlive-fonts-recommended texlive-latex-base texlive-latex-extra \
texlive-latex-recommended texlive-pictures x11proto-core-dev x11proto-dev \
x11proto-xext-dev xorg-sgml-doctools xsltproc xtrans-dev zlib1g-dev

$ sudo apt-get install checkinstall libwebp-dev libopenjp2-7-dev librsvg2-dev \
libde265-dev libheif-dev

Now configure and install:

$ wget https://www.imagemagick.org/download/ImageMagick.tar.gz
$ tar xvzf ImageMagick.tar.gz
$ cd ImageMagick-7.*
$./configure
$ make clean && make

If you get any errors while compiling like above do,

$ sudo apt-get purge libheif-dev
then
$./configure && make

$ sudo make install


$ sudo ldconfig /usr/local/lib
$ magick -version
OR
$ identify -version

Creating SSH keys in EC2 Server and adding as deploying keys to git repo

Come back to our EC2 instance. Create an SSH key for ubuntu user on the server. Copy the public key and add it as a deployment key for the remote repo on Github/Gitlab

$ ssh-keygen -t rsa -b 4096
$ cat ~/.ssh/id_rsa.pub # copy this to clipboard

Goto Gitlab Project > Settings > Repository > Deploykeys -> add copied key and check ‘write allow access’ and submit

Add local pub SSH key to Server authorized_keys

In developer system copy the content of id_rsa.pub

$ cat ~/.ssh/id_rsa.pub

Then, in server add that at

$ vim sudo ~/.ssh/authorized_keys

Creating a Deploy user

Ref: https://gorails.com/deploy/ubuntu/18.04 Login to server and as root user

$ ssh ubuntu@1.2.3.5
$ sudo -i
$ adduser deploy
   Provide the root password(account creation time given password)
  Then fill necessary information
$ adduser deploy sudo
Now user created with sudo permission
Then,
$ exit

Next let's add our SSH key to the server to make it faster to login. We're using a tool called ssh-copy-id for this. If you're on a Mac, you may need to install ssh-copy-id with ‘homebrew first: $ brew install ssh-copy-id Then in Local Machine

$ ssh-copy-id ubuntu@1.2.3.5
$ ssh-copy-id deploy@1.2.3.5

Now you can login as either root or deploy without having to type in a password! For the rest of this tutorial, we want to be logged in as deploy to setup everything. Let's SSH in as deploy now and we shouldn't be prompted for a password this time $ ssh deploy@1.2.3.5

Installing Nginx

$ sudo apt-get install nginx
$ systemctl status nginx
$ nginx -v
    —> nginx version: nginx/1.14.0 (Ubuntu)
$ sudo systemctl restart nginx
$ sudo systemctl enable nginx

Mina-deploy

In Local Machine Add gem and add configuration file update content with staging env config/deploy.rb with below conten

https://gist.github.com/upender-devulapally/9168c2627c3bfefbcb482d67737e6b52

$ mina staging setup

It will creates the shared paths:

Then login into server add required files

Change permissions of 'shared' dir

$ sudo chmod -R 777 /home/ubuntu/apps/my-rails-app/staging/shared

$ cd /home/ubuntu/apps/my-rails-app/staging/shared

Then create master.key, database.yml, puma.rb

Create puma.rb with below content:

https://gist.github.com/upender-devulapally/d82b8d7e59be2cbfd62a8f797ec648de

After that

$ mina staging setup
$ mina staging deploy

Setting puma with Nginx:

  $ cd /etc/systemd/system
  $ sudo vim puma-my-rails-app-staging.service

Add the below content

 https://gist.github.com/upender-devulapally/d831d70efd8743b3c775715df04499e1

then reload systemd * IMP *

$ sudo systemctl daemon-reload

after saving above start and enable it

 $ sudo systemctl start puma-my-rails-app-staging.service
 $ sudo systemctl enable puma-my-rails-app-staging.service
 $ sudo systemctl stop puma-my-rails-app-staging.service
 $ sudo systemctl restart puma-my-rails-app-staging.service
 # check if it runnung
 $ ps aux | grep puma

Point puma to Nginx

$ cd /etc/nginx/sites-available

Imp (then comment all content in 'default' file)

$ sudo vim my-rails-app-staging

then add below content to it

https://gist.github.com/upender-devulapally/b6410f70464e64f3fd92b6ac1910463e

Imp enable it by symlinking

$ sudo ln -s /etc/nginx/sites-available/my-rails-app-staging /etc/nginx/sites-enabled/

Then finally restart nginx & puma

  $ sudo systemctl restart puma-my-rails-app-staging.service
  $ sudo systemctl restart nginx.service
 Reloading Nginx:
    $ sudo systemctl reload nginx

Sidekiq (backgroud jobs) as service systemd

$ cd /etc/systemd/system
$ sudo vim sidekiq.service
And past below 
-> https://gist.github.com/upender-devulapally/081b5886e6024844e0f411e7a294df40 
 
$ sudo systemctl daemon-reload
$ sudo systemctl enable sidekiq.service
$ sudo systemctl start sidekiq.service
$ sudo systemctl restart sidekiq.service

Adding SSL using LetsEncrypt

How To Secure Nginx with Let's Encrypt on Ubuntu 18.04

Follow below tutorials

https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04

OR

https://sunscrapers.com/blog/setting-up-https-on-nginx-with-certbot-and-letsencrypt/

Installation

 $ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
 $ sudo add-apt-repository ppa:certbot/certbot
  Press Enter
 $ sudo apt-get update
 $ sudo apt install python-certbot-nginx

Single Domain:

$ sudo certbot --nginx -d my-rails-app.net -d www.my-rails-app.net

Multiple subdomains

sudo certbot --nginx --expand -d my-rails-app.net -d my-rails-app.net -d api.my-rails-app.net -d admin.my-rails-app.net

The O/P:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/my-rails-app.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/my-rails-app.net/privkey.pem
   Your cert will expire on 2019-12-23. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Verifying Certbot Auto-RenewaL

 $ sudo certbot renew --dry-run

Note: To Delete letsencrypt Certbot Certificate by Domain Name

$ sudo certbot delete --cert-name example.com

If getting error like below for nginx

Restarting nginx: nginx: [warn] conflicting server name "" on 0.0.0.0:443, ignored nginx

Then Find where the ceritcates are using with below and remove them

grep -R /etc/letsencrypt/live/vendor.stuzee.in/fullchain.pem /etc/nginx/

Imp

And finally replace sites-available/my-rails-app-staging with below content

To work all SSL related and http to https redirection

https://gist.github.com/upender-devulapally/5ea7f2c36f21cc3d3a37267a20b65b35

Uploading Files to S3 using shrine gem

Directly uploading to S3 uppy-js

https://uppy.io/docs/aws-s3/ https://medium.com/@maxencemalbois/rails-multiple-direct-uploads-to-s3-with-shrine-and-uppy-with-live-versionning-thanks-to-an-ajax-6482b85dad2b

Fix for uploading directly to S3

Create S3 bucket with no public access

And add below cors rules to it,

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
    <AllowedHeader>x-amz-date</AllowedHeader>
    <AllowedHeader>x-amz-content-sha256</AllowedHeader>
    <AllowedHeader>content-type</AllowedHeader>
</CORSRule>
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>

Clear Shrine Uploads of cashe dir in S3

In mina deploy

invoke :'whenever:update'

But it creating duplicate cronjobs ittobe fixed Once created crontab then comment above line in next deployments

Work on fix for above

Logrotate

In Server

$ sudo apt-get update && sudo apt-get install logrotate
$ sudo vim /etc/logrotate.d/my-rails-app-staging

Then add below content,

   /home/ubuntu/apps/my-rails-app/staging/current/log/*.log {
        su root adm
        daily
        rotate 90
        dateext
        missingok
        compress
        delaycompress
        notifempty
        copytruncate
 }

Try to rotate your logs by running:

$ sudo logrotate /etc/logrotate.d/my-rails-app-staging

Run below to see which logs got rotated and when.

$ cat /var/lib/logrotate/status

Run below to create cron config for logrotate.

$ sudo vim /etc/cron.daily/logrotate

And save file exit

It seem now all set, your Rails app logs should be rotated on daily basis

Install Elasticsearch (ignore if not used)

ref: https://www.evernote.com/shard/s550/client/snv?noteGuid=f2161e91-5082-4f71-9f83-713595799288&noteKey=afe5ad60a65d149338d55f3e10e57e98&sn=https%3A%2F%2Fwww.evernote.com%2Fshard%2Fs550%2Fsh%2Ff2161e91-5082-4f71-9f83-713595799288%2Fafe5ad60a65d149338d55f3e10e57e98&title=Elasticsearch

 $ sudo apt update && sudo apt install apt-transport-https
 # Install Java JDK 
 $ sudo apt install openjdk-8-jdk
 # Verify the Java installation
 $ java -version
 # Import the repository’s GPG
 $ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
 # Add the Elasticsearch repository to the system by issuing
 $ sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'
 # Then Install
 $ sudo apt update && sudo apt install elasticsearch
 # Start the service and enable the service run:
 $ sudo systemctl enable elasticsearch.service
 $ sudo systemctl start elasticsearch.service

Verify elasticsearch is installed correctly $ curl -X GET "localhost:9200/"

O/P:

{
"name" : "ip-172-31-19-195",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "yap9U522R0OIwZY7LA6DuQ",
"version" : {
  "number" : "7.5.1",
  "build_flavor" : "default",
  "build_type" : "deb",
  "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
  "build_date" : "2019-12-16T22:57:37.835892Z",
  "build_snapshot" : false,
  "lucene_version" : "8.3.0",
  "minimum_wire_compatibility_version" : "6.8.0",
  "minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Adding CDN to assets using cloudfront

Ref : https://medium.com/@tranduchanh.ms/optimize-rails-app-performance-with-rails-amazon-cloudfront-e3b305f1e86c

follow above reference make sure below changes made

1. Create cloudfront distribution by selecting web

2. Add Domain Origin name by entering your site name. ex: www.example.com

3. Add below headers to whitelist

   Access-Control-Allow-Origin, Access-Control-Request-Method, Origin

4. After distribution creation select 'Origin & Origin Groups' tab and select 'Origin' and click on edit then, update 'Origin Protocol Policy' to 'Match Viewer' and click on 'Yes Edit'

5. In Rails app add below to application.rb or production.rb

  config.action_controller.asset_host = 'd1axxxxxxx.cloudfront.net' if Rails.env.production? 

** Important notes

in nginx server block packs location add below to resolve fonts issues while applying CDN

location ~* ^/packs/ {
   
   ### Caching through Headers #######
   add_header Cache-Control public;
   access_log off;
   # Per RFC2616 - 1 year maximum expiry
   # expires 1y;
   expires max; 
   ###END Browwser Caching #######
   
   # Configure CORS to resolve web fonts isssues (CDN Cloudfront)
    location ~* \.(ttf|ttc|otf|eot|woff|woff2|svg|font.css)$ {
      add_header Access-Control-Allow-Origin *;
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
      add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    }

   break;
  } 

Invalidate CloudFront Distribution

ref: https://deliciousbrains.com/wp-offload-media/doc/configure-cors-to-resolve-web-font-issues/

Once the changes have been made you will need to invalidate the CloudFront cache with a path of “/*”.