README.md

Overview

This project consists of web services that is deployed with capistrano and docker-compose on a single server, and a set of interfaces, like a browser extension

Services

1. backup: regularily backup and restore the database
2. crawler: main system crawler, writes data to database and analysis
3. web: the web interface that server HTML pages for whoispopulartoday.com
4. images: images web server
5. search: full text search web server
6. mysql: database for syncing with wikipedia
7. postgres: main database
8. memcached: a caching server for webserver and crawler
9. pg_hero: service for monitoring postgres database performance

Guidelines

• Microservices written by us doesn't interact with each other
• Microservices can interact with Databases and filesystem only
• Writing to database is done by the crawler only
• Writing for filesystem is done by the crawler, backup and postgres
• Any service can read from database or filesystem

Running the system localy

Foreman is used to run multiple services with one comman, issue foreman start to begin all required services to have the system up on you machine.

Provisioning

Provisioning is done with Ansible, so you need to install ansible first, the server directory has :

• hosts file which contains the server IP address
• provision.yml it's an ansible playbook you can use to provision the servers
• files a directory that contains some server configuration files

You provision the server by executing:

ansible-playbook -i hosts provision.yml

Deployment

to deploy this application to production you need to be in the root directory where Capfile exists and execute

cap production deploy

that will invoke capistrano to run a set of commands against production server, updates the files there, uses capistrano-decompose gem to build docker images, and run some tasks inside the new containers, like migrating the database, then bring old containers down and run the new ones.

Useful Snippets

Tunnel to pg_hero

ssh -L 8003:localhost:8003 vps

Show the logs for a service

ssh vps docker logs -f news_web_1

Setting the logger of rails active record on production

ActiveRecord::Base.logger = Logger.new(STDOUT)

Generating an SSL Command:

/root/.local/share/letsencrypt/bin/letsencrypt certonly --webroot --webroot-path /var/www/html/ --renew-by-default --email emad.elsaid.hamed@gmail.com --text --agree-tos -d whoispopulartoday.com -d www.whoispopulartoday.com