Scrapers

scrapers-server.org

Boostrap

Ubuntu 16.04

Login to remote instance using ssh and install mosh and zsh shell. Default shell change require user password.

sudo apt-get install mosh
sudo apt-get install git-core apache2-utils

Install required dependencies, and reboot.

sudo apt-get install software-properties-common
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:neovim-ppa/unstable
sudo apt-get update
sudo apt-get install neovim
sudo apt-get install python-dev python-pip python3-dev python3-pip
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'
sudo apt-get update
apt-cache policy docker-engine
sudo apt-get install -y docker-engine
sudo systemctl status docker
sudo usermod -aG docker $(whoami)
sudo shutdown -r 0

Install the linux-image-extra-* kernel packages, which allows you use the aufs storage driver.

sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual

Additional libs:

sudo apt install libcurl4-openssl-dev
sudo apt install libxml2-dev libxslt1-dev python-dev zlib1g-dev libssl-dev

Disable transparent huge pages (warning on mongodb and redis)

sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled
sudo echo never > /sys/kernel/mm/transparent_hugepage/defrag