Set up a server from scratch with wpull and PhantomJS

gistfile1.md

Set up a server from scratch with wpull and PhantomJS and youtube-dl

Basic server set up

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install fail2ban

Bring on the packages!

If you're on Ubuntu 14.04 (like at Digital Ocean), you'll need to do this to get ffmpeg, which is helpful for youtube-dl stuff:

sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install ffmpeg

If you're not on Ubuntu 14.04, just do this instead:

sudo apt-get install ffmpeg

Then add the other stuff:

sudo apt-get install build-essential autoconf libtool python3-dev python3-pip git youtube-dl htop

Then add everything you'll need for phantomjs to work correctly:

sudo apt-get install build-essential chrpath libssl-dev libxft-dev fontconfig ttf-mscorefonts-installer libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev
cd ~
export PHANTOM_JS="phantomjs-2.1.1-linux-x86_64"
wget https://bitbucket.org/ariya/phantomjs/downloads/$PHANTOM_JS.tar.bz2
sudo tar xvjf $PHANTOM_JS.tar.bz2
sudo mv $PHANTOM_JS /usr/local/share
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/share/phantomjs
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/bin/phantomjs
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/bin/phantomjs
# Adapted from this website: https://gist.github.com/julionc/7476620

Finally, install wpull (and a utility called psutil)

sudo pip3 install wpull
sudo pip3 install psutil

Start a screen session, in case you get disconnected from your SSH (which you probably will) you can keep the web crawl going

screen