Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Set up a server from scratch with wpull and PhantomJS

Set up a server from scratch with wpull and PhantomJS and youtube-dl

Basic server set up

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install fail2ban

Bring on the packages!

If you're on Ubuntu 14.04 (like at Digital Ocean), you'll need to do this to get ffmpeg, which is helpful for youtube-dl stuff:

sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install ffmpeg

If you're not on Ubuntu 14.04, just do this instead:

sudo apt-get install ffmpeg

Then add the other stuff:

sudo apt-get install build-essential autoconf libtool python3-dev python3-pip git youtube-dl htop

Then add everything you'll need for phantomjs to work correctly:

sudo apt-get install build-essential chrpath libssl-dev libxft-dev fontconfig ttf-mscorefonts-installer libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev
cd ~
export PHANTOM_JS="phantomjs-2.1.1-linux-x86_64"
wget https://bitbucket.org/ariya/phantomjs/downloads/$PHANTOM_JS.tar.bz2
sudo tar xvjf $PHANTOM_JS.tar.bz2
sudo mv $PHANTOM_JS /usr/local/share
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/share/phantomjs
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/bin/phantomjs
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/bin/phantomjs
# Adapted from this website: https://gist.github.com/julionc/7476620

Finally, install wpull (and a utility called psutil)

sudo pip3 install wpull
sudo pip3 install psutil

Start a screen session, in case you get disconnected from your SSH (which you probably will) you can keep the web crawl going

screen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment