Skip to content

Instantly share code, notes, and snippets.

@vagrantism
Created July 4, 2021 16:27
Show Gist options
  • Save vagrantism/2ef510b9c9e6e28807134b03759dcd5b to your computer and use it in GitHub Desktop.
Save vagrantism/2ef510b9c9e6e28807134b03759dcd5b to your computer and use it in GitHub Desktop.
Spark Worker on Google Colab
!npm install -g localtunnel
!lt --port 8081 > localtunnel.log 2>&1 &
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
!tar -xvf spark-3.1.2-bin-hadoop3.2.tgz > /dev/null
!cat localtunnel.log
!tail -f "$(/content/spark-3.1.2-bin-hadoop3.2/sbin/start-slave.sh --memory 10G spark://xx.xx.xx.xx:7077 | cut -d ' ' -f 5)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment