Skip to content

Instantly share code, notes, and snippets.

@vagrantism
vagrantism / spark_worker_google_colab.py
Created July 4, 2021 16:27
Spark Worker on Google Colab
!npm install -g localtunnel
!lt --port 8081 > localtunnel.log 2>&1 &
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
!tar -xvf spark-3.1.2-bin-hadoop3.2.tgz > /dev/null
!cat localtunnel.log
!tail -f "$(/content/spark-3.1.2-bin-hadoop3.2/sbin/start-slave.sh --memory 10G spark://xx.xx.xx.xx:7077 | cut -d ' ' -f 5)"