Skip to content

Instantly share code, notes, and snippets.

@darkjh

darkjh/gist:d232d4379e803283045d62af37c8b3d0

Created June 16, 2021 15:06
Show Gist options
  • Save darkjh/d232d4379e803283045d62af37c8b3d0 to your computer and use it in GitHub Desktop.
Save darkjh/d232d4379e803283045d62af37c8b3d0 to your computer and use it in GitHub Desktop.
Download ZIP
Init horovod on EMR master
Raw
gistfile1.txt
conda activate tf2
pip install pyspark==2.4.3
pip install cluster-pack
pip install s3fs
pip install petastorm==0.11.1
conda install gxx_linux-64
conda install cmake
sudo yum install hadoop-libhdfs
HOROVOD_CMAKE=/mnt/conda_envs/tf2/bin/cmake HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 HOROVOD_WITH_GLOO=1 HOROVOD_WITHOUT_MPI=1 pip install git+https://github.com/darkjh/horovod@horovod-spark-executable-arg
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment