Skip to content

Instantly share code, notes, and snippets.

@ohjho
Last active March 3, 2021 07:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ohjho/46577004719337545b3fd470adeb9b43 to your computer and use it in GitHub Desktop.
Save ohjho/46577004719337545b3fd470adeb9b43 to your computer and use it in GitHub Desktop.
EFS mounting on AWS EC2 for Data Scientist

Create EFS

make sure it's in the same reigon in the same VPC as your EC2. Furthermore, your EFS also has to be in the same Security Group (with NFS access)

Mounting the EFS

follow this guide on how to mount your EFS on to a new EC2 instance

you can double check that it's mounted by df -h or mount | grep efs

your newly launched EC2 instance should have the EFS mount instructions in /etc/fstab

for instructions on how to mount the EFS manually see here

Once the EFS is mounted you'll need to change permissions to add files:

cd ~/efs-mount-point
sudo chmod go+rw .

If things failed, check here

Get python to use EFS

pip cache

find your pip install cache dir:

pip cache dir

which usually is at ~/.cache/pip. You can then create a symlink of it to a location on your EFS

virtualenv location

First create a symlink to your mount point in your home directory:

cd ~
ln -s /mnt/efs/fs1 efs-store

then just add this to your ~/.bashrc:

# Python VirtualEnvWrapper                                                                                                                      
export WORKON_HOME=$HOME/efs-store/.virtualenvs
export PROJECT_HOME=$HOME/efs-store/Devel
export VIRTUALENVWRAPPER_PYTHON=~/anaconda3/bin/python
source /home/ubuntu/anaconda3/bin/virtualenvwrapper.sh
alias mkvirtualenv3='mkvirtualenv --python=`which python3` '

Then virtualenvwrapper will build all virtualenv on your EFS

Resources

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment