Mount S3 bucket to EC2 instance via S3fs using FUSE

mount-s3-bucket.md

Mount S3 Bucket to AWS EC2 Instance

1. Connect to the EC2 instance.
2. Update package manager to be able to find s3fs: sudo apt-get update
3. Install s3fs and awscli: sudo apt install s3fs awscli -y

   which s3fs
   /usr/bin/s3fs

4. Setup access(if you do NOT have credentials please check the Give permission to EC2 to access S3 section below ):

   echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs
   chmod 600 ${HOME}/.passwd-s3fs

5. Remove '#' of the allow_other in /etc/fuse.conf to allow to 'allow_other' option.

   sudo nano /etc/fuse.conf

6. Mount the bucket

   s3fs s3-bucket ~/local-s3-bucket -o passwd_file=~/.passwd-s3fs -o use_cache=/tmp -o allow_other -o uid=1000 -o mp_umask=002

7. You should see mounted bucket in the file system:

   df -h

   and files in the ~/local-s3-bucket.

Give permission to EC2 to access S3

1. Create IAM user: https://linuxbeast.com/tutorials/aws/create-a-new-iam-users-on-aws-console/
2. Dowload credentials within the following format:

   User name,Password,Access key ID,Secret access key,Console login link
   s3fs,,ACCESS_KEY_ID,SECRET_ACCESS_KEY,https://URL.signin.aws.amazon.com/console

3. That's it! We use this credentials to setup connection between ec2 and s3.

Useful commands

• Unmount the bucket

  - umount s3-bucket

• Check size of the bucket

  aws s3 ls s3://s3-bucket --recursive --human-readable --summarize

• Copy file from EC2 to your local machine

  scp -i ~/.ssh/aws_permission_file.pem ubuntu@ec2-ip-address.eu-central-1.compute.amazonaws.com:"/home/ubuntu/path_some_file" ~/Desktop/name_file_in_your_local_machine

• Copy file from your local machine to EC2

  scp -i ~/.ssh/aws_permission_file.pem ~/Desktop/name_file_in_your_local_machine ubuntu@ec2-ip-address.eu-central-1.compute.amazonaws.com:"/home/ubuntu/path_some_file" 

• Give read permission to files. ('a' refers to all. 'r' refers to read. To give write permission: 'a+w' or both: 'a+rw')

  sudo chmod -R a+r .

• Screen commands
  • List available screens: screen -ls
  • Switched to the screen named 'screen-xyz': screen -rd screen-xyz
  • Create new screen: screen -S watch-tensorboard