Skip to content

Instantly share code, notes, and snippets.

@noahtren

noahtren/Colab-S3.md

Last active Jun 27, 2020
Embed
What would you like to do?
Run code hosted on S3 in a Colab Notebook

Colab + S3 For Syncing Local Code

Scripts to add to your Colab Notebook to sync with an S3 bucket that contains code to run.


  1. Write AWS credentials to disk in order to use s3fs
%%writefile ~/.passwd-s3fs
<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>
!chmod 600 ~/.passwd-s3fs
  1. Set environment variables with AWS credentials
import os
os.environ['AWS_ACCESS_KEY_ID'] = '<AWS_ACCESS_KEY_ID>'
os.environ['AWS_SECRET_ACCESS_KEY'] = '<AWS_SECRET_ACCESS_KEY>'
os.environ['AWS_DEFAULT_REGION'] = 'us-east-1'
  1. Install and use s3fs
!apt install s3fs
!mkdir /s3
!chmod 777 /s3
!mkdir /s3/<BUCKET_NAME>
!s3fs <BUCKET_NAME> /s3/<BUCKET_NAME>
  1. Check that the path exists (this should print all the files in the top level of your S3 bucket)
ls /s3/<BUCKET_NAME>
  1. Finally, add the code from S3 to your PYTHONPATH so that it can be accessed.
sys.path.append('/s3/<BUCKET_NAME>')

Example Code

Run the run function defined in the main.py module.

import main
main.run()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment