Skip to content

Instantly share code, notes, and snippets.

@noahtren
Last active June 27, 2020 00:27
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save noahtren/d18d8c28f2f68f3b4ad20c964bfdef9c to your computer and use it in GitHub Desktop.
Run code hosted on S3 in a Colab Notebook

Colab + S3 For Syncing Local Code

Scripts to add to your Colab Notebook to sync with an S3 bucket that contains code to run.


  1. Write AWS credentials to disk in order to use s3fs
%%writefile ~/.passwd-s3fs
<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>
!chmod 600 ~/.passwd-s3fs
  1. Set environment variables with AWS credentials
import os
os.environ['AWS_ACCESS_KEY_ID'] = '<AWS_ACCESS_KEY_ID>'
os.environ['AWS_SECRET_ACCESS_KEY'] = '<AWS_SECRET_ACCESS_KEY>'
os.environ['AWS_DEFAULT_REGION'] = 'us-east-1'
  1. Install and use s3fs
!apt install s3fs
!mkdir /s3
!chmod 777 /s3
!mkdir /s3/<BUCKET_NAME>
!s3fs <BUCKET_NAME> /s3/<BUCKET_NAME>
  1. Check that the path exists (this should print all the files in the top level of your S3 bucket)
ls /s3/<BUCKET_NAME>
  1. Finally, add the code from S3 to your PYTHONPATH so that it can be accessed.
sys.path.append('/s3/<BUCKET_NAME>')

Example Code

Run the run function defined in the main.py module.

import main
main.run()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment