Skip to content

Instantly share code, notes, and snippets.

@danlamanna
Last active August 2, 2019 13:36
Show Gist options
  • Save danlamanna/e0ed311e989e6b414a9e73cbc3b53b7c to your computer and use it in GitHub Desktop.
Save danlamanna/e0ed311e989e6b414a9e73cbc3b53b7c to your computer and use it in GitHub Desktop.
ingesting activities into stumpf

The way to ingest activities is to take the relevant json files from the m1-annotations repo and run a python script over them that uploads them to Stumpf.

To set this up on your machine you need a few things:

  • Python and pip (if running the command 'pip' or 'python -m pip' prints the help page, you're good)

  • The m1-annotations repo
    Typically you'll want to be on the latest version of master without uncommitted changes, since the files are going from your computer to Stumpf.

  • A few python packages
    These can be installed via pip, so running one of the 2 commands should work:
    pip install --user requests awscli boto3
    python -m pip install --user requests awscli boto3

  • AWS credentials
    Once you have those python packages, you should be able to run the 'aws configure' command (this might be 'aws.cmd configure' if you're on Windows, I'm not sure). It'll ask you a few questions, most you can just accept the defaults for (press enter). The primary concern is when it asks you for an access key id and secret access key you use the credentials obtained under "Access Keys" here: https://console.aws.amazon.com/iam/home?region=us-east-2#/security_credentials

At this point, you have all the dependencies for running the script.

  1. Go to the main directory inside the m1-annotations repository

  2. Create a text file with the list of activities to import, for example:
    m1-annotations/master_grouper/2018-03-08/11/2018-03-08.10-55-00.11-00-00.bus.G327/refiner_activities_pass2

  3. Run 'python tools/stumpf-ingest.py $file' where $file is the location of the text file with your activities.

Here there's 2 likely outcomes:

  1. It spits out an error about the data
    It won't accept when there's an invalid track/activity type (often capitalization errors), or if there's a file with no tracks in it. In this case, fixing the json and trying again is best. If it says videos aren't ingested, then contact Jon/Dan/Brandon/Mike with the list to ingest.

  2. It says 'Uploaded all activities...' - in this case, you're all set.
    Nothing left to do. You can send an email to Dan or Jon Beezley if you want these processed immediately, but we're currently checking the queue every day - so it'll get picked up from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment