Skip to content

Instantly share code, notes, and snippets.

@mrchrisadams

mrchrisadams/things.we.need.md

Last active August 29, 2015 14:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mrchrisadams/11357150 to your computer and use it in GitHub Desktop.
Save mrchrisadams/11357150 to your computer and use it in GitHub Desktop.
Download ZIP
A list of things we need to get another instance of the amee platform up online
Raw
things.we.need.md

Hi guys,

This weekend, I tried to get a version of the amee platform running locally, and spent a bit of time looking through the code, and trying to understand what it would take to get it running on new infrastructure.

As such, I'm sharing this summary of what shape the various bits are in.

Can you chime in where this summary needs any corrections?

Running the API servers themselves

David's done work to get the amee platform api to at least start when deployed to herok. You can see the code for this branch here:

https://github.com/OpenAMEE/amee.platform.api/tree/heroku

However to run this, and make it a publicly accessible platform again we would need to overcome a few more obstacles. I'm listing them here - please add any further info

1. Authentication

We were relying on myAMEE to issue credentials to users. However, we might not need this to make the api available, and might be abl to get by with a much simpler system for managing users, keys and passwords.

2. Database

The amee platform used a cluster of MySQL servers in a master-master replication arrangement before. It's conceivable that we could make the amee platform api connect to an Amazon RDS endpoint or something similar, and let amazon take care of the complexity.

3. Message queue

The platform relies on message queue, RabbitMQ. I couldn't find many details about how this needs to be set up as part of the amee platform. Are there any details in the repos to give guidelines here?

4. Lucene

We use Lucene for caching in the amee platform API. Lucene assumes it has unfettered access to the file system, which is a problem for heroku, as the cache typically needs to be quite large.

The size of this cache is likely to be large (as in gigabytes), but not huge, less than 5GB, for all the data, if we're not including ecoinvent's proprietary datasets.

It's feasible we could use Redis instad of the file system for this, but most hosted Redis offerings have a very affordble way to host the size of cache we would need, and the library that would provide this, RedisDirectory is unmaintained, experimental code.

Data

Finally, if we were to find services to provide:

  • a message queue
  • lucene with file system access
  • simple MySQL hosting
  • a minimal way to administer usernames, passwords and keys for ameeconnect to use

If we had this, we'd need to be able to import the data with the same UIDs as was on the previous platform, to allow users new and old to make the same kinds of credentials. We might do this by:

  • using a tweaked version of ameem, to run the initial data imports again visible at https://github.com/OpenAMEE/datasets
  • running a set of hand written SQL queries to fix UIDs to what they were before

Is this largely it to get it running again? Is there anything glaringly obvious that we're missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment