ungoldman/koop_roadmap_proposal.md

## koop_roadmap_proposal.md

      
    Raw
  

              koop_roadmap_proposal.md
            
          
    Yay!

Koop is now deployable to Heroku.

But wait!

There are still some challenges remaining before koop can be more easily adopted.

1. Versioning

Currently koop requires installation of providers from github tarballs, like so:
# install github
npm install https://github.com/chelm/koop-github/tarball/master --save

# install gist
npm install https://github.com/chelm/koop-gist/tarball/master --save
Current Problem

Breaks caching

This breaks npm caching in all the environments I know of -- if master is changed and an update is needed, the npm cache will have to be manually wiped on the server. This task can be especially challenging when you don't have root access to the server, as in the case of Heroku deploys.
Not reliable

This situation doesn't really help users out either -- if master is the only reliable source for deploys, and master changes, there's no way to for users to know what changed, if there are any breaking changes, if they should upgrade, how they can upgrade...
Proposed Solution

Tag releases

For now we can very easily tag releases, so at the very least users can lock to version tags on github, like so:
npm install git://github.com/Esri/koop.git#v1.0.0 --save
cd koop
npm install git://github.com/Esri/koop-github.git#v1.0.0 --save
npm install git://github.com/Esri/koop-gist.git#v1.0.0 --save
The version tag should match the final commit in a series of commits related to a version bump in package.json.
Use a CHANGELOG.md file to track changes

A common practice that helps development teams just as much as users is a CHANGELOG.md file at the root of the repo. Koop should have one too! this way changes can be logged from version to version and there's no ambiguity as to what's going on.
# Koop Changelog

## v2.0.0
- change that breaks compatibility

## v1.1.0
- feature update
- backwards-compatible change

## v1.0.1
- bug fix

## v1.0.0
- first release

Publish releases to npm

Once koop is tagging version releases and logging changes, publishing releases to npm (in addition to github) is the ultimate power move. After making sure you have the right npm credentials, all you have to do is run npm publish after you've tagged a release.
# ensure version in package.json and changelog is updated, release is ready to go
git tag v1.33.7
git push upstream v1.33.7 ## http://git-scm.com/book/en/v2/Git-Basics-Tagging#Sharing-Tags
npm publish
Using Github's Releases interface takes care of tagging versions and is more reliable/less weird than git's tagging mechanisms (shown above) (tags don't transfer across forks, so pull requests won't do that step).
2. Packaging

If we want this to be a proper node module, it should act like one!
Current Problem

Doesn't act like a node module

To get koop running right now I have to fork it.
git clone git@github.com:Esri/koop.git
cd koop
npm install https://github.com/chelm/koop-github/tarball/master --save
npm install https://github.com/chelm/koop-gist/tarball/master --save
git commit -am 'add my changes to koop repo'
Ideally I should be able to install it as a node module and use its many powers in any context I choose.
Providers can't be configured from project root config

I noticed in the current workflow on the wiki you end up with these warnings at the server log level:
2014-11-03T18:33:14.775504+00:00 app[web.1]: No config file found for koop-gist. Please copy the models/config.js.example to models/config.js.
2014-11-03T18:33:14.852846+00:00 app[web.1]: No config file found for koop-github. Please copy the models/config.js.example to models/config.js.

These config files aren't modifiable since they're installed with dependencies inside node_modules.
Proposed Solution

Just Modularize It™

Export koop's functionality as a node module, allow it to be required, configured, and started as a first class node dependency. Internally, this could look something like
module.exports.start = function () {
  app.listen(process.env.PORT || config.server.port,  function() {
    console.log("Listening at http://%s:%d/", this.address().address, this.address().port);
  });
}

module.exports.stop = function () {
  app.close();
}
The logic would be a bit more complicated since you're doing that cluster.fork() business but I think this communicates the gist of the idea.
With this pattern, I would be able to install it like I would any other package on npm.
mkdir my-koop
cd my-koop
npm init # initialize node project & create package.json file
npm install koop koop-github koop-gist --save # add dependencies and save to package.json
touch index.js # create js file to start koop
touch config.json # create koop config file
git commit -am 'init'
The index.js could look something like
var config = require('config.json');
var koop = require('koop')(config);

koop.start();
Farther down the road, it would be great to decouple Koop from a specific third-party web framework altogether and expose it as a set of tools that can be used in any web framework (or other contexts -- e.g. a node desktop app).
Make all config happen at the top

Ideally the config.json file would also handle configuration for providers.
{
  "server": {
    "port": 1337
  },
  "processes": 1,
  "db": {
    "sqlite": {
      "filename": ":memory:"
    }
  },
  "providers": {
    "github": {
      "token": "XXXXXX"
    },
    "gist": {
      "token": "XXXXXX"
    }
  }
}
Further recommendations

Token security

Committing a secret like a token to version control is an anti-pattern (e.g. not recommended by wizards, sorcerers, necromancers, etc) -- in most cases I've seen, items like tokens and client secrets are kept out of version control by being declared as environmental variables. That way malicious ne'er-do-well types can't access your precious credentials by sniffing around your project source if it's compromised. This can be easily done in the context of Heroku or any Amazon EC2 instance or VPS as well.
export KOOP_GITHUB_TOKEN="XXXXXX"

If we're recommending people fork a public repo and add their API tokens for various providers, this will be a problem (and we'll be responsible for it). So let's be smart system designers and help people out before they make that mistake!

Version roadmap

Once versioning is instituted it would be great to have a version roadmap to allow for planning future releases and work related to koop -- for example pulling out logic for specific to tasks (like consuming feature services and exposing them as GeoJSON) into their own packages. I know of some folks who would love to be able to use a simple module to point at a feature service endpoint and get back GeoJSON without anything else going on in the background. This would also help with breaking tasks up and delegating work, allowing contributors inside and outside of Esri to take part in the project's improvement.

Edit: I see there's already a roadmap for 1.0 - https://github.com/Esri/koop/wiki/Koop:-Version-1.0