I'm giving a talk later this week at a Dat Meetup, so I figured I'd dump this here, get some feedback then put together a deck for Thursday, once I've had a chance to think it over.

amee and a npm for enviromental data.md

Making sense of the AMEE platform and an npm for scientific models

Hi folks.

I'm going to try sharing my understanding of how the AMEE platform works, because I'm giving a talk about this subject later in the week, at this event, where a load of open data and javascript geeks will be, and i figured that might be interested in it.

Anatomy of a calculation on the AMEE platform

Let's says I want to see the impact dairy cows have on the environment:

http://discover.amee.com/categories/Enteric_fermentation

I can run a calculation here, to find out:

http://discover.amee.com/categories/Enteric_fermentation/data/dairy%20cattle/North%20America/-/calculator

But what's happening under the hood?

Looking at the javascript

I don’t know java, but thankfully the main algorithm for working out these in this default.js file on github, in the documentation category for Enteric Fermentation

// edited for brevity
ch4Emissions = livestockNumber * massCH4PerHeadPerTime;

var ch4GWP = parseFloat(dataFinder.getDataItemValue('planet/greenhousegases/gwp','gas=CH4','GWP'));

co2eEmissions = ch4Emissions * ch4GWP;

Hang about, where do livestockNumber amd massCH4PerHeadPerTime come from?

If we look in the itemdef.csv for this , we’ll see livestockNumber and massCH4PerHeadPerTime.

(See the enteric-fermentation-itemdef.csv file below)

In this case, we can see that livestockNumber is false for isDataItemValue, so we know it's passed into the context where this js algorithm is eval'd. However, is massCH4PerHeadPerTime has True for this value, so we know it's not directly passed in as a number, but comes form somewhere else.

Where does it come from though?

The values are in the data.csv file here:

(See the enteric-fermentation-itemdef.csv file below)

Okay, so lets revisit the original algo again. It's not unreasonable to think the drill down option defines which row we should use in data.csv, so if we were looking at dairy cattle, the algo would be doing something like this:

// livestock number is what we put into the calculator
// massCH4PerHeadPerTime is 128 for our North American dairy cattle
ch4Emissions = livestockNumber * massCH4PerHeadPerTime;

Returning a number

Now we know C4 emissions, we migh want to return it, so it looks like we can do this with returnValues.putValue('CH4', 'kg','year', ch4Emissions);

returnValues.putValue('CH4', 'kg','year', ch4Emissions);

Okay, so:

• default.js outlines the algorithm in js to use.
• itemdef.csv tells us what variables we might have passed into the algorithm, and how they would be used by any dropdown form widget thing, and also lets us specify the algorithm file if need be.
• data.csv tells us which values to drop into the algoritm if they're not explicitly passed in by the widget itself
• returnValues in the algorithm is an object or hash of what gets returned

Turning up the difficulty

This appears to be largely how it works for algorithms that don't use other algorithms.

However, it's not uncommmon to see snippets like this:

var ch4GWP = parseFloat(dataFinder.getDataItemValue('planet/greenhousegases/gwp','gas=CH4','GWP'));

co2eEmissions = ch4Emissions * ch4GWP;

WHat's going on there? So, where does that come from?

Well, we can follow the path to planet/greenhousegases/gwp, and find that there'a once a again, a default.js file, an itemdef.csv and data.csv

Which just has:

emissionRate * GWP

Where do these numbers come from?

Lets run through the steps again. In gwp itemdef.csv file, we see these emissionRate and GWP.

(see global-warming-potentials-itemdefs.csv below)

Okay, so once again, we can see that gas is a dataItemValue to be looked up, and but emissionRate isn't. But lets look at this js snippet again:

ch4Emissions = livestockNumber * massCH4PerHeadPerTime;

// we know the result of this look up to be 21.00 now
var ch4GWP = parseFloat(dataFinder.getDataItemValue('planet/greenhousegases/gwp','gas=CH4','GWP'));

The final argument is GWP which if we look for the corresponding row in gwp data.csv, we see as 21.00

(see the global-warming-potentials-data.csv file below )

Putting it all together

// we've looked up massCH4PerHeadPerTime in the csv table, and we know livestockNumber is passed in
ch4Emissions = livestockNumber * massCH4PerHeadPerTime;

// we've shown this to be 21.00
var ch4GWP = parseFloat(dataFinder.getDataItemValue('planet/greenhousegases/gwp','gas=CH4','GWP'));

// we now take the methane emissions, and convert them to Carbon Dioxide equivalent - co2e
co2eEmissions = ch4Emissions * ch4GWP;

returnValues.putValue('CH4', 'kg','year', ch4Emissions);
returnValues.addNote('comment', 'CH4 emissions converted to CO2e using a global warming potential of '+ch4GWP);
returnValues.putValue('CO2e', 'kg','year', co2eEmissions);
returnValues.setDefaultType('CO2e');

Thoughts

In this essay by Bret Viktor, "What can a Technologist do about Climate Change", Brett asks an interesting question that's relevant to AMEE's history:

What if there were an “npm” for scientific models?

Given that it's possible to run javascript on the command line with node or rhino and other tools, and as long as you had a way to look up 'planet/greenhousegases/gwp', and pass in numbers, it doesn't sound like it would be impossible to literally have the npm for scientific models, that anyone could publish to the same way people can public to npm, rubygems, pip, and so on.

Caveats.

So, my understanding so far is that AMEE is full of algebraic equations that power the calculations. There are other kinds of equations for making calculations like this.

Bigger models for bigger calculations

I recently came across Modelica after seeing it name-checked, in the "What can a Technologist do about Climate Change" essay. I looked into it some more, and based on probably incomplete understanding of what Michael Tiller was talking about in this talk about Modelica at Strange loop, it looks like for more complex calculations with thousands of variables, what you end up doing is not 'just' doing a algebraic equation, but something more like a hybrid, non-linear, simultaneous differential algebraic equation.

For this kind of stuff tools like Modelica are a better fit, as they have a compile step to make running these in a sensible amount of time.

Everything that can be expressed in javascript, eventually will be

So, Mr Tiller does refer to and npm for scientific models in his talk. He shows a demo of a modelica model, existing as a npm package - the artillery Demo.

It's not obvious how he did this, but having a poke around on his github profile it turns out that Modelica models can be compiled to Javascript, using emscripten.

My guess is that he's using that to take modelica code, then compile it to javascript, then bundle it as an npm module.

I think.

Anyway, is this largely how the platform works?

I had a poke around the codebase, but as I said before, I can't speak java, and this isn't too meaningful to me, but these parts seem to be of some interest.

• run the algorithm in rhino or something: https://github.com/OpenAMEE/amee.platform.api/blob/multi-module/amee-platform-service/src/main/java/com/amee/calculation/service/CalculationService.java#L115
• getDataItemValue: https://github.com/OpenAMEE/amee.platform.api/blob/multi-module/amee-platform-service/src/main/java/com/amee/calculation/service/DataFinder.java#L87
• return the values after evaling the js somewhere: https://github.com/OpenAMEE/amee.platform.api/blob/multi-module/amee-platform-service/src/main/java/com/amee/calculation/service/CalculationService.java#L85

enteric-fermentation-data.csv

enteric-fermentation-itemdef.csv

global-warming-potentials-data.csv

global-warming-potentials-itemdefs.csv