using uglify to turn javascript functions into DynamoDB query language

api.js

// just a quick brainstorm for my dynamo API: https://github.com/jed/dynamo

// is it worth it to use uglify-js to parse functions into ASTs, to be transformed into dynamo's non-standard query language?

// the good:
// - no need to learn new API, just use javascript to query
// - API ends up resembling well-known .map and .filter style
// - would be entirely optional, compiling into specifiable query objects

// the bad:
// - kinda hacky
// - only a subset of JS could be supported
// - would be difficult to query by variable

// would love to hear some feedback at @jedschmidt

items.filter("age", "GT", 18) // would become
items.filter(function(x){ return x.age > 18 })

items.filter("age", "LT", 18) // would become
items.filter(function(x){ return x.age < 18 })

items.filter("age", "GE", 18) // would become
items.filter(function(x){ return x.age >= 18 })

items.filter("age", "LE", 18) // would become
items.filter(function(x){ return x.age <= 18 })

items.filter("age", "EQ", 18) // would become
items.filter(function(x){ return x.age == 18 })

items.filter("age", "NE", 18) // would become
items.filter(function(x){ return x.age != 18 })

items.filter("age", "NULL") // would become
items.filter(function(x){ return !("age" in x) })

items.filter("age", "NOT_NULL") // would become
items.filter(function(x){ return "age" in x })

items.filter("tags", "CONTAINS", "fun") // would become
items.filter(function(x){ return x.tags.indexOf("fun") >= 0 })

items.filter("tags", "NOT_CONTAINS", "fun") // would become
items.filter(function(x){ return x.tags.indexOf("fun") < 0 })

items.filter("name", "BEGINS_WITH", "Mc") // would become
items.filter(function(x){ return x.name.indexOf("Mc") == 0 })

items.filter("name", "IN", ["peter", "paul", "mary"]) // would become
items.filter(function(x){ return ["peter", "paul", "mary"].indexOf(x.name) >= 0 })

items.filter("age", "BETWEEN", [18, 30]) // would become
items.filter(function(x){ return x.age >= 18 && x.age <= 30 })

I gave some thought to this topic (js as the query language) in projects I work on with @pedrogte and @hij1nx. Our latest conclusion is to have functions that expose bare indexes and functions that operate on those indexes:

db.fulltext("nuno")
[{id: 123, score: 345}, {id: 234, score: 456}]

Now you can do multiple searches and merge results in javascript. With these bare metal constructs you can then have a language like you describe that uses javascript to do free form queries. Then if you like you can use the AST and start doing query optimization - which looks super fun but at the same time it is a topic that takes time to do perfectly - like if you look at a query optimizer from a traditional RDBMS.

Seeing what you have here it seems like ideas align, plus you didn't make the direct access to indexes look too ugly :) Just make sure they are composable!

Hth

awesome, thanks nuno.

thing is, dynamo's query language is very limited. i'm more looking at this as a way to use javascript itself as the query DSL. i'd avoid composition because in this case i think it's better to limit the API to one function call == one http request (i'm not a huge fan of overambitious ORMs).

another possibility is just translate the concepts but scrap the ASTs:

items.filter("age", "GT", 18) // would become
items.filter("age", ">", 18)

items.filter("age", "LT", 18) // would become
items.filter("age", "<", 18)

items.filter("age", "GE", 18) // would become
items.filter("age", ">=", 18)

items.filter("age", "LE", 18) // would become
items.filter("age", "<=", 18)

items.filter("age", "EQ", 18) // would become
items.filter("age", "==", 18)

items.filter("age", "NE", 18) // would become
items.filter("age", "!=", 18)

items.filter("age", "NULL") // would become
items.filter("age", "==", null)

items.filter("age", "NOT_NULL") // would become
items.filter("age", "!=", null)

items.filter("tags", "CONTAINS", "fun") // would become
items.filter("tags", "contains", "fun")

items.filter("tags", "NOT_CONTAINS", "fun") // would become
items.filter("tags", "!contains", "fun")

items.filter("name", "BEGINS_WITH", "Mc") // would become
items.filter("name", "beginsWith", "Mc")

items.filter("name", "IN", ["peter", "paul", "mary"]) // would become
items.filter("name", "in", ["peter", "paul", "mary"])

items.filter("age", "BETWEEN", [18, 30]) // would become
items.filter("age", ">=", [18, 30])

I see what you mean. Our thoughts have mostly been geared to IDB and LevelDB - so no REST calls :)

Good luck with this seems like an awesome project. Would def. use when/if using dynamo.

EDIT: I like the function approach better :)

It's an interesting idea worth thinking about.

Is it possible to combine filters in dynamo queries? If yes, how?
If the answer is "yes, but only simple boolean concatenations" or "no" then the limited choice and composability of dynamo operators makes it not worthwhile.
Then there must be a better way to represent the application of dynamo operators.

the good:

• no need to learn new API, just use javascript to query

How good is that if the second thing library users learn is that they can only do so much and - by design - there can never be more?

• API ends up resembling well-known .map and .filter style

The protocol of map, filter, ... is function application on collections.
The protocol of the above is very limited function application on collections with a translation step in between.
They're not the same.

Compare this to databases with JavaScript M/R capabilities where good libraries simply stringify the functions. Also limited to a subset of available libraries, not limited to a subset of JavaScript OR the capabilities of the database.

I wish I had more background on dynamo to give you a better answer but I guess we'll discuss. Thank you for making me think.

@evilhackerdude:

the (pretty lousy) docs don't seem to indicate that filters can be combined, but if they could i'd imagine it'd be limited to &&.

i realize the queries are limited, but that's the price you pay with dynamo. i guess the idea here is that you can provide a simple M/R-like interface. any queries too complex would just throw. so you could have an immediately grokkable hello world like this:

dynamo = require("dynamo")
db = dynamo.createClient()

db.tables
  .get("recipes")
  .items
    .filter(function(x){ x.user == "jim" && x.ingredients.indexOf("bacon") >= 0 })
    .map(function(recipe){ return {name: recipe.name, time: recipe.time } })
    .fetch(console.log)

the filter call sets the query parameters, and the map call could be used to tell dynamo which attributes to return, to save bandwidth.

@jed

.filter(function(x){ x.user == "jim" && x.ingredients.indexOf("bacon") >= 0 })

What about operators where there are many obvious different ways to express the same thing?
For example, BEGINS_WITH could be done with match, indexOf, slice + comparison among others.
You would have to document a list of accepted expressions even though at first glance one would expect every possible way to work.

I really liked your idea of translating the concepts without the AST stuff.
Doesn't go into the realm of leaky ORM abstractions and is much better than the raw dynamo API.

@jed - I agree with the general sentiment of not hiding too much after having worked with too many ORMs and query extensions (ie, LINQ) and spending far too long debugging the twisted ASTs they generate. DynamoDB is one of those beasts where it pays to have an understanding of exactly what's happening under the hood given its limited interface and potential for costly hidden queries.

So far my favourite is the "translate the concepts but scrap the ASTs" idea. BETWEEN could potentially be >=<= to distinguish/clarify it, but the presence of the array may be enough to make it clear anyway.

Personally, I don't mind the approach taken by @mranney w/ Redis where each command maps to a function. Perhaps that's because it's very easy to jump to and from the Redis CLI, where you get familiarity with the commands.

So with DynamoDB, something like:

db.get('recipes').where('name').beginsWith('mc').fetch('name', 'time'); // or .select('name', 'time')

or this would be shorter, but doesn't scan quite as well:

db.get('recipes').beginsWith('name', 'mc').fetch('name', 'time');

I realise this is veering from your filter/map idea - but I'm just not sure how far you can stretch the native feel of enumerations to a rigid database API.

thanks for the feedback, all.

@evilhackerdude, you're right. there's no way to enumerate the myriad ways that someone can check a substring, even if i do limit folks to indexOf.

@mhart, i think i'll start with generic query, scan, and filter methods that take the comparison operator names like above, and eventually specialize it into methods if it seems natural.