Skip to content

Instantly share code, notes, and snippets.

@scottmessinger

scottmessinger/00_plan.md

Last active Aug 29, 2015
Embed
What would you like to do?
Payload Builder

Suggested Plan to migrate from AR to ES in our Rails app

Clean up Queries

  • Change the specs and the queries to move filters from filter to query.filtered.filter
  • Move {term: {budget_lea_id: budget_lea_id} from query to query.filtered.filter

Remove JBuilder

  • Create PayloadBuilder library to build payloads
  • Convert JBuilder code to PayloadBuilder library

Remove AR

  • Move authorization from AR to ES.*
  • Move ordering from PG to ES
  • Remove results.records**
  • Move pagination from PG to ES

Footnotes

*Originally, I thought we could just use a Ruby function which rejects items in the results set. For pagination (as a filter could change the pagination), better to do in ES

** Due to the fact we call results.records and then use Postgres to filter, we can't use ElasticSearch for pagination until we remove results.records.

# ============================
# PROPOSAL for PAYLOAD BUILDER
# ============================
# ==============================
# On the organization of queries
# ==============================
# All of our queries have the same basic form (below). The methods assume we're passing a payload around that starts like this.
{
"query": {
"filtered": {
"filter": {
"and": []
},
"query": {
}
}
}
}
# ==========
# Philosophy
# ==========
# ES queries can be nested (queries in filters in queries in filters, etc). Thus, we shouldn't assume that when we add a filter, we're operating on the root of our payload.
# However, we also want to make it easy to add a query to the right place without having to repeat ourselves.
# Thus, I propose each function has a param that let's the location of the operation be changed
# Method Signature:
PB::Module::type.method(payload, hash_or_array, location)
# =====
# FILTER
# =====
# Add filter to query. Will be added by default to "query.filtered.filter"
PB::Filter.and.add(payload, filter)
# Add filter to an aggregation
PB::Filter.and.add(payload, filter, [:aggs, :segment_filter_agg, :filter])
# Add filter to post_filter at root of hash
PB::Filter.and.add(payload, filter, [:post_filter])
# Add filter to a arbitrary hash. For instance, if you're building up a filter in a complex aggregation, you want to use this function to add the hash
PB::Filter.and.add(some_hash, filter, [:filters])
# =====
# QUERY
# =====
# Add query to query
PB::Query.merge(payload, and_query)
# Add fuzzy_like_this
PB::Query.fuzzy_like_this.add(payload, hash)
# Add boolean#must
PB::Query::Boolean.must.add(payload, must_array)
# Add boolean#should
PB::Query::Boolean.should.add(payload, should_array)
@khristian

This comment has been minimized.

Copy link

@khristian khristian commented Jul 21, 2015

I like it. Looks good. Didn't see any use of the boolean OR operator on filters. Also, may want to use 'multi-match' and 'query' instead of 'fuzzy_like_this' (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.