Payload Builder

00_plan.md

Suggested Plan to migrate from AR to ES in our Rails app

Clean up Queries

• Change the specs and the queries to move filters from filter to query.filtered.filter
• Move {term: {budget_lea_id: budget_lea_id} from query to query.filtered.filter

Remove JBuilder

• Create PayloadBuilder library to build payloads
• Convert JBuilder code to PayloadBuilder library

Remove AR

• Move authorization from AR to ES.*
• Move ordering from PG to ES
• Remove results.records**
• Move pagination from PG to ES

Footnotes

*Originally, I thought we could just use a Ruby function which rejects items in the results set. For pagination (as a filter could change the pagination), better to do in ES

** Due to the fact we call results.records and then use Postgres to filter, we can't use ElasticSearch for pagination until we remove results.records.

01_payload_builder.rb

# ============================
# PROPOSAL for PAYLOAD BUILDER
# ============================


# ==============================
# On the organization of queries
# ==============================
# All of our queries have the same basic form (below). The methods assume we're passing a payload around that starts like this.
{
  "query": {
    "filtered": {
      "filter": {
        "and": []
      },
      "query": {
        
      }
    }
  }
}

# ==========
# Philosophy
# ==========

# ES queries can be nested (queries in filters in queries in filters, etc). Thus, we shouldn't assume that when we add a filter, we're operating on the root of our payload.
# However, we also want to make it easy to add a query to the right place without having to repeat ourselves.
# Thus, I propose each function has a param that let's the location of the operation be changed

# Method Signature:
PB::Module::type.method(payload, hash_or_array, location)


# =====
# FILTER
# =====

# Add filter to query. Will be added by default to "query.filtered.filter"
PB::Filter.and.add(payload, filter)

# Add filter to an aggregation
PB::Filter.and.add(payload, filter, [:aggs, :segment_filter_agg, :filter])

# Add filter to post_filter at root of hash
PB::Filter.and.add(payload, filter, [:post_filter])

# Add filter to a arbitrary hash. For instance, if you're building up a filter in a complex aggregation, you want to use this function to add the hash
PB::Filter.and.add(some_hash, filter, [:filters])




# =====
# QUERY
# =====

# Add query to query
PB::Query.merge(payload, and_query)

# Add fuzzy_like_this
PB::Query.fuzzy_like_this.add(payload, hash)

# Add boolean#must
PB::Query::Boolean.must.add(payload, must_array)

# Add boolean#should
PB::Query::Boolean.should.add(payload, should_array)

I like it. Looks good. Didn't see any use of the boolean OR operator on filters. Also, may want to use 'multi-match' and 'query' instead of 'fuzzy_like_this' (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html)