public
Last active

Configuring Tire to work with Bonsai

  • Download Gist
configuring tire for bonsai.md
Markdown

1. Configure Tire to use the Bonsai ElasticSearch Heroku add-on

gem 'tire'

config/initializers/bonsai.rb

ENV['ELASTICSEARCH_URL'] = ENV['BONSAI_URL']

app/models/article.rb

class Article
  include Tire::Model::Search
  include Tire::Model::Callbacks
end

2. Create the index and import your documents

rake environment tire:import CLASS=Article FORCE=true

Known issues

There are no "known issues" with Tire and Bonsai at the moment. Having trouble with something in particular? Drop us a line at support@bonsai.io.

  • Custom index analyzers must be set at index creation time. Ability to dynamically create, modify and destroy indexes.
  • Bulk import uses cluster-level /_bulk handler rather than the index-level _bulk handler, causing bulk imports to fail. Issue 327
  • Multi-model search is not scoped within the index. Issue 322
  • Undefined method [] for nil:NilClass on index.settings. Bonsai shared cluster indices are mapped to a random identifier, whereas Tire expects the logical index name in the _index response. To be fixed within Bonsai. Email support@bonsai.io if this is affecting you. Issue 386
  • ES Alias API not fully supported in Bonsai. Email support@bonsai.io if you would like to beta test that.

Tire.configuration doesn't seem to work in Tire's latest version.

I've used:

Tire.configure do
  url "http://index.bonsai.io"
end

Thanks Ernesto, I'll fix that next time I'm back at a computer.

Still prodding @karmi to go with one index per app by default… :)

Hey,
I got a "undefined method `URL' for main:Object" . does one of you have a hint?

thanks

@kikouli, use

 URI.parse(ENV['BONSAI_INDEX_URL']).path[1..-1]

thanks a lot ernesto . it works now!

Silly question but this now fails in my development environment throwing the obvious error

ruby-1.9.2-p318/lib/ruby/1.9.1/uri/common.rb:156:in `split': bad URI(is not URI?):  (URI::InvalidURIError)

What's the fix to getting development back please

Hi @jayinteractive

This configuration uses the BONSAI_INDEX_URL environment variable set up the bonsai add-on in heroku.

You'll need to set the variable in your development environment with the URL to your development ElasticSearch index.

In my case, rather than setting up another environment variable, I installed ElasticSearch in my development machine and modified the initializer:

if ENV['BONSAI_INDEX_URL']
  bonsai_uri = URI.parse(ENV['BONSAI_INDEX_URL'])
  Tire.configure do
    url "http://index.bonsai.io"
  end
  BONSAI_INDEX_NAME = bonsai_uri.path[1..-1]
else
  BONSAI_INDEX_NAME = "my_index"
end

Thanks for all the discussion here, folks. Open source is awesome. Keep it up!

Updated the gist to drop the URI.parse in favor of String#[Regexp] based on the issues mentioned above with it.

Also made BONSAI_INDEX_NAME conditional on the presence of ENV['BONSAI_INDEX_URL'] as per @ernesto-jimenez's comment. My version sets its value to the application name and environment. So your index for AcmeBlog in development would be acme-blog-development.

I tried following these instructions and I'm getting this in the Heroku logs:

2012-04-21T23:18:00+00:00 app[web.1]: Started GET "/search.json?q=janice" for 67.224.81.78 at 2012-04-21 19:18:00 -0400
2012-04-21T23:18:01+00:00 app[web.1]: Processing by SearchController#index as JSON
2012-04-21T23:18:01+00:00 app[web.1]:   Parameters: {"q"=>"janice"}
2012-04-21T23:18:01+00:00 app[web.1]: [REQUEST FAILED] curl -X GET "http://index.bonsai.io/artists,users/_search?pretty=true" -d '{"query":{"query_string":{"query":"janice"}}}'
2012-04-21T23:18:01+00:00 app[web.1]: Completed 500 Internal Server Error in 25ms
2012-04-21T23:18:01+00:00 app[web.1]: 
2012-04-21T23:18:01+00:00 app[web.1]: Tire::Search::SearchRequestFailed (401 : {"error": "Not authorized: Some endpoints are admin-only, ask support@onemorecloud.com."}
2012-04-21T23:18:01+00:00 app[web.1]: ):
2012-04-21T23:18:01+00:00 app[web.1]:   app/models/user.rb:141:in `search_for'
2012-04-21T23:18:01+00:00 app[web.1]:   app/controllers/search_controller.rb:6:in `index'

@rahilsondhi, your URL there (…index.bonsai.io/artists,users…) is omitting your index name, which is the random 20-digit base-36 string from the BONSAI_INDEX_URL environment variable. That's turning your search into a multi-index search across two indexes that don't exist. (Which we also wouldn't allow at this point, since no app should need to search across multiple indexes anyway.)

Another cluster-level Tire method call that @karmi should know about?

So what do you recommend I do @nz? My initializer and models are set up like the instructions ask for.

You should probably open an issue for Tire and cc me on it there.

Thanks for the continued feedback folks. I've updated the gist with some Tire known problems, and their Issues on GitHub:

Fix for bulk importing awaiting testing, merge and release: https://github.com/karmi/tire/pull/328 — feedback welcome!

Hey all, Tire 0.4.1 was just shipped with some fixes for bulk imports. Upgrade and import and drop me a line here or info@onemorecloud.com with any feedback.

there is a problem with this technique

if your setting the index_name in the class you will run into the problem that the Callbacks are not working since the instance.index != Class.index

the work around for this is to create your own observer for after_save and after_destroy

def after_save(record)
   Class.index.store(record)
end

def after_destroy(record)
   Class.index.remove(record)
end

since I only have 1 model i am using this works for me but it can be pretty tedious if you have multiple model maybe you can use active record observer that is not inferred

read more about it here. http://api.rubyonrails.org/classes/ActiveRecord/Observer.html

so, I followed instructions in the gist, adding this to my initializers/bonsai.rb

if ENV['BONSAI_INDEX_URL']
Tire.configure do
url "http://index.bonsai.io"
end
BONSAI_INDEX_NAME = ENV['BONSAI_INDEX_URL'][/[^\/]+$/]
else
app_name = Rails.application.class.parent_name.underscore.dasherize
app_env = Rails.env
BONSAI_INDEX_NAME = "#{app_name}-#{app_env}"
end

Also added this to the model

index_name BONSAI_INDEX_NAME

Now, when I run my app locally in development I get the following error

[REQUEST FAILED] curl -X GET "http://localhost:9200/dynamite-urbanite-development/city/_search?load=true&pretty=true" -d '{"query":{"query_string":{"query":"texas","default_operator":"AND"}}}'

Started GET "/cities?utf8=%E2%9C%93&query=texas" for 127.0.0.1 at 2012-07-03 22:12:02 -0400
Connecting to database specified by database.yml
Processing by CitiesController#index as HTML
Parameters: {"utf8"=>"✓", "query"=>"texas"}
Completed 500 Internal Server Error in 3ms

Tire::Search::SearchRequestFailed (404 : {"error":"IndexMissingException[[dynamite-urbanite-development] missing]","status":404}):
app/models/city.rb:20:in search'
app/controllers/cities_controller.rb:3:in
index'

So, I fixed the above problem by re - indexing my City model. My app works in development, but when I pushed the changes to Heroku, the application now crashes. Any ideas? The error message from Heroku logs are not help enough, all they tell me is that my app did indeed crash.

Have setup the initializer as above and when I curl the URL for BONSAI_INDEX_URL found in heroku config the test response is successful but when interacting with a model I get the infamous connection refused error. Is there a bug with Tire 0.4.2 or something?

For debugging:

Trying to curl locally works, but trying to curl from heroku returns the following:

heroku run curl -XPOST http://index.bonsai.io/e2ljko3c8qhc53zmdf0/test/hello -d '{"title":"Hello world"}' gives this...

Runningcurl -XPOST http://index.bonsai.io/e2ljko3c8qhc53zmdf0/test/hello -d {"title":"Hello world"}attached to terminal... up, run.1
{"error":"RemoteTransportException[[Polaris][inet[/10.31.156.223:9300]][index]]; nested: MapperParsingException[Failed to parse]; nested: JsonParseException[Unexpected character ('H' (code 72)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B@1f8fbddc; line: 1, column: 117]]; ","status":400}

Using Tire 0.4.3, I'm receiving the following when trying to bulk import records:

#

502 Bad Gateway

For those interested, I just soft deployed a big change to how provisioning works, which should be much more compatible with how Tire operates.

New addons will get a different environment variable: BONSAI_URL, which can be set directly for the Tire URL. You can now create indexes at this URL using the usual ElasticSearch Index APIs. Specifying the index name is now optional, though recommended, since I still think that applications should only need a single index per environment, as with your database.

I'll announce more about that on Monday, in particular about number of indexes and shards allowed within our production plans. In the mean time, check out the revised gist, and let me know if you have any questions. You can always email me directly at nick@onemorecloud.com as well.

Cheers, folks, and thanks for all the great feedback throughout the beta! Looking forward to launching this service very soon :-)

/cc @bryanmtl @pawel2105 @verdi327 @zacksiri @rahilsondhi @ernesto-jimenez @jayinteractive @kikouli

Can't wait to see updated notes :)

@nz, btw do you use latest Tire version from github or just latest stable release?

@nz is it possible to set the default analyzer on the _all field? I have tried this with the following settings in an ActiveModel

settings index: { analysis: { analyzer: { default: { type: 'snowball', language: 'english' } } } }

...but it looks like Bonsai is silently rejecting them...

curl -XGET 'http://juniper-1234.us-east-1.bonsai.io/artworks_staging/_settings'                             
{"xxxx1234":{"settings":{"index.number_of_replicas":"1","index.number_of_shards":"1","index.version.created":"190999"}}}

I need to find a way to have the _all field analyzed with snowball. Do you/anyone know a solution to this?

The instructions on

https://devcenter.heroku.com/articles/bonsai#using-tire-with-rails-3x 

does not mention setting up a custom index 'index_name INDEX_NAME' for every model. Are the instructions in this gist still necessary for the current version of Bonsai Search?

@digitalplaywright: the "shared index" approach — specifying the same index_name per model — is now optional. You may now create multiple indexes, subject to the total shards allowed per plan. (Currently set to two during beta.)

@pietia any relatively recent version of Tire should be fine, I'm working to support what's popular

@zefer can you drop me an email (support@onemorecloud.com) with the full index URL, and perhaps a curl command that's trying to POST or PUT the settings? We do filter that settings payload to track/enforce shards and replicas, but analyzers should pass through as provided.

Adding the following to an initializer will set the index prefix for all models:

Tire::Model::Search.index_prefix "#{Rails.application.class.parent_name.downcase}_#{Rails.env.to_s.downcase}"

Nice tip, @evanwhalen. I wonder if something like that could be suggested for a contribution upstream to Tire proper. Scoping by application and environment in some way or another seems like a no-brainer for local development and testing with multiple applications.

I still have trouble with search with multiple indexes.

> Tire.search("profiles,companies") { query { string "*world*"} }

[REQUEST FAILED] curl -X GET 'http://username@juniper-2764614.us-east-1.bonsai.io/profiles,companies/_search?pretty' -d '{"query":{"query_string":{"query":"*world*"}}}'
Tire::Search::SearchRequestFailed: 404 : {"error": "Index not found", "status": 404}

Is it possible to use Bonsai with multiple indexes search?

miry, I have the same problem as well. I'm looking into creating an index that the merged version of the index I'd like to search.

@miry and @Will-Sommers, this issue is on my end. I still have some work to do for supporting multi-index search syntax in Elasticsearch. I'll take a stab at that this week.

@miry @Will-Sommers — in the mean time, merging into a combined index (as per the "optional" in the instructions) should work fine for your purposes. You can then execute searches with:

Tire.search(INDEX_NAME) { query { string 'hello world' } }

Just to note, I've had to write a shell script to create my index with the proper analyzers and scoring. When importing multiple types into a single index only the first model takes settings in my experience.

Can anyone point me in the direction of an explanation / more information regarding this please? -

# Optional, but recommended: use a single index per application per environment.
# Caveat: This convention not be entirely supported throughout Tire's API.

Is the default to have an index for each model?

What's the advantages/disadvantages of a single index for the app?

How many indexes do Bonsai allow?

Thanks,

Ian

This was a helpful gist, thanks for posting it up.

@ichilton—

Is the default to have an index for each model?

This is Tire's default, yes.

What's the advantages/disadvantages of a single index for the app?

Each index carries its own overhead, and multiple indices can be wasteful for small applications. It is negligible, but in a shared cluster environment, that adds up. Creating an index per model can help as an additional level of natural partitioning, but in most cases that is premature optimization, and you will be better served by consolidating into a single index per environment.

The main disadvantage for using a single index with Tire is that it doesn't always fit within Tire's conventions. If you ever have trouble related to index naming conventions, you should just stick with Tire's defaults.

How many indexes do Bonsai allow?

The total primary shards varies per plan. All plans should be able to comfortable index 2–3 models. We're still working on exactly where to set those numbers, so email me at support@bonsai.io if you ever bump into those limits.

@aarongray—

Thanks! Happy to help.

I have the same problem as @pawel2105 had and wonder what is the solution? Via curl locally all works fine. But I get this error when I try to import.

→ heroku run rake environment tire:import CLASS=Initiative FORCE=true
Running rake environment tire:import CLASS=Initiative FORCE=true attached to terminal... up, run.5891
[IMPORT] Deleting index 'stadt-gestalten-production'
rake aborted!
Server broke connection
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:182:in rescue in transmit'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:140:in
transmit'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in execute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute'
/app/vendor/bundle/ruby/2.0.0/gems/rest-client-1.6.7/lib/restclient.rb:88:in head'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/http/client.rb:43:in
head'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/index.rb:18:in exists?'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/tasks.rb:14:in
create_index'
/app/vendor/bundle/ruby/2.0.0/gems/tire-0.6.0/lib/tire/tasks.rb:110:in `block (3 levels) in '
Tasks: TOP => tire:import => tire:import:model

Hi @criscrossed, you should email support@bonsai.io with your account information so I can check that out.

Updated the setup instructions to remove the "optional" combined index strategy. Our plans offer sufficient shards these days to support most common index-per-model sharding strategies.

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.