Skip to content

Instantly share code, notes, and snippets.

@nz
Last active July 3, 2018 22:51
Show Gist options
  • Star 12 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nz/1980980 to your computer and use it in GitHub Desktop.
Save nz/1980980 to your computer and use it in GitHub Desktop.
Bonsai.io - The ElasticSearch add-on for Heroku

Bonsai offers a Heroku add-on for ElasticSearch.

ElasticSearch is the latest generation search engine built on the powerful Lucene library. Its API is freshly designed around modern RESTful JSON conventions, making it easy to learn and understand. ElasticSearch will automatically index your JSON data with sensible defaults, which means that you can get started without writing a single line of configuration. ElasticSearch is the perfect place to start for anyone new to full-text search engine concepts, or for anyone who is looking for more clarity and less ceremony in their search engine.

The ElasticSearch internals are designed from the ground up with data distribution in mind. This means you get greater scalability, performance and reliability in a cloud hosted environment. An ElasticSearch index can automatically shard your data, to scale to the largest indexes and support high volumes of write traffic. All of our plans also offer replication by default, with transparent hot failover, and the ability to scale your replicas for high volumes of read traffic.

You have more important things to work on than managing and scaling another set of servers. With Bonsai, you get solid, scalable technology that is easy to use and understand, and managed by experts. All you have to worry about is putting it to use to build great features for your customers.

The Bonsai add-on provides an ElasticSearch index to your application, which is available to any language and platform that can send JSON to its RESTful HTTP API. You may also opt to use one of the many open source ElasticSearch clients for tighter integration with languages and frameworks such as, in no particular order, Ruby, Ruby on Rails, Python, Django, PHP, Erlang, Clojure, Java, Play! and Perl.

Installing the add-on

Bonsai ElasticSearch can be installed to a Heroku application via the heroku CLI:

Our free plan for development and testing allows a single index with one shard, and no replication. When you are ready for production, a list of all available plans and capacities can be found at [http://addons.heroku.com/bonsai](http://addons.heroku.com/bonsai).
:::term
$ heroku addons:add bonsai
-----> Adding bonsai to sharp-mountain-4005... done, v18 (free)

Once Bonsai ElasticSearch has been added, the BONSAI_URL environment variable will be available in your app's configuration. This will contain the canonical URL and credentials to your ElasticSearch Cluster. You can check the value of this variable with the heroku config comand:

:::term
$ heroku config | grep BONSAI
BONSAI_URL     => http://ql9lsrn8:img5ndnsbtaahloy@redwood-94865.us-east-1.bonsai.io/

After installing Bonsai ElasticSearch, your application may create an index and start using it right away.

Creating your first index

Many ElasticSearch clients will take care of creating an index for you. You should review your client's documentation for more information on its index usage conventions. If you don't know how many indexes your application needs, we recommend creating one index per application per environment, to correspond with your database.

Let's create an example index called acme-production from the command line with curl. Note that your application's BONSAI_URL will contain a username and a password which is used to authenticate index creation, modification and deletion.

You may want to check out [httpie](https://github.com/jkbr/httpie) as a nice alternative to `curl` that provides syntax highlighting.
:::term
$ curl -X POST http://ql9lsrn8:img5ndnsbtaahloy@redwood-94865.us-east-1.bonsai.io/acme-production
{"ok":true,"acknowledged":true}

Using your new index

Let's insert a "Hello, world" test document to verify that your new index is available, and to highlight some basic ElasticSearch concepts.

First and foremost, ElasticSearch is a JSON document store. You can use HTTP REST methods to create, update, fetch and destroy JSON documents. Every JSON document should specify an id and a type. You you may specify these values with the _id and the _type keys, or ElasticSearch will infer them from the URL of its API endpoints.

In the following example, we HTTP POST a simple JSON document to a URL which specifies a _type of test and an _id of hello. You should replace the sample URL in this document with your own index URL to follow along.

:::term
$ curl -XPOST http://redwood-94865.us-east-1.bonsai.io/acme-production/test/hello -d '{"title":"Hello world"}'
{
  "ok" : true,
  "_index" : "acme-production",
  "_type" : "test",
  "_id" : "hello",
  "_version" : 1
}

ElasticSearch will index the document you provide based on sensible defaults for later searching. You can also see some of the values you provided echoed back in the output, along with a few other default fields, which will be explained elsewhere.

Next, you may view this document by issuing a HTTP GET request to its URL.

Note the `_source` key, which contains a copy of your original document. ElasticSearch makes an excellent general-purpose document store!
:::term
$ curl -XGET 'http://redwood-94865.us-east-1.bonsai.io/acme-production/test/hello'                             
{
  "_index" : "acme-production",
  "_type" : "test",
  "_id" : "hello",
  "_version" : 1,
  "exists" : true,
  "_source" : { "title":"hello world" }
}

To learn more about about the operations supported by your index, you should read the ElasticSearch Index API documentation. Note that some operations mentioned in the documentation (such as "Automatic Index Creation") are for administrative purposes, and will be restricted in our environment.

Local workstation setup

ElasticSearch is open source software, and you can download and install your own instance for a local development and testing environment.

OS X and Homebrew

:::term
$ brew install elasticsearch

Other

You may download the latest ElasticSearch release from http://www.elasticsearch.org/download/. The downloaded package will contain a standalone elasticsearch executable.

:::term
$ curl -k -L -O http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.10.tar.gz
$ tar -zxvf elasticsearch-0.19.10.tar.gz
$ ./elasticsearch-0.19.10/bin/elasticsearch -f

Developing with Foreman

We also recommend using Foreman for starting and starting ElasticSearch locally. A sample Procfile may look like this:

elasticsearch: path/to/elasticsearch -f

You should also run heroku ps:scale elasticsearch=0 on your next deploy to avoid attempting to run this command on Heroku.

Using Tire with Rails 3.x

Tire is a popular Ruby on Rails client for ElasticSearch. To integrate Tire with ElasticSearch, you will need to set the ELASTICSEARCH_URL config variable to BONSAI_URL. To do so, add the following to config/initializers/bonsai.rb:

:::ruby
ENV['ELASTICSEARCH_URL'] = ENV['BONSAI_URL']

Please refer to https://gist.github.com/2041121 for more details, and to offer feedback, on Tire integration.

Bonsai ElasticSearch Dashboard

The Bonsai ElasticSearch dashboard shows you basic information about your account and. The dashboard can be accessed via the CLI:

:::term
$ heroku addons:open bonsai
Opening bonsai for sharp-mountain-4005…

Or by selecting Bonsai ElasticSearch from your application's page on the the Heroku dashboard.

Troubleshooting

HTTP 401 Authorization Required

Some ElasticSearch actions have been limited for administrative use only within our shared cluster. In general, this applies to any action outside of your index, or to updating certain system-level attributes of your index (such as shards or replicas).

Notably, if you receive this error when you try to PUT a new mapping to the root of your index, you should instead use the _mapping action within your index.

"No handler found for uri … and method …"

Note that the provided URL is for an index. Your request should be to a URL in the form of one of the following:

  1. http://account.region.bonsai.io/index_name
  2. http://account.region.bonsai.io/index_name/_verb
  3. http://account.region.bonsai.io/index_name/type_name
  4. http://account.region.bonsai.io/index_name/type_name/_verb
  5. http://account.region.bonsai.io/index_name/type_name/document_id
  6. http://account.region.bonsai.io/index_name/type_name/document_id/_verb

We also provide three Cluster-level actions at this time:

  1. http://region.bonsai.io/ — cluster root URL, for cluster health checks
  2. http://account.region.bonsai.io/ — cluster root URL, for cluster health checks
  3. http://account.region.bonsai.io/_bulk — cluster bulk import handler
  4. http://account.region.bonsai.io/_search/scroll — scrolling search handler

Migrating between plans

Application owners should carefully manage the migration timing to ensure proper application function during the migration process.

Our plans are primarily metered on replication level, and also on the number of shards allowed to a plan.

Indexes created within our starter-level plans have no replication, and are not intended for production traffic. An index created within our production-level plans will be replicated for higher availability and higher search traffic.

For greater data capacity, an index may be partitioned into many shards. Each shard is distributed to a different server within our cluster to allocate more resources for each search, and help scale to ever increasing quantities of data.

When changing your plan, we will adjust the replication level of any existing indices, as well as add or remove shards as per your new plan. If you have upgraded to a plan with additional shards, you may create a new index and must migrate your data

Removing the add-on

Bonsai ElasticSearch can be removed via the CLI.

This will destroy all associated data and cannot be undone!
:::term
$ heroku addons:remove bonsai
-----> Removing bonsai from sharp-mountain-4005... done, v20 (free)

Support

All Bonsai ElasticSearch support and runtime issues should be logged with Heroku Support at https://support.heroku.com. Please include your app name and index URL, and if possible, a reproduction of the issue with curl.

Any product feedback or non-support related issues is welcome via Twitter at @bonsaisearch or email to info@onemorecloud.com.

Additional resources

Suggestions for our docs?

Found a typo? Have a client to recommend, or an interesting question? Comment on this gist.

@hale
Copy link

hale commented Apr 2, 2012

Are you sure this is the official ElasticSearch homepage? http://bonsai.org/ looks like it's all about little trees :P. (Even more of a give away - it's backed by Joomla )

@gsf
Copy link

gsf commented Apr 29, 2012

The first commenter is correct. Also, the source code is at https://github.com/elasticsearch/elasticsearch.

@nz
Copy link
Author

nz commented Apr 30, 2012

Thanks, @hale and @gsf. Bad global find and replace :)

@d2kagw
Copy link

d2kagw commented Sep 10, 2014

Hey @nz, this doesn't seem to work for us - we're getting timeouts when connecting to ES via heroku.
We're not using elasticsearch-rails though, just the elasticsearch-transport as we're not building a rails app.
Could that be causing an issue?

@kbouw
Copy link

kbouw commented Mar 8, 2016

Hey @nz,

I was going through the setup process and came across the following typo under Troubleshooting:

screen shot 2016-03-07 at 7 30 27 pm 2

The gem names are case sensitive which renders an error for those who might copy/paste the gem name in the documentation.

@robsears
Copy link

@kbouw Thanks for the heads up. That was my bad (coincidentally due to a bad global find/replace). I fixed that, as well as a number of other spots with incorrect capitalization, outdated links, etc.

We're working on overhauling much of our documentation to make it more current and cohesive, so there should be some major improvements to the content soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment