How to use Amazon AWS Elasticsearch

how-to-use-aws-elasticsearch.md

How to use Amazon AWS Elasticsearch

The good news: you can get it running on the free tier (with a tiny instance).

The bad news: it's stuck on Elasticsearch 1.5.2 and dynamic scripting (Groovy) is disabled.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Authentication: the safest option is to create a brand new IAM user (using the tool at https://console.aws.amazon.com/iam/home?region=us-east-1 ) with its own access key and secret key. Then when you create the Elasticsearch instance you can paste in the following IAM string:

arn:aws:iam::YOUR_AWS_ACCOUNT_ID:user/YOUR_IAM_USERNAME

You'll need to look up YOUR_AWS_ACCOUNT_ID - http://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html

Having done all of the above, here's the secret recipe to getting Python to talk to your new Elasticsearch instance:

import requests
from requests_aws4auth import AWS4Auth

endpoint = 'https://search-your-endpoint.us-east-1.es.amazonaws.com'
auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es')

print requests.get(endpoint, auth=auth).json()
print requests.get(endpoint + '/_aliases', auth=auth).json()
# etc

If you're using the official Python client for Elasticsearch, the recipe looks like this:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es = Elasticsearch(
    'default',
    hosts=['search-your-endpoint.us-east-1.es.amazonaws.com'],
    http_auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es'),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

See also elastic/elasticsearch-py#280

For anyone finding this gist, I had to add port=433 in order to use https:

es = Elasticsearch(
    'default',
    hosts=['search-your-endpoint.us-east-1.es.amazonaws.com'],
    http_auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es'),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    port=433  # Add this for HTTPS
)

Using the official python client version 7.1.0

FYI, this doesn't work if you use AsyncElasticsearch instead of Elasticsearch