Skip to content

Instantly share code, notes, and snippets.

@simonw
Last active November 28, 2023 20:01
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save simonw/f99c70588f4d7447a223 to your computer and use it in GitHub Desktop.
Save simonw/f99c70588f4d7447a223 to your computer and use it in GitHub Desktop.
How to use Amazon AWS Elasticsearch

How to use Amazon AWS Elasticsearch

The good news: you can get it running on the free tier (with a tiny instance).

The bad news: it's stuck on Elasticsearch 1.5.2 and dynamic scripting (Groovy) is disabled.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

Authentication: the safest option is to create a brand new IAM user (using the tool at https://console.aws.amazon.com/iam/home?region=us-east-1 ) with its own access key and secret key. Then when you create the Elasticsearch instance you can paste in the following IAM string:

arn:aws:iam::YOUR_AWS_ACCOUNT_ID:user/YOUR_IAM_USERNAME

You'll need to look up YOUR_AWS_ACCOUNT_ID - http://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html

Having done all of the above, here's the secret recipe to getting Python to talk to your new Elasticsearch instance:

import requests
from requests_aws4auth import AWS4Auth

endpoint = 'https://search-your-endpoint.us-east-1.es.amazonaws.com'
auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es')

print requests.get(endpoint, auth=auth).json()
print requests.get(endpoint + '/_aliases', auth=auth).json()
# etc

If you're using the official Python client for Elasticsearch, the recipe looks like this:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es = Elasticsearch(
    'default',
    hosts=['search-your-endpoint.us-east-1.es.amazonaws.com'],
    http_auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es'),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

See also elastic/elasticsearch-py#280

@quiin
Copy link

quiin commented Apr 27, 2020

For anyone finding this gist, I had to add port=433 in order to use https:

es = Elasticsearch(
    'default',
    hosts=['search-your-endpoint.us-east-1.es.amazonaws.com'],
    http_auth=AWS4Auth(ACCESS_ID, ACCESS_SECRET, 'us-east-1', 'es'),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    port=433  # Add this for HTTPS
)

Using the official python client version 7.1.0

@ciurlaro42
Copy link

FYI, this doesn't work if you use AsyncElasticsearch instead of Elasticsearch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment