Elasticsearch 101 | Using elasticsearch_dsl with Python from CRUD

ElasticSearch.md

Elasticsearch

Definition

From official website, Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

Concept

Data in Elasticsearch is organized into indices. Each index is made up of one or more shards. Each shard is an instance of a Lucene index, which you can think of as a self-contained search engine that indexes and handles queries for a subset of the data in an Elasticsearch cluster. For details on shards and indices, see here

Elasticsearch	RDBMS
Cluster	Database
Shard	Shard
Index	Table
Field	Column
Document	Row

Start Elastic on Docker

Here, we are using docker to start ELS rather than installing it locally and on a single node.

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.6.2

Create an index with mapping

curl -X PUT "localhost:9200/my-first-index?pretty" -H 'Content-Type: application/json' -d' 
{
  "mappings": {
    "properties": {
      "message": { "type": "text" },
      "time": { "type": "date"  }
    }
  }
}
'

Check all available indices

curl -XGET 'http://localhost:9200/_cat/indices?v&pretty'

Check the index created and mapping

curl -X GET 'http://localhost:9200/my-first-index?pretty'

Add some data

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/my-first-index/_doc -d {"message": "message one", "time": "2020-04-30T11:54:14Z"}'

or,

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/my-first-index/typename/1 -d '{"message": "message one", "time": "2020-04-30T11:54:14Z"}'

See standard difference between a HTTP POST vs PUT - POST is creating/generating a new ID for you and a new doc; PUT will amend an existing doc with id e.g. 1 here ***

Add some more data so that we can have some fun while we retrieve.

GET data from ELS

all docs sorted by a time field (as it's a date)

curl -X GET "localhost:9200/my-first-index_search?pretty" -H 'Content-Type: application/json' -d'
{
  "sort": { "time": "asc"} 
}
'

or,

curl -X GET 'http://localhost:9200/my-first-index/_search?sort=time:desc&pretty=true'

all doc filtered by field message having "hello"

curl -X GET "localhost:9200/my-first-index/_search?pretty" -H 'Content-Type: application/json' -d'
{
	"query": {
		"query_string": {
			"query": "*Hello*",
			"fields": ["message"]
		}
	}
}
'

all docs with message conatianing "error" and time filtered by last 5 mins (see AND operator used)

curl -X GET 'http://localhost:9200/my-first-index/_source?sort=time:desc&pretty=true' -H 'Content-Type: application/json' -d'
{
	"query": {
		"bool": {
			"must": [{
				"match": {
					"message": {
						"query": "*error*",
						"operator": "AND"
					}
				}
			}, {
				"range": {
					"time": {
						"gte": "now-5m",
						"lt": "now"
					}
				}
			}]

		}
	},
	"_source": [
		"error_message",
		"time"
	],
	"size": 1
}
'

Store and Get data in ELS using Python

Using elasticsearch_dsl you can easily CRUD ELS. See els_crud.py for example.

els_crud.py

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search

client = Elasticsearch()

s = Search(using=client, index="my-first-index").filter('range', time={'gte': 'now-1d', 'lt': 'now'}).query("match", message="*Error*")

s = s.sort({"time": {"order": "desc"}})

response = s.execute()

for hit in response:
    print(hit.name)