cdahlqvist

# Nginx configuration for stubbing _bulk requests and proxying everything 
# else to a local Elasticsearch instance.
#
# This can be used to test performance of network and the load generator
# for pure bulk indexing benchmarks.
#
# This configuration has been tested with Rally (https://github.com/elastic/rally).

# nginx-extras for the more-include headers module
#   sudo apt-get install nginx nginx-extras

cdahlqvist

# Ingest pipeline that records the timestamp the event was processed (`@received`)
# by the ingest pipeline and calculates the difference in milliseconds compared to 
# the event timestamp (`@timestamp`).

POST _scripts/calculate_ingest_delay
{
  "script": {
    "lang": "painless",
    "source": "SimpleDateFormat sdf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); ctx.ingest_delay = (sdf.parse(ctx['received']).getTime() - sdf.parse(ctx['@timestamp']).getTime()) / 1000.0"
  }

cdahlqvist

Bulk Rejections Test

This Rally track is used to test the relationship between bulk indexing rejections and the following parameters:

• Number of concurrent clients indexing into Elasticsearch
• Number of shards actively being indexed into
• Number of data nodes in the cluster
• Size of bulk requests

The track contains a number of challenges, each indexing into an index with a set number of shards using a increasing number of concurrent client connections and two different bulk sizes.

cdahlqvist

#!/usr/bin/env python

import json
import sys
import md5
import collections
import datetime
import re

epoch = datetime.datetime.utcfromtimestamp(0)

cdahlqvist

Access log size test

This gist contains supporting files for evaluating Elasticsearch index sizes for web access logs.

Prerequisites

• Machine with Linux or Mac OS X.
• Local Elasticsearch 5.3.x instance accessible via 127.0.0.1:9200
• The local Elasticsearch 5.3.x instance must have the geoip and useragent ingest plugins installed
• Local installation of Filebeat 5.3.x with environment variable FILEBEAT_HOME pointing to the directory containing the filebeat binary.

cdahlqvist

def triggered_time = ctx.trigger.triggered_time; 

def failure_counts = new HashMap(); 

ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count)); 

def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList()); 

def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList()); 

cdahlqvist

## Painless transform script:

POST _scripts/painless/alerting_index_transform
{
  "script": "def triggered_time = ctx.trigger.triggered_time; def failure_counts = new HashMap(); ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count)); def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList()); def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList()); def success_and_failure_records = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'HIGH','successful_auths':e.doc_count,'failed_auths':failure_counts.get(e.key)]).collect(Collectors.toList()); success_and_failure_records.addAll(failure_only_re

cdahlqvist

[elastic@localhost rally]$ rally --pipeline=benchmark-only --target-hosts=localhost:9200 --track=elasticlogs --challenge=test
Auto-updating Rally

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

cdahlqvist

{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "create" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value3" }

cdahlqvist

Code samples for X-Pack enablement

Watcher, Step 3-1

PUT _watcher/watch/manage_history
{
    "metadata": {
        "keep_history_days": 2
    },