Skip to content

Instantly share code, notes, and snippets.

Christian Dahlqvist cdahlqvist

Block or report user

Report or block cdahlqvist

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View nginx_es_bulk_stub.conf
# Nginx configuration for stubbing _bulk requests and proxying everything
# else to a local Elasticsearch instance.
#
# This can be used to test performance of network and the load generator
# for pure bulk indexing benchmarks.
#
# This configuration has been tested with Rally (https://github.com/elastic/rally).
# nginx-extras for the more-include headers module
# sudo apt-get install nginx nginx-extras
@cdahlqvist
cdahlqvist / ingest_pipeline_delay
Last active Jan 15, 2020
Ingest pipeline definition for measuring ingest delay based on @timestamp field
View ingest_pipeline_delay
# Ingest pipeline that records the timestamp the event was processed (`@received`)
# by the ingest pipeline and calculates the difference in milliseconds compared to
# the event timestamp (`@timestamp`).
POST _scripts/calculate_ingest_delay
{
"script": {
"lang": "painless",
"source": "SimpleDateFormat sdf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); ctx.ingest_delay = (sdf.parse(ctx['received']).getTime() - sdf.parse(ctx['@timestamp']).getTime()) / 1000.0"
}
@cdahlqvist
cdahlqvist / bulk_rejections.md
Last active Oct 23, 2019
rally-bulk-rejections-track
View bulk_rejections.md

Bulk Rejections Test

This Rally track is used to test the relationship between bulk indexing rejections and the following parameters:

  • Number of concurrent clients indexing into Elasticsearch
  • Number of shards actively being indexed into
  • Number of data nodes in the cluster
  • Size of bulk requests

The track contains a number of challenges, each indexing into an index with a set number of shards using a increasing number of concurrent client connections and two different bulk sizes.

@cdahlqvist
cdahlqvist / transform_system_metrics.py
Last active May 17, 2017
Script to transform Metricbeat system metrics into multiple formats for benchmarking
View transform_system_metrics.py
#!/usr/bin/env python
import json
import sys
import md5
import collections
import datetime
import re
epoch = datetime.datetime.utcfromtimestamp(0)
@cdahlqvist
cdahlqvist / README.md
Created Apr 23, 2017
Access log index size test
View README.md

Access log size test

This gist contains supporting files for evaluating Elasticsearch index sizes for web access logs.

Prerequisites

  • Machine with Linux or Mac OS X.
  • Local Elasticsearch 5.3.x instance accessible via 127.0.0.1:9200
  • The local Elasticsearch 5.3.x instance must have the geoip and useragent ingest plugins installed
  • Local installation of Filebeat 5.3.x with environment variable FILEBEAT_HOME pointing to the directory containing the filebeat binary.
View painless_transform
def triggered_time = ctx.trigger.triggered_time;
def failure_counts = new HashMap();
ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count));
def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList());
def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList());
View painless_transform_script_debug.txt
## Painless transform script:
POST _scripts/painless/alerting_index_transform
{
"script": "def triggered_time = ctx.trigger.triggered_time; def failure_counts = new HashMap(); ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count)); def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList()); def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList()); def success_and_failure_records = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'HIGH','successful_auths':e.doc_count,'failed_auths':failure_counts.get(e.key)]).collect(Collectors.toList()); success_and_failure_records.addAll(failure_only_re
@cdahlqvist
cdahlqvist / rally_out.log
Created Dec 5, 2016
Rally error with custom track
View rally_out.log
[elastic@localhost rally]$ rally --pipeline=benchmark-only --target-hosts=localhost:9200 --track=elasticlogs --challenge=test
Auto-updating Rally
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
@cdahlqvist
cdahlqvist / bulk.txt
Created Sep 26, 2016
ES 5.0.0-beta1 issue?
View bulk.txt
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "create" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value3" }
View X-Pack_Enablement_235.md

Code samples for X-Pack enablement

Watcher, Step 3-1

PUT _watcher/watch/manage_history
{
    "metadata": {
        "keep_history_days": 2
    },
You can’t perform that action at this time.