Skip to content

Instantly share code, notes, and snippets.

View cdahlqvist's full-sized avatar

Christian Dahlqvist cdahlqvist

18 followers · 1 following
  • Independent
  • Valencia, Spain
View GitHub Profile
@cdahlqvist
cdahlqvist / rally_split_indexing_challenge.json
Created September 16, 2018 08:41
Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput
{% set p_bulk_indexing_clients = (bulk_indexing_clients | default(20)) %}
{% set p_duration = bulk_indexing_duration | default(900) %}
{
"name": "split_indexing",
"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
"meta": {
"client_count": {{ p_bulk_indexing_clients }},
"benchmark_type": "split-indexing"
},
@cdahlqvist
cdahlqvist / recover_error_message.txt
Created July 20, 2018 12:23
[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner
@cdahlqvist
cdahlqvist / cold_index_stats.md
Last active July 20, 2018 11:56

250GB shards

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_all" : {
@cdahlqvist
cdahlqvist / Kibana API example
Created February 25, 2018 10:23
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=d25f3e20-041d-11e8-af22-05a1f7ea412f > dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=97aa8e60-041d-11e8-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=51fb4e10-ff75-11e7-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=a45b4e10-041d-11e8-af22-05a1f7ea412f >> dashboards.json
@cdahlqvist
cdahlqvist / nginx_es_bulk_stub.conf
Last active November 23, 2017 09:25
# Nginx configuration for stubbing _bulk requests and proxying everything
# else to a local Elasticsearch instance.
#
# This can be used to test performance of network and the load generator
# for pure bulk indexing benchmarks.
#
# This configuration has been tested with Rally (https://github.com/elastic/rally).
# nginx-extras for the more-include headers module
# sudo apt-get install nginx nginx-extras
@cdahlqvist
cdahlqvist / transform_system_metrics.py
Last active May 17, 2017 10:50
Script to transform Metricbeat system metrics into multiple formats for benchmarking
#!/usr/bin/env python
import json
import sys
import md5
import collections
import datetime
import re
epoch = datetime.datetime.utcfromtimestamp(0)
@cdahlqvist
cdahlqvist / shards_analysis
Last active April 4, 2017 12:52
#!/usr/bin/env perl
use POSIX;
my $KB = 1024;
my $MB = $KB*1024;
my $GB = $MB*1024;
my $TB = $GB*1024;
my %index_stats;
@cdahlqvist
cdahlqvist / painless_transform_script_debug.txt
Last active January 6, 2017 15:00
## Painless transform script:
POST _scripts/painless/alerting_index_transform
{
"script": "def triggered_time = ctx.trigger.triggered_time; def failure_counts = new HashMap(); ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count)); def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList()); def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList()); def success_and_failure_records = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'HIGH','successful_auths':e.doc_count,'failed_auths':failure_counts.get(e.key)]).collect(Collectors.toList()); success_and_failure_records.addAll(failure_only_re
@cdahlqvist
cdahlqvist / painless_transform
Last active January 6, 2017 14:37
def triggered_time = ctx.trigger.triggered_time;
def failure_counts = new HashMap();
ctx.payload.failed_auth_users.aggregations.users.buckets.stream().map(p -> failure_counts.put(p.key,p.doc_count));
def successful_users = ctx.payload.success_auth_users.aggregations.users.buckets.stream().map(p -> p.key).collect(Collectors.toList());
def failure_only_records = ctx.payload.failed_auth_users.aggregations.users.buckets.stream().filter(p -> !successful_users.contains(p.key)).map(e -> ['@timestamp':triggered_time,'user':e.key,'severity':'MEDIUM','failed_auths':e.doc_count]).collect(Collectors.toList());
@cdahlqvist
cdahlqvist / rally_out.log
Created December 5, 2016 19:00
Rally error with custom track
[elastic@localhost rally]$ rally --pipeline=benchmark-only --target-hosts=localhost:9200 --track=elasticlogs --challenge=test
Auto-updating Rally
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/