Skip to content

Instantly share code, notes, and snippets.

Christian Dahlqvist cdahlqvist

Block or report user

Report or block cdahlqvist

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@cdahlqvist
cdahlqvist / ingest_pipeline_delay
Last active Jan 15, 2020
Ingest pipeline definition for measuring ingest delay based on @timestamp field
View ingest_pipeline_delay
# Ingest pipeline that records the timestamp the event was processed (`@received`)
# by the ingest pipeline and calculates the difference in milliseconds compared to
# the event timestamp (`@timestamp`).
POST _scripts/calculate_ingest_delay
{
"script": {
"lang": "painless",
"source": "SimpleDateFormat sdf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); ctx.ingest_delay = (sdf.parse(ctx['received']).getTime() - sdf.parse(ctx['@timestamp']).getTime()) / 1000.0"
}
@cdahlqvist
cdahlqvist / bulk_rejections.md
Last active Oct 23, 2019
rally-bulk-rejections-track
View bulk_rejections.md

Bulk Rejections Test

This Rally track is used to test the relationship between bulk indexing rejections and the following parameters:

  • Number of concurrent clients indexing into Elasticsearch
  • Number of shards actively being indexed into
  • Number of data nodes in the cluster
  • Size of bulk requests

The track contains a number of challenges, each indexing into an index with a set number of shards using a increasing number of concurrent client connections and two different bulk sizes.

@cdahlqvist
cdahlqvist / epoch_prefixed_md5_identifier.conf
Last active May 22, 2019
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
View epoch_prefixed_md5_identifier.conf
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
View restore_snapshot.sh
#/bin/bash
TIMESTAMP=$(date +%s)
ES_HOST=$1
REPOSITORY=$2
INDEX_NAME=$3
SNAPSHOT_ID=$4
NEW_INDEX_NAME=$5
@cdahlqvist
cdahlqvist / README.md
Created Apr 23, 2017
Access log index size test
View README.md

Access log size test

This gist contains supporting files for evaluating Elasticsearch index sizes for web access logs.

Prerequisites

  • Machine with Linux or Mac OS X.
  • Local Elasticsearch 5.3.x instance accessible via 127.0.0.1:9200
  • The local Elasticsearch 5.3.x instance must have the geoip and useragent ingest plugins installed
  • Local installation of Filebeat 5.3.x with environment variable FILEBEAT_HOME pointing to the directory containing the filebeat binary.
View create_repositories.sh
#/bin/bash
echo $(date) "Create snapshot repositories"
curl -X PUT "localhost:9200/_snapshot/elasticlogs-nofm" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/data/snapshots/elasticlogs-nofm"
}
View ccr_watch
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"http" : {
"request" : {
"host" : "127.0.0.1:9200",
View filter_logs.conf
input {
stdin {}
}
filter {
grok {
match => { "message" => [ '%{IP:ip}" %{GREEDYDATA:a}',
'%{IP:ip1}, %{IP:ip}" %{GREEDYDATA:a}' ] }
}
@cdahlqvist
cdahlqvist / rally_split_indexing_challenge.json
Created Sep 16, 2018
Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput
View rally_split_indexing_challenge.json
{% set p_bulk_indexing_clients = (bulk_indexing_clients | default(20)) %}
{% set p_duration = bulk_indexing_duration | default(900) %}
{
"name": "split_indexing",
"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
"meta": {
"client_count": {{ p_bulk_indexing_clients }},
"benchmark_type": "split-indexing"
},
View recover_error_message.txt
[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner
You can’t perform that action at this time.