Skip to content

Instantly share code, notes, and snippets.

View cdahlqvist's full-sized avatar

Christian Dahlqvist cdahlqvist

  • Independent
  • Valencia, Spain
View GitHub Profile
#/bin/bash
TIMESTAMP=$(date +%s)
ES_HOST=$1
REPOSITORY=$2
INDEX_NAME=$3
SNAPSHOT_ID=$4
NEW_INDEX_NAME=$5
@cdahlqvist
cdahlqvist / create_repositories.sh
Created November 26, 2018 11:48
Frozen indices benchmark
#/bin/bash
echo $(date) "Create snapshot repositories"
curl -X PUT "localhost:9200/_snapshot/elasticlogs-nofm" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/data/snapshots/elasticlogs-nofm"
}
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"http" : {
"request" : {
"host" : "127.0.0.1:9200",
@cdahlqvist
cdahlqvist / filter_logs.conf
Created October 23, 2018 20:34
HTTP log replayer
input {
stdin {}
}
filter {
grok {
match => { "message" => [ '%{IP:ip}" %{GREEDYDATA:a}',
'%{IP:ip1}, %{IP:ip}" %{GREEDYDATA:a}' ] }
}
@cdahlqvist
cdahlqvist / epoch_prefixed_md5_identifier.conf
Last active July 3, 2020 03:50
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
@cdahlqvist
cdahlqvist / rally_split_indexing_challenge.json
Created September 16, 2018 08:41
Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput
{% set p_bulk_indexing_clients = (bulk_indexing_clients | default(20)) %}
{% set p_duration = bulk_indexing_duration | default(900) %}
{
"name": "split_indexing",
"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
"meta": {
"client_count": {{ p_bulk_indexing_clients }},
"benchmark_type": "split-indexing"
},
[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner

250GB shards

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_all" : {
@cdahlqvist
cdahlqvist / gdpr_access_controls.txt
Last active March 30, 2020 08:45
Securing GDPR Personal Data with Access Controls
# Tested with version 6.2.x of the Elastic Stack
# Add index templates
PUT _template/identity_store
{
"index_patterns": ["identity_store"],
"settings": {
"number_of_shards": 1
},
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=d25f3e20-041d-11e8-af22-05a1f7ea412f > dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=97aa8e60-041d-11e8-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=51fb4e10-ff75-11e7-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=a45b4e10-041d-11e8-af22-05a1f7ea412f >> dashboards.json