Skip to content

Instantly share code, notes, and snippets.

Avatar

Christian Dahlqvist cdahlqvist

  • Independent
  • Cambridge
View GitHub Profile
View restore_snapshot.sh
#/bin/bash
TIMESTAMP=$(date +%s)
ES_HOST=$1
REPOSITORY=$2
INDEX_NAME=$3
SNAPSHOT_ID=$4
NEW_INDEX_NAME=$5
View create_repositories.sh
#/bin/bash
echo $(date) "Create snapshot repositories"
curl -X PUT "localhost:9200/_snapshot/elasticlogs-nofm" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/data/snapshots/elasticlogs-nofm"
}
View ccr_watch
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"http" : {
"request" : {
"host" : "127.0.0.1:9200",
View filter_logs.conf
input {
stdin {}
}
filter {
grok {
match => { "message" => [ '%{IP:ip}" %{GREEDYDATA:a}',
'%{IP:ip1}, %{IP:ip}" %{GREEDYDATA:a}' ] }
}
@cdahlqvist
cdahlqvist / epoch_prefixed_md5_identifier.conf
Last active Jul 3, 2020
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
View epoch_prefixed_md5_identifier.conf
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
@cdahlqvist
cdahlqvist / rally_split_indexing_challenge.json
Created Sep 16, 2018
Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput
View rally_split_indexing_challenge.json
{% set p_bulk_indexing_clients = (bulk_indexing_clients | default(20)) %}
{% set p_duration = bulk_indexing_duration | default(900) %}
{
"name": "split_indexing",
"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
"meta": {
"client_count": {{ p_bulk_indexing_clients }},
"benchmark_type": "split-indexing"
},
View recover_error_message.txt
[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner
View cold_index_stats.md

250GB shards

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_all" : {
@cdahlqvist
cdahlqvist / gdpr_access_controls.txt
Last active Mar 30, 2020
Securing GDPR Personal Data with Access Controls
View gdpr_access_controls.txt
# Tested with version 6.2.x of the Elastic Stack
# Add index templates
PUT _template/identity_store
{
"index_patterns": ["identity_store"],
"settings": {
"number_of_shards": 1
},
View Kibana API example
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=d25f3e20-041d-11e8-af22-05a1f7ea412f > dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=97aa8e60-041d-11e8-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=51fb4e10-ff75-11e7-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=a45b4e10-041d-11e8-af22-05a1f7ea412f >> dashboards.json