Skip to content

Instantly share code, notes, and snippets.

Christian Dahlqvist cdahlqvist

Block or report user

Report or block cdahlqvist

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View restore_snapshot.sh
#/bin/bash
TIMESTAMP=$(date +%s)
ES_HOST=$1
REPOSITORY=$2
INDEX_NAME=$3
SNAPSHOT_ID=$4
NEW_INDEX_NAME=$5
View create_repositories.sh
#/bin/bash
echo $(date) "Create snapshot repositories"
curl -X PUT "localhost:9200/_snapshot/elasticlogs-nofm" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/data/snapshots/elasticlogs-nofm"
}
View ccr_watch
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"http" : {
"request" : {
"host" : "127.0.0.1:9200",
View filter_logs.conf
input {
stdin {}
}
filter {
grok {
match => { "message" => [ '%{IP:ip}" %{GREEDYDATA:a}',
'%{IP:ip1}, %{IP:ip}" %{GREEDYDATA:a}' ] }
}
@cdahlqvist
cdahlqvist / epoch_prefixed_md5_identifier.conf
Last active May 22, 2019
Logstash config showing how to create a document identifier built from MD5 hash prefixed by hex formatted epoch date
View epoch_prefixed_md5_identifier.conf
input {
generator {
lines => ['2011-04-19T03:44:01.103Z testlog1',
'2011-04-19T03:44:02.035Z testlog2',
'2011-04-19T03:44:03.654Z testlog3',
'2011-04-19T03:44:03.654Z testlog3']
count => 1
}
}
@cdahlqvist
cdahlqvist / rally_split_indexing_challenge.json
Created Sep 16, 2018
Challenge for the rally-eventdata-track to look at the impact of `index.number_of_routing_shards` setting in indexing throughput
View rally_split_indexing_challenge.json
{% set p_bulk_indexing_clients = (bulk_indexing_clients | default(20)) %}
{% set p_duration = bulk_indexing_duration | default(900) %}
{
"name": "split_indexing",
"description": "Index data into indices with and without `number_of_routing_shards` set. IDs are autogenerated by Elasticsearch, meaning there are no conflicts.",
"meta": {
"client_count": {{ p_bulk_indexing_clients }},
"benchmark_type": "split-indexing"
},
View recover_error_message.txt
[2018-07-19T08:48:47,631][WARN ][o.e.i.c.IndicesClusterStateService] [iZBcaR9] [[elasticlogs-2-2018.01.02-44][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2-2018.01.02-44][1]: Recovery failed from {Q8apBcz}{Q8apBcz2QeS7wXq3tIrvLA}{gY6uDjozSmWxadECk2VenQ}{192.168.1.32}{192.168.1.32:9300}{ml.machine_memory=128847142912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {iZBcaR9}{iZBcaR9JTk6f8OhyT7yL4A}{9v9sDmqZTniJCFvk_MSk6g}{192.168.1.33}{192.168.1.33:9300}{ml.machine_memory=128847142912, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner
View cold_index_stats.md

250GB shards

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_all" : {
@cdahlqvist
cdahlqvist / gdpr_access_controls.txt
Last active Mar 26, 2018
Securing GDPR Personal Data with Access Controls
View gdpr_access_controls.txt
# Tested with version 6.2.x of the Elastic Stack
# Add index templates
PUT _template/identity_store
{
"index_patterns": ["identity_store"],
"settings": {
"number_of_shards": 1
},
View Kibana API example
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=d25f3e20-041d-11e8-af22-05a1f7ea412f > dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=97aa8e60-041d-11e8-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=51fb4e10-ff75-11e7-af22-05a1f7ea412f >> dashboards.json
curl -k -u "uuu:ppp" -XGET https://356a27e883d143769ddbd5xxxxxxxxxx.demo.elastic.co:9243/api/kibana/dashboards/export?dashboard=a45b4e10-041d-11e8-af22-05a1f7ea412f >> dashboards.json
You can’t perform that action at this time.