Skip to content

Instantly share code, notes, and snippets.

View renshuki's full-sized avatar
💭
🇫🇷 🇯🇵 🇺🇸

Francois-Clement Brossard renshuki

💭
🇫🇷 🇯🇵 🇺🇸
22 followers · 37 following
View GitHub Profile
@renshuki
renshuki / logstash-output-elasticsearch-bulk-request-size-monitoring.md
Created June 25, 2021 06:03
Logstash - output.elasticsearch bulk requests size monitoring hack

Command:

tcpdump -A 'tcp port 9200 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' -i lo0 | egrep -A 5 -i "POST /_bulk"

Expected output (w/ Content-Length):

...U...5POST /_bulk HTTP/1.1
Connection: Keep-Alive
@renshuki
renshuki / elasticsearch-adjust-scoring.md
Created June 25, 2021 03:25
Elasticsearch - Curation of resources / methods to adjust scoring in Elasticsearch
@renshuki
renshuki / logstash-log4j2-logs-delete-after.md
Created June 24, 2021 23:24
Logstash - log4j2 logs deletion after x

e.g Delete Logstash logs after 45 days

appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfFileName
appender.rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 45D
@renshuki
renshuki / elasticsearch_watcher_percentile_ranks.md
Created July 8, 2019 10:49
Elasticsearch Watcher example using "Percentile Ranks"

This watcher trigger an alert when less than 80% of page responses are under 500ms. (Time range not present in the sample below and need to be added )

Index sample

POST latency/_doc
{
  "response_time": 100
}
@renshuki
renshuki / upsert_by_query.js
Last active September 11, 2020 08:05
[Elasticsearch - Node.js] Upsert by query
// - Try to update matching documents using _update_by_query (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html)
// - If no matching documents, index a new one with desired data
//
// Install Elasticsearch Node.js client: npm install @elastic/elasticsearch
const { Client } = require('@elastic/elasticsearch')
const client = new Client({ node: 'http://localhost:9200' })
client.updateByQuery({
@renshuki
renshuki / elasticsearch_watcher_logstash_monitoring_events.md
Last active August 14, 2020 09:51
Elasticsearch Threshold Alert (Watcher) to check Logstash instance monitoring incoming events

This watcher check every 5 minutes if the number of Logstash monitoring events are below 1 which means Logstash monitoring is in trouble.

"watch" : {
    "trigger" : {
      "schedule" : {
        "interval" : "5m"
      }
    },
    "input" : {
@renshuki
renshuki / watcher-get-number-of-shards.md
Created April 8, 2020 06:12
Elasticsearch / Watcher - Watcher to retrieve number of shards from _cluster/health 
{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input" : {
    "http" : {
 "request" : {
@renshuki
renshuki / keybase.md
Created March 13, 2020 02:51

Keybase proof

I hereby claim:

  • I am renshuki on github.
  • I am fcbrossard (https://keybase.io/fcbrossard) on keybase.
  • I have a public key ASDqj90a23IOGVKTIh8wjEk0cLLD-OgGhihG__X3o-s2gAo

To claim this, I am signing this object:

@renshuki
renshuki / watcher-transform-script-extract-duplicate.md
Created March 4, 2020 06:45
Elasticsearch / Watcher - Transform script to extract duplicates (doc_count > 1) 
"transform": {
  "script": {
    "source": "return [ 'dups': ctx.payload.aggregations.my_buckets.buckets.stream().filter(t -> { return t.doc_count > 1 }).map(t -> { return ['key': t.key ] }).collect(Collectors.toList()) ]",
    "lang": "painless"
  }
}
@renshuki
renshuki / latency-of-curl-command.md
Created February 28, 2020 01:51
How to retrieve latency of a cURL command 
curl -o /dev/null -s -w 'Establish Connection: %{time_connect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n'  https://your_endpoint_url:port