Skip to content

Instantly share code, notes, and snippets.

Avatar
💭
🇫🇷 🇯🇵

François-Clément Brossard renshuki

💭
🇫🇷 🇯🇵
View GitHub Profile
@renshuki
renshuki / logstash-output-elasticsearch-bulk-request-size-monitoring.md
Created Jun 25, 2021
Logstash - output.elasticsearch bulk requests size monitoring hack
View logstash-output-elasticsearch-bulk-request-size-monitoring.md

Command:

tcpdump -A 'tcp port 9200 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' -i lo0 | egrep -A 5 -i "POST /_bulk"

Expected output (w/ Content-Length):

...U...5POST /_bulk HTTP/1.1
Connection: Keep-Alive
@renshuki
renshuki / elasticsearch-adjust-scoring.md
Created Jun 25, 2021
Elasticsearch - Curation of resources / methods to adjust scoring in Elasticsearch
@renshuki
renshuki / logstash-log4j2-logs-delete-after.md
Created Jun 24, 2021
Logstash - log4j2 logs deletion after x
View logstash-log4j2-logs-delete-after.md

e.g Delete Logstash logs after 45 days

appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:ls.logs}
appender.rolling.strategy.action.condition.type = IfFileName
appender.rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 45D
@renshuki
renshuki / elasticsearch-threadpool-size-details.md
Last active Jun 9, 2021
Elasticsearch threadpool size details using _cat/thread_pool API
View elasticsearch-threadpool-size-details.md

Run GET _cat/thread_pool?v&h=node_name,name,max,pool_size,size,type,queue_size

Gives you details on Elasticsearch threadpool size, type, queue_size.

Example output

node_name           name                                    max pool_size size type                  queue_size
instance-0000000009 analyze                                             0    1 fixed                         16
instance-0000000009 ccr                                                32   32 fixed                        100
@renshuki
renshuki / upsert_by_query.js
Last active Sep 11, 2020
[Elasticsearch - Node.js] Upsert by query
View upsert_by_query.js
// - Try to update matching documents using _update_by_query (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html)
// - If no matching documents, index a new one with desired data
//
// Install Elasticsearch Node.js client: npm install @elastic/elasticsearch
const { Client } = require('@elastic/elasticsearch')
const client = new Client({ node: 'http://localhost:9200' })
client.updateByQuery({
@renshuki
renshuki / watcher-get-number-of-shards.md
Created Apr 8, 2020
Elasticsearch / Watcher - Watcher to retrieve number of shards from _cluster/health
View watcher-get-number-of-shards.md
{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input" : {
    "http" : {
      "request" : {
View keybase.md

Keybase proof

I hereby claim:

  • I am renshuki on github.
  • I am fcbrossard (https://keybase.io/fcbrossard) on keybase.
  • I have a public key ASDqj90a23IOGVKTIh8wjEk0cLLD-OgGhihG__X3o-s2gAo

To claim this, I am signing this object:

@renshuki
renshuki / watcher-transform-script-extract-duplicate.md
Created Mar 4, 2020
Elasticsearch / Watcher - Transform script to extract duplicates (doc_count > 1)
View watcher-transform-script-extract-duplicate.md
"transform": {
  "script": {
    "source": "return [ 'dups': ctx.payload.aggregations.my_buckets.buckets.stream().filter(t -> { return t.doc_count > 1 }).map(t -> { return ['key': t.key ] }).collect(Collectors.toList()) ]",
    "lang": "painless"
  }
}
@renshuki
renshuki / latency-of-curl-command.md
Created Feb 28, 2020
How to retrieve latency of a cURL command
View latency-of-curl-command.md
curl -o /dev/null -s -w 'Establish Connection: %{time_connect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n'  https://your_endpoint_url:port
@renshuki
renshuki / kibana-how-to-set-max-memory.md
Created Feb 13, 2020
Kibana - How to set instance max memory / heap (NodeJS)
View kibana-how-to-set-max-memory.md

You'll need to set NODE_OPTIONS in your environment variables.

export NODE_OPTIONS="--max-old-space-size=2048"

(which equivalent to 2GB for example)

Please note a Kibana restart is needed