Skip to content

Instantly share code, notes, and snippets.

View ijokarumawak's full-sized avatar
🎯
Focusing

Koji Kawamura ijokarumawak

🎯
Focusing
83 followers · 16 following
  • Yokohama Japan
View GitHub Profile
@ijokarumawak
ijokarumawak / can_we_change_coerce.md
Last active June 22, 2021 10:45
Can we turn off coercing without having to reindex? Yes, we can.

Create the num field as long with auto-mapping.

PUT coerce_test/_doc/1
{
  "num": 1
}

By default, coerce is true. That means a string value which can be converted into a number, can be ingested.

@ijokarumawak
ijokarumawak / serialize_test.go
Created June 16, 2021 03:05
Libbeat diskqueue cannot handle multi-byte message correctly.
package diskqueue
import (
"encoding/json"
"testing"
"github.com/elastic/beats/v7/libbeat/beat"
"github.com/elastic/beats/v7/libbeat/common"
"github.com/elastic/beats/v7/libbeat/publisher"
"github.com/stretchr/testify/assert"
@ijokarumawak
ijokarumawak / 0_create_index.md
Created May 20, 2021 04:23
Wikipedia 日本語ページを Elasticsearch に登録する Logstash サンプル

Kuromoji で本文を解析できるようにマッピングを作成。

PUT jawiki
{
  "mappings": {
    "properties": {
      "doc": {
        "properties": {
 "revision": {
@ijokarumawak
ijokarumawak / README.md
Last active April 13, 2021 03:12
How does Kibana calculate metric values for other bucket?

How does Kibana calculate metric values for other bucket?

In order to render the Other slice, Kibana uses the sum_other_doc_count value in the response. That works if the metric is Count. But what if we use different metric aggretation to define the size of each slice?

For example, this pie chart uses sum of field age. But still, Kibana is able to show the Other slice as expected.

screen-shot

When I looked at Chrome dev tool network tab, there were 2 _search requests shown as below.

@ijokarumawak
ijokarumawak / example.txt
Created April 8, 2021 04:14
How Elasticsearch calculate average if there are multiple shards
# Create an index with 2 primary shards.
PUT avg-avg
{
"settings": {
"number_of_shards": 2,
"number_of_replicas": 0
}
}
# Add some documents.
@ijokarumawak
ijokarumawak / README.md
Last active February 15, 2022 10:58
README.md

Simplest Filebeat config to test processors

This filebeat.yml config will receive input from stdout, do some processing and write output to stdout.

How to run

echo "message" | filebeat -c `pwd`/filebeat.yml -e 2> /dev/null
@ijokarumawak
ijokarumawak / @PainlessExamples.md
Last active February 23, 2021 12:18
Painless examples

Painless Examples

Elasticsearch 7.11 から Runtime Field が使えるようになったり (まだ Beta だけど)、 Kibana の Painless Lab でオートコンプリートが使えたりするので、 Painless の出番は今後ますます増えてくるでしょう。練習も兼ねて、使えそうなスニペットをまとめていきたいと思います。

7.12 で Grok と Dissect が使えるようになるみたい、楽しみだ！

@ijokarumawak
ijokarumawak / forum_237985.ndjson
Created July 23, 2020 04:40
Sample Kibana visualization for the forum question 237985
{"attributes":{"fields":"[{\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"esTypes\":[\"_type\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"id\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"id.keywor
@ijokarumawak
ijokarumawak / 0.README.md
Last active September 9, 2023 03:01
The simplest NiFi Wait Notify example

This flow using Wait processor to wait for the all split FlowFiles to be processed. When it runs, following log messages can be seen:

2019-04-05 10:02:58,752 INFO [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 1/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,769 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 2/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,770 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 3/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:58,770 INFO [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.LogMessage LogMessage[id=eafe4890-0169-1000-9927-a81fbbafafc6] Processing 4/5 message for 875bd765-209d-4136-a75d-5d14241384fe
2019-04-05 10:02:5
@ijokarumawak
ijokarumawak / Wait_Notify_multiple_counter_names.xml
Created January 7, 2019 02:42
NiFi template: Wait Notify multiple counter names
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
<description></description>
<groupId>2615750e-0168-1000-cd37-5162c89dd01d</groupId>
<name>Wait Notify multiple counter names</name>
<snippet>
<controllerServices>
<id>a557c12c-476b-391d-0000-000000000000</id>
<parentGroupId>cd5035a3-f241-317a-0000-000000000000</parentGroupId>
<bundle>