Skip to content

Instantly share code, notes, and snippets.

Avatar
🎯
Focusing

Koji Kawamura ijokarumawak

🎯
Focusing
View GitHub Profile
@ijokarumawak
ijokarumawak / 0_README.md
Last active Jun 15, 2022
Use Logstash ruby filter to rename field name based on value type to avoid Elasticsearch mapping conflict
View 0_README.md

This is Logstash ruby filter example to rename field names within a specified hash, based on each value type. The original question was asked at this Elastic Discuss.

How to test the filter

With -t, the filter can be tested:

logstash -e "filter { ruby { path => '/{path_of_the_ruby_script}/rename_by_data_type.rb' script_params => { 'field' => 'the_name_of_target_field' } } }" -t

Output:

@ijokarumawak
ijokarumawak / 0.Runtime Field Dashboard sample.md
Last active Apr 12, 2022
Runtime Field を利用した Kibana ダッシュボードサンプル
View 0.Runtime Field Dashboard sample.md

Runtime Field を利用した Kibana ダッシュボードサンプル

"socks_a" のように、カテゴリ名_会社名という値を Runtime field を使い、カテゴリ名、会社名と別々のフィールドとして扱うサンプル。

# 1. サンプルデータの登録
POST items-sample/_bulk
{"index": {}}
{"sku":"socks_a","price":500}
{"index": {}}
@ijokarumawak
ijokarumawak / ReusableTimeUnitExtractionPainlessScript.md
Last active Mar 11, 2022
A stored Painless script example for Elasticsearch Ingest node pipeline to extract certain time unit from a timestamp value.
View ReusableTimeUnitExtractionPainlessScript.md
# Create a stored script
PUT _scripts/getTimestampUnitValue
{
  "script": {
    "lang": "painless", 
    "source": """
    def value = ctx[params['field']];
    if (value == null) {
      return;
@ijokarumawak
ijokarumawak / Query timestamp only by hours.md
Created Dec 15, 2021
A runtime field example at query time.
View Query timestamp only by hours.md
GET blogs/_search
{
  "runtime_mappings": {
    "publish_hour": {
      "type": "long",
      "script": {
        "source": "emit(doc['publish_date'].value.hour)"
      }
 }
@ijokarumawak
ijokarumawak / Kibana-Lens-multi-metrics.md
Last active Oct 19, 2021
Show multiple metrics in the same Kibana Lens chart
View Kibana-Lens-multi-metrics.md

Show two metrics using different chart styles.

  • Use two layers, line chart and bar chart
  • Use the same x-axis among the two layers, order_date date histogram in this example
  • Use different colors
  • Separate y-axis
View test-beats-processors.md

How to test Beats processors

To test Beats processors, beats playground can be helpful.

Input:

2019-09-29 STATUS_OK Server started normally. 92ms
2019-09-30 STATUS_OK Server started normally. 120ms
2019-10-01 STATUS_NG Server stopped abnormally. Timed out. 3010ms
View README.md

Set up a Data Stream

# Delete the stream in order to clear old data
DELETE _data_stream/my-event-stream

# Define a template
PUT /_index_template/my-event-stream-template
{
  "index_patterns": [
@ijokarumawak
ijokarumawak / can_we_change_coerce.md
Last active Jun 22, 2021
Can we turn off coercing without having to reindex? Yes, we can.
View can_we_change_coerce.md

Create the num field as long with auto-mapping.

PUT coerce_test/_doc/1
{
  "num": 1
}

By default, coerce is true. That means a string value which can be converted into a number, can be ingested.

@ijokarumawak
ijokarumawak / serialize_test.go
Created Jun 16, 2021
Libbeat diskqueue cannot handle multi-byte message correctly.
View serialize_test.go
package diskqueue
import (
"encoding/json"
"testing"
"github.com/elastic/beats/v7/libbeat/beat"
"github.com/elastic/beats/v7/libbeat/common"
"github.com/elastic/beats/v7/libbeat/publisher"
"github.com/stretchr/testify/assert"
@ijokarumawak
ijokarumawak / 0_create_index.md
Created May 20, 2021
Wikipedia 日本語ページを Elasticsearch に登録する Logstash サンプル
View 0_create_index.md

Kuromoji で本文を解析できるようにマッピングを作成。

PUT jawiki
{
  "mappings": {
    "properties": {
      "doc": {
        "properties": {
 "revision": {