A simple Logstash example.

0_README.md

Simple Logstash example

• Load CSV data
• Apply some filters
• Changing index name besed on value

How to run

Clone this Gist on your host.

git clone https://gist.github.com/db7636ef860b3be7da4fc0ed7b2d1fe6.git logstash-demo

Install Logstash. The official doc has more installation options.

Installation via curl is convenient for testing purpose:

curl -OL https://artifacts.elastic.co/downloads/logstash/logstash-8.1.0-linux-x86_64.tar.gz
tar xvf logstash-8.1.0-linux-x86_64.tar.gz

Configure logstash.conf:

• Update the CSV file path
• Update Elasicsearch connection

Then, execute logstash:

{logstash-installation-full-path}/bin/logstash -f {logstash-demo-full-path}/logstash.conf

Multiple indices will be populated if it runs successfully: ls-demo-a, ls-demo-b ... etc

data.csv

logstash.conf

input {
  file {
    path => "/home/elastic/logstash-demo/data.csv"
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => [ "timestamp", "account", "amount" ]
    skip_header => true
    skip_empty_columns => true
  }

  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "Asia/Tokyo"
  }

  mutate {
    convert => {
      "amount" => "integer"
    }
    add_field => {"[user][id]" => "sample"}
  }

  mutate {
    remove_field => ["timestamp"]
  }
}

output {
  stdout {}

  elasticsearch {
    index => "ls-demo-%{[account]}"
    # If your elasticsearch is secured with SSL, enable ssl
    # ssl => true
    # If your elasticsearch uses a self-signed cert, and you can trust it, disable verification
    # ssl_certificate_verification => false
    hosts => "localhost"
    user => "changeme"
    password => "changeme"
  }
}