iAPのリクエストログをembulkでelasticsearchに読み込んでみる

README.md

セットアップ

• JAVA_HOMEを設定しておく
• KibanaをDL
• elasticsearchをDL
• それぞれ展開してコマンドラインから起動

embulkのインストールとセットアップ

PowerShell -Command "& {Invoke-WebRequest http://dl.embulk.org/embulk-latest.jar -OutFile embulk.bat}"
embulk mkbundle iaplog-bundle
echo gem 'embulk-output-elasticsearch' >> iaplog-bundle
embulk bundle install -b iaplog-bundle

設定ファイルを作成後プレビュー実施

embulk.bat preview -G iaplogs\iap-log-load.yaml -b iaplog-bundle

実行

embulk.bat run -b iaplog-bundle iaplogs\iap-log-load.yaml -o iaplogs\iap-log-load.yaml 

iap-request-log-load-to-es.yaml

in:
  type: file
  path_prefix: C:\path\to\resin-4.0.47\webapps\imart\WEB-INF\log\platform\request
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: "\t"
    quote: '"'
    escape: '"'
    trim_if_not_quoted: false
    skip_header_lines: 0
    allow_extra_columns: false
    allow_optional_columns: false
    columns:
    - {name: timstamp, type: timestamp, format: '[%Y-%m-%d %H:%M:%S.%L]'}
    - {name: thread, type: string}
    - {name: sequence, type: long}
    - {name: level, type: string}
    - {name: logger, type: string}
    - {name: tenant, type: string}
    - {name: log_id, type: string}
    - {name: no_entry, type: string}
    - {name: session, type: string}
    - {name: remote_host, type: string}
    - {name: http_method, type: string}
    - {name: url, type: string}
    - {name: query_string, type: string}
    - {name: referer, type: string}
    - {name: page_time, type: long}
    - {name: accept_tyme, type: timestamp, format: '%Y-%m-%d %H:%M:%S,%L'}
    - {name: request, type: string}
  last_path: C:\path\to\resin-4.0.47\webapps\imart\WEB-INF\log\platform\request.log
out:
  type: elasticsearch
  nodes:
  - {host: localhost, port: 9300}
  index: iap-request-logs
  index_type: iap-request-logs