Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
iAPのリクエストログをembulkでelasticsearchに読み込んでみる

セットアップ

  • JAVA_HOMEを設定しておく
  • KibanaをDL
  • elasticsearchをDL
  • それぞれ展開してコマンドラインから起動
embulkのインストールとセットアップ
PowerShell -Command "& {Invoke-WebRequest http://dl.embulk.org/embulk-latest.jar -OutFile embulk.bat}"
embulk mkbundle iaplog-bundle
echo gem 'embulk-output-elasticsearch' >> iaplog-bundle
embulk bundle install -b iaplog-bundle
設定ファイルを作成後プレビュー実施
embulk.bat preview -G iaplogs\iap-log-load.yaml -b iaplog-bundle
実行
embulk.bat run -b iaplog-bundle iaplogs\iap-log-load.yaml -o iaplogs\iap-log-load.yaml 
in:
type: file
path_prefix: C:\path\to\resin-4.0.47\webapps\imart\WEB-INF\log\platform\request
parser:
charset: UTF-8
newline: CRLF
type: csv
delimiter: "\t"
quote: '"'
escape: '"'
trim_if_not_quoted: false
skip_header_lines: 0
allow_extra_columns: false
allow_optional_columns: false
columns:
- {name: timstamp, type: timestamp, format: '[%Y-%m-%d %H:%M:%S.%L]'}
- {name: thread, type: string}
- {name: sequence, type: long}
- {name: level, type: string}
- {name: logger, type: string}
- {name: tenant, type: string}
- {name: log_id, type: string}
- {name: no_entry, type: string}
- {name: session, type: string}
- {name: remote_host, type: string}
- {name: http_method, type: string}
- {name: url, type: string}
- {name: query_string, type: string}
- {name: referer, type: string}
- {name: page_time, type: long}
- {name: accept_tyme, type: timestamp, format: '%Y-%m-%d %H:%M:%S,%L'}
- {name: request, type: string}
last_path: C:\path\to\resin-4.0.47\webapps\imart\WEB-INF\log\platform\request.log
out:
type: elasticsearch
nodes:
- {host: localhost, port: 9300}
index: iap-request-logs
index_type: iap-request-logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment