Skip to content

Instantly share code, notes, and snippets.

@hjhart
Last active March 15, 2023 15:04
Show Gist options
  • Save hjhart/6057dd1feadc1c99b91f to your computer and use it in GitHub Desktop.
Save hjhart/6057dd1feadc1c99b91f to your computer and use it in GitHub Desktop.
Trial Fluentd configuration for parsing HAProxy logs from Syslog
<source>
type tail
path /var/log/haproxy.log
pos_file /opt/fluentd/haproxy.log.post
format /^(?<syslog_time>.+) (?<source_ip>.+) (?<ps>\w+)\[(?<pid>\d+)\]: (?<c_ip>[\w\.]+):(?<c_port>\d+) \[(?<time>.+)\] (?<f_end>[\w\.-]+) (?<b_end>[\w\.-]+)\/(?<b_server>[\w\.-]+) (?<tq>\d+)\/(?<tw>\d+)\/(?<tc>\d+)\/(?<tr>\d+)\/(?<tt>\d+) (?<status_code>\d+) (?<bytes_read>\d+) (?<captured_request_cookie>.+) (?<captured_response_cookie>.+) (?<termination_state>.+) (?<actconn>\d+)\/(?<feconn>\d+)\/(?<beconn>\d+)\/(?<srv_conn>\d+)\/(?<retries>\d+) (?<srv_queue>\d+)\/(?<backend_queue>\d+) "(?<http_request>.+)"$/
tag debug.haproxy.http
time_format %d/%B/%Y:%H:%M:%S
</source>
<match debug.**>
type stdout
</match>
<match debug.**>
type elasticsearch
port 9200
host localhost
index_name fluentd
type_name fluentd
logstash_format true
flush_interval 10s # for testing
</match>
source 'https://rubygems.org'
gem 'fluentd'
gem 'fluent-plugin-td'
gem 'fluent-plugin-elasticsearch'
# Installing fluentd on SmartOS
```
sudo rm /var/db/pkgin/pkgin.db
sudo pkgin update # I hate you pkgin
sudo pkgin in ruby-2.1.2
ruby -v
```
Install gems
```
sudo gem install bundler
bundle --path vendor/bundle
```
Configure fluentd
```
cd /opt/fluentd
vi Gemfile
```
Setup and run!
```
bundle exec fluentd --setup .
bundle exec fluentd -c ./fluent.conf -vv
```
@hjhart
Copy link
Author

hjhart commented Sep 27, 2014

Particularly, HTTP default logs from HAProxy as defined in section 8.2.3. HTTP log format in http://www.haproxy.org/download/1.4/doc/configuration.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment