Skip to content

Instantly share code, notes, and snippets.

@wallyqs

wallyqs/a-sample.org

Last active August 29, 2015 13:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wallyqs/9878741 to your computer and use it in GitHub Desktop.
Save wallyqs/9878741 to your computer and use it in GitHub Desktop.
Download ZIP
Current status of org-converge tool
Raw
a-sample.org

Setting up Fluentd example

Introduction

The following will setup and configure fluentd on a node using the org-converge tool.

It can be installed via Rubygems:

gem install org-converge

Setup

Fluentd can be installed either as a package from treasure data or just as a gem. In this example, we only install as a gem:

gem list | grep fluentd
fluentd_installed=$?
if [ $fluentd_installed -ne 0 ]; then
  echo "Installing fluentd...";
  gem install fluentd
fi

gem list | grep fluent-logger
fluentd_logger_installed=$?
if [ $fluentd_logger_installed -ne 0 ]; then
  echo "Installing fluent-logger for Ruby...";
  gem install fluent-logger
fi

Configuration

Set Fluentd to listen on the fluentd_port port and to write logs to fluentd_filepath.

<source>
  type forward
  port 4224
</source>

<match **>
  type file
  path here.log
  flush_interval 0s
</match>

Flusher

Send some logs:

require 'fluent-logger'

sleep 2 # Wait for fluentd to start at the beginning

Fluent::Logger::FluentLogger.open(nil, :host => 'localhost', :port => 4224)

100.times do |n|
  Fluent::Logger.post("myapp.access", {"agent"=>"foo - #{n}"})
  sleep 0.1
end

Tailer

And yet another process which will be tailing the file that we are sending logs too:

tail -f here.*

Start which was configured here

fluentd -c fluentd.conf -vvv
Raw
output
$ gem install org-converge
Successfully installed org-converge-0.0.2
1 gem installed
Installing ri documentation for org-converge-0.0.2...
Building YARD (yri) index for org-converge-0.0.2...
Installing RDoc documentation for org-converge-0.0.2...
$ org-converge setup.org
I, [2014-03-31T04:52:52.748576 #54964] INFO -- : Tangling 1 files...
I, [2014-03-31T04:52:52.748758 #54964] INFO -- : BEGIN(fluentd.conf): Tangling 10 lines at 'fluentd.conf'
W, [2014-03-31T04:52:52.748832 #54964] WARN -- : File already exists at fluentd.conf, it will be overwritten
I, [2014-03-31T04:52:52.749087 #54964] INFO -- : END(fluentd.conf): done.
I, [2014-03-31T04:52:52.749131 #54964] INFO -- : Tangling succeeded!
I, [2014-03-31T04:52:52.749507 #54964] INFO -- : Tangling 4 scripts within directory: /Users/mariko/repos/org-converge/examples/fluentd/run...
I, [2014-03-31T04:52:52.750992 #54964] INFO -- : Running code blocks now! (4 runnable blocks found in total)
I, [2014-03-31T04:52:52.814933 #54964] INFO -- : sh(54979) -- started with pid 54979
I, [2014-03-31T04:52:52.815106 #54964] INFO -- : ruby(54980) -- started with pid 54980
I, [2014-03-31T04:52:52.815215 #54964] INFO -- : sh(54981) -- started with pid 54981
I, [2014-03-31T04:52:52.815313 #54964] INFO -- : bash(54984) -- started with pid 54984
I, [2014-03-31T04:52:52.855450 #54964] INFO -- : sh(54981) -- tail: here.*: No such file or directory
I, [2014-03-31T04:52:53.039736 #54964] INFO -- : sh(54981) -- exited with code 1
I, [2014-03-31T04:52:54.231428 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [info]: fluent/supervisor.rb:171:supervise: starting fluentd-0.10.45
I, [2014-03-31T04:52:54.245321 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [info]: fluent/supervisor.rb:273:read_config: reading config file path="fluentd.conf"
I, [2014-03-31T04:52:54.284529 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'file'
I, [2014-03-31T04:52:54.291990 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered buffer plugin 'memory'
I, [2014-03-31T04:52:54.292173 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'debug_agent'
I, [2014-03-31T04:52:54.435863 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'exec'
I, [2014-03-31T04:52:54.436022 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'forward'
I, [2014-03-31T04:52:54.730910 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'gc_stat'
I, [2014-03-31T04:52:54.731207 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'http'
I, [2014-03-31T04:52:54.829701 #54964] INFO -- : sh(54979) -- fluentd (0.10.45)
I, [2014-03-31T04:52:54.829999 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:54 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'monitor_agent'
I, [2014-03-31T04:52:55.181676 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'object_space'
I, [2014-03-31T04:52:55.181849 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'status'
I, [2014-03-31T04:52:55.292744 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'tcp'
I, [2014-03-31T04:52:55.292910 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'unix'
I, [2014-03-31T04:52:55.501097 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'syslog'
I, [2014-03-31T04:52:55.507507 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered input plugin 'tail'
I, [2014-03-31T04:52:55.586127 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'copy'
I, [2014-03-31T04:52:55.586298 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'exec'
I, [2014-03-31T04:52:55.714930 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'exec_filter'
I, [2014-03-31T04:52:55.715107 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'file'
I, [2014-03-31T04:52:55.800036 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'forward'
I, [2014-03-31T04:52:55.800427 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'null'
I, [2014-03-31T04:52:55.890905 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'roundrobin'
I, [2014-03-31T04:52:55.891283 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'stdout'
I, [2014-03-31T04:52:56.024332 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'tcp'
I, [2014-03-31T04:52:56.025385 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'unix'
I, [2014-03-31T04:52:56.099541 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: fluent/plugin.rb:87:register_impl: registered output plugin 'test'
I, [2014-03-31T04:52:56.099714 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [info]: fluent/engine.rb:86:block in configure: gem 'fluentd' version '0.10.45'
I, [2014-03-31T04:52:56.206337 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [info]: fluent/engine.rb:90:configure: using configuration file: <ROOT>
I, [2014-03-31T04:52:56.206491 #54964] INFO -- : bash(54984) -- <source>
I, [2014-03-31T04:52:56.344324 #54964] INFO -- : bash(54984) -- type forward
I, [2014-03-31T04:52:56.344491 #54964] INFO -- : bash(54984) -- port 4224
I, [2014-03-31T04:52:56.344635 #54964] INFO -- : bash(54984) -- </source>
I, [2014-03-31T04:52:56.471526 #54964] INFO -- : bash(54984) -- <match **>
I, [2014-03-31T04:52:56.471708 #54964] INFO -- : bash(54984) -- type file
I, [2014-03-31T04:52:56.471844 #54964] INFO -- : bash(54984) -- path here.log
I, [2014-03-31T04:52:56.655853 #54964] INFO -- : sh(54979) -- fluent-logger (0.4.7)
I, [2014-03-31T04:52:56.655980 #54964] INFO -- : bash(54984) -- flush_interval 0s
I, [2014-03-31T04:52:56.656136 #54964] INFO -- : bash(54984) -- </match>
I, [2014-03-31T04:52:56.774518 #54964] INFO -- : bash(54984) -- </ROOT>
I, [2014-03-31T04:52:56.774684 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [info]: fluent/engine.rb:100:block in configure: adding source type="forward"
I, [2014-03-31T04:52:56.774833 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [info]: fluent/engine.rb:116:block in configure: adding match pattern="**" type="file"
I, [2014-03-31T04:52:56.917506 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [info]: plugin/in_forward.rb:68:listen: listening fluent socket on 0.0.0.0:4224
I, [2014-03-31T04:52:56.917682 #54964] INFO -- : bash(54984) -- 2014-03-31 04:52:55 +0900 [trace]: plugin/in_forward.rb:158:initialize: accepted fluent socket from '127.0.0.1:50118': object_id=70106442023200
I, [2014-03-31T04:52:57.017640 #54964] INFO -- : sh(54979) -- exited with code 0
I, [2014-03-31T04:53:05.823168 #54964] INFO -- : bash(54984) -- 2014-03-31 04:53:05 +0900 [trace]: plugin/in_forward.rb:201:on_close: closed fluent socket object_id=70106442023200
I, [2014-03-31T04:53:05.828068 #54964] INFO -- : ruby(54980) -- exited with code 0
C-c C-cSIGINT received
I, [2014-03-31T04:53:25.783240 #54964] INFO -- : system -- sending SIGTERM to all processes
I, [2014-03-31T04:53:25.784163 #54964] INFO -- : bash(54984) -- terminated by SIGTERM
I, [2014-03-31T04:53:25.784239 #54964] INFO -- : Run has completed successfully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment