Skip to content

Instantly share code, notes, and snippets.

@hiroakis

hiroakis/error

Last active August 29, 2015 14:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hiroakis/5cad57a7ba3cecaa4ff5 to your computer and use it in GitHub Desktop.
Save hiroakis/5cad57a7ba3cecaa4ff5 to your computer and use it in GitHub Desktop.
Download ZIP
fluentd 0.10.51 error occurred when "include" tag exists in td-agent.conf
Raw
error
[root@xxxxx gems]# /etc/init.d/td-agent start
Starting td-agent: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:57:in `process_include': uninitialized constant Fluent::Config::Parser::URI (NameError)
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:41:in `parse!'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:8:in `parse'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config.rb:33:in `parse'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/supervisor.rb:364:in `apply_system_config'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/supervisor.rb:110:in `initialize'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/command/fluentd.rb:160:in `new'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/command/fluentd.rb:160:in `<top (required)>'
from /usr/lib64/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
from /usr/lib64/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/bin/fluentd:6:in `<top (required)>'
from /usr/lib64/fluent/ruby/bin/fluentd:23:in `load'
from /usr/lib64/fluent/ruby/bin/fluentd:23:in `<top (required)>'
from /usr/sbin/td-agent:7:in `load'
from /usr/sbin/td-agent:7:in `<main>'
[失敗]
Raw
memo.md

memo

  • 現象の発生したサーバ(下記環境欄参照)にはfluentd (0.10.36)がRPMからインストールされている。
  • このサーバで、bigqueryプラグインを入れた。 /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-bigquery
  • これによりプラグインがインストールされるとともに、fluentdが0.10.51にアップグレードされた。ここで、td-agent.confにincludeディレクティブが含まれていると、td-agent起動時に前述したエラーが発生した。
  • なお、問題切り分けのために、上記fluent-plugin-bigqueryをインストールせずに、/usr/lib64/fluent/ruby/bin/fluent-gem install fluentdでfluentdを最新版(現時点では0.10.51)に上げただけでも、同様のエラーが発生した。

解消

  • 次のように、0.10.50を入れて、0.10.51をアンインストールすると、現象は解消した。
/usr/lib64/fluent/ruby/bin/fluent-gem install fluentd -v 0.10.50
/usr/lib64/fluent/ruby/bin/fluent-gem uninstall fluentd -v 0.10.51

環境(OS, RPM)

$ cat /etc/issue
CentOS release 5.4 (Final)

$ uname -r
2.6.18-164.el5

$ sudo yum info td-agent
Name       : td-agent
Arch       : x86_64
Version    : 1.1.15
Release    : 0
Size       : 83 M
Repo       : installed
Summary    : td-agent
URL        : http://treasure-data.com/
License    : APL2
Description:

環境(Ruby/fluentd)

  • /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-xxxxを行う前(fluentd-0.10.51にupgradeされる前)
$ /usr/lib64/fluent/ruby/bin/ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

$ /usr/lib64/fluent/ruby/bin/fluentd -c /etc/td-agent/td-agent.conf
2014-07-09 02:56:21 +0900 [info]: starting fluentd-0.10.36
2014-07-09 02:56:21 +0900 [info]: reading config file path="/etc/td-agent/td-agent.conf"
  • /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-xxxxを行った後(fluentd-0.10.51にupgradeされた後)
$ /usr/lib64/fluent/ruby/bin/ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

$ /usr/lib64/fluent/ruby/bin/fluentd -c /etc/td-agent/td-agent.conf
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:57:in `process_include': uninitialized constant Fluent::Config::Parser::URI (NameError)
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:41:in `parse!'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config/parser.rb:8:in `parse'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/config.rb:33:in `parse'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/supervisor.rb:364:in `apply_system_config'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/supervisor.rb:110:in `initialize'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/command/fluentd.rb:160:in `new'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/lib/fluent/command/fluentd.rb:160:in `<top (required)>'
        from /usr/lib64/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
        from /usr/lib64/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
        from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.51/bin/fluentd:6:in `<top (required)>'
        from /usr/lib64/fluent/ruby/bin/fluentd:23:in `load'
        from /usr/lib64/fluent/ruby/bin/fluentd:23:in `<main>'
Raw
td-agent.conf
<source>
type tail
format /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/
time_format %d/%b/%Y:%H:%M:%S %z
path /usr/local/apache/logs/xxxxx
pos_file /var/log/td-agent/xxxx.pos
tag tail.accesslog
</source>
<match tail.accesslog>
type copy
<store>
type datacounter
count_interval 10m
aggregate all
count_key status
pattern1 4xx ^4\d\d$
pattern2 5xx ^5\d\d$
tag datacounter.accesslog
</store>
<store>
type bigquery
auth_method private_key
email xxxxxxxxxxxxxxx
private_key_path xxxxxxxxxxx
project xxxxxx
dataset xxxxxx
tables xxxxx,xxxx,xxxx
flush_interval 1
buffer_chunk_records_limit 1000
buffer_queue_limit 1024
num_threads 50
time_format %s
time_field time
field_string host,user,method,path,referer,agent
field_integer time,response_time,status,size
</store>
</match>
<match datacounter.accesslog>
type file
path /var/log/td-agent/apache/status
</match>
include conf.d/*.conf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment