Skip to content

Instantly share code, notes, and snippets.

kiyoto kiyoto

Block or report user

Report or block kiyoto

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
kiyoto / sample.log
Created Feb 5, 2014
Testing multiline for Windows
View sample.log
2013-3-03 14:27:33 [main] INFO Main - Start
2013-3-03 14:27:33 [main] ERROR Main - Exception null
at Main.main( ~[bin/:na]
2013-3-03 14:27:33 [main] INFO Main - End
type tail
format apache2
path /var/log/apache2/access_log
pos_file /var/log/fluentd/apache2.access_log.pos
tag riak.apache
<match riak.**>
type riak
View Dockerfile
FROM base
MAINTAINER repeatedly ""
RUN apt-get install -y --force-yes curl
RUN curl -L | sh
RUN sed -i 's/ulimit/# ulimit/g' /etc/init.d/td-agent
RUN echo "ulimit -n 8192" >> /etc/default/td-agent
kiyoto / in_heroku_http.rb
Created Mar 14, 2014
Heroku HTTP logdrain input plugin for Fluentd
View in_heroku_http.rb
module Fluent
class HerokuHttpInput < Input
Plugin.register_input('heroku_http', self)
include DetachMultiProcessMixin
require 'http/parser'
def initialize
require 'webrick/httputils'
kiyoto /
Last active Aug 29, 2015



個人的にはext_format_<format name>という感じがいいと思っています。<format name>に関しては、全部一か所に突っ込んでしまうと、場合によってはいらないパーサーもロードされるので、ある程度グルーピングするとよいかなと。例えばMTA系ののフォーマットに関するものはext_format_mailとか、MongoDBに関するフォーマットはext_format_mongoとか。でもって必要なext_format_<format name>をPluginディレクトリに置く・gem installするという感じ。



kiyoto / forest-example.conf
Created Apr 20, 2014
fluent-plugin-forest example
View forest-example.conf
type forward
<match nodelog.*>
type forest
subtype file
path test/nodelog.__TAG__
flush_interval 3s
kiyoto /
Last active Aug 29, 2015
Kiyoto's take on Fluentd's philosophy

Buried in code, comments and issues is philosophy. Philosophy is important, for it informs judgment, which in turn informs decisions. Everything boils down to philosophy. Here is my take on it.

Fluentd's Philosophy (as I see it)

  • Keep the core lean and mean: we should strive to keep Fluentd itself simple and small. Whenever we feel like adding a new feature, let's remember the programming adage "the least buggy code is the code that's not written." Let's not be afraid to say "no" for feature requests into the core, especially if the end goal can be achieved with plugins.
  • Plugins should be easy to write and even easier to reuse: it's cool when someone can write their own plugin to fulfill their needs. It's even cooler if many more people can re-use that plugin. So, make it easy to write plugins but make it even easier to find and use existing plugins.
  • Be sincere and helpful to everyone: helping others and documenting features/use cases are just as important to the project as writing code
kiyoto / hn.rb
Last active Aug 29, 2015
hacker news top page scraper
View hn.rb
require 'nokogiri'
require 'open-uri'
require 'json'
class HNParseError < RuntimeError; end
$stdout.sync = true
url = ""
kiyoto / example.conf
Last active Aug 29, 2015
Selecting a particular field for Fluentd to process further
View example.conf
type forward
<match test.**>
type select_fields
add_tag_prefix fields_selected
selected_field foo
kiyoto / in_steam_friends_online.rb
Last active Aug 29, 2015
Steam friend online status plugin
View in_steam_friends_online.rb
module Fluent
class SteamFriendsOnlineInput < Input
Plugin.register_input('steam_friends_online', self)
config_param :steam_id, :integer
config_param :tag, :string, default:"steam_stats"
config_param :friend_nicknames, :string
config_param :stats_interval, :time, default:300
def initialize
You can’t perform that action at this time.