Skip to content

Instantly share code, notes, and snippets.

kiyoto kiyoto

Block or report user

Report or block kiyoto

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@kiyoto
kiyoto / sample.log
Created Feb 5, 2014
Testing multiline for Windows
View sample.log
2013-3-03 14:27:33 [main] INFO Main - Start
2013-3-03 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
at Main.main(Main.java:16) ~[bin/:na]
2013-3-03 14:27:33 [main] INFO Main - End
View Fluentd1.sh
<source>
type tail
format apache2
path /var/log/apache2/access_log
pos_file /var/log/fluentd/apache2.access_log.pos
tag riak.apache
</source>
<match riak.**>
type riak
View Dockerfile
FROM base
MAINTAINER repeatedly "https://github.com/repeatedly"
RUN apt-get install -y --force-yes curl
RUN curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh
RUN sed -i 's/ulimit/# ulimit/g' /etc/init.d/td-agent
RUN echo "ulimit -n 8192" >> /etc/default/td-agent
@kiyoto
kiyoto / in_heroku_http.rb
Created Mar 14, 2014
Heroku HTTP logdrain input plugin for Fluentd
View in_heroku_http.rb
module Fluent
class HerokuHttpInput < Input
Plugin.register_input('heroku_http', self)
include DetachMultiProcessMixin
require 'http/parser'
def initialize
require 'webrick/httputils'
@kiyoto
kiyoto / ext_parser.md
Last active Aug 29, 2015
Fluentdのparser/format拡張について
View ext_parser.md

#FluentdのFormat追加の名前について

##名前に関する規則

個人的にはext_format_<format name>という感じがいいと思っています。<format name>に関しては、全部一か所に突っ込んでしまうと、場合によってはいらないパーサーもロードされるので、ある程度グルーピングするとよいかなと。例えばMTA系ののフォーマットに関するものはext_format_mailとか、MongoDBに関するフォーマットはext_format_mongoとか。でもって必要なext_format_<format name>をPluginディレクトリに置く・gem installするという感じ。

このアプローチの最大の問題は、ext_format_*系が量産される可能性があること。これに関しては、ある程度使われるとわかったものは、随時Coreにマージしてもよいかもしれない。

##フォルダ構成

@kiyoto
kiyoto / forest-example.conf
Created Apr 20, 2014
fluent-plugin-forest example
View forest-example.conf
<source>
type forward
</source>
<match nodelog.*>
type forest
subtype file
<template>
path test/nodelog.__TAG__
flush_interval 3s
@kiyoto
kiyoto / fluentd_philosophy.md
Last active Aug 29, 2015
Kiyoto's take on Fluentd's philosophy
View fluentd_philosophy.md

Buried in code, comments and issues is philosophy. Philosophy is important, for it informs judgment, which in turn informs decisions. Everything boils down to philosophy. Here is my take on it.

Fluentd's Philosophy (as I see it)

  • Keep the core lean and mean: we should strive to keep Fluentd itself simple and small. Whenever we feel like adding a new feature, let's remember the programming adage "the least buggy code is the code that's not written." Let's not be afraid to say "no" for feature requests into the core, especially if the end goal can be achieved with plugins.
  • Plugins should be easy to write and even easier to reuse: it's cool when someone can write their own plugin to fulfill their needs. It's even cooler if many more people can re-use that plugin. So, make it easy to write plugins but make it even easier to find and use existing plugins.
  • Be sincere and helpful to everyone: helping others and documenting features/use cases are just as important to the project as writing code
@kiyoto
kiyoto / hn.rb
Last active Aug 29, 2015
hacker news top page scraper
View hn.rb
require 'nokogiri'
require 'open-uri'
require 'json'
class HNParseError < RuntimeError; end
$stdout.sync = true
url = "https://news.ycombinator.com"
@kiyoto
kiyoto / example.conf
Last active Aug 29, 2015
Selecting a particular field for Fluentd to process further
View example.conf
<source>
type forward
</source>
<match test.**>
type select_fields
add_tag_prefix fields_selected
selected_field foo
</match>
@kiyoto
kiyoto / in_steam_friends_online.rb
Last active Aug 29, 2015
Steam friend online status plugin
View in_steam_friends_online.rb
module Fluent
class SteamFriendsOnlineInput < Input
Plugin.register_input('steam_friends_online', self)
config_param :steam_id, :integer
config_param :tag, :string, default:"steam_stats"
config_param :friend_nicknames, :string
config_param :stats_interval, :time, default:300
def initialize
You can’t perform that action at this time.