Skip to content

Instantly share code, notes, and snippets.

kiyoto kiyoto

Block or report user

Report or block kiyoto

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@kiyoto
kiyoto / failed_plugin_tests.log
Created Feb 1, 2015
Failed plugin test for Fluentd Windows in the test suite @v0.10.46
View failed_plugin_tests.log
-------------------------------------------------------------------
test_in_exec.rb
-------------------------------------------------------------------
Run options:
# Running tests:
......
Finished tests in 8.276824s, 0.7249 tests/s, 1.8123 assertions/s.
@kiyoto
kiyoto / fluent.conf
Created Jan 27, 2015
EFK for Apache
View fluent.conf
<source>
type tail
tag apache.access
format apache2
path /path/to/your/log
read_from_head true #過去ログ用。ファイルの先頭から読む
</source>
<match apache.access>
type elasticsearch
View README.md

With this config, if your hostname is koala and if you have files like

log/t1.log
   /t2.log
   /...

They are sent to

View fluentd-recover.rb
require 'fluent-logger'
# specify the host and port to forward data to
logger = Fluent::Logger.new(nil, :host => "localhost", :port => 24224)
fn = ARGV[0]
fh = File.new(fn)
msgpack_unpacker = MessagePack::Unpacker.new(fh)
View filter_if.rb
module Fluent
class IfFilter < Filter
Fluent::Plugin.register_filter('if', self)
config_param :if, :string
def configure(conf)
super
@placeholder_expander = RubyPlaceholderExpander.new(log)
@kiyoto
kiyoto / file0.txt
Last active Aug 29, 2015
DevNullParserで始めるFluentd パーサプラグイン入門 ref: http://qiita.com/kiyoto/items/9903718147adde4d6c32
View file0.txt
<source>
type tail
format json
path /path/to/file
tag aoi.yu
</source>
@kiyoto
kiyoto / fluentd-docs-versioning.markdown
Last active Aug 29, 2015
Versioning Fluentd's Documentation
View fluentd-docs-versioning.markdown

Versioning Fluentd's Documentation

As Fluentd starts rolling out new major uprates (v0.12 & v0.14) and inches towards the v1 release, I want to re-organize and version the docs website

Here is my plan.

  1. All current URLs will start pointing to the latest release. This does NOT include preview releases.
  2. For each page, the version information is embedded as follows: https://docs.fluentd.org/<LANGUAGE>/<VERSION>/<PAGE_NAME>. Ex. https://docs.fluentd.org/ja/v0-12/config-file. The pages that have version info in the URL are called Versioned Pages.
  3. For each Versioned Page that is not the latest version, the following warning shows up on the page:
View tunguz.rb
require 'nokogiri'
require 'open-uri'
require 'json'
BASE_URL = 'http://www.tomtunguz.com'
class URLFormatError < RuntimeError; end
def fetch_categories(base_url)
html = open(base_url).read
@kiyoto
kiyoto / update_flume_plugin.markdown
Last active Aug 29, 2015
Going about updateing fluent-plugin-flume
View update_flume_plugin.markdown

This is a little note about how I would go about updating fluent-plugin-flume to resolve this issue.

Cause

The current fluent-plugin-flume supports 0.9 protocol. However, most curernt deployments are on the 1.5 branch. This causes a compatibility issue because the Thrift definition for Flume changed between 0.9 and 1.5.

How to fix it

  1. We need to update the flume.thrift file. The Thrift file definition for Flume 1.5 can be found here.
  2. The actual Thrift server code in Ruby should be generated using Thrift compiler. The 0.9.1 release should work. Use the updated flume.thrift to generate the server code (in lib/fluent/plugin/thrift)
@kiyoto
kiyoto / fluent.conf
Last active Aug 29, 2015
hsinclar-test.conf
View fluent.conf
# sender td-agent.conf
#---------------------------------------------------
<source>
type forward
</source>
<match apache.*>
type secure_forward
shared_key somekey
You can’t perform that action at this time.