Skip to content

Instantly share code, notes, and snippets.

@jordansissel
Created July 11, 2012 07:03
Show Gist options
  • Save jordansissel/3088552 to your computer and use it in GitHub Desktop.
Save jordansissel/3088552 to your computer and use it in GitHub Desktop.
logstash + you + me.

logstash needs full time love.

There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.

I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.

Unfortunately, I've never used logstash in production. (Embarrassing, I know!)

I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."

It's bullshit, really. That changes now.

Right now, the big competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).

In 3-5 years, I want logstash to dominate everything in the logging space not simply by cost, but in form and function.

To get to that point, I need a faster feedback cycle as well as strong incentive to improve the project. To that end...

Hire me.

I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth.

Choosing logstash and hiring one or two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.

Join me.

Not hiring? The logstash community is growing super super fast and has some already amazing members who help out on IRC and the mailing list. Everyone is welcome.

Compete with me.

Can't join because you're the competition? Better get your shit together, because logstash will be winning the race very soon. It will be far more fun if we push each other, through competition, to solve problems in this area.

Gauntlet thrown, but not for violence. Let's do some awesome!

Roadmap

This is my rough roadmap for things to improve in logstash. Many (analytics, throughput, etc) are super important and will directly benefit anyone using logstash.

Search, Analytics, and Visuals

  • Pick up working on the vxin project again.
  • Port kibana to ruby to help ship it with logstash as well as gain more contributors (simply by logstash already being ruby)

Reduce Cost of Operation

  • Improve agent pipeline speeds
  • Find pipeline bottlenecks. Fix.
  • Improve on-disk data sizes when using elasticsearch.

Improve Releases

  • Make testing easier.
  • Release frequently.

Improve Management

  • Run-time API managing.
  • Make the agent monitorable (metrics!)
  • Make external plugins super easy to include.

Improve Documentation

  • Push hard for new cookbook recipes
  • Write more cookbook recipes

Improve Integration

  • Find relevant open source and commercial targets for input, filter, and output integration.
  • Find relevant targets for search/analytics integration.

Improve Agility

  • Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
  • Interactive multiline filter tool for the same.
  • Find any other things that slow folks down building logstash deployments, and fix them.

Improve Deployment

  • Logstash may be too fat to run in some places. Provide a light-weight log forwarding tool (similar to my syslog-shipper project)
  • Ship RPM and Deb packages
  • Ship upstart, systemd, smf, and sysv init configurations.
  • Puppet modules, cfengine thingies, chef cookbooks, vagrant vms, etc.

Grow the Community

  • If a newbie has a bad time, it's a bug.
  • If something doesn't work towards expectations, it's a bug - in documentation, code, somewhere.
  • Grow active IRC users
  • Grow active mailing list users
  • Grow active ticket users.
  • Build a community culture of hugs and support.

Research

  • Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
@jordansissel
Copy link
Author

@jaimegago - Yup! It's staying open source and my goal is to grow the project! :)

@tsenart
Copy link

tsenart commented Jul 22, 2012

Hey Jordan. Would you be interested in working for SoundCloud?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment