There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.
I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.
Unfortunately, I've never used logstash in production. (Embarrassing, I know!)
I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."
It's bullshit, really. That changes now.
Right now, the big competition is Splunk, and at present logstash makes a better value proposition only (based on discussions with folks who have experience with both). It lacks many features (though surpassing in other areas).
In 3-5 years, I want logstash to dominate everything in the logging space not simply by cost, but in form and function.
To get to that point, I need a faster feedback cycle as well as strong incentive to improve the project. To that end...
I'm looking for somewhere that does billions of events per day, would benefit from logstash, and would benefit from investing in engineering time in it. I need this to gather production experience with logstash for the purposes of fast feedback, incentive to code, and performance growth.
Choosing logstash and hiring one or two full-time engineers is cheaper (and actually provides increasing ROI) than a Splunk license.
Not hiring? The logstash community is growing super super fast and has some already amazing members who help out on IRC and the mailing list. Everyone is welcome.
Can't join because you're the competition? Better get your shit together, because logstash will be winning the race very soon. It will be far more fun if we push each other, through competition, to solve problems in this area.
Gauntlet thrown, but not for violence. Let's do some awesome!
This is my rough roadmap for things to improve in logstash. Many (analytics, throughput, etc) are super important and will directly benefit anyone using logstash.
- Pick up working on the vxin project again.
- Port kibana to ruby to help ship it with logstash as well as gain more contributors (simply by logstash already being ruby)
- Improve agent pipeline speeds
- Find pipeline bottlenecks. Fix.
- Improve on-disk data sizes when using elasticsearch.
- Make testing easier.
- Release frequently.
- Run-time API managing.
- Make the agent monitorable (metrics!)
- Make external plugins super easy to include.
- Push hard for new cookbook recipes
- Write more cookbook recipes
- Find relevant open source and commercial targets for input, filter, and output integration.
- Find relevant targets for search/analytics integration.
- Interactive grok tool for helping write new patterns, test and debug broken ones, etc.
- Interactive multiline filter tool for the same.
- Find any other things that slow folks down building logstash deployments, and fix them.
- Logstash may be too fat to run in some places. Provide a light-weight log forwarding tool (similar to my syslog-shipper project)
- Ship RPM and Deb packages
- Ship upstart, systemd, smf, and sysv init configurations.
- Puppet modules, cfengine thingies, chef cookbooks, vagrant vms, etc.
- If a newbie has a bad time, it's a bug.
- If something doesn't work towards expectations, it's a bug - in documentation, code, somewhere.
- Grow active IRC users
- Grow active mailing list users
- Grow active ticket users.
- Build a community culture of hugs and support.
- Grow the 'studies' portion of the cookbook. Learn more about real log examples. Find common patterns. Generalize into code.
Awesome! Great to know about the roadmap,