FWIW: I (@rondy) am not the creator of the content shared here, which is an excerpt from Edmond Lau's book. I simply copied and pasted it from another location and saved it as a personal note, before it gained popularity on news.ycombinator.com. Unfortunately, I cannot recall the exact origin of the original source, nor was I able to find the author's name, so I am can't provide the appropriate credits.
Asynchronicity is the price to pay, you better know what you're paying for...
Let's share some vocabulary first:
Thread: The primitive responsible of executing code on the processor, you can give an existing (or a new) Thread some code, and it will execute it. Normally you can have a few hundreds on a JVM, arguments that you can tweak your way out to thousands. Worth noting that multitasking is achieved when using multiple Threads. Multiple Threads can exist for a single processor in which case multitasking happens when this processor switches between threads, called context switching, which will give the impression of things happenning in parallel. An example of a direct, and probably naive, use of a new Thread in Java:
public class MyRunnable implements Runnable {
public void run(){
System.out.println("MyRunnable running");
| # one way w/o web interface: | |
| # java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf | |
| # another with a web interface(http://localhost:9292) : | |
| # java -jar lib/playbooks/servers/files/logstash-1.1.1-monolithic.jar agent -f lib/playbooks/servers/files/logstash.conf -- web --backend 'elasticsearch:///?local' | |
| # nc localhost 3333 < logfile | |
| # | |
| # When you are all done, you can delete the 'data' folder to delete the database. | |
| input { | |
| tcp { |
There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.
I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.
Unfortunately, I've never used logstash in production. (Embarrassing, I know!)
I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."
| { | |
| "template": "logstash-*", | |
| "settings" : { | |
| "number_of_shards" : 1, | |
| "number_of_replicas" : 0, | |
| "index" : { | |
| "query" : { "default_field" : "@message" }, | |
| "store" : { "compress" : { "stored" : true, "tv": true } } | |
| } | |
| }, |
| #!/bin/sh | |
| # Do elasticsearch optimize on logstash previous day index | |
| # if $1 = all then optimize all indicies | |
| esindex="/opt/elasticsearch/data/elasticsearch/nodes/0/indices" | |
| # Grab yesterday's values | |
| D=`date +%d -d yesterday` | |
| M=`date +%m -d yesterday` | |
| Y=`date +%Y -d yesterday` |