Logstash Elasticsearch Template

logstash-template.json

{
    "template": "logstash-*",
    "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0,
        "index" : {
            "query" : { "default_field" : "@message" },
            "store" : { "compress" : { "stored" : true, "tv": true } }
        }
    },
    "mappings": {
        "_default_": { 
            "_all": { "enabled": false },
            "_source": { "compress": true },
            "dynamic_templates": [
                {
                    "string_template" : { 
                        "match" : "*",
                        "mapping": { "type": "string", "index": "not_analyzed" },
                        "match_mapping_type" : "string"
                     } 
                 }
             ],
             "properties" : {
                "@fields": { "type": "object", "dynamic": true, "path": "full" }, 
                "@message" : { "type" : "string", "index" : "analyzed" },
                "@source" : { "type" : "string", "index" : "not_analyzed" },
                "@source_host" : { "type" : "string", "index" : "not_analyzed" },
                "@source_path" : { "type" : "string", "index" : "not_analyzed" },
                "@tags": { "type": "string", "index" : "not_analyzed" }, 
                "@timestamp" : { "type" : "date", "index" : "not_analyzed" },
                "@type" : { "type" : "string", "index" : "not_analyzed" }
            }
        }
    }
}

set-template.sh

curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d @logstash-template.json

@sunny2385 create a directory called templates/ underneath your config directory and put the JSON file there. It will be auto-loaded next time Elasticsearch starts.

Note: there appears to be a bug whereby templates loaded from config are not visible (under some conditions) when using curl - see this post

Could you explain "store" : { "compress" : { "stored" : true, "tv": true } } a little bit please, I could not found corresponding documentation

@eclipse9614 It's pretty basic, but the settings are mentioned here: http://www.elasticsearch.org/guide/reference/index-modules/store/

Do you use Kibana to query your ES indexes?

I've tried your template against ES 0.90.5 and Kibana 3.0.0milestone4, and any indexes created using the template don't return any data to Kibana. The data is loaded into the index, and I run basic queries against it successfully myself, but if I try the query Kibana runs then no data is returned.

I'll carry on investigating myself, but was interested to know if you've seen the same problem.

For anybody who is interested - the line that was preventing Kibana from querying the index is:
"query" : { "default_field" : "@message" },

Remove that, re-index and Kibana works fine.

FYI i added:
"numeric_detection" : true,
to get automatically numeric fields (integer, long, float...).

@gregmruphy
how re-index? Please tell me the way

thanks !