Slow Search Help

gistfile1.txt

Status
=============================
{
  "cluster_name" : "elasticsearchdp",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 10,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

Mapping for Index with slow queries
=============================
{
  "deal" : {
    "properties" : {
      "categories" : {
        "type" : "string"
      },
      "category_id" : {
        "type" : "string"
      },
      "created_at" : {
        "type" : "date",
        "format" : "dateOptionalTime"
      },
      "featured" : {
        "type" : "boolean"
      },
      "feed_importer_id" : {
        "type" : "string"
      },
      "gender" : {
        "type" : "string"
      },
      "gender_checked" : {
        "type" : "boolean"
      },
      "head_line" : {
        "type" : "string"
      },
      "id" : {
        "type" : "long"
      },
      "long_description" : {
        "type" : "string"
      },
      "percentage" : {
        "type" : "double"
      },
      "sales_price" : {
        "type" : "double"
      },
      "shipping_cost" : {
        "type" : "double"
      },
      "short_description" : {
        "type" : "string"
      },
      "staff_pick" : {
        "type" : "boolean"
      }
    }
  }
}

The deal index is store in 5 shards.


Example from Slow Log
=============================

[2012-04-04 00:16:16,688][TRACE][index.search.slowlog.query] [Yellowjacket] [deals][4] took[656ms], took_millis[656], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"sort":[{"sales_price":"asc"},{"sales_price":"asc"}],"facets":{"categories":{"terms":{"field":"categories","size":1000,"all_terms":false,"min":1}},"gender":{"terms":{"field":"gender","size":1000,"all_terms":false,"min":1}},"featured":{"terms":{"field":"featured","size":1000,"all_terms":false,"min":1}},"staff_pick":{"terms":{"field":"staff_pick","size":1000,"all_terms":false,"min":1}},"created_at":{"terms":{"field":"created_at","size":1000,"all_terms":false,"min":1}},"feed_importer_id":{"terms":{"field":"feed_importer_id","size":1000,"all_terms":false,"min":1}}},"filter":{"terms":{"categories":[1]}},"size":50,"from":6600}extra_source[], 
[2012-04-04 00:16:16,719][TRACE][index.search.slowlog.query] [Yellowjacket] [deals][1] took[704.7ms], took_millis[704], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"sort":[{"sales_price":"asc"},{"sales_price":"asc"}],"facets":{"categories":{"terms":{"field":"categories","size":1000,"all_terms":false,"min":1}},"gender":{"terms":{"field":"gender","size":1000,"all_terms":false,"min":1}},"featured":{"terms":{"field":"featured","size":1000,"all_terms":false,"min":1}},"staff_pick":{"terms":{"field":"staff_pick","size":1000,"all_terms":false,"min":1}},"created_at":{"terms":{"field":"created_at","size":1000,"all_terms":false,"min":1}},"feed_importer_id":{"terms":{"field":"feed_importer_id","size":1000,"all_terms":false,"min":1}}},"filter":{"terms":{"categories":[1]}},"size":50,"from":6600}extra_source[], 
[2012-04-04 00:16:16,739][TRACE][index.search.slowlog.query] [Yellowjacket] [deals][2] took[716.3ms], took_millis[716], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"sort":[{"sales_price":"asc"},{"sales_price":"asc"}],"facets":{"categories":{"terms":{"field":"categories","size":1000,"all_terms":false,"min":1}},"gender":{"terms":{"field":"gender","size":1000,"all_terms":false,"min":1}},"featured":{"terms":{"field":"featured","size":1000,"all_terms":false,"min":1}},"staff_pick":{"terms":{"field":"staff_pick","size":1000,"all_terms":false,"min":1}},"created_at":{"terms":{"field":"created_at","size":1000,"all_terms":false,"min":1}},"feed_importer_id":{"terms":{"field":"feed_importer_id","size":1000,"all_terms":false,"min":1}}},"filter":{"terms":{"categories":[1]}},"size":50,"from":6600}extra_source[],

General Info
=============================
OS : Linux 2.6.35-30-virtual #59-Ubuntu SMP x86_64 GNU/Linux
Memory : 12GB for Elastic Search Process 15GB for System
Index stored on EBS volumes Software Raid 10
4 Virtual Processors on box
18GB Index with 14 Million Documents
java -version
	java version "1.6.0_20"
	OpenJDK Runtime Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2)
	OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
	
1 Node
2 Indexes in Node

Configuration File
=============================
### Only things that I changed other than this it is default
cluster.name: elasticsearchdp
bootstrap.mlockall: true
http.max_content_length: 500mb