Alex-Ikanow

--------------------------------
SHARD1 LOGS:

Soon after adding the second shard:

Mon Feb 18 16:10:31 [conn68] moveChunk updating self version to: 2|1||000000000000000000000000 through { index: "000007bdc6f2a4f19f36149b08cac55" } -> { index: "01d8d8f3fcac4efa684888aa46d9008f" } for collection 'feature.association'

---

Mon Feb 18 16:11:18 [conn29881] assertion 13388 [feature.association] shard version not ok in Client::Context: version mismatch detected for feature.association, stored major version 2 does not match received 1 ( ns : feature.association, received : 1|84||000000000000000000000000, wanted : 2|0||000000000000000000000000, send ) ( ns : feature.association, received : 1|84||000000000000000000000000, wanted : 2|0||000000000000000000000000, send ) ns:feature.$cmd query:{ findandmodify: "association", query: { index: "14be3fa342209dacb92cfa69073829e0", communityId: ObjectId('4c927585d591d31d7b37097a')}, fields: { doccount: 1, entity1: 1, entity2: 1, verb: 1, db_sync_time: 1, db_sync_doccount: 1 },

Alex-Ikanow

{
        "_id" : ObjectId("4f996119e4b053cd3218c2fd"),
        "associations" : [
                {
                        "entity1" : "clay wanta",
                        "entity1_index" : "clay wanta/person",
                        "verb" : "current",
                        "verb_category" : "career",
                        "entity2" : "police detective",
                        "entity2_index" : "police detective/position",

Alex-Ikanow

{
        "sharded" : true,
        "ns" : "doc_metadata.metadata",
        "count" : 3003302,
        "numExtents" : 42,
        "size" : 27260679308,
        "storageSize" : 28334976896,
        "totalIndexSize" : 9223868864,
        "indexSizes" : {
                "_id_" : 92773072,

Alex-Ikanow

QUERY:

{ 
"$or" : [ { "searchCycle_secs" : { "$exists" : false}} , { "$where" : "(null == this.harvest) || (null == this.harvest.harvested) || (null == this.searchCycle_secs) || ((this.harvest.harvested.getTime() + 1000*this.searchCycle_secs) <= 1352397384946)"}] , 

"isApproved" : true , "harvestBadSource" : { "$ne" : true} , "extractType" : { "$in" : [ "Feed" , "feed"]}
}

FIELDS:

Alex-Ikanow

{
    "description": "test",
    "extractType": "File",
    "file": {
        "XmlIgnoreValues": [
            "DefiningCharacteristicList",
            "TargetedCharacteristicList",
            "WeaponTypeList",
            "PerpetratorList",
            "VictimList",

Alex-Ikanow

{
    "authentication": {
        "password": "",
        "username": ""
    },
    "database": {
        "databaseName": "DB_NAME",
        "databaseType": "relique:csv",
        "deleteQuery": "",
        "deltaQuery": "SELECT * FROM TABLE_NAME",

Alex-Ikanow

umount /mnt
mdadm --create /dev/md0 --run --level=0 --chunk=256 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde
pvcreate /dev/md0
vgcreate data_vg /dev/md0
lvcreate -l 100%vg -n data_vol data_vg
mkfs.ext3 -m 0 /dev/data_vg/data_vol
mount -t ext3 /dev/data_vg/data_vol /mnt/

touch /etc/mdadm.conf
grep -F "/dev/md0" /etc/mdadm.conf || mdadm -Es | grep md0 >> /etc/mdadm.conf

Alex-Ikanow

2012-02-08 17:50:41.252 [DEBUG] action.search.type:89 - [Screaming Mimi] [doc_dummy][0], node[x1GaMz1BQBSIzbcmc5GoTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5fdb7adc]
org.elasticsearch.search.SearchParseException: [doc_dummy][0]: from[-1],size[1000]: Parse Failure [Failed to parse source [:)
?ze$??query?ol?st?ustom_score??ould?sted?term?dexXtest systems/industryterm?athGentities?score_modeBmax??scriptJdecayscript?langEnative?params?ram?x)?x)?x)=} c=YIh;%Weu?"??nstant_score?lter??rms?mmunityId?927585d591d31d7b37097aW4c927585d591d31d7c37097bW4e53e755ebcd573d6af5f091W4e6e1deaacd7852244b92fbfW4da6080df891e8e224809c41W4e08f20f0e9a573dd8f74b93??ost(??ript_fields?cay?ecayscriptLEnativeM??x)?x)?x)=} c=YIh;%Weu?#??facets?o?fieldClocs@$??ime?te_histogram?ublishedDate?intervalA1w?vents?Jassoc_index@$??regexI^Event\|.*?ssociations?cts?Jassoc_index@$?]H^Fact\|.*?ssociations?urceTags?Ctags@$??ourceTypes?HmediaType@$??ourceKeys?HsourceKey@$??]]
        at org.elasticsearch.search.

Alex-Ikanow

// First document

{
  "_id": "4f2723ae4f3625d892f5faf4",
  "title": "Profile 1",
  // Lots of other completely mapped fields, including arrays of objects, geo points, strings etc
  "metadata": { // metadata does not have a mapping
    "socialnetworkobject": [
      {
        "field1": {}, // <-- THIS CAUSES THE PROBLEM WHEN A DOC WITH A STRING VALUE HERE IS INSERTED

Alex-Ikanow

// Unfortunately this attempt at a simple recreation doesn't work.
// This differences vs the real world case (that occurs 100% consistently are):
//  - the "obj1" equivalent is not a completely dynamic object, all fields are declared via mapping apart from obj2
//  - the "obj1" mapping has a long and complex set of dynamic templates declared (none of which are used)
//  - many other documents not containing any obj2 fields have been indexed before the first value with a "bad" obj2 is indexed
// (- several GBs of other shards/indexes etc etc are present)

// 1] Create an index, eg something like