Skip to content

Instantly share code, notes, and snippets.

@gokhangirgin

gokhangirgin/definition.md

Last active July 11, 2016 21:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gokhangirgin/5e5dabb0c921696274feac2a2c729747 to your computer and use it in GitHub Desktop.
Save gokhangirgin/5e5dabb0c921696274feac2a2c729747 to your computer and use it in GitHub Desktop.
Download ZIP
Creates subset of geo points (near or far) from index (Elasticsearch)
Raw
definition.md

#Use Case When we need to get only near geo points, no way to filter or get some small amount of geo points in query. Imagine when you have 500 geo points in single document, how would you identify or limit near or far locations?

See https://discuss.elastic.co/t/retrieving-only-matching-locations/54590 and http://stackoverflow.com/questions/37919891/how-to-limit-returning-nested-objects-in-elasticsearch ##Create Index

curl -XPUT localhost:9200/geo

Sample mapping

curl -XPUT /geo/test/_mapping -d '{
   "properties": {
      "locations": {
         "type": "geo_point"
      }
   }
}'

First index

curl -XPOST "localhost:9200/geo/test/1" -d '{
    "locations": [
      {
        "lon": 27.09519515440661,
        "lat": 38.478413567935185
      },
      {
        "lon": 34.80236053466797,
        "lat": 39.81952667236328
      },
      {
        "lon": 39.83582532931803,
        "lat": 40.96827796407719
      },
      {
        "lon": 36.333538930028226,
        "lat": 41.28705790254833
      },
      {
        "lon": 30.37797194349696,
        "lat": 40.77331286046352
      },
      {
        "lon": 28.254863739013672,
        "lat": 37.468238830566406
      },
      {
        "lon": 34.62808663310784,
        "lat": 36.80608547444202
      },
      {
        "lon": 27.417057,
        "lat": 38.61295
      },
      {
        "lon": 30.023533189852515,
        "lat": 39.39730937110715
      },
      {
        "lon": 32.5051864924701,
        "lat": 37.89531395220301
      },
      {
        "lon": 32.508596011506825,
        "lat": 37.8492941159215
      },
      {
        "lon": 32.488811510405185,
        "lat": 37.87698760572874
      }
    ]
}'

Second index

curl -XPOST  "localhost:9200/geo/test/2" -d '{
    "locations": [
      {
        "lon": 32.51198884183816,
        "lat": 37.90110180235243
      },
      {
        "lon": 32.03915426223398,
        "lat": 36.540804011837125
      },
      {
        "lon": 30.750701764736277,
        "lat": 36.890245468561396
      },
      {
        "lon": 30.595000683221542,
        "lat": 36.84250172907872
      },
      {
        "lon": 29.130974405626603,
        "lat": 36.67013612740797
      },
      {
        "lon": 28.442826999999966,
        "lat": 37.021167
      },
      {
        "lon": 27.357373237609863,
        "lat": 37.056478551035404
      },
      {
        "lon": 29.059229208694887,
        "lat": 37.81761128155548
      },
      {
        "lon": 27.260725335147754,
        "lat": 37.843887300205395
      },
      {
        "lon": 27.138333320617676,
        "lat": 38.3168448192717
      }
    ]
}'

Third index

curl -XPOST "localhost:9200/geo/test/3" -d '{
    "locations": [
      {
        "lon": 27.29519515440661,
        "lat": 38.378413567935185
      }
}'

Sample Query which returns all of the documents as expected

curl -XPOST localhost:9200/geo/test/_search -d '{
   "query": {
      "bool": {
         "must": [
            {
               "geo_distance": {
                  "distance": "100km",
                  "locations": {
                     "lon": 28.442826999999966,
                     "lat": 37.121167
                  }
               }
            }
         ]
      }
   },
   "sort": {
      "_geo_distance": {
         "locations": {
            "lon": 28.442826999999966,
            "lat": 37.121167
         },
         "order": "asc",
         "unit": "km"
      }
   }
}'

Move custom script to es scripts dir /etc/elasticsearch/scripts/sortedGeoPoints.groovy

Raw
sample_usage.md

##Sample usage with script_fields

"script_fields": {
  "closest_points": {
    "script": {
        "lang": "groovy", //language
        "file": "sortedGeoPoints", //script name
        "params": {
           "field" : "locations", //attribute
           "lon": 28.442826999999966, //longitude
           "lat": 37.101167, //latitude
           "method" : "PLANE", //Calculation (PLANE, FACTOR, ARC)
           "unit" : "km", //unit km,mi,cm....
           "order" : "asc",
           "limit" : 3
        }
    }
  }
}

Sample Query on top of Geo sample

curl -XPOST /geo/test/_search -d '{
    "fields": [
       "_source",
       "closest_points"
    ], 
   "query": {
      "bool": {
         "must": [
            {
               "geo_distance": {
                  "distance": "100km",
                  "locations": {
                     "lon": 28.442826999999966,
                     "lat": 37.101167
                  }
               }
            }
         ]
      }
   },
   "script_fields": {
         "closest_points": {
            "script": {
                "lang": "groovy",
                "file": "sortedGeoPoints",
                "params": {
                   "field" : "locations",
                   "lon": 28.442826999999966,
                   "lat": 37.101167,
                   "method" : "FACTOR",
                   "unit" : "mi",
                   "order" : "asc",
                   "limit" : 3
                }
            }
         }
      },
   "sort": {
      "_geo_distance": {
         "locations": {
            "lon": 28.442826999999966,
            "lat": 37.101167
         },
         "order": "asc",
         "unit": "km"
      }
   }
}'

###Response

{
   "took": 3,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 3,
      "max_score": null,
      "hits": [
         {
            "_index": "geo",
            "_type": "test",
            "_id": "3",
            "_score": null,
            "_source": {
               "locations": {
                  "lon": 28.442826999999966,
                  "lat": 37.101167
               }
            },
            "fields": {
               "closest_points": [
                  {
                     "lat": 37.101166946813464,
                     "lon": 28.442826848477125
                  }
               ]
            },
            "sort": [
               0
            ]
         },
         {
            "_index": "geo",
            "_type": "test",
            "_id": "2",
            "_score": null,
            "_source": {
               "locations": [
                  {
                     "lon": 32.51198884183816,
                     "lat": 37.90110180235243
                  },
                  {
                     "lon": 32.03915426223398,
                     "lat": 36.540804011837125
                  },
                  {
                     "lon": 30.750701764736277,
                     "lat": 36.890245468561396
                  },
                  {
                     "lon": 30.595000683221542,
                     "lat": 36.84250172907872
                  },
                  {
                     "lon": 29.130974405626603,
                     "lat": 36.67013612740797
                  },
                  {
                     "lon": 28.442826999999966,
                     "lat": 37.021167
                  },
                  {
                     "lon": 27.357373237609863,
                     "lat": 37.056478551035404
                  },
                  {
                     "lon": 29.059229208694887,
                     "lat": 37.81761128155548
                  },
                  {
                     "lon": 27.260725335147754,
                     "lat": 37.843887300205395
                  },
                  {
                     "lon": 27.138333320617676,
                     "lat": 38.3168448192717
                  },
                  {
                     "lon": 27.071256637573242,
                     "lat": 38.48933825650774
                  },
                  {
                     "lon": 26.714625953089467,
                     "lat": 39.32858740058092
                  },
                  {
                     "lon": 27.01315328845908,
                     "lat": 39.58359828963627
                  },
                  {
                     "lon": 30.480773448944092,
                     "lat": 39.7880274728305
                  },
                  {
                     "lon": 28.94373893737793,
                     "lat": 40.21308797314051
                  },
                  {
                     "lon": 29.060039520263672,
                     "lat": 40.21855252401866
                  },
                  {
                     "lon": 29.994049072265625,
                     "lat": 40.75616479199092
                  },
                  {
                     "lon": 29.277684688568115,
                     "lat": 40.86729024334164
                  },
                  {
                     "lon": 29.09432578850101,
                     "lat": 40.984569347469616
                  },
                  {
                     "lon": 29.160798594461085,
                     "lat": 41.00309736951742
                  },
                  {
                     "lon": 28.980817794799805,
                     "lat": 41.08468861630809
                  },
                  {
                     "lon": 28.887250672831783,
                     "lat": 41.04278663253767
                  },
                  {
                     "lon": 28.81338898665183,
                     "lat": 41.03003964605299
                  },
                  {
                     "lon": 28.877512160231504,
                     "lat": 41.01217464724523
                  },
                  {
                     "lon": 28.813502804240443,
                     "lat": 41.02979628210313
                  }
               ]
            },
            "fields": {
               "closest_points": [
                  {
                     "lat": 37.9011017549783,
                     "lon": 32.51198871061206
                  },
                  {
                     "lat": 36.54080394655466,
                     "lon": 32.03915415331721
                  },
                  {
                     "lat": 40.75616472400725,
                     "lon": 29.994049072265625
                  }
               ]
            },
            "sort": [
               8.894771255505198
            ]
         },
         {
            "_index": "geo",
            "_type": "test",
            "_id": "1",
            "_score": null,
            "_source": {
               "locations": [
                  {
                     "lon": 27.09519515440661,
                     "lat": 38.478413567935185
                  },
                  {
                     "lon": 34.80236053466797,
                     "lat": 39.81952667236328
                  },
                  {
                     "lon": 39.83582532931803,
                     "lat": 40.96827796407719
                  },
                  {
                     "lon": 36.333538930028226,
                     "lat": 41.28705790254833
                  },
                  {
                     "lon": 30.37797194349696,
                     "lat": 40.77331286046352
                  },
                  {
                     "lon": 28.254863739013672,
                     "lat": 37.468238830566406
                  },
                  {
                     "lon": 34.62808663310784,
                     "lat": 36.80608547444202
                  },
                  {
                     "lon": 27.417057,
                     "lat": 38.61295
                  },
                  {
                     "lon": 30.023533189852515,
                     "lat": 39.39730937110715
                  },
                  {
                     "lon": 32.5051864924701,
                     "lat": 37.89531395220301
                  },
                  {
                     "lon": 32.508596011506825,
                     "lat": 37.8492941159215
                  },
                  {
                     "lon": 32.488811510405185,
                     "lat": 37.87698760572874
                  },
                  {
                     "lon": 31.165362977346263,
                     "lat": 40.84839903519435
                  },
                  {
                     "lon": 33.502927020483526,
                     "lat": 39.849883588782404
                  },
                  {
                     "lon": 27.11124803491998,
                     "lat": 38.4019190962971
                  },
                  {
                     "lon": 27.113189923774826,
                     "lat": 38.381669540934766
                  },
                  {
                     "lon": 27.056431001957094,
                     "lat": 38.4986724763324
                  },
                  {
                     "lon": 27.05082893371582,
                     "lat": 38.557979583740234
                  },
                  {
                     "lon": 27.38449839752684,
                     "lat": 37.70892968802906
                  },
                  {
                     "lon": 30.715407166381965,
                     "lat": 36.9195390918838
                  },
                  {
                     "lon": 30.715800131973424,
                     "lat": 36.91938670968827
                  },
                  {
                     "lon": 35.322755584584,
                     "lat": 36.998830301305745
                  },
                  {
                     "lon": 35.35866608901438,
                     "lat": 36.997541010459656
                  },
                  {
                     "lon": 32.82055654306532,
                     "lat": 39.866401396487404
                  },
                  {
                     "lon": 32.777273297441525,
                     "lat": 39.97936453719419
                  }
               ]
            },
            "fields": {
               "closest_points": [
                  {
                     "lat": 40.96827792003751,
                     "lon": 39.83582530170679
                  },
                  {
                     "lat": 41.28705787472427,
                     "lon": 36.33353892713785
                  },
                  {
                     "lat": 36.99754097498953,
                     "lon": 35.358666088432074
                  }
               ]
            },
            "sort": [
               44.06946269976184
            ]
         }
      ]
   }
}
Raw
sortedGeoPoints.groovy
import org.elasticsearch.common.geo.GeoDistance;
import org.elasticsearch.common.unit.DistanceUnit;
import org.elasticsearch.common.geo.GeoPoint;
/***
*
* usage
* "script_fields": {
* "closest_points": {
* "script": {
* "lang": "groovy",
* "file": "sortedGeoPoints",
* "params": {
* "field": "lokasyonlar",
* "lon": 26.954897,
* "lat": 38.7762021,
* "method": "PLANE",
* "unit": "km",
* "order": "asc",
* "limit": 5
* }
* }
* }
* }
*
*/
if (doc[field].values.size() < limit){
return doc[field]
}
else{
def distanceUnit
def geoCalculator
switch (method){
case "ARC" :
geoCalculator = GeoDistance.ARC
break
case "FACTOR" :
geoCalculator = GeoDistance.FACTOR
break
default:
geoCalculator = GeoDistance.PLANE
break
}
switch (unit) {
case "in" : //inch
distanceUnit = DistanceUnit.INCH
break
case "yd": //yards
distanceUnit = DistanceUnit.YARD
break
case "ft": //feet
distanceUnit = DistanceUnit.FEET
break
case "nmi": //NAUTICALMILES
distanceUnit = DistanceUnit.NAUTICALMILES
break
case "mm": // MILLIMETERS
distanceUnit = DistanceUnit.MILLIMETERS
break
case "cm": // CENTIMETERS
distanceUnit = DistanceUnit.CENTIMETERS
break
case "mi": // MILES
distanceUnit = DistanceUnit.MILES
break
case "m": // MILES
distanceUnit = DistanceUnit.METERS
break
default:
distanceUnit = DistanceUnit.KILOMETERS
break
}
def sortedGeoPoints = new TreeMap<Double, GeoPoint>()
for(i = 0; i < doc[field].values.size(); i++){
def loc = doc[field].values[i]
sortedGeoPoints.put(geoCalculator.calculate(loc.lon, loc.lat, lon, lat, distanceUnit), loc)
}
def list
if(order == "desc"){
list = new ArrayList<GeoPoint>(sortedGeoPoints.descendingMap().values()) //reversed
}
else{
list = new ArrayList<GeoPoint>(sortedGeoPoints.values())
}
return list.subList(0, limit)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment