GDAL and Elasticsearch examples

README.md

GDAL and Elasticsearch examples

Shapefile

GDAL Shapefile docs

Import shapefile into Elasticsearch

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
http://elastic:changeme@localhost:9200 \
./my-shapefile.shp

Import zipped shapefile into Elasticsearch

Natural Earth Populated Places

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
http://elastic:changeme@localhost:9200 \
/vsizip/./ne_10m_populated_places_simple.zip/ne_10m_populated_places_simple.shp

CSV

GDAL CSV docs

Import CSV with latitude and longitude into Elasticsearch

NYC Emergency Response Incidents Open Data

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
-lco INDEX_NAME=oem-nyc \
-lco OVERWRITE_INDEX=YES \
-oo X_POSSIBLE_NAMES=Longitude \
-oo Y_POSSIBLE_NAMES=Latitude \
-oo KEEP_GEOM_COLUMNS=NO \
-oo EMPTY_STRING_AS_NULL=YES \
http://elastic:changeme@localhost:9200 \
Emergency_Response_Incidents.csv

Generate and use a custom mapping for CSV file

NYC Emergency Response Indicents Open Data

1. Generate a mapping JSON file from the CSV file using GDAL

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
-lco INDEX_NAME=oem-nyc \
-lco OVERWRITE_INDEX=YES \
-lco WRITE_MAPPING='./oem-nyc.json' \
-oo X_POSSIBLE_NAMES=Longitude \
-oo Y_POSSIBLE_NAMES=Latitude \
-oo KEEP_GEOM_COLUMNS=NO \
-oo EMPTY_STRING_AS_NULL=YES \
http://elastic:changeme@localhost:9200 \
Emergency_Response_Incidents.csv

2. Change the Creation Date and Closed Date mappings to type: date in the oem-nyc.json file.

{
  "properties": {
    "Incident Type": {
      "type": "keyword"
    },
    "Location": {
      "type": "keyword"
    },
    "Borough": {
      "type": "keyword"
    },
    "Creation Date": {
-     "type": "keyword"
+     "type": "date",
+     "format": "MM/dd/yyyy HH:mm:ss a"
    },
    "Closed Date": {
-     "type": "keyword"
+     "type": "date",
+     "format": "MM/dd/yyyy HH:mm:ss a"
    },
    "geometry": {
      "properties": {
        "type": {
          "type": "text"
        },
        "coordinates": {
          "type": "geo_point"
        }
      }
    }
  },
  "_meta": {
    "fid": "ogc_fid"
  }
}

3. Use the custom mapping when ingesting the data into Elasticsearch.

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
-lco INDEX_NAME=oem-nyc \
-lco OVERWRITE_INDEX=YES \
-lco MAPPING='./oem-nyc.json' \
-oo X_POSSIBLE_NAMES=Longitude \
-oo Y_POSSIBLE_NAMES=Latitude \
-oo KEEP_GEOM_COLUMNS=NO \
-oo EMPTY_STRING_AS_NULL=YES \
http://elastic:changeme@localhost:9200 \
Emergency_Response_Incidents.csv

PostgreSQL/ PostGIS

GDAL PostgreSQL Docs

Import data from a table

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
-lco INDEX_NAME=my-index \
http://elastic:changeme@localhost:9200 \
PG:"host='localhost' port=5432 user='postgres' dbname='postgres' password='mysecretpassword'" my-pg-table

Import data from a SQL query

ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
-lco INDEX_NAME=my-index \
-lco OVERWRITE_INDEX=YES \
http://elastic:changeme@localhost:9200 \
PG:"host='localhost' port=5432 user='postgres' dbname='postgres' password='mysecretpassword'" \
-sql "select * from my-pg-table where my-field='A'"

Examples using nightly Docker images

These are useful when you want to test the latest GDAL features such as compatibility with Elasticsearch 7.x. Requires Docker. These examples assume Elasticsearch is running on the host container.

Docker Engine (Linux)

docker run --rm --network=host -u $(id -u ${USER}):$(id -g ${USER}) \
-v $(pwd):/data \
osgeo/gdal:alpine-small-latest \
ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
http://elastic:changeme@localhost:9200 \
/data/my-shapefile.shp

Docker for Mac or Windows

why?

docker run --rm --network=host -u $(id -u ${USER}):$(id -g ${USER}) \
-v $(pwd):/data \
osgeo/gdal:alpine-small-latest \
ogr2ogr -f ElasticSearch \
-lco NOT_ANALYZED_FIELDS={ALL} \
http://elastic:changeme@host.docker.internal:9200 \
/data/my-shapefile.shp