karussell

# TO_FOLDER=/something
# FROM=/your-es-installation

DATE=`date +%Y-%m-%d_%H-%M`
TO=$TO_FOLDER/$DATE/
echo "rsync from $FROM to $TO"
# the first times rsync can take a bit long - do not disable flusing
rsync -a $FROM $TO

# now disable flushing and do one manual flushing

karussell

[2011-09-26 16:49:58,710][TRACE][discovery                ] [Razor Fist] waiting for 30s for the initial state to be set by the discovery
[2011-09-26 16:49:59,800][TRACE][discovery.ec2            ] [Razor Fist] building dynamic unicast discovery nodes...
[2011-09-26 16:49:59,821][TRACE][discovery.ec2            ] [Razor Fist] adding i-5a06613a, address PRIVATE_IP, transport_address inet[/PRIVATE_IP:9300]
[2011-09-26 16:49:59,821][DEBUG][discovery.ec2            ] [Razor Fist] using dynamic discovery nodes [[#cloud-i-5a06613a-0][inet[/PRIVATE_IP:9300]]]
[2011-09-26 16:49:59,872][TRACE][discovery.zen.ping.unicast] [Razor Fist] [1] connecting to [#cloud-i-5a06613a-0][inet[/PRIVATE_IP:9300]]
[2011-09-26 16:49:59,966][TRACE][discovery.zen.ping.unicast] [Razor Fist] [1] received response from [#cloud-i-5a06613a-0][inet[/PRIVATE_IP:9300]]: [ping_response{target [[Razor Fist][Otq02_OVRamD0VxK6f71cg][inet[/PUBLIC_IP:9300]]], master [null], cluster_name[elasticsearch]}, ping_response{target [[Razor Fist][Otq02_OVRamD0

karussell

Several times in a month there pop up questions regarding query structure on the ElasticSearch user group.

Although there are good docs explaining this in depth probably the bird view of the Query DSL is necessary to
understand what is written there. There is even already some good external documentation available:
http://www.elasticsearch.org/tutorials/2011/08/28/query-dsl-explained.html

And there were attempts to define a schema:

http://groups.google.com/group/json-schema/browse_thread/thread/ae498ee818155d50
https://gist.github.com/8887766ca0e7052814b0

karussell

Why is there no such DataImportHandler thing in ElasticSearch? Uhm, well ... but because:

1. You should really consider your own scripts 
(be it jvm based, perl, ruby, php, nodejs/javascript) 
to feed ElasticSearch via bulk indexing: 
http://www.elasticsearch.org/guide/reference/java-api/bulk.html

2. There are two projects doing it already:
 * http://code.google.com/p/sql-to-nosql-importer/
 * https://github.com/Aconex/scrutineer (keeps DB in synch with ES or solr!)

karussell

sudo apt-get update
sudo apt-get install openjdk-6-jdk

wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.8.deb
sudo dpkg -i elasticsearch-0.19.8.deb

# be sure you add "action.disable_delete_all_indices" : true to the config!!

# if you want to remove it:
sudo dpkg -r elasticsearch

karussell

/** Just a quick snippet to show how to use the REST ElasticSearch in Java - grab the parts you like ;)
 *  Instead of using the official Java API, where adding the complete ~20MB ElasticSearch jar is necessary. 
 */
public class ElasticInterface {

    public static void main(String[] args) throws Exception {
        Module module = new DefaultModule();
        Injector injector = Guice.createInjector(module);

        ElasticInterface tws = injector.getInstance(ElasticInterface.class);

karussell

HOST="http://localhost:9200"

# rebuild index
curl -XDELETE "$HOST/osm"
curl -XPUT "$HOST/osm"

curl -XPUT "$HOST/osm/way/_mapping" -d '
{
  "way" : {
    "properties" : {

karussell

# install the roll plugin for every server
# https://github.com/karussell/elasticsearch-rollindex/
# call this script ala roll.sh <indexPrefix> <type> <numberOfRemainingIndices>

indexprefix=$1
type=$2
roll=$3
SRC=
host=localhost:9200
# grab the mapping for 'type' from somewhere

karussell

This is a lot easier: https://gist.github.com/piscisaureus/3342247

Step 1: Check out a new branch to test the changes — run this from your project directory

   git checkout -b dajac-jsoup-1.7.1 master

Step 2: Bring in dajac's changes and test ('jsoup-1.7.1' acts as commit message)

    git pull https://github.com/dajac/snacktory.git jsoup-1.7.1

karussell

You'll need /data and /install folders, if this is not desired you'll need to change the script variables for the paths

1. install osmosis 0.43.1 http://bretth.dev.openstreetmap.org/osmosis-build/ to /install/osmosis
2. do git clone https://github.com/graphhopper/graphhopper /install/graphhopper_mapscreation then do git checkout 0.3
3. try cd /install/graphhopper_mapscreation; ./graphhopper.sh web europe_germany_berlin.pbf, this should not fail
4. create a new directory which should be accessible from outside /data/public/maps/ and contains subdirectory '0.4'
5. create /data/osm and /data/mapsforge where pbf and map files will be stored
6. install java to /install/java so that one can execute /install/java/bin/java
7. copy mapsforge writer 0.4.3 to /home//.openstreetmap/osmosis/plugins/mapsforge-map-writer-0.4.3-jar-with-dependencies.jar from http://ci.mapsforge.org/job/release-0.4.3/ or from maven central
8. include as many map names for de_maps, us_maps, eu_maps as possible. The naming