karussell

You'll need /data and /install folders, if this is not desired you'll need to change the script variables for the paths

1. install osmosis 0.43.1 http://bretth.dev.openstreetmap.org/osmosis-build/ to /install/osmosis
2. do git clone https://github.com/graphhopper/graphhopper /install/graphhopper_mapscreation then do git checkout 0.3
3. try cd /install/graphhopper_mapscreation; ./graphhopper.sh web europe_germany_berlin.pbf, this should not fail
4. create a new directory which should be accessible from outside /data/public/maps/ and contains subdirectory '0.4'
5. create /data/osm and /data/mapsforge where pbf and map files will be stored
6. install java to /install/java so that one can execute /install/java/bin/java
7. copy mapsforge writer 0.4.3 to /home//.openstreetmap/osmosis/plugins/mapsforge-map-writer-0.4.3-jar-with-dependencies.jar from http://ci.mapsforge.org/job/release-0.4.3/ or from maven central
8. include as many map names for de_maps, us_maps, eu_maps as possible. The naming

karussell

// dependency: graphhopper 0.3
// JVM_OPTS="-Xms2400m -Xmx2400m"
// 
// result on my oldish machine:
// separate arrays: GET=34.10 and SET=36.34
// single    array: GET=36.34 and SET=34.49
//
// => GET is ~7% faster for single array
// => SET is ~5% faster for single array
//

karussell

package de.jetwick.es;

import org.apache.lucene.analysis.CharArraySet;
import org.apache.lucene.analysis.TokenStream;
import de.jetwick.org.apache.solr.analysis.WordDelimiterFilter;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.Index;
import org.elasticsearch.index.analysis.AbstractTokenFilterFactory;

public class JetwickFilterFactory extends AbstractTokenFilterFactory {

karussell

2011-01-21 00:20:48,468 [elasticsearch[Smuggler II]clusterService#updateTask-pool-6-thread-1] ERROR org.elasticsearch.gateway - [Smuggler II] failed to create index [twindexreal]
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) No implementation for java.util.Map<java.lang.String, org.elasticsearch.index.analysis.AnalyzerProviderFactory> was bound.
  while locating java.util.Map<java.lang.String, org.elasticsearch.index.analysis.AnalyzerProviderFactory>
    for parameter 2 at org.elasticsearch.index.analysis.AnalysisService.<init>(AnalysisService.java:59)
  while locating org.elasticsearch.index.analysis.AnalysisService
    for parameter 3 at org.elasticsearch.index.mapper.MapperService.<init>(MapperService.java:89)
  at org.elasticsearch.index.mapper.MapperServiceModule.configure(MapperServiceModule.java:30)

karussell

for (String fromIndex : indexList) {
    SearchResponse rsp = client.prepareSearch(fromIndex).
            setQuery(QueryBuilders.matchAllQuery()).setSize(hitsPerPage).
            // This is important:
            setSearchType(SearchType.QUERY_AND_FETCH).
            setScroll(TimeValue.timeValueMinutes(30)).execute().actionGet();
    try {
        long total = rsp.hits().totalHits();
        Collection<MyTweet> tweets = collectTweets(rsp);
        bulkUpdate(tweets, intoIndex);

karussell

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import java.util.Collection;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchType;
import org.elasticsearch.common.unit.TimeValue;
import java.util.LinkedHashMap;
import java.util.Map.Entry;

karussell

org.elasticsearch.transport.RemoteTransportException: [Fer-de-Lance][inet[/127.0.0.1:9300]][indices/search]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[VHXZa25OQGCtSDa9zdUyFQ][twindex][0]: QueryPhaseExecutionException[[twindex][0]: query[ConstantScore(org.elasticsearch.common.lucene.search.AndFilter@7ed467ba)],from[0],size[15],sort[<custom:"relevance": org.elasticsearch.index.field.data.doubles.DoubleFieldDataType$1@1afc83a>!]: Query Failed [Failed to execute main query]]; nested: }{[VHXZa25OQGCtSDa9zdUyFQ][twindex][3]: QueryPhaseExecutionException[[twindex][3]: query[ConstantScore(org.elasticsearch.common.lucene.search.AndFilter@7ed467ba)],from[0],size[15],sort[<custom:"relevance": org.elasticsearch.index.field.data.doubles.DoubleFieldDataType$1@1afc83a>!]: Query Failed [Failed to execute main query]]; nested: }
...
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhas

karussell

// we don't want results with an id larger  than maxId
long maxId = lastId;
// we don't want results with an id smaller than sinceId
long sinceId = lastId;

int hitsPerPage = 100;
int maxPages = 1;
boolean breakPaging = false;
for (int page = 0; page < maxPages; page++) {
    Query query = new Query(term);

karussell

a document with a number -> faceting over that

ComparatorType.TOTAL
====================

term:9	 count:2	 total:2.0
there should be 10 terms. there were:1

ComparatorType.COUNT
====================

karussell

    @Test public void testTermsStatsFacetsShouldAllowFilter() throws Exception {
        try {
            client.admin().indices().prepareDelete("test").execute().actionGet();
        } catch (Exception e) {
            // ignore
        }
        client.admin().indices().prepareCreate("test").execute().actionGet();
        client.admin().cluster().prepareHealth().setWaitForGreenStatus().execute().actionGet();

        for (int i = 0; i < 20; i++) {