Skip to content

Instantly share code, notes, and snippets.

@karussell
karussell / doc.md
Last active Aug 29, 2015
mapsforge map creation
View doc.md

You'll need /data and /install folders, if this is not desired you'll need to change the script variables for the paths

  1. install osmosis 0.43.1 http://bretth.dev.openstreetmap.org/osmosis-build/ to /install/osmosis
  2. do git clone https://github.com/graphhopper/graphhopper /install/graphhopper_mapscreation then do git checkout 0.3
  3. try cd /install/graphhopper_mapscreation; ./graphhopper.sh web europe_germany_berlin.pbf, this should not fail
  4. create a new directory which should be accessible from outside /data/public/maps/ and contains subdirectory '0.4'
  5. create /data/osm and /data/mapsforge where pbf and map files will be stored
  6. install java to /install/java so that one can execute /install/java/bin/java
  7. copy mapsforge writer 0.4.3 to /home//.openstreetmap/osmosis/plugins/mapsforge-map-writer-0.4.3-jar-with-dependencies.jar from http://ci.mapsforge.org/job/release-0.4.3/ or from maven central
  8. include as many map names for de_maps, us_maps, eu_maps as possible. The naming
View RAMCacheTest.java
// dependency: graphhopper 0.3
// JVM_OPTS="-Xms2400m -Xmx2400m"
//
// result on my oldish machine:
// separate arrays: GET=34.10 and SET=36.34
// single array: GET=36.34 and SET=34.49
//
// => GET is ~7% faster for single array
// => SET is ~5% faster for single array
//
View JetwickFilterFactory.java
package de.jetwick.es;
import org.apache.lucene.analysis.CharArraySet;
import org.apache.lucene.analysis.TokenStream;
import de.jetwick.org.apache.solr.analysis.WordDelimiterFilter;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.Index;
import org.elasticsearch.index.analysis.AbstractTokenFilterFactory;
public class JetwickFilterFactory extends AbstractTokenFilterFactory {
@karussell
karussell / es-exception-for-custom-filter
Created Jan 20, 2011
elastic search exception for custom filter
View es-exception-for-custom-filter
2011-01-21 00:20:48,468 [elasticsearch[Smuggler II]clusterService#updateTask-pool-6-thread-1] ERROR org.elasticsearch.gateway - [Smuggler II] failed to create index [twindexreal]
org.elasticsearch.common.inject.CreationException: Guice creation errors:
1) No implementation for java.util.Map<java.lang.String, org.elasticsearch.index.analysis.AnalyzerProviderFactory> was bound.
while locating java.util.Map<java.lang.String, org.elasticsearch.index.analysis.AnalyzerProviderFactory>
for parameter 2 at org.elasticsearch.index.analysis.AnalysisService.<init>(AnalysisService.java:59)
while locating org.elasticsearch.index.analysis.AnalysisService
for parameter 3 at org.elasticsearch.index.mapper.MapperService.<init>(MapperService.java:89)
at org.elasticsearch.index.mapper.MapperServiceModule.configure(MapperServiceModule.java:30)
View gist:816297
for (String fromIndex : indexList) {
SearchResponse rsp = client.prepareSearch(fromIndex).
setQuery(QueryBuilders.matchAllQuery()).setSize(hitsPerPage).
// This is important:
setSearchType(SearchType.QUERY_AND_FETCH).
setScroll(TimeValue.timeValueMinutes(30)).execute().actionGet();
try {
long total = rsp.hits().totalHits();
Collection<MyTweet> tweets = collectTweets(rsp);
bulkUpdate(tweets, intoIndex);
@karussell
karussell / TestES.java
Created Feb 8, 2011
Something goes wrong when doing bulkUpdate and retrieving
View TestES.java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import java.util.Collection;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchType;
import org.elasticsearch.common.unit.TimeValue;
import java.util.LinkedHashMap;
import java.util.Map.Entry;
View exception on client
org.elasticsearch.transport.RemoteTransportException: [Fer-de-Lance][inet[/127.0.0.1:9300]][indices/search]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[VHXZa25OQGCtSDa9zdUyFQ][twindex][0]: QueryPhaseExecutionException[[twindex][0]: query[ConstantScore(org.elasticsearch.common.lucene.search.AndFilter@7ed467ba)],from[0],size[15],sort[<custom:"relevance": org.elasticsearch.index.field.data.doubles.DoubleFieldDataType$1@1afc83a>!]: Query Failed [Failed to execute main query]]; nested: }{[VHXZa25OQGCtSDa9zdUyFQ][twindex][3]: QueryPhaseExecutionException[[twindex][3]: query[ConstantScore(org.elasticsearch.common.lucene.search.AndFilter@7ed467ba)],from[0],size[15],sort[<custom:"relevance": org.elasticsearch.index.field.data.doubles.DoubleFieldDataType$1@1afc83a>!]: Query Failed [Failed to execute main query]]; nested: }
...
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhas
@karussell
karussell / search.java
Created Feb 12, 2011
search twitter via maxId
View search.java
// we don't want results with an id larger than maxId
long maxId = lastId;
// we don't want results with an id smaller than sinceId
long sinceId = lastId;
int hitsPerPage = 100;
int maxPages = 1;
boolean breakPaging = false;
for (int page = 0; page < maxPages; page++) {
Query query = new Query(term);
@karussell
karussell / result.txt
Created Feb 22, 2011
different facet entries but only sort changed
View result.txt
a document with a number -> faceting over that
ComparatorType.TOTAL
====================
term:9 count:2 total:2.0
there should be 10 terms. there were:1
ComparatorType.COUNT
====================
View TermsStatsFacetsShouldAllowFilter.java
@Test public void testTermsStatsFacetsShouldAllowFilter() throws Exception {
try {
client.admin().indices().prepareDelete("test").execute().actionGet();
} catch (Exception e) {
// ignore
}
client.admin().indices().prepareCreate("test").execute().actionGet();
client.admin().cluster().prepareHealth().setWaitForGreenStatus().execute().actionGet();
for (int i = 0; i < 20; i++) {
You can’t perform that action at this time.