cordje

Spark Tips & Tricks

Misc. Tips & Tricks

• If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
• Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
• Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

cordje

First verify the version of Java being used is not Sun Java 8 SDK

java -version

Check the latest version a build numbers in http://www.oracle.com/technetwork/java/javase/downloads/index.html

For url http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz, the variables should be defined as:

java_base_version="8"
java_sub_version="11"

cordje

public class DamerauLevenshteinDistanceWithThreshold {

	public static int distance(String source, String target, int threshold) {
		//this code was ported to Java from http://stackoverflow.com/questions/9453731/how-to-calculate-distance-similarity-measure-of-given-2-strings/9454016#9454016

	    int length1 = source.length();
	    int length2 = target.length();

	    // Return trivial case - difference in string lengths exceeds threshhold
	    if (Math.abs(length1 - length2) > threshold) { return 2147483647; }