This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
private static String [] MAIL_LIST_PROJECTIONS = { | |
MessageProvider.MessageColumns._ID, | |
MessageProvider.MessageColumns.SENDER_ADDRESS, | |
MessageProvider.MessageColumns.SENDER, | |
MessageProvider.MessageColumns.SEND_DATE, | |
MessageProvider.MessageColumns.SUBJECT, | |
MessageProvider.MessageColumns.PREVIEW, | |
MessageProvider.MessageColumns.UNREAD, | |
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2016-09-18 20:29:58 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
2016-09-18 20:31:34 INFO deprecation:1173 - session.id is deprecated. Instead, use dfs.metrics.session-id | |
2016-09-18 20:31:34 INFO JvmMetrics:76 - Initializing JVM Metrics with processName=JobTracker, sessionId= | |
2016-09-18 20:31:35 WARN JobResourceUploader:171 - No job jar file set. User classes may not be found. See Job or Job#setJar(String). | |
2016-09-18 20:31:35 INFO FileInputFormat:283 - Total input paths to process : 0 | |
2016-09-18 20:31:35 INFO JobSubmitter:198 - number of splits:0 | |
2016-09-18 20:31:35 INFO JobSubmitter:287 - Submitting tokens for job: job_local1137380161_0001 | |
2016-09-18 20:31:36 INFO Job:1294 - The url to track the job: http://localhost:8080/ | |
2016-09-18 20:31:36 INFO Job:1339 - Running job: job_local1137380161_0001 | |
2016-09-18 20:31:36 INFO LocalJobRunner:471 - OutputCommitter set in config null |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public class PointVector implements WritableComparable<PointVector> { | |
/** | |
* Keep the tfIdf values of the terms of a document | |
*/ | |
private Vector<Double> data = new Vector<>(); | |
public PointVector(double [] values) { | |
this.data = new Vector<>(values.length); | |
this.data.addAll(Doubles.asList(values)); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public class KMeansReducer extends Reducer<PointVector, PointVector, Text, Text> { | |
private double min_dist = Double.MAX_VALUE; | |
@Override | |
public void reduce(PointVector center, Iterable<PointVector> points, Context context) throws IOException, InterruptedException { | |
EuclideanDistance measure = new EuclideanDistance(); | |
double distance = 0.0; | |
int numOfPoints = 0; | |
double centerx = 0; | |
double centery = 0; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public class KMeansMapper extends Mapper<LongWritable, Text, PointVector, PointVector> { | |
private int clusters; | |
private List<ImmutableTriple<Integer, String, PointVector>> centers; | |
@Override | |
protected void setup(Context context) throws IOException, InterruptedException { | |
System.out.println("Inside setup"); | |
this.clusters = Integer.valueOf(context.getConfiguration().get("clusters")); | |
this.centers = new ArrayList<>(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* Implementation of k-means algorithm using Map Reduce paradigm. | |
*/ | |
public class KMeans extends Configured implements Tool { | |
/** | |
* Number of indexed documents | |
*/ | |
private int indexedDocuments; | |
private List<ImmutableTriple<Integer, String, PointVector>> centers; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[denis@denisArch]$ sudo cp INSTALL.txt /etc/ | |
[sudo] password for denis: | |
/usr/bin/cp INSTALL /etc/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## sudoers file. | |
## | |
## This file MUST be edited with the 'visudo' command as root. | |
## Failure to use 'visudo' may result in syntax or file permission errors | |
## that prevent sudo from running. | |
## | |
## See the sudoers man page for the details on how to write a sudoers file. | |
## | |
## |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// taken from http://www.mkyong.com/java/how-to-compress-files-in-zip-format/ | |
// and slightly modified to test my hypothesis. | |
import java.io.File; | |
import java.io.FileInputStream; | |
import java.io.FileOutputStream; | |
import java.io.IOException; | |
import java.io.FileNotFoundException; | |
import java.util.ArrayList; | |
import java.util.List; |