Skip to content

Instantly share code, notes, and snippets.

View mindcrime's full-sized avatar
🌎
Plotting world domination...

Phillip Rhodes mindcrime

🌎
Plotting world domination...
97 followers · 1.1k following
View GitHub Profile
@mindcrime
mindcrime / gist:b697acc97e68b721669353aa8c6b19b2
Created March 30, 2016 00:53
How to add a subdomain with AWS Route 53 using the Java SDK
AWSCredentials credentials = null;
try
{
credentials = new PropertiesFileCredentialsProvider( "credentials.properties" ).getCredentials();
}
catch (Exception e)
{
throw new AmazonClientException(
"Cannot load the credentials from the credential profiles file. " +
"Please make sure that your credentials file is at the correct " +
@mindcrime
mindcrime / gist:27acd60af805a3b05072606b9a7ab83d
Created February 25, 2017 17:38
DL4J error loading data
17/02/25 12:32:41 INFO deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
17/02/25 12:32:41 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
17/02/25 12:32:41 INFO deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
17/02/25 12:32:42 INFO deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
17/02/25 12:32:42 INFO Nd4jBackend: Loaded [CpuBackend] backend
17/02/25 12:32:42 INFO NativeOpsHolder: Number of threads used for NativeOps: 4
17/02/25 12:32:43 INFO Reflections: Reflections took 242 ms to scan 12 urls, producing 29 keys and 176 values
17/02/25 12:32:44 INFO Reflections: Reflections took 595 ms to scan 12 urls, producing 368 keys and 1424 values
17/02/25 12:32:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NumberFormatException: For input string: "[[[[0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00,
@mindcrime
mindcrime / gist:043270fa30ebfe9ba557642585c1609b
Created February 25, 2017 17:43
DL4J code for loading MNIST with Spark
package org.fogbeam.dl4j.spark;
import java.util.Arrays;
import java.util.List;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.input.PortableDataStream;
@mindcrime
mindcrime / gist:eac54feaeb6181d8ed1732ae7ca4ba1f
Created February 25, 2017 19:46
Loading MNIST in Spark (again)
package org.fogbeam.dl4j.spark;
import java.util.Arrays;
import java.util.List;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.input.PortableDataStream;
@mindcrime
mindcrime / gist:b1b11121755df79d1ad8e7f3711c763d
Created February 25, 2017 22:31
Apparently working code for loading images with Spark/DL4J
JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/training/**");
ImageRecordReader irr = new ImageRecordReader(28,28,1,new ParentPathLabelGenerator());
List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
irr.setLabels(labelsList);
RecordReaderFunction rrf = new RecordReaderFunction(irr);
JavaRDD<List<Writable>> rdd = origData.map(rrf);
System.out.println( "DataSet RDD created");
@mindcrime
mindcrime / gist:3ff5b382e64e519889b005699b926f53
Created February 26, 2017 11:02
dl4j scaling isssue
JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/cutdown/0/**");
ImageRecordReader irr = new ImageRecordReader(28, 28, 1, new ParentPathLabelGenerator() );
List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
irr.setLabels(labelsList);
RecordReaderFunction rrf = new RecordReaderFunction(irr);
JavaRDD<List<Writable>> rdd = origData.map(rrf);
System.out.println( "DataSet RDD created");
@mindcrime
mindcrime / gist:6c149827205184617d4edad6f00342f3
Created February 26, 2017 18:53
Attempt to manually use scaler
JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/cutdown/0/**");
ImageRecordReader irr = new ImageRecordReader(28, 28, 1, new ParentPathLabelGenerator() );
List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
irr.setLabels(labelsList);
RecordReaderFunction rrf = new RecordReaderFunction(irr);
JavaRDD<List<Writable>> rdd = origData.map(rrf);
System.out.println( "DataSet RDD created");
@mindcrime
mindcrime / gist:f30b6641a58ecf6f059db9380a865f86
Created February 26, 2017 18:59
Complete example where values seem to not get scaled.
package org.fogbeam.dl4j.spark;
import java.io.File;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
@mindcrime
mindcrime / gist:b440cc83fde2a4cf7735fd37b31a23d8
Created February 26, 2017 19:23
call() method in DataSetDavaVecFunction
public DataSet call(List<Writable> currList) throws Exception {
//allow people to specify label index as -1 and infer the last possible label
int labelIndex = this.labelIndex;
if (numPossibleLabels >= 1 && labelIndex < 0) {
labelIndex = currList.size() - 1;
}
INDArray label = null;
INDArray featureVector = null;
@mindcrime
mindcrime / gist:1f16788a8330b746fbf187d332f932bf
Created February 26, 2017 20:28
manual invocation of the preprocessor
JavaRDD<DataSet> unscaledData = rdd.map(new DataVecDataSetFunction(1,10, false, null, null ));
JavaRDD<DataSet> trainingData = unscaledData.map( new Function<DataSet,DataSet>() {
DataNormalization scaler = new ImagePreProcessingScaler(0,1);
@Override
public DataSet call(DataSet v1) throws Exception {
DataSet newDS = new DataSet( v1.getFeatures(), v1.getLabels());
scaler.preProcess(newDS);