Skip to content

Instantly share code, notes, and snippets.

View crockpotveggies's full-sized avatar
🥕
carrot

Justin Long crockpotveggies

🥕
carrot
View GitHub Profile
/**
* Started to rough this naive S3-native filesystem RDD out because I need to use IAM
* profiles for S3 access and also https://issues.apache.org/jira/browse/HADOOP-3733.
*
* Use at your own risk, bear in mind this is maybe 30 - 45min of work and testing and
* expect it to behave as such.
*
* Feedback/criticism/discussion welcome via Github/Twitter
*
* In addition to Spark 1.0.x, this depends on Amazon's S3 SDK, dependency is as follows:
@crockpotveggies
crockpotveggies / SquAction.scala
Created March 28, 2012 06:29
Play! 2.0 is a meanie head so we slap that bitch with SquAction, a trait binding session to action thread
package controllers
import play.api.mvc._
import org.squeryl._
import org.squeryl.PrimitiveTypeMode._
/**
* bind a session to each action with this nifty Squeryl trait
*/
@crockpotveggies
crockpotveggies / README.markdown
Created April 11, 2012 18:52 — forked from Yavari/README.markdown
Bootstrap's Typeahead plugin extended (allowing for AJAX functionality) among other things

This is a fork of Bootstrap Typeahead that adds minimal but powerful extensions and eliminates matching to prevent over-filtering of AJAX responses.

Forked from https://gist.github.com/1891669.

For example, process typeahead list asynchronously and return objects

  # This example does an AJAX lookup and is in CoffeeScript
  $('.typeahead').typeahead(
    # source can be a function
@crockpotveggies
crockpotveggies / BrowserSocket.scala
Created September 20, 2012 17:32 — forked from minosiants/gist:3163791
Akka MessageBus with WebSockets access wrapped in Actors
/**
* actor wrapping access for browser socket
*/
class BrowserSocket(
val s: WebSocketConnection,
val userId: Long,
val teamId: Long
) extends Actor {
package org.deeplearning4j.VGGwebDemo;
import org.nd4j.linalg.factory.Nd4j;
import org.deeplearning4j.nn.modelimport.keras.trainedmodels.Utils.ImageNetLabels;
import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.nn.graph.ComputationGraph;
import org.deeplearning4j.nn.modelimport.keras.trainedmodels.TrainedModels;
import org.deeplearning4j.util.ModelSerializer;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
@crockpotveggies
crockpotveggies / build-dl4j-stack.sh
Last active March 3, 2017 03:29
Script that cleanly builds and locally installs the Deeplearning4j stack
#!/bin/bash
# helper function that ensures cmd returns 0 exit code
function checkexit {
"$@"
local status=$?
if [ $status -ne 0 ]; then
echo "error with $1" >&2
exit 1
fi
Name ALEXNET
Description CIFAR-10 128x3x224x224
Operating System GNU/Linux Ubuntu 16.04.2 LTS
Devices TITAN X (Pascal) 6 1 12779978752
CPU Cores 12
Backend CUDA
BLAS Vendor CUBLAS
CUDA Version 8000
CUDNN Version 6020
Total Params 20344650
@crockpotveggies
crockpotveggies / PreSave.java
Created April 13, 2017 02:30
PreSaving (preprocessing) a dataset for faster training in Deeplearning4j and DataVec.
import lombok.extern.slf4j.Slf4j;
import org.datavec.api.berkeley.Pair;
import org.datavec.api.io.filters.RandomPathFilter;
import org.datavec.api.io.labels.ParentPathLabelGenerator;
import org.datavec.api.split.FileSplit;
import org.datavec.api.split.InputSplit;
import org.datavec.image.loader.NativeImageLoader;
import org.datavec.image.recordreader.ImageRecordReader;
import org.datavec.image.transform.FlipImageTransform;
import org.datavec.image.transform.ImageTransform;
@crockpotveggies
crockpotveggies / imagemagickcorruptfiles
Created May 18, 2017 16:25
Remove all corrupt PNGs with bad headers using ImageMagick and identify command
for i in $(find ./ -maxdepth 2 -iname '*.png' -type f -print); do if $(identify "$i" 2>&1 |grep -q improper); then echo "$i incorrect"; fi; done
@crockpotveggies
crockpotveggies / GeoAveragingReduction.scala
Last active October 25, 2017 06:12
A DataVec reduction op for averaging a list of coordinates
// our reduction op class that we will need shortly
class GeoAveragingReduction(columnOutputName: String="AveragedLatLon") extends AggregableColumnReduction {
override def reduceColumn(columnData: java.util.List[Writable]): Writable = {
val coordinates = columnData.map( d => d.toString.split(",") ).map{ case Array(lat,lon) => (lat.toDouble,lon.toDouble) }
val (lat,lon) = coordinates.reduceLeft(avgCoordinates)
new Text(s"""$lat,$lon""")
}
private def avgCoordinates(latlon1: (Double, Double), latlon2: (Double, Double)): (Double, Double) = {