Skip to content

Instantly share code, notes, and snippets.

View andypetrella's full-sized avatar

Andy Petrella andypetrella

288 followers · 73 following
View GitHub Profile
@andypetrella
andypetrella / CRDD.scala
Last active August 29, 2015 14:04
CRDD
trait CRDD[S<:Space] {
def continuum:S
def break:S#SpacePoint => (S#SpacePoint, S#SpacePoint)
// the rest follow more or less the DStream definition
}
@andypetrella
andypetrella / README.md
Last active August 29, 2015 14:08
Zip won't work? → "Honi soit qui mal y pense"

This gist is extracting the problem I'm facing from the Spark-Notebook (see here) to deal with a dynamic form that generates new Spark SQL.

It is in a transient status, meaning that it still contain a lot of legacy constructions that I'd like to get rid:

  • Connection
  • Observable
  • Observer But one thing at a time :-D.

The result of this thing will visually be like the image below (or above :-D)

@andypetrella
andypetrella / Test MLlib.snb
Last active August 29, 2015 14:15
import mllib and netlib in notebook
{"content":{"metadata":{"name":"Test MLlib","user_save_timestamp":"1970-01-01T01:00:00.000Z","auto_save_timestamp":"1970-01-01T01:00:00.000Z","language_info":{"name":"scala","file_extension":"scala","codemirror_mode":"text/x-scala"},"trusted":true},"cells":[{"metadata":{"trusted":true,"collapsed":false},"cell_type":"code","source":":local-repo /tmp/spark-notebook","outputs":[{"name":"stdout","output_type":"stream","text":"res1: String = Repo changed to /tmp/spark-notebook!\n"},{"metadata":{},"data":{"text/html":"Repo changed to /tmp/spark-notebook!"},"output_type":"execute_result","execution_count":1}]},{"metadata":{"trusted":true,"collapsed":false},"cell_type":"code","source":":dp org.apache.spark % spark-mllib_2.10 % 1.2.0\n- org.apache.spark % spark-core_2.10 % _\n- org.apache.hadoop % _ % _\ncom.github.fommil.netlib % all % 1.1.2 % pom","outputs":[{"name":"stdout","output_type":"stream","text":"jars: Array[String] = [Ljava.lang.String;@4639adb7\nres1: scala.xml.Elem = \n<pre onclick=\"this.style.display=(
@andypetrella
andypetrella / TOC.cell.js
Last active October 2, 2015 18:26
Drop this cell into one of your Spark Notebook (https://github.com/andypetrella/spark-notebook/) declaring headings, execute, hide output, hide input
:javascript
require(["jquery", "underscore"], function(j, _) {
j(".toc").remove();
var toc = j(document.createElement("div"));
toc.attr("id", "toc")
.addClass("toc")
.css("position", "fixed")
.css("top", "15%")
.css("width", "8%")
.addClass("panel").addClass("panel-info");
@andypetrella
andypetrella / GraphFramesExample.snb
Last active March 4, 2016 18:25 — forked from deanwampler/GraphFramesExample.snb
Databrick's Python example for the new GraphFrame API ported to Scala and Spark Notebook
{
"metadata" : {
"name" : "GraphFramesExample",
"user_save_timestamp" : "1970-01-01T01:00:00.000Z",
"auto_save_timestamp" : "1970-01-01T01:00:00.000Z",
"language_info" : {
"name" : "scala",
"file_extension" : "scala",
"codemirror_mode" : "text/x-scala"
},
@andypetrella
andypetrella / QPlot.snb
Last active March 9, 2016 02:33
Super basic qplot implementation on Scala collection, using R backend as renderer... in 74 lines :-)
{
"metadata" : {
"name" : "QPlot",
"user_save_timestamp" : "1970-01-01T01:00:00.000Z",
"auto_save_timestamp" : "1970-01-01T01:00:00.000Z",
"language_info" : {
"name" : "scala",
"file_extension" : "scala",
"codemirror_mode" : "text/x-scala"
},
@andypetrella
andypetrella / StateStream-Workaround.snb
Created March 14, 2016 19:37
{
"metadata" : {
"name" : "Serializable StateStream",
"user_save_timestamp" : "1970-01-01T01:00:00.000Z",
"auto_save_timestamp" : "1970-01-01T01:00:00.000Z",
"language_info" : {
"name" : "scala",
"file_extension" : "scala",
"codemirror_mode" : "text/x-scala"
},
@andypetrella
andypetrella / buildnativeoperations.sh
Created May 11, 2016 12:39
building libnd4j
root@ip-172-31-15-75:/mnt/libnd4j# ./buildnativeoperations.sh
eval cmake
PACKAGING = none
BUILD = release
CHIP = cpu
LIBRARY TYPE = dynamic
/mnt/libnd4j/blasbuild/cpu
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
@andypetrella
andypetrella / nd4j - ld problem
Created June 17, 2016 08:50
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.nd4j:nd4j-jdbc-api:jar:0.4-rc3.11-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-deploy-plugin is missing. @ org.nd4j:nd4j:0.4-rc3.11-SNAPSHOT, /mnt/nd4j/pom.xml, line 327, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.nd4j:nd4j-jdbc-mysql:jar:0.4-rc3.11-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-deploy-plugin is missing. @ org.nd4j:nd4j:0.4-rc3.11-SNAPSHOT, /mnt/nd4j/pom.xml, line 327, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.nd4j:nd4j-jdbc:pom:0.4-rc3.11-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-deploy-plugin is missing. @ org.nd4j:nd4j:0.4-rc3.11-SNAPSHOT, /mnt/nd4j/pom.xml, line 327, column 21
@andypetrella
andypetrella / libnd4j log
Created June 17, 2016 08:54
root@ip-172-31-15-75:/mnt/nd4j# cd ../libnd4j/
root@ip-172-31-15-75:/mnt/libnd4j# ./buildnativeoperations.sh
eval cmake
PACKAGING = none
BUILD = release
CHIP = cpu
LIBRARY TYPE = dynamic
/mnt/libnd4j/blasbuild/cpu
-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4