Skip to content

Instantly share code, notes, and snippets.

View aperepel's full-sized avatar

Andrew Grande aperepel

53 followers · 12 following
View GitHub Profile
@aperepel
aperepel / Migrate CDE spark-submit.md
Last active December 2, 2021 19:33

Migrate spark-submit to cde spark submit

  • Drop --master
  • Drop --deploy-mode
  • Drop --spark.keytab
  • Drop --spark.yarn.principal
  • Drop any --spark.dynamicAllocation.enabled properties (already enabled in CDE by default)
  • Drop unravel
  • Rename --files to --file (plural syntax will be supported in the upcoming release)
  • Rename --py-files to --py-file (plural syntax will be supported in the upcoming release)
  • Break up comma-separated arguments for --files and --py-files into individual entries:
@aperepel
aperepel / example_http_get.json
Last active February 13, 2018 22:51
{
"snapshotMetadata" : {
"version" : 1,
"timestamp" : 1518561997786,
"author" : "anonymous",
"comments" : ""
},
"flowContents" : {
"identifier" : "9b658263-eec9-333a-9c57-afb2c5c86d83",
"name" : "Example Simple Http Get",
@aperepel
aperepel / NiFi_Flow_Provenance_Stream.json
Created February 13, 2018 22:30
An example NiFi Flow from the Apache Wiki: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
{
"snapshotMetadata" : {
"version" : 1,
"timestamp" : 1518560772117,
"author" : "anonymous",
"comments" : "Initial import from Apache NiFi Wiki"
},
"flowContents" : {
"identifier" : "d4c48e23-4188-3b3e-937c-5c2a3eef9031",
"name" : "Provenance Demo",
@aperepel
aperepel / test_flow.json
Last active February 10, 2018 21:12
Test NiFi Flow URL
{
"snapshotMetadata" : {
"version" : 2,
"timestamp" : 1517693159539,
"author" : "anonymous",
"comments" : "Added a property referencing a variable registry"
},
"flowContents" : {
"identifier" : "eb3b8a22-bc5b-3ece-b26b-01f5f131cfb3",
"name" : "First Flow",
@aperepel
aperepel / MNIST Keras.py
Last active December 8, 2017 23:11
K.clear_session()
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(28, 28, 1)))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Activation('relu'))
model.add(Flatten())
@aperepel
aperepel / HBase REST API Processor
Last active May 25, 2017 21:48
A scripted NiFi processor taking an incoming JSON, extracting key/values according to a user-provided filter, and prepping data according to the HBase REST API format specification.
import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import org.apache.nifi.flowfile.attributes.CoreAttributes
class HBaseRestJsonTransformer implements Processor {
def REL_SUCCESS = new Relationship.Builder()
.name('success')
.description('A JSON document formatted for HBase REST API')
@aperepel
aperepel / AtTinyResponder.cpp
Created February 18, 2016 23:52
#include <SoftSerial.h>
SoftSerial mySerial(1, 0);
void setup() {
mySerial.begin(57600);
}
void loop() {
int lastRead;
@aperepel
aperepel / Dockerfile
Last active February 18, 2016 17:46
Hazelcast Consul
FROM hazelcast/hazelcast
MAINTAINER Andrew Grande
RUN mkdir lib
#RUN wget -nv -P lib/ http://jcenter.bintray.com/org/bitsofinfo/hazelcast-consul-discovery-spi/1.0-RC2/hazelcast-consul-discovery-spi-1.0-RC2.jar
#RUN wget -nv -P lib/ http://central.maven.org/maven2/com/orbitz/consul/consul-client/0.9.16/consul-client-0.9.16.jar
#RUN wget -nv -P lib/ http://central.maven.org/maven2/com/google/guava/guava/18.0/guava-18.0.jar
#RUN wget -nv -P lib/ http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.3/jackson-core-2.6.3.jar
#RUN wget -nv -P lib/ http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.3/jackson-databind-2.6.3.jar
#RUN wget -nv -P lib/ http://central.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-guava/2.6.3/jackson-datatype-guava-2.6.3.jar