Skip to content

Instantly share code, notes, and snippets.

View metadaddy's full-sized avatar

Pat Patterson metadaddy

326 followers · 3 following
View GitHub Profile
@metadaddy
metadaddy / CustomTransformer.scala
Last active September 27, 2018 03:42
Creating a StreamSets Spark Transformer in Scala - after second code expansion
package com.streamsets.spark.scala
import com.streamsets.pipeline.api.Field
import com.streamsets.pipeline.api.Record
import com.streamsets.pipeline.spark.api.SparkTransformer
import com.streamsets.pipeline.spark.api.TransformResult
import org.apache.spark.api.java.JavaPairRDD
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.rdd.RDD
@metadaddy
metadaddy / CustomTransformer.scala
Last active September 27, 2018 03:42
Creating a StreamSets Spark Transformer in Scala - after first code expansion
package com.streamsets.spark.scala
import com.streamsets.pipeline.api.Field
import com.streamsets.pipeline.api.Record
import com.streamsets.pipeline.spark.api.SparkTransformer
import com.streamsets.pipeline.spark.api.TransformResult
import org.apache.spark.api.java.JavaPairRDD
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.api.java.JavaSparkContext
@metadaddy
metadaddy / TaxiData.html
Created June 20, 2017 23:50
Databricks Notebook for Analyzing Taxi Transaction Data
<!DOCTYPE html>
<html>
<head>
<meta name="databricks-html-version" content="1">
<title>TaxiData - Databricks</title>
<meta charset="utf-8">
<meta name="google" content="notranslate">
<meta name="robots" content="nofollow">
<meta http-equiv="Content-Language" content="en">
@metadaddy
metadaddy / DomainNameEL.java
Created April 28, 2017 05:42
Example Custom EL Function for StreamSets Data Collector
/**
* Copyright 2017 StreamSets Inc.
*
* Licensed under the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
@metadaddy
metadaddy / Field Manipulations - Groovy.json
Created April 11, 2017 22:58
Manipulate fields in StreamSets Data Collector via Groovy
{
"pipelineConfig" : {
"schemaVersion" : 2,
"version" : 5,
"uuid" : "1fba4dbe-5855-4765-92ec-722fe832405a",
"title" : "Field Manipulations - Groovy",
"description" : "",
"configuration" : [ {
"name" : "executionMode",
"value" : "STANDALONE"
@metadaddy
metadaddy / Field Manipulations.json
Last active April 11, 2017 23:07
Manipulate fields in StreamSets Data Collector - download and import into SDC 2.4.0.0 or above!
{
"pipelineConfig" : {
"schemaVersion" : 2,
"version" : 5,
"uuid" : "e027d7d1-12db-4d9a-8f9a-827937c7c4df",
"title" : "Field Manipulations",
"description" : "",
"configuration" : [ {
"name" : "executionMode",
"value" : "STANDALONE"
@metadaddy
metadaddy / setup.sh
Last active February 14, 2023 08:06
Setup a SalesforceDX scratch org, open an IP range, check we can log in
#! /bin/bash
#
# Requires jq - see https://stedolan.github.io/jq/
#
# Assumes that the following env vars are set:
#
# CLIENT_ID - the client ID for your OAuth 2 app
# JWT_KEY_FILE - path to the private key for creating a JWT
# HUB_USERNAME - authentication username
@metadaddy
metadaddy / splunk.py
Last active January 18, 2017 02:06
Write data to Splunk from StreamSets Data Collector
import sys
# Set to wherever the requests package lives on your machine
sys.path.append('/Library/Python/2.7/site-packages')
import requests
import json
# Endpoint for Splunk HTTP Event Collector
url = 'http://localhost:8088/services/collector'
# Splunk metadata fields
@metadaddy
metadaddy / GetMeanSD.java
Created August 18, 2016 18:42
Retrieve mean and standard deviation from Cassandra, writing values to a file
package com.streamsets.example.cassandra;
import com.datastax.driver.core.BoundStatement;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.ResultSet;
import com.datastax.driver.core.Row;
import com.datastax.driver.core.Session;
import java.io.PrintWriter;
@metadaddy
metadaddy / keybase.md
Created August 17, 2016 01:52

Keybase proof

I hereby claim:

  • I am metadaddy on github.
  • I am metadaddy (https://keybase.io/metadaddy) on keybase.
  • I have a public key whose fingerprint is 35AF ABB8 08B8 FF6F 1C44 2D28 4EE1 0A59 5D19 24CB

To claim this, I am signing this object: