Skip to content

Instantly share code, notes, and snippets.

View nddipiazza's full-sized avatar

Nicholas DiPiazza nddipiazza

9 followers · 1 following
View GitHub Profile
@nddipiazza
nddipiazza / gist:dc645cd2763d0ff4f4d63de7ca425840
Created February 16, 2017 19:48
package spauth;
import java.io.*;
import java.net.*;
import javax.xml.parsers.*;
import javax.xml.xpath.*;
import org.w3c.dom.Document;
import org.xml.sax.*;
public class LoginManager {
@nddipiazza
nddipiazza / extract-stacktraces.py
Created April 3, 2017 21:33
extract stacktraces from java logs
#!/usr/bin/env python
#
# Extracts exceptions from log files.
#
import sys
import re
from collections import defaultdict
REGEX = re.compile("(^\tat |^Caused by: |^\t... \\d+ more)")
@nddipiazza
nddipiazza / gist:00c5128aba4835bd494af4ed4d3480ad
Created April 11, 2017 13:23
Spark job that crawls database and joins fields into Field String array instead of extra Solr documents
val jdbcUrl = "jdbc:sqlserver://SERVER_IP;user=USERNAME;password=PASSWORD"
val maxId = sqlContext.jdbc(jdbcUrl, "(select max(Person_Number) as maxId from [Export].[dbo].[Person]) as tmp").select("maxId").collect()(0)(0).toString
val dbOpts = Map(
"url" -> jdbcUrl,
"dbtable" -> "[Export].[dbo].[Person]",
"partitionColumn" -> "Person_Number",
"numPartitions" -> "4",
"lowerBound" -> "0",
"upperBound" -> maxId,
"fetchSize" -> "1000"
@nddipiazza
nddipiazza / gist:9b9c007e06965291dab68fe1e630d71e
Created April 11, 2017 15:43
Spark JDBC Job - With Joins into Field String Arrays
val jdbcUrl = "jdbc:sqlserver://SERVER_IP;user=USERNAME;password=PASSWORD"
val maxId = sqlContext.jdbc(jdbcUrl, "(select max(Person_Number) as maxId from [Export].[dbo].[Person]) as tmp").select("maxId").collect()(0)(0).toString
val dbOpts = Map(
"url" -> jdbcUrl,
"dbtable" -> "[Export].[dbo].[Person]",
"partitionColumn" -> "Person_Number",
"numPartitions" -> "4",
"lowerBound" -> "0",
"upperBound" -> maxId,
"fetchSize" -> "1000"
@nddipiazza
nddipiazza / fusion-3node-cluster-setup.sh
Last active July 28, 2017 20:23
A simple bash script that can set up a Linux node for a 3 node cluster
#!/usr/bin/env bash
# Enter each private IP address of each node in this list
zkhost1=IPADDRESS1
zkhost2=IPADDRESS2
zkhost3=IPADDRESS3
zkhost=$zkhost1:9983,$zkhost2:9983,$zkhost3:9983
# Assign an ID to this node.
# This will be the only thing in this script that is unique on each node of the cluster.
# Use increasing IDs starting with 1, 2, 3, etc.
@nddipiazza
nddipiazza / pre-commit-hook.sh
Last active July 30, 2017 17:45
#!/usr/bin/env bash
set -eou pipefail
DIR=$(dirname "$0")
"$DIR/gradlew" --parallel --max-workers=8 -p "$DIR" compileJava compileTestJava
@nddipiazza
nddipiazza / checkstyle.xml for fusion
Last active August 25, 2017 01:17
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://checkstyle.sourceforge.net/dtds/configuration_1_3.dtd">
<!--
Checkstyle configuration that checks the Google coding conventions from Google Java Style
that can be found at https://google.github.io/styleguide/javaguide.html.
Checkstyle is very configurable. Be sure to read the documentation at
@nddipiazza
nddipiazza / HeadlessFirefoxSeleniumExample.java
Created September 1, 2017 17:51
Headless firefox (without xvfb) example
package com.mozilla.example;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxBinary;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import java.util.concurrent.TimeUnit;
@nddipiazza
nddipiazza / build.gradle
Created September 1, 2017 17:52
Gradle file for the headlessfirefox selenium example
group 'com.mozilla'
version '1.0'
apply plugin: 'java'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
@nddipiazza
nddipiazza / javascript_timeout.html
Last active September 12, 2017 14:57
page that will hard loop in the <script> section.
<!DOCTYPE html>
<html>
<body>
<div id="container">Will never display</div>
<script>
while (true) {}
</script>
</body>
</html>