Skip to content

Instantly share code, notes, and snippets.

View kmader's full-sized avatar

Kevin Mader kmader

304 followers · 214 following
  • Zurich, Switzerland
View GitHub Profile
@kmader
kmader / README.md
Last active October 31, 2023 14:21
Beating Serialization in Spark

Serialization

As all objects must be Serializable to be used as part of RDD operations in Spark, it can be difficult to work with libraries which do not implement these featuers.

Java Solutions

Simple Classes

For simple classes, it is easiest to make a wrapper interface that extends Serializable. This means that even though UnserializableObject cannot be serialized we can pass in the following object without any issue

public interface UnserializableWrapper extends Serializable {
 public UnserializableObject create(String parm1, String parm2);
@kmader
kmader / README.md
Last active March 15, 2021 13:25 — forked from bellbind/loader.html
[threejs][html5]STL File Viewer with HTML5 File API
@kmader
kmader / CNN_MNIST_PlaidML.ipynb
Last active September 9, 2020 04:06
PlaidML Keras MNIST
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@kmader
kmader / dup_deleter.py
Created December 18, 2014 21:02
Delete duplicate entries from a bibtex library (only the first)
bibstr='\n'.join(open('library.bib').readlines())
bibents=bibstr.split('@article')
def get_key(tstr):
(stpos,endpos) = (tstr.find('{'),tstr.find(','))
if(stpos>=0) & (endpos>stpos): return tstr[stpos+1:endpos]
else: return tstr # keep everything
biblist=map(lambda x: (get_key(x),x),bibents)
outlist=[]
keyalready={}
@kmader
kmader / CovidSpread.html
Created June 24, 2020 13:16
This file has been truncated, but you can view the full file.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>IPyVolume Widget</title>
</head>
<body>
@kmader
kmader / SparkDemo.scala
Last active June 17, 2020 03:06
A demo script for loading data into Spark and performing a few basic analyzes (from presentation http://4quant.github.io/spark-introduction on slide https://rawgit.com/4Quant/spark-introduction/master/TutorialSlides.html#/26). To get started make a clone of the repository locally (git clone https://gist.github.com/755c2d99c23f4cbe2e74.git) and p…
val inData=sc.textFile("tiny.csv")
// Read the first line and split by commas
import scala.collection.immutable.StringOps
val header= inData.first
val headerLine = header.split(",").map(_.trim.filter(_ != '"'))
// Count lines in the file
inData.count
// Remove the header from the data
val rowData = inData.filter(_ != header).cache
// Divide each row into columns by seperating by commas
@kmader
kmader / prep_convert.py
Last active February 18, 2020 08:10
Convert a preprocessing function into a convolutional layer (in Keras). It takes an input from -127 to 127 and runs it through the preprocess_input function and then returns a convolutional layer with the appropriate weights and biases to reproduce (as closely as possible) the preprocessing step. This allows models to be packaged without worryin…
from keras.applications.mobilenet_v2 import MobileNetV2, preprocess_input
from keras import layers, models
from sklearn.linear_model import LinearRegression
import numpy as np
def prep_to_conv(
in_prep_func: Callable[[np.array], np.array],
*,
min_val: float=-127,
max_val: float=127,
@kmader
kmader / freeze_parameters.ipynb
Created December 5, 2019 14:26
How to freeze parameters in a model using pytorch
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@kmader
kmader / 00215.tif
Last active November 28, 2019 14:01
Tiff Read Error in skimage
@kmader
kmader / more_working.R
Created August 28, 2019 12:59
https://www.r-bloggers.com/stock-market-data-scenario-set-generation-sp-100/
library(quantmod)
library(rvest)
table_id <- 3
url <- "https://en.wikipedia.org/wiki/S%26P_100"
sp100 <- url %>%
read_html() %>%
html_nodes("table") %>%
.[[table_id]] %>%
html_table()