Skip to content

Instantly share code, notes, and snippets.

View skipperkongen's full-sized avatar

Pimin Konstantin Kefaloukos skipperkongen

View GitHub Profile
@skipperkongen
skipperkongen / load_and_query.scala
Last active August 20, 2017 13:10
Load and query data from S3 in Zeppelin (Spark)
// Load two data paths
val df1 = spark.read.load("/path/to/data1") // e.g. parquet files
val df2 = spark.read.load("/path/to/data2") // e.g. parquet files
// Union into single dataframe
df1.createOrReplaceTempView("data1")
df2.createOrReplaceTempView("data2")
val df = spark.sql("""
@skipperkongen
skipperkongen / capture.py
Last active October 11, 2018 11:44
Capture video from web cam on Mac
import cv2
from time import sleep
CLASSES = ['SAFE', 'DANGER']
NEG_IDX = 0
POS_IDX = 1
FRAMES_PER_VIDEO = 100
VIDEOS_PER_CLASS = 2
def capture(num_frames, path='out.avi'):
@skipperkongen
skipperkongen / video_features.py
Created October 11, 2018 11:49
Convert video frames to features
# Create X, y series
import cv2
import numpy as np
from keras.preprocessing import image
from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input
class VGGFramePreprocessor():
def __init__(self, vgg_model):
@skipperkongen
skipperkongen / train_videos.py
Created October 11, 2018 11:56
Train neural network on videos
from keras.models import Sequential, load_model
from keras.layers import Dense, Activation, Dropout
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
MODEL_PATH='model.h5'
EPOCHS = 10
HIDDEN_SIZE = 16
model = Sequential()
@skipperkongen
skipperkongen / inference.py
Created October 11, 2018 11:58
Inference on video
# Infer on live video
from math import ceil
import subprocess
import cv2
TEST_FRAMES = 500
# Initialize camera
cap = cv2.VideoCapture(0)
# Check if camera opened successfully
@skipperkongen
skipperkongen / draw_bipartite.py
Last active January 17, 2019 16:11
Generate and draw a bipartitite graph in NetworkX
%matplotlib inline
import networkx as nx
from networkx.algorithms import bipartite
from networkx.algorithms import community
from matplotlib import pyplot as plt
G = bipartite.gnmk_random_graph(3,5,10)
top = nx.bipartite.sets(G)[0]
pos = nx.bipartite_layout(G, top)
nx.draw_networkx(G,pos)
@skipperkongen
skipperkongen / show_image.py
Created January 25, 2019 10:33
Show numpy array as image in matplotlib/Jupyter
from matplotlib import pyplot as plt
import numpy as np
a = np.random.randn(10,10)
plt.imshow(a, cmap='gray')
@skipperkongen
skipperkongen / scrape-image.py
Last active February 7, 2019 13:35
Scrape images from the Google Images
# pip install icrawler
from icrawler.builtin import GoogleImageCrawler
import argparse
if __name__=='__main__':
parser = argparse.ArgumentParser(description='Scrape some images.')
parser.add_argument('keywords', metavar = 'KEYWORDS', nargs = '+',
help='keywords to download images for')
parser.add_argument('-n', '--max-num', type = int, default = 10,
@skipperkongen
skipperkongen / weeks.sql
Last active March 18, 2019 12:51
Select next weeks with year in Spark SQL
select
year(dt) as departure_year,
weekofyear(dt) as departure_week
from (
select date_add(current_date(), x) as dt
from `table_that_contains_integers_0_to_n_as_x`
where x % 7 = 0
)
order by dt

Keybase proof

I hereby claim:

  • I am skipperkongen on github.
  • I am skipperkongen (https://keybase.io/skipperkongen) on keybase.
  • I have a public key ASBjntm9D5V5uVtiIAogJKMeTmVedwh69aRp5p1fLzPqxQo

To claim this, I am signing this object: