Skip to content

Instantly share code, notes, and snippets.

View tomgrek's full-sized avatar

Tom Grek tomgrek

147 followers · 2 following
View GitHub Profile
@tomgrek
tomgrek / denorm.md
Last active February 25, 2018 22:03
Denormalized Example

Originally, we join a table like this

zip_code data_val
94105 123.45
94106 100.10

with one like this

zip_code|city

@tomgrek
tomgrek / kops.md
Created February 25, 2018 22:53
Snippet to create a private cluster on AWS with Kops
  • Create a t2 micro instance for administering the cluster
  • Give it an appropriate IAM role
  • Create an S3 bucket to persist cluster state

Then:

export NAME=mycluster.k8s.local
export KOPS_STATE_STORE=s3://k8s-state-store-mycluster
kops create cluster --topology=private --networking=flannel --zones us-west-2c mycluster.k8s.local --vpc=vpc-012345ab --ssh-access [my IP address] --network-cidr=[my VPC CIDR block] --associate-public-ip=false --master-size m4.large --node-size m4.large
kops edit ig --name=mycluster.k8s.local nodes (set how many machines)
@tomgrek
tomgrek / datagen.py
Created March 4, 2018 22:34
Iterator returning pairs of NLP training data suitable for PyTorch model with an embedding layer
"""
To use, simply pass a list of ints e.g. [1,2,3,4] will yield (1,2), (2,3), (3,4), StopIteration
"""
class DataGenerator():
def __init__(self, dset):
self.dset = dset
self.len = len(self.dset)
self.idx = 0
def __len__(self):
@tomgrek
tomgrek / model.py
Last active March 5, 2018 04:44
PyTorch model for a simple AI chatbot
class BotBrain(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(len(words), 10)
self.rnn = nn.LSTM(10, 20, 2, dropout=0.5)
self.h = (Variable(torch.zeros(2, 1, 20)), Variable(torch.zeros(2, 1, 20)))
self.l_out = nn.Linear(20, len(words))
def forward(self, cs):
inp = self.embedding(cs)
@tomgrek
tomgrek / trainingloop.py
Last active June 22, 2020 17:11
A PyTorch training loop
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(m.parameters(), lr=0.01)
for epoch in range(0,300):
gen = DataGenerator([words[word.lower()] for word in ' '.join(sentences).replace('?',' <unk>').split(' ')])
for x, y in gen:
m.zero_grad()
output = m(x)
loss = criterion(output, y)
loss.backward()
optimizer.step()
@tomgrek
tomgrek / 2_a.py
Created March 23, 2018 02:36
chatbot article part 2_a
class DataGenerator():
def __init__(self, dset):
self.dset = dset
self.len = len(self.dset)
self.idx = 0
def __len__(self):
return self.len
def __iter__(self):
return self
def __next__(self):
@tomgrek
tomgrek / 2_b.py
Last active March 23, 2018 02:49
2_b
class BotBrain(nn.Module):
def __init__(self):
super().__init__()
self.embedding = nn.Embedding(len(words), 10)
self.rnn = nn.LSTM(20, 30, 2, dropout=0.5)
self.h = (Variable(torch.zeros(2, 1, 30)).cuda(), Variable(torch.zeros(2, 1, 30)))
self.l_out = nn.Linear(30, len(words))
def forward(self, cs):
inp = (self.embedding(cs)).view(1, -1)
@tomgrek
tomgrek / index.html
Last active May 3, 2018 17:09 — forked from mourner/index.html
Mapbox GL JS Puppeteer benchmark
<!doctype html>
<meta charset="utf-8">
<title>Benchmark</title>
<body></body>
<style>html, body, #map { height: 100%; margin: 0; } </style>
<div id="map"></div>
<script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.40.0/mapbox-gl.js'></script>
<!-- <script src="mapbox-gl.js"></script> -->
@tomgrek
tomgrek / Policy.py
Last active September 1, 2018 20:53
My neural network for actor critic financial trading
class Policy(nn.Module):
def __init__(self):
super(Policy, self).__init__()
self.input_layer = nn.Linear(8, 128)
self.hidden_1 = nn.Linear(128, 128)
self.hidden_2 = nn.Linear(32,31)
self.hidden_state = torch.tensor(torch.zeros(2,1,32), requires_grad=False).cuda()
self.rnn = nn.GRU(128, 32, 2)
self.action_head = nn.Linear(31, 5)
self.value_head = nn.Linear(31, 1)
@tomgrek
tomgrek / default
Created November 17, 2018 03:29
nginx front and backend config on same server
server {
root /var/www/xxx;
index index.html;
server_name xxx.xxx;
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain; charset=utf-8';