Skip to content

Instantly share code, notes, and snippets.

View ctivanovich's full-sized avatar

Christopher Ivan ctivanovich

5 followers · 6 following
View GitHub Profile
@ctivanovich
ctivanovich / scrape_wunderground_weather.py
Last active May 6, 2019 08:53
A weather data scraper I made for Wunderground.com, using XPaths and dealing with asynchronously loading page elements.
import bs4
import datetime
import os
import pandas as pd
import psycopg2
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
@ctivanovich
ctivanovich / making_views.sql
Last active May 6, 2019 08:42
Complex Postgresql query using window functions and cases, building views and materialized views
CREATE OR REPLACE VIEW public.centiles_by_small_region
AS SELECT s2.small_pid,
s2.p_region,
percentile_cont(0.30::double precision) WITHIN GROUP (ORDER BY (mode_prices.mode_price::double precision)) AS percentile_30,
percentile_cont(0.60::double precision) WITHIN GROUP (ORDER BY (mode_prices.mode_price::double precision)) AS percentile_60,
percentile_cont(0.90::double precision) WITHIN GROUP (ORDER BY (mode_prices.mode_price::double precision)) AS percentile_90
FROM mode_prices
JOIN ( SELECT product_id_sml_class.product_id,
product_id_sml_class.small_pid,
CASE
@ctivanovich
ctivanovich / Enron_Fraud.ipynb
Last active April 4, 2018 08:50
Machine Learning to identify suspects by financial data for Enron employees
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@ctivanovich
ctivanovich / ML on cryptocurrency trading.ipynb
Last active May 9, 2018 01:14
Predictive modeling of trading data from a cryptocurrency platform
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@ctivanovich
ctivanovich / index.html
Last active March 23, 2018 10:13
A D3/Dimple Line Chart with a tsv source
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<script src="http://d3js.org/d3.v3.min.js"></script>
<script src="http://dimplejs.org/dist/dimple.v2.0.0.min.js"></script>
<style>
circle.dimple-series-1 {
fill: red;
@ctivanovich
ctivanovich / date_from_text_preprocessing.py
Last active February 20, 2018 08:07
def preprocessing(doc):
'''Doc is assumed to be an open file object of English-language text.
Function outputs a dictionary with keys as line numbers from the document,
values as lists containing likely date-element candidates.
Years are assumed to be between 1900-2018 inclusive, but this can be easily changed.'''
import re
maybedatestuff = {i:[] for i in range(len(doc))}
for i, line in enumerate(doc):