Skip to content

Instantly share code, notes, and snippets.

@kovid-rathee
kovid-rathee / pandas_mysql_dataframe.py
Created January 25, 2017 19:05
Script to connect to a MySQL source, fetch data and print loosely formatted data using pandas DataFrame
import MySQLdb as mdb
import pandas as pd
con = mdb.connect(‘127.0.0.1’, ‘root’, ‘password’, ‘database_name’);
with con:
cur = con.cursor()
cur.execute(“select random_number_one, random_number_two, random_number_three from randomness.a_random_table”)
rows = cur.fetchall()
df = pd.DataFrame( [[ij for ij in i] for i in rows] )
df.rename(columns={0: ‘Random Number One’, 1: ‘Random Number Two’, 2: ‘Random Number Three’}, inplace=True);
print(df.head(20))
@kovid-rathee
kovid-rathee / pandas_scatterplot_pyplot.py
Created January 25, 2017 20:11
Create a basic scatter plot using matplotlib on random data set generated by NumPy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# if you want a list of integers as your random data set
df = pd.DataFrame(np.random.randint(0,20,size=(20, 2)), columns=list('PQ'))
# if you want a list of random decimal numbers as your random data set
df = pd.DataFrame(np.random.randn(20,2), columns=list('PQ'))
# plot the data
plt.scatter(df.P,df.Q, alpha=0.5)
# plt.show()
@kovid-rathee
kovid-rathee / etl_mysql_redshift.sh
Last active February 3, 2017 11:15
Script to Load a table from MySQL to Amazon Redshift with Alerting.
#!/bin/sh
credentials()
{
username=loader
password=L0#De&1234
hostname=127.0.0.1
database=yourdatabasename
postgres_user=postgresusr
@kovid-rathee
kovid-rathee / static_server.js
Created February 3, 2017 10:39 — forked from ryanflorence/static_server.js
Node.JS static file web server. Put it in your path to fire up servers in any directory, takes an optional port argument.
var http = require("http"),
url = require("url"),
path = require("path"),
fs = require("fs")
port = process.argv[2] || 8888;
http.createServer(function(request, response) {
var uri = url.parse(request.url).pathname
, filename = path.join(process.cwd(), uri);
@kovid-rathee
kovid-rathee / rds2redshift.sh
Created February 3, 2017 21:15 — forked from erincerys/rds2redshift.sh
Loads a MySQL data dump into a Redshift table. Useful for AWS RDS instances. Dependent on external script to stream MySQL data to file, and postgres psql command line client.
PYTHON_PATH=$(which python)
PSQL_PATH=$(which psql)
MYSQL_SCRIPT='mysql2file.py'
MYSQL_SERVER=
MYSQL_PORT=3306
MYSQL_DATABASE=
MYSQL_USER=
MYSQL_PASSWORD=
@kovid-rathee
kovid-rathee / fetch_organize_instalooter.py
Created February 4, 2017 07:41
Python Script to fetch and organize Instagram photos and videos chronologically using instaLooter
import os
import sys
import subprocess
import instaLooter
basepath = "/Users/kovid.rathee/Desktop/instaLooter/"
accounts = ['rupikaur_','fursty','tropicalratchet','dylankato','asyrafacha','tomashavel','runawayueli','dreamingandwandering']
for account in accounts:
dir = basepath + account
print(os.path.isdir(dir))
if not os.path.exists(dir):
@kovid-rathee
kovid-rathee / mysql-rename-db.sh
Created February 5, 2017 09:21 — forked from tadas-s/mysql-rename-db.sh
MySQL database rename script
#!/bin/bash
# Disclaimer - make backups, use at your own risk.
#
# Based on this comment: http://stackoverflow.com/a/13944924/843067
# Views and stored procedures have to be done separately.
OLDDB="old_db_name"
NEWDB="new_db_name"
MYSQL="mysql -u root -pyour_password "
@kovid-rathee
kovid-rathee / README.md
Created February 5, 2017 09:23 — forked from oodavid/README.md
Restore MySQL from Amazon S3

Restore MySQL from Amazon S3

This is a hands-on way to pull down a set of MySQL dumps from Amazon S3 and restore your database with it

Sister Document - Backup MySQL to Amazon S3 - read that first

1 - Set your MySQL password and S3 bucket, make a temp dir, get a list of snapshots

# Set our variables

export mysqlpass="ROOTPASSWORD"

@kovid-rathee
kovid-rathee / dump.sh
Created February 5, 2017 09:24 — forked from andsens/dump.sh
Backup all MySQL databases into separate files
#!/bin/sh
## backup each mysql db into a different file, rather than one big file
## as with --all-databases. This will make restores easier.
## To backup a single database simply add the db name as a parameter (or multiple dbs)
## Putting the script in /var/backups/mysql seems sensible... on a debian machine that is
## Create the user and directories
# mkdir -p /var/backups/mysql/databases
# useradd --home-dir /var/backups/mysql --gid backup --no-create-home mysql-backup
## Remember to make the script executable, and unreadable by others
@kovid-rathee
kovid-rathee / closure_table.md
Created February 5, 2017 09:33
Persistent tree structure using closure table in MySQL

Using closure tables to manage hierarchical relations in MySQL

Create DB tables

Create a table to represent tree nodes.

CREATE TABLE `tree_node` (
    `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
    `data_body` text,

node_deleted datetime DEFAULT NULL,