Skip to content

Instantly share code, notes, and snippets.

View msampathkumar's full-sized avatar

Sampath Kumar msampathkumar

31 followers · 30 following
View GitHub Profile
@msampathkumar
msampathkumar / gist:36ffc7a0d0651e6b137b
Last active August 29, 2015 14:21 — forked from dmitriykuptsov/gist:4f6faadbcd7b3415d202
#!/usr/bin/python
import subprocess
import re
# Get process info
ps = subprocess.Popen(['ps', '-caxm', '-orss,comm'], stdout=subprocess.PIPE).communicate()[0]
vm = subprocess.Popen(['vm_stat'], stdout=subprocess.PIPE).communicate()[0]
# Iterate processes
@msampathkumar
msampathkumar / gist:5cc4655f9a276da5dce3
Created February 27, 2016 07:33
Sublime Text 2 - Useful Shortcuts

Sublime Text 2 – Useful Shortcuts (PC)

Loosely ordered with the commands I use most towards the top. Sublime also offer full documentation.

Editing

Ctrl+C copy current line (if no selection)
Ctrl+X cut current line (if no selection)
Ctrl+⇧+K delete line
Ctrl+↩ insert line after
@msampathkumar
msampathkumar / Spark Dataframe Cheat Sheet.py
Created January 19, 2017 11:53 — forked from evenv/Spark Dataframe Cheat Sheet.py
Cheat sheet for Spark Dataframes (using Python)
# A simple cheat sheet of Spark Dataframe syntax
# Current for Spark 1.6.1
# import statements
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import *
#creating dataframes
df = sqlContext.createDataFrame([(1, 4), (2, 5), (3, 6)], ["A", "B"]) # from manual data
@msampathkumar
msampathkumar / Anime Updates Script.py
Last active April 26, 2017 05:04
Adding Filename Extension
'''To fetch top 5 lastest Episodes & Release Dates from Anime Websites.'''
import requests
from bs4 import BeautifulSoup
from IPython.core.display import display, HTML
SAMURAI_JACK_SITE_URL = "http://www.toonova.net/samurai-jack-season-5"
MY_HERO_ACADEMIA_SITE_URL = "http://watchmha2.com/"
DBZ_SUPER_SITE_URL ="http://watchdbzsuper.tv/"
@msampathkumar
msampathkumar / pg-pong.py
Created May 25, 2017 10:45 — forked from karpathy/pg-pong.py
Training a Neural Network ATARI Pong agent with Policy Gradients from raw pixels
""" Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. """
import numpy as np
import cPickle as pickle
import gym
# hyperparameters
H = 200 # number of hidden layer neurons
batch_size = 10 # every how many episodes to do a param update?
learning_rate = 1e-4
gamma = 0.99 # discount factor for reward
@msampathkumar
msampathkumar / airflow_json_variables.py
Created September 20, 2019 06:38 — forked from kaxil/airflow_json_variables.py
Using Airflow Json Variables
from airflow.models import Variable
# Common (Not-so-nice way)
# 3 DB connections when the file is parsed
var1 = Variable.get("var1")
var2 = Variable.get("var2")
var3 = Variable.get("var3")
# Recommended Way
# Just 1 Database call
@msampathkumar
msampathkumar / glue_job.py
Created October 31, 2019 06:53
Read & Write data from S3 AWS Python Glue
import io
import pandas as pd
def split_s3_name(s3_path):
bucket, key = s3_path.strip('s3://').split('/', 1)
return (bucket, key)
def read_from_s3(file_name):
@msampathkumar
msampathkumar / inspections.txt
Created January 30, 2020 10:05 — forked from pylover/inspections.txt
PyCharm inspections
# Extracted using: $ unzip -p lib/pycharm.jar com/jetbrains/python/PyBundle.properties | grep -B1 INSP.NAME | grep '^#' | sed 's|Inspection||g' | sed -e 's|#\s\{,1\}|# noinspection |'
# noinspection PyPep8
# noinspection PyPep8Naming
# noinspection PyTypeChecker
# noinspection PyAbstractClass
# noinspection PyArgumentEqualDefault
# noinspection PyArgumentList
# noinspection PyAssignmentToLoopOrWithParameter
# noinspection PyAttributeOutsideInit
@msampathkumar
msampathkumar / PySparkCommandsNotes.md
Last active June 30, 2020 07:33
PySpark Commonly Used Commands

Setup a Spark Context

spark = SparkSession.builder.appName('Magic').getOrCreate()

File IO

df = spark.read.format("csv").option("multiline", True).option("delimiter", ",").load(file_path)
@msampathkumar
msampathkumar / Magic Module - Terraform Samples PR : How to use tftools for PR validations?
Created November 30, 2021 09:29
# Testing https://github.com/GoogleCloudPlatform/magic-modules/pull/5448/files
# Step by Step - Instructions
- Step1: Download files
```
wget https://github.com/GoogleCloudPlatform/magic-modules/raw/e8d547a02b95d7a1a650b52c5968404ee57a7b14/mmv1/products/compute/terraform.yaml
wget https://github.com/GoogleCloudPlatform/magic-modules/raw/e8d547a02b95d7a1a650b52c5968404ee57a7b14/mmv1/templates/terraform/examples/external_ssl_proxy_lb_mig_backend_custom_header.tf.erb