Skip to content

Instantly share code, notes, and snippets.

Samarth Bhargav samarthbhargav

View GitHub Profile
@samarthbhargav
samarthbhargav / CUDA Availability in Torch
Created Sep 24, 2018
One liner to test if cuda is available
View CUDA Availability in Torch
python -c "import torch; print(torch.cuda.is_available())"
@samarthbhargav
samarthbhargav / python_stop_thread.py
Last active Mar 8, 2018
A 'stoppable' thread for Python
View python_stop_thread.py
# source: https://www.safaribooksonline.com/library/view/python-cookbook-2nd/0596007973/ch09s03.html
import threading
class TestThread(threading.Thread):
def _ _init_ _(self, name='TestThread'):
""" constructor, setting initial variables """
self._stopevent = threading.Event( )
self._sleepperiod = 1.0
threading.Thread._ _init_ _(self, name=name)
def run(self):
@samarthbhargav
samarthbhargav / scrape_xkcd.py
Last active Dec 12, 2017
XKCD Scraper - using BeautifulSoup and requests
View scrape_xkcd.py
from bs4 import BeautifulSoup
import requests
import shutil
import glob
from os import listdir
from os.path import isfile, join
def save_image(url, filename):
response = requests.get(url, stream=True)
@samarthbhargav
samarthbhargav / timer.py
Created Dec 18, 2014
a simple profiler for python
View timer.py
# source: http://www.huyng.com/posts/python-performance-analysis/
import time
class Timer(object):
def __init__(self, verbose=False):
self.verbose = verbose
def __enter__(self):
self.start = time.time()
return self
@samarthbhargav
samarthbhargav / powerset.py
Created Dec 4, 2014
Recursive generation of a power set
View powerset.py
# as seen in 6.00.2x, Week 6, Lecture 1, Video 4
def powerset(iterable):
if len(iterable) == 0:
return [[]]
smaller = powerset(iterable[1:])
withElem = []
for s in smaller:
withElem.append( s + [iterable[0]])
@samarthbhargav
samarthbhargav / .bashrc
Last active Mar 3, 2018
bashrc for Hadoop
View .bashrc
# Hadoop Properties
export HADOOP_PREFIX=/usr/local/hadoop # installation location
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export PATH=$PATH:$HADOOP_PREFIX/bin
View yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>
View mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
View hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/hdfs/data/</value>
<description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
View core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost/</value>
<description>NameNode URI</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
You can’t perform that action at this time.