Skip to content

Instantly share code, notes, and snippets.

View rohithreddy's full-sized avatar
🐢
working from dreams

Rohith rohithreddy

🐢
working from dreams
View GitHub Profile
@sQu1rr
sQu1rr / kernel-update.sh
Created September 16, 2017 08:32
gentoo kenel update script
View kernel-update.sh
#!/bin/bash
src='/usr/src'
# Get current version
cur_ver="linux-`uname -r`"
# Get new version
ver=`ls $src | grep linux- | sort -V | tail -1`
@eddies
eddies / setup-notes.md
Created July 29, 2016 08:00
Spark 2.0.0 and Hadoop 2.7 with s3a setup
View setup-notes.md

Standalone Spark 2.0.0 with s3

###Tested with:

  • Spark 2.0.0 pre-built for Hadoop 2.7
  • Mac OS X 10.11
  • Python 3.5.2

Goal

Use s3 within pyspark with minimal hassle.

@bearfrieze
bearfrieze / comprehensions.md
Last active December 23, 2023 22:49
Comprehensions in Python the Jedi way
View comprehensions.md

Comprehensions in Python the Jedi way

by Bjørn Friese

Beautiful is better than ugly. Explicit is better than implicit.

-- The Zen of Python

I frequently deal with collections of things in the programs I write. Collections of droids, jedis, planets, lightsabers, starfighters, etc. When programming in Python, these collections of things are usually represented as lists, sets and dictionaries. Oftentimes, what I want to do with collections is to transform them in various ways. Comprehensions is a powerful syntax for doing just that. I use them extensively, and it's one of the things that keep me coming back to Python. Let me show you a few examples of the incredible usefulness of comprehensions.

@ohaz
ohaz / main.py
Created March 8, 2016 09:07
Parse math formulas in python and put parentheses around Mult/Div
View main.py
from __future__ import print_function
import ast
def recurse(node):
if isinstance(node, ast.BinOp):
if isinstance(node.op, ast.Mult) or isinstance(node.op, ast.Div):
print('(', end='')
recurse(node.left)
recurse(node.op)
@porterjamesj
porterjamesj / hello_mesos.py
Last active March 6, 2018 20:43
the tiniest mesos scheduler
View hello_mesos.py
import logging
import uuid
import time
from mesos.interface import Scheduler
from mesos.native import MesosSchedulerDriver
from mesos.interface import mesos_pb2
logging.basicConfig(level=logging.INFO)
@ankurcha
ankurcha / error1.sh
Last active July 1, 2016 12:36
mesos on hadoop
View error1.sh
STARTUP_MSG: java = 1.8.0_25
************************************************************/
14/10/28 06:27:07 INFO mapred.JobTracker: registered UNIX signal handlers for [TERM, HUP, INT]
14/10/28 06:27:08 FATAL mapred.JobTracker: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:163)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:152)
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:2165)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1764)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1757)
@mangecoeur
mangecoeur / a-conda-workon-tool.md
Last active February 9, 2021 14:53
A "virtualenv activate" for Anaconda environments
View a-conda-workon-tool.md

A "virtualenv activate" for Anaconda environments

I've been using the Anaconda python package from continuum.io recently and found it to be a good way to get all the complex compiled libs you need for a scientific python environment. Even better, their conda tool lets you create environments much like virtualenv, but without having to re-compile stuff like numpy, which gets old very very quickly with virtualenv and can be a nightmare to get correctly set up on OSX.

The only thing missing was an easy way to switch environments - their docs suggest running python executables from the install folder, which I find a bit of a pain. Coincidentally I came across this article - Virtualenv's bin/activate is Doing It Wrong - which desribes a simple way to launch a sub-shell with certain environment variables set. Now simple was the key word for me since my bash-fu isn't very strong, but I managed to come up with the script below. Put this in a text file called conda-work